D6754: contrib: proof of concept script to build Mac packages without system python

[mharbison72 (Matt Harbison)]

mharbison72 added a comment.
mharbison72 abandoned this revision.


  D6846  will download and build from 
source without relying on virtualenv stuff.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D6754/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

To: mharbison72, #hg-reviewers
Cc: durin42, indygreg, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D6754: contrib: proof of concept script to build Mac packages without system python

[mharbison72 (Matt Harbison)]

mharbison72 added a comment.


  In D6754#100159 , @durin42 wrote:
  
  > This is a start, but I suspect where we should end up with this is 
downloading and compiling the whole Python distribution, rather than relying on 
a virtualenv to be portable?
  
  Agreed.  I think we will need to build OpenSSL too, and bundle certifi?
  
  The script for building thg makes it look easy, but the result I ended up 
with failed to support zlib and sqlite3.  (We need the latter too, right?)  
Maybe it's an issue with non-framework builds, as that's the only difference 
I'm aware of from what thg build is doing.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D6754/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

To: mharbison72, #hg-reviewers
Cc: durin42, indygreg, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D6754: contrib: proof of concept script to build Mac packages without system python

[durin42 (Augie Fackler)]

durin42 added a comment.


  This is a start, but I suspect where we should end up with this is 
downloading and compiling the whole Python distribution, rather than relying on 
a virtualenv to be portable?

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D6754/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

To: mharbison72, #hg-reviewers
Cc: durin42, indygreg, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D6754: contrib: proof of concept script to build Mac packages without system python

[mharbison72 (Matt Harbison)]

mharbison72 added a comment.
mharbison72 added a subscriber: durin42.


  In D6754#99255 , @indygreg wrote:
  
  > This looks promising.
  > I don't want to tell you not to work on this, but my bold plan is to get 
Mercurial using PyOxidizer and leaning on PyOxidizer for packaging. This will 
require shipping a Python 3 Mercurial, however.
  
  Thanks for the pointers.
  
  I'm broadly aware of the plans to use PyOxidizer for py3, and hope to use it 
for thg too.  I started this with nothing more in mind than making the 
installer support more than 10.14.  But since it links against the system 
python, setting the MACOS_DEPLOYMENT_TARGET (or whatever it's called) errors 
out.  So I tried allowing PYTHON to be overridden in the makefile, so it could 
be pointed to the python.org install and go back to 10.9.  But @durin42 locked 
it into the system python in da1848f07c6a 
, 
and things just kinda snowballed from there.  He mentioned on IRC that a 
pyinstaller dependency wouldn't be great, and also maybe we don't want to be 
completely tied to PyOxidizer.  (I took it as don't preclude other methods, but 
maybe that needs to be clarified.)
  
  > Until PyOxidizer is ready, you may want to look into Beeware's Briefcase 
tool for packaging macOS applications. It also provides its own self-contained 
Python distribution. I'm not sure if it works with Python 2.7, however.
  
  There at least appears to be a package visible in pip27, but the docs talk 
about building a *.app, which isn't going to be useful for us.
  
  > You may also be interested in 
https://github.com/indygreg/python-build-standalone for self-contained Python 
distributions. Only works with Python 3.7+ at the moment now, though. The 
pre-built distributions on GitHub having a working Python install in them. All 
the Python C extensions are statically linked into the `python` executable.
  
  I'll try to find time to play with this in the next week or so.  I'm not sure 
what it would take to support py2, but I'd rather everybody poured their effort 
in elsewhere at this point.  @durin42 mentioned on IRC that it's long past time 
to get off the system python, but the only real deadline is that it won't be in 
10.16 a year from now.  Hopefully the codebase is still supporting py2 then, 
but maybe the py3 support is good enough that we don't need an official py2 
installer?  End of py2 support doesn't seem like a big deal, unless some 
horrible TLS bug is found.
  
  I suppose a reasonable fix for supporting more OS versions is to spin up a 
10.9 VM and build there.  I don't have one right now, but I do have a machine 
that is going to be setup to host various Mac VMs, and I don't have a problem 
building the installer on it if there's interest.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D6754/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

To: mharbison72, #hg-reviewers
Cc: durin42, indygreg, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D6754: contrib: proof of concept script to build Mac packages without system python

[indygreg (Gregory Szorc)]

indygreg added a comment.


  This looks promising.
  
  I don't want to tell you not to work on this, but my bold plan is to get 
Mercurial using PyOxidizer and leaning on PyOxidizer for packaging. This will 
require shipping a Python 3 Mercurial, however.
  
  Until PyOxidizer is ready, you may want to look into Beeware's Briefcase tool 
for packaging macOS applications. It also provides its own self-contained 
Python distribution. I'm not sure if it works with Python 2.7, however.
  
  You may also be interested in 
https://github.com/indygreg/python-build-standalone for self-contained Python 
distributions. Only works with Python 3.7+ at the moment now, though. The 
pre-built distributions on GitHub having a working Python install in them. All 
the Python C extensions are statically linked into the `python` executable.

INLINE COMMENTS

> make_osx.sh:16
> +# TODO: Don't install this into the virtual environment. Also, use 
> docutils-python3
> +python -m pip install docutils
> +

We should be pinning versions and hashes when installing anything from the 
network. For reproducibility and security.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D6754/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

To: mharbison72, #hg-reviewers
Cc: indygreg, mercurial-devel
___
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

D6754: contrib: proof of concept script to build Mac packages without system python

[mharbison72 (Matt Harbison)]

mharbison72 created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  This isn't close to being ready- I'm just looking for ideas before I get too 
far
  off into the weeds.  It will build a *.pkg that does install, and both
  `hg version` and `hg debuginstall` look healthy.  As noted in the script 
though,
  there are still references to /Library/Frameworks/Python.framework (which is 
the
  python.org installed stuff), so it may not work on a fresh system.
  
  At this point, I'm wondering if we should just curl the source for OpenSSL and
  Python, and build the non-framework version with the destination set to the
  proper directory.  The packaging for thg does this (though it probably doesn't
  have to), and it takes about 10 minutes on a Mac Mini.  5 Minutes of that is
  running the tests on the python build, so maybe there's a way to shut that 
off.
  
  As far as alternatives, I started down the path of using virtualenv here, and
  shipping that.  I'm skeptical of the experimental nature of --relocatable, and
  grepping on the tree showed the build path in the *.pyc files.  I have no idea
  if that matters.  I left pip in place so that things like evolve could be
  installed, but it errored out when specifying `--user`.  I didn't look into 
why.
  
  I also looked at pyinstaller.  I couldn't figure out how to convince it to
  bundle the C extensions, and it appears to append its own output when the hg
  script exits with a non zero value.  I briefly looked at bbfreeze and 
cx_freeze,
  but didn't get as far.  I've had enough frustration with py2exe and py2app 
that
  I'm not wanting to struggle with yet another packager.
  
  Ultimately I think we should keep the target for building against system 
python
  (there was some special functionality unlocked with a specifically crafted 
dummy
  cert), and this should possibly be a different target.  (I don't see a way to
  switch between the two by simply overriding $PYTHON the way this is going.)  I
  basically copy/pasted the makefile as a starting point, but moved it into a
  shell script so the environment could be activated.  This either needs to be
  moved to contrib/, or folded back into the makefile.  I kinda lean towards the
  former, and have that called by make.  The readme in the installer will also
  have to be conditionalize/updated, since it mentions relying on system python.
  
  It would be nice to open up the range of supported platforms back to say, 
10.9.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6754

AFFECTED FILES
  make_osx.sh

CHANGE DETAILS

diff --git a/make_osx.sh b/make_osx.sh
new file mode 100755
--- /dev/null
+++ b/make_osx.sh
@@ -0,0 +1,87 @@
+#!/bin/sh
+
+set -e
+
+# Extra args passed in, not version
+OSXVERSIONFLAGS=
+
+# TODO: default MACOSX_DEPLOYMENT_TARGET=10.9
+# TODO: adjust ${OSXVER} accordingly
+
+rm -rf build/mercurial
+python -m virtualenv build/mercurial/usr/local/mercurial
+source build/mercurial/usr/local/mercurial/bin/activate
+
+# TODO: Don't install this into the virtual environment. Also, use 
docutils-python3
+python -m pip install docutils
+
+python setup.py install --optimize=1  \
+ --root=build/mercurial/ --prefix=/usr/local/mercurial
+
+make -C doc all install DESTDIR="`pwd`/build/mercurial/"
+
+# Place a bogon .DS_Store file in the target dir so we can be
+# sure it doesn't get included in the final package.
+touch build/mercurial/.DS_Store
+
+# install zsh completions - this location appears to be
+# searched by default as of macOS Sierra.
+install -d build/mercurial/usr/local/share/zsh/site-functions/
+install -m 0644 contrib/zsh_completion 
build/mercurial/usr/local/share/zsh/site-functions/_hg
+
+# install bash completions - there doesn't appear to be a
+# place that's searched by default for bash, so we'll follow
+# the lead of Apple's git install and just put it in a
+# location of our own.
+install -d build/mercurial/usr/local/hg/contrib/
+install -m 0644 contrib/bash_completion 
build/mercurial/usr/local/hg/contrib/hg-completion.bash
+
+make -C contrib/chg \
+ HGPATH=/usr/local/bin/hg \
+ PYTHON=python \
+ HGEXTDIR=/usr/local/mercurial/lib/python2.7/site-packages/hgext \
+ DESTDIR=../../build/mercurial \
+ PREFIX=/usr/local \
+ clean install
+
+# Keep pip around so that things like evolve can be installed.
+rm -f build/mercurial/usr/local/mercurial/bin/rst* \
+  build/mercurial/usr/local/mercurial/bin/activate* \
+  build/mercurial/usr/local/mercurial/bin/wheel \
+  build/mercurial/usr/local/mercurial/bin/easy_install
+
+# LANG=C' to avoid "sed: RE error: illegal byte sequence"
+LANG=C sed -i '' -e "s|`pwd`/build/mercurial||g" 
build/mercurial/usr/local/mercurial/bin/*
+
+# The virtualenv contains a symlink back to the real python install.  We can
+# either slim this down somehow (to avoid distributing whatever random