Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Thu, May 28, 2015 at 01:20:06PM -0400, Donald Stufft wrote: I think it’s an issue for all platforms, even when there is a system Python that can be used. Here’s why: * Even on Linux systems Python isn’t always a guaranteed thing to be installed, for instance Debian works just fine without any Python installed. Donald, are you a Linux user? If so, which distro? Because in the Linux world that I'm familiar with, this (and the points you make below) are absolutely not an issue. You let the package manager worry about dependencies: yum install myapp # Red Hat based distros apt-get install myapp # Debian based distros will ensure that the right version of Python is installed. In the circles I move in, installing anything which does not go through the package manager is considered to be quite dubious. In order of preference (best to worst): - install from official distro repositories; - install from a third-party repo; - install from source; - find an alternative application; - do without; - install from some binary format (also known as would you like a root kit with that?). [...] * Even if you have Python installed already, is it the right one? What if it’s an ancient RHEL box that has 2.6 or (heaven forbid) 2.4? What if it’s a not ancient box that has Python 2.7 but you want to deploy your app in Python 3? Most recent distros have both a python2 and python3 package, and when building your rpm or deb file, you specify which is your dependency in the normal fashion. * What if you have Python installed already, but it’s been patched by the place you got it from and now the behavior is different than what you expected? Normally you would write for the version of Python provided by the distros you wish to support. In practice that might mean writing hybrid code targetting (say) 2.6 and 2.7, which covers most recent Red Hat and Debian based systems, and anything else, you provide the source code and let the user work it out. I suppose that in principle you could include whatever version of Python you like *in your application*, but that would be considered an unusual thing to do. I believe that LibreOffice and OpenOffice do that, so they can support Python as a scripting language, but they're generally considered (1) a special case because they're cross-platform, and (2) not proper Unix or Linux apps anyway. The point is, in the Linux circles I move in, this idea of single file installation would be about as popular as a police raid at a rave club. Maybe you move in different circles (perhaps more enterprisey?), but I can already imagine the sort of derogatory comments the sys admins I work with would make about this idea. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Thu, May 28, 2015 at 05:38:49PM +0100, Paul Moore wrote: I suspect single file executables just aren't viewed as a desirable solution on Unix. More of an anti-pattern than a pattern. A single file executable means that when you have a security update, instead of patching one library, you have to patch all fifty applications that include that library. Although Donald referred to a 4K binary, which probably means just a stub exe that depends on system-installed .so files, likely including Python (I'm just guessing here). The machine I'm currently on has a 5.6K Python executable: [steve@ando ~]$ ls -lh /usr/bin/python* -rwxr-xr-x 2 root root 5.6K Jan 9 2013 /usr/bin/python lrwxrwxrwx 1 root root6 Jan 22 2013 /usr/bin/python2 - python -rwxr-xr-x 2 root root 5.6K Jan 9 2013 /usr/bin/python2.4 but that doesn't include libpython: [steve@ando ~]$ ls -lh /usr/lib/libpython2.4.so* lrwxrwxrwx 1 root root 19 Jan 22 2013 /usr/lib/libpython2.4.so - libpython2.4.so.1.0 -r-xr-xr-x 1 root root 1.1M Jan 9 2013 /usr/lib/libpython2.4.so.1.0 or the standard library. -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
Hello, On Fri, 29 May 2015 20:53:53 +1000 Steven D'Aprano st...@pearwood.info wrote: [ insightful statistics skipped ] I think there are some exciting and interesting languages coming up: Swift, Julia, Go, Rust and others. Only those? Every one in a dozen university student comes up with an exciting, interesting language - it has always been like that. Further development and maintenance is what levels it below the common crowd. Why are we threatened by this? Because at least some of them are backed by media companies, who use them as leverage for their advertisement and PR campaigns. Obviously, media companies already have great advertisement influence, and can fool anybody's head with their tricks. Python makes a wonderful glue language. It would be great for Python to glue to more than just C and Fortran code. For scientific users, imagine being able to call Julia code from Python, and vice versa. There always were things like integration of Python and Lua, etc. Did somebody use them? No, they're of interest only to their authors. Instead of thinking of Go as an opponent to beat, wouldn't it be great to be able to write extensions in a modern language like Go, Rust or D instead of creaky old C with all its safety issues? Because very few people use Go, Rust, or D at all. And then they're likely concentrated in a small niche. Going 2-level, using experimental language L in an area A, where A is arbitrary/random, has almost zero interest for majority of population. Let's wait till Rust becomes real rust and talk again. -- Steve -- Best regards, Paul mailto:pmis...@gmail.com ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 29 May 2015 at 09:36, Steven D'Aprano st...@pearwood.info wrote: The point is, in the Linux circles I move in, this idea of single file installation would be about as popular as a police raid at a rave club. Maybe you move in different circles (perhaps more enterprisey?), but I can already imagine the sort of derogatory comments the sys admins I work with would make about this idea. In my environments, we frequently have ancient versions of RHEL installed, sometimes with no Python at all (IIRC) or nothing better than 2.4. The sysadmins won't install newer versions, as Python isn't formally needed, but we'd happily use it for adhoc admin-style scripts (the alternative is typically shell scripts or nothing). It's not precisely acceptable, but being able to create a single-file small executable in the support user's home directory made from an admin script would be immensely useful. It's hardly a core use case, and we generally just live with shell scripts, but such environments *do* exist :-( Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Fri, May 29, 2015 at 07:08:43AM +1000, Nick Coghlan wrote: On 29 May 2015 05:25, Chris Barker chris.bar...@noaa.gov wrote: OK, I'm really confused here: 1) what the heck is so special about go all of a sudden? People have been writing and deploying single file executables built with C and ++, and whatever else? forever. (and indeed, it was a big sticking point for me when I introduced python in my organization) For scientific Python folks, the equivalent conversations I have are about Julia. If you're not used to thinking of Python's competitive position as best orchestration language, solid competitor in any given niche, then the rise of niche specific competitors like Go Julia can feel terrifying, as the relatively narrow user base changes the trade-offs you can make in the language ecosystem design to better optimise them for that purpose. We've been there before, with the Ruby is the Python-killer FUD of a few years ago. If Go is different and does overtake Python, I think it will be due to its privileged position on Android. Personally, I don't pay a lot of attention to language popularity statistics. Who cares whether Python is used by 8% of projects or 10% of projects? Either way, it's huge. But from time to time, it might be useful to look at a few different measurements of popularity. According to CodeEval, Python is still *by far* the most popular language, at 31.2% (second place is below 20%), with Go at #9 with 2.3%. According to Redmonk, Python is stable at #4, while Go has jumped from #21 to #17 in six months. LangPop gives various different measures of popularity, and according to the overall summary, Python is at #6, but Go doesn't appear to be a language they look at. TIOBE has Python moving up two places to #6, and Go (which was the 2009 Hall Of Fame winner) doesn't even appear in the top 100. http://blog.codeeval.com/codeevalblog/2015 http://redmonk.com/sogrady/2015/01/14/language-rankings-1-15/ http://langpop.com/ http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html I think there are some exciting and interesting languages coming up: Swift, Julia, Go, Rust and others. Why are we threatened by this? Python makes a wonderful glue language. It would be great for Python to glue to more than just C and Fortran code. For scientific users, imagine being able to call Julia code from Python, and vice versa. Instead of thinking of Go as an opponent to beat, wouldn't it be great to be able to write extensions in a modern language like Go, Rust or D instead of creaky old C with all its safety issues? -- Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
Paul Moore writes: In my environments, we frequently have ancient versions of RHEL installed, sometimes with no Python at all (IIRC) or nothing better than 2.4. That's pretty advanced as older Red Hat systems go. You're lucky it isn't 1.5.2! Getting serious, Red Hat systems have included Python for about as long as the youngest core committers have been alive, and I'm sure that goes back before they called it Enterprise Linux. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 28 May 2015 at 22:09, Glenn Linderman v+pyt...@g.nevcal.com wrote:
This would be something I could use and benefit from immediately upon it
being available, so I laud your idea, and hope you have a successful
implementation, and look forward to using it. It would largely replace the
need for the py.exe launcher for some classes of applications.
The following proof-of-concept works as is (based on my pretty minimal
testing), and only uses the limited API, so it should work with any
version of Python 3 (I've not tested it with Python 2, but I think the
only new API is PySys_SetArgvEx, which could be replaced with
PySys_SetArgv at a pinch). Excuse the dreadful coding style and lack
of error handling, I hacked it up in about an hour :-)
(Actually, I just tried building on Python 2 - guess what - Unicode
:-) SetProgramName and SetArgvEx won't take Unicode values. The easy
fix is just not to use Unicode, the hard one is to do the encoding
dance, but I'm not going to bother...).
#define UNICODE
#define _UNICODE
#include Python.h
#include windows.h
int
main()
{
TCHAR program[MAX_PATH];
LPWSTR *argv;
int argc;
PyObject *runpy;
PyObject *ret;
argv = CommandLineToArgvW(GetCommandLineW(), argc);
GetModuleFileName(NULL, program, MAX_PATH);
Py_SetProgramName(program); /* optional but recommended */
Py_Initialize();
PySys_SetArgvEx(argc, argv, 0);
runpy = PyImport_ImportModule(runpy);
if (!runpy) PyErr_Print();
ret = PyObject_CallMethod(runpy, run_path, u, program);
if (!ret) PyErr_Print();
Py_Finalize();
return 0;
}
One mildly annoying thing is that python3.dll is only installed in
python install dir\DLLs, which typically isn't on PATH. So actually
using the limited API from your own application fails by default.
Fixing that's mostly a user admin issue, though (and you can just link
to the full API and avoid the whole problem).
Of course, per other disccusions, this doesn't solve the problem for:
A) machine without Python installed
B) programs that need binary extensions
Other discussions have suggested:
3) The stub could offer to download and install Python
A corollary:
4) The stub could offer to download and install the needed binary extensions
as well as Python. This would require the installation uniformity of
something like pip, so perhaps would be restricted to extensions available
via pip. And it would be much enhanced by some technique where the zipapp
would contain metadata readable by the stub, that would declare the list of
binary extensions required. Or, of course, it could even declare non-binary
extension that are not packaged with the zipapp, if the process is smooth,
the modules available via pip, etc., as a tradeoff.
I'm pretty strongly against downloading interpreters or extensions.
Apart from the pretty huge added complexity, as a user I'm not sure
I'd trust a supposedly simple application I'd received if it started
asking to download stuff unexpectedly...
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On May 29, 2015 at 4:37:37 AM, Steven D'Aprano (st...@pearwood.info) wrote: On Thu, May 28, 2015 at 01:20:06PM -0400, Donald Stufft wrote: I think it’s an issue for all platforms, even when there is a system Python that can be used. Here’s why: * Even on Linux systems Python isn’t always a guaranteed thing to be installed, for instance Debian works just fine without any Python installed. Donald, are you a Linux user? If so, which distro? Because in the Linux world that I'm familiar with, this (and the points you make below) are absolutely not an issue. You let the package manager worry about dependencies: yum install myapp # Red Hat based distros apt-get install myapp # Debian based distros will ensure that the right version of Python is installed. In the circles I move in, installing anything which does not go through the package manager is considered to be quite dubious. In order of preference (best to worst): - install from official distro repositories; - install from a third-party repo; - install from source; - find an alternative application; - do without; - install from some binary format (also known as would you like a root kit with that?). [...] * Even if you have Python installed already, is it the right one? What if it’s an ancient RHEL box that has 2.6 or (heaven forbid) 2.4? What if it’s a not ancient box that has Python 2.7 but you want to deploy your app in Python 3? Most recent distros have both a python2 and python3 package, and when building your rpm or deb file, you specify which is your dependency in the normal fashion. * What if you have Python installed already, but it’s been patched by the place you got it from and now the behavior is different than what you expected? Normally you would write for the version of Python provided by the distros you wish to support. In practice that might mean writing hybrid code targetting (say) 2.6 and 2.7, which covers most recent Red Hat and Debian based systems, and anything else, you provide the source code and let the user work it out. I suppose that in principle you could include whatever version of Python you like *in your application*, but that would be considered an unusual thing to do. I believe that LibreOffice and OpenOffice do that, so they can support Python as a scripting language, but they're generally considered (1) a special case because they're cross-platform, and (2) not proper Unix or Linux apps anyway. The point is, in the Linux circles I move in, this idea of single file installation would be about as popular as a police raid at a rave club. Maybe you move in different circles (perhaps more enterprisey?), but I can already imagine the sort of derogatory comments the sys admins I work with would make about this idea. I use Linux for servers yes, I don't stick with a single Distribution and right now I manage services that are running on CentOS, Ubuntu, Debian, Alpine, and FreeBSD (not a Linux, but w/e). Here's the thing though, when you make software that is designed for other people to consume you have two choices. Either you have to try to anticipate every single environment that they might possibly run it in and what is available there so that your software either runs there or you can provide instructions on how to run it there, or you need to depend on as little from the OS as possible. An example of a product that does this is Chef, they install their own Ruby and everything but libc into /opt/chef to completely isolate themselves from the host system. I’m told this made things *much* easier for them as they don't really have to worry at all about what's available on the host system, Chef pretty much just works. Another example is one that I personally worked on recently, where the company I worked for wanted to distribute a CLI to our customers which would just work that they could use to interact with the service we provided. We were writing this client in Python and it was very painful to satisfy the requirement that it work without Python installed as a single file executable. That was a hard requirement as anything else would be more difficult to distribute to end users and be more likely to break. The entire time I was working on that particular piece I was feeling like I should be suggesting to my manager that we throw away the Python code and just write it in Go. As folks may or may not know, I'm heavily involved in pip which is probably one of the most widely used CLIs written in Python. A single file executable won't help pip, however through my experience there I can tell you that a significant portion of our issues come from random weird differences in how a particular person's Python environment is setup. If someone is distributing a CLI app written in Python, being able to remain independent from the OS eliminates a whole class of problems. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Fri, 29 May 2015 18:36:02 +1000 Steven D'Aprano st...@pearwood.info wrote: The point is, in the Linux circles I move in, this idea of single file installation would be about as popular as a police raid at a rave club. This is frankly not true. There are many programs (e.g. games) which are not available as distribution packages, and can't rely on the user's distribution to provide the right versions of the required pieces of infrastructure. Those programs have to bundle the whole stack independently. Perhaps in *your* Linux circle you never use such programs, but they do exist and are part of the ecosystem. Regards Antoine. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 5/28/2015 4:29 PM, Paul Moore wrote: On 28 May 2015 at 20:47, Brett Cannon br...@python.org wrote: I think it's to have a single tool to do it for any platform, not to have the technical nuts and bolts be the same necessarily. I think it's also to figure out if there is anything the interpreter and/or stdlib can do to facilitate this. Precisely. At the moment, the story seems to be if you're on Windows, use py2exe, if you're on OSX use py2app, or on Unix, ..., or... What would be a compelling story is to build your app into a single file executable, do python -m build myapp. The machinery behind the build can be as different as necessary - but being able to use the same command on every platform is the goal. The python-based ren'py visual novel development system has something like this When one is ready to publish, there is an easy option to build single-file downloadable redistributables for any or all of Windows, Linux, and Mac. I know it works as far as it goes because I help my wife use the system not for a novel, but a photo-based tutorial. After testing the resulting files with the help of others on linux and mac systems (we developed on Windows), she put the files up on one of her university pages. As far as I know, the build code should be Python, if anyone want to look at it. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 29 May 2015 at 23:15, Glenn Linderman v+pyt...@g.nevcal.com wrote: I don't presently see any C:\Python34\DLLs or C:\Python34 on my path, but I didn't ask the installer to put it there either. So I'm guessing your option 1 assumes asking the Python installer to put it there? Not automatically but on request, I think? In my c:\Python34\DLLs, I don't see a python34.dll, only python3.dll... so I'm somewhat unclear on your simplified explanation. I'm definitely batting zero today :-( OK, let's try to be clear. I typically do all users installs. The for me only install is slightly different, and the new install for 3.5 may be different again. But what I see is: 1. C:\Python34\DLLs containins python3.dll, which is *never* on PATH (and doesn't need to be for normal use). Anything that wants to use python3.dll needs that directory manually adding to PATH. 2. python34.dll is in C:\Windows\System32. This is always available to all processes, as it's in the Windows system directory. If you say add Python to my PATH you get C:\Python34 added to PATH. For a user install, I believe python34.dll may be in there rather than in C:\Windows\system32, so technically, for an app that uses python34.dll to work, you need *either* an admin install, *or* to have done add Python to PATH. I hope that made sense. Sorry for my garbled previous version. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 5/29/2015 2:45 PM, Paul Moore wrote: On 29 May 2015 at 21:49, Glenn Linderman v+pyt...@g.nevcal.com wrote: That looks interesting, I wonder what compilation environment it would need? I don't think I've even installed a C compiler on my last couple boxes, and the only version of a C compiler I have is, umm... M$VC++6.0, since I've moved to using Python for anything a 5 line batch file can't do... One mildly annoying thing is that python3.dll is only installed in python install dir\DLLs, which typically isn't on PATH. Ah, linking so I guess if I figured out how to create this binary, it would contain a reference to python3.dll that would attempt to be resolved via the PATH, from what you say, and typically fail, due to PATH seldom containing python3.dll. The python launcher gets around that by (1) being installed in %windir%, and going and finding the appropriate Python (per its own configuration file, and command line parameters), and setting up the path to that Python, which, when executed, knows its own directory structure and can thus find its own python3.dll. The launcher, of course, adds an extra layer of process between the shell and the program, because it launches the real Python executable. So actually using the limited API from your own application fails by default. Fixing that's mostly a user admin issue, though (and you can just link to the full API and avoid the whole problem). Do I understand correctly that the user admin issue means add the appropriate python install dir\DLLs to the PATH? What I don't understand here is how linking to the full API avoids the problem... it must put more python library code into the stub executable? Enough to know how to search the registry to find the python install dir for the version of Python from which the full API was obtained? Or something else? Sorry, I assumed more Windows/C knowledge than you have. It is mostly the C/Python interface knowledge that I lack... although my Windows/C knowledge is getting rusty. I'll work on this and produce proper binaries in due course, so you can always wait for them. But you can build the stub with pretty much anything, I suspect - I managed with MSVC 2010 and mingw. I'll add some build docs and get it on github. Using mingw gcc -Wall -O2 -o stub.exe stub.c -I python home\Include C:\Windows\system32\python34.dll strip -s stub.exe Using MSVC cl /Festub.exe /O2 stub.c /Ipython home\Include python home\libs\python34.lib Github sounds good. Binaries sound good. I would have to download the free MSVC10 or Ming and install and learn to use them, etc., to make progress... probably doable, but (1) I'm surviving at the moment with the launcher + zipapp, but it'd be nice for folks I code for to have .exe things, and (2) I'm backlogged in my other projects which don't need me to download a C compiler to make progress. Regarding the DLLs, yes the user admin issue is adding the right directory to PATH. I used the phrase admin issue as it's the aspect that's likely to be far harder than any of the technical issues :-) The reason using the full API helps is that the full API references python34.dll rather than python3.dll. And the Python installer puts python34.dll on PATH automatically, as it's what the python command uses. (For the people with more Windows knowledge, I know this is a simplification, but it's close enough for now). So there are two options with the code I posted. 1. Build an exe that uses a specific version of Python, but which will just work in basically the same way that the python command works. 2. Build an exe that works with any version of Python, but requires some setup from the user. Either approach requires that the Python DLL is on PATH, but that's far more likely with the version-specific one, just because of how the installer does things. I don't presently see any C:\Python34\DLLs or C:\Python34 on my path, but I didn't ask the installer to put it there either. So I'm guessing your option 1 assumes asking the Python installer to put it there? Not automatically but on request, I think? In my c:\Python34\DLLs, I don't see a python34.dll, only python3.dll... so I'm somewhat unclear on your simplified explanation. With extra code, the stub could locate an appropriate Python DLL dynamically, which would simplify usage at the cost of a bit of fiddly code in the stub. This might be a useful addition to the zipapp module for Python 3.6. Indeed. Especially with extra fiddly code if you or someone on Github has the time, it could be very useful for the zipapp module. Paul PS Current launchers (py.exe, the entry point launchers from pip/setuptools, etc) tend to spawn the actual python program in a subprocess. I believe there are *technically* some differences in the runtime environment when you use an embedding approach like this, but I don't know what they are, and they probably won't affect 99.9% of users. Lack of support for
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 5/29/2015 3:28 PM, Paul Moore wrote: On 29 May 2015 at 23:15, Glenn Linderman v+pyt...@g.nevcal.com wrote: I don't presently see any C:\Python34\DLLs or C:\Python34 on my path, but I didn't ask the installer to put it there either. So I'm guessing your option 1 assumes asking the Python installer to put it there? Not automatically but on request, I think? In my c:\Python34\DLLs, I don't see a python34.dll, only python3.dll... so I'm somewhat unclear on your simplified explanation. I'm definitely batting zero today :-( OK, let's try to be clear. I typically do all users installs. The for me only install is slightly different, and the new install for 3.5 may be different again. But what I see is: 1. C:\Python34\DLLs containins python3.dll, which is *never* on PATH (and doesn't need to be for normal use). Anything that wants to use python3.dll needs that directory manually adding to PATH. 2. python34.dll is in C:\Windows\System32. This is always available to all processes, as it's in the Windows system directory. If you say add Python to my PATH you get C:\Python34 added to PATH. For a user install, I believe python34.dll may be in there rather than in C:\Windows\system32, so technically, for an app that uses python34.dll to work, you need *either* an admin install, *or* to have done add Python to PATH. Interesting. In C:\, I have directories Python27, Python32, Python33, Python34. I can't be 100% sure how I answered the install questions, even Python34 was a couple months ago. In C:\Windows\System32, I have python27.dll, python32.dll, python33.dll, pythoncom32.dll, pythoncom33.dll, pythoncomloader32.dll, and pythoncomloader33.dll. But not python34.dll! I finally found that in c:\Windows\SysWOW64, which I guess means that I accidentally installed a 32-bit Python 3.4. Or maybe I had a reason at the time. But does that add another dimension to the picture for the stub? I hope that made sense. Sorry for my garbled previous version. This matches reality better... but whether it makes sense or not is another question. On 5/29/2015 3:33 PM, Steve Dower wrote: Paul Moore wrote: One mildly annoying thing is that python3.dll is only installed in python install dir\DLLs, which typically isn't on PATH. So actually using the limited API from your own application fails by default. Fixing that's mostly a user admin issue, though (and you can just link to the full API and avoid the whole problem). I didn't even notice that 3.4 (and earlier?) were doing that, so I changed/fixed it by accident :) Indeed, and earlier. It apparently started in 3.2 with the definition of the Stable ABI. Python 3.5 installs python3.dll alongside python35.dll, so it'll go into the user's Python directory by default or into the system directory for an all-users installation. The embeddable distro includes python3.dll alongside the other DLLs as well. Cheers, Steve This makes more sense... but will it cause problems with something? It seems to me like it was a bug to put it in the python install dir rather than %windir%\System32 back in Python 3.2 days when it was invented. What good is it to have a stable ABI that is hidden away where it cannot (easily) be used, and the more dynamic API is easier to get to? Sadly, PEP 384 is silent on the location of python3.dll. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
Paul Moore wrote: One mildly annoying thing is that python3.dll is only installed in python install dir\DLLs, which typically isn't on PATH. So actually using the limited API from your own application fails by default. Fixing that's mostly a user admin issue, though (and you can just link to the full API and avoid the whole problem). I didn't even notice that 3.4 (and earlier?) were doing that, so I changed/fixed it by accident :) Python 3.5 installs python3.dll alongside python35.dll, so it'll go into the user's Python directory by default or into the system directory for an all-users installation. The embeddable distro includes python3.dll alongside the other DLLs as well. Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
I did that once; it wasn't worth it. It was no smaller than what PyInstaller would output and required manually adding in the required modules that weren't in the stdlib, along with any extra DLLs (e.g. the Qt DLLs). On Fri, May 29, 2015 at 4:45 PM, Paul Moore p.f.mo...@gmail.com wrote: On 29 May 2015 at 21:49, Glenn Linderman v+pyt...@g.nevcal.com wrote: That looks interesting, I wonder what compilation environment it would need? I don't think I've even installed a C compiler on my last couple boxes, and the only version of a C compiler I have is, umm... M$VC++6.0, since I've moved to using Python for anything a 5 line batch file can't do... One mildly annoying thing is that python3.dll is only installed in python install dir\DLLs, which typically isn't on PATH. Ah, linking so I guess if I figured out how to create this binary, it would contain a reference to python3.dll that would attempt to be resolved via the PATH, from what you say, and typically fail, due to PATH seldom containing python3.dll. The python launcher gets around that by (1) being installed in %windir%, and going and finding the appropriate Python (per its own configuration file, and command line parameters), and setting up the path to that Python, which, when executed, knows its own directory structure and can thus find its own python3.dll. The launcher, of course, adds an extra layer of process between the shell and the program, because it launches the real Python executable. So actually using the limited API from your own application fails by default. Fixing that's mostly a user admin issue, though (and you can just link to the full API and avoid the whole problem). Do I understand correctly that the user admin issue means add the appropriate python install dir\DLLs to the PATH? What I don't understand here is how linking to the full API avoids the problem... it must put more python library code into the stub executable? Enough to know how to search the registry to find the python install dir for the version of Python from which the full API was obtained? Or something else? Sorry, I assumed more Windows/C knowledge than you have. I'll work on this and produce proper binaries in due course, so you can always wait for them. But you can build the stub with pretty much anything, I suspect - I managed with MSVC 2010 and mingw. I'll add some build docs and get it on github. Using mingw gcc -Wall -O2 -o stub.exe stub.c -I python home\Include C:\Windows\system32\python34.dll strip -s stub.exe Using MSVC cl /Festub.exe /O2 stub.c /Ipython home\Include python home\libs\python34.lib Regarding the DLLs, yes the user admin issue is adding the right directory to PATH. I used the phrase admin issue as it's the aspect that's likely to be far harder than any of the technical issues :-) The reason using the full API helps is that the full API references python34.dll rather than python3.dll. And the Python installer puts python34.dll on PATH automatically, as it's what the python command uses. (For the people with more Windows knowledge, I know this is a simplification, but it's close enough for now). So there are two options with the code I posted. 1. Build an exe that uses a specific version of Python, but which will just work in basically the same way that the python command works. 2. Build an exe that works with any version of Python, but requires some setup from the user. Either approach requires that the Python DLL is on PATH, but that's far more likely with the version-specific one, just because of how the installer does things. With extra code, the stub could locate an appropriate Python DLL dynamically, which would simplify usage at the cost of a bit of fiddly code in the stub. This might be a useful addition to the zipapp module for Python 3.6. Paul PS Current launchers (py.exe, the entry point launchers from pip/setuptools, etc) tend to spawn the actual python program in a subprocess. I believe there are *technically* some differences in the runtime environment when you use an embedding approach like this, but I don't know what they are, and they probably won't affect 99.9% of users. Lack of support for binary extensions is likely to be way more significant. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 5/29/2015 3:33 AM, Paul Moore wrote:
On 28 May 2015 at 22:09, Glenn Linderman v+pyt...@g.nevcal.com wrote:
This would be something I could use and benefit from immediately upon it
being available, so I laud your idea, and hope you have a successful
implementation, and look forward to using it. It would largely replace the
need for the py.exe launcher for some classes of applications.
The following proof-of-concept works as is (based on my pretty minimal
testing), and only uses the limited API, so it should work with any
version of Python 3 (I've not tested it with Python 2, but I think the
only new API is PySys_SetArgvEx, which could be replaced with
PySys_SetArgv at a pinch). Excuse the dreadful coding style and lack
of error handling, I hacked it up in about an hour :-)
I have no particular interest in Python 2, having started with Python 3,
and only used Python 2 for cases where dependent packages required it,
and I've now reached the nirvana of all my dependency packages being
ported to Python 3, although I have yet to port/test one remaining
application to prove that. So I only mentioned Python 2 because it still
could be useful for other people :)
(Actually, I just tried building on Python 2 - guess what - Unicode
:-) SetProgramName and SetArgvEx won't take Unicode values. The easy
fix is just not to use Unicode, the hard one is to do the encoding
dance, but I'm not going to bother...).
One approach would be to support Unicode arguments only for Python 3,
but that would really be only paying lip service to Python 2 support.
Another approach might be to not #define UNICODE for the Python 2
version, and use the 8-bit Windows APIs, allowing Windows and the C
runtime to do the encoding dance for you? Although I'm not sure what
Python 2 requires in that respect.
#define UNICODE
#define _UNICODE
#include Python.h
#include windows.h
int
main()
{
TCHAR program[MAX_PATH];
LPWSTR *argv;
int argc;
PyObject *runpy;
PyObject *ret;
argv = CommandLineToArgvW(GetCommandLineW(), argc);
GetModuleFileName(NULL, program, MAX_PATH);
Py_SetProgramName(program); /* optional but recommended */
Py_Initialize();
PySys_SetArgvEx(argc, argv, 0);
runpy = PyImport_ImportModule(runpy);
if (!runpy) PyErr_Print();
ret = PyObject_CallMethod(runpy, run_path, u, program);
if (!ret) PyErr_Print();
Py_Finalize();
return 0;
}
That looks interesting, I wonder what compilation environment it would
need? I don't think I've even installed a C compiler on my last couple
boxes, and the only version of a C compiler I have is, umm... M$VC++6.0,
since I've moved to using Python for anything a 5 line batch file can't
do...
One mildly annoying thing is that python3.dll is only installed in
python install dir\DLLs, which typically isn't on PATH.
Ah, linking so I guess if I figured out how to create this binary,
it would contain a reference to python3.dll that would attempt to be
resolved via the PATH, from what you say, and typically fail, due to
PATH seldom containing python3.dll. The python launcher gets around
that by (1) being installed in %windir%, and going and finding the
appropriate Python (per its own configuration file, and command line
parameters), and setting up the path to that Python, which, when
executed, knows its own directory structure and can thus find its own
python3.dll.
The launcher, of course, adds an extra layer of process between the
shell and the program, because it launches the real Python executable.
So actually using the limited API from your own application fails by default.
Fixing that's mostly a user admin issue, though (and you can just link
to the full API and avoid the whole problem).
Do I understand correctly that the user admin issue means add the
appropriate python install dir\DLLs to the PATH?
What I don't understand here is how linking to the full API avoids the
problem... it must put more python library code into the stub
executable? Enough to know how to search the registry to find the
python install dir for the version of Python from which the full API
was obtained? Or something else?
Are there other alternatives? Assuming that the reference to the
missing DLL is not required until the point at which a symbol from it is
first referenced, so that the stub would have some ability to do
something before that first call, maybe...
1. The stub above could be enhanced to contained a hard coded
directory that it adds to the PATH itself?
2. The stub above could be enhanced to define that its first parameter
is the python install dir, and tweak its PATH.
3. These days, the Python installer does offer to optionally add itself
to the PATH. Is that sufficient to make the stub work?
4. The launcher could be used, assuming it is installed, but then you
don't need a stub, and you get the extra process layer.
5. stubpy.cmd could be created, a four line
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 29 May 2015 at 21:49, Glenn Linderman v+pyt...@g.nevcal.com wrote: That looks interesting, I wonder what compilation environment it would need? I don't think I've even installed a C compiler on my last couple boxes, and the only version of a C compiler I have is, umm... M$VC++6.0, since I've moved to using Python for anything a 5 line batch file can't do... One mildly annoying thing is that python3.dll is only installed in python install dir\DLLs, which typically isn't on PATH. Ah, linking so I guess if I figured out how to create this binary, it would contain a reference to python3.dll that would attempt to be resolved via the PATH, from what you say, and typically fail, due to PATH seldom containing python3.dll. The python launcher gets around that by (1) being installed in %windir%, and going and finding the appropriate Python (per its own configuration file, and command line parameters), and setting up the path to that Python, which, when executed, knows its own directory structure and can thus find its own python3.dll. The launcher, of course, adds an extra layer of process between the shell and the program, because it launches the real Python executable. So actually using the limited API from your own application fails by default. Fixing that's mostly a user admin issue, though (and you can just link to the full API and avoid the whole problem). Do I understand correctly that the user admin issue means add the appropriate python install dir\DLLs to the PATH? What I don't understand here is how linking to the full API avoids the problem... it must put more python library code into the stub executable? Enough to know how to search the registry to find the python install dir for the version of Python from which the full API was obtained? Or something else? Sorry, I assumed more Windows/C knowledge than you have. I'll work on this and produce proper binaries in due course, so you can always wait for them. But you can build the stub with pretty much anything, I suspect - I managed with MSVC 2010 and mingw. I'll add some build docs and get it on github. Using mingw gcc -Wall -O2 -o stub.exe stub.c -I python home\Include C:\Windows\system32\python34.dll strip -s stub.exe Using MSVC cl /Festub.exe /O2 stub.c /Ipython home\Include python home\libs\python34.lib Regarding the DLLs, yes the user admin issue is adding the right directory to PATH. I used the phrase admin issue as it's the aspect that's likely to be far harder than any of the technical issues :-) The reason using the full API helps is that the full API references python34.dll rather than python3.dll. And the Python installer puts python34.dll on PATH automatically, as it's what the python command uses. (For the people with more Windows knowledge, I know this is a simplification, but it's close enough for now). So there are two options with the code I posted. 1. Build an exe that uses a specific version of Python, but which will just work in basically the same way that the python command works. 2. Build an exe that works with any version of Python, but requires some setup from the user. Either approach requires that the Python DLL is on PATH, but that's far more likely with the version-specific one, just because of how the installer does things. With extra code, the stub could locate an appropriate Python DLL dynamically, which would simplify usage at the cost of a bit of fiddly code in the stub. This might be a useful addition to the zipapp module for Python 3.6. Paul PS Current launchers (py.exe, the entry point launchers from pip/setuptools, etc) tend to spawn the actual python program in a subprocess. I believe there are *technically* some differences in the runtime environment when you use an embedding approach like this, but I don't know what they are, and they probably won't affect 99.9% of users. Lack of support for binary extensions is likely to be way more significant. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 29 May 2015 05:25, Chris Barker chris.bar...@noaa.gov wrote: OK, I'm really confused here: 1) what the heck is so special about go all of a sudden? People have been writing and deploying single file executables built with C and ++, and whatever else? forever. (and indeed, it was a big sticking point for me when I introduced python in my organization) For scientific Python folks, the equivalent conversations I have are about Julia. If you're not used to thinking of Python's competitive position as best orchestration language, solid competitor in any given niche, then the rise of niche specific competitors like Go Julia can feel terrifying, as the relatively narrow user base changes the trade-offs you can make in the language ecosystem design to better optimise them for that purpose. We don't need to debate the accuracy of that perception of risk, though. If it motivates folks to invest time energy into providing one-obvious-way to do cross-platform single-file distribution, lower barriers to adoption for PyPy, and work on a Rust-style memory ownership based model for concurrent execution of subinterpreters across multiple cores, then the community wins regardless :) Cheers, Nick. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 5/28/2015 12:26 PM, Paul Moore wrote: On 28 May 2015 at 19:22, Chris Angelico ros...@gmail.com wrote: Unfortunately (and believe me, I've been down this road many times) on Windows *only* the exe format is a first-class executable. Executable scripts and shebangs are very useful, but there are always corner cases where they don't work *quite* like an exe. On Windows, you have to be prepared to ship an exe if you want to compete with languages that generate exes. I'm aware of that. When I said a tiny stub, I was thinking in terms of a small executable. The idea is that its sole purpose is to locate Python someplace else, and chain to it; that has to be actual executable code, complete with the 512-byte MZ header and everything, to ensure compatibility. But it should be able to be small, tight, and easy to verify correctness of, so there aren't (in theory!) security exploits in the header itself. OK, cool. I'm sort of working on that as a bit of a side project - a tiny stub exe that you can prepend to a Python zipapp which runs it via the standard embedding APIs. It's little more than an idea at the moment, but I don't think it'll be too hard to implement... Paul, I've been using zipapps for some years now, and it would be really cool to have what I think you are talking about here: 1) Build zipapp as normal. It likely depends on some minimal Python version. 2) Prepend stub .exe (Windows) or !# line (Unix) that declares a version of Python to actually use. This can be as broad as Python 2 or Python 3 (not sure how to specify that either works, especially on Windows), or more detailed/restrictive by specifying 2.n or 3.n. On Windows, it would find the newest qualifying installation and run it, and Unix, the symlinks are set up to do that already, is my understanding, so the proper !# line does that for you. This would be something I could use and benefit from immediately upon it being available, so I laud your idea, and hope you have a successful implementation, and look forward to using it. It would largely replace the need for the py.exe launcher for some classes of applications. Of course, per other disccusions, this doesn't solve the problem for: A) machine without Python installed B) programs that need binary extensions Other discussions have suggested: 3) The stub could offer to download and install Python A corollary: 4) The stub could offer to download and install the needed binary extensions as well as Python. This would require the installation uniformity of something like pip, so perhaps would be restricted to extensions available via pip. And it would be much enhanced by some technique where the zipapp would contain metadata readable by the stub, that would declare the list of binary extensions required. Or, of course, it could even declare non-binary extension that are not packaged with the zipapp, if the process is smooth, the modules available via pip, etc., as a tradeoff. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 28 May 2015 at 20:47, Brett Cannon br...@python.org wrote: I think it's to have a single tool to do it for any platform, not to have the technical nuts and bolts be the same necessarily. I think it's also to figure out if there is anything the interpreter and/or stdlib can do to facilitate this. Precisely. At the moment, the story seems to be if you're on Windows, use py2exe, if you're on OSX use py2app, or on Unix, ..., or... What would be a compelling story is to build your app into a single file executable, do python -m build myapp. The machinery behind the build can be as different as necessary - but being able to use the same command on every platform is the goal. zipapp is a start down that direction, but there's a *lot* more to be done before we have a story good enough to address the growing trend towards wanting a strong single-file deployment solution. Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Thu, May 28, 2015 at 10:32 AM, Ryan Gonzalez rym...@gmail.com wrote: py2exe tends to invoke DLL hell if you have various versions of VS or Office or both installed. Because Windows. uh, yes -- Windows applications invoke dll hell..nothign to be done about that! -Chris On May 28, 2015 11:23:57 AM CDT, Chris Barker chris.bar...@noaa.gov wrote: I'm confused: Doesn't py2exe (optionally) create a single file executable? And py2app on the Mac creates an application bundle, but that is more-or-less the equivalent on OS-X (you may not even be able to have a single file executable that can access the Window Manager, for instance) Depending on what extra packages you need, py2exe's single file doesn't always work, but last I tried, it worked for a fair bit (I think all of the stdlib). I don't know what PyInstaller or others create. And I have no idea if there is a linux option -- but it seems like the standard of practice for an application for linux is a bunch of files scattered over the system anyway :-) Yes, the resulting exe is pretty big, but it does try to include only those modules and packages that are used, and that kind of optimization could be improved in any case. So is something different being asked for here? Barry Warsaw wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? And I'd note that getting a good way to use Python to develop for iOS, Android, and Mobile Windows is FAR more critical! -- maybe that's the same problem ? -Chris On Thu, May 28, 2015 at 8:39 AM, Donald Stufft don...@stufft.io wrote: On May 28, 2015 at 11:30:37 AM, Steve Dower (steve.do...@microsoft.com) wrote: Donald Stufft wrote: Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need. For contrast, here are the things you need on Windows to be able to get to an interactive prompt (I don't know how other platforms get this down to 4KB...): * python.exe (or some equivalent launcher) 39KB * python35.dll 3,788KB * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not redistributable so doesn't count here) * 26 files in Lib 343KB This gets you to , and basically everything after that is going to fail for some reason. That's an unavoidable 4,257KB. The rest of the stdlib adds another ~16MB once you exclude the test suite, so a fully functioning Python is not cheap. (Using compressed .pyc's in a zip file can make a big difference here though, assuming you're willing to trade CPU for HDD.) Cheers, Steve You don’t need a fully functioning Python for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc. Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov -- Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Fri, May 29, 2015 at 3:04 AM, Brian Curtin br...@python.org wrote: Donald mentioned one earlier: command line utilities. I want a single CLI I can deploy to my customers that doesn't make them have to install Python or even know it's Python at all. My users write code in all types of languages on all OSes, but I should be able to produce one thing that they can all use. Donald himself initiated the CLI in particular I'm talking about, but Go is picking up steam here as we have other utilities that quickly solved the write one thing, every user can run it immediately, no one knows/cares what it's written in Unix-like systems have this courtesy of the shebang, so as long as there's some sort of Python installed, people don't need to know or care that /usr/local/bin/mailmail is implemented in Python. Maybe the solution is to push for Windows to always include a Python interpreter, which would allow a tiny stub to go and call on that? Obviously a full shebang concept would be a huge change to Windows, but if a 4KB executable can go and locate the rest of Python, or open up a web browser saying Please install OS update KB123456, that would do it for most end users. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Fri, May 29, 2015 at 3:52 AM, Paul Moore p.f.mo...@gmail.com wrote: On 28 May 2015 at 18:15, Chris Angelico ros...@gmail.com wrote: Unix-like systems have this courtesy of the shebang, so as long as there's some sort of Python installed, people don't need to know or care that /usr/local/bin/mailmail is implemented in Python. Maybe the solution is to push for Windows to always include a Python interpreter, which would allow a tiny stub to go and call on that? Unfortunately (and believe me, I've been down this road many times) on Windows *only* the exe format is a first-class executable. Executable scripts and shebangs are very useful, but there are always corner cases where they don't work *quite* like an exe. On Windows, you have to be prepared to ship an exe if you want to compete with languages that generate exes. I'm aware of that. When I said a tiny stub, I was thinking in terms of a small executable. The idea is that its sole purpose is to locate Python someplace else, and chain to it; that has to be actual executable code, complete with the 512-byte MZ header and everything, to ensure compatibility. But it should be able to be small, tight, and easy to verify correctness of, so there aren't (in theory!) security exploits in the header itself. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 28 May 2015 at 19:22, Chris Angelico ros...@gmail.com wrote: Unfortunately (and believe me, I've been down this road many times) on Windows *only* the exe format is a first-class executable. Executable scripts and shebangs are very useful, but there are always corner cases where they don't work *quite* like an exe. On Windows, you have to be prepared to ship an exe if you want to compete with languages that generate exes. I'm aware of that. When I said a tiny stub, I was thinking in terms of a small executable. The idea is that its sole purpose is to locate Python someplace else, and chain to it; that has to be actual executable code, complete with the 512-byte MZ header and everything, to ensure compatibility. But it should be able to be small, tight, and easy to verify correctness of, so there aren't (in theory!) security exploits in the header itself. OK, cool. I'm sort of working on that as a bit of a side project - a tiny stub exe that you can prepend to a Python zipapp which runs it via the standard embedding APIs. It's little more than an idea at the moment, but I don't think it'll be too hard to implement... Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 5/28/2015 12:44 PM, Donald Stufft wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. I completely agree. I talk to a lot of people about packaging of things, and while I think there are some serious problems with huge parts of Go’s packaging and distribution story the static linking and compiling down to a “single” file is not one of them. How about the following compromise between assuming the presence of installed python and ignoring the possibility of installed python: we provide (as least for Windows) a C startup file that checks for the needed version of python and offers to download and start the installer if not present. A user would see 'Python 3.5 is needed to run this file; shall I download and install it?' just once on a particular machine. -- Terry Jan Reedy ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
OK, I'm really confused here: 1) what the heck is so special about go all of a sudden? People have been writing and deploying single file executables built with C and ++, and whatever else? forever. (and indeed, it was a big sticking point for me when I introduced python in my organization) 2) Why the sudden interest in this as core a Python issue? I've been using Python for desktop apps, on primarily Windows and the Mac for years -- and had to deal with py2exe, py2app, etc. forever. And it has been a very very common question on the various mailing lists for ages: how do I deploy this? how do I make it easy to install? The answer from the developers of cPython itself has always been that that's a third party problem -- and go look for py2exe and friends to solve it. And that it is a solved-enough problem. The biggest unsolved issues are that you get a really big application. Don't get me wrong -- I've wanted for years for it to be easier to deploy python-based apps as a single thinking for users to easily install and uninstall where they don't need to know it's python -- but what the heck is different now? 3) There was mention of a platform-neutral way to do this. Isn't that simply impossible? The platforms are different in all the ways that matter for this problem: both technical differences, and conventions. Which isn't to say you couldn't have one API to produce a single thing executable, so it would look like one solution for multiple platforms to the user. But the end product should be (would have to be) a different beast altogether. And doesn't PyInstaller already provide that (may it can't do single-file...) Anyway -- if there really is a new interest in this problem such that people will put some time into, here are some thoughts I've had for ages: The big one is Separation of concerns: in order to build a single thing executable, you need three things: a) An API to for the developer to specify what they want b) Figure out what needs to be included -- what extra modules, etc. c) A way to package it all up: App bundle on the Mac, single file executable on Windows (statically linked? zip file, ???) That third one -- (c) is inherently platform dependent -- and there is more than one way to do it even on one platform. But it sure would be nice if the API between a) b), and c) could be unified so we could mix and match different implementations. And, of course, if cPython itself could be built in a way that makes step(c) easier/less kludgy great! -Chris On Thu, May 28, 2015 at 9:54 AM, Donald Stufft don...@stufft.io wrote: On May 28, 2015 at 12:24:42 PM, Chris Barker (chris.bar...@noaa.gov) wrote: I'm confused: Doesn't py2exe (optionally) create a single file executable? And py2app on the Mac creates an application bundle, but that is more-or-less the equivalent on OS-X (you may not even be able to have a single file executable that can access the Window Manager, for instance) Depending on what extra packages you need, py2exe's single file doesn't always work, but last I tried, it worked for a fair bit (I think all of the stdlib). I don't know what PyInstaller or others create. And I have no idea if there is a linux option -- but it seems like the standard of practice for an application for linux is a bunch of files scattered over the system anyway :-) Yes, the resulting exe is pretty big, but it does try to include only those modules and packages that are used, and that kind of optimization could be improved in any case. So is something different being asked for here? All of those solutions “work” to varying degrees of work, almost all of them rely on hacks in order to make things “work” because the ability to do it isn’t built into Python itself. If the critical pieces to execute in this way was built into Python itself, then those tools would work a whole lot better than they currently do. Barry Warsaw wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? The desktop isn’t dying, Mobile is becoming a very important thing of course, but that’s just because people are using devices *more* to account for the use of Mobile, they aren’t really using their Desktop’s less. See: http://blogs.wsj.com/cmo/2015/05/26/mobile-isnt-killing-the-desktop-internet/ And I'd note that getting a good way to use Python to develop for iOS, Android, and Mobile Windows is FAR more critical! -- maybe that's the same problem ? It’s not the same problem, but it’s also not very relevant. Volunteer time isn’t fungible, you get what people are willing to work on regardless of whether it will help Python as a whole. It’s also not an either/or proposition, we can both improve our ability to develop under iOS/Android/etc and improve our
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Thu, May 28, 2015 at 3:25 PM Chris Barker chris.bar...@noaa.gov wrote: OK, I'm really confused here: 1) what the heck is so special about go all of a sudden? People have been writing and deploying single file executables built with C and ++, and whatever else? forever. (and indeed, it was a big sticking point for me when I introduced python in my organization) Because Go is much easier to use to write those CLI apps than C or C++. The barrier now is low enough that the ease of development plus the ability to do statically compiled binaries is enticing folks to drop Python and Ruby for Go when making a CLI app. 2) Why the sudden interest in this as core a Python issue? I've been using Python for desktop apps, on primarily Windows and the Mac for years -- and had to deal with py2exe, py2app, etc. forever. And it has been a very very common question on the various mailing lists for ages: how do I deploy this? how do I make it easy to install? The answer from the developers of cPython itself has always been that that's a third party problem -- and go look for py2exe and friends to solve it. And that it is a solved-enough problem. The biggest unsolved issues are that you get a really big application. Anecdotal evidence suggests Go's user base has a decent amount of converts from Python and it's currently the only language that seems to be siphoning people out of our community. You do hear stories of people skipping Python 3 and going to Go, but considering how much more work that is than writing Python 2/3 code I believe that typically happens when the organization wanted to jump ship anyway and the Python 3 transition just gave them an excuse to rewrite their stuff (plus we all know how enticing it can be to play with a shiny new piece of tech if given the chance). Don't get me wrong -- I've wanted for years for it to be easier to deploy python-based apps as a single thinking for users to easily install and uninstall where they don't need to know it's python -- but what the heck is different now? Active user loss where the biggest reason people are leaving that we can actively fix now is easy app deployment (the other is performance; some might argue concurrency but concurrent.futures and async/await water that down somewhat for me). 3) There was mention of a platform-neutral way to do this. Isn't that simply impossible? The platforms are different in all the ways that matter for this problem: both technical differences, and conventions. Which isn't to say you couldn't have one API to produce a single thing executable, so it would look like one solution for multiple platforms to the user. But the end product should be (would have to be) a different beast altogether. I think it's to have a single tool to do it for any platform, not to have the technical nuts and bolts be the same necessarily. I think it's also to figure out if there is anything the interpreter and/or stdlib can do to facilitate this. -Brett ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Fri, May 29, 2015 at 3:20 AM, Donald Stufft don...@stufft.io wrote: On May 28, 2015 at 12:54:34 PM, Chris Angelico (ros...@gmail.com) wrote: Is this a Windows-specific issue, or is it also intended for Linux and Mac OS, where there'll already be a system Python (so a single-file-executable would be used to be independent of the system Python)? I think it’s an issue for all platforms, even when there is a system Python that can be used. Here’s why: * Even on Linux systems Python isn’t always a guaranteed thing to be installed, for instance Debian works just fine without any Python installed. * On OS X you have the system Python yes, but that is in an unknown state. It could have any number of changes made to it or things installed or what have you. * Even if you have Python installed already, is it the right one? What if it’s an ancient RHEL box that has 2.6 or (heaven forbid) 2.4? What if it’s a not ancient box that has Python 2.7 but you want to deploy your app in Python 3? * What if you have Python installed already, but it’s been patched by the place you got it from and now the behavior is different than what you expected? The independence argument. Yep, reasonable. The trouble is that if everyone seeks to be independent of the system Python, it makes the system Python redundant with every single Python app; and the bundled Python will have to be platform-specific (not just OS, but architecture, word size, possibly OS version in some cases, etc, etc). And that independence also means you miss out on security/compatibility updates from upstream, so you're having to manage your own updates. I'd still much rather see a small stub that goes and calls on a system Python than something that has to duplicate all of Python into every binary, but it's a choice for the application developer to make. If there can at least be an easy way to strip off the self-executability header and get back the base Python application, that would be a big help - you could have a script that goes through all your executables, looks for the signature that says hi, I'm a Python 2.7.10 self-runner, and strips it off in favour of a 2.7.11 that fixes some critical issue. Or you could strip that off and run the underlying Python code on a different OS. Binary blobs are so often unworkable. ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Thu, May 28, 2015 at 11:23 AM, Chris Barker chris.bar...@noaa.gov wrote: I'm confused: Doesn't py2exe (optionally) create a single file executable? And py2app on the Mac creates an application bundle, but that is more-or-less the equivalent on OS-X (you may not even be able to have a single file executable that can access the Window Manager, for instance) Depending on what extra packages you need, py2exe's single file doesn't always work, but last I tried, it worked for a fair bit (I think all of the stdlib). I don't know what PyInstaller or others create. And I have no idea if there is a linux option -- but it seems like the standard of practice for an application for linux is a bunch of files scattered over the system anyway :-) Yes, the resulting exe is pretty big, but it does try to include only those modules and packages that are used, and that kind of optimization could be improved in any case. So is something different being asked for here? Barry Warsaw wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? Donald mentioned one earlier: command line utilities. I want a single CLI I can deploy to my customers that doesn't make them have to install Python or even know it's Python at all. My users write code in all types of languages on all OSes, but I should be able to produce one thing that they can all use. Donald himself initiated the CLI in particular I'm talking about, but Go is picking up steam here as we have other utilities that quickly solved the write one thing, every user can run it immediately, no one knows/cares what it's written in When I worked on Ubuntu One, I was the Windows guy responsible for making sure the end-user experience was the same there as it was on Ubuntu. On Ubuntu we were a part of the base install and didn't have to worry about much. On Windows we had none of that, not even the C runtime, so we had some pre-installer work to do, and then a bunch of py2exe hacking to make everything play nicely and transparently. ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
Donald Stufft wrote: On May 28, 2015 at 11:30:37 AM, Steve Dower (steve.do...@microsoft.com) wrote: Donald Stufft wrote: Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need. For contrast, here are the things you need on Windows to be able to get to an interactive prompt (I don't know how other platforms get this down to 4KB...): * python.exe (or some equivalent launcher) 39KB * python35.dll 3,788KB * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not redistributable so doesn't count here) * 26 files in Lib 343KB This gets you to , and basically everything after that is going to fail for some reason. That's an unavoidable 4,257KB. The rest of the stdlib adds another ~16MB once you exclude the test suite, so a fully functioning Python is not cheap. (Using compressed .pyc's in a zip file can make a big difference here though, assuming you're willing to trade CPU for HDD.) Cheers, Steve You don’t need a fully functioning Python for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc. Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes. Agreed, but the minimally functioning Python is barely under 5MB. That will be considered bloated and won't help us compete with Go, so we should find a better way to fix Python application distribution and stop getting so hung up on putting everything into a single executable file. Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 28 May 2015 at 18:04, Brian Curtin br...@python.org wrote: Donald mentioned one earlier: command line utilities. I want a single CLI I can deploy to my customers that doesn't make them have to install Python or even know it's Python at all. Yep, that's the killer for me as well. I know it's unrealistic in some sense, but my benchmark is what does a Hello, world program look like? In C, it's a 38K executable with no external dependencies. What does it look like in Python? (I'm not too worried if it's a bit bigger, but if it's a *lot* bigger that starts to be noticeable - Python generates bloated exes). What I'd like to be able to do is to write Python ports of a range of core Unix utilities (comm, cut, join, od, seq, tr, ...) and have them be viable alternatives to my current C builds (a series of single-file 100-200k static exes). On 28 May 2015 at 18:15, Chris Angelico ros...@gmail.com wrote: Unix-like systems have this courtesy of the shebang, so as long as there's some sort of Python installed, people don't need to know or care that /usr/local/bin/mailmail is implemented in Python. Maybe the solution is to push for Windows to always include a Python interpreter, which would allow a tiny stub to go and call on that? Unfortunately (and believe me, I've been down this road many times) on Windows *only* the exe format is a first-class executable. Executable scripts and shebangs are very useful, but there are always corner cases where they don't work *quite* like an exe. On Windows, you have to be prepared to ship an exe if you want to compete with languages that generate exes. Having said that, the trick of appending a zipfile to an exe (or similar) is already common practice in the Python world, and works really well. Vinay Sanjip's pyzzer is a good example of this approach. On 28 May 2015 at 16:45, Steve Dower steve.do...@microsoft.com wrote: Agreed, but the minimally functioning Python is barely under 5MB. That will be considered bloated and won't help us compete with Go, so we should find a better way to fix Python application distribution and stop getting so hung up on putting everything into a single executable file. There's a perception issue here. You can compile C# code into small exes (that's how people think) and all you need is the .net runtime installed. If we shipped a pyc compiler that compiled Python code into small exes that just needed the Python runtime installed would that feel the same to people? Would they be happy to view that as comparable to go compiled executables? (I assume go *doesn't* rely on a separate runtime, though). Nevertheless, I would like to understand how Unix can manage to have a Python 3.4.3 binary at 4kb. Does that *really* have no external dependencies (other than the C library)? Are we really comparing like with like here? Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On May 28, 2015 at 12:54:34 PM, Chris Angelico (ros...@gmail.com) wrote: On Fri, May 29, 2015 at 2:28 AM, Chris Barker wrote: oops, sorry -- I see this was addressed in another thread. Though I guess I still don't see why single file is critical, over single thing to install -- like a OS-X app bundle that can just be dragged into the Applications folder. There's also single thing to uninstall, which IMO is more important. If I download a tiny program that's supposed to just do one tiny thing, and it has to install itself into Program Files, Common Files, Windows\System32, and Documents Settings\my-user-name\Applications, then I have to hope it has a proper uninstaller. If it's a single executable that just does its stuff (or, failing that, a single zip file that I extract to anywhere and run the program), I can expect that deleting that file (or directory) will get rid of it all. Of course, it's entirely possible that it's gone and left its droppings all over the system, but that's a matter of trust - a legit program won't lie about that. Is this a Windows-specific issue, or is it also intended for Linux and Mac OS, where there'll already be a system Python (so a single-file-executable would be used to be independent of the system Python)? I think it’s an issue for all platforms, even when there is a system Python that can be used. Here’s why: * Even on Linux systems Python isn’t always a guaranteed thing to be installed, for instance Debian works just fine without any Python installed. * On OS X you have the system Python yes, but that is in an unknown state. It could have any number of changes made to it or things installed or what have you. * Even if you have Python installed already, is it the right one? What if it’s an ancient RHEL box that has 2.6 or (heaven forbid) 2.4? What if it’s a not ancient box that has Python 2.7 but you want to deploy your app in Python 3? * What if you have Python installed already, but it’s been patched by the place you got it from and now the behavior is different than what you expected? etc etc. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Thu, May 28, 2015 at 11:38 AM, Paul Moore p.f.mo...@gmail.com wrote: On 28 May 2015 at 16:58, Barry Warsaw ba...@python.org wrote: On May 28, 2015, at 11:39 AM, Donald Stufft wrote: You don’t need a fully functioning Python for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc. There are actually two related but different use cases to single file executables. The first is nicely solved by tools like pex, where you don't need to include a fully functional Python at the head of the zip file because the environment you're deploying it into will have enough Python to make the zip work. This can certainly result in smaller zip files. This is the approach I took with Snappy Ubuntu Core support for Python 3, based on the current situation that the atomic upgrade client is written in Python 3. If that changes and Python 3 is removed from the image, then this approach won't work. pex (and others) does a great job at this, so unless there are things better refactored into upstream Python, I don't think we need to do much here. One problem with pex is that it doesn't appear to work on Windows (I just gave it a try, and got errors because it relies on symlinks). IMO, any solution to distributing Python applications that is intended to compete with the idea that go produces nice single-file executables needs to be cross-platform. At the moment, zipapp (and in general, the core support for running applications from a zip file) handles this for the case where you're allowed to assume an already installed Python interpreter. The proviso here, as Donald pointed out, is that it doesn't handle C extensions. The biggest problem with 3rd-party solutions is that they don't always support the full range of platforms that Python supports. That's fine for a 3rd party tool, but if we want to have a response to people asking how to bundle their application written in Python, we need a better answer than if you're on Windows, use py2exe, or if you're on Unix use pex, or maybe... Python has core support for the equivalent of Java's jar format in zipapp. It's not well promoted (and doesn't support C extensions) but it's a pretty viable option for a lot of situations. The second use case is as you describe: put a complete functional Python environment at the head of the zip file so you don't need anything in the target deployment environment. Complete can easily mean the entire stdlib, and although that would usually be more bloated than you normally need, hey, it's just some extra unused bits so who cares? wink. I think this would be an excellent starting point which can be optimized to trim unnecessary bits later, maybe by third party tools. Tools like py2exe and cx_Freeze do this, and are pretty commonly used on Windows. An obvious example of use is Mercurial. If you're looking at this scenario, a good place to start would probably be understanding why cx_Freeze isn't more commonly used on Unix (AFAIK, it supports Unix, but I've only ever really heard of it being used on Windows). Esky https://github.com/cloudmatrix/esky/ * supports py2exe, py2app, cxfreeze and bbfreeze * builds a zip archive containing an .exe * manages (failed) [auto-]updates PEX https://pantsbuild.github.io/pex_design.html * adds an executable header to a (topo-sorted?) ZIP file with a minimal path * pipsi https://github.com/mitsuhiko/pipsi/blob/master/pipsi.py * installs packages with console_scripts into separate virtualenvs with minimal sys.paths and ~/.local/bin) At the end of the day I still need packaging or configmgmt or NIX for checksums (a manifest wrapped around the executable wrapper). I suspect single file executables just aren't viewed as a desirable solution on Unix. Although Donald referred to a 4K binary, which probably means just a stub exe that depends on system-installed .so files, likely including Python (I'm just guessing here). It's easy to do something similar on Windows, but it's *not* what most Windows users think of when you say a single file executable for a Python program (because there's no system package manager doing dependencies for you). NuGet, Chocolatey, - OneGet It's a topologically sorted adjacency list + build + install + uninstall scripts. Again, platform-specific answers are one thing, and are relatively common, but having a good cross-platform answer at the language level (a section on docs.python.org How to ship your Python program) is much harder. Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On May 28, 2015, at 12:44 PM, Donald Stufft wrote: Pex would be improved by having native support for importing .so’s from within a zipfile via zipimport. It would also be improved by having good, built in support for extraneous resources in the stdlib too. Completely agree on both points. Having an API for importing .so's from a zip would be really useful. Today that can be implemented as copy to tempdir and tomorrow the implementation could optionally dlopen_from_memory() without any client code changing. Right, it would be great to get it built into Python itself, but I consider that less important than getting the critical pieces into Python. If those pieces are there then we can iterate outside of the standard library and try different approaches to *building* such a file, and eventually take a look at the landscape and bless one approach (or not, if we don’t want to). Sounds good to me! Cheers, -Barry pgp1R37cL1HEu.pgp Description: OpenPGP digital signature ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
You might want to have a look at eGenix PyRun, which gives you an almost complete Python runtime in 4-13MB (depending on what startup performance needs you have): http://www.egenix.com/products/python/PyRun/ On 28.05.2015 17:58, Barry Warsaw wrote: On May 28, 2015, at 11:39 AM, Donald Stufft wrote: You don’t need a fully functioning Python for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc. There are actually two related but different use cases to single file executables. The first is nicely solved by tools like pex, where you don't need to include a fully functional Python at the head of the zip file because the environment you're deploying it into will have enough Python to make the zip work. This can certainly result in smaller zip files. This is the approach I took with Snappy Ubuntu Core support for Python 3, based on the current situation that the atomic upgrade client is written in Python 3. If that changes and Python 3 is removed from the image, then this approach won't work. pex (and others) does a great job at this, so unless there are things better refactored into upstream Python, I don't think we need to do much here. The second use case is as you describe: put a complete functional Python environment at the head of the zip file so you don't need anything in the target deployment environment. Complete can easily mean the entire stdlib, and although that would usually be more bloated than you normally need, hey, it's just some extra unused bits so who cares? wink. I think this would be an excellent starting point which can be optimized to trim unnecessary bits later, maybe by third party tools. See above. This is what eGenix PyRun provides. Our main motivation is to have a binary which works on all Unix platforms, without having to rely on the way too many system dependent Python distribution (with all their quirks and whistles ;-)). On Windows, we use py2exe at the moment, but a port of PyRun to Windows would be possible as well. You'd still need the separate Python DLL, though in order to stay compatible to C extensions which link against this. As for application packaging: we don't have a need to put everything into a single ZIP file or even concatenate such a ZIP file to PyRun (which is possible: just add sys.executable to sys.path to import from the executable). We have plans to create a tool to make such packaging possible, though, since it's handy to have for building small executable apps, e.g. to drive installations or larger applications. Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes. I'd love to see Python itself gain such a tool, but if it had the critical pieces to execute in this way, that would enable a common approach to supporting this in third party tools, on a variety of platforms. I do think single-file executables are an important piece to Python's long-term competitiveness. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 28 2015) Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ : Try our mxODBC.Connect Python Database Interface for free ! :: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On May 28, 2015, at 09:23 AM, Chris Barker wrote: Barry Warsaw wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? And I'd note that getting a good way to use Python to develop for iOS, Android, and Mobile Windows is FAR more critical! -- maybe that's the same problem ? Well, in my world they are the same problem! With mobile, IoT, etc. you can't or shouldn't assume that Python will be available in the base environment. There are ways to deploy to the various new world of devices that perhaps don't require single-file executables, but those are more complicated to manage, and they still have to ship the Python environment along with the application code. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
py2exe tends to invoke DLL hell if you have various versions of VS or Office or both installed. Because Windows. On May 28, 2015 11:23:57 AM CDT, Chris Barker chris.bar...@noaa.gov wrote: I'm confused: Doesn't py2exe (optionally) create a single file executable? And py2app on the Mac creates an application bundle, but that is more-or-less the equivalent on OS-X (you may not even be able to have a single file executable that can access the Window Manager, for instance) Depending on what extra packages you need, py2exe's single file doesn't always work, but last I tried, it worked for a fair bit (I think all of the stdlib). I don't know what PyInstaller or others create. And I have no idea if there is a linux option -- but it seems like the standard of practice for an application for linux is a bunch of files scattered over the system anyway :-) Yes, the resulting exe is pretty big, but it does try to include only those modules and packages that are used, and that kind of optimization could be improved in any case. So is something different being asked for here? Barry Warsaw wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? And I'd note that getting a good way to use Python to develop for iOS, Android, and Mobile Windows is FAR more critical! -- maybe that's the same problem ? -Chris On Thu, May 28, 2015 at 8:39 AM, Donald Stufft don...@stufft.io wrote: On May 28, 2015 at 11:30:37 AM, Steve Dower (steve.do...@microsoft.com) wrote: Donald Stufft wrote: Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need. For contrast, here are the things you need on Windows to be able to get to an interactive prompt (I don't know how other platforms get this down to 4KB...): * python.exe (or some equivalent launcher) 39KB * python35.dll 3,788KB * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not redistributable so doesn't count here) * 26 files in Lib 343KB This gets you to , and basically everything after that is going to fail for some reason. That's an unavoidable 4,257KB. The rest of the stdlib adds another ~16MB once you exclude the test suite, so a fully functioning Python is not cheap. (Using compressed .pyc's in a zip file can make a big difference here though, assuming you're willing to trade CPU for HDD.) Cheers, Steve You don’t need a fully functioning Python for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc. Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
I agree that size is an issue, but is it really that bad? Just compare it to the recent web surge where everyone is writing desktop apps in HTML5+CSS+JS and bundling a huge WebKit engine in their apps binary. Python on Windows is seriously in a bad state. IMO, what needs to be prioritized is the ability to make exes that *actually work* with nicer GUI abilities. py2exe gives too many random DLL errors and PyInstaller is an ugly hack that just shoves 20 DLLs with your executable. Mixed with the fact that TkInter looks even uglier when built via py2exe and almost everything else (PyGI, PySide, etc.) requires yet another 20 DLLs (PySide threw in Qt DLLs that I didn't even use!), it's sad. Really sad. This is most of the reason I write programs that I plan on distributing to various crowds in some statically compiled language (C++, Nim, Felix, not Go) with (when necessary) a statically-linked GUI library. Less DLL hell, less free files, etc. Oh yeah, and add to that the problems with running both Python 2 and 3 on Windows while using some binaries that want 3 and others that want 2. It's painful. On May 28, 2015 10:30:33 AM CDT, Steve Dower steve.do...@microsoft.com wrote: Donald Stufft wrote: Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need. For contrast, here are the things you need on Windows to be able to get to an interactive prompt (I don't know how other platforms get this down to 4KB...): * python.exe (or some equivalent launcher) 39KB * python35.dll 3,788KB * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not redistributable so doesn't count here) * 26 files in Lib 343KB This gets you to , and basically everything after that is going to fail for some reason. That's an unavoidable 4,257KB. The rest of the stdlib adds another ~16MB once you exclude the test suite, so a fully functioning Python is not cheap. (Using compressed .pyc's in a zip file can make a big difference here though, assuming you're willing to trade CPU for HDD.) Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com -- Sent from my Android device with K-9 Mail. Please excuse my brevity.___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
Getting lost as to what thread this belongs in... But another tack to take toward a single executable is Cython's embedding option: https://github.com/cython/cython/wiki/EmbeddingCython This is a quick and dirty way to create a C executable that will then run the cythonized code, all linked to the python run time. At this point, it still requires the python shared lib, and I think any other compiled extension is shared, too. And if you run Cython on all the python code and modules you use, you'll have a LOT of shared libs. But perhaps one could re-do the linking step of all that and get a single compiled exe. and IIUC, the way Windows dll hell works, if you stuff the whole pile into one dir -- you will get a single executable directory, if not a single file. and about a 2X performance boost, as well, when you cythonize pure Python, at least in my limited experience. Just a thought. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Thu, May 28, 2015 at 9:23 AM, Chris Barker chris.bar...@noaa.gov wrote: Barry Warsaw wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? oops, sorry -- I see this was addressed in another thread. Though I guess I still don't see why single file is critical, over single thing to install -- like a OS-X app bundle that can just be dragged into the Applications folder. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 28 May 2015 at 16:58, Barry Warsaw ba...@python.org wrote: On May 28, 2015, at 11:39 AM, Donald Stufft wrote: You don’t need a fully functioning Python for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc. There are actually two related but different use cases to single file executables. The first is nicely solved by tools like pex, where you don't need to include a fully functional Python at the head of the zip file because the environment you're deploying it into will have enough Python to make the zip work. This can certainly result in smaller zip files. This is the approach I took with Snappy Ubuntu Core support for Python 3, based on the current situation that the atomic upgrade client is written in Python 3. If that changes and Python 3 is removed from the image, then this approach won't work. pex (and others) does a great job at this, so unless there are things better refactored into upstream Python, I don't think we need to do much here. One problem with pex is that it doesn't appear to work on Windows (I just gave it a try, and got errors because it relies on symlinks). IMO, any solution to distributing Python applications that is intended to compete with the idea that go produces nice single-file executables needs to be cross-platform. At the moment, zipapp (and in general, the core support for running applications from a zip file) handles this for the case where you're allowed to assume an already installed Python interpreter. The proviso here, as Donald pointed out, is that it doesn't handle C extensions. The biggest problem with 3rd-party solutions is that they don't always support the full range of platforms that Python supports. That's fine for a 3rd party tool, but if we want to have a response to people asking how to bundle their application written in Python, we need a better answer than if you're on Windows, use py2exe, or if you're on Unix use pex, or maybe... Python has core support for the equivalent of Java's jar format in zipapp. It's not well promoted (and doesn't support C extensions) but it's a pretty viable option for a lot of situations. The second use case is as you describe: put a complete functional Python environment at the head of the zip file so you don't need anything in the target deployment environment. Complete can easily mean the entire stdlib, and although that would usually be more bloated than you normally need, hey, it's just some extra unused bits so who cares? wink. I think this would be an excellent starting point which can be optimized to trim unnecessary bits later, maybe by third party tools. Tools like py2exe and cx_Freeze do this, and are pretty commonly used on Windows. An obvious example of use is Mercurial. If you're looking at this scenario, a good place to start would probably be understanding why cx_Freeze isn't more commonly used on Unix (AFAIK, it supports Unix, but I've only ever really heard of it being used on Windows). I suspect single file executables just aren't viewed as a desirable solution on Unix. Although Donald referred to a 4K binary, which probably means just a stub exe that depends on system-installed .so files, likely including Python (I'm just guessing here). It's easy to do something similar on Windows, but it's *not* what most Windows users think of when you say a single file executable for a Python program (because there's no system package manager doing dependencies for you). Again, platform-specific answers are one thing, and are relatively common, but having a good cross-platform answer at the language level (a section on docs.python.org How to ship your Python program) is much harder. Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes. I'd love to see Python itself gain such a tool, but if it had the critical pieces to execute in this way, that would enable a common approach to supporting this in third party tools, on a variety of platforms. Stripping out unused code is a hard problem in a language as dynamic as Python. It would be great to see it happen, but I'm not sure how much better we can do than existing tools
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On May 28, 2015, at 11:39 AM, Donald Stufft wrote: You don’t need a fully functioning Python for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc. There are actually two related but different use cases to single file executables. The first is nicely solved by tools like pex, where you don't need to include a fully functional Python at the head of the zip file because the environment you're deploying it into will have enough Python to make the zip work. This can certainly result in smaller zip files. This is the approach I took with Snappy Ubuntu Core support for Python 3, based on the current situation that the atomic upgrade client is written in Python 3. If that changes and Python 3 is removed from the image, then this approach won't work. pex (and others) does a great job at this, so unless there are things better refactored into upstream Python, I don't think we need to do much here. The second use case is as you describe: put a complete functional Python environment at the head of the zip file so you don't need anything in the target deployment environment. Complete can easily mean the entire stdlib, and although that would usually be more bloated than you normally need, hey, it's just some extra unused bits so who cares? wink. I think this would be an excellent starting point which can be optimized to trim unnecessary bits later, maybe by third party tools. Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes. I'd love to see Python itself gain such a tool, but if it had the critical pieces to execute in this way, that would enable a common approach to supporting this in third party tools, on a variety of platforms. I do think single-file executables are an important piece to Python's long-term competitiveness. Cheers, -Barry ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On May 28, 2015 at 11:30:37 AM, Steve Dower (steve.do...@microsoft.com) wrote: Donald Stufft wrote: Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need. For contrast, here are the things you need on Windows to be able to get to an interactive prompt (I don't know how other platforms get this down to 4KB...): * python.exe (or some equivalent launcher) 39KB * python35.dll 3,788KB * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not redistributable so doesn't count here) * 26 files in Lib 343KB This gets you to , and basically everything after that is going to fail for some reason. That's an unavoidable 4,257KB. The rest of the stdlib adds another ~16MB once you exclude the test suite, so a fully functioning Python is not cheap. (Using compressed .pyc's in a zip file can make a big difference here though, assuming you're willing to trade CPU for HDD.) Cheers, Steve You don’t need a fully functioning Python for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc. Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On 28 May 2015 at 17:28, Chris Barker chris.bar...@noaa.gov wrote: On Thu, May 28, 2015 at 9:23 AM, Chris Barker chris.bar...@noaa.gov wrote: Barry Warsaw wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? oops, sorry -- I see this was addressed in another thread. Though I guess I still don't see why single file is critical, over single thing to install -- like a OS-X app bundle that can just be dragged into the Applications folder. On Windows, there's a strong interest in no install download-and-run applications (.Net has xcopy deployment, portable applications are a big deal to some people). Just unzip the distribution somewhere and run the exe is definitely a selling point for that audience. And download an exe and just run it is even more so. But the latter is definitely an incremental improvement - the former is the big deal (IMO). Paul ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Fri, May 29, 2015 at 1:28 AM, Chris Barker chris.bar...@noaa.gov wrote: On Thu, May 28, 2015 at 9:23 AM, Chris Barker chris.bar...@noaa.gov wrote: Barry Warsaw wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? oops, sorry -- I see this was addressed in another thread. Though I guess I still don't see why single file is critical, over single thing to install -- like a OS-X app bundle that can just be dragged into the Applications folder. It is much simpler to deploy in an automated, recoverable way (and also much faster), because you can't have parts of the artefact unsynchronized with another part of the program. Note also that moving a python installation in your fs is actually quite unlikely to work in interesting usecases on unix because of the relocatability issue. Another advantage: it makes it impossible for users to tamper an application's content and be surprised things don't work anymore (a very common source of issues, familiar to anybody deploying complex python applications in the enterprise world). I recently started using some services written in go, and the single file approach is definitely a big +. It makes *using* applications written in it so much easier than python, even though I am complete newbie in go and relatively comfortable with python. One should keep in mind that go has some inherent advantages over python in those contexts even if python were to gain single file distribution tomorrow. Most of go stdlib is written in go now I believe, and it is much more portable across linux systems on a given CPU arch compared to python. IOW, it is more robust against ABI variability. David ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On May 28, 2015 at 12:01:22 PM, Barry Warsaw (ba...@python.org) wrote: On May 28, 2015, at 11:39 AM, Donald Stufft wrote: You don’t need a fully functioning Python for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc. There are actually two related but different use cases to single file executables. The first is nicely solved by tools like pex, where you don't need to include a fully functional Python at the head of the zip file because the environment you're deploying it into will have enough Python to make the zip work. This can certainly result in smaller zip files. This is the approach I took with Snappy Ubuntu Core support for Python 3, based on the current situation that the atomic upgrade client is written in Python 3. If that changes and Python 3 is removed from the image, then this approach won't work. pex (and others) does a great job at this, so unless there are things better refactored into upstream Python, I don't think we need to do much here. Pex would be improved by having native support for importing .so’s from within a zipfile via zipimport. It would also be improved by having good, built in support for extraneous resources in the stdlib too. It’s doing pretty well on it’s own though besides those quality of life improvements. The second use case is as you describe: put a complete functional Python environment at the head of the zip file so you don't need anything in the target deployment environment. Complete can easily mean the entire stdlib, and although that would usually be more bloated than you normally need, hey, it's just some extra unused bits so who cares? . I think this would be an excellent starting point which can be optimized to trim unnecessary bits later, maybe by third party tools. Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes. I'd love to see Python itself gain such a tool, but if it had the critical pieces to execute in this way, that would enable a common approach to supporting this in third party tools, on a variety of platforms. Right, it would be great to get it built into Python itself, but I consider that less important than getting the critical pieces into Python. If those pieces are there then we can iterate outside of the standard library and try different approaches to *building* such a file, and eventually take a look at the landscape and bless one approach (or not, if we don’t want to). I do think single-file executables are an important piece to Python's long-term competitiveness. I completely agree. I talk to a lot of people about packaging of things, and while I think there are some serious problems with huge parts of Go’s packaging and distribution story the static linking and compiling down to a “single” file is not one of them. People *really* enjoy this and it simplifies a ton of things for people. They don’t have to worry about making sure a whole set of files are deployed, they don’t have to worry about what version of Python is installed (or if any version is installed), they don’t have to worry about what other things have been installed, they just copy an executable and then they are good to go. I think we could potentially do *better* than Go in this regard, because we can make it possible to do both “static” and “dynamic” dependencies, as well as provide the ecosystem around that to do things like provide visibility into what versions of libraries are “compiled” into that single file executable, etc. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
I'm confused: Doesn't py2exe (optionally) create a single file executable? And py2app on the Mac creates an application bundle, but that is more-or-less the equivalent on OS-X (you may not even be able to have a single file executable that can access the Window Manager, for instance) Depending on what extra packages you need, py2exe's single file doesn't always work, but last I tried, it worked for a fair bit (I think all of the stdlib). I don't know what PyInstaller or others create. And I have no idea if there is a linux option -- but it seems like the standard of practice for an application for linux is a bunch of files scattered over the system anyway :-) Yes, the resulting exe is pretty big, but it does try to include only those modules and packages that are used, and that kind of optimization could be improved in any case. So is something different being asked for here? Barry Warsaw wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? And I'd note that getting a good way to use Python to develop for iOS, Android, and Mobile Windows is FAR more critical! -- maybe that's the same problem ? -Chris On Thu, May 28, 2015 at 8:39 AM, Donald Stufft don...@stufft.io wrote: On May 28, 2015 at 11:30:37 AM, Steve Dower (steve.do...@microsoft.com) wrote: Donald Stufft wrote: Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need. For contrast, here are the things you need on Windows to be able to get to an interactive prompt (I don't know how other platforms get this down to 4KB...): * python.exe (or some equivalent launcher) 39KB * python35.dll 3,788KB * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not redistributable so doesn't count here) * 26 files in Lib 343KB This gets you to , and basically everything after that is going to fail for some reason. That's an unavoidable 4,257KB. The rest of the stdlib adds another ~16MB once you exclude the test suite, so a fully functioning Python is not cheap. (Using compressed .pyc's in a zip file can make a big difference here though, assuming you're willing to trade CPU for HDD.) Cheers, Steve You don’t need a fully functioning Python for a single file binary, you only need enough to actually run your application. For example, if you're making an application that can download files over HTTP, you don't need to include parts of the stdlib like xmlrpc, pickle, shelve, marshall, sqlite, csv, email, mailcap, mailbox, imaplib, nntplib, etc. Of course deciding which pieces you include in the zip file you're appending to the end of Python is up to whatever tool builds this executable which doesn't need to be part of Python itself. If Python itself gained the ability to operate in that manner than third party tools could handle trying to do the optimizations where it only includes the things it actually needs in the stdlib and excludes things it doesn't. The key thing here is that since you're doing a single file binary, you don't need to have a Python which is suitable to execute random Python code, you only need one that is suitable to execute this particular code so you can specialize what that includes. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On Fri, May 29, 2015 at 2:28 AM, Chris Barker chris.bar...@noaa.gov wrote: oops, sorry -- I see this was addressed in another thread. Though I guess I still don't see why single file is critical, over single thing to install -- like a OS-X app bundle that can just be dragged into the Applications folder. There's also single thing to uninstall, which IMO is more important. If I download a tiny program that's supposed to just do one tiny thing, and it has to install itself into Program Files, Common Files, Windows\System32, and Documents Settings\my-user-name\Applications, then I have to hope it has a proper uninstaller. If it's a single executable that just does its stuff (or, failing that, a single zip file that I extract to anywhere and run the program), I can expect that deleting that file (or directory) will get rid of it all. Of course, it's entirely possible that it's gone and left its droppings all over the system, but that's a matter of trust - a legit program won't lie about that. Is this a Windows-specific issue, or is it also intended for Linux and Mac OS, where there'll already be a system Python (so a single-file-executable would be used to be independent of the system Python)? ChrisA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
On May 28, 2015 at 12:24:42 PM, Chris Barker (chris.bar...@noaa.gov) wrote: I'm confused: Doesn't py2exe (optionally) create a single file executable? And py2app on the Mac creates an application bundle, but that is more-or-less the equivalent on OS-X (you may not even be able to have a single file executable that can access the Window Manager, for instance) Depending on what extra packages you need, py2exe's single file doesn't always work, but last I tried, it worked for a fair bit (I think all of the stdlib). I don't know what PyInstaller or others create. And I have no idea if there is a linux option -- but it seems like the standard of practice for an application for linux is a bunch of files scattered over the system anyway :-) Yes, the resulting exe is pretty big, but it does try to include only those modules and packages that are used, and that kind of optimization could be improved in any case. So is something different being asked for here? All of those solutions “work” to varying degrees of work, almost all of them rely on hacks in order to make things “work” because the ability to do it isn’t built into Python itself. If the critical pieces to execute in this way was built into Python itself, then those tools would work a whole lot better than they currently do. Barry Warsaw wrote: I do think single-file executables are an important piece to Python's long-term competitiveness. Really? It seems to me that desktop development is dying. What are the critical use-cases for a single file executable? The desktop isn’t dying, Mobile is becoming a very important thing of course, but that’s just because people are using devices *more* to account for the use of Mobile, they aren’t really using their Desktop’s less. See: http://blogs.wsj.com/cmo/2015/05/26/mobile-isnt-killing-the-desktop-internet/ And I'd note that getting a good way to use Python to develop for iOS, Android, and Mobile Windows is FAR more critical! -- maybe that's the same problem ? It’s not the same problem, but it’s also not very relevant. Volunteer time isn’t fungible, you get what people are willing to work on regardless of whether it will help Python as a whole. It’s also not an either/or proposition, we can both improve our ability to develop under iOS/Android/etc and improve our ability to handle desktop applications. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Single-file Python executables (was: Computed Goto dispatch for Python 2)
Donald Stufft wrote: Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need. For contrast, here are the things you need on Windows to be able to get to an interactive prompt (I don't know how other platforms get this down to 4KB...): * python.exe (or some equivalent launcher) 39KB * python35.dll 3,788KB * vcruntime140.dll 87KB (the rest of the CRT is about 1MB, but is not redistributable so doesn't count here) * 26 files in Lib 343KB This gets you to , and basically everything after that is going to fail for some reason. That's an unavoidable 4,257KB. The rest of the stdlib adds another ~16MB once you exclude the test suite, so a fully functioning Python is not cheap. (Using compressed .pyc's in a zip file can make a big difference here though, assuming you're willing to trade CPU for HDD.) Cheers, Steve ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
