On 3/19/2011 7:38 PM, Mark Hammond wrote:
Thanks for the feedback!
And thanks for more complete explanations.
Sadly I was offline when writing my first response, and couldn't view
the man page for execve you referred to. Having just read it, I think
it would be total gibberish to a Windows user who has only learned
Python 2.x programming, and is now faced with migrating to 3.x, and
wants to use this launcher to help make that a gradual process, instead
of the full, instant migration required by the installation of Python
3.x wiping out the associations from Python 2.x leaving his system with
2.x unusable (from a naïve point of view). I have not found any support
either from the installer or the documentation (but maybe I missed some
things) that tell how to make Python 2.x and 3.x coexist happily on
Windows, short of simply reinstalling the one you want to use.
A Windows user who has only learned Python 2.x programming would not
necessarily have ever heard of execve, would not realize execve(2) means
it is from the 2nd chapter of the Unix man pages, meaning an API call,
would not know C programming, would not know what #include <unistd.h>
means, would not understand syntax like "const" or "[]". The
description is a bit friendlier, if they get that far, but they may
still be quite confused by wondering how to create and provide argv and
envp in the right forms, wondering what SIGCHLD, PID, set-user-ID bit,
SIG_IGN, SIG_DFL, SIGTRAP, set-group-ID, effective id, a.out, binary
executable, shared library stubs, ld.so(8), ELF executable, PT_INTERP,
and /lib/ld-unix.so.1 (and .2) are.
In short, anyone that is a Unix C programmer can easily understand this
stuff, but anyone that is a Windows Python-only programmer will feel
like understanding the launcher by reading that man page is similar in
effort to getting their masters degree, and that if it is that hard to
slowly migrate their 300 Python scripts to use Python 3.x while still
using Python 2.x for those that are not converted, that they might as
well just stick with Python 2.x.
Yes, the argument could be made that parts of Python also require some
Unix knowledge and basic C knowledge to use effectively... but there are
many Python programs that can be written without using the parts that
require that knowledge.
That said, I don't think it would be an onerous task to extract a
reasonable description of #! usage from the execve man page, and
eliminate the Unix masters degree requirement. In fact, only the
following two paragraph fragments seem to be relevant to #! lines:
... a script starting with a line of the form "*#! /interpreter/*
[arg]". ... the interpreter must be a valid pathname for an
executable which is not itself a script, which will be invoked as
*interpreter* [arg] /filename/.
A maximum line length of 127 characters is allowed for the first line
in a #! executable shell script.
More below in context...
On 19/03/2011 7:44 PM, Glenn Linderman wrote:
Not all of the ideas below are complementary to each other, some are
either or, to allow different thoughts to be inspired or different
directions to be taken.
Thanks for starting a PEP.
On 3/18/2011 11:02 PM, Mark Hammond wrote:
The launcher should be as simple as possible (but no simpler.)
The launcher could be simpler if it isn't used for launching interactive
interpreters as well as script references via Windows associations (more
about that after the next quote).
The launcher could be simpler if it just read through the file of its
first parameter until it finds a line starting with #@ (process as a
Windows version of Unix #!) or starting without a # (error case). This
avoids the need parse such lines.
I don't agree with that at all. The rules for the shebang parsing are
very simple - see the man-page I reference in the pep - the file must
start with the characters '#!' and the entire command must be < 127
chars. The only real "parsing" needed is to check if the specified
command starts with one of 2 fixed strings. I believe this is simpler
than parsing multiple lines of the file.
Ah, but Mark! You are confusing (and maybe I do in some of my comments
too) the complexity of the launcher versus the complexity of describing
the launcher. It is really the complexity of describing the launcher
(and remember, Unix doesn't need a launcher, it has #!, so you need to
describe it to Windows users for Windows users) that matters most. I
find I often rewrite portions of my programs when I write the
documentation to make them easier to describe. And some say the docs
should be written first, but then I find that I have to rewrite the docs
to make the code possible :) Whatever the order, the goal should be to
make the program as useful as possible to a variety of users, with as
little complexity as possible in the documentation.
So I still claim that it is easier to tell a Windows user that the
launcher looks at comment lines starting from line 1 until it finds a
line starting with #@ or finds a non-comment line, and executes it like
the first paragraph I extracted from execve above, than to explain about
how Unix works, why Unix uses / instead of \, why there is a special
case for /usr/bin/python and /usr/bin/env python, and why there is a 127
character limit (which need not be the case for a Windows specific
line). After describing that, the only reference to Unix that is
necessary is to say that Unix implements a similar feature[footnote to
execve man page] in the operating system with a line starting with #!
which must be the very first line and limited to 127 characters, and
that this Windows launcher is more flexible to accommodate using both
techniques for cross-platform scripts.
> Remember, the typical
Windows user is not likely to place a #! line in their scripts in the
first place, so teaching them what a Unix #! line is, and how the
parameter after it should be something that Windows doesn't even use,
and the launcher has to work hard to interpret, is not as simple as
possible.
I disagree with various aspects of that - the "typical Windows user"
is not going to add a shebang or variation of to their Python source
files, period.
IMO, when faced with migrating from 2.x to 3.x incrementally, the
"typical Windows user" will look to the Python provided launcher
solution to help (presuming that there is such a thing). If it doesn't
require a Unix masters degree, they'll give it whirl. Something like
Install the launcher feature from Python 3.3 (or available separately
somewhere), and add
#@C:\Python3.3\python.exe before the first non-comment line in the new
Python 3 scripts.
In fact, this description just inspired me to suggest that the launcher
should launch a Python 2 if it cannot find a #@ (or #! if I can't
convince you of #@) line... that way old scripts stick with Python 2
until they are converted.
IMO, the kind of user who would is already somewhat likely to know of
the shebang convention, so it would not be foreign. Those remaining
who are not familiar with it can simply be pointed at existing docs
etc for the shebang line and their new knowledge now works on more
than Windows (and more than Python on non-Windows platforms)
I agree that the users that already understand #! would gladly add it to
their Windows Python scripts and understand it. And I agree that those
are probably the only users that would understand the execve Unix
masters degree well enough to figure out a launcher that is described by
pointed to that man page. But I think a simple description of a
launcher that uses Windows terminology (Unix users don't need a
launcher) would be acceptable to a much larger subset of Windows users,
that have learned a bit of Python scripting. And those are the ones
that need it. The ones that understand #! have also figured out Windows
assoc and ftype baloney, and have figured out how to switch back and
forth from one version of Python (or anything else). Or have given up
using XXXX.py and simply always invoke scripts with
C:\PythonX.y\python.exe XXXX.py in frustration at the lack of a Python
aid in this area.
The launcher could be simpler if the Python installer placed versioned
Python executables on the PATH. Unfortunately, historically it hasn't.
If it did, would, or the launcher installer would place them there for
pre-existing versions of Python, then the launcher could work by
launching the appropriate version of Python, expecting Windows to find
it on the PATH. The PEP doesn't address the level of internal complexity
of the launcher necessary to find where Python has been installed,
neither for CPython, nor for the alternate implementations to be
supported.
The PEP intentionally doesn't, but the implementation does - it
already does exactly that for CPython. Other implementation may need
a different strategy, but we can cross that bridge when we come to it.
The PEP could therefore be clearer by discussing the CPython
implementation strategy, even if it points out that other
implementations may need a different strategy. Putting PythonX.Y on the
PATH eliminates the need for the launchers (py.exe and pyw.exe) to be on
the path. They are then only needed in the Windows associations, as I
suggested somewhere.
The launcher could be simpler if a directory \usr\bin were created under
Windows Program Files, placed on the PATH, and %ProgramFiles% prepended
to the Unix #! line, with the Python/Jython/Cython installers placing
appropriately versioned executables in that directory. Could even start
a trend for programs ported from Unix. One could even place an "env"
program there, for more simplicity.
Again, I disagree - I think in practice the code would be more
complex, and having Python assert it "owns" such directories and
executables is a can of worms we should avoid.
This was a random thought I had, which went a "more Unix" direction,
instead of the "less Unix" direction of #@. I didn't particularly like
this one either, but it seemed to be a possible alternative, so I didn't
want it to go unmentioned, in case thousands of other Python users
thought it would be a great idea if only they had thought of it... :)
* When used to launch an interactive Python interpreter, the
launcher
will support the first command-line argument optionally be a
version specifier in the form "-n[.n]" (where n is a single
integer) to nominate a specific version be used. For example,
while "py.exe" may locate and launch the latest Python 2.x
implementation installed, a command-line such as "py.exe -3"
could
specify the latest Python 3.x implementation be launched, while
"py.exe -2.6" could specify Python 2.6 be located and launched.
If a Python 2.x implementation is desired to be launched with
the
-3 flag, the command-line would need to be similar to "py.exe -2
-3" (or the specific version of Python could obviously be
launched manually without use of this launcher.)
I think that a python launcher that is "on the PATH" that could be used
to launch an interactive Python, should be different than one that is
used to launch XXXX.py[w] scripts.
I believe that if you know the Python you want is already on the PATH,
you should just use 'python.exe' instead of 'py.exe'. I don't see any
reason to use this launcher for interactive Python sessions where this
is the case.
1) python should be invoked interactively by typing "python" or
"pythonX[.Y]" at the CMD prompt, not "py". This can be done without a
launcher, if appropriate versioned pythons are placed on the PATH. The
launcher is really and only needed for handling XXXX.py[w] scripts,
which, in the Windows way of thinking, can only be associated with one
specific, system-wide configured version of Python (presently, the
latest one wins). The script itself is not examined to modify such an
association. The Unix !# line provides such modification on Unix.
OK, I think we found something we can agree on :) If the python.exe
you want is on your path, just ignore this launcher for anything other
than file associations.
So I think we are agreeing here, mostly. And you mentioned having the
CPython implementation PythonX.Y get on the PATH somehow. So just keep
the python launcher off the PATH, remove its then unnecessary option
syntax, and we agree totally :)
2) If the launcher provides command line options for the "benefit" of
launching interactive Python interpreters, those command line options
can have data puns with script names, or can conflict with actual Python
options. I believe Python 2 already has a -3 option, for example. And
Windows users are not trained that "-" introduces an option syntax, but
rather "/". Most _programmer_ users would probably be aware of "-" as an
option syntax, but Python is used as a language for non-programmers in
some circles, and few Windows non-programmers understand "/" much less
"-" and not even command lines very well. So not using a launcher for
launching interactive Python sidesteps all that: Python itself is
introduced and taught, and no need to teach about (or even have)
launcher options that could possibly conflict and confuse, in addition
to Python options that may conflict with script names already. (I have
seen lots of Windows users use leading punctuation characters on
filenames to affect sort order and grouping of files in My Documents,
not knowing they can create subdirectories/subfolders, or not wanting to
bother with them, since all their applications default to opening things
from My Documents.)
I'm not 100% sure of the points you are trying to make above, but the
gist of it seems to be the same as (1) - use python.exe directly if
you prefer - in which case I agree. Obviously if people choose to use
the new launcher interactively they are doing so because they see some
benefit and would be willing to understand how it works. So I don't
see any problems here as I'm not advocating it would replace
"python.exe" in interactive scenarios except where the user actively
chooses to.
Does anyone see any benefit to using a launcher, if PythonX.Y is on the
PATH? I don't. I see putting PythonX.Y as an extremely useful feature
available for Unix Python that is totally missing from Windows Python,
to its large detriment. But that is not a feature the launcher should
have to supply, but since Python hasn't been, a launcher installer may
need to, to compensate.
3) Unix !# lines can have embedded options after the program name on the
line. Such options would be another source of potential conflict with
launcher options, if the launcher has options for use with launching
interactive interpreters.
Item 3 is also an issue for the PEP even apart from its use as an
interactive Python launcher; since options may exist on the Unix #!
line, a discussion of how and if they are handled by the launcher should
be included in the PEP.
I believe a reference to the execve man-page and the note in the PEP
that we will support the description there, including limitations, is
enough - but I'm happy to change this if people agree it is
underspecified or confusing.
Per the beginning of this email, I think you need to stay far away from
referencing the execve man page in a description of a Windows launcher,
except perhaps as a footnote.
Until there is a benefit seen to using a launcher from the command line,
I think it should (1) not be on the PATH (2) have no option syntax,
which gets things back to a simple description.
* Environment varialbes will be used to override the semantics for
determining exactly what version of Python will be used. For
example, while a shebang line of "/usr/bin/python2" will
automatically locate a Python 2.x implementation, an environment
variable can override exactly which Python 2.x implementation
will
be chosen. Similarly for "/usr/bin/python" etc.
Clarify if environment variables can be used to override semantics for
shebang lines of the form "/usr/bin/python2.x".
On re-reading the PEP, I notice I deleted something which is important
on the mistaken belief the execve man page would make it clear. Only
the first command-line argument will be checked for a shebang line.
If the first argument is an option (ie, starts with a '-'), nothing is
examined for a shebang line. This is what the reference
implementation does and I'll be sure to update the PEP to reflect this.
The PEP does state the optional -N.N argument must be the first
argument. Therefore, the presence of the -N.N argument will avoid any
shebang processing at all so can't impact the version selected via the
shebang line. This makes sense to me as someone explicitly executing
"py.exe -3 foo.py" is explicitly overriding the version they want to
run the script with.
Not using the launcher from the command line, and it not having any
available options, would sidestep this issue nicely.
Here you didn't respond to the question about overriding semantics of
/usr/bin/pythonX.Y, but your reply to Dj did. Please clarify it in the
PEP, however. Thanks.
If alternate implementations are to be supported, additional virtual
commands will be required, not just these two. Each one adds complexity
to the launcher.
This is true - each implementation would need custom code to sniff out
the requested implementation version. I don't think this is a burden
though and can be worked around by using fully-qualified executable
names in the shebang line.
No, the workaround of using fully-qualified executable names in the
shebang line makes the script now dysfunctional on Unix. So it would
not be possible to use CPython on Unix, and IronPython on Windows, nor
jpython on Unix and CPython on Windows, nor any other
cross-implementation scripts, should different implementations be used
on different platforms in a particular environment. This is one of the
reasons I like a separate #@ line for a Windows launcher. It permits
the flexibility of specifying both the implementation and the version
separately for the different platforms, which would be complex to
achieve if both platforms look at the same #! line.
Non-virtual shebang lines should be discouraged as they make the
script specific to a specific Windows installation. However, they
are supported for maximum flexibility.
This is a false statement.
I believe it is a true statement - needing words like "likely" and
"most" to dispute it just demonstrates that IMO. Note that I don't
dispute your statements, but instead believe that "most" simply isn't
good enough - I want "always".
I don't think you'll get "always", because of environments that use
different implementations on different platforms. To get always, you
need the flexibility to specify implementation and version for each
platform separately... two lines. I recall an environment I once worked
in where yacc was used on Unix, and bison on Windows, for example... as
a counter-argument for anyone that wants to jump in a say that such a
split implementation environment would surely never exist. And
differing C implementations between Unix/Windows also, in another
environment. I wasn't using Python in those days, but I recall having
different versions of Perl available between Unix/Windows. Etc.
It may be true that it is better to rely on the Windows path,
specifying #@pythonX.Y instead of #@C:\PythonX.Y\python.exe if
Python, or the launcher installer, can arrange to get all the pythonX.Y
on the PATH.
Unfortunately, it is clear we disagree on some fundamental points and
I can't see a way to change my PEP that would keep both you and I
happy. Therefore, the only reasonable resolution would be for you to
draft a competing PEP and reference implementation which python-dev
can informally vote on.
Let's enumerate... and maybe if we keep talking we'll increase the
agreement list and decrease the disagreement list.
New ideas in this message:
1) If launcher doesn't find a #!/#@ line, it should run python2 to
support scripts without such lines.
Agreements:
1) Python needs PythonX.Y to be on the PATH
2) Need a launcher to handle Windows associations that otherwise only
allow one version for one extension
3) Need a separate launcher for .py and .pyw
4) Support for as many environments as possible, and as many
installations as possible for each environment.
Disagreements:
1) #! only, with "virtual" paths versus #! for Unix, #@ for Windows
to support different implementations and versions on the different
platforms.
2) Use launcher from command line versus use PythonX.Y from the
command line for interactive Python interpreters, don't put launcher on
the PATH, its interactive option just adds complexity, not benefit. (Not
sure if we disagree all that much here, my position is very close to +/-
0 on this one -- just wondering if there is any benefit to the added
complexity of documenting it for command line use)
3) Reference execve man page and expect Windows users to understand it
versus document one paragraph for how a similar feature works on Windows
4) The launcher should be as simple as possible (but no simpler)
versus The launcher should be as useful as possible to a variety of
users, with as little complexity as possible in the documentation.
What else? Can I borrow and tweak your reference implementation? I'm
seriously awash in multiple projects, most of which want to use
cross-platform Python, so I see this topic as extremely important for my
future projects, but haven't written C in so long I'd hate to try to
commit to more than a reference implementation in Python.
Cheers,
Mark
Hopefully you are still cheery :)
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
