Re: MBF for deprecating Python2 usage
On Mon, 7 Aug 2017, Steve Langasek wrote: But I am concerned about possible implementation strategies here. exec() is a very expensive syscall, and python is a frequently used interpreter. If this were implemented as a wrapper that checked isatty(), printed a banner, and then re-execed the real python, that could have a measurable performance impact on some applications. There is a reason /usr/bin/python, /usr/bin/gcc, etc. are always symlinks to the real interpreter on Debian, not wrapper scripts - other distributions have tried to do this as a wrapper script and the result wasn't pretty. Python itself is not very fast to start up, though, so exec is not the dominating cost. Some ad-hoc testing with the current version of pythonmux (which uses the re-exec approach) shows that it takes about 13.8 ms to print("Hello world") with /usr/bin/python and 15.5 ms to print("Hello world") with pythonmux on my 2010-ish laptop. For the sake of argument, I just pushed a branch of pythonmux that links libpython3.5 and calls Py_Main instead of execing python3.5 (that is, if the target interpreter is python3.5, it will not actually exec): https://github.com/geofft/pythonmux/commits/libpython But it's quite a bit _slower_ with this approach (about 26.9 ms per run), regardless of whether I link libpython3.5m.so or libpython3.5m.a. This confuses me and I'm definitely curious to figure this out if that's worth doing, but I think this is getting into a lot of weird complexity to save 1.7 ms on a mere hello world -- for comparison, if I add `import requests` to my hello world program, each run now takes about 342 ms. By the way, I do think I made pythonmux do too much since I wanted to make it useful in the general case; in particular I don't think it's super helpful to support scripts that say "I can run on Python 3.2 but not any newer version". Given the way the discussion is going (both here and on linux-sig) and how close we are to Python 2's end-of-life, I think it's sufficient to check isatty(0) and the shebang line if one exists, call libpython3's Py_Main if that seems reasonable, print a warning and attempt to exec python2 otherwise, and print an error message about how to install python2 if the exec fails. -- Geoffrey Thomas https://ldpreload.com geo...@ldpreload.com
Re: MBF for deprecating Python2 usage
On Tuesday, August 08, 2017 08:58:04 PM Ole Streicher wrote: > Diane Trout writes: > >> What I am opposing is the suggestion to install, in the near to > >> medium > >> term, a command of exactly the same name that has subtly similar but > >> incompatible behaviour, when that behaviour *already* has a command – > >> ‘python3’ – that is widely used by those who need it. > > > > my problem with that plan is all of the printed documentation saying to > > learn python, type "python". > > All *actual* printed documentation? Which refers to Python 3? Could you > give an example here? > > Otherwise, (if using a Python 2 book), one of the next steps would be > "print 'hello world'", and nothing would prevent from the horrible > experience that the book is outdated anyway. > > Sorry, I don't see the problem with "printed documentation". Having recently (late last year) searched for a beginner's Python book for one of my kids, I can tell you that current offers range between "Only a few people use Python 3, so you're better off ignoring this odd duck while you learn" to "Python 3 is the future, so you may as well start be learning the future rather than the past". I don't recall any current books that didn't at least acknowledge the existence of this Python 3 thing. Based on that (admittedly incomplete) survey, I think confusion about the existence of Python/Python 3 is a problem with current dead tree learning Python editions. If you're trying to learn something new with old documentation, I don't think it should be surprising there are a few bumps along the way. I agree. Scott K
Re: MBF for deprecating Python2 usage
Diane Trout writes: >> What I am opposing is the suggestion to install, in the near to >> medium >> term, a command of exactly the same name that has subtly similar but >> incompatible behaviour, when that behaviour *already* has a command – >> ‘python3’ – that is widely used by those who need it. >> > > my problem with that plan is all of the printed documentation saying to > learn python, type "python". All *actual* printed documentation? Which refers to Python 3? Could you give an example here? Otherwise, (if using a Python 2 book), one of the next steps would be "print 'hello world'", and nothing would prevent from the horrible experience that the book is outdated anyway. Sorry, I don't see the problem with "printed documentation". Best regards Ole
Re: MBF for deprecating Python2 usage
Steve Langasek writes: > FWIW I don't share Ben's concern about this being a "backwards > incompatible" change (who is relying on the precise contents of the > banner when running 'python' interactively?!). That's not a backward incompatibility anyone would seriously rely on, and I don't have that concern. The backward incompatibility I'm concerned about is: to have ‘python’, which today invokes Python 2 and some uncountable number of scripts rely on, change on a near-to-medium future upgrade to instead invoke the backward-incompatible Python 3 hence breaking those scripts. Merely noting a change in the interactive prompt's banner is, I agree, inconsequential by comparison. So that's not the objection. -- \“All opinions are not equal. Some are a very great deal more | `\ robust, sophisticated, and well supported in logic and argument | _o__) than others.” —Douglas Adams | Ben Finney
Re: MBF for deprecating Python2 usage
On Mon, Aug 07, 2017 at 08:11:40PM -0700, Diane Trout wrote: > > What I am opposing is the suggestion to install, in the near to > > medium > > term, a command of exactly the same name that has subtly similar but > > incompatible behaviour, when that behaviour *already* has a command – > > ‘python3’ – that is widely used by those who need it. > my problem with that plan is all of the printed documentation saying to > learn python, type "python". > At the very least there needs to a usr/bin/python that prints > instructions about what you should run. FWIW I don't share Ben's concern about this being a "backwards incompatible" change (who is relying on the precise contents of the banner when running 'python' interactively?!). But I am concerned about possible implementation strategies here. exec() is a very expensive syscall, and python is a frequently used interpreter. If this were implemented as a wrapper that checked isatty(), printed a banner, and then re-execed the real python, that could have a measurable performance impact on some applications. There is a reason /usr/bin/python, /usr/bin/gcc, etc. are always symlinks to the real interpreter on Debian, not wrapper scripts - other distributions have tried to do this as a wrapper script and the result wasn't pretty. Avoiding the performance hit would require that any changes to the banner be made in the python source itself. -- Steve Langasek Give me a lever long enough and a Free OS Debian Developer to set it on, and I can move the world. Ubuntu Developerhttp://www.debian.org/ slanga...@ubuntu.com vor...@debian.org signature.asc Description: PGP signature
Re: MBF for deprecating Python2 usage
On Aug 7, 2017, at 20:52, Ben Finney wrote: > > So IMO the conservative assumption is that they expect Debian to provide > the stability it is famous for, and not to break the behaviour of > commands unnecessarily. There’s another dimension to that too: it’s people who expect no changes to the command called /usr/bin/python *and* want to track the latest and greatest Debian releases. Maybe that changes the equation and maybe not. Certainly any large organization with tons of Python 2 laying around is going to be much more conservative about upgrading their entire distro than individual developers who just use unstable and get whatever today’s snapshot looks like. And to reiterate, I’m not arguing that /usr/bin/python change today or even at the end of Pycon 2020 when 2.7 is officially EOL’d. I’m saying that it *will* change some day and that we should plan that change, if for no other reason than to set the expectations of exactly those folks who are not part of this conversation. Cheers, -Barry signature.asc Description: Message signed with OpenPGP
Re: MBF for deprecating Python2 usage
Diane Trout writes: > On Tue, 2017-08-08 at 13:24 +1000, Ben Finney wrote: > > > > Those people, not party [to] this conversation, have reasonable > > expectation that such breakage will not happen without very good > > reason. > > Good reason would entail, as an example, that there is no better > > alternative. > > Why not ask? Ask whom? Because the issue is about the expectations of those who *don't want* to be bothered by technical discussions like this, asking those few who make themselves available for such questions will not resolve the issue. At best, we would have a collection of anecdotes and no idea how representative they are of those who choose not to be contactable (but still expect Debian to avoid breaking things unnecessarily). At worst, we would have a collection of anecdotes that would be a strong temptation for *falsely assuming* they represent the silent majority. Barry, earlier in this thread, was correct when he pointed out that, by the nature of the issue, we *can't* know with confidence what the majority of people expect about this change. So IMO the conservative assumption is that they expect Debian to provide the stability it is famous for, and not to break the behaviour of commands unnecessarily. -- \ “Some people have a problem, and they think “I know, I'll use | `\ Perl!”. Now they have some number of problems but they're not | _o__) sure whether it's a string or an integer.” —Benno Rice, 2011 | Ben Finney
Re: MBF for deprecating Python2 usage
On Tue, 2017-08-08 at 13:24 +1000, Ben Finney wrote: > > Those people, not party ot this conversation, have reasonable > expectation that such breakage will not happen without very good > reason. > Good reason would entail, as an example, that there is no better > alternative. > Why not ask? I know people in training groups like software carpentry? Does anyone know sites with large python deployments? I'd love to know what Apple is going to do about this issue. Also if we do come up with a migration it should happen at a release and be clearly reported in the release notes. Diane
Re: MBF for deprecating Python2 usage
Diane Trout writes: > my problem with that plan is all of the printed documentation saying > to learn python, type "python". I agree that's a problem. I don't agree that making a backward incompatible breakage is justified by widespread documentation giving bad advice. -- \ “There are no chaplains in foxholes.” —Sergeant Justin | `\ Griffith, 2011-07-27 | _o__) | Ben Finney
Re: MBF for deprecating Python2 usage
Diane Trout writes: > Personally, I'm ready for python to point to python3 now. I appreciate that. In many of my hours I even concur. Both of us can have that, for our own personal environment. That doesn't answer the question of changing the behaviour of the *default* ‘python’ command installed by Debian packages, for people who have not asked for this breaking change. Those people, not party ot this conversation, have reasonable expectation that such breakage will not happen without very good reason. Good reason would entail, as an example, that there is no better alternative. The use cases provided in favour of “make an incompatible change in what the ‘python’ command does, by default” so far do not meet that standard. People who *actively want* ‘python’ to mean Python 3 can set up their environment that way today, without affecting others who have not asked for anything to change. So the desires of those people – and I can even count myself one of them! – should weigh less than those who expect the stability promised by Debian. > Python has been remarkably good at avoiding breakage, but I've seen > other scripting languages have serious incompatibilities with far less > warning. That is not an argument for changes that break default behaviour. > It might be useful to add an option to the interpreter where if a > python script is launched without a usr/bin/python2 or usr/bin/python3 > header it reports a deprecation warning (either to console or syslog) > so it's easier to find programs that still need updating. That sounds good to me. -- \ “I was stopped by the police for speeding; they said ‘Don't you | `\ know the speed limit is 55 miles an hour?’ I said ‘Yeah I know, | _o__) but I wasn't going to be out that long.’” —Steven Wright | Ben Finney
Re: MBF for deprecating Python2 usage
> What I am opposing is the suggestion to install, in the near to > medium > term, a command of exactly the same name that has subtly similar but > incompatible behaviour, when that behaviour *already* has a command – > ‘python3’ – that is widely used by those who need it. > my problem with that plan is all of the printed documentation saying to learn python, type "python". At the very least there needs to a usr/bin/python that prints instructions about what you should run.
Re: MBF for deprecating Python2 usage
> What tearing need is there to change what the command ‘python’ does, > in > a backward-incompatible way? Personally, I'm ready for python to point to python3 now. I'm tired of writing python 2/3 compatible code because someone _might_ launch a script with "python my_python3_script.py instead of ./my_python3_script.py Python has been remarkably good at avoiding breakage, but I've seen other scripting languages have serious incompatibilities with far less warning. For example R packages need to be recompiled for point release updates. Some perl scripts we inherited broke somewhere between perl 5.8 and 5.12. Users have tools like virtualenv or conda to deal with the incompatibilities. It might be useful to add an option to the interpreter where if a python script is launched without a usr/bin/python2 or usr/bin/python3 header it reports a deprecation warning (either to console or syslog) so it's easier to find programs that still need updating. Diane
Re: MBF for deprecating Python2 usage
Geoffrey Thomas writes: > We will not indefinitely provide a /usr/bin/python that runs Python 2; > we probably will do so for at most one more stable release. I'm fine with removing the ‘python’ command, because that command means Python 2 and Python 2 is going away. What I am opposing is the suggestion to install, in the near to medium term, a command of exactly the same name that has subtly similar but incompatible behaviour, when that behaviour *already* has a command – ‘python3’ – that is widely used by those who need it. -- \ “From the moment I picked your book up until I laid it down I | `\was convulsed with laughter. Someday I intend reading it.” | _o__)—Groucho Marx | Ben Finney
Re: MBF for deprecating Python2 usage
Barry Warsaw writes: > On Aug 7, 2017, at 18:28, Ben Finney wrote: > > > That day [when we know that the vast majority of scripts in the wild > > no longer expect Python 2 when invoking the ‘python’ command] might > > never come, in which case the ‘python’ command will forever mean > > Python 2. That is, I'm saying, better than breaking that command in > > the near or medium future. > > I wonder if you’ll say the same thing in 2030 when Python 2.7 is 10 > years dead, can literally cannot be built on any then-modern Debian, > let alone Linux distro? Why do you wonder that, in the context of this proposal? That proposal is the context of my statement above. In the context of the proposal now before us, to change the behaviour of the ‘python’ command in the near-to-medium term, it seems irrelevant to wonder what I'll say on the matter in 2030. Of course, I'm open to Python 2 dwindling into the past; it is inevitable as the years go by. But in 2020 that won't be the case, and we should be very conservative about declaring it to be the case because – as you point out – even in 2020 we cannot be confident that we'll know what the vast majority of scripts out there expect. > Sure, today. And tomorrow. And many tomorrows until at least 2020. > Then I think we’ll be having very different conversations about it. > Anybody who will still be running Python 2.7 code will have some > difficult choices to make. 1) port to Python 3; 2) run old versions of > distros that still have Python 2.7 around and live with the > unsupported nature of it; 3) roll their own Python 2.7s; 4) pay some > commercial vendors to keep their stuff alive and secure. I see no good reason to make that job *more difficult* by wantonly changing what interpreter is invoked with the ‘python’ command. Especially because it isn't even something needed for Python 3 to flourish. What tearing need is there to change what the command ‘python’ does, in a backward-incompatible way? We already have a ‘python3’ command. Scripts written today must explicitly choose ‘python3’ to get that version of the language. Learners learning today must explicitly ask for Python version 3. > IMHO, there will absolutely be a day when it makes no sense to point > /usr/bin/python to Python 2. We agree on that fact. There is some disagreement about how many people, whose voices are not part of this conversation, can be dismissed before that day arrives. And when they can reasonably be dismissed. -- \ “We demand rigidly defined areas of doubt and uncertainty!” | `\—Vroomfondel, _The Hitch-Hiker's Guide To The Galaxy_, Douglas | _o__)Adams | Ben Finney
Re: MBF for deprecating Python2 usage
On Tue, 8 Aug 2017, Ben Finney wrote: The issue is: Invoking ‘python’ today gets an interpereter, Python 2, that will work with some code and not others. It should not tomorrow invoke an incompatible interpreter without *knowing* that the vast majority of scripts in the wild no longer expect Python 2 to come from that command. That day might never come, in which case the ‘python’ command will forever mean Python 2. That is, I'm saying, better than breaking that command in the near or medium future. I'm saying it's a bad idea for ‘python’ tomorrow to get an incompatible intepreter that won't run the same Python code. That interpreter is named ‘python3’ and should never be installed as ‘python’, because ‘python’ is a command that many scripts expect to invoke Python 2. This conflates the role of the interactive Python interpreter (which is what Diane was referring to) and a script interpreter. #!/usr/bin/python and `python -c` are ABI referring to Python 2, yes, and can't start running Python 3 any more than /lib/libc.so.5 can start being glibc. But there's no reason this should prevent typing "python" from giving you a Python 3 interpreter. Since I think it hasn't been linked recently on this list, I'll re-link this project, which separates the two roles: https://github.com/geofft/pythonmux https://ldpreload.com/blog/usr-bin-python-23 Keep in mind that we do intend to break the ability for /usr/bin/python to run Python 2 code in less than three years (unless we're interested in providing security support for Python 2 when it's dead upstream, which I hope nobody wants to do). We will not indefinitely provide a /usr/bin/python that runs Python 2; we probably will do so for at most one more stable release. The only question is whether it should break by giving you an error message (either ENOENT or something printed on stderr), or break by trying to run the code as Python 3. -- Geoffrey Thomas https://ldpreload.com geo...@ldpreload.com
Re: MBF for deprecating Python2 usage
On Aug 7, 2017, at 18:28, Ben Finney wrote: > The issue is: Invoking ‘python’ today gets an interpereter, Python 2, > that will work with some code and not others. It should not tomorrow > invoke an incompatible interpreter without *knowing* that the vast > majority of scripts in the wild no longer expect Python 2 to come from > that command. That’s unknowable. We have no idea what people do with any package of Debian outside of the archive. We can’t even quantify what “vast” means. So under the above definition, you’re right, that day will never come. > That day might never come, in which case the ‘python’ command will > forever mean Python 2. That is, I'm saying, better than breaking that > command in the near or medium future. I wonder if you’ll say the same thing in 2030 when Python 2.7 is 10 years dead, can literally cannot be built on any then-modern Debian, let alone Linux distro? That’s not hyperbole; one of the current upstream exceptions to the 2.7 branch is to adapt to newer build systems and library versions. That will stop in 2020, but of course everything else will continue to evolve. It might even come sooner. Once Python 2.7 is EOL’d, it won’t even get source releases, and may not even get security releases. If folks want to keep it alive, it’ll take active downstream resources to maintain. People may pay for commercial support from RedHat, Canonical, and others, but how much will volunteers want to keep it secure and working? I can remember when ‘python’ meant Python 1.5.2. So that name doesn’t have to be tied to a specific major version. > I'm saying it's a bad idea for ‘python’ tomorrow to get an incompatible > intepreter that won't run the same Python code. That interpreter is > named ‘python3’ and should never be installed as ‘python’, because > ‘python’ is a command that many scripts expect to invoke Python 2. Sure, today. And tomorrow. And many tomorrows until at least 2020. Then I think we’ll be having very different conversations about it. Anybody who will still be running Python 2.7 code will have some difficult choices to make. 1) port to Python 3; 2) run old versions of distros that still have Python 2.7 around and live with the unsupported nature of it; 3) roll their own Python 2.7s; 4) pay some commercial vendors to keep their stuff alive and secure. IMHO, there will absolutely be a day when it makes no sense to point /usr/bin/python to Python 2. Cheers, -Barry signature.asc Description: Message signed with OpenPGP
Re: MBF for deprecating Python2 usage
Diane Trout writes: > My suggestion was "the startup banner should print what command to run > to get Python 2." Thanks for the suggestion. I think it's a bad idea. > I was thinking of the case of the end-user trying to follow a Python > tutorial. They'd still need to exit and run the python2 command if they > wanted 2. That's not the issue. The issue is: Invoking ‘python’ today gets an interpereter, Python 2, that will work with some code and not others. It should not tomorrow invoke an incompatible interpreter without *knowing* that the vast majority of scripts in the wild no longer expect Python 2 to come from that command. That day might never come, in which case the ‘python’ command will forever mean Python 2. That is, I'm saying, better than breaking that command in the near or medium future. I'm saying it's a bad idea for ‘python’ tomorrow to get an incompatible intepreter that won't run the same Python code. That interpreter is named ‘python3’ and should never be installed as ‘python’, because ‘python’ is a command that many scripts expect to invoke Python 2. -- \ “Reality must take precedence over public relations, for nature | `\ cannot be fooled.” —Richard P. Feynman, _Rogers' Commission | _o__) Report into the Challenger Crash_, 1986-06 | Ben Finney
Re: MBF for deprecating Python2 usage
> I disagree, it's a bad idea to actively take steps to make the same > command invoke *incompatible* programs depending on the time and > host. My suggestion was "the startup banner should print what command to run to get Python 2." I was thinking of the case of the end-user trying to follow a Python tutorial. They'd still need to exit and run the python2 command if they wanted 2. Diane
Re: MBF for deprecating Python2 usage
Barry Warsaw writes: > On Aug 7, 2017, at 14:52, Diane Trout wrote: > > Perhaps when launching via the command "python" the interpreter could > > hint python2 was available? Something like: > > > > $ python > > Python 3.5.4rc1 (default, Jul 25 2017, 08:53:34) > > [GCC 6.4.0 20170704] on linux > > Type "python2" for Python 2.7.13 > > Type "help", "copyright", "credits" or "license" for more information. > > Good idea Diane! I disagree, it's a bad idea to actively take steps to make the same command invoke *incompatible* programs depending on the time and host. (good sigmonster; have a cookie.) -- \ “I've always wanted to be somebody, but I see now that I should | `\ have been more specific.” —Jane Wagner, via Lily Tomlin | _o__) | Ben Finney
Re: MBF for deprecating Python2 usage
On Fri, 4 Aug 2017, Scott Kitterman wrote: If the primary concern is what happens when a user types "python", then can we address that in command-not-found and leave /usr/bin/python out of it? Debian doesn't ship command-not-found by default. One other approach here would be to have /usr/bin/python exec python3 if it's run interactively -- that is, if isatty(0) -- and print an error message and exit otherwise. I don't think it is too bad to print an error message instead of failing the exec, since that's equivalent to what happens if you're missing a shared library or something: the exec succeeds, but the dynamic loader prints an error and exits nonzero. Ship this /usr/bin/python with python3-minimal, and have the python-minimal package divert it and replace it with the current /usr/bin/python (a symlink to python2.7). This way we can build systems with Python 3 only that behave reasonably, but also allow doing `apt-get install python` to cause /usr/bin/python to start working again. However, I feel like another reason to care about /usr/bin/python is so that sysadmins responsible for multiple platforms can continue writing #!/usr/bin/python (or #!/usr/bin/env python) scripts and use the same script on all their machines. If I can't do this and need platform-specific scripts, I might as well use some other language or ship compiled binaries. Keeping Python viable for this use case is a large part of why I originally proposed pythonmux: I can write scripts that declare that they're polyglot Python 2/3 and work on both Python 2-only and Python 3-only systems, but a Python 2 script using #!/usr/bin/python will give me an error (as it should!) if there's no Python 2 installed on the machine. Last time I checked (2015), entirely reasonable versions of RHEL and macOS both shipped only Python 2 and not Python 3. Is it likely to be the case that all UNIXy OSes within security support in 2019 or 2020 will have a /usr/bin/python3 (preferably 3.3+)? If so, that's a viable option. I just want my best migration path from #!/usr/bin/python to not involve rewriting my scripts in Go. -- Geoffrey Thomas https://ldpreload.com geo...@ldpreload.com
Re: MBF for deprecating Python2 usage
On Aug 7, 2017, at 14:52, Diane Trout wrote: > Perhaps when launching via the command "python" the interpreter could > hint python2 was available? Something like: > > $ python > Python 3.5.4rc1 (default, Jul 25 2017, 08:53:34) > [GCC 6.4.0 20170704] on linux > Type "python2" for Python 2.7.13 > Type "help", "copyright", "credits" or "license" for more information. Good idea Diane! -Barry signature.asc Description: Message signed with OpenPGP
Re: MBF for deprecating Python2 usage
> * Plan for a date at which /usr/bin/python will point to Python 3. I > know that’s the most controversial bit, but I do think that as time > goes on and we’re past 2020, it will be the choice that gives our > users the best experience. I agree the default should change. Perhaps when launching via the command "python" the interpreter could hint python2 was available? Something like: $ python Python 3.5.4rc1 (default, Jul 25 2017, 08:53:34) [GCC 6.4.0 20170704] on linux Type "python2" for Python 2.7.13 Type "help", "copyright", "credits" or "license" for more information. >>> Diane signature.asc Description: This is a digitally signed message part
Re: MBF for deprecating Python2 usage
Hi Matthias, On 08/03/2017 11:57 PM, Matthias Klose wrote: > It might not be possible to drop Python2 for the next release, Even if all Python-related packages in Debian were compatible with Python3, I don't think it's a good idea to drop Python2 support in Buster, there are still far too many third-party users of Python2 out there. I also don't think there has been enough communication to users of Debian that Python2 is going to be removed at some point - this has mainly been discussed in development channels so far. And while there's writing on the wall due to upstream's EOL of Python2 I do get the impression that many people are not aware of that. Therefore I would strongly recommend to keep Python2 in Buster, but add a big fat comment to the release notes that it will be the last Debian release that will support Python2 - so that it can be dropped in Bullseye. That said, I do think we should strive to get rid of all the Python2-only packages in Buster if possible, so I do appreciate your efforts in this regard. Regards, Christian
Re: MBF for deprecating Python2 usage
On Thu, Aug 03, 2017 at 05:57:01PM -0400, Matthias Klose wrote: > While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be > done > to deprecate Python2 usage within the distribution. It might not be possible > to > drop Python2 for the next release, but there are still too many issues with > packages. For now we identified some categories which need fixing. These are > documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, resulting > in > more than 3000 bug reports. That's a high number, on the other hand we won't > make much progress if the issues are not named. My intent is to bring that up > in the Python BoF next week at DebConf and then filing these bug reports with > the user tags mentiond on the wiki page. For python3-app I would actually consider it a bad idea to make it visible at this point in time (except for native packages): I can clearly see how for example python3-package from your list is an issue that can become painful for users in buster, but whether a program uses Python2, Python3 or Perl is usually irrelevant for the user. In 2020 when Python2 is already EOL and other distributions have already removed Python2, Debian will get this in most cases automatically just by upgrading to the latest upstream version. Trying to do anything about python3-app already in 2017 would only create extra work with close to zero benefits to users of Debian. And what should a maintainer do based on a python3-app bug or lintian warning? "Please convert to Python3, the 2021 release of Debian will drop Python2" sounds hilarious if sent to upstream today. > Matthias cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed
Re: MBF for deprecating Python2 usage
On Thu, Aug 03, 2017 at 05:57:01PM -0400, Matthias Klose wrote: > While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be > done > to deprecate Python2 usage within the distribution. It might not be possible > to > drop Python2 for the next release, but there are still too many issues with > packages. For now we identified some categories which need fixing. These are > documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, resulting > in > more than 3000 bug reports. That's a high number, on the other hand we won't > make much progress if the issues are not named. My intent is to bring that up > in the Python BoF next week at DebConf and then filing these bug reports with > the user tags mentiond on the wiki page. I do not see the benefits of the MBF. Visibility will already be provided both to you and to package maintainers by lintian, with automated checking by lintian actually more reliable than information in the BTS. And maintainers who ignore lintian results in their packages tend to be the same people who ignore non-RC bugs in their packages. > Matthias cu Adrian -- "Is there not promise of rain?" Ling Tan asked suddenly out of the darkness. There had been need of rain for many days. "Only a promise," Lao Er said. Pearl S. Buck - Dragon Seed
Re: MBF for deprecating Python2 usage
On August 4, 2017 9:37:18 PM EDT, Barry Warsaw wrote: >Scott Kitterman wrote: > >> If the primary concern is what happens when a user types "python", >then can we >> address that in command-not-found and leave /usr/bin/python out of >it? > >I'm definitely willing to for now. It's clearly not time to remove the >link or point it elsewhere anyway. I think it'll still be a good idea >some day, but I'll leave it at that. > >Can we agree though to start using /usr/bin/python2 in the shebang line >instead of /usr/bin/python, for system installed scripts? > >-Barry I think that ship has already sailed, so sure, but I don't think it's anything more than a wishlist bug. Scott K
Re: MBF for deprecating Python2 usage
Scott Kitterman wrote: > If the primary concern is what happens when a user types "python", then can > we > address that in command-not-found and leave /usr/bin/python out of it? I'm definitely willing to for now. It's clearly not time to remove the link or point it elsewhere anyway. I think it'll still be a good idea some day, but I'll leave it at that. Can we agree though to start using /usr/bin/python2 in the shebang line instead of /usr/bin/python, for system installed scripts? -Barry signature.asc Description: OpenPGP digital signature
Re: MBF for deprecating Python2 usage
On Friday, August 04, 2017 10:13:00 AM ba...@debian.org wrote: > On Aug 3, 2017, at 23:23, Scott Kitterman wrote: > > Read it. I remain completely convinced that /usr/bin/python pointing at a > > python3 version is utterly wrong and a disservice to our users. Even > > after we remove python2.7, people will be locally compiling it and using > > it for a decade. > I think it’s inevitable, and not doing so *eventually* will be a greater > disservice. Sure, not today, maybe not in 2020, but I don’t think we’ll be > having this discussion in 2025. If by then, 5 years after Python 2.7 EOL, > users read on SO or some Learning Python site that they can just type > ‘python' to get an interactive prompt, but it doesn’t work on Debian, > they’ll think Debian is broken, not Python. So I think it makes sense to > consider how the transition will work. > > You’re right that folks will still need Python 2 after 2020, and that their > best option may be to build it themselves, but I don’t think they’ll be > building and installing some old Debian package. Instead they may build > from source, which means they’ll be installing it into /usr/local/bin not > /usr/bin, and they’ll have to change their shebang lines anyway. And there > will come a time when even Python 2.7 is unbuildable as toolchains and > libraries evolve. Upstream will stop tracking those changes so even the > upstream git repo won’t give you buildable source. There may be a bunch of > third party forks, but I don’t think it’s our responsibility to ensure that > Debian aligns with any of those hypotheticals. Folks needing to stay on > Python 2 will probably also just elect to not upgrade their OS, so what we > do in future releases won’t matter to them. > > Upstream and Debian already install a `python2` alias for `python`, and I > think we need to join the chorus that in the future, the way to run Python > 2 is via `python2`. Yes, people will have legacy stuff around for a long > time, but it’s better to begin the transition now rather than make a major > break years from now. > > Such a change would be actively user hostile. > > > > When python2.7 goes away, /usr/bin/python should go too. > > Maybe in the short term (as Matthias suggests), but I’m not so sure it’s > best to disappear /usr/bin/python forever. /usr/bin/python is not just the > common shebang (and we should be actively transitioning to /usr/bin/python2 > for that), but it’s also the command line UI for people wanting to learn > Python, or just wanting to get an interactive interpreter to play with. > There will be a day when, if they get a command not found when typing > `python` at a shell prompt, they will wonder why Debian is broken. > > I think the transition should roughly be: > > * Update policy and tooling so that any scripts that require Python 2 use > /usr/bin/python2 in their shebang. > > * Educate our users that they should be using `python2` for anything that > has not been ported. > > * Port as much as possible to Python 3 (eventually, everything maintained in > Debian) and /usr/bin/python3 in their shebang. > > * Plan for a date at which /usr/bin/python will point to Python 3. I know > that’s the most controversial bit, but I do think that as time goes on and > we’re past 2020, it will be the choice that gives our users the best > experience. > > The discussion on linux-sig is a way to align the entire Python Linux > ecosystem to the eventual goal, giving individual distros the leeway they > need to time such transitions as they see fit to best serve their users. > I’m hoping more Debian Pythonistas will join that discussion, otherwise the > outcome will be heavily weighted toward other Linux distros and Debian may > find itself without a voice in the matter. PEP-394 convinced me that upstream wasn't concerned about the same things Debian is concerned about. Even though they did manage to say in the PEP: > The main barrier to a distribution switching the python command from python2 > to python3 isn't breakage within the distribution, but instead breakage of > private third party scripts developed by sysadmins and other users. > Updating the python command to invoke python3 by default indicates that a > distribution is willing to break such scripts with errors that are > potentially quite confusing for users that aren't yet familiar with the > backwards incompatible changes in Python 3. they don't actually consider the implications of that statement. The current discussion only seems to compound the error. I don't see a lot of value in personally expending time on the upstream discussion. If the primary concern is what happens when a user types "python", then can we address that in command-not-found and leave /usr/bin/python out of it? Scott K signature.asc Description: This is a digitally signed message part.
Re: MBF for deprecating Python2 usage
Ole Streicher wrote: > It is very usual to use "#!/usr/bin/env python" as shebang, exactly for > the case that python is not installed in /usr/bin. Sure, but then all bets are off. The script so shebanged can't assume anything about $PATH so it gets whatever it gets. Using /usr/bin/env in system provided scripts has been known to break stuff, so it's a very bad idea. Plus, if that's what you use for your own scripts, then in the future you'll be in luck because all you'd have to do is change your $PATH to point to wherever your custom built and installed Python 2 lives first, and then you'll get exactly the version you want. > That is -- at least in my environmet -- "ipython", or (for Python 3) > "ipython3". > > IMO it would be better to communicate that the best way for an > interactive session is "ipython3" (or "python3", if you insist). I would > wonder when todays tutorials (that cover Python 3) recommend to use > plain "python". But ipython doesn't come with upstream Python so that would require that Something Extra be installed. That's generally why Getting Started guides and tutorials usually recommend the built-in interactive prompt. >> * Port as much as possible to Python 3 (eventually, everything >> maintained in Debian) and /usr/bin/python3 in their shebang. > I disagree here, and I don't see an advantage over letting > /usr/bin/python just die with Python 2. /usr/bin/python didn't die with Python 2 (who else remembers 1.5.2 as an effectively LTS release? :) and I don't believe it should with Python 3. Maybe go on vacation, but eventually return home. > I would also like to point PEP 394, which has a number of arguments why > /usr/bin/python should remain as Python 2. > > https://www.python.org/dev/peps/pep-0394/ Right, but the ecosystem has evolved since then, so the discussions going on in linux-sig and the *draft* PR at https://github.com/python/peps/pull/315 are about updating the recommendations now that we're somewhere less than three years away from Python 2 EOL. Fedora is moving forward with or without us. I bet that other Linux distros will too. We can certainly do our own thing, but I think that'll make Debian the odd distro out eventually. Cheers, -Barry
Re: MBF for deprecating Python2 usage
ba...@debian.org writes: > On Aug 3, 2017, at 23:23, Scott Kitterman wrote: > You’re right that folks will still need Python 2 after 2020, and that > their best option may be to build it themselves, but I don’t think > they’ll be building and installing some old Debian package. Instead > they may build from source, which means they’ll be installing it into > /usr/local/bin not /usr/bin, and they’ll have to change their shebang > lines anyway. It is very usual to use "#!/usr/bin/env python" as shebang, exactly for the case that python is not installed in /usr/bin. >> Such a change would be actively user hostile. >> >> When python2.7 goes away, /usr/bin/python should go too. > > Maybe in the short term (as Matthias suggests), but I’m not so sure > it’s best to disappear /usr/bin/python forever. /usr/bin/python is > not just the common shebang (and we should be actively transitioning > to /usr/bin/python2 for that), but it’s also the command line UI for > people wanting to learn Python, or just wanting to get an interactive > interpreter to play with. That is -- at least in my environmet -- "ipython", or (for Python 3) "ipython3". > There will be a day when, if they get a command not found when typing > `python` at a shell prompt, they will wonder why Debian is broken. IMO it would be better to communicate that the best way for an interactive session is "ipython3" (or "python3", if you insist). I would wonder when todays tutorials (that cover Python 3) recommend to use plain "python". > I think the transition should roughly be: > > * Update policy and tooling so that any scripts that require Python 2 > use /usr/bin/python2 in their shebang. > > * Educate our users that they should be using `python2` for anything > that has not been ported. > > * Port as much as possible to Python 3 (eventually, everything > maintained in Debian) and /usr/bin/python3 in their shebang. I disagree here, and I don't see an advantage over letting /usr/bin/python just die with Python 2. I would also like to point PEP 394, which has a number of arguments why /usr/bin/python should remain as Python 2. https://www.python.org/dev/peps/pep-0394/ Best regards Ole
Re: MBF for deprecating Python2 usage
On Aug 3, 2017, at 23:23, Scott Kitterman wrote: > > Read it. I remain completely convinced that /usr/bin/python pointing at a > python3 version is utterly wrong and a disservice to our users. Even after > we remove python2.7, people will be locally compiling it and using it for a > decade. I think it’s inevitable, and not doing so *eventually* will be a greater disservice. Sure, not today, maybe not in 2020, but I don’t think we’ll be having this discussion in 2025. If by then, 5 years after Python 2.7 EOL, users read on SO or some Learning Python site that they can just type ‘python' to get an interactive prompt, but it doesn’t work on Debian, they’ll think Debian is broken, not Python. So I think it makes sense to consider how the transition will work. You’re right that folks will still need Python 2 after 2020, and that their best option may be to build it themselves, but I don’t think they’ll be building and installing some old Debian package. Instead they may build from source, which means they’ll be installing it into /usr/local/bin not /usr/bin, and they’ll have to change their shebang lines anyway. And there will come a time when even Python 2.7 is unbuildable as toolchains and libraries evolve. Upstream will stop tracking those changes so even the upstream git repo won’t give you buildable source. There may be a bunch of third party forks, but I don’t think it’s our responsibility to ensure that Debian aligns with any of those hypotheticals. Folks needing to stay on Python 2 will probably also just elect to not upgrade their OS, so what we do in future releases won’t matter to them. Upstream and Debian already install a `python2` alias for `python`, and I think we need to join the chorus that in the future, the way to run Python 2 is via `python2`. Yes, people will have legacy stuff around for a long time, but it’s better to begin the transition now rather than make a major break years from now. > Such a change would be actively user hostile. > > When python2.7 goes away, /usr/bin/python should go too. Maybe in the short term (as Matthias suggests), but I’m not so sure it’s best to disappear /usr/bin/python forever. /usr/bin/python is not just the common shebang (and we should be actively transitioning to /usr/bin/python2 for that), but it’s also the command line UI for people wanting to learn Python, or just wanting to get an interactive interpreter to play with. There will be a day when, if they get a command not found when typing `python` at a shell prompt, they will wonder why Debian is broken. I think the transition should roughly be: * Update policy and tooling so that any scripts that require Python 2 use /usr/bin/python2 in their shebang. * Educate our users that they should be using `python2` for anything that has not been ported. * Port as much as possible to Python 3 (eventually, everything maintained in Debian) and /usr/bin/python3 in their shebang. * Plan for a date at which /usr/bin/python will point to Python 3. I know that’s the most controversial bit, but I do think that as time goes on and we’re past 2020, it will be the choice that gives our users the best experience. The discussion on linux-sig is a way to align the entire Python Linux ecosystem to the eventual goal, giving individual distros the leeway they need to time such transitions as they see fit to best serve their users. I’m hoping more Debian Pythonistas will join that discussion, otherwise the outcome will be heavily weighted toward other Linux distros and Debian may find itself without a voice in the matter. Cheers, -Barry signature.asc Description: Message signed with OpenPGP
Re: MBF for deprecating Python2 usage
On August 4, 2017 6:49:23 AM EDT, Matthias Klose wrote: >On 03.08.2017 21:08, ba...@debian.org wrote: >> On Aug 3, 2017, at 17:57, Matthias Klose wrote: >>> >>> While at DebCamp, Stefano Rivera and I sat down to analyze what >needs to be done >>> to deprecate Python2 usage within the distribution. It might not be >possible to >>> drop Python2 for the next release, but there are still too many >issues with >>> packages. For now we identified some categories which need fixing. >These are >>> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, >resulting in >>> more than 3000 bug reports. That's a high number, on the other hand >we won't >>> make much progress if the issues are not named. My intent is to >bring that up >>> in the Python BoF next week at DebConf and then filing these bug >reports with >>> the user tags mentiond on the wiki page. >> >> Great to hear that you guys talked about it. >> >> Just a quick note that PEP 394 discussions have revived, lead by the >Fedora folks. Please do check out the new thread, especially if you >have opinions about what /usr/bin/python should do once Python 2.7 is >EOL. >> >> https://mail.python.org/pipermail/linux-sig/2017-August/thread.html > >I replied to this thread. I think there should be one release which is >not >shipping /usr/bin/python before /usr/bin/python should be reused and >pointed at >python (>> 2). This should be good enough to get all scripts actively >converted >which are not part of the distribution. If that release is buster, we >should >require the use of python2 instead of python now, document that in >policy and >provide a lintian check for that. > >Matthias I disagree. There will be users whose reaction to removal of python2.7 will be to compile their own. Reintroducing /usr/bin/python as a python3 version risks their systems for no benefit (since all python3 stuff points to /usr/bin/python3 and works fine). Just let it go and don't bring it back. As far as when to drop it goes, I believe that the python-django maintainers are planning on releasing Buster with Django 1.11 as bilingual python2/python3. Then they'll go to Django 2, which is python3 only, early in the Bullseye cycle. For the chunk of the Python ecosystem that is built around Django, keeping python2.7 in for Buster would be a significant aid to transition. Scott K
Re: MBF for deprecating Python2 usage
On 03.08.2017 21:08, ba...@debian.org wrote: > On Aug 3, 2017, at 17:57, Matthias Klose wrote: >> >> While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be >> done >> to deprecate Python2 usage within the distribution. It might not be >> possible to >> drop Python2 for the next release, but there are still too many issues with >> packages. For now we identified some categories which need fixing. These are >> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, >> resulting in >> more than 3000 bug reports. That's a high number, on the other hand we won't >> make much progress if the issues are not named. My intent is to bring that >> up >> in the Python BoF next week at DebConf and then filing these bug reports with >> the user tags mentiond on the wiki page. > > Great to hear that you guys talked about it. > > Just a quick note that PEP 394 discussions have revived, lead by the Fedora > folks. Please do check out the new thread, especially if you have opinions > about what /usr/bin/python should do once Python 2.7 is EOL. > > https://mail.python.org/pipermail/linux-sig/2017-August/thread.html I replied to this thread. I think there should be one release which is not shipping /usr/bin/python before /usr/bin/python should be reused and pointed at python (>> 2). This should be good enough to get all scripts actively converted which are not part of the distribution. If that release is buster, we should require the use of python2 instead of python now, document that in policy and provide a lintian check for that. Matthias
Re: MBF for deprecating Python2 usage
[Scott Kitterman, 2017-08-04] > Read it. I remain completely convinced that /usr/bin/python pointing > at a python3 version is utterly wrong and a disservice to our users. > Even after we remove python2.7, people will be locally compiling it > and using it for a decade. > > Such a change would be actively user hostile. > > When python2.7 goes away, /usr/bin/python should go too. > > Before python2.7 goes away, any python3 scripts/apps need to work with > /usr/bin/python3. Making them work with /usr/bin/python too doesn't > actually help anything. All it does is look pretty for us old timers. > > I've said before and I've yet to see anything to cause me to have any > doubt on the matter, that as long as I am one of the python-defaults > maintainers, /usr/bin/python will not point at a python3 version. same here. Way too many not packaged /usr/bin/python scripts around me -- GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
Re: MBF for deprecating Python2 usage
Dropped d-devel. On August 3, 2017 9:08:10 PM EDT, ba...@debian.org wrote: >On Aug 3, 2017, at 17:57, Matthias Klose wrote: >> >> While at DebCamp, Stefano Rivera and I sat down to analyze what needs >to be done >> to deprecate Python2 usage within the distribution. It might not be >possible to >> drop Python2 for the next release, but there are still too many >issues with >> packages. For now we identified some categories which need fixing. >These are >> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, >resulting in >> more than 3000 bug reports. That's a high number, on the other hand >we won't >> make much progress if the issues are not named. My intent is to >bring that up >> in the Python BoF next week at DebConf and then filing these bug >reports with >> the user tags mentiond on the wiki page. > >Great to hear that you guys talked about it. > >Just a quick note that PEP 394 discussions have revived, lead by the >Fedora folks. Please do check out the new thread, especially if you >have opinions about what /usr/bin/python should do once Python 2.7 is >EOL. > >https://mail.python.org/pipermail/linux-sig/2017-August/thread.html > >Cheers, >-Barry Read it. I remain completely convinced that /usr/bin/python pointing at a python3 version is utterly wrong and a disservice to our users. Even after we remove python2.7, people will be locally compiling it and using it for a decade. Such a change would be actively user hostile. When python2.7 goes away, /usr/bin/python should go too. Before python2.7 goes away, any python3 scripts/apps need to work with /usr/bin/python3. Making them work with /usr/bin/python too doesn't actually help anything. All it does is look pretty for us old timers. I've said before and I've yet to see anything to cause me to have any doubt on the matter, that as long as I am one of the python-defaults maintainers, /usr/bin/python will not point at a python3 version. Scott K
Re: MBF for deprecating Python2 usage
On Aug 3, 2017, at 17:57, Matthias Klose wrote: > > While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be > done > to deprecate Python2 usage within the distribution. It might not be possible > to > drop Python2 for the next release, but there are still too many issues with > packages. For now we identified some categories which need fixing. These are > documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, resulting > in > more than 3000 bug reports. That's a high number, on the other hand we won't > make much progress if the issues are not named. My intent is to bring that up > in the Python BoF next week at DebConf and then filing these bug reports with > the user tags mentiond on the wiki page. Great to hear that you guys talked about it. Just a quick note that PEP 394 discussions have revived, lead by the Fedora folks. Please do check out the new thread, especially if you have opinions about what /usr/bin/python should do once Python 2.7 is EOL. https://mail.python.org/pipermail/linux-sig/2017-August/thread.html Cheers, -Barry signature.asc Description: Message signed with OpenPGP
MBF for deprecating Python2 usage
While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be done to deprecate Python2 usage within the distribution. It might not be possible to drop Python2 for the next release, but there are still too many issues with packages. For now we identified some categories which need fixing. These are documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, resulting in more than 3000 bug reports. That's a high number, on the other hand we won't make much progress if the issues are not named. My intent is to bring that up in the Python BoF next week at DebConf and then filing these bug reports with the user tags mentiond on the wiki page. Matthias