Re: MBF for deprecating Python2 usage

[Geoffrey Thomas]

On Mon, 7 Aug 2017, Steve Langasek wrote:


But I am concerned about possible implementation strategies here.  exec() is
a very expensive syscall, and python is a frequently used interpreter.  If
this were implemented as a wrapper that checked isatty(), printed a banner,
and then re-execed the real python, that could have a measurable performance
impact on some applications.  There is a reason /usr/bin/python,
/usr/bin/gcc, etc. are always symlinks to the real interpreter on Debian,
not wrapper scripts - other distributions have tried to do this as a wrapper
script and the result wasn't pretty.


Python itself is not very fast to start up, though, so exec is not the 
dominating cost. Some ad-hoc testing with the current version of pythonmux 
(which uses the re-exec approach) shows that it takes about 13.8 ms to 
print("Hello world") with /usr/bin/python and 15.5 ms to print("Hello 
world") with pythonmux on my 2010-ish laptop.


For the sake of argument, I just pushed a branch of pythonmux that links 
libpython3.5 and calls Py_Main instead of execing python3.5 (that is, if 
the target interpreter is python3.5, it will not actually exec):


  https://github.com/geofft/pythonmux/commits/libpython

But it's quite a bit _slower_ with this approach (about 26.9 ms per run), 
regardless of whether I link libpython3.5m.so or libpython3.5m.a. This 
confuses me and I'm definitely curious to figure this out if that's worth 
doing, but I think this is getting into a lot of weird complexity to save 
1.7 ms on a mere hello world -- for comparison, if I add `import requests` 
to my hello world program, each run now takes about 342 ms.


By the way, I do think I made pythonmux do too much since I wanted to make 
it useful in the general case; in particular I don't think it's super 
helpful to support scripts that say "I can run on Python 3.2 but not any 
newer version". Given the way the discussion is going (both here and on 
linux-sig) and how close we are to Python 2's end-of-life, I think it's 
sufficient to check isatty(0) and the shebang line if one exists, call 
libpython3's Py_Main if that seems reasonable, print a warning and attempt 
to exec python2 otherwise, and print an error message about how to install 
python2 if the exec fails.


--
Geoffrey Thomas
https://ldpreload.com
geo...@ldpreload.com

Re: MBF for deprecating Python2 usage

[Scott Kitterman]

On Tuesday, August 08, 2017 08:58:04 PM Ole Streicher wrote:
> Diane Trout  writes:
> >> What I am opposing is the suggestion to install, in the near to
> >> medium
> >> term, a command of exactly the same name that has subtly similar but
> >> incompatible behaviour, when that behaviour *already* has a command –
> >> ‘python3’ – that is widely used by those who need it.
> > 
> > my problem with that plan is all of the printed documentation saying to
> > learn python, type "python".
> 
> All *actual* printed documentation? Which refers to Python 3? Could you
> give an example here?
> 
> Otherwise, (if using a Python 2 book), one of the next steps would be
> "print 'hello world'", and nothing would prevent from the horrible
> experience that the book is outdated anyway.
> 
> Sorry, I don't see the problem with "printed documentation".

Having recently (late last year) searched for a beginner's Python book for one 
of my kids, I can tell you that current offers range between "Only a few 
people use Python 3, so you're better off ignoring this odd duck while you 
learn" to "Python 3 is the future, so you may as well start be learning the 
future rather than the past".  I don't recall any current books that didn't at 
least acknowledge the existence of this Python 3 thing.

Based on that (admittedly incomplete) survey, I think confusion about the 
existence of Python/Python 3 is a problem with current dead tree learning 
Python editions.  If you're trying to learn something new with old 
documentation, I don't think it should be surprising there are a few bumps 
along the way.

I agree.

Scott K

Re: MBF for deprecating Python2 usage

[Ole Streicher]

Diane Trout  writes:
>> What I am opposing is the suggestion to install, in the near to
>> medium
>> term, a command of exactly the same name that has subtly similar but
>> incompatible behaviour, when that behaviour *already* has a command –
>> ‘python3’ – that is widely used by those who need it.
>> 
>
> my problem with that plan is all of the printed documentation saying to
> learn python, type "python".

All *actual* printed documentation? Which refers to Python 3? Could you
give an example here?

Otherwise, (if using a Python 2 book), one of the next steps would be
"print 'hello world'", and nothing would prevent from the horrible
experience that the book is outdated anyway.

Sorry, I don't see the problem with "printed documentation".

Best regards

Ole

Re: MBF for deprecating Python2 usage

[Ben Finney]

Steve Langasek  writes:

> FWIW I don't share Ben's concern about this being a "backwards
> incompatible" change (who is relying on the precise contents of the
> banner when running 'python' interactively?!).

That's not a backward incompatibility anyone would seriously rely on,
and I don't have that concern.

The backward incompatibility I'm concerned about is: to have ‘python’,
which today invokes Python 2 and some uncountable number of scripts rely
on, change on a near-to-medium future upgrade to instead invoke the
backward-incompatible Python 3 hence breaking those scripts.

Merely noting a change in the interactive prompt's banner is, I agree,
inconsequential by comparison. So that's not the objection.

-- 
 \“All opinions are not equal. Some are a very great deal more |
  `\   robust, sophisticated, and well supported in logic and argument |
_o__) than others.” —Douglas Adams |
Ben Finney

Re: MBF for deprecating Python2 usage

[Steve Langasek]

On Mon, Aug 07, 2017 at 08:11:40PM -0700, Diane Trout wrote:

> > What I am opposing is the suggestion to install, in the near to
> > medium
> > term, a command of exactly the same name that has subtly similar but
> > incompatible behaviour, when that behaviour *already* has a command –
> > ‘python3’ – that is widely used by those who need it.

> my problem with that plan is all of the printed documentation saying to
> learn python, type "python".

> At the very least there needs to a usr/bin/python that prints
> instructions about what you should run.

FWIW I don't share Ben's concern about this being a "backwards incompatible"
change (who is relying on the precise contents of the banner when running
'python' interactively?!).

But I am concerned about possible implementation strategies here.  exec() is
a very expensive syscall, and python is a frequently used interpreter.  If
this were implemented as a wrapper that checked isatty(), printed a banner,
and then re-execed the real python, that could have a measurable performance
impact on some applications.  There is a reason /usr/bin/python,
/usr/bin/gcc, etc. are always symlinks to the real interpreter on Debian,
not wrapper scripts - other distributions have tried to do this as a wrapper
script and the result wasn't pretty.

Avoiding the performance hit would require that any changes to the banner
be made in the python source itself.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: PGP signature

Re: MBF for deprecating Python2 usage

[Barry Warsaw]

On Aug 7, 2017, at 20:52, Ben Finney  wrote:
> 
> So IMO the conservative assumption is that they expect Debian to provide
> the stability it is famous for, and not to break the behaviour of
> commands unnecessarily.

There’s another dimension to that too: it’s people who expect no changes to the 
command called /usr/bin/python *and* want to track the latest and greatest 
Debian releases.  Maybe that changes the equation and maybe not.  Certainly any 
large organization with tons of Python 2 laying around is going to be much more 
conservative about upgrading their entire distro than individual developers who 
just use unstable and get whatever today’s snapshot looks like.

And to reiterate, I’m not arguing that /usr/bin/python change today or even at 
the end of Pycon 2020 when 2.7 is officially EOL’d.  I’m saying that it *will* 
change some day and that we should plan that change, if for no other reason 
than to set the expectations of exactly those folks who are not part of this 
conversation.

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP

Re: MBF for deprecating Python2 usage

[Ben Finney]

Diane Trout  writes:

> On Tue, 2017-08-08 at 13:24 +1000, Ben Finney wrote:
> > 
> > Those people, not party [to] this conversation, have reasonable
> > expectation that such breakage will not happen without very good
> > reason.
> > Good reason would entail, as an example, that there is no better
> > alternative.
>
> Why not ask?

Ask whom? Because the issue is about the expectations of those who
*don't want* to be bothered by technical discussions like this, asking
those few who make themselves available for such questions will not
resolve the issue.

At best, we would have a collection of anecdotes and no idea how
representative they are of those who choose not to be contactable (but
still expect Debian to avoid breaking things unnecessarily).

At worst, we would have a collection of anecdotes that would be a strong
temptation for *falsely assuming* they represent the silent majority.

Barry, earlier in this thread, was correct when he pointed out that, by
the nature of the issue, we *can't* know with confidence what the
majority of people expect about this change.

So IMO the conservative assumption is that they expect Debian to provide
the stability it is famous for, and not to break the behaviour of
commands unnecessarily.

-- 
 \   “Some people have a problem, and they think “I know, I'll use |
  `\ Perl!”. Now they have some number of problems but they're not |
_o__) sure whether it's a string or an integer.” —Benno Rice, 2011 |
Ben Finney

Re: MBF for deprecating Python2 usage

[Diane Trout]

On Tue, 2017-08-08 at 13:24 +1000, Ben Finney wrote:
> 
> Those people, not party ot this conversation, have reasonable
> expectation that such breakage will not happen without very good
> reason.
> Good reason would entail, as an example, that there is no better
> alternative.
> 

Why not ask?

I know people in training groups like software carpentry? Does anyone
know sites with large python deployments?

I'd love to know what Apple is going to do about this issue.

Also if we do come up with a migration it should happen at a release
and be clearly reported in the release notes.

Diane

Re: MBF for deprecating Python2 usage

[Ben Finney]

Diane Trout  writes:

> my problem with that plan is all of the printed documentation saying
> to learn python, type "python".

I agree that's a problem.

I don't agree that making a backward incompatible breakage is justified
by widespread documentation giving bad advice.

-- 
 \  “There are no chaplains in foxholes.” —Sergeant Justin |
  `\  Griffith, 2011-07-27 |
_o__)  |
Ben Finney

Re: MBF for deprecating Python2 usage

[Ben Finney]

Diane Trout  writes:

> Personally, I'm ready for python to point to python3 now.

I appreciate that. In many of my hours I even concur.

Both of us can have that, for our own personal environment.

That doesn't answer the question of changing the behaviour of the
*default* ‘python’ command installed by Debian packages, for people who
have not asked for this breaking change.

Those people, not party ot this conversation, have reasonable
expectation that such breakage will not happen without very good reason.
Good reason would entail, as an example, that there is no better
alternative.

The use cases provided in favour of “make an incompatible change in what
the ‘python’ command does, by default” so far do not meet that standard.

People who *actively want* ‘python’ to mean Python 3 can set up their
environment that way today, without affecting others who have not asked
for anything to change. So the desires of those people – and I can even
count myself one of them! – should weigh less than those who expect the
stability promised by Debian.

> Python has been remarkably good at avoiding breakage, but I've seen
> other scripting languages have serious incompatibilities with far less
> warning.

That is not an argument for changes that break default behaviour.

> It might be useful to add an option to the interpreter where if a
> python script is launched without a usr/bin/python2 or usr/bin/python3
> header it reports a deprecation warning (either to console or syslog)
> so it's easier to find programs that still need updating.

That sounds good to me.

-- 
 \ “I was stopped by the police for speeding; they said ‘Don't you |
  `\   know the speed limit is 55 miles an hour?’ I said ‘Yeah I know, |
_o__) but I wasn't going to be out that long.’” —Steven Wright |
Ben Finney

Re: MBF for deprecating Python2 usage

[Diane Trout]

> What I am opposing is the suggestion to install, in the near to
> medium
> term, a command of exactly the same name that has subtly similar but
> incompatible behaviour, when that behaviour *already* has a command –
> ‘python3’ – that is widely used by those who need it.
> 

my problem with that plan is all of the printed documentation saying to
learn python, type "python".

At the very least there needs to a usr/bin/python that prints
instructions about what you should run.

Re: MBF for deprecating Python2 usage

[Diane Trout]

> What tearing need is there to change what the command ‘python’ does,
> in
> a backward-incompatible way?

Personally, I'm ready for python to point to python3 now.

I'm tired of writing python 2/3 compatible code because someone _might_
launch a script with "python my_python3_script.py instead of
./my_python3_script.py

Python has been remarkably good at avoiding breakage, but I've seen
other scripting languages have serious incompatibilities with far less
warning.

For example R packages need to be recompiled for point release updates.
Some perl scripts we inherited broke somewhere between perl 5.8 and
5.12.

Users have tools like virtualenv or conda to deal with the
incompatibilities.

It might be useful to add an option to the interpreter where if a
python script is launched without a usr/bin/python2 or usr/bin/python3
header it reports a deprecation warning (either to console or syslog)
so it's easier to find programs that still need updating.

Diane

Re: MBF for deprecating Python2 usage

[Ben Finney]

Geoffrey Thomas  writes:

> We will not indefinitely provide a /usr/bin/python that runs Python 2;
> we probably will do so for at most one more stable release.

I'm fine with removing the ‘python’ command, because that command means
Python 2 and Python 2 is going away.

What I am opposing is the suggestion to install, in the near to medium
term, a command of exactly the same name that has subtly similar but
incompatible behaviour, when that behaviour *already* has a command –
‘python3’ – that is widely used by those who need it.

-- 
 \   “From the moment I picked your book up until I laid it down I |
  `\was convulsed with laughter. Someday I intend reading it.” |
_o__)—Groucho Marx |
Ben Finney

Re: MBF for deprecating Python2 usage

[Ben Finney]

Barry Warsaw  writes:

> On Aug 7, 2017, at 18:28, Ben Finney  wrote:
>
> > That day [when we know that the vast majority of scripts in the wild
> > no longer expect Python 2 when invoking the ‘python’ command] might
> > never come, in which case the ‘python’ command will forever mean
> > Python 2. That is, I'm saying, better than breaking that command in
> > the near or medium future.
>
> I wonder if you’ll say the same thing in 2030 when Python 2.7 is 10
> years dead, can literally cannot be built on any then-modern Debian,
> let alone Linux distro?

Why do you wonder that, in the context of this proposal? That proposal
is the context of my statement above.

In the context of the proposal now before us, to change the behaviour of
the ‘python’ command in the near-to-medium term, it seems irrelevant to
wonder what I'll say on the matter in 2030.

Of course, I'm open to Python 2 dwindling into the past; it is
inevitable as the years go by. But in 2020 that won't be the case, and
we should be very conservative about declaring it to be the case because
– as you point out – even in 2020 we cannot be confident that we'll know
what the vast majority of scripts out there expect.

> Sure, today. And tomorrow. And many tomorrows until at least 2020.
> Then I think we’ll be having very different conversations about it.
> Anybody who will still be running Python 2.7 code will have some
> difficult choices to make. 1) port to Python 3; 2) run old versions of
> distros that still have Python 2.7 around and live with the
> unsupported nature of it; 3) roll their own Python 2.7s; 4) pay some
> commercial vendors to keep their stuff alive and secure.

I see no good reason to make that job *more difficult* by wantonly
changing what interpreter is invoked with the ‘python’ command.
Especially because it isn't even something needed for Python 3 to
flourish.

What tearing need is there to change what the command ‘python’ does, in
a backward-incompatible way?

We already have a ‘python3’ command. Scripts written today must
explicitly choose ‘python3’ to get that version of the language.
Learners learning today must explicitly ask for Python version 3.

> IMHO, there will absolutely be a day when it makes no sense to point
> /usr/bin/python to Python 2.

We agree on that fact. There is some disagreement about how many people,
whose voices are not part of this conversation, can be dismissed before
that day arrives. And when they can reasonably be dismissed.

-- 
 \ “We demand rigidly defined areas of doubt and uncertainty!” |
  `\—Vroomfondel, _The Hitch-Hiker's Guide To The Galaxy_, Douglas |
_o__)Adams |
Ben Finney

Re: MBF for deprecating Python2 usage

[Geoffrey Thomas]

On Tue, 8 Aug 2017, Ben Finney wrote:


The issue is: Invoking ‘python’ today gets an interpereter, Python 2,
that will work with some code and not others. It should not tomorrow
invoke an incompatible interpreter without *knowing* that the vast
majority of scripts in the wild no longer expect Python 2 to come from
that command.

That day might never come, in which case the ‘python’ command will
forever mean Python 2. That is, I'm saying, better than breaking that
command in the near or medium future.

I'm saying it's a bad idea for ‘python’ tomorrow to get an incompatible
intepreter that won't run the same Python code. That interpreter is
named ‘python3’ and should never be installed as ‘python’, because
‘python’ is a command that many scripts expect to invoke Python 2.


This conflates the role of the interactive Python interpreter (which is 
what Diane was referring to) and a script interpreter. #!/usr/bin/python 
and `python -c` are ABI referring to Python 2, yes, and can't start 
running Python 3 any more than /lib/libc.so.5 can start being glibc. But 
there's no reason this should prevent typing "python" from giving you a 
Python 3 interpreter.


Since I think it hasn't been linked recently on this list, I'll re-link 
this project, which separates the two roles:


https://github.com/geofft/pythonmux
https://ldpreload.com/blog/usr-bin-python-23

Keep in mind that we do intend to break the ability for /usr/bin/python to 
run Python 2 code in less than three years (unless we're interested in 
providing security support for Python 2 when it's dead upstream, which I 
hope nobody wants to do). We will not indefinitely provide a 
/usr/bin/python that runs Python 2; we probably will do so for at most one 
more stable release.


The only question is whether it should break by giving you an error 
message (either ENOENT or something printed on stderr), or break by trying 
to run the code as Python 3.


--
Geoffrey Thomas
https://ldpreload.com
geo...@ldpreload.com

Re: MBF for deprecating Python2 usage

[Barry Warsaw]

On Aug 7, 2017, at 18:28, Ben Finney  wrote:

> The issue is: Invoking ‘python’ today gets an interpereter, Python 2,
> that will work with some code and not others. It should not tomorrow
> invoke an incompatible interpreter without *knowing* that the vast
> majority of scripts in the wild no longer expect Python 2 to come from
> that command.

That’s unknowable.  We have no idea what people do with any package of Debian 
outside of the archive.  We can’t even quantify what “vast” means.  So under 
the above definition, you’re right, that day will never come.

> That day might never come, in which case the ‘python’ command will
> forever mean Python 2. That is, I'm saying, better than breaking that
> command in the near or medium future.

I wonder if you’ll say the same thing in 2030 when Python 2.7 is 10 years dead, 
can literally cannot be built on any then-modern Debian, let alone Linux 
distro? That’s not hyperbole; one of the current upstream exceptions to the 2.7 
branch is to adapt to newer build systems and library versions.  That will stop 
in 2020, but of course everything else will continue to evolve.

It might even come sooner.  Once Python 2.7 is EOL’d, it won’t even get source 
releases, and may not even get security releases.  If folks want to keep it 
alive, it’ll take active downstream resources to maintain.  People may pay for 
commercial support from RedHat, Canonical, and others, but how much will 
volunteers want to keep it secure and working?

I can remember when ‘python’ meant Python 1.5.2.  So that name doesn’t have to 
be tied to a specific major version.

> I'm saying it's a bad idea for ‘python’ tomorrow to get an incompatible
> intepreter that won't run the same Python code. That interpreter is
> named ‘python3’ and should never be installed as ‘python’, because
> ‘python’ is a command that many scripts expect to invoke Python 2.

Sure, today.  And tomorrow.  And many tomorrows until at least 2020.  Then I 
think we’ll be having very different conversations about it.  Anybody who will 
still be running Python 2.7 code will have some difficult choices to make.  1) 
port to Python 3; 2) run old versions of distros that still have Python 2.7 
around and live with the unsupported nature of it; 3) roll their own Python 
2.7s; 4) pay some commercial vendors to keep their stuff alive and secure.

IMHO, there will absolutely be a day when it makes no sense to point 
/usr/bin/python to Python 2.

Cheers,
-Barry


signature.asc
Description: Message signed with OpenPGP

Re: MBF for deprecating Python2 usage

[Ben Finney]

Diane Trout  writes:

> My suggestion was "the startup banner should print what command to run
> to get Python 2."

Thanks for the suggestion. I think it's a bad idea.

> I was thinking of the case of the end-user trying to follow a Python
> tutorial. They'd still need to exit and run the python2 command if they
> wanted 2.

That's not the issue.

The issue is: Invoking ‘python’ today gets an interpereter, Python 2,
that will work with some code and not others. It should not tomorrow
invoke an incompatible interpreter without *knowing* that the vast
majority of scripts in the wild no longer expect Python 2 to come from
that command.

That day might never come, in which case the ‘python’ command will
forever mean Python 2. That is, I'm saying, better than breaking that
command in the near or medium future.

I'm saying it's a bad idea for ‘python’ tomorrow to get an incompatible
intepreter that won't run the same Python code. That interpreter is
named ‘python3’ and should never be installed as ‘python’, because
‘python’ is a command that many scripts expect to invoke Python 2.

-- 
 \ “Reality must take precedence over public relations, for nature |
  `\   cannot be fooled.” —Richard P. Feynman, _Rogers' Commission |
_o__)   Report into the Challenger Crash_, 1986-06 |
Ben Finney

Re: MBF for deprecating Python2 usage

[Diane Trout]

> I disagree, it's a bad idea to actively take steps to make the same
> command invoke *incompatible* programs depending on the time and
> host.

My suggestion was "the startup banner should print what command to run
to get Python 2."

I was thinking of the case of the end-user trying to follow a Python
tutorial. They'd still need to exit and run the python2 command if they
wanted 2.

Diane

Re: MBF for deprecating Python2 usage

[Ben Finney]

Barry Warsaw  writes:

> On Aug 7, 2017, at 14:52, Diane Trout  wrote:
> > Perhaps when launching via the command "python" the interpreter could
> > hint python2 was available? Something like:
> > 
> > $ python
> > Python 3.5.4rc1 (default, Jul 25 2017, 08:53:34)
> > [GCC 6.4.0 20170704] on linux
> > Type "python2" for Python 2.7.13
> > Type "help", "copyright", "credits" or "license" for more information.
>
> Good idea Diane!

I disagree, it's a bad idea to actively take steps to make the same
command invoke *incompatible* programs depending on the time and host.

(good sigmonster; have a cookie.)

-- 
 \ “I've always wanted to be somebody, but I see now that I should |
  `\   have been more specific.” —Jane Wagner, via Lily Tomlin |
_o__)  |
Ben Finney

Re: MBF for deprecating Python2 usage

[Geoffrey Thomas]

On Fri, 4 Aug 2017, Scott Kitterman wrote:


If the primary concern is what happens when a user types "python", then can we
address that in command-not-found and leave /usr/bin/python out of it?


Debian doesn't ship command-not-found by default. One other approach here 
would be to have /usr/bin/python exec python3 if it's run interactively -- 
that is, if isatty(0) -- and print an error message and exit otherwise. 
I don't think it is too bad to print an error message instead of failing 
the exec, since that's equivalent to what happens if you're missing a 
shared library or something: the exec succeeds, but the dynamic loader 
prints an error and exits nonzero.


Ship this /usr/bin/python with python3-minimal, and have the 
python-minimal package divert it and replace it with the current 
/usr/bin/python (a symlink to python2.7). This way we can build systems 
with Python 3 only that behave reasonably, but also allow doing `apt-get 
install python` to cause /usr/bin/python to start working again.



However, I feel like another reason to care about /usr/bin/python is so 
that sysadmins responsible for multiple platforms can continue writing 
#!/usr/bin/python (or #!/usr/bin/env python) scripts and use the same 
script on all their machines. If I can't do this and need 
platform-specific scripts, I might as well use some other language or ship 
compiled binaries. Keeping Python viable for this use case is a large part 
of why I originally proposed pythonmux: I can write scripts that declare 
that they're polyglot Python 2/3 and work on both Python 2-only and Python 
3-only systems, but a Python 2 script using #!/usr/bin/python will give me 
an error (as it should!) if there's no Python 2 installed on the machine.


Last time I checked (2015), entirely reasonable versions of RHEL and macOS 
both shipped only Python 2 and not Python 3. Is it likely to be the case 
that all UNIXy OSes within security support in 2019 or 2020 will have a 
/usr/bin/python3 (preferably 3.3+)? If so, that's a viable option. I just 
want my best migration path from #!/usr/bin/python to not involve 
rewriting my scripts in Go.


--
Geoffrey Thomas
https://ldpreload.com
geo...@ldpreload.com

Re: MBF for deprecating Python2 usage

[Barry Warsaw]

On Aug 7, 2017, at 14:52, Diane Trout  wrote:
> Perhaps when launching via the command "python" the interpreter could
> hint python2 was available? Something like:
> 
> $ python
> Python 3.5.4rc1 (default, Jul 25 2017, 08:53:34)
> [GCC 6.4.0 20170704] on linux
> Type "python2" for Python 2.7.13
> Type "help", "copyright", "credits" or "license" for more information.

Good idea Diane!
-Barry




signature.asc
Description: Message signed with OpenPGP

Re: MBF for deprecating Python2 usage

[Diane Trout]

> * Plan for a date at which /usr/bin/python will point to Python 3.  I
> know that’s the most controversial bit, but I do think that as time
> goes on and we’re past 2020, it will be the choice that gives our
> users the best experience.

I agree the default should change.

Perhaps when launching via the command "python" the interpreter could
hint python2 was available? Something like:

$ python
Python 3.5.4rc1 (default, Jul 25 2017, 08:53:34) 
[GCC 6.4.0 20170704] on linux
Type "python2" for Python 2.7.13
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Diane

signature.asc
Description: This is a digitally signed message part

Re: MBF for deprecating Python2 usage

[Christian Seiler]

Hi Matthias,

On 08/03/2017 11:57 PM, Matthias Klose wrote:
> It might not be possible to drop Python2 for the next release,

Even if all Python-related packages in Debian were compatible with
Python3, I don't think it's a good idea to drop Python2 support in
Buster, there are still far too many third-party users of Python2
out there. I also don't think there has been enough communication
to users of Debian that Python2 is going to be removed at some
point - this has mainly been discussed in development channels so
far. And while there's writing on the wall due to upstream's EOL
of Python2 I do get the impression that many people are not aware
of that.

Therefore I would strongly recommend to keep Python2 in Buster,
but add a big fat comment to the release notes that it will be the
last Debian release that will support Python2 - so that it can be
dropped in Bullseye.

That said, I do think we should strive to get rid of all the
Python2-only packages in Buster if possible, so I do appreciate
your efforts in this regard.

Regards,
Christian

Re: MBF for deprecating Python2 usage

[Adrian Bunk]

On Thu, Aug 03, 2017 at 05:57:01PM -0400, Matthias Klose wrote:
> While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be 
> done
> to deprecate Python2 usage within the distribution.  It might not be possible 
> to
> drop Python2 for the next release, but there are still too many issues with
> packages.  For now we identified some categories which need fixing. These are
> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, resulting 
> in
> more than 3000 bug reports.  That's a high number, on the other hand we won't
> make much progress if the issues are not named.  My intent is to bring that up
> in the Python BoF next week at DebConf and then filing these bug reports with
> the user tags mentiond on the wiki page.

I do not see the benefits of the MBF.

Visibility will already be provided both to you and to package 
maintainers by lintian, with automated checking by lintian
actually more reliable than information in the BTS.

And maintainers who ignore lintian results in their packages tend to be 
the same people who ignore non-RC bugs in their packages.

> Matthias

cu
Adrian

-- 

   "Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
   "Only a promise," Lao Er said.
   Pearl S. Buck - Dragon Seed

Re: MBF for deprecating Python2 usage

[Scott Kitterman]

On August 4, 2017 9:37:18 PM EDT, Barry Warsaw  wrote:
>Scott Kitterman wrote:
>
>> If the primary concern is what happens when a user types "python",
>then can we 
>> address that in command-not-found and leave /usr/bin/python out of
>it?
>
>I'm definitely willing to for now.  It's clearly not time to remove the
>link or point it elsewhere anyway.  I think it'll still be a good idea
>some day, but I'll leave it at that.
>
>Can we agree though to start using /usr/bin/python2 in the shebang line
>instead of /usr/bin/python, for system installed scripts?
>
>-Barry

I think that ship has already sailed, so sure, but I don't think it's anything 
more than a wishlist bug.

Scott K

Re: MBF for deprecating Python2 usage

[Barry Warsaw]

Scott Kitterman wrote:

> If the primary concern is what happens when a user types "python", then can 
> we 
> address that in command-not-found and leave /usr/bin/python out of it?

I'm definitely willing to for now.  It's clearly not time to remove the
link or point it elsewhere anyway.  I think it'll still be a good idea
some day, but I'll leave it at that.

Can we agree though to start using /usr/bin/python2 in the shebang line
instead of /usr/bin/python, for system installed scripts?

-Barry




signature.asc
Description: OpenPGP digital signature

Re: MBF for deprecating Python2 usage

[Scott Kitterman]

On Friday, August 04, 2017 10:13:00 AM ba...@debian.org wrote:
> On Aug 3, 2017, at 23:23, Scott Kitterman  wrote:
> > Read it.  I remain completely convinced that /usr/bin/python pointing at a
> > python3 version is utterly wrong and a disservice to our users.  Even
> > after we remove python2.7, people will be locally compiling it and using
> > it for a decade.
> I think it’s inevitable, and not doing so *eventually* will be a greater
> disservice.  Sure, not today, maybe not in 2020, but I don’t think we’ll be
> having this discussion in 2025.  If by then, 5 years after Python 2.7 EOL,
> users read on SO or some Learning Python site that they can just type
> ‘python' to get an interactive prompt, but it doesn’t work on Debian,
> they’ll think Debian is broken, not Python.  So I think it makes sense to
> consider how the transition will work.
> 
> You’re right that folks will still need Python 2 after 2020, and that their
> best option may be to build it themselves, but I don’t think they’ll be
> building and installing some old Debian package.  Instead they may build
> from source, which means they’ll be installing it into /usr/local/bin not
> /usr/bin, and they’ll have to change their shebang lines anyway.  And there
> will come a time when even Python 2.7 is unbuildable as toolchains and
> libraries evolve.  Upstream will stop tracking those changes so even the
> upstream git repo won’t give you buildable source.  There may be a bunch of
> third party forks, but I don’t think it’s our responsibility to ensure that
> Debian aligns with any of those hypotheticals.  Folks needing to stay on
> Python 2 will probably also just elect to not upgrade their OS, so what we
> do in future releases won’t matter to them.
> 
> Upstream and Debian already install a `python2` alias for `python`, and I
> think we need to join the chorus that in the future, the way to run Python
> 2 is via `python2`.  Yes, people will have legacy stuff around for a long
> time, but it’s better to begin the transition now rather than make a major
> break years from now.
> > Such a change would be actively user hostile.
> > 
> > When python2.7 goes away, /usr/bin/python should go too.
> 
> Maybe in the short term (as Matthias suggests), but I’m not so sure it’s
> best to disappear /usr/bin/python forever.  /usr/bin/python is not just the
> common shebang (and we should be actively transitioning to /usr/bin/python2
> for that), but it’s also the command line UI for people wanting to learn
> Python, or just wanting to get an interactive interpreter to play with.
> There will be a day when, if they get a command not found when typing
> `python` at a shell prompt, they will wonder why Debian is broken.
> 
> I think the transition should roughly be:
> 
> * Update policy and tooling so that any scripts that require Python 2 use
> /usr/bin/python2 in their shebang.
> 
> * Educate our users that they should be using `python2` for anything that
> has not been ported.
> 
> * Port as much as possible to Python 3 (eventually, everything maintained in
> Debian) and /usr/bin/python3 in their shebang.
> 
> * Plan for a date at which /usr/bin/python will point to Python 3.  I know
> that’s the most controversial bit, but I do think that as time goes on and
> we’re past 2020, it will be the choice that gives our users the best
> experience.
> 
> The discussion on linux-sig is a way to align the entire Python Linux
> ecosystem to the eventual goal, giving individual distros the leeway they
> need to time such transitions as they see fit to best serve their users. 
> I’m hoping more Debian Pythonistas will join that discussion, otherwise the
> outcome will be heavily weighted toward other Linux distros and Debian may
> find itself without a voice in the matter.

PEP-394 convinced me that upstream wasn't concerned about the same things 
Debian is concerned about.  Even though they did manage to say in the PEP:

> The main barrier to a distribution switching the python command from python2
> to python3 isn't breakage within the distribution, but instead breakage of
> private third party scripts developed by sysadmins and other users.
> Updating the python command to invoke python3 by default indicates that a
> distribution is willing to break such scripts with errors that are
> potentially quite confusing for users that aren't yet familiar with the
> backwards incompatible changes in Python 3.

they don't actually consider the implications of that statement.

The current discussion only seems to compound the error.  I don't see a lot of 
value in personally expending time on the upstream discussion.

If the primary concern is what happens when a user types "python", then can we 
address that in command-not-found and leave /usr/bin/python out of it?

Scott K

signature.asc
Description: This is a digitally signed message part.

Re: MBF for deprecating Python2 usage

[Barry Warsaw]

Ole Streicher wrote:

> It is very usual to use "#!/usr/bin/env python" as shebang, exactly for
> the case that python is not installed in /usr/bin.

Sure, but then all bets are off.  The script so shebanged can't assume
anything about $PATH so it gets whatever it gets.  Using /usr/bin/env in
system provided scripts has been known to break stuff, so it's a very
bad idea.

Plus, if that's what you use for your own scripts, then in the future
you'll be in luck because all you'd have to do is change your $PATH
to point to wherever your custom built and installed Python 2 lives
first, and then you'll get exactly the version you want.

> That is -- at least in my environmet -- "ipython", or (for Python 3)
> "ipython3".
>
> IMO it would be better to communicate that the best way for an
> interactive session is "ipython3" (or "python3", if you insist). I would
> wonder when todays tutorials (that cover Python 3) recommend to use
> plain "python".

But ipython doesn't come with upstream Python so that would require that
Something Extra be installed.  That's generally why Getting Started
guides and tutorials usually recommend the built-in interactive prompt.

>> * Port as much as possible to Python 3 (eventually, everything
>> maintained in Debian) and /usr/bin/python3 in their shebang.
> I disagree here, and I don't see an advantage over letting
> /usr/bin/python just die with Python 2.

/usr/bin/python didn't die with Python 2 (who else remembers 1.5.2 as an
effectively LTS release? :) and I don't believe it should with Python 3.
Maybe go on vacation, but eventually return home.

> I would also like to point PEP 394, which has a number of arguments why
> /usr/bin/python should remain as Python 2.
>
> https://www.python.org/dev/peps/pep-0394/

Right, but the ecosystem has evolved since then, so the discussions
going on in linux-sig and the *draft* PR at
https://github.com/python/peps/pull/315 are about updating the
recommendations now that we're somewhere less than three years away from
Python 2 EOL.

Fedora is moving forward with or without us.  I bet that other Linux
distros will too.  We can certainly do our own thing, but I think
that'll make Debian the odd distro out eventually.

Cheers,
-Barry

Re: MBF for deprecating Python2 usage

[Ole Streicher]

ba...@debian.org writes:
> On Aug 3, 2017, at 23:23, Scott Kitterman  wrote:
> You’re right that folks will still need Python 2 after 2020, and that
> their best option may be to build it themselves, but I don’t think
> they’ll be building and installing some old Debian package.  Instead
> they may build from source, which means they’ll be installing it into
> /usr/local/bin not /usr/bin, and they’ll have to change their shebang
> lines anyway.

It is very usual to use "#!/usr/bin/env python" as shebang, exactly for
the case that python is not installed in /usr/bin.

>> Such a change would be actively user hostile.
>> 
>> When python2.7 goes away, /usr/bin/python should go too.
>
> Maybe in the short term (as Matthias suggests), but I’m not so sure
> it’s best to disappear /usr/bin/python forever.  /usr/bin/python is
> not just the common shebang (and we should be actively transitioning
> to /usr/bin/python2 for that), but it’s also the command line UI for
> people wanting to learn Python, or just wanting to get an interactive
> interpreter to play with.

That is -- at least in my environmet -- "ipython", or (for Python 3)
"ipython3".

> There will be a day when, if they get a command not found when typing
> `python` at a shell prompt, they will wonder why Debian is broken.

IMO it would be better to communicate that the best way for an
interactive session is "ipython3" (or "python3", if you insist). I would
wonder when todays tutorials (that cover Python 3) recommend to use
plain "python".

> I think the transition should roughly be:
>
> * Update policy and tooling so that any scripts that require Python 2
> use /usr/bin/python2 in their shebang.
>
> * Educate our users that they should be using `python2` for anything
> that has not been ported.
>
> * Port as much as possible to Python 3 (eventually, everything
> maintained in Debian) and /usr/bin/python3 in their shebang.

I disagree here, and I don't see an advantage over letting
/usr/bin/python just die with Python 2.

I would also like to point PEP 394, which has a number of arguments why
/usr/bin/python should remain as Python 2.

https://www.python.org/dev/peps/pep-0394/

Best regards

Ole

Re: MBF for deprecating Python2 usage

[barry]

On Aug 3, 2017, at 23:23, Scott Kitterman  wrote:
> 
> Read it.  I remain completely convinced that /usr/bin/python pointing at a 
> python3 version is utterly wrong and a disservice to our users.  Even after 
> we remove python2.7, people will be locally compiling it and using it for a 
> decade.

I think it’s inevitable, and not doing so *eventually* will be a greater 
disservice.  Sure, not today, maybe not in 2020, but I don’t think we’ll be 
having this discussion in 2025.  If by then, 5 years after Python 2.7 EOL, 
users read on SO or some Learning Python site that they can just type ‘python' 
to get an interactive prompt, but it doesn’t work on Debian, they’ll think 
Debian is broken, not Python.  So I think it makes sense to consider how the 
transition will work.

You’re right that folks will still need Python 2 after 2020, and that their 
best option may be to build it themselves, but I don’t think they’ll be 
building and installing some old Debian package.  Instead they may build from 
source, which means they’ll be installing it into /usr/local/bin not /usr/bin, 
and they’ll have to change their shebang lines anyway.  And there will come a 
time when even Python 2.7 is unbuildable as toolchains and libraries evolve.  
Upstream will stop tracking those changes so even the upstream git repo won’t 
give you buildable source.  There may be a bunch of third party forks, but I 
don’t think it’s our responsibility to ensure that Debian aligns with any of 
those hypotheticals.  Folks needing to stay on Python 2 will probably also just 
elect to not upgrade their OS, so what we do in future releases won’t matter to 
them.

Upstream and Debian already install a `python2` alias for `python`, and I think 
we need to join the chorus that in the future, the way to run Python 2 is via 
`python2`.  Yes, people will have legacy stuff around for a long time, but it’s 
better to begin the transition now rather than make a major break years from 
now.

> Such a change would be actively user hostile.
> 
> When python2.7 goes away, /usr/bin/python should go too.

Maybe in the short term (as Matthias suggests), but I’m not so sure it’s best 
to disappear /usr/bin/python forever.  /usr/bin/python is not just the common 
shebang (and we should be actively transitioning to /usr/bin/python2 for that), 
but it’s also the command line UI for people wanting to learn Python, or just 
wanting to get an interactive interpreter to play with. There will be a day 
when, if they get a command not found when typing `python` at a shell prompt, 
they will wonder why Debian is broken.

I think the transition should roughly be:

* Update policy and tooling so that any scripts that require Python 2 use 
/usr/bin/python2 in their shebang.

* Educate our users that they should be using `python2` for anything that has 
not been ported.

* Port as much as possible to Python 3 (eventually, everything maintained in 
Debian) and /usr/bin/python3 in their shebang.

* Plan for a date at which /usr/bin/python will point to Python 3.  I know 
that’s the most controversial bit, but I do think that as time goes on and 
we’re past 2020, it will be the choice that gives our users the best experience.

The discussion on linux-sig is a way to align the entire Python Linux ecosystem 
to the eventual goal, giving individual distros the leeway they need to time 
such transitions as they see fit to best serve their users.  I’m hoping more 
Debian Pythonistas will join that discussion, otherwise the outcome will be 
heavily weighted toward other Linux distros and Debian may find itself without 
a voice in the matter.

Cheers,
-Barry




signature.asc
Description: Message signed with OpenPGP

Re: MBF for deprecating Python2 usage

[Scott Kitterman]

On August 4, 2017 6:49:23 AM EDT, Matthias Klose  wrote:
>On 03.08.2017 21:08, ba...@debian.org wrote:
>> On Aug 3, 2017, at 17:57, Matthias Klose  wrote:
>>>
>>> While at DebCamp, Stefano Rivera and I sat down to analyze what
>needs to be done
>>> to deprecate Python2 usage within the distribution.  It might not be
>possible to
>>> drop Python2 for the next release, but there are still too many
>issues with
>>> packages.  For now we identified some categories which need fixing.
>These are
>>> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal,
>resulting in
>>> more than 3000 bug reports.  That's a high number, on the other hand
>we won't
>>> make much progress if the issues are not named.  My intent is to
>bring that up
>>> in the Python BoF next week at DebConf and then filing these bug
>reports with
>>> the user tags mentiond on the wiki page.
>> 
>> Great to hear that you guys talked about it.
>> 
>> Just a quick note that PEP 394 discussions have revived, lead by the
>Fedora folks.  Please do check out the new thread, especially if you
>have opinions about what /usr/bin/python should do once Python 2.7 is
>EOL.
>> 
>> https://mail.python.org/pipermail/linux-sig/2017-August/thread.html
>
>I replied to this thread.  I think there should be one release which is
>not
>shipping /usr/bin/python before /usr/bin/python should be reused and
>pointed at
>python (>> 2). This should be good enough to get all scripts actively
>converted
>which are not part of the distribution.  If that release is buster, we
>should
>require the use of python2 instead of python now, document that in
>policy and
>provide a lintian check for that.
>
>Matthias

I disagree.

There will be users whose reaction to removal of python2.7 will be to compile 
their own.

Reintroducing /usr/bin/python as a python3 version risks their systems for no 
benefit (since all python3 stuff points to /usr/bin/python3 and works fine).  
Just let it go and don't bring it back.

As far as when to drop it goes, I believe that the python-django maintainers 
are planning on releasing Buster with Django 1.11 as bilingual python2/python3. 
 Then they'll go to Django 2, which is python3 only, early in the Bullseye 
cycle.

For the chunk of the Python ecosystem that is built around Django, keeping 
python2.7 in for Buster would be a significant aid to transition.

Scott K

Re: MBF for deprecating Python2 usage

[Matthias Klose]

On 03.08.2017 21:08, ba...@debian.org wrote:
> On Aug 3, 2017, at 17:57, Matthias Klose  wrote:
>>
>> While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be 
>> done
>> to deprecate Python2 usage within the distribution.  It might not be 
>> possible to
>> drop Python2 for the next release, but there are still too many issues with
>> packages.  For now we identified some categories which need fixing. These are
>> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, 
>> resulting in
>> more than 3000 bug reports.  That's a high number, on the other hand we won't
>> make much progress if the issues are not named.  My intent is to bring that 
>> up
>> in the Python BoF next week at DebConf and then filing these bug reports with
>> the user tags mentiond on the wiki page.
> 
> Great to hear that you guys talked about it.
> 
> Just a quick note that PEP 394 discussions have revived, lead by the Fedora 
> folks.  Please do check out the new thread, especially if you have opinions 
> about what /usr/bin/python should do once Python 2.7 is EOL.
> 
> https://mail.python.org/pipermail/linux-sig/2017-August/thread.html

I replied to this thread.  I think there should be one release which is not
shipping /usr/bin/python before /usr/bin/python should be reused and pointed at
python (>> 2). This should be good enough to get all scripts actively converted
which are not part of the distribution.  If that release is buster, we should
require the use of python2 instead of python now, document that in policy and
provide a lintian check for that.

Matthias

Re: MBF for deprecating Python2 usage

[Piotr Ożarowski]

[Scott Kitterman, 2017-08-04]
> Read it.  I remain completely convinced that /usr/bin/python pointing
> at a python3 version is utterly wrong and a disservice to our users.
> Even after we remove python2.7, people will be locally compiling it
> and using it for a decade.
> 
> Such a change would be actively user hostile.
> 
> When python2.7 goes away, /usr/bin/python should go too.
> 
> Before python2.7 goes away, any python3 scripts/apps need to work with
> /usr/bin/python3.  Making them work with /usr/bin/python too doesn't
> actually help anything.  All it does is look pretty for us old timers.
> 
> I've said before and I've yet to see anything to cause me to have any
> doubt on the matter, that as long as I am one of the python-defaults
> maintainers, /usr/bin/python will not point at a python3 version.

same here. Way too many not packaged /usr/bin/python scripts around me
-- 
GPG: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

Re: MBF for deprecating Python2 usage

[Scott Kitterman]

Dropped d-devel.

On August 3, 2017 9:08:10 PM EDT, ba...@debian.org wrote:
>On Aug 3, 2017, at 17:57, Matthias Klose  wrote:
>> 
>> While at DebCamp, Stefano Rivera and I sat down to analyze what needs
>to be done
>> to deprecate Python2 usage within the distribution.  It might not be
>possible to
>> drop Python2 for the next release, but there are still too many
>issues with
>> packages.  For now we identified some categories which need fixing.
>These are
>> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal,
>resulting in
>> more than 3000 bug reports.  That's a high number, on the other hand
>we won't
>> make much progress if the issues are not named.  My intent is to
>bring that up
>> in the Python BoF next week at DebConf and then filing these bug
>reports with
>> the user tags mentiond on the wiki page.
>
>Great to hear that you guys talked about it.
>
>Just a quick note that PEP 394 discussions have revived, lead by the
>Fedora folks.  Please do check out the new thread, especially if you
>have opinions about what /usr/bin/python should do once Python 2.7 is
>EOL.
>
>https://mail.python.org/pipermail/linux-sig/2017-August/thread.html
>
>Cheers,
>-Barry

Read it.  I remain completely convinced that /usr/bin/python pointing at a 
python3 version is utterly wrong and a disservice to our users.  Even after we 
remove python2.7, people will be locally compiling it and using it for a decade.

Such a change would be actively user hostile.

When python2.7 goes away, /usr/bin/python should go too.

Before python2.7 goes away, any python3 scripts/apps need to work with 
/usr/bin/python3.  Making them work with /usr/bin/python too doesn't actually 
help anything.  All it does is look pretty for us old timers.

I've said before and I've yet to see anything to cause me to have any doubt on 
the matter, that as long as I am one of the python-defaults maintainers, 
/usr/bin/python will not point at a python3 version.

Scott K

Re: MBF for deprecating Python2 usage

[barry]

On Aug 3, 2017, at 17:57, Matthias Klose  wrote:
> 
> While at DebCamp, Stefano Rivera and I sat down to analyze what needs to be 
> done
> to deprecate Python2 usage within the distribution.  It might not be possible 
> to
> drop Python2 for the next release, but there are still too many issues with
> packages.  For now we identified some categories which need fixing. These are
> documented at https://wiki.debian.org/Sprints/2017/Python3Montreal, resulting 
> in
> more than 3000 bug reports.  That's a high number, on the other hand we won't
> make much progress if the issues are not named.  My intent is to bring that up
> in the Python BoF next week at DebConf and then filing these bug reports with
> the user tags mentiond on the wiki page.

Great to hear that you guys talked about it.

Just a quick note that PEP 394 discussions have revived, lead by the Fedora 
folks.  Please do check out the new thread, especially if you have opinions 
about what /usr/bin/python should do once Python 2.7 is EOL.

https://mail.python.org/pipermail/linux-sig/2017-August/thread.html

Cheers,
-Barry



signature.asc
Description: Message signed with OpenPGP