Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Richard van der Hoff]

On 08/06/2020 08:04, Glyph wrote:
> 

I'm going to start here by saying: I agree with almost all of what you 
wrote, but at the end of the day, I don't get to determine our 
customers' policies. You can try to explain to them why their policies 
are misguided, but particularly when you're working with a large 
organisation, change can be very slow. So you end up working around the 
policy, whether you agree with it or not. In practical terms, that means 
that for now at least we need to support Python 3.5.


As Erik said, we certainly have no right to demand that Twisted continue 
to support 3.5: indeed, if dropping support will deliver value to the 
project, then I'd encourage you to go for it; and as you've already 
said, the whole thing is probably moot anyway given the timescales we're 
talking about.


I believe in this case its a general desire to keep track of what 
packages are running and where they've come from. They basically 
trust that packages from official Debian repositories are probably 
safe from being tampered with, whereas random tarballs of code from 
the web are not safe (unless they're signed by someone they trust or 
whatever).


I think this sounds like a misunderstanding of Debian's vetting 
process?  It's not like there's a ton of additional auditing that goes 
into packaging something.  There's definitely an authentication 
process for both Twisted and Python, although this attestation could 
be somewhat stronger and less centralized, PyPI does quite a bit of 
heavy lifting there.


I think it's less that they think that Debian does extra vetting, and 
more that, especially if you're managing whole fleets of servers, then 
if everything runs the same version, it's easier to keep track of what 
you need to upgrade when there's a "security" bug. And yes, there are 
plenty of counterarguments to this, but that's the reasoning.



 1. Many non-"security" bugs are in fact security bugs that nobody has
noticed you can exploit.
 2. Many "low-severity" or "un-exploiable" security bugs are in fact
exploitable
 3. "supported" distros rarely take care to backport many patches for
their software, and when they do, they often make undetected
errors (like debian's infamous ssh bug) which are analyzed by far
fewer security analysts than the upstream source code.

These are probably all true, but taken to their logical extreme, the 
conclusion seems to be "you should always run the bleeding edge of all 
software, to make sure you've got all the latest bug fixes". I don't 
think you're really arguing for that, so the point is: we end up 
nominating "stable" versions, and trying to make an assessment as to 
which bugfixes are worth backporting. That latter part is a subjective 
decision, and the question is who you trust to make it. You may not 
trust Debian to make that decision on your behalf (with perfectly valid 
reasons), but plenty of others do.
So I feel like the folks making the decision to stick with these old 
"supported" distros are only getting half the story - sure, it won't 
break, but are they /actually/ getting the security fixes that they 
think they are?  Debian's staff are stretched pretty thin as it is.


A counterpoint here is that the Python in oldstable has had several 
years of bugfixes, and of course it was the primary Debian-supported 
version of Python for a good couple of years. Again, I know that 
peoples' assessment of Debian's ability or competence varies, but I 
don't think it's *unreasonable* to assume that by this point the worst 
problems with that version of Python have been shaken out (and that if a 
significant new problem arises, a fix will be made).


In terms of twisted dropping support of 3.5, I guess the question is 
to what extent do you want applications to be hassle free to deploy 
on the more "enterprise" style environment?


One other confusion I have about these environments is why they want 
very-old Python but don't /also/ want very-old Twisted.


Well, again, it comes down to fleet management, and responsibility for 
"security". From the customer's point of view, they want to provide a 
python interpreter which runs our application. So the Python interpreter 
is their responsibility (hence: use Debian oldstable everywhere), 
whereas the stuff running on it is ours. Plus, since there's only one 
thing using Twisted in their network, it's inherently easier to maintain 
a single version.


(I also fear that there is a misguided belief that security 
vulnerabilities are "worse" if they are in the Python interpreter, 
because that runs native code, whereas Twisted can't possibly do 
anything that bad because "interpreted language". I mention this only 
for completeness, and fully realise it's nonsense.)


But supporting old Pythons, old service_identity modules, old 
OpenSSL's, etc, has been seeming more and more to me like a disservice 
to the community, because it facilitates the adoption of slow, 
insecure, dangerous 

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

> On May 28, 2020, at 8:31 AM, Erik Johnston  wrote:
> 
> 
> On 23/05/2020 06:39, Glyph wrote:
>> 
>>> On May 19, 2020, at 1:52 AM, Richard van der Hoff >> > wrote:
>>> 
>>> On 16/05/2020 06:56, Glyph wrote:
 
 
> On May 15, 2020, at 8:40 PM, Craig Rodrigues  > wrote:
> 
> Maybe it would be OK to do one more release of Twisted and announce that 
> as the last release supporting Python 3.5, before
> dropping support?  
 
 Yeah; whenever we drop a Python version we should always support at least 
 one more release, so that people have some notice before they lose access 
 to the next set of security updates.
 
 Any 3.5 users on this list who would want to postpone it longer than this?
 
>>> Sadly we have an important customer whose servers run debian oldstable, 
>>> which means we need to stay compatible with 3.5 until we can persuade them 
>>> to upgrade, and it's taken a couple of years to get them off python 2.7...
>>> 
>>> I'm not sure that should necessarily affect your plans, but I doubt we're 
>>> alone in this situation.
>>> 
>> 
>> I guess one thing I'm curious about is why your application would need to be 
>> installed along with the system Python on those OS versions?  It seems like 
>> a packaging strategy that ignored the fossilized versions that Debian 
>> packages with the system and just built its own Python would be more 
>> reliable and allow for upgrading at least most Python dependencies well 
>> beyond what the system would allow by policy.  Or, for that matter, why not 
>> just run in a Docker container?
>> 
>> Matrix is a pretty big user, and so in some sense I care about this specific 
>> case, but I also find the general question interesting, because I have 
>> difficulty reasoning about how long to support older versions of things in 
>> the modern application packaging environment where containers, virtualenvs, 
>> and associated tooling make it possible to effectively ignore the base 
>> environment. When & why do you have to pay attention to it?
>> 
>> -glyph
> 
> I believe in this case its a general desire to keep track of what packages 
> are running and where they've come from. They basically trust that packages 
> from official Debian repositories are probably safe from being tampered with, 
> whereas random tarballs of code from the web are not safe (unless they're 
> signed by someone they trust or whatever).
> 
I think this sounds like a misunderstanding of Debian's vetting process?  It's 
not like there's a ton of additional auditing that goes into packaging 
something.  There's definitely an authentication process for both Twisted and 
Python, although this attestation could be somewhat stronger and less 
centralized, PyPI does quite a bit of heavy lifting there.
> Now, I think it would be possible to get a newer version of Python on their 
> infrastructure if we needed, but I'm sure there would be hoops that would 
> need to be jumped through and justifications given, etc, which would 
> undoubtedly take some time. So really it just means extra faff for them and 
> us, especially since we're only a small part of their overall infrastructure.
> 
It's worth noting that Python 3.5 itself is out of security support (per 
https://www.python.org/dev/peps/pep-0478/ 
) and it's not like Debian employs a 
team of Python security-support engineers to continue maintaining this dead 
branch.  If their infrastructure were to try to comply with various security 
standards such as PCI-DSS, you might bump into the need to upgrade to a newer 
version with available future patch releases anyway.
> Then there is the fact that they're not unique. While oldstable is, well, 
> old, its still very much supported and so there's going to be a bunch of 
> "enterprise" (for want of a better term) customers who will still be using 
> it, and we'll need to go through the faff each and every time, which is quite 
> tedious. Come the Autumn when oldstable stops being supported (or at least, 
> goes into LTS mode),
> 
You say "Autumn", but...  oldstable is supported for the purposes of migrating 
off of it... it's not really even supported by Debian proper.  To quote from 
https://wiki.debian.org/DebianOldStable 
 :

>>> The security team tries to support a stable distribution for about one year 
>>> after the next stable distribution has been released, except when another 
>>> stable distribution is released within this year. It is not possible to 
>>> support three distributions; supporting two simultaneously is already 
>>> difficult enough.

According to https://wiki.debian.org/DebianBuster 
, Buster was released on 2019-07-06, 
which means we've now got less than one month of even this small amount of 
support that is offered; i.e. the expectation from 

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

> On May 28, 2020, at 3:13 PM, Tom Most  wrote:
> 
> On Fri, May 22, 2020, at 10:54 PM, Glyph wrote:
>> So, pidfd's cool, we should totally use it. Also we should use posix_spawn 
>> and maybe some other stuff too. But I wonder if there's any heuristic we 
>> could use to speed up our current strategy, like ordering the to-reap list 
>> by putting things with no open FDs at the front of it? And optimistically 
>> assuming that once we've found something to reap, maybe we can stop? And 
>> maybe it should run in a cooperator, rather than just blocking the reactor 
>> indefinitely?
> 
> Those all sound like reasonable optimizations with little downside to me.

Great!  Anybody feel like filing some tickets? :)

>> Honestly it had not occurred to me that people were managing 20k+ python 
>> interpreters at a time with spawnProcess (although, yikes, amazing, you've 
>> gotta talk more about this application, and what kind of hardware you're on!)
> 
> Nonono, that was a synthetic stress test. It did spawn Python processes, but 
> they immediately called `exec sleep` to conserve memory.
> 
> The result of the test was that the service would run out of memory _well_ 
> before inefficiencies in Twisted's reaping scheme became a problem.


Aah, that's less exciting ;).

-g
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Tom Most]

On Fri, May 22, 2020, at 10:54 PM, Glyph wrote:
> So, pidfd's cool, we should totally use it. Also we should use posix_spawn 
> and maybe some other stuff too. But I wonder if there's any heuristic we 
> could use to speed up our current strategy, like ordering the to-reap list by 
> putting things with no open FDs at the front of it? And optimistically 
> assuming that once we've found something to reap, maybe we can stop? And 
> maybe it should run in a cooperator, rather than just blocking the reactor 
> indefinitely?

Those all sound like reasonable optimizations with little downside to me.

> Honestly it had not occurred to me that people were managing 20k+ python 
> interpreters at a time with spawnProcess (although, yikes, amazing, you've 
> gotta talk more about this application, and what kind of hardware you're on!)

Nonono, that was a synthetic stress test. It did spawn Python processes, but 
they immediately called `exec sleep` to conserve memory.

The result of the test was that the service would run out of memory _well_ 
before inefficiencies in Twisted's reaping scheme became a problem.
 
---Tom

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

> On May 19, 2020, at 11:14 AM, Tom Most  wrote:
> 
> On Tue, May 19, 2020, at 5:41 AM, Jean-Paul Calderone wrote:
>> On Tue, May 19, 2020 at 8:33 AM Barry Scott > > wrote:
>> Why not waitpid for all children (pid=0 or pid=-1) in a single call? 
>> 
>> Because then you might reap a child process someone launched with a 
>> different API - eg the Python subprocess module.
> 
> Exactly. I am currently using this as a workaround, but it's not a general 
> solution. You need to be careful because subprocess use hides in many places. 
> For example, beware uuid.uuid1().

So, pidfd's cool, we should totally use it.  Also we should use posix_spawn and 
maybe some other stuff too.  But I wonder if there's any heuristic we could use 
to speed up our current strategy, like ordering the to-reap list by putting 
things with no open FDs at the front of it?  And optimistically assuming that 
once we've found something to reap, maybe we can stop?  And maybe it should run 
in a cooperator, rather than just blocking the reactor indefinitely?  Honestly 
it had not occurred to me that people were managing 20k+ python interpreters at 
a time with spawnProcess (although, yikes, amazing, you've gotta talk more 
about this application, and what kind of hardware you're on!)

Anyway there's lot of exciting room for performance improvements here, it would 
be nice to have some benchmarks for this kind of high-load process-monitor 
application.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

> On May 19, 2020, at 5:41 AM, Jean-Paul Calderone  
> wrote:
> 
> On Tue, May 19, 2020 at 8:33 AM Barry Scott  > wrote:
> On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
> 
> > 
> > If looking improve process management in Twisted, I think that the
> > lowest-hanging fruit is to watch children with pidfd, rather than reaping
> > all children on SIGCHLD. This O(n) reaping --- one waitpid() call per
> > running process --- starts to consume substantial CPU. IIRC it pegged a
> > core at 20,000 processes or so.
> 
> Why not waitpid for all children (pid=0 or pid=-1) in a single call?
> 
> Barry
> 
> Because then you might reap a child process someone launched with a different 
> API - eg the Python subprocess module.

In case it is not clear from context - we used to do exactly this.  It broke a 
lot of stuff.  The current approach is slow, but robust.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

> On May 19, 2020, at 1:52 AM, Richard van der Hoff  wrote:
> 
> On 16/05/2020 06:56, Glyph wrote:
>> 
>> 
>>> On May 15, 2020, at 8:40 PM, Craig Rodrigues >> > wrote:
>>> 
>>> Maybe it would be OK to do one more release of Twisted and announce that as 
>>> the last release supporting Python 3.5, before
>>> dropping support?  
>> 
>> Yeah; whenever we drop a Python version we should always support at least 
>> one more release, so that people have some notice before they lose access to 
>> the next set of security updates.
>> 
>> Any 3.5 users on this list who would want to postpone it longer than this?
>> 
> Sadly we have an important customer whose servers run debian oldstable, which 
> means we need to stay compatible with 3.5 until we can persuade them to 
> upgrade, and it's taken a couple of years to get them off python 2.7...
> 
> I'm not sure that should necessarily affect your plans, but I doubt we're 
> alone in this situation.
> 

I guess one thing I'm curious about is why your application would need to be 
installed along with the system Python on those OS versions?  It seems like a 
packaging strategy that ignored the fossilized versions that Debian packages 
with the system and just built its own Python would be more reliable and allow 
for upgrading at least most Python dependencies well beyond what the system 
would allow by policy.  Or, for that matter, why not just run in a Docker 
container?

Matrix is a pretty big user, and so in some sense I care about this specific 
case, but I also find the general question interesting, because I have 
difficulty reasoning about how long to support older versions of things in the 
modern application packaging environment where containers, virtualenvs, and 
associated tooling make it possible to effectively ignore the base environment. 
When & why do you have to pay attention to it?

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Tom Most]

On Tue, May 19, 2020, at 5:41 AM, Jean-Paul Calderone wrote:
> On Tue, May 19, 2020 at 8:33 AM Barry Scott  
> wrote:
>> Why not waitpid for all children (pid=0 or pid=-1) in a single call? 
> 
> Because then you might reap a child process someone launched with a different 
> API - eg the Python subprocess module.

Exactly. I am currently using this as a workaround, but it's not a general 
solution. You need to be careful because subprocess use hides in many places. 
For example, beware uuid.uuid1().

---Tom

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Jean-Paul Calderone]

On Tue, May 19, 2020 at 8:33 AM Barry Scott 
wrote:

> On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
>
> >
> > If looking improve process management in Twisted, I think that the
> > lowest-hanging fruit is to watch children with pidfd, rather than reaping
> > all children on SIGCHLD. This O(n) reaping --- one waitpid() call per
> > running process --- starts to consume substantial CPU. IIRC it pegged a
> > core at 20,000 processes or so.
>
> Why not waitpid for all children (pid=0 or pid=-1) in a single call?
>
> Barry
>

Because then you might reap a child process someone launched with a
different API - eg the Python subprocess module.

Jean-Paul
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Barry Scott]

On Monday, 18 May 2020 22:21:16 BST Tom Most wrote:
> On Sun, May 17, 2020, at 12:22 PM, Glyph wrote:
> >> In the one case that may be complicated I think that pinning Twisted will
> >> be acceptable, since the service in question is a process supervisor
> >> without direct external exposure.> 
> > Well in that case I'll just do my best to add lots of goodies (posix_spawn
> > support!) that will make you want to upgrade that to current twisted and
> > python 3.8 :).
> Oh it's not for lack of wanting. :-)
> 
> The service in question is actually spawning Python processes, so
> posix_spawn support doesn't seem likely to help much: fork() + exec() is a
> drop in the bucket compared to Python imports, particularly in Python 3
> (I'm hoping to explore oxidized-importer [1] to help with this at some
> point).

Would putting all the .pyc/.pyo into a zip file help with the import problem?
You only have one file to open, the .zip, and no compile as its only 
.pyc/.pyo.


> 
> If looking improve process management in Twisted, I think that the
> lowest-hanging fruit is to watch children with pidfd, rather than reaping
> all children on SIGCHLD. This O(n) reaping --- one waitpid() call per
> running process --- starts to consume substantial CPU. IIRC it pegged a
> core at 20,000 processes or so.

Why not waitpid for all children (pid=0 or pid=-1) in a single call?

Barry
> 
> ---Tom
> 
> [1]: https://pypi.org/project/oxidized-importer/




___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Richard van der Hoff]

On 16/05/2020 06:56, Glyph wrote:



On May 15, 2020, at 8:40 PM, Craig Rodrigues > wrote:


Maybe it would be OK to do one more release of Twisted and announce 
that as the last release supporting Python 3.5, before

dropping support?


Yeah; whenever we drop a Python version we should always support at 
least one more release, so that people have some notice before they 
lose access to the next set of security updates.


Any 3.5 users on this list who would want to postpone it longer than this?

Sadly we have an important customer whose servers run debian oldstable, 
which means we need to stay compatible with 3.5 until we can persuade 
them to upgrade, and it's taken a couple of years to get them off python 
2.7...


I'm not sure that should necessarily affect your plans, but I doubt 
we're alone in this situation.



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Tom Most]

On Sun, May 17, 2020, at 12:22 PM, Glyph wrote:
>> In the one case that may be complicated I think that pinning Twisted will be 
>> acceptable, since the service in question is a process supervisor without 
>> direct external exposure.
> Well in that case I'll just do my best to add lots of goodies (posix_spawn 
> support!) that will make you want to upgrade that to current twisted and 
> python 3.8 :).

Oh it's not for lack of wanting. :-)

The service in question is actually spawning Python processes, so posix_spawn 
support doesn't seem likely to help much: fork() + exec() is a drop in the 
bucket compared to Python imports, particularly in Python 3 (I'm hoping to 
explore oxidized-importer [1] to help with this at some point).

If looking improve process management in Twisted, I think that the 
lowest-hanging fruit is to watch children with pidfd, rather than reaping all 
children on SIGCHLD. This O(n) reaping --- one waitpid() call per running 
process --- starts to consume substantial CPU. IIRC it pegged a core at 20,000 
processes or so.

---Tom

[1]: https://pypi.org/project/oxidized-importer/___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

> On May 17, 2020, at 12:10 PM, Tom Most  wrote:
> 
> On Sun, May 17, 2020, at 11:31 AM, Glyph wrote:
>> 
>>> On May 16, 2020, at 3:12 PM, Tom Most >> > wrote:
>>> I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's 
>>> Python 3.5. However, I am fine with dropping Python 3.5 support. I think 
>>> it's certainly worth it for variable annotations and contextvars.
>> 
>> Would your plan to be to just keep twisted pinned until you can EOL ubuntu? 
>> I don't want to be putting (non-trivial numbers of) users into the position 
>> of having to skip a bunch of updates.
>> 
>> -glyph
> 
> Yes, I'm already working towards EOL of Ubuntu 16.04. I expect to have that 
> work completed by the end of the year.

Glad to hear it :).

> In the one case that may be complicated I think that pinning Twisted will be 
> acceptable, since the service in question is a process supervisor without 
> direct external exposure.

Well in that case I'll just do my best to add lots of goodies (posix_spawn 
support!) that will make you want to upgrade that to current twisted and python 
3.8 :).

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Tom Most]

On Sun, May 17, 2020, at 11:31 AM, Glyph wrote:
> 
>> On May 16, 2020, at 3:12 PM, Tom Most  wrote:
>> I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's 
>> Python 3.5. However, I am fine with dropping Python 3.5 support. I think 
>> it's certainly worth it for variable annotations and contextvars.
> 
> Would your plan to be to just keep twisted pinned until you can EOL ubuntu? I 
> don't want to be putting (non-trivial numbers of) users into the position of 
> having to skip a bunch of updates.
> 
> -glyph

Yes, I'm already working towards EOL of Ubuntu 16.04. I expect to have that 
work completed by the end of the year.

In the one case that may be complicated I think that pinning Twisted will be 
acceptable, since the service in question is a process supervisor without 
direct external exposure.

---Tom___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

> On May 16, 2020, at 3:12 PM, Tom Most  wrote:
> 
> 
> On Fri, May 15, 2020, at 10:56 PM, Glyph wrote:
>> 
>> 
>>> On May 15, 2020, at 8:40 PM, Craig Rodrigues >> > wrote:
>>> 
>>> Maybe it would be OK to do one more release of Twisted and announce that as 
>>> the last release supporting Python 3.5, before
>>> dropping support?  
>> 
>> Yeah; whenever we drop a Python version we should always support at least 
>> one more release, so that people have some notice before they lose access to 
>> the next set of security updates.
>> 
>> Any 3.5 users on this list who would want to postpone it longer than this?
>> 
>> -glyph

> I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's Python 
> 3.5. However, I am fine with dropping Python 3.5 support. I think it's 
> certainly worth it for variable annotations and contextvars.

Would your plan to be to just keep twisted pinned until you can EOL ubuntu? I 
don't want to be putting (non-trivial numbers of) users into the position of 
having to skip a bunch of updates.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Tom Most]

I'm (unfortunately) still shipping some Twisted apps on Ubuntu 16.04's Python 
3.5. However, I am fine with dropping Python 3.5 support. I think it's 
certainly worth it for variable annotations and contextvars.

---Tom

On Fri, May 15, 2020, at 10:56 PM, Glyph wrote:
> 
> 
>> On May 15, 2020, at 8:40 PM, Craig Rodrigues  wrote:
>> 
>> Maybe it would be OK to do one more release of Twisted and announce that as 
>> the last release supporting Python 3.5, before
>> dropping support? 
> Yeah; whenever we drop a Python version we should always support at least one 
> more release, so that people have some notice before they lose access to the 
> next set of security updates.
> 
> Any 3.5 users on this list who would want to postpone it longer than this?
> 
> -glyph
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

> On May 15, 2020, at 8:40 PM, Craig Rodrigues  wrote:
> 
> Maybe it would be OK to do one more release of Twisted and announce that as 
> the last release supporting Python 3.5, before
> dropping support?  

Yeah; whenever we drop a Python version we should always support at least one 
more release, so that people have some notice before they lose access to the 
next set of security updates.

Any 3.5 users on this list who would want to postpone it longer than this?

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Craig Rodrigues]

Personally, I don't have any dependencies on Python 3.5.
I just suggested once year's notice to give people time to adjust,
but if no one really cares, dropping 3.5 support some time this year is
fine by me.

Maybe it would be OK to do one more release of Twisted and announce that as
the last release supporting Python 3.5, before
dropping support?
--
Craig


On Wed, May 13, 2020 at 10:37 PM Amber Brown 
wrote:

> Xenial's 3.5.1/3.5.2 sucks and doesn't have some asyncio functionality
> (due to it being provisional then?) which means things like Eliot doesn't
> work due to aiocontextvars barfing on missing functionality. There's also
> 20.04 LTS now, making 16.04 old-old-LTS. I don't think it's worth it.
>
> - Amber
>
> On Thu, 14 May 2020, at 15:18, Moshe Zadka wrote:
> > May 2021 seems to be right around when Xenial (Which has 3.5) falls
> > into ESM, so maybe we can wait?
> > I don't have strong feelings, but it is a datapoint.
> >
> > Moshe Z.
> >
> > On Wed, May 13, 2020, at 22:13, Glyph wrote:
> > > On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) <
> hawk...@atleastfornow.net> wrote:
> > > >
> > > > On 14/5/20 5:10 am, Craig Rodrigues wrote:
> > > >> I would like to propose that Twisted drop support for Python 3.5
> one year from now in 2021.
> > > >
> > > > I was considering proposing dropping it soon (I was thinking when we
> had 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
> > >
> > >
> > > I'm also inclined to say sooner rather than later; I don't know the
> numbers but my impression is that 3.x uptake began for real at 3.4 and in
> earnest at 3.6, so there aren't that many people still on 3.5.
> > >
> > > The lack of variable annotations, in particular, seems like a sticking
> point with 3.5 as we attempt to adopt mypy.
> > >
> > > -g
> > >
> > >
> > > ___
> > > Twisted-Python mailing list
> > > Twisted-Python@twistedmatrix.com
> > > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> > >
> >
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> >
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Craig Rodrigues]

On Wed, May 13, 2020 at 10:05 PM Amber Brown (hawkowl) <
hawk...@atleastfornow.net> wrote:

>
> So, I guess if we have 3.8 support, the next release can announce the
> dropping of 3.5, and then the release after that can be the final one
> with support.
>

In trunk, on Python 3.8, all the tests in CI pass on Mac and Linux.
With Windows and Python 3.8, there are 6 errors due to AyncioReactorTests
failing:
https://dev.azure.com/twistedmatrix/twisted/_build/results?buildId=1771=logs=ea01aad7-d5f3-5f0e-2fda-71d72e596cac=f47a629b-ce24-5ee0-314a-24c887b72299=16071

most likely due to this: https://twistedmatrix.com/trac/ticket/9766

where in Python 3.8, the ProactorEventLoop was made the default on Windows.

So once this is fixed on Windows, I think we are pretty close to having
clean CI runs on Python 3.8 on all platforms.
--
Craig
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

> On May 14, 2020, at 3:55 AM, Amber Brown  wrote:
> 
> 
> 
> On Thu, 14 May 2020, at 16:54, Glyph wrote:
>> 
>>> $ /usr/bin/python3
>>> Python 3.7.3 (default, Apr 7 2020, 14:06:47)
>>> [Clang 11.0.3 (clang-1103.0.32.59)] on darwin
>>> Type "help", "copyright", "credits" or "license" for more information.
>> import sys
>> sys.executable
>>> '/Applications/Xcode.app/Contents/Developer/usr/bin/python3'
>> 
> 
> wow thanks I hate it

ITYM 'Wow thanks I hate it.app'

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Amber Brown]

On Thu, 14 May 2020, at 16:54, Glyph wrote:
> 
> > $ /usr/bin/python3
> > Python 3.7.3 (default, Apr 7 2020, 14:06:47)
> > [Clang 11.0.3 (clang-1103.0.32.59)] on darwin
> > Type "help", "copyright", "credits" or "license" for more information.
> > >>> import sys
> > >>> sys.executable
> > '/Applications/Xcode.app/Contents/Developer/usr/bin/python3'
> > >>>

wow thanks I hate it

> -g
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

> On May 13, 2020, at 11:13 PM, Amber Brown  wrote:
> 
> On Thu, 14 May 2020, at 16:09, Hynek Schlawack wrote:
>> 
>> 
>>> On 14. May 2020, at 07:04, Amber Brown (hawkowl) 
>>>  wrote:
>>> 
>>> - MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.
>> 
>> FWIW this is not true anymore. Catalina comes – for all its faults – 
>> with Python 3.7:
>> 
> 
> *puts fingers in ears* sorry I can't hear you, I'm in my happy place where I 
> don't have to think of software in ".framework" folders la la la la

Oh hawkowl.  Either you die an a.out, or you live long enough to become a .dylib

>  But, it's good to know, anyway. 

Unfortunately it's only kind of true!

$ /usr/bin/python3
Python 3.7.3 (default, Apr  7 2020, 14:06:47)
[Clang 11.0.3 (clang-1103.0.32.59)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/Applications/Xcode.app/Contents/Developer/usr/bin/python3'
>>>

You can also get one from /Library/Developer/CommandLineTools/usr/bin/python3 
if you do `xcode-select --install` but it's not present in the macOS base image 
and you can't use it from applications.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Amber Brown]

On Thu, 14 May 2020, at 16:09, Hynek Schlawack wrote:
> 
> 
> > On 14. May 2020, at 07:04, Amber Brown (hawkowl) 
> >  wrote:
> > 
> > - MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.
> 
> FWIW this is not true anymore. Catalina comes – for all its faults – 
> with Python 3.7:
> 

*puts fingers in ears* sorry I can't hear you, I'm in my happy place where I 
don't have to think of software in ".framework" folders la la la la

 But, it's good to know, anyway. 

- Amber

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Hynek Schlawack]

> On 14. May 2020, at 07:04, Amber Brown (hawkowl)  
> wrote:
> 
> - MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.

FWIW this is not true anymore. Catalina comes – for all its faults – with 
Python 3.7:

❯ /usr/bin/python2

WARNING: Python 2.7 is not recommended.
This version is included in macOS for compatibility with legacy software.
Future versions of macOS will not include Python 2.7.
Instead, it is recommended that you transition to using 'python3' from within 
Terminal.

Python 2.7.16 (default, Feb 29 2020, 01:55:37)
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- 
on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> ^D

~ took 6s
❯ /usr/bin/python3
Python 3.7.3 (default, Apr  7 2020, 14:06:47)
[Clang 11.0.3 (clang-1103.0.32.59)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Amber Brown]

On Thu, 14 May 2020, at 15:13, Glyph wrote:
> On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) 
>  wrote:
> > 
> > On 14/5/20 5:10 am, Craig Rodrigues wrote:
> >> I would like to propose that Twisted drop support for Python 3.5 one year 
> >> from now in 2021.
> > 
> > I was considering proposing dropping it soon (I was thinking when we had 
> > 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
> 
> 
> I'm also inclined to say sooner rather than later; I don't know the 
> numbers but my impression is that 3.x uptake began for real at 3.4 and 
> in earnest at 3.6, so there aren't that many people still on 3.5.

A year ago (https://atleastfornow.net/posts/py2-py3-twisted-downloads/), those 
numbers said that 3.5 was 6.7% of users for the currently released version of 
Twisted, and 4.37% of all Twisted downloads that month. I could probably rerun 
them, but I think it's kind of obvious what the results would be.

> 
> The lack of variable annotations, in particular, seems like a sticking 
> point with 3.5 as we attempt to adopt mypy.
> 
> -g
> 
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Amber Brown]

Xenial's 3.5.1/3.5.2 sucks and doesn't have some asyncio functionality (due to 
it being provisional then?) which means things like Eliot doesn't work due to 
aiocontextvars barfing on missing functionality. There's also 20.04 LTS now, 
making 16.04 old-old-LTS. I don't think it's worth it. 

- Amber

On Thu, 14 May 2020, at 15:18, Moshe Zadka wrote:
> May 2021 seems to be right around when Xenial (Which has 3.5) falls 
> into ESM, so maybe we can wait?
> I don't have strong feelings, but it is a datapoint.
> 
> Moshe Z.
> 
> On Wed, May 13, 2020, at 22:13, Glyph wrote:
> > On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) 
> >  wrote:
> > > 
> > > On 14/5/20 5:10 am, Craig Rodrigues wrote:
> > >> I would like to propose that Twisted drop support for Python 3.5 one 
> > >> year from now in 2021.
> > > 
> > > I was considering proposing dropping it soon (I was thinking when we had 
> > > 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
> > 
> > 
> > I'm also inclined to say sooner rather than later; I don't know the numbers 
> > but my impression is that 3.x uptake began for real at 3.4 and in earnest 
> > at 3.6, so there aren't that many people still on 3.5.
> > 
> > The lack of variable annotations, in particular, seems like a sticking 
> > point with 3.5 as we attempt to adopt mypy.
> > 
> > -g
> > 
> > 
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> > 
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Moshe Zadka]

May 2021 seems to be right around when Xenial (Which has 3.5) falls into ESM, 
so maybe we can wait?
I don't have strong feelings, but it is a datapoint.

Moshe Z.

On Wed, May 13, 2020, at 22:13, Glyph wrote:
> On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl) 
>  wrote:
> > 
> > On 14/5/20 5:10 am, Craig Rodrigues wrote:
> >> I would like to propose that Twisted drop support for Python 3.5 one year 
> >> from now in 2021.
> > 
> > I was considering proposing dropping it soon (I was thinking when we had 
> > 3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).
> 
> 
> I'm also inclined to say sooner rather than later; I don't know the numbers 
> but my impression is that 3.x uptake began for real at 3.4 and in earnest at 
> 3.6, so there aren't that many people still on 3.5.
> 
> The lack of variable annotations, in particular, seems like a sticking point 
> with 3.5 as we attempt to adopt mypy.
> 
> -g
> 
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Glyph]

On May 13, 2020, at 10:04 PM, Amber Brown (hawkowl)  
wrote:
> 
> On 14/5/20 5:10 am, Craig Rodrigues wrote:
>> I would like to propose that Twisted drop support for Python 3.5 one year 
>> from now in 2021.
> 
> I was considering proposing dropping it soon (I was thinking when we had 3.8 
> support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).


I'm also inclined to say sooner rather than later; I don't know the numbers but 
my impression is that 3.x uptake began for real at 3.4 and in earnest at 3.6, 
so there aren't that many people still on 3.5.

The lack of variable annotations, in particular, seems like a sticking point 
with 3.5 as we attempt to adopt mypy.

-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Amber Brown (hawkowl)]

I was considering proposing dropping it soon (I was thinking when we had 
3.8 support, that 3.6/3.7/3.8 is a reasonable compatibility matrix).


Why I think this is probably fine:

- Debian Stable (buster) has 3.7. oldstable (stretch) has 3.5.3, but I 
don't think we care about oldstable.
- Ubuntu 18.04 has Python 3.6. 16.04 has 3.5.1, which is a sucky, buggy 
version that people shouldn't use anyway. Plus, getting a modern Python 
on Ubuntu isn't shockingly hard (deadsnakes makes it trivial).
- CentOS and RHEL 6 and 7 have Software Collections which makes Python 
3.6+ readily available.

- Windows has 3.8 available in the Store.
- MacOS doesn't ship a Python 3, but homebrew/python.org offer 3.8 easily.
- For everyone else, there's always Docker.

So, I guess if we have 3.8 support, the next release can announce the 
dropping of 3.5, and then the release after that can be the final one 
with support.


- Amber

On 14/5/20 5:10 am, Craig Rodrigues wrote:
I would like to propose that Twisted drop support for Python 3.5 one 
year from now in 2021.


Reasons:

1.  Python 3.5.9 will be the final release of Python 3.5, in November 
2019 ( https://www.python.org/dev/peps/pep-0478/ )
2.  Python 3.6.0 was released in December 2016 ( 
https://www.python.org/dev/peps/pep-0494/ )
3.  PyPy 7.1.1  introduced Python 3.6 support in April 2019 ( 
https://morepypy.blogspot.com/2019/04/pypy-711-bug-fix-release.html ).
     PyPy 7.3.1 is the latest PyPy version ( 
https://morepypy.blogspot.com/2020/04/pypy-731-released.html ).


4.  Python 3.6 introduced support for variable annotations ( 
https://www.python.org/dev/peps/pep-0526/ )
5.  Python 3.6 introduced new string interpolation (f-string) support ( 
https://www.python.org/dev/peps/pep-0498/ )
6.  Python 3.6 introduced support for asynchronous generators ( 
https://www.python.org/dev/peps/pep-0525/ ) and asynchronous 
comprehensions ( https://www.python.org/dev/peps/pep-0530/ )




Based on item 2., Python 3.6.0 has been around for almost 4 years, which 
is a reasonable amount of time.

Based on item 3., PyPy supports Python 3.6.

It might be nice to take advantage of some of the language features 
introduces in items 4-6.


I think dropping support one year from now in 2021, would give 
sufficient notice, and be in line with
the Twisted Compatibility Policy ( 
https://twistedmatrix.com/documents/current/core/development/policy/compatibility-policy.html 
).


None of these items are critial, but I thought I would ask, to get 
people's opinions.


--
Craig

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Craig Rodrigues]

I would like to propose that Twisted drop support for Python 3.5 one year
from now in 2021.

Reasons:

1.  Python 3.5.9 will be the final release of Python 3.5, in November 2019
( https://www.python.org/dev/peps/pep-0478/ )
2.  Python 3.6.0 was released in December 2016 (
https://www.python.org/dev/peps/pep-0494/ )
3.  PyPy 7.1.1  introduced Python 3.6 support in April 2019 (
https://morepypy.blogspot.com/2019/04/pypy-711-bug-fix-release.html ).
PyPy 7.3.1 is the latest PyPy version (
https://morepypy.blogspot.com/2020/04/pypy-731-released.html ).

4.  Python 3.6 introduced support for variable annotations (
https://www.python.org/dev/peps/pep-0526/ )
5.  Python 3.6 introduced new string interpolation (f-string) support (
https://www.python.org/dev/peps/pep-0498/ )
6.  Python 3.6 introduced support for asynchronous generators (
https://www.python.org/dev/peps/pep-0525/ ) and asynchronous comprehensions
( https://www.python.org/dev/peps/pep-0530/ )



Based on item 2., Python 3.6.0 has been around for almost 4 years, which is
a reasonable amount of time.
Based on item 3., PyPy supports Python 3.6.

It might be nice to take advantage of some of the language features
introduces in items 4-6.

I think dropping support one year from now in 2021, would give sufficient
notice, and be in line with
the Twisted Compatibility Policy (
https://twistedmatrix.com/documents/current/core/development/policy/compatibility-policy.html
).

None of these items are critial, but I thought I would ask, to get people's
opinions.

--
Craig
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python