Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement

[Richard van der Hoff]

On 16/07/2021 20:27, Glyph wrote:


One particular problem I came across was the type annotation on 
inlineCallbacks. I've filed 
https://twistedmatrix.com/trac/ticket/10231 about it - would 
appreciate thoughts.


This definitely looks wrong; there should be a TypeVar in there.  Adi, 
I'd go so far as to say that this should be a release blocker, 
although the change should be fairly minimal.


Richard, could you please make a proper PR for this to get CI kicked 
off and make sure the new annotation doesn't cause any failures?

Done, in the form of https://github.com/twisted/twisted/pull/1632.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement

[Richard van der Hoff]

On 16/07/2021 00:18, Glyph wrote:




On Jul 15, 2021, at 9:00 AM, Richard van der Hoff <mailto:rich...@matrix.org>> wrote:


We can't just go and add type annotations because we need to maintain 
compatibility with older Twisted (to make it possible to package in 
Debian et al).


Any suggestions for keeping mypy happy?

Are you saying you need it to typecheck against older versions or just 
run against them?


Ah, this gave me the clue I needed. We just need to run against them. 
Which means I can put type hints in comments, where they will be ignored 
at runtime. It's fiddly, but it will work well enough.


Thanks Glyph, and thanks to Adi and Barry for your suggestions too.

One particular problem I came across was the type annotation on 
inlineCallbacks. I've filed https://twistedmatrix.com/trac/ticket/10231 
about it - would appreciate thoughts.




___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement

[Richard van der Hoff]

On 11/07/2021 00:49, Glyph wrote:
Thank you Adi!  Very glad to see us collectively getting back on the 
release-management horse again!  And it was super encouraging to see 
how quickly you were able to get this out, with all the automation 
that everyone (not least of all yourself) has been building to make 
this process faster and simpler.



To echo this: thank you for your work, Adi!

It appears we've buried the lede on one of our biggest features in 
this release though - 
https://github.com/twisted/twisted/pull/1448/files 
 had no newsfile 
that shows up in the changelog, but it properly made Deferred into a 
generic, so I think this is the first release of Twisted where you 
might /reasonably/ be able to use mypy without your own custom stub 
files! :)


This looks very cool, but presents us with a problem.

mypy now complains about instantiations of Deferred without a type 
annotation:


> d = defer.Deferred()

results in an error:

> synapse/util/async_helpers.py:124: error: Need type annotation for 
'd'  [var-annotated]


We can't just go and add type annotations because we need to maintain 
compatibility with older Twisted (to make it possible to package in 
Debian et al).


Any suggestions for keeping mypy happy?



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] startTLS errors not propagating to Factory

[Richard van der Hoff]

On 29/04/2021 07:36, Glyph wrote:


In my tests at least, TLSMemoryBIOProtocol.connectionLost is doing 
exactly the right thing - it is called with an unhelpful reason, but 
substitutes back in the helpful reason which has already been stashed.


Rather, the problem, as I see it, is that it's not 
TLSMemoryBIOProtocol.connectionLost that calls 
Factory.clientConnectionLost. That is done by 
tcp.Client.connectionLost, via one of tcp.Client's myriad of base 
classes, at 
https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/internet/tcp.py#L508. 
Of course, that doesn't get the benefit of TLSMemoryBIOProtocol's 
reason switcheroo.


I'm still not quite sure who is in the wrong here.


Aah, yeah, this is a weird quirk of the ancient-style layering in the 
SMTP code :-|.  The way this /should/ work is by using HostnameEndpoint.


I'm not sure /exactly/ where we're going off the rails, but by using 
/both /the old 'startTLS' style of starting a TLS connection, as well 
as relying on ClientFactory rather than an Endpoint of some kind, 
means that we're getting this duplicate notification; the one that you 
get to Protocol.connectionLost will come from TLS and have useful 
information, but the one that goes to the Connector will be coming 
straight from TCP.


The right thing to fix here, I think, is to ignore 
clientConnectionLost entirely, and instead to have the protocol object 
relay its failure to some other differently-named method on 
SMTPSenderFactory.


Right! That sounds plausible, and certainly gives me some places to 
poke. I'll have another look later. Thanks very much!


R
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] startTLS errors not propagating to Factory

[Richard van der Hoff]

On 28/04/2021 07:06, Glyph wrote:


Is the SMTP code holding the Factory 
wrong? Or is it reasonable to expect the verification error to 
propagate into clientConnectionFailed - in which case, how could this 
work?


Thanks for your thoughts!


Hi Richard,

Sorry for the delayed response here.

This is a bug in Twisted, and I think it boils down to this line: 
https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/protocols/tls.py#L391 



The code as written here appears to be expecting this sequence:

 1. failVerification is called with a reason containing a helpful
OpenSSL verification error
 2. we save that reason as `self._reason` for reporting later, calling
abortConnection()
 3. since the connection got aborted, we expect our underlying transport
to call loseConnection on us
 4. we will then get a falsey reason [?!?!] and as such we will use
self._reason instead


Assumption 4 is nonsense and has never been true within Twisted as far 
as I know; connectionLost always gets a Failure, Failures are never 
falsey, so we will never use self._reason.  To fix this we need to 
actually honor self._reason under the conditions we expect, i.e. it's a 
ConnectionAborted 
https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/internet/error.py#L209 


Thanks very much for your thoughts on this, Glyph - it's always helpful 
to have an insight into the intended design when trying to resolve this 
sort of thing.


I don't follow your reasoning though. I think you might have misread the 
line you point to 
(https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/protocols/tls.py#L391). 
It is "self._reason or reason" - ie, if self._reason is already set, it 
will take precedence over reason.


In my tests at least, TLSMemoryBIOProtocol.connectionLost is doing 
exactly the right thing - it is called with an unhelpful reason, but 
substitutes back in the helpful reason which has already been stashed.


Rather, the problem, as I see it, is that it's not 
TLSMemoryBIOProtocol.connectionLost that calls 
Factory.clientConnectionLost. That is done by tcp.Client.connectionLost, 
via one of tcp.Client's myriad of base classes, at 
https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/internet/tcp.py#L508. 
Of course, that doesn't get the benefit of TLSMemoryBIOProtocol's reason 
switcheroo.


I'm still not quite sure who is in the wrong here.

Cheers

Richard


PS: yes, once we figure out what's going wrong here, I'll at least write 
up an issue...


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] startTLS errors not propagating to Factory

[Richard van der Hoff]

Hi folks,

I've been investigating 
https://github.com/matrix-org/synapse/issues/9566, which amounts to: 
"when there is a TLS error connecting to the SMTP server, the resultant 
exception is unreadable".


I think I've traced the problem to the fact that 
SMTPSenderFactory.clientConnectionFailed is being called with an 
unhelpful ConnectionAborted rather than anything more descriptive.


I've then reproduced that with a simpler test case: see 
https://gist.github.com/richvdh/909761ff5dab23f0873eeddd7936a740. As you 
can see, the output is: "Factory lost connection. Reason: Connection was 
aborted locally using ITCPTransport.abortConnection."


This seems to be thanks to TLSMemoryBIOProtocol.failVerification, which 
stashes the error and calls abortConnection(): 
https://github.com/twisted/twisted/blob/trunk/src/twisted/protocols/tls.py#L427.


At this point I'm struggling. Is the SMTP code holding the Factory 
wrong? Or is it reasonable to expect the verification error to propagate 
into clientConnectionFailed - in which case, how could this work?


Thanks for your thoughts!

Richard

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

[Richard van der Hoff]

On 03/03/2021 18:47, Glyph wrote:


I'll take this to the Synapse team to discuss further, but we could 
probably easily arrange for one of our CI runs to install Twisted 
trunk from git instead of pypi, which might be a good start.


This is specifically the approach I'd really rather /not/ take :) and 
here's why:


 1. You want to provide stability for you contributors so that if a
problem is introduced, you don't halt development on that
unrelated feature to diagnose the upstream issue.

 2. You want to ensure that when /users/ install your software, it
works with everything that's currently released.

I'm not sure I follow this. We'd still have CI runs that test against 
*release* Twisted; I'm just proposing that we would *also* test against 
development Twisted.


Your point about not stopping development when there's a problem is well 
noted though. A CI pipeline that runs on a timer might work fine. I'll 
discuss with the team.



but we have had enough difficulty keeping our CI configuration current 
based on what cloud provider is falling over this month ;-).


Yes, CI seems to be universally one of those things that is conceptually 
simple but somehow takes hours and hours to maintain.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

[Richard van der Hoff]

On 03/03/2021 08:07, Glyph wrote:


If dependencies could start testing against Twisted trunk in some 
capacity, we could get notified close to when unintentionally breaking 
changes occur, and dependencies can let us know well before the 
release happens, and we can either revert or they can fix things if 
the error is on their end.  If initially, say, crossbar and matrix 
would like to work with us to set up some kind of repeatable pattern 
we can suggest to others, that would be great.


I'll take this to the Synapse team to discuss further, but we could 
probably easily arrange for one of our CI runs to install Twisted trunk 
from git instead of pypi, which might be a good start.



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.2.0rc1 Release Candidate Announcement

[Richard van der Hoff]

On 28/02/2021 07:38, Craig Rodrigues wrote:


Thomas Grainger and Adi Roiban have worked together to bump up the 
minimum Python version to 3.5.3 for this
release ( https://github.com/twisted/twisted/pull/1524 ), and are not 
approving the PR that I have submitted
in PR 1525.  Even though you approved PR 1525, I cannot merge it 
because only PR reviews by people with

write access to the Twisted repo can be merged to the Twisted repo.

I am going to proceed with the release as-is with Python 3.5.3 as the 
minimum version.  I hope that is OK for your needs.


It looks to me like https://github.com/twisted/twisted/pull/1524 sets 
the minimum python version to 3.5.4, so I think we're good and #1525 
should just be closed as a duplicate.


Thanks for getting the release out!


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.2.0rc1 Release Candidate Announcement

[Richard van der Hoff]

On 21/02/2021 01:36, Craig Rodrigues wrote:


Awww, this is unfortunate.
It looks like our CI on GitHub Actions is testing *Python 3.5.10*.

Over the past year, we have added a lot of type annotations to the 
codebase, and it has
been a struggle to still keep things working with Python 3.5, but 
looks like between Python 3.5.2 and 3.5.10

there were changes to Python in this area.


As it happens, it also fails (for an import of typing.Deque) on Python 
3.5.3, as used by Debian oldstable.


Craig, what is your intention here? I think it's ok to drop support for 
these ancient versions of Python 3.5, but please could you make sure 
that python_requires gets set to 3.5.10 if that's what's been tested 
against?



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.2.0rc1 Release Candidate Announcement

[Richard van der Hoff]

On 21/02/2021 00:31, Richard van der Hoff wrote:


On 15/02/2021 02:53, Craig Rodrigues wrote:

It's time for another Twisted release!

There are two major announcements for this release:

- *Python 2.7 support has been dropped*.  Twisted 21.2.0 supports 
Python 3.5 and higher only

- *This will be the last Twisted release to support Python 3.5*.


Thanks Craig for getting this release ready.

It looks like it is incompatible with Python 3.5.2: exception as 
below. There's no real reason you should care about this, except that 
it is (still!) what Ubuntu 16.04LTS ships with, and Ubuntu claim to 
still be supporting 16.04 until April this year, so this might catch a 
few people by surprise, as it did me[1]. Anyway, I suggest bumping 
`python_requires` in `setup.cfg` 
(https://github.com/twisted/twisted/blob/trunk/setup.cfg#L27), to stop 
the new version being installed on systems where it won't work.


It also breaks compat with treq 20.4 and earlier. This may or may not be 
a thing you care about:


  File "/home/rav/work/synapse/synapse/http/client.py", line 33, in 


    import treq
  File 
"/home/rav/work/synapse/env3/lib/python3.8/site-packages/treq/__init__.py", 
line 5, in 

    from treq.api import head, get, post, put, patch, delete, request
  File 
"/home/rav/work/synapse/env3/lib/python3.8/site-packages/treq/api.py", 
line 5, in 

    from treq.client import HTTPClient
  File 
"/home/rav/work/synapse/env3/lib/python3.8/site-packages/treq/client.py", 
line 11, in 

    from twisted.python.compat import _PY3, unicode
ImportError: cannot import name '_PY3' from 'twisted.python.compat' 
(unknown location)



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.2.0rc1 Release Candidate Announcement

[Richard van der Hoff]

On 15/02/2021 02:53, Craig Rodrigues wrote:

It's time for another Twisted release!

There are two major announcements for this release:

- *Python 2.7 support has been dropped*.  Twisted 21.2.0 supports 
Python 3.5 and higher only

- *This will be the last Twisted release to support Python 3.5*.


Thanks Craig for getting this release ready.

It looks like it is incompatible with Python 3.5.2: exception as below. 
There's no real reason you should care about this, except that it is 
(still!) what Ubuntu 16.04LTS ships with, and Ubuntu claim to still be 
supporting 16.04 until April this year, so this might catch a few people 
by surprise, as it did me[1]. Anyway, I suggest bumping 
`python_requires` in `setup.cfg` 
(https://github.com/twisted/twisted/blob/trunk/setup.cfg#L27), to stop 
the new version being installed on systems where it won't work.


(env3)rav@faith:~$ python3
Python 3.5.2 (default, Oct  7 2020, 17:19:02)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twisted.python import log
Traceback (most recent call last):
  File "", line 1, in 
  File 
"/home/synapse/env3/lib/python3.5/site-packages/twisted/python/log.py", 
line 20, in 

    from twisted.python import reflect
  File 
"/home/synapse/env3/lib/python3.5/site-packages/twisted/python/reflect.py", 
line 377, in 
    def _safeFormat(formatter: Union[types.FunctionType, Type[str]], o: 
object) -> str:

  File "/usr/lib/python3.5/typing.py", line 552, in __getitem__
    dict(self.__dict__), parameters, _root=True)
  File "/usr/lib/python3.5/typing.py", line 512, in __new__
    for t2 in all_params - {t1} if not isinstance(t2, TypeVar)):
  File "/usr/lib/python3.5/typing.py", line 512, in 
    for t2 in all_params - {t1} if not isinstance(t2, TypeVar)):
  File "/usr/lib/python3.5/typing.py", line 1077, in __subclasscheck__
    if super().__subclasscheck__(cls):
  File "/usr/lib/python3.5/abc.py", line 225, in __subclasscheck__
    for scls in cls.__subclasses__():
TypeError: descriptor '__subclasses__' of 'type' object needs an argument


[1] Yes, yes, I know. I'm going to fix it real soon, I promise.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Richard van der Hoff]

On 08/06/2020 08:04, Glyph wrote:
> 

I'm going to start here by saying: I agree with almost all of what you 
wrote, but at the end of the day, I don't get to determine our 
customers' policies. You can try to explain to them why their policies 
are misguided, but particularly when you're working with a large 
organisation, change can be very slow. So you end up working around the 
policy, whether you agree with it or not. In practical terms, that means 
that for now at least we need to support Python 3.5.


As Erik said, we certainly have no right to demand that Twisted continue 
to support 3.5: indeed, if dropping support will deliver value to the 
project, then I'd encourage you to go for it; and as you've already 
said, the whole thing is probably moot anyway given the timescales we're 
talking about.


I believe in this case its a general desire to keep track of what 
packages are running and where they've come from. They basically 
trust that packages from official Debian repositories are probably 
safe from being tampered with, whereas random tarballs of code from 
the web are not safe (unless they're signed by someone they trust or 
whatever).


I think this sounds like a misunderstanding of Debian's vetting 
process?  It's not like there's a ton of additional auditing that goes 
into packaging something.  There's definitely an authentication 
process for both Twisted and Python, although this attestation could 
be somewhat stronger and less centralized, PyPI does quite a bit of 
heavy lifting there.


I think it's less that they think that Debian does extra vetting, and 
more that, especially if you're managing whole fleets of servers, then 
if everything runs the same version, it's easier to keep track of what 
you need to upgrade when there's a "security" bug. And yes, there are 
plenty of counterarguments to this, but that's the reasoning.



 1. Many non-"security" bugs are in fact security bugs that nobody has
noticed you can exploit.
 2. Many "low-severity" or "un-exploiable" security bugs are in fact
exploitable
 3. "supported" distros rarely take care to backport many patches for
their software, and when they do, they often make undetected
errors (like debian's infamous ssh bug) which are analyzed by far
fewer security analysts than the upstream source code.

These are probably all true, but taken to their logical extreme, the 
conclusion seems to be "you should always run the bleeding edge of all 
software, to make sure you've got all the latest bug fixes". I don't 
think you're really arguing for that, so the point is: we end up 
nominating "stable" versions, and trying to make an assessment as to 
which bugfixes are worth backporting. That latter part is a subjective 
decision, and the question is who you trust to make it. You may not 
trust Debian to make that decision on your behalf (with perfectly valid 
reasons), but plenty of others do.
So I feel like the folks making the decision to stick with these old 
"supported" distros are only getting half the story - sure, it won't 
break, but are they /actually/ getting the security fixes that they 
think they are?  Debian's staff are stretched pretty thin as it is.


A counterpoint here is that the Python in oldstable has had several 
years of bugfixes, and of course it was the primary Debian-supported 
version of Python for a good couple of years. Again, I know that 
peoples' assessment of Debian's ability or competence varies, but I 
don't think it's *unreasonable* to assume that by this point the worst 
problems with that version of Python have been shaken out (and that if a 
significant new problem arises, a fix will be made).


In terms of twisted dropping support of 3.5, I guess the question is 
to what extent do you want applications to be hassle free to deploy 
on the more "enterprise" style environment?


One other confusion I have about these environments is why they want 
very-old Python but don't /also/ want very-old Twisted.


Well, again, it comes down to fleet management, and responsibility for 
"security". From the customer's point of view, they want to provide a 
python interpreter which runs our application. So the Python interpreter 
is their responsibility (hence: use Debian oldstable everywhere), 
whereas the stuff running on it is ours. Plus, since there's only one 
thing using Twisted in their network, it's inherently easier to maintain 
a single version.


(I also fear that there is a misguided belief that security 
vulnerabilities are "worse" if they are in the Python interpreter, 
because that runs native code, whereas Twisted can't possibly do 
anything that bad because "interpreted language". I mention this only 
for completeness, and fully realise it's nonsense.)


But supporting old Pythons, old service_identity modules, old 
OpenSSL's, etc, has been seeming more and more to me like a disservice 
to the community, because it facilitates the adoption of slow, 
insecure, dangerous 

Re: [Twisted-Python] [RFC] Drop support for Python 3.5 sometime after May 2021?

[Richard van der Hoff]

On 16/05/2020 06:56, Glyph wrote:



On May 15, 2020, at 8:40 PM, Craig Rodrigues > wrote:


Maybe it would be OK to do one more release of Twisted and announce 
that as the last release supporting Python 3.5, before

dropping support?


Yeah; whenever we drop a Python version we should always support at 
least one more release, so that people have some notice before they 
lose access to the next set of security updates.


Any 3.5 users on this list who would want to postpone it longer than this?

Sadly we have an important customer whose servers run debian oldstable, 
which means we need to stay compatible with 3.5 until we can persuade 
them to upgrade, and it's taken a couple of years to get them off python 
2.7...


I'm not sure that should necessarily affect your plans, but I doubt 
we're alone in this situation.



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Help with twisted.conch.manhole

[Richard van der Hoff]

Or peek in matrix-synapse's code, which has a telnet-based one.


synapse's manhole is SSH-based :)

(dunno why, given it uses hardcoded auth, so all security depends on the 
manhole port being locked down, but still)


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] IReactorTime.seconds: epoch time or no?

[Richard van der Hoff]

On 26/03/2020 09:23, Barry Scott wrote:

Why does Twisted need to duplicate the built in python time features? > What 
was wrong with using time.time() for the access logs?


I feel like glyph's already answered this: use of `time` makes for poor 
testability. You might argue you don't care for access logs, since we 
aren't going to check the exact output, but (a) we probably *should* 
check the output matches the given timestamp, and (b) if a pattern's 
worth following, it's worth following everywhere.



Surely reactors must be use monotonic time to avoid breaking protocol
timing across wall-clock time being adjusted by ntp etc.


Again, I think glyph's answered this. There *should* be a method for 
getting a monotonic-time, and things that do scheduling should use it; 
but that method shouldn't be `seconds()`.


Thanks to Amber and glyph for answering my original question: I've 
opened https://twistedmatrix.com/trac/ticket/9787 to track the issue.


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] IReactorTime.seconds: epoch time or no?

[Richard van der Hoff]

HTTPFactory seems to think that `reactor.seconds` is reliably an epoch 
time (see 
https://github.com/twisted/twisted/blob/trunk/src/twisted/web/http.py#L3110, 
etc).


On the other hand. AsyncioSelectorReactor.seconds() returns a monotonic 
time:


python3 -c 'from twisted.internet import asyncioreactor; 
print(asyncioreactor.AsyncioSelectorReactor().seconds())'

41116.763594412

One of these is wrong... I think it's HTTPFactory making bad 
assumptions, but can anyone confirm the intention here?


R


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 20.3.0rc1 Release Candidate Announcement

[Richard van der Hoff]

Noted. And thanks to Amber for cutting the release!

On 20/03/2020 06:48, Glyph wrote:

Hi Richard!

Thanks for your testing!

One thing I always like to remind people is that if you want a 
particular ticket to get addressed quickly (in this case, the release) 
one thing you can do is to go to https://twisted.reviews/ and review a 
couple of project-member-submitted contributions; this drains the 
review queue and makes it more likely that project members with review 
privileges will review the thing you want reviewed, whenever they have 
spare cycles.


-glyph

On Mar 18, 2020, at 4:42 AM, Richard van der Hoff <mailto:rich...@matrix.org>> wrote:


Hi Amber and team!

Firstly, I can confirm that I've been using 20.3.0rc1 on a production 
server, and all seems fine.


So... I'm keen to see a final release, particularly in light of the 
request smuggling attacks. Any idea what sort of timeframe looks likely?


R

On 14/03/2020 07:03, Glyph wrote:
Thanks for doing another release, Amber!  Glad to see more important 
HTTP security issues get squashed; looking forward to seeing this on 
store shelves soon!

-g
On Mar 9, 2020, at 2:39 AM, Amber Brown (hawkowl) 
mailto:hawk...@atleastfornow.net>> wrote:


It's time for another Twisted release!

Twisted 20.3.0rc1 brings the following:

- curve25519-sha256 key exchange algorithm support in Conch.
- "openssh-key-v1" key format support in Conch.
- Security fixes to twisted.web, including preventing request 
smuggling attacks and rejecting malformed headers. CVE-2020-10108 
and CVE-2020-10109 were assigned for these issues, see the NEWS 
file for full details.

- `twist dns --secondary` now works on Python 3.
- The deprecation of twisted.news.
- ...and various other fixes, with 28 tickets closed in total.

You can get the tarball and the NEWS file at 
https://twistedmatrix.com/Releases/rc/20.3.0rc1/ , or you can try 
it out from PyPI:


python -m pip install Twisted==20.3.0rc1

Please test it, and let me know how your applications fare, good or 
bad! If nothing comes up, 20.3 will release very soon.


Twisted regards,

Amber Brown (hawkowl)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com 
<mailto:Twisted-Python@twistedmatrix.com>

https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com 
<mailto:Twisted-Python@twistedmatrix.com>

https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com 
<mailto:Twisted-Python@twistedmatrix.com>

https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 20.3.0rc1 Release Candidate Announcement

[Richard van der Hoff]

Hi Amber and team!

Firstly, I can confirm that I've been using 20.3.0rc1 on a production 
server, and all seems fine.


So... I'm keen to see a final release, particularly in light of the 
request smuggling attacks. Any idea what sort of timeframe looks likely?


R

On 14/03/2020 07:03, Glyph wrote:

Thanks for doing another release, Amber!  Glad to see more important HTTP 
security issues get squashed; looking forward to seeing this on store shelves 
soon!

-g


On Mar 9, 2020, at 2:39 AM, Amber Brown (hawkowl)  
wrote:

It's time for another Twisted release!

Twisted 20.3.0rc1 brings the following:

- curve25519-sha256 key exchange algorithm support in Conch.
- "openssh-key-v1" key format support in Conch.
- Security fixes to twisted.web, including preventing request smuggling attacks 
and rejecting malformed headers. CVE-2020-10108 and CVE-2020-10109 were 
assigned for these issues, see the NEWS file for full details.
- `twist dns --secondary` now works on Python 3.
- The deprecation of twisted.news.
- ...and various other fixes, with 28 tickets closed in total.

You can get the tarball and the NEWS file at 
https://twistedmatrix.com/Releases/rc/20.3.0rc1/ , or you can try it out from 
PyPI:

 python -m pip install Twisted==20.3.0rc1

Please test it, and let me know how your applications fare, good or bad! If 
nothing comes up, 20.3 will release very soon.

Twisted regards,

Amber Brown (hawkowl)

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Waiting for a contended resource

[Richard van der Hoff]

Thank you for all for all the answers so far, particularly to Ilya and 
Jean-Paul who provided some very helpful code samples.


It's interesting to realise that, by avoiding locking, we can end up 
with a much more efficient implementation. I'll have to figure out how 
widely we can apply this technique - and how often it's going to be 
worth rewriting things to allow that. Thanks for some useful pointers!


Richard


On 12/03/18 20:00, Jean-Paul Calderone wrote:
On Mon, Mar 12, 2018 at 3:52 PM, Ilya Skriblovsky 
> wrote:


Hi, Richard,

I've used class like this to cache the result of Expensive
Calculation:

class DeferredCache:
pending = None
result = None
failure = None

def __init__(self, expensive_func):
  self.expensive_func = expensive_func

def __call__(self):
  if self.pending is None:
      def on_ready(result):
          self.result = result
      def on_fail(failure):
          self.failure = failure

      self.pending =
defer.maybeDeferred(self.expensive_func).addCallbacks(on_ready,
on_fail)

  return self.pending.addCallback(self._return_result)


This seems like basically a correct answer to me. However, I suggest 
one small change.


You probably want to create and return a new Deferred for each 
result.  If you don't, then your internal `pending` Deferred is now 
reachable by application code.


As written, an application might (very, very reasonably):

    d = getResource()
    d.addCallback(long_async_operation)

Now `pending` has `long_async_operation` as a callback on its chain.  
This will prevent anyone else from getting a result until 
`long_async_operation` is done.


You can fix this by:

    result = Deferred()
self.pending.addCallback(self._return_result).chainDeferred(result)
    return result

Now the application can only reach `result`.  Nothing they do to 
`result` will make much difference to `pending` because 
`chainDeferred` only puts `callback` (and `errback`) onto `pending`'s 
callback chain.  `callback` and `errback` don't wait on anything.


You have to be a little careful with `chainDeferred` because it 
doesn't have the recursion-avoidance logic that implicit chaining 
has.  However, that doesn't matter in this particular case because the 
chain depth is fixed at two (`pending` and `result`).  The problems 
only arise if you extend the chain out in this direction without bound.


Jean-Paul

def _return_result(self, _):
  return self.failure or self.result

Using it you can get rid of DeferredLocks:

    deferred_cache = DeferredCache(do_expensive_calculation)

    def getResource():
        return deferred_cache()

It will start `expensive_func` on the first call. The second and
consequtive calls will return deferreds that resolves with the
result when expensive_func is done. If you call it when result is
already here, it will return alread-fired deferred.

Of course, it will require some more work if you need to pass
arguments to `expensive_func` and memoize results per arguments
values.

-- ilya



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Waiting for a contended resource

[Richard van der Hoff]

Hi folks,

I thought I'd poll the list on the best way to approach a problem in 
Twisted.


The background is that we have a number of resources which can be 
requested by a REST client, and which are calculated on demand. The 
calculation is moderately expensive (can take multiple seconds), so the 
results of the calculation are cached so multiple lookups of the same 
resource are more efficient.


The problem comes in trying to handle multiple clients requesting the 
same resource at once. Obviously if 200 clients all request the same 
resource at the same time, we don't want to fire off 200 calculation 
requests.


The approach we adopted was, effectively, to maintain a lock for each 
resource:



lock = defer.DeferredLock()
cached_result = None

@defer.inlineCallbacks
def getResource():
 yield lock.acquire()
 try:
 if cached_result is None:
 cached_result = yield do_expensive_calculation()
 defer.returnValue(cached_result)
 finally:
 lock.release()


(Of course one can optimise the above to avoid getting the lock if we 
already have the cached result - I've omitted that for simplicity.)


That's all very well, but it falls down when we get more than about 200 
requests for the same resource: once the calculation completes, we can 
suddenly serve all the requests, and the Deferreds returned by 
DeferredLock end up chaining together in a way that overflows the stack.


I reported this as http://twistedmatrix.com/trac/ticket/9304 and, at the 
time, worked around it by adding a call to reactor.callLater(0) into our 
implementation. However, Jean-Paul's comments on that bug implied that 
we were approaching the problem in completely the wrong way, and instead 
we should be avoiding queuing up work like this in the first place.


It's worth reiterating that the requests arrive from REST clients which 
we have no direct control over. We *could* keep track of the number of 
waiting clients, and make the API respond with a 5xx error or similar if 
that number gets too high, with the expectation that the client retries 
- but one concern would be that the load from the additional HTTP 
traffic would outweigh any efficiency gained by not stacking up Deferreds.


So, I'd welcome any advice on better ways to approach the problem.

Richard

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python