Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

2021-03-02 Thread Tobias Oberstein 

Hi Jean-Paul,

It seems like this change does not actually violate Twisted's 
compatibility policy.  This is not to say introducing such regressions 
is desirable, but when constructing such work-arounds it might be worth 
paying closer-than-average attention to what the policy actually is.


The policy is currently documented at 
https://twistedmatrix.com/documents/current/core/development/policy/compatibility-policy.html#new-attributes 


Thanks! I wasn't aware (well, I was, but I never read all the docs;)

That indeed quite precisely hints at sth we did in our workaround: 
relying on the output of dir()


IOW, we should have done

except (AttributeError, NotImplementedError):

in the first place, because that catches both.

on an different angle, I wonder about the relation between both 
exceptions independent of Twisted.


eg "attribute not defined" certainly is an "attribute error"? and in any 
case, a missing attribute means it cannot be implemented?


so it cannot be implemented without making attribute access succeed, right?

but even if that would all be defined, it could not be expressed in 
python. or even statically checked. because that would require some 
machine readable language / decorators / whatever to exhaustively state 
what exceptions can be raised in the first place.


and then, an app developer (crossbar in this case) can make sure and 
test that it does actually have code to treat _all_ exceptions that can 
ever be raised.


anyways, just some ramblings .. python won't give me that, yes.



(and that anchor takes you to the section that I think allows the change).

Policy aside, this change doesn't seem like much of an improvement to 
me.  If I were to guess, I would guess the change was made to satisfy 
some check Mypy is now being asked to make about Twisted.  If that's the 
case, it seems unfortunate that real-world software is suffering so that 
a synthetic goal can be achieved.  I do recognize there is a perception 
that practical value can come from attending to the errors Mypy 
reports.  It would probably benefit everyone if more care were taken to 
consider the real-world consequences of changes that are made to satisfy 
the non-real-world goalposts set by tools like Mypy.


fwiw, I highly welcome the use of mypy! even with some fallout;) we use 
it too (in crossbarfx).


but ^ might indeed be what happened practically. np, I can deal with it. 
Twisted has been a very reliable, and central piece of our stack - and 
is here to stay! happy user=) and, I guess, all workarounds (mine) 
eventually come back and bite. that's life;)


Cheers,
/Tobias




Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com

https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python



--

Tobias Oberstein - phone +49 176 2375 2055 - tobias.oberst...@crossbario.com
Crossbar.io GmbH - Waldstrasse 18 - 91054 Erlangen
HRB 15870 - Amtsgericht Fuerth - Geschäftsfuehrer/CEO - Tobias Oberstein

https://crossbar.io
https://crossbario.com

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

2021-03-02 Thread Tobias Oberstein 

Hi Craig,

thanks for your answer, tips and questions! much appreciated

Am 03.03.21 um 02:58 schrieb Craig Rodrigues:


On Tue, Mar 2, 2021 at 12:12 PM Tobias Oberstein 
mailto:tobias.oberst...@gmail.com>> wrote:



this fix broke our workaround for the bug now fixed;)


I had to read that sentence a few times before I fully understood it, 
and my mind did a backflip. :)


I hope the grammar is fine (not a native speaker), but it seems to 
express what I wanted to say .. and that has some recursive, mind 
stretching aspect to it, yeah ;)


there are a couple of layers .. eg the specific problem is for sth where 
Twisted itself defines an interface (zope-interface), but then fails to 
implement it.


However, declaring a zope.interface without CI testing it is just 
"documentation". best effort.


But that isn't the last layer: without declaring exactly all interfaces 
formally in the first place, there will always be a gray area, where 
users accidently rely on behavior-as-is, even when that behavior is a 
bug. Then how to reconceile fixing the bug vs stability?


IOW: how can one declare "stability" if the object of that (the API) 
isn't exhaustively and formally defined?


it took me 10min to figure out and fix the bug, but I am still thinking 
about the "why" and the "should" .. and about Rust ;)




Is there a quick and dirty way to run Crossbar's CI against a 
pre-release version of Twisted?


Actually, there was a time when we ran Twisted trunk as part of the CI.

Due to complexity managing our deps (~130), and CI run-time, we stopped 
that .. well, not so good. I think this would have warned us early on.


https://github.com/crossbario/crossbar/blob/master/tox.ini



For Buildbot (a project that I work with a lot), I just tweaked their CI 
like this to run against a branch of Twisted: 
https://github.com/buildbot/buildbot/pull/5739
When those tests failed, that helped give some info, and helped me 
isolate one thing to fix in Twisted before doing the release.
Povilas Kanapickas and Pierre Tardy from the Buildbot project were very 
helpful with this.


That sounds pretty good! congrats.



I did not run the test suite for Matrix Synapse, but Richard van der 
Hoff was very helpful in terms of giving feedback.


Twisted's internal test suite is quite good and comprehensive, but it 
doesn't catch everything.

I find it useful to do quick tests against existing projects where possible.

Twisted has been around for around 20 years, and is used *all over the 
place*!

--
Craig


--

Tobias Oberstein - phone +49 176 2375 2055 - tobias.oberst...@crossbario.com
Crossbar.io GmbH - Waldstrasse 18 - 91054 Erlangen
HRB 15870 - Amtsgericht Fuerth - Geschäftsfuehrer/CEO - Tobias Oberstein

https://crossbar.io
https://crossbario.com

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

2021-03-02 Thread Jean-Paul Calderone 
On Tue, Mar 2, 2021 at 3:09 PM Tobias Oberstein 
wrote:

> Hi all,
>
> rgd Twisted v21.2.0, thanks a lot to all involved in this release! the
> release, mypy, more py3, .. +1! really appreciated.
>
> now, fwiw, just wanted to let you know (sorry, too late) about a
> regression with
>
> https://github.com/crossbario/crossbar
>
> that actually originates from
>
> https://github.com/crossbario/autobahn-python/issues/1470
>
> =>
>
> with Twisted v20.3.0 calling
> twisted.internet.protocol.ProcessProtocol.getPeer raises an AttributeError
>
> with Twisted v21.2.0 calling the same method raises an
> NotImplementedError
>
> =>
>
> this fix broke our workaround for the bug now fixed;)
>
> https://github.com/crossbario/autobahn-python/pull/1469/files
>
> and released in:
>
> https://pypi.org/project/autobahn/21.3.1/
> https://pypi.org/project/crossbar/21.3.1/
>
> Cheers,
> /Tobias
>
>
It seems like this change does not actually violate Twisted's compatibility
policy.  This is not to say introducing such regressions is desirable, but
when constructing such work-arounds it might be worth paying
closer-than-average attention to what the policy actually is.

The policy is currently documented at
https://twistedmatrix.com/documents/current/core/development/policy/compatibility-policy.html#new-attributes
(and that anchor takes you to the section that I think allows the change).

Policy aside, this change doesn't seem like much of an improvement to me.
If I were to guess, I would guess the change was made to satisfy some check
Mypy is now being asked to make about Twisted.  If that's the case, it
seems unfortunate that real-world software is suffering so that a synthetic
goal can be achieved.  I do recognize there is a perception that practical
value can come from attending to the errors Mypy reports.  It would
probably benefit everyone if more care were taken to consider the
real-world consequences of changes that are made to satisfy the
non-real-world goalposts set by tools like Mypy.

Jean-Paul



> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

2021-03-02 Thread Craig Rodrigues 
On Tue, Mar 2, 2021 at 12:12 PM Tobias Oberstein 
wrote:

>
> this fix broke our workaround for the bug now fixed;)
>
>
I had to read that sentence a few times before I fully understood it, and
my mind did a backflip. :)

Is there a quick and dirty way to run Crossbar's CI against a pre-release
version of Twisted?

For Buildbot (a project that I work with a lot), I just tweaked their CI
like this to run against a branch of Twisted:
https://github.com/buildbot/buildbot/pull/5739
When those tests failed, that helped give some info, and helped me isolate
one thing to fix in Twisted before doing the release.
Povilas Kanapickas and Pierre Tardy from the Buildbot project were very
helpful with this.

I did not run the test suite for Matrix Synapse, but Richard van der Hoff
was very helpful in terms of giving feedback.

Twisted's internal test suite is quite good and comprehensive, but it
doesn't catch everything.
I find it useful to do quick tests against existing projects where possible.

Twisted has been around for around 20 years, and is used *all over the
place*!
--
Craig
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.2.0 Release Announcement

2021-03-02 Thread Werner Thie 
On 3/1/21 18:35, Craig Rodrigues wrote:

> Glyph,
>
> The 21.2.0 release branch is ready for review so that it can be merged
> back to trunk:
>
> https://github.com/twisted/twisted/pull/1514
> 
>
> Thanks.
>
> --
> Craig
>
Mahalo nui loa for the latest and greatest! I can report that all my
software is now running on Python3 with nevow/athena too, which might
give me some time to dig into the work on chunking the changes in nevow
and feeding them back into the standard twisted development process.

Werner

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

2021-03-02 Thread Tobias Oberstein 

Hi all,

rgd Twisted v21.2.0, thanks a lot to all involved in this release! the 
release, mypy, more py3, .. +1! really appreciated.


now, fwiw, just wanted to let you know (sorry, too late) about a 
regression with


https://github.com/crossbario/crossbar

that actually originates from

https://github.com/crossbario/autobahn-python/issues/1470

=>

with Twisted v20.3.0 calling 
twisted.internet.protocol.ProcessProtocol.getPeer raises an AttributeError


with Twisted v21.2.0 calling the same method raises an
NotImplementedError

=>

this fix broke our workaround for the bug now fixed;)

https://github.com/crossbario/autobahn-python/pull/1469/files

and released in:

https://pypi.org/project/autobahn/21.3.1/
https://pypi.org/project/crossbar/21.3.1/

Cheers,
/Tobias

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python