[Twisted-Python] email list downtime

[Glyph]

Hello all,

This mailing list will be shut down in a few minutes, and should be back online 
in about 12 hours at the latest.  When it comes back, the preferred address 
will be twis...@python.org .

Thanks,

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Moving the Twisted-Python and Twisted-Web mailing lists too Python mailman3

[Glyph]

> On Aug 2, 2021, at 1:10 AM, Thomas Grainger  wrote:
> 
> I actually think it's better to move twisted-web first and see what it
> looks, then mess about there with redirects, then do the main mailing
> list. I think the redirects would be fairly easy as the URL structure
> is the same

Sounds like a good plan.

Speaking of using testing areas rather than the most heavily used production 
stuff:

One thing that perhaps bears mentioning here is that we do have a second domain 
name that I bought at some expense for the project - in part to celebrate its 
20th anniversary at … PyCon 2020 (sob): twisted.org .  If 
this minor emergency is sparking anyone's interest in slowly building some new 
infrastructure (like a new website) on clean, modern infra, without the 
pressure to keep something working continuously, we could host it all there and 
cut over twistedmatrix.com  to be a redirect at the 
appropriate time.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] bad news about twistedmatrix.com's hosting

[Glyph]

> On Aug 1, 2021, at 2:49 AM, Adi Roiban  <mailto:adiroi...@gmail.com>> wrote:
> 
> Hi Glyph
> 
> Thanks for sending the message.

As a wise computer once sung, "We do what we must, because we can".

> This is bad, but I am confident that we can do the migration in 60 days, as 
> long as we don't expect to have the exact same Twisted infrastructure.

It's certainly annoying (and I'm very sad to see tummy go, they were great) but 
I think we may be happy with the opportunity to reduce the amount of stuff we 
operate.

> We have a monthly credit of 150 USD from MS to spend on Azure services.

This is great news!  Presumably I … already have access to this, somehow?  Is 
it in the 1password team vault?  How do I see it?  Do we have a commitment from 
MS to actually maintain this credit?

> A quick hack would be to create a new VM in Azure with Ubuntu 18.04 and 
> migrate the existing server to it.

If we need to provision a new server anyway, could we provision it on the most 
recent ubuntu?  Presumably most of the stuff we require will still be on there? 
 If it's a hassle let's fall back to 18.04 but let's not set ourselves up to be 
outdated as long as we're doing this work.

> Do we have other servers hosted with tummy.com <http://tummy.com/>, other 
> than dornkirk.twistedmatrix.com <http://dornkirk.twistedmatrix.com/> ?

Nope.  We used to have a VPS called Wolfwood, but in an act of nominative 
determinism, it had to die for the greater good.  (It was our source-control 
server, back when that was a thing you had to run yourself.  We shut it down 
when we migrated to github.)

> I will not have too much time to do the migration, but I can ask a sysadmin 
> from my work to help with that and my company will cover that costs.

Thanks Adi, this is very generous.  Let me know if I'm needed to give this 
sysadmin any requisite privileges.

> I think that for now, we should see who is interested with helping with the 
> migration and after 1 week start to work with that team.

Looks like a team is coming together :).

> Maybe we will need to schedule an online meeting, but I hope we can do async 
> work via GitHub Issues (braid) and IRC/Gitter.

Yeah, async seems to be working fine.

> I saw that you have already considered using containers for Twisted infra 
> https://github.com/twisted-infra/braid/issues/265 
> <https://github.com/twisted-infra/braid/issues/265>
Yes. I'd love to do this, and if such a migration reduces effort, then let's do 
it by all means - I just don't want to get caught up in trying to do this and 
ending up with things broken when the clock runs out.

> In parallel with migrating  dornkirk.twistedmatrix.com 
> <http://dornkirk.twistedmatrix.com/>, we can look at outsourcing some of our 
> services.
> For example
> 
> * Move mailing list to the main Python mailing list server

Looks like this is full steam ahead!

> * Move DNS to a cloud provider (Azure DNS)

I've already pulled the trigger to migrate our DNS to Gandi, since they're our 
registrar and I already know I can provision certificates against their LiveDNS 
API :).  We do still have to get a few other domains off of this infrastructure 
though, I'll do the ones we control (Divunal) and contact folks at the others.

Longer-term, I would like to host some DNS, since I want to make sure that 
twisted.names does not bit-rot.  However, perhaps a prerequisite to setting 
this up again will be (A) containerizing it and (B) providing something like 
Sentry for us to track exceptions, so that the dog-fooding here is interesting 
and not just a tedious obligation that produces gigabytes full of 
tracback-laden log files that nobody ever looks at or fixes.

> * Move Trac Wiki to GitHub Wiki

The trac wiki's main purpose is not really to serve as a wiki-style editable 
document repository, but rather to host the website.  I think migrating it to 
something like Pelican, or some other static-site management thing might make 
more sense.

> * Move Trac Tickets to GitHub Issues

Sure, sure, as soon as we figure out a way to preserve all the links :).

> * Move static file hosting to GitHub pages

I wonder if it would make sense for www.twistedmatrix.com 
<http://www.twistedmatrix.com/> itself to be hosted by 

> * Move highscore to a container and update it to pull info from GitHub hooks

+1.  This thing might be interesting 
<https://github.com/leereilly/github-high-scores 
<https://github.com/leereilly/github-high-scores>> in that it might be a better 
starting point than our own code (which is not even open source because, among 
other reasons, it is such trash) this version gets the point of the scoring 
wrong, which is to weight 

> * IRC bot to some sort of integration to Gitter and then IRC bridge

Given that this is already broken right now, I'd say "don't bo

Re: [Twisted-Python] Moving the Twisted-Python and Twisted-Web mailing lists too Python mailman3

[Glyph]

Great news.  I'd be happy to move the mailing lists over to let someone else 
maintain their infrastructure.  Particularly if we could get off of Mailman 2, 
which has been no fun to maintain.

Many personal email addresses of core team members (including, um, this one) 
are forwarded by our mail hosting provider, Mailgun, and so moving the MX 
records directly is a non-starter, but we could easily point 
lists.twistedmatrix.com  at python.org 
, or we could just use @python.org addresses and be quite 
happy.  twis...@python.org  would be very cool.  We 
could also maintain a forwarding alias for compatibility for a little while, 
but I'd kinda rather deprecate posting to twisted-python@twistedmatrix.com 
 for a couple of other reasons anyway.

Do we have a way to get our pipermail archive URLs (which are linked in 
tickets, in source code, etc) to redirect to the fancy new hyperkitty 
interface, or should those just be static HTML?  (I'm kinda fine with either, 
honestly, although the former would be nice if we can manage it easily.)

I am happy to coordinate off-list about a secure way to exchange these data 
files (or anyone with administrator access to Dornkirk can just do it 
directly).  Please send the announcement of the new address here first so we 
know when to cut over posting, and so that folks aren't surprised to see a new 
list-id.

(At this point I think I'm happy to just nuke twisted-web as part of this 
migration; we can keep the archive up as static HTML for posterity, but it's 
mostly unused at this point, and the distinction doesn't make sense.)

-g

> On Aug 2, 2021, at 12:56 AM, Thomas Grainger  wrote:
> 
> Apologies I meant to forward this mail instead:
> 
> On 8/1/21 1:16 PM, Thomas Grainger wrote:
>> Hello,
>> 
>> The twisted ( https://pypi.org/project/Twisted/ ) web hosting provider is 
>> winding
>> down and we're looking for a new home.
>> 
>> Would it be possible to move our mailing lists and archives onto the
>> https://mail.python.org/mailman3/lists/ infrastructure?
>> 
>> These are the current lists
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo
> 
> 
> If you can get me copies of Mailman's lists/twisted-python/config.pck
> and lists/twisted-web/config.pck configuration files and
> archives/private/twisted-python.mbox/twisted-python.mbox and
> archives/private/twisted-web.mbox/twisted-web.mbox archive mailboxes, I
> can import these on mail.python.org.
> 
> The twisted-web list seems pretty inactive currently, but twisted-python
> seems fairly active. Ideally we can arrange a time for you to shut down
> the lists and get me the files and get the lists up on mail.python.org.
> Otherwise, if the lists are still operational while I'm getting the
> files and doing the imports, messages and perhaps other changes will be
> lost, although it is possible to import additional messages to the
> archives after the fact.
> 
> I will be available to do this during the next two weeks, but I will not
> be available for the rest of August and the first two weeks of September.
> 
> Thomas Grainger
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] new success story: Battlehouse Games

[Glyph]

I ran across a PyCon APAC talk about this, and the speaker generously offered 
to write a blurb for our site, which I've posted here: 
https://twistedmatrix.com/trac/wiki/SuccessStories#BattlehouseGames 


> Battlehouse creates and operates massively-multiplayer strategy games for 
> browser and desktop platforms. Our most popular titles are nearly 10 years 
> old and have served over 4 million users.
> 
> Twisted plays an important role in our Python-based server stack. We use 
> Twisted to handle many protocols including HTTP, WebSockets, and SQL. The 
> asynchronous coroutine model helps us deliver low-latency gameplay without 
> adding too much complexity to the codebase. Across 10 years of continuous 
> evolution, Twisted has proven to be the most stable, well-documented, and 
> flexible low-level networking library for Python.
> 
> -- Dan Maas, CTO, Battlehouse Games

Do you have your own success story?  Please contact succ...@twistedmatrix.com 
 - it's always nice to hear from users!

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] bad news about twistedmatrix.com's hosting

[Glyph]

Hello Twistors,

Our venerable hosting provider, tummy.com , will be 
terminating their business operations on September 30, 2021.

>>> datetime.date(2021, 9, 30) - datetime.date.today()
datetime.timedelta(days=60)

In addition to being a bit personally sad - Tummy has been an avid supporter of 
the Twisted community since the very beginning - this means we have just about 
60 days to fully migrate our website, mailing list, and DNS's hosting from 
Dornkirk, the machine where it's been running continuously since 2013, to ... 
something else.

There are 2 problems here:

We need a sponsor to help us find a new hosting or cloud provider where things 
can be hosted.  We do pay tummy currently, and we can probably ask the SFC to 
pay a different hosting provider a similar price for equivalent services, but 
ideally we'd find someone willing to donate something so as not to put a drain 
on those resources.

I will need significant help with the operational aspects of the migration.  I 
typically do a bunch of behind-the-scenes system administration for 
twistedmatrix.com  to keep the whole thing clunking 
along, but after my brain has been fried by a year and a half's worth of 
pandemic stress, I simply don't have the time, energy, or mental capacity to do 
this.  I know I've asked for help before and rarely received any, but if I have 
to do this entirely by myself, the most likely outcome is that I'll migrate DNS 
to some free cloud thing so at least my email address doesn't break (I migrated 
SMTP, and thereby everything associated with personal email addresses to 
Mailgun some years ago, so Tummy is not in that loop for that), and the website 
(and this mailing list) will simply stop working sometime between now and 
October.  So if you'd like to keep www.twistedmatrix.com 
, speed.twistedmatrix.com 
, our IRC bot (which is offline anyway right 
now due to Freenode shutting down), https://twistedmatrix.com/highscores/ 
, this mailing list, or our dogfooding 
instance for Twisted Names DNS, please volunteer so we can start to coordinate.

If you've been wanting to just use Github Issues and don't care about 
preserving any of the data in Trac, congratulations, you're about to win that 
argument by default ;-).

In the worst case scenario, I will download a backup image before things get 
turned off in case someone wants to deal with this 

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.7.0 Final Release Announcement

[Glyph]

Thanks again, Adi!  I've posted this release to https://labs.twistedmatrix.com/ 
 and https://twitter.com/twistedmatrix 


-g


> On Jul 28, 2021, at 2:07 AM, Adi Roiban  wrote:
> 
> On behalf of the Twisted contributors I announce the final release of Twisted 
> 21.7.0
> 
> This is mostly a bugfix release.
> 
> Python 3.5 is no longer a supported platform.
> The minimum supported platform is Python 3.6.7.
> 
> The notable features are:
> 
> - Python 3.10 beta is now a supported platform and should be ready for the 
> final 3.10 release.
> - twisted.web.template.renderElement() now accepts any IRequest implementer 
> instead of only twisted.web.server.Request. Add type hints to 
> twisted.web.template. (#10184)
> -  Type hinting was added to twisted.internet.defer, making this the first 
> release  of Twisted where you might reasonably be able to use mypy without 
> your own custom stub files. (#10017)
> 
> The full release notes are available at
> 
> https://github.com/twisted/twisted/releases/tag/twisted-21.7.0 
> 
> 
> Documentation is available at
> 
> https://docs.twistedmatrix.com/en/stable/ 
> 
> 
> Wheels for the release candidate are available on PyPI
> 
> https://pypi.org/project/Twisted/21.7.0/ 
> 
> 
> python -m pip install Twisted==21.7.0
> 
> Many thanks to everyone who had a part in Twisted - the supporters of the 
> Twisted Software Foundation, the developers, and all the people testing and 
> building great things with Twisted!
> 
> Enjoy the release!
> 
> -- 
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reviews

[Glyph]

Update: I went through the queue and gardened it substantially. There were a 
bunch of tickets where folks had forgotten to remove the review keyword upon 
closure, some with malformatted commit messages so they weren't closed (this is 
very hard to enforce mechanically, but please, use the template in the PR 
boilerplate), and some with merge conflicts that needed addressing before 
review.

I also pushed the "update branch" button on a whole lot of PRs so hopefully 
some CI detritus that made tests fail for spurious reasons will be cleared out, 
and everything will be reasonably up to date.

We are now down to 17 matches, but this does give me a commanding lead on this 
month's leaderboard:



I look forward to being dethroned.

-g

> On Jul 19, 2021, at 9:38 PM, Glyph  wrote:
> 
> Hello Twisted friends,
> 
> This is just a friendly reminder that https://twisted.reviews 
> <https://twisted.reviews/> is full to bursting - 34 open reviews right now, 
> some dating back as far as May of last year.  If you've got a few free 
> minutes, can you find a ticket on that list and review the associated PR?  (I 
> recently fixed the report so it's actually in the order you should review 
> them...)  A short review queue is a happy review queue.
> 
> Remember also that this isn't for nothing, if you do code reviews on Twisted 
> you get internet points and bragging rights, at 
> https://twistedmatrix.com/highscores/ <https://twistedmatrix.com/highscores/>.
> 
> This isn't just for project members; if you did not already know, the rule is 
> that if you're a project member you can review anything, but if you're an 
> external contributor, you can review submissions by project members (and it's 
> up to them to decide if your review is adequate before acting on it).  
> Project members (should) have a "*" next to their name in the "submitted by" 
> list.  (If they don't I think someone needs to go manually update a list of 
> strings, feel free to report issues if that's wrong.)
> 
> Happy hacking,
> 
> -glyph
> 

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reviews

[Glyph]

> On Jul 20, 2021, at 5:21 AM, Adi Roiban  wrote:
> 
> Hi
> 
> Thanks Barry for reporting the issue.
> 
> It should be back online.
> 
> The twisted-web process was down.
> I have started it.

If anyone has the time to deal with it, we *do* have a free subscription 
donated by the friendly folks over at Sentry; if we set up the integration in 
all of our services it might lead to the discovery of some interesting bugs.

Let me know if you need credentials or an invite or something.

-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Error using "python -m twisted.conch.stdio" command in command prompt

[Glyph]

> On Jul 18, 2021, at 11:39 PM, Melanie Molifie  wrote:
> 
> Hi guys
> 
> I'm new to twisted and would appreciate some beginner guidance.
> 
Hi Melanie!  Welcome to the community. I hope we can get you set up better.

> I'm trying to use the "python -m twisted.conch.stdio" command in command 
> prompt but it does not work and gives the following error:
> 
> C:\WINDOWS\system32>python -m twisted.conch 
> C:\Users\Melanie\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\python.exe:
>  No module named twisted.conch.main; 'twisted.conch' is a package and cannot 
> be directly executed
> 
In this example you're doing `python -m twisted.conch`, not `python -m 
twisted.conch.stdio`, which will not work anywhere.
Unfortunately, the Conch standard-IO library (and therefore the conch terminal 
client, and twisted.conch.stdio) doesn't work on Windows and never has: 
https://twistedmatrix.com/trac/ticket/2157 
<https://twistedmatrix.com/trac/ticket/2157>.  However, you may be able to get 
it up and running with the Windows Subsystem for Linux, which I believe will 
present a Linux-like interface since as far as the application knows it's 
running on Linux: https://docs.microsoft.com/en-us/windows/wsl/about 
<https://docs.microsoft.com/en-us/windows/wsl/about>
> I'm hoping that this will solve the rest of the error messages I'm having. I 
> want to eventually be ably to run :
> 
> >>> from twisted.names import client
> >>> client.getHostByName('www.example.com <http://www.example.com/>')
> 
> >>> _
> 
> 
> >>> client.lookupMailExchange('twistedmatrix.com <http://twistedmatrix.com/>')
> 
> >>> _
>  <http://twistedmatrix.com/> type=MX class=IN ttl=1s auth=False>], [], [])>
Is the goal here specifically to do this with Conch, or would doing it with 
Jupyter Notebook (using something like 
https://twitter.com/glyph/status/1417360623818575876 
<https://twitter.com/glyph/status/1417360623818575876> which I found while 
researching a response to this email!) be sufficient?

Or, for that matter, could you just write a little program?  A sample Python 
script that runs the reactor is probably easier to set up than a 
reactor-enabled REPL, particularly on Windows.  (There are various options for 
a reactor-enabled REPL, it's definitely doable, but it might be a bit rough as 
a first step; as you're discovering.)

> The aim is to get the deferred responses as shown in previous code, but all I 
> get is:
> 
> >>> from twisted.names import client
> >>> client.getHostByName('www.example.com <http://www.example.com/>')
> 
> >>> _
> 
> >>> client.lookupMailExchange('twistedmatrix.com <http://twistedmatrix.com/>')
> 
> >>> _
> 
The reason you're getting this with a normal Python interactive interpreter is 
that the event loop which actually performs the I/O (does the DNS queries, in 
this case) isn't running at all, so the Deferreds are created, but never 
resolved.

> Please help. I'm trying to send this to the twisted mailing group. If I'm 
> wrong please nudge me in the right direction.
> 
This is as good a place as any :).

> Thanks in advance.
> 
> M
> 
> On Mon, Jul 19, 2021 at 12:43 AM Richard van der Hoff  <mailto:rich...@matrix.org>> wrote:
> On 16/07/2021 20:27, Glyph wrote:
> >
> >> One particular problem I came across was the type annotation on 
> >> inlineCallbacks. I've filed 
> >> https://twistedmatrix.com/trac/ticket/10231 
> >> <https://twistedmatrix.com/trac/ticket/10231> about it - would 
> >> appreciate thoughts.
> >>
> > This definitely looks wrong; there should be a TypeVar in there.  Adi, 
> > I'd go so far as to say that this should be a release blocker, 
> > although the change should be fairly minimal.
> >
> > Richard, could you please make a proper PR for this to get CI kicked 
> > off and make sure the new annotation doesn't cause any failures?
> Done, in the form of https://github.com/twisted/twisted/pull/1632 
> <https://github.com/twisted/twisted/pull/1632>.
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com>
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
> <https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] reviews

[Glyph]

Hello Twisted friends,

This is just a friendly reminder that https://twisted.reviews 
<https://twisted.reviews/> is full to bursting - 34 open reviews right now, 
some dating back as far as May of last year.  If you've got a few free minutes, 
can you find a ticket on that list and review the associated PR?  (I recently 
fixed the report so it's actually in the order you should review them...)  A 
short review queue is a happy review queue.

Remember also that this isn't for nothing, if you do code reviews on Twisted 
you get internet points and bragging rights, at 
https://twistedmatrix.com/highscores/ <https://twistedmatrix.com/highscores/>.

This isn't just for project members; if you did not already know, the rule is 
that if you're a project member you can review anything, but if you're an 
external contributor, you can review submissions by project members (and it's 
up to them to decide if your review is adequate before acting on it).  Project 
members (should) have a "*" next to their name in the "submitted by" list.  (If 
they don't I think someone needs to go manually update a list of strings, feel 
free to report issues if that's wrong.)

Happy hacking,

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Enable auto-merge for twisted/twisted

[Glyph]

I think we should enable it everywhere.

-g

> On Jul 19, 2021, at 1:23 AM, Adi Roiban  wrote:
> 
> 
> 
> On Sun, 18 Jul 2021 at 01:04, Glyph  <mailto:gl...@twistedmatrix.com>> wrote:
> Nope, I'm fully in favor. When we're done reviewing, and waiting for CI, 
> reviewers should be able to set auto-merge and walk away.  Thank you for 
> raising the issue!
> 
> I have enabled auto-merge for twisted/twisted
> 
> Let me know if you want to be enabled for other repos.
> 
> -- 
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted-Infra org - twm added as maintainer

[Glyph]

Hi Adi,

This, and adding Tom to dornkirk/SSH, both sound good to me. I wish we had a 
more formal process for these various permissions but I think whatever the 
standard we would set, Tom has exceeded the bar by now.

Thanks.

> On Jul 14, 2021, at 5:27 AM, Adi Roiban  wrote:
> 
> Hi,
> 
> I tried to send this email 9 days ago... but I was not subscribed to the list 
> using this email... and have not seen the bounce message... if any.
> 
> --
> 
> I just wanted to let you know that I have added Tom Most as a "core" team 
> member to the twisted-infra organization.
> 
> This is more of an information.
> Let me know if I did something wrong
> I don't know if there is any process in place for adding new members to 
> twisted-infra org.
> 
> Tom can merge the code but without access to the actual production server 
> there is not much he can do.
> 
> I think that we can give him access to the servers to help with the 
> infrastructure.
> 
> I have not added the keys yet, but if any other member of the team think that 
> this is a good idea please check with Tom what are the preferred key/keys to 
> be added to ~/.ssh/authorized_keys
> 
> Tom's SSH Keys https://api.github.com/users/twm/keys 
> 
> 
> I can also add the keys once I get some feedback.
> 
> Cheers
> -- 
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Enable auto-merge for twisted/twisted

[Glyph]

Nope, I'm fully in favor. When we're done reviewing, and waiting for CI, 
reviewers should be able to set auto-merge and walk away.  Thank you for 
raising the issue!

> On Jul 14, 2021, at 5:33 AM, Adi Roiban  wrote:
> 
> Hi,
> 
> Do you see any issues if we enable the auto-merge feature for twisted/twisted?
> 
> The feature is documented here
> 
> https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/automatically-merging-a-pull-request
>  
> 
> 
> With this, in theory you don't have to wait for all the CI to be green before 
> a merge.
> You got your reviews, you fix some typos that don't require a re-review and 
> then set the merge message and your are done.
> Or there was a flaky test failure and just re-run the job and the merge will 
> be done once all the jobs are green again.
> 
> -
> 
> This can be abused if someone will enable auto-merge before the PR is 
> approved by a reviewer.
> As most of the time the reviewer will accept the PR but will request or 
> suggest minor changes.
> If the auto-merge is enabled, the PR is mergesd as soon as it is reviewed.
> 
> -
> The config looks like this:
> 
> You can allow setting pull requests to merge automatically once all required 
> reviews and status checks have passed.
> 
> Allow auto-merge
> Waits for merge requirements to be met and then merges automatically.
> 
> 
> 
> 
> There is also the option to block a merge only after all the conversations 
> are marked as resolved.
> I am using this for my project and it works ok... a bit annoying but can help 
> if you accedentally forgot to address a comment.
> 
> Cheers
> 
> 
> -- 
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted.web HTTPS client certificate

[Glyph]

> On Jul 14, 2021, at 7:05 AM, Barry Scott  wrote:
> 
> On Tuesday, 13 July 2021 22:48:18 BST Glyph wrote:
>> 
>>> On Jul 13, 2021, at 2:09 AM, Barry Scott  wrote:
>>> 
>>> On Monday, 12 July 2021 09:27:19 BST Glyph wrote:
>>>> FWIW I would avoid calling the pyOpenSSL APIs for this, since hopefully 
>>>> we'll manage to move away from OpenSSL and at least somewhat abstract away 
>>>> the transition. 
>>> 
>>> Are you thinking to use the Windows and macOS provided crypto API and only 
>>> use openssl on Unix systems?
>>> 
>>> Barry
>> 
>> It would be nice to have a system for backends so that we could do this when 
>> certain specific situations call for it (usually related to TLS clients, 
>> rather than servers, although having both would be great), but no, the main 
>> motivation is to drop OpenSSL entirely in favor of Rustls, as recommended by 
>> the ISRG: 
>> <https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/ 
>> <https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/>>.
> 
> That is a great goal for Twisted.

I'm glad you think so! I think it's a great goal for everybody, really ;-).

-g
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement

[Glyph]

> On Jul 17, 2021, at 3:47 AM, Adi Roiban  wrote:
> 
> 
> 
> On Fri, 16 Jul 2021 at 20:27, Glyph  <mailto:gl...@twistedmatrix.com>> wrote:
> 
> 
>> On Jul 16, 2021, at 2:20 AM, Richard van der Hoff > <mailto:rich...@matrix.org>> wrote:
>> 
>> On 16/07/2021 00:18, Glyph wrote:
>> 
>>> 
>>> 
>>>> On Jul 15, 2021, at 9:00 AM, Richard van der Hoff >>> <mailto:rich...@matrix.org>> wrote:
>>>> 
>>>> We can't just go and add type annotations because we need to maintain 
>>>> compatibility with older Twisted (to make it possible to package in Debian 
>>>> et al).
>>>> 
>>>> Any suggestions for keeping mypy happy?
>>>> 
>>> 
>>> Are you saying you need it to typecheck against older versions or just run 
>>> against them?
>> Ah, this gave me the clue I needed. We just need to run against them. Which 
>> means I can put type hints in comments, where they will be ignored at 
>> runtime. It's fiddly, but it will work well enough.
>> 
> You can also do 'from __future__ import annotations' to avoid the annotations 
> getting evaluated, which might be slightly less awkward.
>> 
>> Thanks Glyph, and thanks to Adi and Barry for your suggestions too.
>> 
>> One particular problem I came across was the type annotation on 
>> inlineCallbacks. I've filed https://twistedmatrix.com/trac/ticket/10231 
>> <https://twistedmatrix.com/trac/ticket/10231> about it - would appreciate 
>> thoughts.
>> 
> 
> This definitely looks wrong; there should be a TypeVar in there.  Adi, I'd go 
> so far as to say that this should be a release blocker, although the change 
> should be fairly minimal.
> 
> Richard, could you please make a proper PR for this to get CI kicked off and 
> make sure the new annotation doesn't cause any failures?
> 
> 
> I think that I will do a RC2 release candidate, since due to this issue the 
> GHA tests on RC1 are red - https://github.com/twisted/twisted/pull/1628 
> <https://github.com/twisted/twisted/pull/1628>
> So I plan to cherry-pick those changes and do a RC2.
> 
> Regarding  the release blocker, I will block the release for any ticket from 
> https://twistedmatrix.com/trac/report/26 
> <https://twistedmatrix.com/trac/report/26>
> 
> Right now, I have added https://twistedmatrix.com/trac/ticket/10231 
> <https://twistedmatrix.com/trac/ticket/10231> to that report.
> 
> So, if anyone knows how to fix https://twistedmatrix.com/trac/ticket/10231 
> <https://twistedmatrix.com/trac/ticket/10231> please help to unblock the 
> release.
> 
> Or if you think that https://twistedmatrix.com/trac/ticket/10231 
> <https://twistedmatrix.com/trac/ticket/10231> should not be a release 
> blocker, please add your comments.
> 
> For now, I will delay the release of RC2 to wait to see what is the 
> resolution for #10231

This depends largely upon your available time and the difficulty of making 
these changes, but I'd suggest doing an RC2 as soon as the fixes are ready for 
the new python versions, to ensure folks can test with those, and then do an 
RC3 as soon as the annotation fix is ready.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement

[Glyph]

> On Jul 16, 2021, at 2:20 AM, Richard van der Hoff  wrote:
> 
> On 16/07/2021 00:18, Glyph wrote:
> 
>> 
>> 
>>> On Jul 15, 2021, at 9:00 AM, Richard van der Hoff >> <mailto:rich...@matrix.org>> wrote:
>>> 
>>> We can't just go and add type annotations because we need to maintain 
>>> compatibility with older Twisted (to make it possible to package in Debian 
>>> et al).
>>> 
>>> Any suggestions for keeping mypy happy?
>>> 
>> 
>> Are you saying you need it to typecheck against older versions or just run 
>> against them?
> Ah, this gave me the clue I needed. We just need to run against them. Which 
> means I can put type hints in comments, where they will be ignored at 
> runtime. It's fiddly, but it will work well enough.
> 
You can also do 'from __future__ import annotations' to avoid the annotations 
getting evaluated, which might be slightly less awkward.
> Thanks Glyph, and thanks to Adi and Barry for your suggestions too.
> 
> One particular problem I came across was the type annotation on 
> inlineCallbacks. I've filed https://twistedmatrix.com/trac/ticket/10231 
> <https://twistedmatrix.com/trac/ticket/10231> about it - would appreciate 
> thoughts.
> 

This definitely looks wrong; there should be a TypeVar in there.  Adi, I'd go 
so far as to say that this should be a release blocker, although the change 
should be fairly minimal.

Richard, could you please make a proper PR for this to get CI kicked off and 
make sure the new annotation doesn't cause any failures?

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement

[Glyph]

> On Jul 15, 2021, at 9:00 AM, Richard van der Hoff  wrote:
> 
> We can't just go and add type annotations because we need to maintain 
> compatibility with older Twisted (to make it possible to package in Debian et 
> al).
> 
> Any suggestions for keeping mypy happy?
> 

Are you saying you need it to typecheck against older versions or just run 
against them?

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Enable pre-commit.ci for twisted/twisted

[Glyph]

> On Jul 15, 2021, at 7:35 AM, Jean-Paul Calderone  
> wrote:
> 
> Can you elaborate on this a little bit?  First time contributors won't be 
> able to push a branch to twisted/twisted.  Does giving pre-commit.ci 
>  write access to twisted/twisted also give it write 
> access to forks of twisted/twisted?  Or maybe only to branches in forks with 
> PRs targeting twisted/twisted?
> 

I believe so.  As a Twisted contributor you can actually push to branches of 
any user targeting a PR in Twisted.  I'm pretty sure repo:write for a github 
app grants the app similar access to a human org member.

I can't remember the exact mechanics of how this is configured, but I think 
that it's a side-effect of the "Require branches to be up to date before 
merging" setting being enabled, because that setting adds the "update branch" 
button to out-of-date PRs which implicitly gives the org push permission.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Enable pre-commit.ci for twisted/twisted

[Glyph]

> On Jul 15, 2021, at 7:25 AM, Adi Roiban  wrote:
> 
> I am +0 on this change due to security reasons...but I do think that it will 
> reduce a bit of the frustration for first time contributors.
> 

I'm +1; we've accepted a lot of cloud risk already, and this seems like a lot 
of benefit for minimal additional risk.  Is this asottile running the show?  
The website is not entirely clear.

It would be extra nice if we could restrict it from pushing to `trunk` on its 
own though, or running any release automation.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted.web HTTPS client certificate

[Glyph]

> On Jul 13, 2021, at 2:09 AM, Barry Scott  wrote:
> 
> On Monday, 12 July 2021 09:27:19 BST Glyph wrote:
>> FWIW I would avoid calling the pyOpenSSL APIs for this, since hopefully 
>> we'll manage to move away from OpenSSL and at least somewhat abstract away 
>> the transition. 
> 
> Are you thinking to use the Windows and macOS provided crypto API and only 
> use openssl on Unix systems?
> 
> Barry

It would be nice to have a system for backends so that we could do this when 
certain specific situations call for it (usually related to TLS clients, rather 
than servers, although having both would be great), but no, the main motivation 
is to drop OpenSSL entirely in favor of Rustls, as recommended by the ISRG: 
<https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/ 
<https://www.abetterinternet.org/post/preparing-rustls-for-wider-adoption/>>.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted.web HTTPS client certificate

[Glyph]

> On Jul 12, 2021, at 1:05 AM, Adi Roiban  wrote:
> 
> On Mon, 12 Jul 2021 at 08:09, Ian Haywood  <mailto:i...@haywood.id.au>> wrote:
> I am trying to work out how to retrieve on the server a X.509 
> certificate presented by the HTTPS client.  This code tries to tell me 
> the transport has no peer certificate.
> 
> same error when I use wget as a client so I think my problem is in the 
> server code. I'm using self-signed certificates
> 
> Any clues as to what I'm doing wrong?
> 
> Ian
> 
> 
> As commented by Glyph you will first need  to setup one or a list of accepted 
> CAs.
> 
> Then setup custom server-side context.
> 
> Add the CA via 
> https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context.load_verify_locations
>  
> <https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context.load_verify_locations>
> 
> This is done via 
> https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context.set_verify
>  
> <https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context.set_verify>
>  to define a path or dir
> or implement a custom one via 
> https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context.get_cert_store
>  
> <https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context.get_cert_store>

FWIW I would avoid calling the pyOpenSSL APIs for this, since hopefully we'll 
manage to move away from OpenSSL and at least somewhat abstract away the 
transition.  These map to the 'caCerts' and 'verify=True' arguments to 
CertificateOptions, if you need more flexibility than the tutorial 
documentation that I linked previously: 
https://twistedmatrix.com/documents/20.3.0/api/twisted.internet.ssl.CertificateOptions.html
 
<https://twistedmatrix.com/documents/20.3.0/api/twisted.internet.ssl.CertificateOptions.html>
> Without set_verify, during the TLS/SSL handshake the server will not ask the 
> client to send its own certificate.
> 
> -
>  
> I am using X509 authentication as a  passwordless authentication for 
> automated transfers, similar to the SSH key authentication.
> 
> The x509 certificate authentication is used by the Spanish government across 
> many of their services.
> Taxes, customs, health service ...
> 
> -- 
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted.web HTTPS client certificate

[Glyph]

> On Jul 12, 2021, at 12:07 AM, Ian Haywood  wrote:
> 
> I am trying to work out how to retrieve on the server a X.509 certificate 
> presented by the HTTPS client.  This code tries to tell me the transport has 
> no peer certificate.
> 
> same error when I use wget as a client so I think my problem is in the server 
> code. I'm using self-signed certificates
> 
> Any clues as to what I'm doing wrong?

It's been quite a while since I tried to do this, but:

For starters, you need to specify the certificate authority that will be 
validating your client's certificates.  I forget the precise specifics, but I 
believe if you don't specify a CA that will be used, it won't prompt the client 
to present a certificate in the first place, and also there's nothing for your 
endpoint to validate against, so what would it even do if the client did 
present one, other than let you retrieve it? The point is not to inspect the 
certificate but to authenticate it, an API that let you inspect it without 
first validating it against a CA would be a security nightmare.  And generally 
client certs are not understood to be validated by webtrust CAs, so "just 
validate it like usual" doesn't apply, since you can't check the SANs for a 
hostname.

There's a guide to doing this programmatically here: 
https://docs.twistedmatrix.com/en/twisted-21.2.0/core/howto/ssl.html#tls-server-with-client-authentication-via-client-certificate-verification
 

 - I believe that this edge-case is not supported by endpoints.serverFromString.

If it's within your control to avoid, don't use client certificate 
authentication from HTTPS clients.  It unnecessarily leaks a bunch of 
peripheral information to the service you're authenticating to, the UX is a 
disaster on basically every browser, and most of the big players have stopped 
caring about this use-case in favor of things like webauthn that properly exist 
entirely outside of the setup process for a secure channel.  I confess I used 
to think client cert auth was really cool myself, but luckily nothing I did 
with it ever took off :-).

That said: as long as the use-case exists, Twisted should have good support for 
it.  Adding a clientCA argument to twisted.internet.endpoints._parseSSL so that 
serverFromString would support it would be a pretty simple PR to put together, 
so if you've got to bite the bullet on this, a contribution to close the gap 
here would be appreciated.  Twisted is nothing if not a tool that should make 
it seamless to integrate with every bad idea anyone's had in a protocol design 
;-).

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement

[Glyph]

Thank you Adi!  Very glad to see us collectively getting back on the 
release-management horse again!  And it was super encouraging to see how 
quickly you were able to get this out, with all the automation that everyone 
(not least of all yourself) has been building to make this process faster and 
simpler.

It appears we've buried the lede on one of our biggest features in this release 
though - https://github.com/twisted/twisted/pull/1448/files 
 had no newsfile that shows 
up in the changelog, but it properly made Deferred into a generic, so I think 
this is the first release of Twisted where you might reasonably be able to use 
mypy without your own custom stub files! :)

-g

> On Jul 10, 2021, at 4:50 AM, Adi Roiban  wrote:
> 
> On behalf of the Twisted contributors I announce the release candidate of 
> Twisted 21.7.0
> 
> This is mostly a bugfix release.
> 
> Python 3.5 is no longer a supported platform.
> The minimum supported platform is Python 3.6.7.
> 
> The notable features are:
> 
> - Python 3.10 beta is now a supported platform.
> - twisted.web.template.renderElement() now accepts any IRequest implementer 
> instead of only twisted.web.server.Request. Add type hints to 
> twisted.web.template. (#10184)
> 
> The release and NEWS file is available for review at
> 
> https://github.com/twisted/twisted/pull/1614/files 
> 
> 
> Release candidate documentation is available at
> 
> https://twisted--1614.org.readthedocs.build/en/1614/ 
> 
> 
> Wheels for the release candidate are available on PyPI
> 
> https://pypi.org/project/Twisted/21.7.0rc1/ 
> 
> 
> python -m pip install Twisted==21.7.0rc1
> 
> Please test it and report any issues.
> If nothing comes up in 1 week, 21.7.0 will be released based on the latest 
> release candidate.
> 
> Many thanks to everyone who had a part in Twisted - the supporters of the 
> Twisted Software Foundation, the developers, and all the people testing and 
> building great things with Twisted!
> 
> -- 
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted Release Planning

[Glyph]

On July 5, 2021 at 3:55:38 PM, Adi Roiban 
(a...@roiban.ro(mailto:a...@roiban.ro)) wrote:

> Hi,
>  
> I don't have much time or much "need" but if needed I can help with a new 
> Twisted release.  

Hooray! I’m hype for 21.7!

>  
> I am still working on py3 migration as an urgent task so I am stuck with 
> 20.3.0.  
>  
> There was a lot of confusion / miscommunication / non-cooperation lately with 
> the Twisted release process.  
>  
> For example, I don't know when a new release is required / appropriate.  
>  
> Maybe we can look at doing 2 releases per year around December and June, or 
> sync with Ubuntu releases?  
> What do you think?.

My own preference is to just do them as often as we have capacity for. More 
releases means peoples’ work gets into the world faster, there’s more 
motivation to work on Twisted, more features means more users and more users 
means more potential contributors and it’s a virtuous cycle.


However since I personally have the bandwidth to do zero (0) releases per year 
myself, this does mean that others get to dictate the schedule ;-). If you’d 
like to do these two, other volunteers can do more (and more than one planned 
release manager at a time would be great). If at all possible, I think one per 
quarter, so four per year, would be a good lower bound to aim for. (Among other 
things, a quarterly cycle makes it easier to align with a popular style of 
resource planning in many companies.)

>  
> My plan is to have the release process documented and automated so that 
> anyone can take the role of the release manager for a certain release.  

So say we all! Thanks for all your work towards making this happen so far.

>  
> The current documentation for the release is here  
> https://docs.twistedmatrix.com/en/twisted-21.2.0/core/development/policy/release-process.html
>  
> Happy to recessive feedback and PRs for the current documentation.  
>  
>   
>  
> I don't see any tickets flagged as regressions in Trac 
> (https://twistedmatrix.com/trac/report/26) so I guess that we can cut a 
> release right away.
>  
> --  
>  
> I am thinking of a scenario in which person X sends a PR that is merged and 
> that person X needs a new release so the same person X can act as release 
> manager for that release.
> I don't know if that will work :)  

Looking forward to crossing that bridge when we come to it.

>  
> Cheers
> --
> Adi Roiban ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] IRC still alive?

[Glyph]

On July 5, 2021 at 4:44:26 PM, Adi Roiban 
(a...@roiban.ro(mailto:a...@roiban.ro)) wrote:

> Hi
> 
> On Tue, 6 Jul 2021 at 00:02, Tom Most 
> mailto:t...@freecog.net)> wrote:
> > Hi Adi,
> > 
> > IRC has moved to Libera.Chat(https://libera.chat/) due to the Freenode 
> > implosion(https://lwn.net/Articles/857140/). The channel names are the same 
> > --- #twisted, #twisted-dev, #twisted.web.
> > 
> > ---Tom
> > 
> 
> Thanks for the info. 
> 
> I guess that the website can be updated to no longer mention freenode - 
> https://twistedmatrix.com/trac/ 
> 
> 
> 
> 
> 
> 


That, and kenaan needs to be reconfigured to point at the new network. Please 
scrub any mention of freenode that you find from the docs & website.

> 
> Is there a web log? 
> 
> 
> 
> 
> 
> 


Not yet, but feel free to set one up.

> 
> Cheers 
> 
> > On Mon, Jul 5, 2021, at 3:45 PM, Adi Roiban wrote:
> > > Hi,
> > > 
> > > Only now I had time to check the Twisted IRC channel and it looks like 
> > > the channels are no longer registered ... and the IRC bot is not 
> > > recording the logs.
> > > 
> > > Should we continue to use IRC or move to gitter or something else that 
> > > has history and offline messages by default?
> > > 
> > > Cheers
> > > --
> > > Adi Roiban
> > > ___
> > > Twisted-Python mailing list
> > > Twisted-Python@twistedmatrix.com(mailto:Twisted-Python@twistedmatrix.com)
> > > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> > > 
> > 
> > ___
> > Twisted-Python mailing list
> > Twisted-Python@twistedmatrix.com(mailto:Twisted-Python@twistedmatrix.com)
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] IRC still alive?

[Glyph]

Freenode is dead, but Twisted has registered #twisted and #twisted-dev on 
libera.chat.

-g

> On Jul 5, 2021, at 3:45 PM, Adi Roiban  wrote:
> 
> Hi,
> 
> Only now I had time to check the Twisted IRC channel and it looks like the 
> channels are no longer registered ... and the IRC bot is not recording the 
> logs.
> 
> Should we continue to use IRC or move to gitter or something else that has 
> history and offline messages by default?
> 
> Cheers
> --
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] itervalues in pypi package

[Glyph]

Yes, this is expected. We haven't had a release in the last 3 months.  The main 
issue is that we don't currently have any volunteers to be release manager.

-g

> On Jul 2, 2021, at 2:27 PM, Nathaniel Haggard  wrote:
> 
> When making a python3 virtualenv and installing twisted I get t/i/process.py 
> that has .itervalues() in it, but this was fixed 3 months ago 
> (https://twistedmatrix.com/trac/ticket/9933 
> ).
> 
> ```
> curl 
> https://files.pythonhosted.org/packages/c2/41/3f30da0f7025480eff8feb9ef0927c6db6bbbf6e64985cac77ee0210a903/Twisted-21.2.0.tar.gz
>  
> 
>  > t
> tar xvzf t
> grep itervalues Twisted-21.2.0/src/twisted/internet/process.py
> for p in self.pipes.itervalues():
> for p in self.pipes.itervalues():
> ```
> 
> Thanks,
> Nate
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] startTLS errors not propagating to Factory

[Glyph]

> On Apr 29, 2021, at 3:09 AM, Richard van der Hoff  wrote:
> Right! That sounds plausible, and certainly gives me some places to poke. 
> I'll have another look later. Thanks very much!
> 

Glad to help. Looking forward to the resolution on this!

For what it's worth: it may also be useful to explicitly account for the 
Connector object in the transport.startTLS subsystem, it would be good to have 
the same reason object delivered in both places.  So fixing the SMTP code isn't 
necessarily the only path forward here.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] startTLS errors not propagating to Factory

[Glyph]

> On Apr 28, 2021, at 2:43 PM, Richard van der Hoff  wrote:
> 
> On 28/04/2021 07:06, Glyph wrote:
>>> Is the SMTP code holding the Factory wrong? Or is it reasonable to expect 
>>> the verification error to propagate into clientConnectionFailed - in which 
>>> case, how could this work?
>>> 
>>> Thanks for your thoughts!
>> Hi Richard,
>> Sorry for the delayed response here.
>> This is a bug in Twisted, and I think it boils down to this line: 
>> https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/protocols/tls.py#L391
>>  The code as written here appears to be expecting this sequence:
>> 1. failVerification is called with a reason containing a helpful
>>OpenSSL verification error
>> 2. we save that reason as `self._reason` for reporting later, calling
>>abortConnection()
>> 3. since the connection got aborted, we expect our underlying transport
>>to call loseConnection on us
>> 4. we will then get a falsey reason [?!?!] and as such we will use
>>self._reason instead
>> Assumption 4 is nonsense and has never been true within Twisted as far as I 
>> know; connectionLost always gets a Failure, Failures are never falsey, so we 
>> will never use self._reason.  To fix this we need to actually honor 
>> self._reason under the conditions we expect, i.e. it's a ConnectionAborted 
>> https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/internet/error.py#L209
>>  
> 
> Thanks very much for your thoughts on this, Glyph - it's always helpful to 
> have an insight into the intended design when trying to resolve this sort of 
> thing.
> 
> I don't follow your reasoning though. I think you might have misread the line 
> you point to 
> (https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/protocols/tls.py#L391).
>  It is "self._reason or reason" - ie, if self._reason is already set, it will 
> take precedence over reason.

Sigh. You're right, I read it backwards.

> In my tests at least, TLSMemoryBIOProtocol.connectionLost is doing exactly 
> the right thing - it is called with an unhelpful reason, but substitutes back 
> in the helpful reason which has already been stashed.
> 
> Rather, the problem, as I see it, is that it's not 
> TLSMemoryBIOProtocol.connectionLost that calls Factory.clientConnectionLost. 
> That is done by tcp.Client.connectionLost, via one of tcp.Client's myriad of 
> base classes, at 
> https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/internet/tcp.py#L508.
>  Of course, that doesn't get the benefit of TLSMemoryBIOProtocol's reason 
> switcheroo.
> 
> I'm still not quite sure who is in the wrong here.

Aah, yeah, this is a weird quirk of the ancient-style layering in the SMTP code 
:-|.  The way this should work is by using HostnameEndpoint.

I'm not sure exactly where we're going off the rails, but by using both the old 
'startTLS' style of starting a TLS connection, as well as relying on 
ClientFactory rather than an Endpoint of some kind, means that we're getting 
this duplicate notification; the one that you get to Protocol.connectionLost 
will come from TLS and have useful information, but the one that goes to the 
Connector will be coming straight from TCP.

The right thing to fix here, I think, is to ignore clientConnectionLost 
entirely, and instead to have the protocol object relay its failure to some 
other differently-named method on SMTPSenderFactory.

-g

> 
> Cheers
> 
> Richard
> 
> 
> PS: yes, once we figure out what's going wrong here, I'll at least write up 
> an issue...
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Does anyone know why trunk is failing CI on PyPy 7.3.4?

[Glyph]

It's merged.

On Wed, Apr 28, 2021, at 3:07 AM, Adi Roiban wrote:
> Hi
> 
> PR at https://github.com/twisted/twisted/pull/1592
> 
> I hope we can get a quick review and make the trunk green again :)
> 
> Cheeers
> 
> On Wed, 28 Apr 2021 at 10:30, Adi Roiban  wrote:
>> Hi,
>> 
>> A quick diff to fix it... and a ticket 
>> https://twistedmatrix.com/trac/ticket/10187
>> 
>> I will create a PR and we can refine the fix during the review.
>> 
>> 
>> diff --git a/src/twisted/words/protocols/irc.py 
>> b/src/twisted/words/protocols/irc.py
>> index 75300019b..a97abce2e 100644
>> --- a/src/twisted/words/protocols/irc.py
>> +++ b/src/twisted/words/protocols/irc.py
>> @@ -3678,10 +3678,10 @@ def ctcpExtract(message):
>>  normal_messages.append(messages.pop(0))
>>  odd = not odd
>>  
>> -extended_messages[:] = filter(None, extended_messages)
>> -normal_messages[:] = filter(None, normal_messages)
>> +extended_messages[:] = list(filter(None, extended_messages))
>> +normal_messages[:] = list(filter(None, normal_messages))
>>  
>> -extended_messages[:] = map(ctcpDequote, extended_messages)
>> +extended_messages[:] = list(map(ctcpDequote, extended_messages))
>> 
>> 
>> On Wed, 28 Apr 2021 at 10:01, Adi Roiban  wrote:
>>> Hi Wim,
>>> 
>>> Thanks for the report
>>> 
>>> On Tue, 27 Apr 2021 at 08:14, Wim Lewis  wrote:
 TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")?
 
 I don't have time right now to set up a PyPy-capable environment and try 
 to reproduce this, but perhaps someone does?
 
 Long version:
 
 Trunk has been broken since the last merge a week or so ago, but I 
 don't think the breakage is due to that merge.  As an experiment I made 
 a PR based off the last successful version of trunk, with a whitespace 
 change, and it now fails CI as well. So I think the failure must be due 
 to some change that isn't in Twisted or controlled-for by tox.
 
 The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. 
 Unlike our usual CI problems it doesn't seem to be a random failure: it 
 fails all the time in the same place. But the place doesn't make sense 
 to me. It's in the IRC CTCP tests, and they fail in the same ways each 
 time (an expected response is not received).
 
 The pair of CI runs closest to the change are these:
 
 run 5793: https://github.com/twisted/twisted/runs/2328450554
 run 5809: https://github.com/twisted/twisted/runs/2360415474
 
 There are a lot of differences, but sys.version went from 3.7.9 to 
 3.7.10 between those runs, so that seems like the most likely culprit.
 
 Last working:
 >  sys.version  : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52)
 >  [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]
 >  sys.prefix   : /opt/hostedtoolcache/PyPy/3.7.9/x64
 >  sys.exec_prefix  : /opt/hostedtoolcache/PyPy/3.7.9/x64
 >  sys.executable   : 
 > /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python
 
 First broken:
 >  sys.version  : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34)
 >  [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)]
 >  sys.prefix   : /opt/hostedtoolcache/PyPy/3.7.10/x64
 >  sys.exec_prefix  : /opt/hostedtoolcache/PyPy/3.7.10/x64
 >  sys.executable   : 
 > /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python
 
 PyPy's release notes for 7.3.4 don't list anything that jumps out at me:
 
 https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html
 
 My guess would be some latent timing bug in Twisted that was uncovered 
 by pypy execution time changes (I don't imagine that the CTCP code gets 
 exercised very heavily these days) or perhaps PyPy got a bug.
 
 
 Wim.
 
>>> 
>>> I have setup a pyp3.7.4 and I can reproduce it.
>>> 
>>> I see 3 options:
>>> 
>>> * Option A: Skip those tests on pypy  and open a separate ticket to fix the 
>>> test
>>> * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the 
>>> test and unpin it
>>> * Option C: Just fix the tests :)
>>> 
>>> I am looking at option C for one hour... if I can't find a fix will look 
>>> into option A.
>>> 
>>> Cheers
>>> 
>>> -- 
>>> Adi Roiban
>> 
>> 
>> -- 
>> Adi Roiban
> 
> 
> -- 
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com 
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] startTLS errors not propagating to Factory

[Glyph]

> On Apr 9, 2021, at 4:07 PM, Richard van der Hoff  wrote:
> 
> Hi folks,
> 
> I've been investigating https://github.com/matrix-org/synapse/issues/9566, 
> which amounts to: "when there is a TLS error connecting to the SMTP server, 
> the resultant exception is unreadable".
> 
> I think I've traced the problem to the fact that 
> SMTPSenderFactory.clientConnectionFailed is being called with an unhelpful 
> ConnectionAborted rather than anything more descriptive.
> 
> I've then reproduced that with a simpler test case: see 
> https://gist.github.com/richvdh/909761ff5dab23f0873eeddd7936a740. As you can 
> see, the output is: "Factory lost connection. Reason: Connection was aborted 
> locally using ITCPTransport.abortConnection."
> 
> This seems to be thanks to TLSMemoryBIOProtocol.failVerification, which 
> stashes the error and calls abortConnection(): 
> https://github.com/twisted/twisted/blob/trunk/src/twisted/protocols/tls.py#L427.
> 
> At this point I'm struggling. Is the SMTP code holding the Factory wrong? Or 
> is it reasonable to expect the verification error to propagate into 
> clientConnectionFailed - in which case, how could this work?
> 
> Thanks for your thoughts!

Hi Richard,

Sorry for the delayed response here.

This is a bug in Twisted, and I think it boils down to this line: 
https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/protocols/tls.py#L391
 

 

The code as written here appears to be expecting this sequence:

failVerification is called with a reason containing a helpful OpenSSL 
verification error
we save that reason as `self._reason` for reporting later, calling 
abortConnection()
since the connection got aborted, we expect our underlying transport to call 
loseConnection on us
we will then get a falsey reason [?!?!] and as such we will use self._reason 
instead

Assumption 4 is nonsense and has never been true within Twisted as far as I 
know; connectionLost always gets a Failure, Failures are never falsey, so we 
will never use self._reason.  To fix this we need to actually honor 
self._reason under the conditions we expect, i.e. it's a ConnectionAborted 
https://github.com/twisted/twisted/blob/3c868ac11786eef7ea269caa3056f00854128957/src/twisted/internet/error.py#L209
 

 

Does this make sense?  (Will you be able to file / fix this bug?)

Thanks for the report,

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] IDNA problem in twisted

[Glyph]

> On Apr 27, 2021, at 8:58 PM, Wim Lewis  wrote:
> 
> On Thursday, April 8, 2021 8:43:35 AM PDT, Barry Scott wrote:
>> We just added a patch to our twisted to prevent twisted from doing idna 
>> validation.
>> _idnaBytes and _idnaText not convert from bytes to unicode based on the type 
>> of
>> the provided arg.
>> 
>> We had to do this because there are domain names that youtube.com uses that 
>> are
>> not valid under IDNA-2008 https://tools.ietf.org/html/rfc5891#section-4.2.3.1
> 
> My reading of the RFC is that the YouTube domain you mention 
> (r2---sn-aigzrn7e.googlevideo.com) is an invalid "U-Label", but that doesn't 
> mean it's an entirely invaid domain label. It just means you can't legally 
> run it through IDNA and turn it into "xn--r2---sn-aigzrn7e-". The intent, as 
> I understand it, is to forbid any possibility of double-encoding or 
> double-decoding a label, not to forbid the possibility of using labels like 
> the one you mention.

I agree with this reading.

>> I can see why a UI would need to do IDNA-2008 converts and validation
>> but I'm not clear why its of value deep in the guts of twisted.
> 
> My guess is that this is just an accident of the way that the 
> bytes/characters distinction and the IDNA features were added to Twisted, and 
> is probably a bug.

+1.

We also have other issues with the Python IDNA library: 
https://github.com/kjd/idna/issues/18  
and would generally like to reduce our strictness via whatever mechanisms we 
can, even for things that genuinely require it (which this does not).

>> Why is this code needed at all in twisted?
>> If its for a high level API then why isn't it being called at the
>> edge of the high level API calls?
> 
> I'd argue that resolving URLs is in fact a high level API (from the point of 
> view of the name resoution system) but even so, it seems to me that Twisted 
> is doing the wrong thing here. The format of that label should prevent it 
> from ever being transformed by IDNA, but shouldn't prevent it from being 
> passed through unchanged, since it doesn't contain any codepoints outside of 
> the usual ASCII range.

Also agreed with all of this.

>> The key idea here is that its human input that will be converted.
>> But the code is used deep in the _sslverify.py where no human
>> input is entered.
> 
> _sslverify has to check whether the information in the server's certificate 
> matches the URL that the user supplied. Certificates can contain Unicode text 
> — at least in the (completely obsolete) CN-as-domain-name situation — so 
> _sslverify probably picked up the requirement for IDNA transformations from 
> that. (I don't remember whether dNSName SANs can contain unicode.)

Yep.

> What is the patch you decided to add to your version? Where in _sslverify did 
> the problem surface?

I am also very curious about this :).

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Codecov.io security incident

[Glyph]

> On Apr 16, 2021, at 11:26 AM, Adi Roiban  > wrote:
> 
> For twisted/twisted and I think that other repos the main secret available 
> for GitHub Action is the PYPY upload token.

Just to make sure here - you mean PyPI, right?

> I guess that what we can do is stop using the codecov.io  
> bash uploaded and
> switch back to python uploader.
> 
> Any other ideas ?

I think we are actually OK given the constraints on the env vars, but just to 
be safe, we should invalidate / rotate the PyPI upload token. Any admins have a 
few spare minutes to do that?  (And like… check to make sure nobody uploaded 
anything surprising on our project page ;-)).

-g




___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] GitHub Actions parallelism limit increase

[Glyph]

> On Mar 30, 2021, at 7:57 AM, Kyle Altendorf  wrote:
> 
> Hi All,
> 
> Has anyone contacted GitHub to see if they would be willing to increase the 
> parallelism limit in Actions?  My understanding is that we maintain two CI 
> systems (GitHub Actions and Azure Pipelines) for the sake of more 
> parallelism.  While perhaps worthwhile, this doesn't seem fun.  Maybe GitHub 
> would be willing to help out.

Not as far as I know. Do you want to give it a shot?

-g
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Enable and use GitHub Discussions for twisted/twisted

[Glyph]

> On Mar 29, 2021, at 7:27 AM, Adi Roiban  wrote:
> 
> Hi,
> 
> Should I enable GitHub discussions for twisted/twisted repo?

Rather than phrasing this in terms of advantages & disadvantages, I think it 
might be more fruitful to talk about what problem we would ostensibly be trying 
to solve by using it.  Right now I don't think the list is high-traffic enough 
that the big advantage (you can have a discussion without blasting out a 
message to too many people) would really solve any issues we're currently 
having.

Personally I am -0 because I don't think that Github's structure for 
notifications are particularly conducive to an Inbox Zero style approach, i.e. 
one where everyone involved with he project knows where to go check for things 
that they might want to pay attention to.  The mailing list makes this very 
straightforward; you just check your email, which you're probably doing anyway.

Github does have some notion of "inboxes", i.e. 
https://github.com/twisted/twisted/pulls/review-requested/@me 
 could probably 
replace https://twisted.reviews , but even though I 
am a person who thinks way too much about inboxes 
, I find 
https://github.com/notifications  difficult 
to approach.  If we were going to turn it on we should probably have a 
recommended way to check in on messages for contributors.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Small amount of feedback for a contributor to address?

[Glyph]

I just reviewed https://twistedmatrix.com/trac/ticket/10052#comment:5 and it’s 
almost landable, it just needs a small amount of feedback addressed to land. 
Given that the submitter had to wait months for this review I don’t know if 
they’re tracking it closely, so I figured I’d raise it here to see if anyone 
else had a moment to nudge a good bugfix here over the line.

Thanks in advance to whoever picks it up,  

-g ___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Strange recursion error with twisted.web

[Glyph]

> On Mar 16, 2021, at 4:20 AM, Peter Westlake  wrote:
> 
> On Sun, 14 Mar 2021, at 02:08, Glyph wrote:
>> 
>> I did indeed have a half-finished experiment to fix this.  The fix is now 
>> finished and in review here: 
>> https://twistedmatrix.com/trac/ticket/10125#comment:1 
>> <https://twistedmatrix.com/trac/ticket/10125#comment:1> .
>> 
>> If you could review it thoroughly enough, I could land it.
> 
> Excellent, thank you!
> 
> I think I understand how it works, but one thing puzzles me: how does the 
> queue in the jump() function ever get longer than one element? Is it because 
> it might contain a recursive call, which is activated by the unpause()?

This was actually a really insightful review - after thinking about it for a 
moment, I realized that the actual trampoline was already in 
twisted.internet.defer._inlineCallbacks, and this function was totally useless 
complexity!  Thank you!

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] automat question

[Glyph]

> On Mar 16, 2021, at 1:09 AM, Chris Withers  wrote:
> 
> 
> On 15/03/2021 09:34, Glyph wrote:
>>> Right, but as best I can tell, outputs in automat have to be defined as 
>>> part of the state machine class, I need targets only available after class 
>>> instantiation to be notified.
>> Outputs are just methods, so it sort of depends what you mean by "after 
>> class instantiation".  You can call whatever you want /in/ an output.
> 
> Right, but then I'd have to maintain my own list of subscribers and figure 
> out a way to duplicate in that logic in each of the output methods defined on 
> the machine.
> 
>>>> Again... have outputs.  I think there's something you're leaving out about 
>>>> how you want to have some /generalized/ output, but without knowing a bit 
>>>> more it's hard to say how it could help more :).
>>> 
>>> I think it's what I said above. TBH, I'm probably going to end up writing 
>>> "yet another state machine class" that does just what I need, but I think 
>>> that's okay - not everything has to work for every situation :-)
>> It sounds like what you /might/ want here is soemthing for constructing 
>> arbitrary machines at runtime, 
> 
> The states and transitions are very rigid.
> 
>> and automat is all about enforcing that your machine is static so that (for 
>> example) it can be visualized statically.  So yeah in that case, different 
>> library.
> 
> Take this example:
> 
> class LightSwitch(object):
>_machine = MethodicalMachine()
> 
>@_machine.state(serialized="on")
>def on_state(self):
>"the switch is on"
> 
>@_machine.state(serialized="off", initial=True)
>def off_state(self):
>"the switch is off"
> 
>@_machine.input()
>def flip(self):
>"flip the switch"
> 
>on_state.upon(flip, enter=off_state, outputs=[])
>off_state.upon(flip, enter=on_state, outputs=[])
> 
> 
> What I'm looking to do is something along the lines of:
> 
> aSwitch = LightSwitch()
> aSwitch.flip.addListener(myCallable)
> 
> myCallable might well be a deferred that something else is waiting on...

Thanks for the example.

My opinion is definitely that this should be an output produced by both edges 
here, and that output should manage an observer which does this extra 
functionality.  By modeling it as an output, you can see in your visual DFA 
where observers will be notified; and managing the list of observers is 
something that might have its own state-controlled lifecycle.

You can disagree of course but I think that's a very different design idiom 
than automat :)

-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] automat question

[Glyph]

> On Mar 15, 2021, at 1:18 AM, Chris Withers  wrote:
> 
> On 09/03/2021 18:53, Glyph wrote:
>>> On Mar 9, 2021, at 3:18 AM, Chris Withers >> <mailto:ch...@withers.org>> wrote:
>>> 
>>> I'm not sure we're quite on the same page: I'm not looking to inspect the 
>>> state, but more be notified when certain edges are traversed.
>> The way you get notified when an edge is traversed is you add an output to 
>> that edge :).
> 
> Right, but as best I can tell, outputs in automat have to be defined as part 
> of the state machine class, I need targets only available after class 
> instantiation to be notified.

Outputs are just methods, so it sort of depends what you mean by "after class 
instantiation".  You can call whatever you want in an output.

>> Again... have outputs.  I think there's something you're leaving out about 
>> how you want to have some /generalized/ output, but without knowing a bit 
>> more it's hard to say how it could help more :).
> 
> I think it's what I said above. TBH, I'm probably going to end up writing 
> "yet another state machine class" that does just what I need, but I think 
> that's okay - not everything has to work for every situation :-)

It sounds like what you might want here is soemthing for constructing arbitrary 
machines at runtime, and automat is all about enforcing that your machine is 
static so that (for example) it can be visualized statically.  So yeah in that 
case, different library.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Strange recursion error with twisted.web

[Glyph]

> On Mar 11, 2021, at 4:12 AM, Peter Westlake  wrote:
> 
> On Wed, 10 Mar 2021, at 23:11, Glyph wrote:
>> 
>> Would you mind filing a ticket in trac, or digging one up if you can 
>> find it? This problem rings a bell, and I think I might actually have 
>> some code for this lying around somewhere already.
> 
> Here is it: https://twistedmatrix.com/trac/ticket/10125#ticket
> 
> thanks,
> 
> Peter.

I did indeed have a half-finished experiment to fix this.  The fix is now 
finished and in review here: 
https://twistedmatrix.com/trac/ticket/10125#comment:1 
<https://twistedmatrix.com/trac/ticket/10125#comment:1> .

If you could review it thoroughly enough, I could land it.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Strange recursion error with twisted.web

[Glyph]

> On Mar 10, 2021, at 3:09 AM, Peter Westlake  wrote:
> 
> On Tue, 9 Mar 2021, at 19:28, Glyph wrote:
>> 
>> 
>>> On Mar 9, 2021, at 4:54 AM, Peter Westlake  wrote:
>>> 
>>> I'm getting a "maximum recursion depth exceeded" error that appears to be 
>>> coming from flatten(). The odd thing is that it only happens sometimes. The 
>>> HTML that's being flattened does have a few Deferreds in it. Those come 
>>> from function calls, which cache the results, which might explain why I 
>>> only see the error on the first visit to the page (as far as I can tell). 
>>> 
>>> The system recursion limit is the standard 1000. My HTML is only nested a 
>>> few tags deep, two orders of magnitude short of that. Is there anything 
>>> about the way flatten() works that might cause this behaviour?
>> 
>> flatten() can definitely result in some deep recursive stacks, 
>> particularly in combination with synchronous Deferreds which have their 
>> own accumulating stack costs. I'd be interested to see a minimal 
>> reproducer for this though, I'm sure we could do a lot better.
> 
> Here it is:
> 
> import sys
> from twisted.internet import reactor, defer, task
> from twisted.web.template import flatten
> 
> def output(stuff):
>sys.stdout.write(stuff.decode())
> 
> def sync(reactor):
>return flatten(None, [defer.succeed(str(i)+'\n') for i in range(1000)], 
> output)
> 
> task.react(sync)
> 
> 
> It fails after printing 197 lines. The same sort of thing using deferLater 
> instead of defer.succeed printed 1000 without error.
> 
> Peter.

Would you mind filing a ticket in trac, or digging one up if you can find it? 
This problem rings a bell, and I think I might actually have some code for this 
lying around somewhere already.

-g
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Strange recursion error with twisted.web

[Glyph]

> On Mar 9, 2021, at 4:54 AM, Peter Westlake  wrote:
> 
> I'm getting a "maximum recursion depth exceeded" error that appears to be 
> coming from flatten(). The odd thing is that it only happens sometimes. The 
> HTML that's being flattened does have a few Deferreds in it. Those come from 
> function calls, which cache the results, which might explain why I only see 
> the error on the first visit to the page (as far as I can tell). 
> 
> The system recursion limit is the standard 1000. My HTML is only nested a few 
> tags deep, two orders of magnitude short of that. Is there anything about the 
> way flatten() works that might cause this behaviour?

flatten() can definitely result in some deep recursive stacks, particularly in 
combination with synchronous Deferreds which have their own accumulating stack 
costs. I'd be interested to see a minimal reproducer for this though, I'm sure 
we could do a lot better.

-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] automat question

[Glyph]

> On Mar 9, 2021, at 3:18 AM, Chris Withers  wrote:
> 
> I'm not sure we're quite on the same page: I'm not looking to inspect the 
> state, but more be notified when certain edges are traversed.

The way you get notified when an edge is traversed is you add an output to that 
edge :).

When I was talking about "internal state", what I meant was not the literal 
current state atom of the machine, but the presence and names of states and 
edges themselves.  Within the abstraction of the state machine, the way you 
find out that an edge was traversed by having an output run.

> That said: what's the best practice when you want to visualize the current 
> state of a machine? In this case, the machine is for a unit of work in a 
> scheduling app, and the UI has a list of these units where it'd make sense to 
> show the current state.

This is definitely a gap.  I think today the best way to do this would be to 
have an event emitted by an output that invokes the serialization code to 
discover the state and have the UI show that.

> The unit of work also forms the boundary API between the from end object 
> model and the pool/scheduling back end. As such, each side can naturally call 
> inputs methods on the machine, as I believe automat intends, but it's how to 
> get notified by an output?

The output can just... call a UI observer?

> So, the front end may want to say "I no longer need this UoW done, please 
> kill it", the back end then needs to run the code to kill whatever's running 
> the UoW.
> Likewise, when the backend finishes a UoW, it wants to call either 
> machine.result(...) or machine.error(...) to say what happened, but how do I 
> wire in the front end bit that needs to get called when this happens?

Again... have outputs.  I think there's something you're leaving out about how 
you want to have some generalized output, but without knowing a bit more it's 
hard to say how it could help more :).

-g

> 
> cheers,
> 
> Chris
> 
> On 07/03/2021 22:34, Glyph wrote:
>> Automat is designed to make this sort of thing intentionally annoying, as 
>> you have discovered:).
>> 
>> The idea is that if you want to know this sort of internal state, it’s for a 
>> specific reason. That's not a blanket "No" — see for example how automat 
>> deals with serialization — but each such interface should be minimal and 
>> thoughtfully designed. Otherwise a state machine library just becomes a 
>> bunch of complex infrastructure around making an arbitrary series of 
>> function calls, and loses all of its helpful formalisms.
>> 
>> So probably we do need to make a change to automat if you really need to do 
>> this, but first it's important to know what your use-case is. In a lot of 
>> cases the answer is "just make an output. Stop trying to make the 
>> application see into the guts of the framework." But without knowing it’s 
>> impossible to say!
>> 
>> -g
>>> Hi,
>>> 
>>> Apologies if there's a better list for this, please let me know where...
>>> 
>>> I've grown to like Glyph's automat package a lot, but in a current 
>>> project, I have observers that need to know when a machine changes state.
>>> 
>>> What's the best way to let those observers know?
>>> 
>>> cheers,
>>> 
>>> Chris
>>> 
>>> ___
>>> Twisted-Python mailing list
>>> Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com>
>>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
>>> <https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
>> 
>> 
>> 
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com>
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
>> <https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
> 

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] automat question

[Glyph]

Automat is designed to make this sort of thing intentionally annoying, as you 
have discovered:).  

The idea is that if you want to know this sort of internal state, it’s for a 
specific reason. That's not a blanket "No" — see for example how automat deals 
with serialization — but each such interface should be minimal and thoughtfully 
designed. Otherwise a state machine library just becomes a bunch of complex 
infrastructure around making an arbitrary series of function calls, and loses 
all of its helpful formalisms.  

So probably we do need to make a change to automat if you really need to do 
this, but first it's important to know what your use-case is. In a lot of cases 
the answer is "just make an output. Stop trying to make the application see 
into the guts of the framework." But without knowing it’s impossible to say!  

-g
> Hi,
>  
> Apologies if there's a better list for this, please let me know where...
>  
> I've grown to like Glyph's automat package a lot, but in a current
> project, I have observers that need to know when a machine changes state.
>  
> What's the best way to let those observers know?
>  
> cheers,
>  
> Chris
>  
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] conch bytes/str traceback when /etc/ssh/moduli is not present

[Glyph]

> On Mar 5, 2021, at 3:24 AM, Colin Watson  wrote:
> 
> On Thu, Mar 04, 2021 at 04:16:59PM -0800, Glyph wrote:
>> There are a bunch of tickets you could file here:
>> 
>> Fixing the search path to comport with modern standards
>> Automatically generating a new one in a writable location if none exists
>> Better handle the case where it really truly doesn't exist and can't be 
>> generated (read-only filesystem or no readily discoverable, secure 
>> read/write locations)
>> 
>> and in fact probably all of these are valid :)
> 
> Probably not the second.  Generating a new set of suitable DH moduli
> takes a while (IIRC hours) - it's not something you'd want to do as part
> of any kind of interactive process.

For what it's worth, `ssh-keygen -G` on my laptop took 2 and a half minutes, 
`ssh-keygen -T` took 5.  It's slow, maybe even too slow for interactive use, 
but not quite "hours".

-g
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Klein?

[Glyph]

> On Mar 4, 2021, at 9:01 PM, Robert DiFalco  wrote:
> 
> Thanks! The trick will be figuring out how to handle Python exception vs 
> werkzeug exception, and the branched, error handling routes etc. Currently 
> this is the sort of data structure I'm posting to Data Dog. But I can make it 
> a pluggable observer-type idiom so anything can receive these metrics. 
> 
> tags=[
> "path:{}.{}.{}".format(module, className, method),
> "blocking:{}".format(isBlocking),
> "module:{}".format(module),
> "class:{}".format(className),
> "method:{}".format(method),
> "status_code:{}".format(request_code),
> "status_code_class:{}xx".format(status_code_class),
> ]
> metrics.timing("my.rest.endpoint", elapsed, tags=tags)
> I don't have it quite right yet. It's hard to get the status code, deal with 
> unhandled exceptions, etc. But I'll post questions here when I'm closer. 
> 

Nifty. Thanks for working on this.  I look forward to your PR!

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] conch bytes/str traceback when /etc/ssh/moduli is not present

[Glyph]

> On Mar 4, 2021, at 3:02 PM, Werner Thie  wrote:
> 
> On 3/4/21 08:51, Glyph wrote:
> 
>> Even if we want a traceback, a TypeError on str/bytes seems like the wrong 
>> kind to have.  Please file a bug (and open a PR, if you can :-)).
>> 
>> -g
> 
> Investigating I would say that with all the possibilities to configure
> for the moduli file to be found it is my fault to not have configured it
> properly for all the platforms I wanted to deploy to. On the other hand
> the basic assumption that moduli lives on BSDs in the /usr/local/etc/ssh
> directory seems now to become outdated, so the only change would be
> changing the default search location for the moduli file which again
> seems not to be warranted.
> 
> Mahalo, Werner

There are a bunch of tickets you could file here:

Fixing the search path to comport with modern standards
Automatically generating a new one in a writable location if none exists
Better handle the case where it really truly doesn't exist and can't be 
generated (read-only filesystem or no readily discoverable, secure read/write 
locations)

and in fact probably all of these are valid :)

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] conch bytes/str traceback when /etc/ssh/moduli is not present

[Glyph]

Even if we want a traceback, a TypeError on str/bytes seems like the wrong kind 
to have.  Please file a bug (and open a PR, if you can :-)).

-g

> On Mar 4, 2021, at 10:24 AM, Werner Thie  wrote:
> 
> Aloha
> 
> should the case of a missing moduli file be handled more gracefully than
> with a traceback or is this a bug?
> 
> I was running into this problem when installing on different OSs with
> OSX and FreeBSD not having an /etc/ssh/moduli file by default.
> 
> Mahalo, Werner
> 
> 2021-03-04T15:04:51+0100 [builtins.ConchSSHFactory#info] disabling
> non-fixed-group key exchange algorithms because we cannot find moduli file
> 2021-03-04T15:04:51+0100 [builtins.ConchSSHFactory] Unhandled Error
> Traceback (most recent call last):
>   File
> "/Users/anon/.pyenv/versions/3.7.10/lib/python3.7/site-packages/twisted/python/log.py",
> line 85, in callWithContext
> return context.call({ILogContext: newCtx}, func, *args, **kw)
>   File
> "/Users/anon/.pyenv/versions/3.7.10/lib/python3.7/site-packages/twisted/python/context.py",
> line 118, in callWithContext
> return self.currentContext().callWithContext(ctx, func, *args,
> **kw)
>   File
> "/Users/anon/.pyenv/versions/3.7.10/lib/python3.7/site-packages/twisted/python/context.py",
> line 83, in callWithContext
> return func(*args, **kw)
>   File
> "/Users/anon/.pyenv/versions/3.7.10/lib/python3.7/site-packages/twisted/internet/selectreactor.py",
> line 149, in _doReadOrWrite
> why = getattr(selectable, method)()
> ---  ---
>   File
> "/Users/anon/.pyenv/versions/3.7.10/lib/python3.7/site-packages/twisted/internet/tcp.py",
> line 1403, in doRead
> protocol.makeConnection(transport)
>   File
> "/Users/anon/.pyenv/versions/3.7.10/lib/python3.7/site-packages/twisted/internet/protocol.py",
> line 508, in makeConnection
> self.connectionMade()
>   File
> "/Users/anon/.pyenv/versions/3.7.10/lib/python3.7/site-packages/twisted/conch/ssh/transport.py",
> line 512, in connectionMade
> self.sendKexInit()
>   File
> "/Users/anon/.pyenv/versions/3.7.10/lib/python3.7/site-packages/twisted/conch/ssh/transport.py",
> line 535, in sendKexInit
> NS(b",".join(self.supportedPublicKeys)),
> builtins.TypeError: sequence item 0: expected a bytes-like object,
> str found
> 
> 2021-03-04T15:04:51+0100
> [twisted.conch.ssh.transport.SSHServerTransport#info] Disconnecting with
> error, code 3
> reason: b"couldn't match all kex parts"
> 2021-03-04T15:04:51+0100
> [twisted.conch.ssh.transport.SSHServerTransport#info] connection lost
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Developer docs should be updated on wiki when steps changed in code?

[Glyph]

> On Mar 1, 2021, at 5:30 AM, Adi Roiban  wrote:
> 
> 
> 
> On Wed, 3 Feb 2021 at 18:09, Glyph  <mailto:gl...@twistedmatrix.com>> wrote:
> 
> [snip]
> 
> I can't quickly find the place where we agreed to this, but I think several 
> years ago at this point we had a discussion about moving all these docs into 
> the source tree.  (If they're on the wiki, there's no review process or even 
> a place where updates can be staged for commentary before going live and 
> becoming "official".)
> 
>> Until this is fully completed, what is the correct thing to do when there is 
>> a mismatch between docs on the wiki, and docs in the tree?
> 
> Docs in the tree always win.
> 
>> I still refer to a lot of docs on the wiki, especially for process related 
>> things,
>> so I think it would be nice if the wiki docs were kept up to date, until the 
>> day that
>> they are fully deleted.
> 
> Let's start deleting them now, and replacing them with links to the in-tree 
> docs, rather than updating them.  They've been skewing out of date for a long 
> time.  When I was looking for information about how to do a revert, I found 
> wiki docs about linking to revisions in Subversion which didn't mention 
> Github, which gives a flavor for how outdated some of this stuff is.
> 
> Thomas, would you mind doing the honors for this document?  Links for process 
> docs should probably be to https://docs.twistedmatrix.com/en/latest/ 
> <https://docs.twistedmatrix.com/en/latest/> since, for process information 
> (unlike API information), trunk should be authoritative, not the latest 
> release.
> 
> 
> I don't know what wiki page we are talking about here. Any link would help.

Upthread (in the [snip]'d part), Craig referenced 
https://twistedmatrix.com/trac/wiki/TwistedDevelopment 
<https://twistedmatrix.com/trac/wiki/TwistedDevelopment>, and I thought we'd 
migrated some stuff from there.  If not, never mind.

> I see this wiki page and it already has a redirection: 
> https://twistedmatrix.com/trac/wiki/ReleaseProcess 
> <https://twistedmatrix.com/trac/wiki/ReleaseProcess>
> 
> My suggestion is that next time a dev needs to update a wiki page, 
> she/he/they should consider moving that page to twisted/twisted repo
> narrative docs and create a PR for that change.

Sounds good to me.

> From this wiki page https://twistedmatrix.com/trac/wiki/TwistedDevelopment 
> <https://twistedmatrix.com/trac/wiki/TwistedDevelopment>
> 
> I see that Security and Review Process are not yet migrated to narrativedocs.

Does someone want to go file tickets for those?

> And I think that "Contributor Advancement Path" can be removed as we now have 
> an informal processes for giving write access to the repo.

Perhaps this proposal has been rejected but I think we should leave a stub in 
there saying that we could really use some help with a formal process.  The 
informal process is sort of necessary at the moment, but implicit processes 
tend to overlook quieter people, and people who are less sure of themselves in 
the open source social context.  However, improving this process is going to be 
some work, and we'll have to find someone willing to do it.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Klein?

[Glyph]

> On Mar 1, 2021, at 12:51 PM, Robert DiFalco  wrote:
> 
> Is this the right place to ask klein questions?

Absolutely, it's a Twisted org project.

> I'm writing a metrics plugin for Klein and I can't figure out how to inject a 
> metrics handler so that I can get route, path, duration, and status code. 
> What I'm doing now sucks because Klein and twisted interact in complex ways 
> on Failure and status codes. 
> # Replace the klein _call with the metrics generating call
> _app._call = _callWithMetrics
> Rather than replace _call with my version of _call I was hoping there was a 
> cleaner way to get the start and stop with the result code of a route 
> invocation. Thoughts?


It sounds like you should contribute a patch that makes this an 
explicitly-supported pluggable entrypoint, rather than relying on a hack.  No 
need to figure out a way to make it work with existing versions, the magic of 
open source is that you can change it :).

Feel free to ping here when it's done to remind folks to do a review.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] [Twisted-web] Twisted 21.2.0 Release Announcement

[Glyph]

> On Feb 28, 2021, at 2:27 AM, Craig Rodrigues  wrote:
> 
> On behalf of Twisted Matrix Laboratories, I am honored to announce the
> release of Twisted 21.2.0!

Thank you so much for shepherding this release to completion, Craig.  It's so 
good to have a recent release out in the world again!  Looking forward to the 
many process improvements that have landed as well, and having more frequent 
releases produced by automation!

-g



___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

[Glyph]

> On Mar 3, 2021, at 4:58 AM, Jean-Paul Calderone  
> wrote:
> 
> Broadly, I agree.  But not with this part.  It seems like there is clearly a 
> trade-off that is better for everyone.  The trade-off represented by #1298:
> Breaks application code without providing any new functionality or fixing any 
> bugs
> Captures a long list of TODOs as an arbitrarily complex collection of sources 
> spread across the entire codebase
> All the work of addressing the problems still remains to be done 
> Contrast this with any basic ratchet-style approach (for example, capture the 
> list of mypy errors, then allow any PR that either removes some or doesn't 
> add any new ones):

In principle, I agree.  And some coarse-grained ratchets are absolutely worth 
doing, when they're supported by the tools - for example, we have one of these 
with the way we're handling `disallow_untyped_defs` right now.  I wish tools 
like Mypy would build in ratcheting infrastructure that was easy to use and 
convenient to deploy.

But in practice this type of ratchet involves a "# type: ignore" at every 
single call site, as well as delaying shipping stubs or maintaining separate 
stubs.  As well as building the maladaptive habit that #type:ignore or 
cast()ing your way out of type errors is the right way to address them in 
individual features and getting every code reviewer used to that.
> At the outset, no application code breaks because nothing has actually changed
> As work towards fixing the TODOs progresses, if it does break application 
> code then at least applications have a chance at some benefit
> As the work towards fixing the TODOs progresses, maintainers can easily look 
> up what issues remain by consulting the list of errors that feeds the 
> ratchet-style job.
> The exact mechanism for the ratchet isn't the point here.  Maybe Mypy plays 
> more nicely with inline comments telling it to ignore something, I don't 
> know.  The point is:

The "exact mechanism for the ratchet" introduces enough hard problems that it 
inevitably delays tooling adoption for months or years, reduces the benefits of 
the tooling adoption (what Twisted users really want are comprehensive stubs, 
less than Twisted to type-check internally, I think, and this lets us ship them 
sooner in at least a mostly-usable state).
> A centralized list of stuff left to do is better than a mess smeared across 
> the whole codebase
> If we're going to risk breaking applications we should at least try to offer 
> some value to them
> We shouldn't make applications deal with every change twice when we could 
> just disturb them once

Overall I very much appreciate your criticism and I wish that I could provide 
clear and concrete steps towards a functioning ratchet for all of these types 
of tools, but in practice the minor issues caused by the boil-the-ocean 
approach are an order of magnitude easier to address (and cheaper to leave 
un-addressed) than the effort required to build ratchets for our contributors 
that aren't a total nightmare to maintain and interact with.

Case in point: running `black` was a giant earthquake in the codebase but the 
pain associated with that was a drop in the bucket compared to the lingering, 
years-long nightmare of our attempt to build a style ratchet with 
twistedchecker.  In principle what we were trying to do with twistedchecker was 
very simple, but the practicalities ate us alive.

You could of course quite rightly argue that there were multiple conflated 
issues at play there!  But that's also the point: there's always a lot of 
complexity that comes along with trying to do codemod migrations "the right 
way" when that's going against the grain of the rest of the ecosystem.

I don't love that we've foisted this off onto our users, but I suspect Tobias 
spent a lot less time fixing the bug here than we've spent writing emails about 
it at this point ;-).

I look forward to the day where I'm wrong about all of this and every new code 
quality tool comes with a super easy-to-deploy ratchet, though.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

[Glyph]

> On Mar 3, 2021, at 8:25 AM, Richard van der Hoff  wrote:
> 
> 
> On 03/03/2021 08:07, Glyph wrote:
>> 
>> If dependencies could start testing against Twisted trunk in some capacity, 
>> we could get notified close to when unintentionally breaking changes occur, 
>> and dependencies can let us know well before the release happens, and we can 
>> either revert or they can fix things if the error is on their end.  If 
>> initially, say, crossbar and matrix would like to work with us to set up 
>> some kind of repeatable pattern we can suggest to others, that would be 
>> great.
> 
> I'll take this to the Synapse team to discuss further, but we could probably 
> easily arrange for one of our CI runs to install Twisted trunk from git 
> instead of pypi, which might be a good start.

This is specifically the approach I'd really rather not take :) and here's why:

You want to provide stability for you contributors so that if a problem is 
introduced, you don't halt development on that unrelated feature to diagnose 
the upstream issue.

You want to ensure that when users install your software, it works with 
everything that's currently released.

Ideally your CI configuration would be a timed job which builds against trunk 
and PR builds that build against a release, and then some process in place for 
reacting to regressions and filing a ticket upstream with us so we can work out 
what to do.

Easier said than done though, I know!  I wish we had something like this on 
Twisted for all of our dependencies, but we have had enough difficulty keeping 
our CI configuration current based on what cloud provider is falling over this 
month ;-).

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted v21.2.0 breaks Crossbar.io

[Glyph]

> On Mar 2, 2021, at 6:10 PM, Jean-Paul Calderone  
> wrote:
> 
> Policy aside, this change doesn't seem like much of an improvement to me.  If 
> I were to guess, I would guess the change was made to satisfy some check Mypy 
> is now being asked to make about Twisted.  If that's the case, it seems 
> unfortunate that real-world software is suffering so that a synthetic goal 
> can be achieved.  I do recognize there is a perception that practical value 
> can come from attending to the errors Mypy reports.  It would probably 
> benefit everyone if more care were taken to consider the real-world 
> consequences of changes that are made to satisfy the non-real-world goalposts 
> set by tools like Mypy.

I do think that Mypy was likely highlighting a real issue here, and it should 
have been fixed.  We had documented interfaces already, and we were failing to 
adhere to them properly, and Mypy was complaining about that.

For easy reference, the change that made these fixes is here 
https://github.com/twisted/twisted/pull/1298 


This genre of fix is definitely something we should unwind over time, although 
it does make it easier to start mypy-clean rather than have hundreds of 
exclusion lines that need to be manually adjusted, so on balance I agree with 
this tradeoff at the point it was made.

Specifically what I mean by "this genre of fix" is that you can always make 
mypy happy by transforming code like this:

class Thing:
pass

Thing().stuff()

into code like this:

class Thing:
@property
def stuff(self):
raise AttributeError("no such thing as `stuff`")

Thing().stuff()

The runtime behavior is the same, but now Mypy can't tell you about this class 
of bug any more.

So, at some point, we should slowly unwind all `NotImplementedError` methods 
and find ways to either implement them for real or make their required use 
raise at import time.

Finally: let's not develop an aversion to new tooling and change because it 
might create problems; experience over the last few years has shown me that 
Mypy can catch tons of real bugs and it's well worth getting the codebase to 
type check.  If we want to prevent breakages like this in the future, the 
answer is not to stop trying to get linters and typecheckers to run cleanly 
with arbitrary changes, but to figure out some kind of continuous integration 
solution that's actually continuous with our downstream dependencies

If dependencies could start testing against Twisted trunk in some capacity, we 
could get notified close to when unintentionally breaking changes occur, and 
dependencies can let us know well before the release happens, and we can either 
revert or they can fix things if the error is on their end.  If initially, say, 
crossbar and matrix would like to work with us to set up some kind of 
repeatable pattern we can suggest to others, that would be great.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Need help releasing new version of Twisted incremental

[Glyph]

> On Mar 1, 2021, at 11:13 AM, Craig Rodrigues  wrote:
> 
> 
> 
> On Mon, Mar 1, 2021 at 9:13 AM Colin Watson  > wrote:
> 
> Isn't this just because incremental hasn't had a release since 17.5.0?
> I added post= support way back in
> https://github.com/twisted/incremental/pull/37 
> , but that was after
> 17.5.0.
> 
> 
> Oh wow!  You are right.
> 
> Can someone help me release a new version of Twisted incremental?
> I don't know how to do it.
> 
> According to https://pypi.org/project/incremental/ 
>  , the latest release was
> done in 2017, which does not have Colin's fix.

Incremental doesn't have any maintainers except Hawkowl, unfortunately, and her 
responsiveness since COVID has been even lower than mine.  If I see her active 
online anywhere I'll mention that this is a problem.

-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted 21.2.0rc1 Release Candidate Announcement

[Glyph]

Thank you Craig for getting this out!  On Valentines day, no less :-) ❤️.

Very excited that the release train is back on the tracks! 

It's been a very rough year.  Thank you to everyone who contributed to the 225 
tickets that have been closed during this release, and thanks also to everyone 
else who is trying to accelerate the progress of the next release so the next 
one won't be such a massive effort to get out!

-g

> On Feb 14, 2021, at 6:53 PM, Craig Rodrigues  wrote:
> 
> It's time for another Twisted release!
> 
> There are two major announcements for this release:
> 
> - Python 2.7 support has been dropped.  Twisted 21.2.0 supports Python 3.5 
> and higher only
> - This will be the last Twisted release to support Python 3.5.
> 
> Twisted 21.2.0rc1 brings the following:
> - twist dns --pyzone now works on Python 3
> - twisted.web.twcgi now works on Python 3
> - all tests pass on all platforms on Python 3.8, including asyncio tests on 
> Windows
> - tests pass on Python 3.9 (thanks to Michał Górny)
> - type hints, as specified in PEP 484, are now used in functions throughout 
> the codebase
> - mypy (static type checker) is now run over the codebase during CI to help 
> detect problems, and help improve the use of types
> - code in is now formatted with Black (thanks to Tom Most)
> - many CI and automation improvements (thanks to Adi Roiban and Thomas 
> Grainger)
> - many more fixes (225 tickets closed!)
> 
> 
> You can install by running this command:
> 
>   python -m pip install Twisted==21.2.0rc1
> 
> The full tarball is available at:
> https://pypi.org/project/Twisted/21.2.0rc1/#files 
> 
> 
> The full NEWS file with all changes is at:
> https://github.com/twisted/twisted/blob/twisted-21.2.0rc1/NEWS.rst 
> 
> 
> Please test it, and let me know how your applications fare, good or bad!
> If nothing comes up, 21.2 will release very soon.
> 
> --
> Craig
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Glyph]

> On Feb 11, 2021, at 2:09 PM, Glyph  wrote:
> 
> 
> 
>> On Feb 11, 2021, at 1:04 PM, Glyph > <mailto:gl...@twistedmatrix.com>> wrote:
>> 
>> 
>>> On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega >> <mailto:wsanc...@wsanchez.net>> wrote:
>>> 
>>>>  My concern here is that Twisted may have added a dependency on requests, 
>>>> and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini 
>>>> does, that you aren’t allowed to.  That seems weak; I’d say a bug.  No?
>>> 
>>>  Note this is only cause when installing treq, so the issue seems to be 
>>> there, though for some reason, it’s only a problem with Twisted trunk, 
>>> which kind of confuses me, but I haven’t dug into it beyond trying to 
>>> figure out a bit about what fails to install.
>>> 
>>>  So I agree probably it’s not a blocker for Twisted, and it’s only annoying 
>>> to Klein for its own testing, not it’s clients, but if I were a client of 
>>> treq (and Klein is), I’d think this is a bug.
>> 
>> Attempting to reproduce this locally was a real adventure for me.  I had a 
>> bunch of failure-to-terminate cases from the new pip resolver, until I gave 
>> up, download the newest security update of each release of python, upgraded 
>> pip, tox, setuptools, and virtualenv for every python version, fully cleaned 
>> my git checkout, and re-ran.
>> 
>> Now everything passes except for coverage-py39-twtrunk, which still fails to 
>> terminate.
>> 
>> Under -, it spends a very long time printing messages like these over 
>> and over again:
>> 
>> INFO: pip is looking at multiple versions of setuptools to determine which 
>> version is compatible with other requirements. This could take a while.
>> INFO: This is taking longer than usual. You might need to provide the 
>> dependency resolver with stricter constraints to reduce runtime. If you want 
>> to abort this run, you can press Ctrl + C to do so. To improve how pip 
>> performs, tell us what happened here: 
>> https://pip.pypa.io/surveys/backtracking 
>> <https://pip.pypa.io/surveys/backtracking>
>> 
>> as it installs every version of setuptools and six that has ever existed.
>> 
>> It seems like Klein has backed itself into a very weird and complex corner 
>> with the new dependency resolver, but it doesn't appear unique to Twisted 
>> trunk, just perhaps tickled ever so slightly worse.
> 
> I've now been running `tox - -r -e coverage-py39-twtrunk` for well over 
> an hour, so I think we may have some pip resolver bugs to report.

I filed https://github.com/pypa/pip/issues/9601 
<https://github.com/pypa/pip/issues/9601> for this since it seems like it's 
well outside of our control.  If anyone else can add more information that 
would be great.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Glyph]

> On Feb 11, 2021, at 1:04 PM, Glyph  wrote:
> 
> 
>> On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega > <mailto:wsanc...@wsanchez.net>> wrote:
>> 
>>>  My concern here is that Twisted may have added a dependency on requests, 
>>> and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini 
>>> does, that you aren’t allowed to.  That seems weak; I’d say a bug.  No?
>> 
>>  Note this is only cause when installing treq, so the issue seems to be 
>> there, though for some reason, it’s only a problem with Twisted trunk, which 
>> kind of confuses me, but I haven’t dug into it beyond trying to figure out a 
>> bit about what fails to install.
>> 
>>  So I agree probably it’s not a blocker for Twisted, and it’s only annoying 
>> to Klein for its own testing, not it’s clients, but if I were a client of 
>> treq (and Klein is), I’d think this is a bug.
> 
> Attempting to reproduce this locally was a real adventure for me.  I had a 
> bunch of failure-to-terminate cases from the new pip resolver, until I gave 
> up, download the newest security update of each release of python, upgraded 
> pip, tox, setuptools, and virtualenv for every python version, fully cleaned 
> my git checkout, and re-ran.
> 
> Now everything passes except for coverage-py39-twtrunk, which still fails to 
> terminate.
> 
> Under -, it spends a very long time printing messages like these over and 
> over again:
> 
> INFO: pip is looking at multiple versions of setuptools to determine which 
> version is compatible with other requirements. This could take a while.
> INFO: This is taking longer than usual. You might need to provide the 
> dependency resolver with stricter constraints to reduce runtime. If you want 
> to abort this run, you can press Ctrl + C to do so. To improve how pip 
> performs, tell us what happened here: 
> https://pip.pypa.io/surveys/backtracking 
> <https://pip.pypa.io/surveys/backtracking>
> 
> as it installs every version of setuptools and six that has ever existed.
> 
> It seems like Klein has backed itself into a very weird and complex corner 
> with the new dependency resolver, but it doesn't appear unique to Twisted 
> trunk, just perhaps tickled ever so slightly worse.

I've now been running `tox - -r -e coverage-py39-twtrunk` for well over an 
hour, so I think we may have some pip resolver bugs to report.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Glyph]

> On Feb 11, 2021, at 1:15 PM, Adi Roiban  wrote:
> 
> I guess that we can remove idna from setup.cfg TLS section in Twisted
> 

Nope; we use it directly, in 
https://github.com/twisted/twisted/blob/7cf6c8bc320ac5fd96b4784f6feb932ea819856d/src/twisted/internet/_idna.py
 
.

In fact, if anything, we should bump it up, since we use it even in non-tls 
configurations: 
https://github.com/twisted/twisted/blob/cd97222df2ca7032bbff2fe9a8793d7b42dee8b7/src/twisted/internet/_resolver.py#L208
 

 .

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Glyph]

> On Feb 11, 2021, at 11:41 AM, Wilfredo Sánchez Vega  
> wrote:
> 
>>  My concern here is that Twisted may have added a dependency on requests, 
>> and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini 
>> does, that you aren’t allowed to.  That seems weak; I’d say a bug.  No?
> 
>  Note this is only cause when installing treq, so the issue seems to be 
> there, though for some reason, it’s only a problem with Twisted trunk, which 
> kind of confuses me, but I haven’t dug into it beyond trying to figure out a 
> bit about what fails to install.
> 
>  So I agree probably it’s not a blocker for Twisted, and it’s only annoying 
> to Klein for its own testing, not it’s clients, but if I were a client of 
> treq (and Klein is), I’d think this is a bug.

Attempting to reproduce this locally was a real adventure for me.  I had a 
bunch of failure-to-terminate cases from the new pip resolver, until I gave up, 
download the newest security update of each release of python, upgraded pip, 
tox, setuptools, and virtualenv for every python version, fully cleaned my git 
checkout, and re-ran.

Now everything passes except for coverage-py39-twtrunk, which still fails to 
terminate.

Under -, it spends a very long time printing messages like these over and 
over again:

INFO: pip is looking at multiple versions of setuptools to determine which 
version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the 
dependency resolver with stricter constraints to reduce runtime. If you want to 
abort this run, you can press Ctrl + C to do so. To improve how pip performs, 
tell us what happened here: https://pip.pypa.io/surveys/backtracking 


as it installs every version of setuptools and six that has ever existed.

It seems like Klein has backed itself into a very weird and complex corner with 
the new dependency resolver, but it doesn't appear unique to Twisted trunk, 
just perhaps tickled ever so slightly worse.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Twisted trunk and klein?

[Glyph]

> On Feb 10, 2021, at 3:18 PM, Craig Rodrigues  wrote:
> 
> Hi,
> 
> I quickly asked Wilfredo if Twisted trunk worked with Klein, and he mentioned
> that he saw this dependency issue on the Twisted klein side when used
> with Twisted trunk:
> 
> The conflict is caused by:
>The user requested idna==3.1
>hyperlink 21.0.0 depends on idna>=2.5
>requests 2.25.1 depends on idna<3 and >=2.5
> 
> 
> Is that problem purely for klein to deal with, or is there
> any issue that should be fixed in Twisted trunk?

This isn't even a bug in Klein, it's an issue with a version pin in its 
tox.ini:  
https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c6d/tox.ini#L36
 

 

The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are perfectly 
compatible; one's just a subset of the other.

So it might be nice to submit a Klein PR at some point soon, but it's not a 
release blocker.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Moving iocpsupport to an external package? Implications on Windows?

[Glyph]

> On Feb 10, 2021, at 6:45 AM, Craig Rodrigues  wrote:
> 
> 
> 
> On Wednesday, February 10, 2021, Glyph  <mailto:gl...@twistedmatrix.com>> wrote:
> 
> 
>> On Feb 10, 2021, at 12:40 AM, Craig Rodrigues > <mailto:rodr...@crodrigues.org>> wrote:
>> 
>> How will this patch change for people installing Twisted on Windows?
>> 
>> Twisted is still used by a lot of users on Windows, and the IOCP Reactor is
>> still used on this platform.
> 
> This line is what does the trick:
> 
> https://github.com/twisted/twisted/pull/1446/files#diff-fa602a8a75dc9dcc92261bac5f533c2a85e34fcceaff63b3a3a81d9acde2fc52R34
>  
> <https://github.com/twisted/twisted/pull/1446/files#diff-fa602a8a75dc9dcc92261bac5f533c2a85e34fcceaff63b3a3a81d9acde2fc52R34>
> 
> Basically, on Windows, you'll automatically get the `twisted-iocpsupport` 
> module as a hard dependency.  On other platforms, you won't.  Unless you're 
> using a truly ancient `pip` (which is less likely on Windows, where people 
> tend to install things themselves rather than getting them from a calcified 
> platfom) this will just work ✨ magically 
> 
> 
>  Ok, thanks for the clarification.  That is worth calling out in the release 
> announcement, since that is a major change for Windows users.

I don't disagree that it's worth calling out, but I might quibble on "major" 
change - it's a small build-configuration change that mostly affects Twisted 
contributors (in a positive way), and those with heavily locked-down build 
environments.  From the perspective of a Twisted user on Windows who wants to 
make use of the IOCP reactor and is installing with a regular `pip install`, 
nothing changes except a new line in `pip freeze`.

> Is the https://github.com/twisted/twisted-iocpsupport 
> <https://github.com/twisted/twisted-iocpsupport> repository subject to the 
> same rules as the Twisted core repository such as:
> - must have a Trac ticket

Yep! You can see in the README.rst on the repo, and also on its PyPI page: 
https://pypi.org/project/twisted-iocpsupport/ 
<https://pypi.org/project/twisted-iocpsupport/>

> - PR can only be merged after review approval from a reviewer with write 
> access to the https://github.com/twisted/twisted-iocpsupport 
> <https://github.com/twisted/twisted-iocpsupport> repository.
> etc.?

The full @twisted/twisted-contributors team has write access to this repo so 
it's all set for any Twisted reviewer who has the time and inclination.

I just noticed that PyPI just had me and graingert, though.  Sadly, PyPI 
doesn't have teams, so I synced up the management permissions on the PyPI 
project to match Twisted.  (The current list of uploaders isn't particularly 
principled or up to date, but better to have one list than two.  And I don't 
want a bus-factor of 1...)

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Getting ready to do a release of Twisted

[Glyph]

> On Feb 10, 2021, at 12:49 AM, Craig Rodrigues  wrote:
> 
> Thanks to review work by Tom Most,
> I have modified and merged https://github.com/twisted/twisted/pull/1502 
> <https://github.com/twisted/twisted/pull/1502>
> which addresses test failures encountered against Buildbot's regression suite.
> 
> I would like to proceed with doing a Twisted release.

Yay!  Let's go!

> Does anyone have any PR's that they are working on that they would like
> to get into the tree now?

Unless it's a regression, let's not delay any further.  The next release will 
be faster!

> Otherwise, I would request Twisted committers to hold off merging to trunk 
> for the next few weeks until the release is complete.

That's not how the release process is supposed to work.  Make the release 
branch, and development should proceed as usual.  (Not having any code freezes 
on trunk is the whole reason the process specifically uses a branch that is 
merged after the fact.)

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Moving iocpsupport to an external package? Implications on Windows?

[Glyph]

> On Feb 10, 2021, at 12:40 AM, Craig Rodrigues  wrote:
> 
> How will this patch change for people installing Twisted on Windows?
> 
> Twisted is still used by a lot of users on Windows, and the IOCP Reactor is
> still used on this platform.

This line is what does the trick:

https://github.com/twisted/twisted/pull/1446/files#diff-fa602a8a75dc9dcc92261bac5f533c2a85e34fcceaff63b3a3a81d9acde2fc52R34
 


Basically, on Windows, you'll automatically get the `twisted-iocpsupport` 
module as a hard dependency.  On other platforms, you won't.  Unless you're 
using a truly ancient `pip` (which is less likely on Windows, where people tend 
to install things themselves rather than getting them from a calcified platfom) 
this will just work ✨ magically ✨.

(Personally I think that in a future change, we should seriously consider 
moving this to an extra, since for some Windows users the simplicity of a 
pure-python install outweighs the performance improvements of a binary 
dependency.  And it should be possible to install Twisted without this 
accelerator.  But that is a very minor point compared to the benefits of not 
having to drag platform-variant wheels around for all other platforms!)

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Developer docs should be updated on wiki when steps changed in code?

[Glyph]

> On Feb 3, 2021, at 4:51 AM, Craig Rodrigues  wrote:
> 
> In this PR: https://github.com/twisted/twisted/pull/1461 
> 
> some changes were made to the Twisted development process
> for running the various linters before committing code.
> 
> I noticed that the changes in PR 1461 were not reflected
> in docs on the wiki, specifically here: 
> https://twistedmatrix.com/trac/wiki/TwistedDevelopment 
> 
> 
> which mentions tox environments that have been deleted in PR 1461.
> 
> I noticed this, when I recently tried to work on a branch from recent trunk,
> and when I tried to run some of the commands on that wiki page
> they didn't work, because they were deleted.
> 
> In this post-review comment:
> https://github.com/twisted/twisted/pull/1461#issuecomment-753695414 
> 
> 
> I asked Thomas if he could modify the wiki to reflect the changes he made in 
> trunk,
> but this hasn't been done yet.
> 
> My impression in the comments in PR 1461 is that Thomas and Adi want
> to move all this documentation into the source tree, and delete the stuff on 
> the wiki?
> 
> Is this the long term plan?

I can't quickly find the place where we agreed to this, but I think several 
years ago at this point we had a discussion about moving all these docs into 
the source tree.  (If they're on the wiki, there's no review process or even a 
place where updates can be staged for commentary before going live and becoming 
"official".)

> Until this is fully completed, what is the correct thing to do when there is 
> a mismatch between docs on the wiki, and docs in the tree?

Docs in the tree always win.

> I still refer to a lot of docs on the wiki, especially for process related 
> things,
> so I think it would be nice if the wiki docs were kept up to date, until the 
> day that
> they are fully deleted.

Let's start deleting them now, and replacing them with links to the in-tree 
docs, rather than updating them.  They've been skewing out of date for a long 
time.  When I was looking for information about how to do a revert, I found 
wiki docs about linking to revisions in Subversion which didn't mention Github, 
which gives a flavor for how outdated some of this stuff is.

Thomas, would you mind doing the honors for this document?  Links for process 
docs should probably be to https://docs.twistedmatrix.com/en/latest/ 
 since, for process information 
(unlike API information), trunk should be authoritative, not the latest release.

-g

> --
> Craig
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] resolving release management conflict

[Glyph]

Hello Twisted community!

Recently we've had some emotional language around the release process.  On 
https://github.com/twisted/twisted/pull/1464 
, Craig said several times that 
he found Adi's work and process changes "very aggressive".  Obviously, to have 
suggested this, Craig would have found some things Adi said to be emotionally 
loaded as well.

I think we need to resolve this conflict in order to move forward productively 
and avoid wasting more volunteer energy on negative feelings.  In order to do 
that, I think we need to centralize the discussion here.

At this point I think we need to all begin with a presumption of good faith.  
Adi, Thomas Grainger, Tom Most, Maarten, Povilas and Craig are all trying to 
address big problems in Twisted.  Nobody's trying to be aggressive, even if 
it's coming across that way.

(Personally, I should acknowledge that I don't see Adi's actions as aggressive, 
but I think we have to address Craig's perception that they are.  Part of the 
presumption of good faith is presuming that everyone has valid reasons for 
their reactions.)

Here's the status as I understand it:

Adi, Thomas, and Maarten are trying to address release automation issues to 
make future releases go faster.  They've merged several changes, including 
https://github.com/twisted/twisted/pull/1464 
, to that end.
Craig is trying to get out a release according to the current process, which 
means addressing https://twistedmatrix.com/trac/ticket/10069 
 as it is a release blocker.  He 
has a fix for this, but that fix is blocked.

While we've been waiting on #2, the changes in #1 proceed apace.  Craig's also 
noted several times that I've given permission for him to do the release and so 
he feels that breaking things mid-way is reneging on a promise from the project.

I believe the proximate cause of the conflict here is that we're dealing with 
this regression incorrectly.  When a regression is introduced, there's a 
process for dealing with it, originally documented here: 
https://twistedmatrix.com/trac/wiki/ReviewProcess#Revertingachange 
. [1]  The 
principle we need to outline clearly is this: if we have a regression, the 
change that introduced it should be reverted as soon as possible.  If we can't 
revert it directly because of changes to trunk in the meanwhile that introduce 
conflicts, ideally the change that we would make would be the closest possible 
thing to reverting it, even if that reintroduces a known bug that we have to 
reopen.

In the case where a lot of work has happened in the interim, and it's less work 
to just "fix forward" on top of trunk, it's fine to do a small fix as an end 
run around this process.  The goal of this process is to minimize the work 
required to restore trunk to a releasable state.

Right now we have three competing fixes which have all been flagged as 
incorrect in various ways:

from Tom: https://github.com/twisted/twisted/pull/1499 

Craig rejected this one by saying it will break "certain versions" of Python, 
but I don't understand why it was closed rather than reviewed / reassigned as 
normal.  (Or for that matter which versions are the problems, or how it breaks 
them.)
from Adi: https://github.com/twisted/twisted/pull/1501 

Craig rejected this one by saying "please stop working on this" but it also 
doesn't have any technical reason that this solution is incorrect.
from Craig: https://github.com/twisted/twisted/pull/1502 

This was originally given a passing review by Povilas from the Buildbot 
project, but twm added an additional blocking review shortly thereafter 
explaining that it introduced an additional regression.  It seems as though 
this one is now actively being worked on?

Craig: you closed two of these issues on the basis that you were "already 
working on a fix", but that's not a valid reason to close a PR.  Or, for that 
matter, to give it a failing review.  In the future, please comment on PRs that 
you don't want merged by pointing out their technical insufficiency, so that we 
can take the opportunity to learn for the benefit of future maintenance.  In 
this case, the underlying issue is clearly quite nuanced and requires expertise 
that touches platform differences in Python's core APIs, so having multiple 
people working on different fixes is totally fine; this effort is not wasted if 
everybody learns a little bit about the code in question, and it's definitely 
not wasted if we have to synthesize our approaches in the end anyway because 
they address different cases.

If you are pressed for time and don't have the ability to enumerate a large 
number of subtle problems, it's fine to 

Re: [Twisted-Python] Thread Cancelled

[Glyph]

If you're dealing with lots of clients on the public internet, sometimes this 
is just gonna happen, for a variety of reasons; it's normal.  We would welcome 
better error reporting for this scenario so it doesn't require the kind of 
debugging you just did :-).

-g

> On Jan 24, 2021, at 2:58 PM, Robert DiFalco  wrote:
> 
> That makes sense, thank you. A timeout seems unlikely but maybe the client is 
> closing the connection due to a network issue. This is an extremely rare 
> occurrence.
> 
> On Sun, Jan 24, 2021 at 2:41 PM Glyph  <mailto:gl...@twistedmatrix.com>> wrote:
> While a socket is open and receiving data, recv() will either give you a 
> non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN) 
> if no bytes are ready.  A result of zero bytes (the empty string) means "end 
> of file" - the other end has closed the socket.
> 
> So what's happening here is your client is timing out or otherwise canceling 
> its request by closing the socket, and this is the correct, intentional 
> response to that scenario.
> 
> -g
> 
>> On Jan 24, 2021, at 11:57 AM, Robert DiFalco > <mailto:robert.difa...@gmail.com>> wrote:
>> 
>> You're absolutely right, I meant "cancel the deferred". I don't grok server 
>> sockets very well so maybe someone can help. But apparently, klein does a 
>> .doRead from our server socket (getting the request from the client?). This 
>> returns a "why" of "connection done" so that closes the connection before we 
>> have written our response to the client, and that cancels the deferred SQS 
>> write.
>> 
>> https://github.com/racker/python-twisted-core/blob/master/twisted/internet/selectreactor.py#L148-L155
>>  
>> <https://github.com/racker/python-twisted-core/blob/master/twisted/internet/selectreactor.py#L148-L155>
>> 
>> The method above is "doRead". Which calls this:
>> https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L239
>>  
>> <https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L239>
>> 
>> I guess if If socket.rcv() returns an empty string it simply closes the 
>> connection.
>> https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L249-L250
>>  
>> <https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L249-L250>
>> 
>> Is that normal? I mean I guess it must be but then why is the read getting 
>> an empty string and closing the connection? I can't really account for it? 
>> Some kind of back pressure due to load? 
>> 
>> Thanks for any thoughts.
>> 
>> 
>> 
>> On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau > <mailto:colin.dunk...@gmail.com>> wrote:
>> 
>> 
>> On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco > <mailto:robert.difa...@gmail.com>> wrote:
>> Hi, I apologize this question is a little vague. I'm looking for pointers. I 
>> have a klein route that makes an underlying deferToThread call with a simple 
>> single thread (an IO based sync call I can't change, a boto3 sqs write). The 
>> thread pool is simple, just a couple of threads, nothing fancy. 
>> 
>> VERY rarely it appears that Klein cancels the thread. What techniques can I 
>> use to figure out why my thread is being Canceled? There's nothing in the 
>> failure to tell me "who, why, or where" it was canceled. Also, I cannot get 
>> this down to a reproducible case, but here's the boto3 sqs wrapper, this 
>> fall back works fine, but it's a band-aide for an error I can't track down.:
>> def write(self, payload):
>> """
>> Write message to SQS async from thread pool. If twisted cancels the
>> thread, instead write synchronously.
>> 
>> def _retrySynchronously(error):
>> if error.type != CancelledError:
>> return error
>> 
>> log.warn("Async SQS write cancelled. Calling synchronously.")
>> return defer.succeed(self._writeSyncFallback(payload))
>> 
>> deferredCall = self._deferToThread(self.sqs.write, payload)
>> deferredCall.addErrback(_retrySynchronously)
>> return deferredCall
>> 
>> def _writeSyncFallback(self, payload):
>> return self.sqs.write(payload)
>> 
>> The _deferToThread call just uses my own thread pool with 2 threads, but is 
>> otherwise stock. 
>> 
>> Is there a level of logging I'm missing or some other thing that would tell 
>> me why the thread is being canceled? The retry works great and Klein does 
>>

Re: [Twisted-Python] Thread Cancelled

[Glyph]

While a socket is open and receiving data, recv() will either give you a 
non-zero number of bytes if bytes are ready, or an EWOULDBLOCK (AKA EAGAIN) if 
no bytes are ready.  A result of zero bytes (the empty string) means "end of 
file" - the other end has closed the socket.

So what's happening here is your client is timing out or otherwise canceling 
its request by closing the socket, and this is the correct, intentional 
response to that scenario.

-g

> On Jan 24, 2021, at 11:57 AM, Robert DiFalco  wrote:
> 
> You're absolutely right, I meant "cancel the deferred". I don't grok server 
> sockets very well so maybe someone can help. But apparently, klein does a 
> .doRead from our server socket (getting the request from the client?). This 
> returns a "why" of "connection done" so that closes the connection before we 
> have written our response to the client, and that cancels the deferred SQS 
> write.
> 
> https://github.com/racker/python-twisted-core/blob/master/twisted/internet/selectreactor.py#L148-L155
>  
> 
> 
> The method above is "doRead". Which calls this:
> https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L239
>  
> 
> 
> I guess if If socket.rcv() returns an empty string it simply closes the 
> connection.
> https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/tcp.py#L249-L250
>  
> 
> 
> Is that normal? I mean I guess it must be but then why is the read getting an 
> empty string and closing the connection? I can't really account for it? Some 
> kind of back pressure due to load? 
> 
> Thanks for any thoughts.
> 
> 
> 
> On Sun, Jan 24, 2021 at 11:32 AM Colin Dunklau  > wrote:
> 
> 
> On Sun, Jan 24, 2021 at 11:45 AM Robert DiFalco  > wrote:
> Hi, I apologize this question is a little vague. I'm looking for pointers. I 
> have a klein route that makes an underlying deferToThread call with a simple 
> single thread (an IO based sync call I can't change, a boto3 sqs write). The 
> thread pool is simple, just a couple of threads, nothing fancy. 
> 
> VERY rarely it appears that Klein cancels the thread. What techniques can I 
> use to figure out why my thread is being Canceled? There's nothing in the 
> failure to tell me "who, why, or where" it was canceled. Also, I cannot get 
> this down to a reproducible case, but here's the boto3 sqs wrapper, this fall 
> back works fine, but it's a band-aide for an error I can't track down.:
> def write(self, payload):
> """
> Write message to SQS async from thread pool. If twisted cancels the
> thread, instead write synchronously.
> 
> def _retrySynchronously(error):
> if error.type != CancelledError:
> return error
> 
> log.warn("Async SQS write cancelled. Calling synchronously.")
> return defer.succeed(self._writeSyncFallback(payload))
> 
> deferredCall = self._deferToThread(self.sqs.write, payload)
> deferredCall.addErrback(_retrySynchronously)
> return deferredCall
> 
> def _writeSyncFallback(self, payload):
> return self.sqs.write(payload)
> 
> The _deferToThread call just uses my own thread pool with 2 threads, but is 
> otherwise stock. 
> 
> Is there a level of logging I'm missing or some other thing that would tell 
> me why the thread is being canceled? The retry works great and Klein does not 
> return an error from the route.
> 
> Thanks in advance.
> 
> 
> I think we'll need to see more code for this, specifically the caller of that 
> `write` method, and its callers, etc. Note that the thread itself isn't being 
> cancelled, the Deferred you get from _deferToThread is... so you'll most 
> likely need to find out what code interacts with that object to progress in 
> isolating this.
> 
> In my quick skim of the deferToThread and ThreadPool source, I can't find any 
> explicit cancellations. While that certainly doesn't rule it out, it does 
> make me think you're more likely to find the issue by inspecting the callers 
> involved.
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com 
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Questions about Project Leadership committee

[Glyph]

> On Jan 24, 2021, at 5:48 AM, Adi Roiban  wrote:
> 
> It's just a personal frustration that we need to make extra effort, when we 
> don't have much time and energy for code reviews.
> 

I feel this every single day :)

Thanks,

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Questions about Project Leadership committee

[Glyph]

> On Jan 24, 2021, at 1:24 AM, Adi Roiban  wrote:
> 
> Hi,
> 
> On Sun, 24 Jan 2021 at 04:29, Glyph  <mailto:gl...@twistedmatrix.com>> wrote:
> 
> On January 23, 2021 at 6:10:04 PM, Craig Rodrigues (rodr...@crodrigues.org 
> <mailto:rodr...@crodrigues.org>) wrote:
> 
>> 
>> 
>> On Sat, Jan 23, 2021 at 4:54 PM Glyph > <mailto:gl...@twistedmatrix.com>> wrote:
>> At the time the committee was created, I don’t remember if this was public, 
>> so I don’t feel comfortable sharing identities publicly because it wasn’t 
>> part of the deal at the time. This is not the way I’d structure things now, 
>> but given that several members are unresponsive and don’t seem to want to be 
>> associated with the project any more, I can’t ask them if it’s OK with them. 
>> If you can find any old public documentation feel free to share though; the 
>> issue is that I’m not sure it *has* been public. If it has, it can by all 
>> means remain public.
>> 
>> I can at least share that as I recall there are 6 members and I’m one of 
>> them. But you’d want to confirm this with the conservancy, so please join 
>> Adi’s private thread with them for further discussion.
>> 
>> 
>> Thanks for the clarification.
>> Have you recently contacted the 5 other committee members to confirm if they 
>> want to remain on the committee or not,
>> and either received confirmation (or feedback timeout)?
> Not recently although I’ve reached out several times in the past. I have no 
> plans to reach out again until someone has worked out with the SFC what our 
> options are and proposed a concrete plan.
> 
> I was in contact with SFC over IRC.
> 
> The first thing that someone need to do is send a message to 
> twis...@sfconservancy.org <mailto:twis...@sfconservancy.org>
> Only after no response is received in time (I don't know how long that 
> is...maybe 3 weeks) we can contact SFC and they
> will allocate extra resources to help solve this issue.
> 
> I have not sent a message to that list.
> For now, I don't plan to do it.
> I feel there is no consensus across the current active Twisted developers.
> 
> I encourage anyone else who wants to do it, to send the message to 
> twis...@sfconservancy.org <mailto:twis...@sfconservancy.org>
> 
> Good luck

Thanks for the update, Adi; I appreciate your taking the initiative on this.

If we want to have a private deliberation among committers about next steps, I 
believe https://github.com/orgs/twisted/teams/twisted-contributors/discussions 
<https://github.com/orgs/twisted/teams/twisted-contributors/discussions> makes 
it possible to do that, so that might be a tool to use.

(To be clear, I think any conclusions from this discussions need to be made 
public and transparent, but for reasons previously mentioned in this 
conversation, we may want to be able to share ideas less publicly before we 
pick one to avoid muddling things like future bidding on work.)

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Questions about Project Leadership committee

[Glyph]

On January 23, 2021 at 6:10:04 PM, Craig Rodrigues 
(rodr...@crodrigues.org(mailto:rodr...@crodrigues.org)) wrote:

>  
>  
> On Sat, Jan 23, 2021 at 4:54 PM Glyph 
> mailto:gl...@twistedmatrix.com)> wrote:  
> >  
> >  
> > At the time the committee was created, I don’t remember if this was public, 
> > so I don’t feel comfortable sharing identities publicly because it wasn’t 
> > part of the deal at the time. This is not the way I’d structure things now, 
> > but given that several members are unresponsive and don’t seem to want to 
> > be associated with the project any more, I can’t ask them if it’s OK with 
> > them. If you can find any old public documentation feel free to share 
> > though; the issue is that I’m not sure it *has* been public. If it has, it 
> > can by all means remain public.
> >  
> >  
> > I can at least share that as I recall there are 6 members and I’m one of 
> > them. But you’d want to confirm this with the conservancy, so please join 
> > Adi’s private thread with them for further discussion.
> >  
> >  
>  
>  
> Thanks for the clarification.  
> Have you recently contacted the 5 other committee members to confirm if they 
> want to remain on the committee or not,
> and either received confirmation (or feedback timeout)?
>  
>  
>  
>  


Not recently although I’ve reached out several times in the past. I have no 
plans to reach out again until someone has worked out with the SFC what our 
options are and proposed a concrete plan.

>  
> In this e-mail: 
> https://twistedmatrix.com/pipermail/twisted-python/2020-December/065364.html 
> , I mentioned that I  
> tried to look for documentation about this committee at 
> https://twistedmatrix.com and could not find any.
>  
> So if you are not aware of any public documentation regarding this committee, 
> then I think it is reasonable  
> to assume that such documentation does not exist.
> --
> Craig
>  
>  
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Questions about Project Leadership committee

[Glyph]

On January 23, 2021 at 1:36:40 PM, Craig Rodrigues 
(rodr...@crodrigues.org(mailto:rodr...@crodrigues.org)) wrote:

> Hi,
> I asked some questions about the Twisted project leadership committee here:
>  
> https://twistedmatrix.com/pipermail/twisted-python/2020-December/065367.html
>  
> but it looks like my questions got lost in the thread, and no one aso I will 
> ask again.
> Who are the current members of the committee?  

At the time the committee was created, I don’t remember if this was public, so 
I don’t feel comfortable sharing identities publicly because it wasn’t part of 
the deal at the time. This is not the way I’d structure things now, but given 
that several members are unresponsive and don’t seem to want to be associated 
with the project any more, I can’t ask them if it’s OK with them. If you can 
find any old public documentation feel free to share though; the issue is that 
I’m not sure it *has* been public. If it has, it can by all means remain public.


I can at least share that as I recall there are 6 members and I’m one of them. 
But you’d want to confirm this with the conservancy, so please join Adi’s 
private thread with them for further discussion.

>  
> What are the titles/roles/responsibilities of the current committee members?  

There are no titles or roles.

>  
> Is there a mailing list, IRC channel, or some other venue which the current 
> committee members belong to
> so that they can communicate amongst themselves, and also with the outside 
> world, such as the Software Freedom Conservancy?  

There’s theoretically a mailing list but it hasn’t seen a post in more than 4 
years. I’m not even sure if it still works in our new mail service 
configuration. So functionally, “no”.

>  
>  
> --
> Craig
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Upcoming Twisted Release

[Glyph]

> On Dec 28, 2020, at 9:53 AM, Glyph  wrote:
> 
>> It might take me a few days to complete this, since I am in the middle of 
>> holidays now, 
>> so won't be able to complete a release until the beginning of January.
> 
> Everybody should be able to take a break on a holiday (or heck, whenever you 
> want, for as long as you want, we're all volunteers here) but "please don't 
> make progress while I'm gone" is a surefire recipe for the person asking to 
> complete the thing themselves getting distracted and wandering away.  The 
> release is stuck enough as it is with CI issues and regressions, so if 
> someone has energy to make some forward progress let's please not hold it up.

Following up on this, since we are now officially past the "beginning" of 
January: Craig, is a release imminent or can someone else pick up some tasks 
yet?

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor for Linux io_uring

[Glyph]

> On Jan 10, 2021, at 4:42 PM, Ian Haywood  wrote:
> 
> 
> 
> On 8/01/2021 7:23 am, Glyph wrote:
>> 
>> The mess of ctypes stuff
> unclear what you mean: either the aio implementation or statx.py 
> 
Both, really, but I was mainly talking about statx.py.
>> seems somewhat irrelevant to the "asynchronous filesystem" part of this PR - 
>> do you think you could do a smaller version of this which decouples it from 
>> smb and ctypes?
> but yes both can be spun out easily, so it's just the interface and a plain 
> portable threads-based implementation. Where in the twisted tree should the 
> interface and implementation go?

I'd say probably twisted.internet?  (This is something that might get modified 
as folks review & comment)

>> (It would also be nice to have an interface that acts as an IProducer to 
>> integrate more natively with Twisted's support for backpressure rather than 
>> only having a custom readChunk method.  I think readChunk is still necessary 
>> for completeness since you need to be able to seek and offset, though.)
> certainly will look at this. It fits well with FTP's filesystem interaction. 
> 
> 
Thank you so much!

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor for Linux io_uring

[Glyph]

> On Jan 7, 2021, at 3:24 AM, Ian Haywood  wrote:
> 
> For async file I/O my plan would be to export a new IFilesystem (which is 
> closely based on conch.interfaces.ISFTPServer) apps would be have to be 
> written to use it, conch could with minimal tweaking,  and unsurprisingly the 
> SMB server will. 
> 
> Apps can fallback to thread-based or aio(7) based implementations that 
> already  exists as PoC:https://github.com/twisted/twisted/pull/1420 
>   
> 

The mess of ctypes stuff seems somewhat irrelevant to the "asynchronous 
filesystem" part of this PR - do you think you could do a smaller version of 
this which decouples it from smb and ctypes?  It would be very nice for 
twisted.web.static.File to be able to use this as well.  And it would be much 
easier to review in isolation.  (And also easier to review as concrete "clear 
asynchronous file I/O interface" functionality, rather than tying it to the 
term "VFS" which has a fairly checkered past within Twisted :)).

(It would also be nice to have an interface that acts as an IProducer to 
integrate more natively with Twisted's support for backpressure rather than 
only having a custom readChunk method.  I think readChunk is still necessary 
for completeness since you need to be able to seek and offset, though.)

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor for Linux io_uring

[Glyph]

I suspect that this may require somewhat less... cognitive surface area than 
your other contributions :).

And hey, we have a vaccine now, which means that maybe things will go back to 
normal or close enough to it that I'll have enough capacity to get back to it 
myself :)

-g

> On Jan 3, 2021, at 8:01 PM, Ian Haywood  wrote:
> 
> In investigating async file I/O I came across this. In a nutshell it's the 
> new epoll()
> 
> It's marginally more efficient although this is only apparent at very high 
> loads. What's more interesting is that io_uring accepts files as well as 
> network/pipe handles: avoiding the need for threads.
> 
> Here's a good intro: https://unixism.net/loti/index.html
> 
> If people think an IoUringReactor is worthwhile I'll open a ticket and make a 
> start.
> 
> However it will need a reviewer... :-)
> 
> Ian
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Plan/Goal for GitHub Sponsors

[Glyph]

It’s complicated and I’m not a lawyer, so maybe it is indeed not a problem. But 
in brief it’s like trademark protection, kind of, in that it becomes SFC’s 
problem to be aware that you’ve said these things and tell you not to say them. 
The twisted project (which is a bit of an amorphous concept to begin with) has 
authorized SFC to be its fiscal sponsor, the SFC has gone through the 
rigamarole with the IRS to ensure this is an exempt-able public benefit 
activity, and now someone is making claims about the project hiring, which 
they’re on the hook for. How does the IRS know your status of affiliation with 
the project or the conservancy for sure? Somebody has to investigate it, 
investigating means asking a bunch of questions and sucking up the SFC’s time 
and energy, even if no enforcement action is ever formally taken.  

In short: talk to the SFC first about the project’s status, get an actual 
official recommendation and not my random opinion about what may or may not be 
a problem, before doing anything related to fundraising. I can’t say anything 
authoritative about what is allowed, because as far as I understand it, 
*nothing* is allowed without untangling the PLC/approval process first. :-)  

(Except to raise money for the already-authorized expenses related to the 
continued hosting of twistedmatrix.com, of course.)  

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Plan/Goal for GitHub Sponsors

[Glyph]

(It might be generally good to stop discussing these priorities on a public 
list, and instead make a private list that anyone can request to join, but e.g. 
having joined precludes or disqualifies one from bidding on any resulting 
funded efforts. The big thing to avoid here is self dealing, ie having a clown 
showing up and advocating really aggressively that the thing Twisted really 
needs is a bunch of balloon animals that conveniently only they know how to 
make. Having a private group with documented members for discussion provides a 
nice audit trail that this isn’t happening.)

On January 3, 2021 at 3:32:56 PM, Glyph 
(gl...@twistedmatrix.com(mailto:gl...@twistedmatrix.com)) wrote:

> You can’t fundraise for a job opening that you haven’t cleared with the SFC 
> as mission-aligned and properly transparent; this could get them in trouble 
> with the IRS. You’ll need to clear this by writing a grant proposal and 
> having it approved first. Please delete this posting from the sponsors page 
> as soon as you can, since even posting to this list probably constitutes 
> public advertising.
>  
> On January 3, 2021 at 2:46:42 PM, Adi Roiban 
> (a...@roiban.ro(mailto:a...@roiban.ro)) wrote:
>  
> >  
> >  
> > On Sun, 3 Jan 2021 at 20:30, Jean-Paul Calderone 
> > mailto:exar...@twistedmatrix.com)> wrote:
> > > On Wed, Dec 23, 2020 at 4:41 PM Adi Roiban 
> > > mailto:a...@roiban.ro)> wrote:
> > > > Hi,  
> > > >  
> > > > I started a separate discussion to confirm the goal for a future 
> > > > fundraising.  
> > > >  
> > > > Top priority - Pay someone to help with the review queue  
> > > > Nice to have - Move server/services to Azure VM - We have at least $100 
> > > > monthly allowance for Azure not sure if we still have the huge 
> > > > $2000 allowance on rackspace.
> > > > Nice to have - Migrate Trac wiki to GitHub Wiki
> > > > Nice to have - Migrate Trac Ticket to GitHub Issues
> > >  
> > > I have a suggestion for another priority to be inserted above that top 
> > > priority - pay someone to look after the day-to-day logistics of the 
> > > project. For example: to triage tickets in the issue tracker (identify 
> > > duplicates to avoid redundant effort, classify issues by feature request 
> > > vs defect vs regression, etc), keep track of the release process so 
> > > releases are completed in a timely manner, identify and eliminate 
> > > friction in the development process, and identify big-picture directions 
> > > / priorities / roadmap items and track and coordinate efforts to achieve 
> > > them. I don't think someone could be hired to set the Twisted roadmap but 
> > > someone could be hired to solicit this information from core developers 
> > > and the wider community and organize it into a coherent plan.  
> > >  
> >  
> > Many thanks for your comments.  
> > I agree.
> >  
> > In this case, I think that for any near-future fundraising effort 
> > (including GitHub Sponsors) we should have a single goal:  
> >  
> > 1. Raise money to hire a project manager.  
> >  
> > That will be a part time job and the person will work on other projects.
> >  
> > The job activities will be (non exhaustive list - feel free to suggest):  
> >  
> > * triage tickets
> > * act as the release manager (the actual release has some automation) so 
> > this is more about communication
> > * identify friction in the development process and document and try to get 
> > consensus for a propose solution (the actual implementation can be done by 
> > someone else)
> > * identify big-picture directions / priorities / roadmap items and track 
> > and coordinate efforts to achieve them.
> > * help with fundraising / communication to future sponsors  
> >  
> > 
> >  
> > Do you have any idea of the required effort for a project manager for 
> > Twisted?
> > Maybe we can start with raising money to hire someone for 10 hours per 
> > week.  
> > Please suggest a different number of hours if you think that 10 hours are 
> > not a good start.
> >  
> > I think that at least for the first month, 10 hours per week is not much... 
> >  
> > but maybe after the second month, we can have 5 hours per week for project 
> > management
> > and use extra money for the review queue or implement the top priority 
> > tasks identified by the project manager.
> >  
> > ---
> >  
> > Then, if we raise more than 10 hours per week, we

Re: [Twisted-Python] Plan/Goal for GitHub Sponsors

[Glyph]

You can’t fundraise for a job opening that you haven’t cleared with the SFC as 
mission-aligned and properly transparent; this could get them in trouble with 
the IRS. You’ll need to clear this by writing a grant proposal and having it 
approved first. Please delete this posting from the sponsors page as soon as 
you can, since even posting to this list probably constitutes public 
advertising.

On January 3, 2021 at 2:46:42 PM, Adi Roiban 
(a...@roiban.ro(mailto:a...@roiban.ro)) wrote:

>  
>  
> On Sun, 3 Jan 2021 at 20:30, Jean-Paul Calderone 
> mailto:exar...@twistedmatrix.com)> wrote:
> > On Wed, Dec 23, 2020 at 4:41 PM Adi Roiban 
> > mailto:a...@roiban.ro)> wrote:
> > > Hi,  
> > >  
> > > I started a separate discussion to confirm the goal for a future 
> > > fundraising.  
> > >  
> > > Top priority - Pay someone to help with the review queue  
> > > Nice to have - Move server/services to Azure VM - We have at least $100 
> > > monthly allowance for Azure not sure if we still have the huge $2000 
> > > allowance on rackspace.
> > > Nice to have - Migrate Trac wiki to GitHub Wiki
> > > Nice to have - Migrate Trac Ticket to GitHub Issues
> >  
> > I have a suggestion for another priority to be inserted above that top 
> > priority - pay someone to look after the day-to-day logistics of the 
> > project. For example: to triage tickets in the issue tracker (identify 
> > duplicates to avoid redundant effort, classify issues by feature request vs 
> > defect vs regression, etc), keep track of the release process so releases 
> > are completed in a timely manner, identify and eliminate friction in the 
> > development process, and identify big-picture directions / priorities / 
> > roadmap items and track and coordinate efforts to achieve them. I don't 
> > think someone could be hired to set the Twisted roadmap but someone could 
> > be hired to solicit this information from core developers and the wider 
> > community and organize it into a coherent plan.  
> >  
>  
> Many thanks for your comments.  
> I agree.
>  
> In this case, I think that for any near-future fundraising effort (including 
> GitHub Sponsors) we should have a single goal:  
>  
> 1. Raise money to hire a project manager.  
>  
> That will be a part time job and the person will work on other projects.
>  
> The job activities will be (non exhaustive list - feel free to suggest):  
>  
> * triage tickets
> * act as the release manager (the actual release has some automation) so this 
> is more about communication
> * identify friction in the development process and document and try to get 
> consensus for a propose solution (the actual implementation can be done by 
> someone else)
> * identify big-picture directions / priorities / roadmap items and track and 
> coordinate efforts to achieve them.
> * help with fundraising / communication to future sponsors  
>  
> 
>  
> Do you have any idea of the required effort for a project manager for Twisted?
> Maybe we can start with raising money to hire someone for 10 hours per week.  
> Please suggest a different number of hours if you think that 10 hours are not 
> a good start.
>  
> I think that at least for the first month, 10 hours per week is not much...  
> but maybe after the second month, we can have 5 hours per week for project 
> management
> and use extra money for the review queue or implement the top priority tasks 
> identified by the project manager.
>  
> ---
>  
> Then, if we raise more than 10 hours per week, we can dedicate that money to 
> code review.  
>  
> Then, if we ran out of reviews in the queue, use the leftover money for 
> reducing operation overhead / removing roadblock.
>  
> > Ideally this person could also look after fundraising efforts to ensure 
> > that there are funds to continue to support their other activities.
> >  
>  
> We have a catch-22 situation here ... we need to hire someone to work on 
> fundraising ... we need to raise funds to hire someone :)
>  
> I can volunteer to bootstrap this effort and try to raise initial money to 
> find a project manager that can look after future fundraising.
> I don't have much free time and I am not a good project manager or 
> communication manager :)
>  
> Right now, I don't know who we could hire and what could be the selection 
> criteria ...
>  
> But I think that we can focus to see if we can raise 10 hours per week and 
> then worry about finding the right person :)  
>  
> > Helping with the review queue is great but it's a purely reactive activity. 
> > This is fine so far as it goes but it leaves the project without a coherent 
> > direction, which in turn makes less productive use of the resources 
> > available. The project should continue to operate reactively to address 
> > issues raised by the community but to really stay relevant, the core 
> > Twisted team itself also needs to identify coherent future goals and work 
> > to achieve them.  
> >  
> > Messing with 

Re: [Twisted-Python] Logging and SpawnProcess

[Glyph]

One minor point, if this is within your power: don't marshal data over stdout; 
sometimes libraries scribble on it.  Open an arbitrary additional file 
descriptor - say, 7 - and put your marshalled data there.

Definitely never use stderr for marshaling anything but lines of text - you're 
not gonna get away with that :-).  Even well-behaved libraries sometimes dump 
random text to stderr, that's what it's there for, so you don't do it with 
stdout!

Subunit has some really interesting ideas here - it's not Twisted code, but it 
does deal with communicating structured data over stdio streams in such a way 
that random overzealous SSH MOTD messages won't ruin your day:

https://github.com/testing-cabal/subunit/tree/0e9f67b9683bf2c8c82edb4e511d9b2c7708d900#version-2
 


Hope this is helpful,

-g

> On Dec 30, 2020, at 9:12 PM, Tom Most  wrote:
> 
> I'm not entirely following the question (is the log output intermixed with 
> other data on stdout?) but I think that you might find the implementation of 
> twisted.internet.endpoints.ProcessEndpoint a useful example. It relays stderr 
> to the logging stream: 
> https://github.com/twisted/twisted/blob/c8064075a207af1fc9ce19d8f885a986bc5ab00c/src/twisted/internet/endpoints.py#L336-L479
>  
> 
> 
> ---Tom
> 
> On Wed, Dec 30, 2020, at 11:38 AM, Robert DiFalco wrote:
>> Can anyone point me to some sample code for handling logging with 
>> SpawnProcess? Right now I'm using an EchoProtocol that takes stdout and 
>> resends it to the log. But it's kludgy because some sub-processes use 
>> stdin/stderr for marshaling data. What I'd most like to do is separate data 
>> transfer from log output in a clean way. 
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com 
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
>> 
>> 
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com 
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
> 
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Updating the project leadership committee

[Glyph]

It’s never been publicly documented; it was originally an internal interface 
between the discussions on this list and the SFC when the project was much 
smaller, so it’s probably written down in some internal conservancy document, 
maybe. If we’re going to do something new, obviously it should be a bit more 
transparent.

On December 28, 2020 at 5:52:08 PM, Craig Rodrigues 
(rodr...@crodrigues.org(mailto:rodr...@crodrigues.org)) wrote:

>  
>  
> On Sat, Dec 26, 2020 at 2:01 PM Glyph 
> mailto:gl...@twistedmatrix.com)> wrote:
> >  
> > I am extremely enthusiastic for your future reign on a new and improved 
> > project leadership committee :-). Thank you for working on this!  
> >  
>  
> Where is the documentation for the existing Twisted Project Leadership 
> Committee? I couldn't find it on https://twistedmatrix.com.  
> Where are the members for the existing committee listed?
>  
> --  
> Craig
>  
>  
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Upcoming Twisted Release

[Glyph]

> On Dec 28, 2020, at 9:50 AM, Glyph  wrote:
> 
> I'm also against this.  Inactivity for a week or two is not enough reason to 
> allow a known regression to be present in a release.
> 
I should clarify that maybe I haven't processed the full context for this 
specific issue though.  I do think it's acceptable to time out on an issue 
where a release blocker is claimed to exist if e.g. we can't reproduce the 
problem, or if it's a request for an above-and-beyond compatibility thing (like 
"please restore this private API, its removal breaks our application").  As a 
courtesy we might block a release for a small amount of time while waiting for 
a reproducer or a short-term private-API compatibility shim but the onus there 
is really on the reporter.

From what I can see though, this one is a pretty straightforward case of us 
just introducing a bug into a perfectly valid configuration though, just not 
one we happen to have in our test matrix right now.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Upcoming Twisted Release

[Glyph]

> On Dec 28, 2020, at 7:45 AM, Craig Rodrigues  wrote:
> 
> 
> 
> On Sun, Dec 27, 2020 at 3:59 PM Adi Roiban  <mailto:a...@roiban.ro>> wrote:
> Hi Craig,
> 
> On Sun, 27 Dec 2020 at 20:10, Craig Rodrigues  <mailto:rodr...@crodrigues.org>> wrote:
> 
> 
> On Sat, Dec 26, 2020 at 3:50 PM Adi Roiban  <mailto:a...@roiban.ro>> wrote:
> Hi,
> 
> I plan to act as a release manager for the next release and follow the plan 
> documented at
> 
> https://docs.twistedmatrix.com/en/latest/core/development/policy/release-process.html
>  
> <https://docs.twistedmatrix.com/en/latest/core/development/policy/release-process.html>
> 
> 
> I was previously working on releasing Twisted.  I was running into various 
> roadblocks, but was moving forward,
> and got permission from Glyph to move forward with this.
> Has this changed?
> 
> If you want to do the release, I am more than happy to not have to do the 
> release myself :)
> 
> Well since Glyph gave me permission to do this, I would like to complete this.

I'm not clear why Adi could not pick up your work where you left off?
 
> OK. I closed https://twistedmatrix.com/trac/ticket/10069 
> <https://twistedmatrix.com/trac/ticket/10069> as I think it's a duplicate.
> 
> Do you or Pierre plan to fix that ticket?
> 
> 
> I'm working on this now.
> 
> It might take me a few days to complete this, since I am in the middle of 
> holidays now, 
> so won't be able to complete a release until the beginning of January.

Everybody should be able to take a break on a holiday (or heck, whenever you 
want, for as long as you want, we're all volunteers here) but "please don't 
make progress while I'm gone" is a surefire recipe for the person asking to 
complete the thing themselves getting distracted and wandering away.  The 
release is stuck enough as it is with CI issues and regressions, so if someone 
has energy to make some forward progress let's please not hold it up.

-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Upcoming Twisted Release

[Glyph]

> On Dec 28, 2020, at 3:38 AM, Jean-Paul Calderone  
> wrote:
> 
> I guess there are no comments against removing a ticket from the 
> release-blocking list if the ticket is not active for 1 or 2 weeks.
> 
> Commenting against this was the main reason for my earlier reply.  I've left 
> my quoted reply above.

I'm also against this.  Inactivity for a week or two is not enough reason to 
allow a known regression to be present in a release.

Is there a reason we can't identify the commit that caused the problem and 
simply revert before doing the release, then fix it afterwards?

If we're in a position where trunk has drifted so far between the regression 
and its identification that it is too labor-intensive to revert and we need to 
fix it forward, then the release is stuck - that's the whole point of the 
"release blocker" category.  This is a very unfortunate situation but less 
unfortunate than shipping buggy releases that are known to break big users of 
Twisted.

-g

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Updating the project leadership committee

[Glyph]

Hi Adi,

The best next step would be to reach out to Conservancy directly—probably Karen 
Sandler, based on what the website says now—and describe the situation we are 
in. The problem is that the process we currently have in place is that the old 
committee nominates the new one and as far as I’m aware it doesn’t support any 
other options. The folks at the conservancy are probably the only ones who can 
change this.

The plans you propose all sound pretty reasonable to me, I just don’t have the 
time or energy to do all the planning and coordinating required to make them 
happen.

If you did have a fully fleshed out, written up proposal for what to do next, 
volunteers ready to be the next PLC, and all we needed was a PLC up-or-down 
vote, I might be able to help by tracking down a majority of the existing PLC 
and needling them to vote just once; however a better solution would be to get 
a more sustainable process in place with the SFC that takes this situation into 
account.

I am extremely enthusiastic for your future reign on a new and improved project 
leadership committee :-). Thank you for working on this!

-g

> On Dec 16, 2020, at 3:33 AM, Adi Roiban  wrote:
> 
> 
> Hi,
> 
> Based on a comment from Glyph
> 
> > The current "project leadership committee" is more than half made up of 
> > people who are a. no longer affiliated with the project and b. never answer 
> > their email at this point,
> 
> I started this discussion to see if we can update / refresh the project 
> leadership committee.
> 
> I remember that in 2015 I read the same comment.
> 
> Glypy, do you know the required number of members for the committee?
> 
> My suggestion is to try to arrange for a committee that is updated every 2 
> years... or maybe every year.
> I can organize the communication and project management stuff.
> 
> Here is the list of committers for the last to years
> 
> https://github.com/twisted/twisted/graphs/contributors?from=2018-12-16=2020-12-16=c
> 
> This is a quantitative approach... and does not imply quality or dedication, 
> but I think that is a good starting point for searching for candidates for 
> the committee.
> 
> I can also volunteer to be part of the committee.
> 
> Maybe we can create a new list of members for the commitee and then see if 
> this list will be accepted by theSoftware Freedom Conservancy.
> 
> I can also volunteer to do a bit of noise and see if we can raise more money.
> 
> I will not apply for the fellowship, so I hope that in this way we will not 
> have any complaints about my intentions :)
> 
> PS: I have many patches for Twisted that are not commited. I could just hire 
> another person to "review" them, but I don't think that this is fair.
> In fact, all the patches already passed an internal review, done by I person 
> paid by my company.
> So, I hope that we can hire an independent reviewer 
> 
> -- 
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Unjellying and circular references?

[Glyph]

Out official bug tracker is at https://twistedmatrix.com/trac/newticket — if 
you could file it there it would be great.  (Github login required.)

Thanks!

> On Dec 16, 2020, at 2:21 AM, Jasper Phillips  wrote:
> 
> 
> 
> 
>> On Fri, Dec 11, 2020 at 6:08 AM Jean-Paul Calderone 
>>  wrote:
>>> On Fri, Dec 11, 2020 at 8:19 AM Jasper Phillips  
>>> wrote:
>> 
>>> I'm using perspective broker to transfer objects in a networked game, which 
>>> I'm having trouble unjellying -- the remote versions wind up with dangling 
>>> twisted.persisted.crefutil._Dereference instances, so don't match the 
>>> original objects.
>>> 
>>> I'm seeing this for objects that have circular references to each other. 
>>> I've refactored things to mostly avoid circular references and sidestep 
>>> this, but have one remaining case where I find circular references mean 
>>> clearer code that I'm reluctant to refactor.
>>> 
>>> Is there some trick I'm missing to avoid _Dereferences?
>> 
>> No, it's supposed to Just Work™ so you've found a bug in some part of the 
>> implementation.  If you can produce a minimal reproducing example then it 
>> may be worth a bug report.  But after that, I'd suggest investigating HTTP 
>> or AMP as a replacement for PB.
> 
> Here's a test case demonstrating the bug:
> 
> import sys
> from twisted.persisted.crefutil import _Dereference
> from twisted.spread import pb, jelly
> 
> class RemoteCopyable( pb.Copyable, pb.RemoteCopy ):  pass
> jelly.globalSecurity.allowInstancesOf( RemoteCopyable )
> pb.setCopierForClassTree( sys.modules[__name__], pb.Copyable )
> 
> if __name__ == '__main__':
> # circular object ref
> cell   = RemoteCopyable()
> cell.link  = RemoteCopyable()
> cell.link.cell = cell
> 
> # Mimic sending across network
> broker = pb.Broker()
> serializedCell = broker.serialize( cell )
> remoteCell = broker.unserialize( serializedCell )
> 
> print( _Dereference is type(remoteCell.link.cell) )
> 
> This bug stems from twisted.spread.flavors.RemoteCopy.setCopyableState(), 
> which works fine for Python 2! But the UTF fix for Python 3 broke circular 
> references. It looks like this:
> 
> def setCopyableState(self, state):
> if _PY3:
> state = {x.decode('utf8') if isinstance(x, bytes)
>  else x:y for x,y in state.items()}
> self.__dict__ = state
> 
> Here is a simple tweak that fixes the problem:
> 
> def setCopyableState(self, state):
> if _PY3:
> for x, y in list(state.items()):
> if isinstance(x, bytes):
> del state[x]
> state[x.decode('utf8')] = y
> self.__dict__ = state
> 
> 
> Basically the reference unwinding that takes place in 
> twisted.spread.jelly._Unjellier._unjelly_reference()'s call to 
> ref.resolveDependants() relies upon setCopyableStates()'s passed in state 
> being used directly, such that all matching references' __dict__ point to the 
> same object.
> 
> Hope this helps clarify. Is there some more formal location than this list 
> that you'd like a bug report filed?
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Participating to GitHub Sponsors

[Glyph]

The money is first allocated to keeping the lights on (i.e.: paying for the 
infrastructure required to host twistedmatrix.com ).

If we manage to raise enough to go beyond that, then if we can clear the 
procedural hurdles necessary to spend money (i.e.: the current "project 
leadership committee" is more than half made up of people who are a. no longer 
affiliated with the project and b. never answer their email at this point, so 
we need to sort out some way to resolve that with the Conservancy) we will 
likely run another Fellowship program - you can see the results of previous 
programs on https://labs.twistedmatrix.com , 
and yes, it's mostly hiring someone to accelerate code review.

-g

> On Dec 15, 2020, at 6:48 PM, Ian Haywood  wrote:
> 
> No objection, but any plans for the money? I'd ask for more reviewers, I'd 
> like to contribute more to twisted but there's little point without access to 
> code review in a reasonable timeframe. 
> 
> On 16/12/2020 11:56 am, Adi Roiban wrote:
>> Hi,
>> 
>> Does anyone have anything against applying so that Twisted can be on the 
>> waitlist of the GitHub Sponsors ?  
>> 
>> https://github.com/sponsors/twisted/waitlist 
>> 
>> 
>> As far as I know, the money for the Twisted project is managed by the 
>> Software Freedom Conservancy organization and they are already approved 
>> hosts for GitHub Sponsors.
>> 
>> At this point, we only need to press the green button.
>> The bank account and other administrative issues are already sorted out.
>> 
>> 
>> 
>> If there are no complaints , I plan to press the green button in 1 week.
>> 
>> We will then have to see how to access those funds via Software Freedom 
>> Conservancy... but that is part 2.
>> 
>> Cheers
>> -- 
>> Adi Roiban
>> 
>> 
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com 
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
>> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Current twisted dns client just doesn't work

[Glyph]

Thank you for your contribution. I'm looking forward to getting this bug fixed!

-g

> On Dec 11, 2020, at 7:59 AM, spam tam  wrote:
> 
> I will try to pass the rest of the checklist
> 
> On Fri, Dec 11, 2020 at 6:45 PM Jean-Paul Calderone 
> mailto:exar...@twistedmatrix.com>> wrote:
> On Fri, Dec 11, 2020 at 10:31 AM spam tam  > wrote:
> Yes. I would like to replace ANY with A and  requests.
> I created pull request: https://github.com/twisted/twisted/pull/1488 
> 
> It's an easy solution. I just request for A records and if A doesn't provide 
> IP I create  request.
> 
> Thanks for your work on this so far.  Are you interested in finishing up the 
> PR (at least go down the rest of the checklist)?  If so, wonderful.  If not, 
> it would be good to know and maybe someone else can pick up the task from 
> here.
> 
> Jean-Paul
>  
> 
> On Fri, Dec 11, 2020 at 6:03 PM Barry Scott  > wrote:
> On Friday, 11 December 2020 14:23:49 GMT spam tam wrote:
> > Dis you read the whole my email?
> > Did you read this:
> > https://blog.cloudflare.com/rfc8482-saying-goodbye-to-any/ 
> > 
> > 
> > ANY is not supported by internet. Sometime works sometime not
> 
> Oh I missed that. That is very interesting.
> Is that what you are trying to fix in twisted? The use of ANY?
> 
> Barry
> 
> > 
> > Пт, 11 дек. 2020 г. в 12:26, Barry Scott  > >:
> > 
> > > On Thursday, 10 December 2020 23:29:33 GMT spam tam wrote:
> > > > I would like to find problems with ANY. But I think that there is no
> > > > problem.
> > > > DNS servers don't provide standard response for ANY request. My local
> > > > machine doesn't provide correct response for request:
> > >
> > > So you need to fix your network infra not twisted right?
> > >
> > > Barry
> > >
> > >
> > > >
> > > > $ dig amazon.in  any
> > > >
> > > > ; <<>> DiG 9.16.1-Ubuntu <<>> amazon.in  any
> > > > ;; global options: +cmd
> > > > ;; connection timed out; no servers could be reached
> > > >
> > > > My VPS server provide such response:
> > > >
> > > > ; <<>> DiG 9.10.3-P4-Ubuntu <<>> amazon.in  any
> > > > ;; global options: +cmd
> > > > ;; Got answer:
> > > > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54130
> > > > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
> > > >
> > > > ;; OPT PSEUDOSECTION:
> > > > ; EDNS: version: 0, flags:; udp: 512
> > > > ;; QUESTION SECTION:
> > > > ;amazon.in . IN  ANY
> > > >
> > > > ;; ANSWER SECTION:
> > > > amazon.in .  3599IN  HINFO   
> > > > "RFC8482" ""
> > > >
> > > > ;; Query time: 40 msec
> > > > ;; SERVER: 8.8.8.8#53(8.8.8.8)
> > > > ;; WHEN: Thu Dec 10 22:10:39 UTC 2020
> > > > ;; MSG SIZE  rcvd: 59
> > > >
> > > > It sometimes provides another response. But the problem is that 
> > > > behaviour
> > > > with ANY is not stable.
> > > > The present and the future of ANY are hazy. Read more here:
> > > > https://blog.cloudflare.com/rfc8482-saying-goodbye-to-any/ 
> > > > 
> > > >
> > > > If you don't see the problem please provide your opinion. I would like 
> > > > to
> > > > find solution with ANY but it seems it is impossible.
> > > >
> > > > So what do you think?
> > > >
> > > > On Thu, Dec 10, 2020 at 8:30 PM Barry Scott  > > > >
> > > > wrote:
> > > >
> > > > > Answers to but your emails in line.
> > > > >
> > > > > I've added the twisted list back in so others can comment.
> > > > >
> > > > > On Wednesday, 9 December 2020 21:17:51 GMT spam tam wrote:
> > > > > > As additional information you can check how operation system works
> > > with
> > > > > dns.
> > > > > > You can run
> > > > > >
> > > > > > *sudo tcpdump -i lo -v port 53*
> > > > > > as UDP local sniffer.
> > > > >
> > > > > Agreed great tool to debug this stuff with.
> > > > > And use wireshark to decode the output.
> > > > >
> > > > > > And run
> > > > > >
> > > > > > *dig google.com   > > > > > >*
> > > > > >
> > > > > > And you will see that it makes A request. Not ANY
> > > > >
> > > > > That is the dig default to use A. Use this to do a any query.
> > > > >
> > > > >dig google.com  any
> > > > >
> > > > > I checked the man page to see if anything extra can be printed but it
> > > > > looks like
> > > > > the default is to print everything dig knows how to print. The options
> > > only
> > > > > remove output it seems.
> > > > >
> > > > > > On Wed, Dec 9, 2020 at 11:42 PM spam tam  > > > > > >
> > > wrote:
> > > > > >
> > > > > > > Yes. You are correct. My local dns just is not stable. But 

Re: [Twisted-Python] Current twisted dns client just doesn't work

[Glyph]

Hi Barry!

> On Dec 8, 2020, at 6:57 AM, Barry Scott  wrote:
> 
> I recall that the DNS caching code cannot work and we replaced all the
> logic.

Did you open a ticket for this?  I realize we're a bit bottlenecked on reviewer 
bandwidth and stuff isn't getting integrated quickly, but "cannot work" seems 
like something we should at least have written down :).

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] revert / rework of "AMPv2" change?

[Glyph]

> On Oct 21, 2020, at 10:34 AM, Jonathan Bastien-Filiatrault  
> wrote:
> 
> On 2020-10-20 04 h 57, adi at roiban.ro (Adi Roiban) wrote:
>> Just my 2 cents
>> I think that whether this should be reverted or reworked should be between
>> the person raising this issue (Glyph) and the author of the PR (Jonathan).
> Glyph has asked me to submit the type annotation work on AMP, I will do
> this. I also plan on resubmitting a reworked patch that addresses the
> issues that glyph raised.
>> 
>> I see that Jonathan has responded to the feedback inside the merged PR.I
>> hope it can be reworked.
>> 
>> Since AMP v2 is not standardized maybe is ok to have it as a separate
>> experimental top level name.
>> 
>> I have never used AMP ...I tried to use it via ampoule but ended up using
>> the stdlib process pool as I was not able see any performance difference.
>> 
>> Since `amp` is used by `trial -j`  we need to keep it inside twisted.
>> 
>> But since AMP v2 is experimental, I think that is best to have it as a
>> separate txamp2 project.
>> In this way, the twisted AMP v2 project can follow its own rules.
>> Once AMP v2 is "mature" it can be moved to core twisted
>> 
>> So my take is to revert it and move it to twisted/txamp2.
> 
> I have no issue with changing the name or if this patch lives as a
> project besides twisted. I might even rework it into something more
> forward-compatible as suggested by glyph. I think netstrings are a
> decent option as it would make long value handling easier since values
> would always be contiguous without "continuation markers" interspersed.
> The main thing I dislike about netstrings are the variable-length ASCII
> numbers, but that is not a big complexity.

I originally didn't like this either, but since I designed AMP I've come around:

The thing about variable-length ASCII numbers is that they give you a lot of 
leeway to error out in case the protocol is not being respected.  With packed 
binary values, any 2-byte sequence is a valid length, so if something opens a 
socket and says "EHLO", it starts waiting for 17734 more bytes which it will 
never receive.  By contrast, netstrings have built-in redudancy; for a very low 
overhead (most length prefixes end up being 2-3 bytes rather than 2) you get 
the ability to immediately error out with a clear exception as soon as you see 
"E".  Anyone implementing this in a static context where they don't want heap 
allocations for the length prefix itself should not allocate arbitrary data, 
but rather, allocate a very small fixed buffer (6 bytes will get you 
considerably beyond the built-in limitations of AMP already) and again, 
immediately error out if that is exceeded.  Similarly the trailing comma 
provides an in-band sanity check where buggy applications will promptly crash 
rather than bleeding into the next message.

> Could the next AMP version
> use values framed using 32 bit or 64 bit integers ?

I'd really rather go with netstrings for all the reasons listed above.

> Transmitting long-ish values is a real use case and whatever extension
> we do must retain the dead-simple, easy to implement nature of the
> current AMP protocol. I now realise that I somewhat hijacked the
> "streaming values" bug (2529). IMHO, streaming should be handled at the
> application level, but I digress.

Yeah, that bug is more about making it easy to do the application-level stuff 
than implementing something implicit within the protocol.

>> 
>> 
>> BTW this is also my suggestion for the PR trying to introduce support for
>> SMB have twisted SMB implementation in a separate project.
>> 
> All the best,
> 
> Jonathan

Same to you.  Thanks for your contributions, and sorry for the slow pace of 
communication here; 2020 is a heck of a year.

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] smb component progress

[Glyph]

> On Sep 8, 2020, at 4:18 AM, Ian Haywood  wrote:
> 
> I have continued to work slowly on this project. Named pipes are supported, 
> the samba client can connect and list available shares.
> 
> i am currently working on a vfs layer using deferToThread to provide 
> asynchronous file access
> 
>  I have tried to keep the code divided into logically separate chunks . my 
> first chunk received comments from glyph 2 months ago but has remained in the 
> review queue since.
> 
> https://twistedmatrix.com/trac/ticket/9818
> 
> is there anyone available to review this code to progress it ?

Hi Ian,

I am sorry I haven't really had any time in this crazy year we've been having 
to review this myself.

Judging by the growing length of https://twisted.reviews 
<https://twisted.reviews/>, the pandemic has not been friendly to most other 
Twisted reviewers' schedules either.

The whole point of folding this into Twisted was supposed to make it easier to 
find reviewers, but "easier" remains a relative statement :).

If anyone else out there has a few spare cycles to look at these PRs, they're 
all still in review, and it would be very cool to get a memory-safe SMB 
implementation out there, particularly if we could make it part of Twisted.

Thanks again for this enormously valuable contribution; I hope we can find a 
way to integrate it.

-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] twisted vs asyncio vs tornado (and other event loops?)

[Glyph]

I still don't personally have time to execute on this, but I still think it's 
an important thing we should do if anyone has time to hack on it :)

-g

> On Nov 23, 2020, at 3:10 AM, Chris Withers  wrote:
> 
> Hi All,
> 
> Just bumped into Glyph's post from last year:
> https://twistedmatrix.com/pipermail/twisted-python/2019-April/032280.html
> 
> Doesn't look like there was much of a response, so wondered what had happened 
> with the plans laid out?
> 
> Also interested in how people see crochet fitting into this world...
> 
> cheers,
> 
> Chris
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Re-run Azure tests

[Glyph]

Also, since this is a common failure mode, the way to fix your login is to 
clear all cookies for:

microsoft.com <http://microsoft.com/>
live.com <http://live.com/>
microsoftonline.com <http://microsoftonline.com/>
s-microsoft.com <http://s-microsoft.com/>
skype.com <http://skype.com/>
visualstudio.com <http://visualstudio.com/>
bing.com <http://bing.com/>
azure.com <http://azure.com/>

(and any subdomains thereof, particularly login.live.com 
<http://login.live.com/> and account.microsoft.com 
<http://account.microsoft.com/>)

I have to do this about once a month these days, which is progress; I used to 
have to do it on every login.  After doing so (making sure no windows are open 
to a Microsoft property such as Azure Pipelines as I'm doing so) I can reliably 
get in.

To be clear this is not a joke about Microsoft's login process or the confusing 
mess of domains they use; this is literally what I have to do to get my login 
to work.

You probably *don't* need to clear them for Github since GH's own login is not 
broken in the same horrible way that MS's is.  At least, I haven't had to.

Happy logins, all,

-glyph

> On Oct 24, 2020, at 5:21 AM, Adi Roiban  wrote:
> 
> Hi,
> 
> 
> 
> On Mon, 19 Oct 2020 at 05:47, Wilfredo Sánchez Vega  <mailto:wsanc...@wsanchez.net>> wrote:
>  Yeah, log in with GH has never worked for me. Always results in what claims 
> to be a transient error. 
> 
> 
> They worked for me and I like them more than buildbot logs :)
> 
> Maybe we can try something to reduce the load for logs.
> For example the run report into a file to contain only the failed + todo.
> Use a separate step to show the content...have that step fail on errors.
> When you click on a build, it should get you to the short summary. 
>  
> Cheers
>   -wsv
> 
>> On Oct 13, 2020, at 9:40 PM, Glyph > <mailto:gl...@twistedmatrix.com>> wrote:
>> 
>> 
>> 
>>> On Oct 13, 2020, at 4:26 PM, Adi Roiban >> <mailto:a...@roiban.ro>> wrote:
>>> 
>>> Thanks. So you need to be logged with github account and not your normal 
>>> Azure/Devops account.
>>> I guess most of us are connected to Azure using the Azure AD account.
>>> 
>>> I was not expecting to see "Login via Github" on the MS login page.
>>> 
>>> After the Github login, the github re-run works. It automatically re-runs 
>>> all the failed tests...not only a single job.
>> 
>> After reading this I tried for like 20 minutes to try and figure out what 
>> microsoft is doing when I "log in"; it seems it's figured out my azure 
>> account is my github account, I could have sworn I had 2 but now I only have 
>> 1.
>> 
>> Not sure how one gets into this situation though, and the fact that large 
>> parts of the MS / live.com <http://live.com/> auth stack are just broken 
>> (right now the logout button doesn't work at all, I have to delete all my 
>> cookies) makes this tricky to debug.  So I'm giving up now :).
>> 
>> -g
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com>
>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
>> <https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com>
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python 
> <https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
> 
> 
> -- 
> Adi Roiban
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] towncrier review policies and etiquette (and maybe releases too)

[Glyph]

I'd love to update these policy documents to have something explicitly 
org-wide, but until somebody has the time to go through and do that properly, 
this is generally a good assumption to have.

> On Oct 21, 2020, at 1:59 PM, Tom Most  wrote:
> 
> I think that it's fine to fall back to the twisted/twisted policies[1] for 
> any project in the Twisted GitHub organization. The workflow won't apply 
> directly --- no Trac, etc. --- but the core "someone must review all changes" 
> and "test all the things" principles are relevant. That's how I treat Treq.
> 
> ---Tom
> 
> [1]: https://twistedmatrix.com/trac/wiki/ContributingToTwistedLabs
> 
> On Wed, Oct 21, 2020, at 12:11 PM, Kyle Altendorf wrote:
>> Where does towncrier stand on review policies and etiquette?  I 
>> generally don't like to just jump into new projects and start reviewing 
>> and merging but I don't see other activity on that front nor do I see 
>> any guidance on how it should be done.  If I know how to proceed 
>> properly then I might dedicate some review time to it.
>> 
>> Cheers,
>> -kyle
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] revert / rework of "AMPv2" change?

[Glyph]

I just glanced at https://github.com/twisted/twisted/pull/1417 
<https://github.com/twisted/twisted/pull/1417> after it was merged and noticed 
a few problems:

- It exposes a new top-level name rather than setting the protocol version as a 
parameter
- The NEWS files are malformed which is going to lead to some confusing 
duplication in the changelog
- AMPv2 itself does not appear to be "standardized" - the page at 
https://amp-protocol.net/conversations_v2.html 
<https://amp-protocol.net/conversations_v2.html> appears to be a preliminary 
suggestion for how long values might be handled rather than something broadly 
implemented; for one thing, I'd never heard of it before and for another the 
site itself doesn't link the document. This would be fine as a first draft, but 
I think it needs to be given some more revisions, given that (for example) 
there are a number of backwards-compatible ways to do this.
- The layering is wrong because it puts the protocol-parsing into a leaf class 
in the hierarchy, when the parsing logic was deliberately isolated to a lower 
level to facilitate different framing mechanisms.
- As JP pointed out, the tests have a potential bug where they can leak errors 
between cases.

I don't normally like to revert folks' work once it's been reviewed and 
accepted, particularly when there's no process violation (broken CI, lack of 
code coverage etc), but I'm particularly concerned about lending Twisted's 
imprimatur to this protocol extension as a whole new version without much more 
context on who is implementing it and what other options were considered, 
particularly when I personally (nominally the inventor of AMP) don't like the 
direction it has taken :).

What do other folks think?  Anyone else have more of a finger on the pulse of 
where the "v2" conversation has been happening?

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Re-run Azure tests

[Glyph]

> On Oct 13, 2020, at 4:26 PM, Adi Roiban  wrote:
> 
> Thanks. So you need to be logged with github account and not your normal 
> Azure/Devops account.
> I guess most of us are connected to Azure using the Azure AD account.
> 
> I was not expecting to see "Login via Github" on the MS login page.
> 
> After the Github login, the github re-run works. It automatically re-runs all 
> the failed tests...not only a single job.

After reading this I tried for like 20 minutes to try and figure out what 
microsoft is doing when I "log in"; it seems it's figured out my azure account 
is my github account, I could have sworn I had 2 but now I only have 1.

Not sure how one gets into this situation though, and the fact that large parts 
of the MS / live.com  auth stack are just broken (right now 
the logout button doesn't work at all, I have to delete all my cookies) makes 
this tricky to debug.  So I'm giving up now :).

-g___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Mailgun Email Service Replacement Proposal

[Glyph]

> On Oct 2, 2020, at 11:35 PM, Glyph  wrote:
> 
> Right now, I'm dealing with a minor issue where our "plan" (which, again, 
> Mailgun has generously given us for free) no longer includes the ability to 
> process incoming email... so, for example, this email is only reaching you 
> because the pre-existing inbound routes are grandfathered in, and continue to 
> work (whew!); however, I can no longer add new @twistedmatrix.com email 
> addresses nor change the configuration of existing ones.
> 
> Now, for this particular issue I've opened a ticket with Mailgun and 
> hopefully they'll be kind enough to extend their support to us yet again, but 
> in the longer term, it feels like we might want to go back to hosting our own 
> thing; ideally a thing that allows us to dogfood Twisted and maybe learn 
> interesting things about our SMTP support.

Update: the severity on this issue is back down to "low", as Mailgun's 
excellent support has re-enabled our inbound email processing for the 
forseeable future.  Thanks Mailgun!

Even so, though, I think it would be helpful to build a thing like what I 
described to dogfood our email code in a more realistic environment once again.

-g


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Request for new Twisted release?

[Glyph]

> On Oct 3, 2020, at 4:15 PM, Craig Rodrigues  wrote:
> 
> 
> 
> On Wed, Sep 23, 2020 at 10:11 AM Glyph  <mailto:gl...@twistedmatrix.com>> wrote:
> 
> 
>> On Sep 22, 2020, at 3:42 PM, Craig Rodrigues > <mailto:rodr...@crodrigues.org>> wrote:
>> 
>> Amber,
>> 
>> Can we have a new Twisted release within the next two months, say in 
>> Nov./Dec. timeframe, or
>> sooner if you'd like?
> 
> Given that Amber hasn't been super responsive lately, and that I firmly 
> believe it should be possible to take breaks from open source (particularly 
> amidst the, you know, global collapse of civilization and stuff), I will take 
> the opportunity point out yet again that 
> http://twisted.readthedocs.org/en/latest/core/development/policy/release-process.html
>  
> <http://twisted.readthedocs.org/en/latest/core/development/policy/release-process.html>
>  documents the release process and any sufficiently-motivated volunteer (with 
> at least a bit of history & trust within the community) can step forward and 
> receive all the permissions necessary to do all of these steps.
> 
> I will personally volunteer to fill in the permissions gaps for any 
> interested parties, although I don't have much bandwidth for the release 
> itself.
> 
> Amber, if you want to do it, go ahead and get started, I'm just saying we 
> don't need to put this on your shoulders :).
> 
> 
> I haven't heard any response from Amber, so I assume that Amber is busy (and 
> I hope is OK during all the pandemic).

I have heard from Amber occasionally and I believe she's healthy and safe, but 
if she wants to share more than that it's her own business :).  (Are any of us 
really "OK"?)

> If it is OK with Glyph and Amber, I can focus on pushing out the next Twisted 
> release, and hopefully get help from Adi.

Absolutely OK with me.  More than OK.  Please do this!  I can't wait until we 
have the level of automation you've been working on for anybody to cut the 
release.  I take it this means you'll officially be the release manager for 
20.10.0?

Let me know if you need any more permission bits to make progress.

> I'm working now with Adi to automate more of the release actions via GitHub 
> Actions, and have gotten pretty far,
> such as being able to build wheels of Twisted on GitHub Actions for Linux, 
> MacOS, and Windows.

This is fantastic news!  I really appreciate all the effort you're putting in 
to simplify and streamline the CI situation.

-g

> --
> Craig
> 
>  
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Mailgun Email Service Replacement Proposal

[Glyph]

Hello Twistors,

As many of you know, Twisted's email infrastructure for the past few years has 
been generously provided by Mailgun.  This was a huge upgrade to our sender 
reputation over running our own Exim instance at the time, and I'm tremendously 
grateful for all the effort it's saved the project.

However, Mailgun has never been a super close fit for what we actually want as 
a project (see for example https://github.com/glyph/mg2dsn 
<https://github.com/glyph/mg2dsn> which papers over some of the differences 
between what they provide and what we actually want), and over the years there 
have been several hiccups in service as Mailgun tweaks its pricing and their 
plan-upgrade machinery doesn't know what to do with our discounted account.

Right now, I'm dealing with a minor issue where our "plan" (which, again, 
Mailgun has generously given us for free) no longer includes the ability to 
process incoming email... so, for example, this email is only reaching you 
because the pre-existing inbound routes are grandfathered in, and continue to 
work (whew!); however, I can no longer add new @twistedmatrix.com email 
addresses nor change the configuration of existing ones.

Now, for this particular issue I've opened a ticket with Mailgun and hopefully 
they'll be kind enough to extend their support to us yet again, but in the 
longer term, it feels like we might want to go back to hosting our own thing; 
ideally a thing that allows us to dogfood Twisted and maybe learn interesting 
things about our SMTP support.

What we need is not trivial, but it's also not too complicated.  Critically we 
do not need to host a full-featured mail storage service, as we only do 
forwarding to and from other services.  So what we need is a thing that can:

listen on TLS ports with a certificate
generate an RSA key for DKIM
spit out something we can plug into our twisted.names configuration instead of 
this - 
https://github.com/twisted-infra/braid/blob/1df63c5d8b44e079487be2f0bf099108a77872e5/services/t-names/zones/twistedmatrix.com#L58-L63
 
<https://github.com/twisted-infra/braid/blob/1df63c5d8b44e079487be2f0bf099108a77872e5/services/t-names/zones/twistedmatrix.com#L58-L63>
distinguish between inbound-forwarded and outbound-sent email
authenticate users to send for particular addresses (i.e. set "From" and 
"Sender" headers, and confirm consistency with MAIL FROM & auth, reject if 
anything doesn't match; with a caveat for messages forwarded via mailman); sign 
these outbound messages with https://pypi.org/project/dkimpy/ 
<https://pypi.org/project/dkimpy/>
forward inbound messages which have DMARC alignment
maybe run spambayes and junk stuff that's obviously spammy before forwarding if 
we want to get fancy
DKIM sign messages on their way through
take over port 25, somehow talk to mailman (either via talking to Debian's Exim 
on some alternate port or by running mailman's receipt scripts itself).

Basically, a signing / authenticating MX relay.

Anyone interested in attempting to write such a thing with Twisted? :)

-glyph___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python