[Twisted-Python] Announcing the migration to the Python Org Mailing Lists hosting
Hi, This is a quick message to let you know that the current mailing list service will migrate from the Twisted own hosted services to the mailing list service provided by the Python organization. The migration is scheduled to start on the 5th of August, 2021 17:00 UTC time (10:00am PDT / 18:00 BST) The following email address will no longer be available: twis...@twistedmatrix.com The new address will be: twis...@python.org The new archive will be hosted at the following link that will be available after the migration: https://mail.python.org/archives/list/twis...@python.org/ All existing subscribers will be migrated. If you only interact with the mailing list using your email you only need to update your existing filters and update your email contact address for the list. The migration is expected to take about 3 hours. After the migration, Glyph plans to implement an email redirection, so even if you send emails to the old address we should still receive them. Subject to availability :) We also plan to redirect (via HTTP 308) the old mailing list pages and archives to the new pages. This will also happen after the migration. Have a nice day... and talk to you soon on the new mailing list. PS: while we do the migration you can still get in touch using IRC on Libera #twisted channel Matrix/Gitter at https://gitter.im/twisted/twisted -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] bad news about twistedmatrix.com's hosting
is the documentation that I am aware of: > https://twistedmatrix.com/trac/wiki/Infrastructure There is not much documentation... we need to rely on memories :) So don't bother to search for documentation, and just ask :) > > Particularly braid: https://github.com/twisted-infra/braid > > Yes. braid is our deployment as code thing... is the best we have in terms of documenting the infrastructure. "braid" is (was?) the Twisted customization on top of fabric for doing primitive automated deployment. > Is there anything else to look at? Something known to not be > supported in braid / only partially supported? > The plan is to stop using braid and do all the deployment via images and containers. > Would you rather schedule some online voice session where I poke > your brain at what needs to be done (IIRC something similar was > done about 5 years ago) and take notes that can be used (also by > someone else) while you let your mind flow happily? > We are split between Europe time zone (UTC -> UTC+2) and US ( something like UTC-6.. but also UTC-7) For now, we have done only ad-hoc meetings. > > > If you've been wanting to just use Github Issues and don't care > > about preserving any of the data in Trac, congratulations, > > you're about to win that argument by default ;-). > > > > In the worst case scenario, I will download a backup image > > before things get turned off in case someone wants to deal with > > this > > > I personally hope this doesn't happen; particularly the mailing > list is a great low-bandwidth way of staying mostly up-to-date > with the project. > I think that the plan is to migrate the mailing list to the Python Org mailing list hosting. We will need some help with setting up archive redirections. --- I think the current problem is finding or deciding for the next hosting / VM / container provider. We have 150 USD per month from MS to spend on Azure, but this is renewed any year without any guarantee that next year we will receive the free credit. The subscription was requested by Python Software Foundation and was granted by MS, but it was associated to the Azure AD of my organization. I have no idea how to create an Azure AD for Twisted organization, so that I can transfer the subscription. So...if anyone has any experience with Azure AD, it would help to get this thing sorted. For now, using Azure is plan B... so if we don't manage to migrate the services somewhere else, we (I) can just create an Ubuntu VM in Azure and migrate the existing dornkirk services there. The DNS was already migrated by Glyph. Thanks Glyph :) The mailing lists will be migrated soon to Python Org... we only have to see how to do it in a seamless way. So Evilham, if you want to help, try to see if you can hang out over IRC or Gitter and we can see how you can help. Thanks! -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] bad news about twistedmatrix.com's hosting
On Sun, 1 Aug 2021 at 08:42, Glyph wrote: > Hello Twistors, > > Our venerable hosting provider, tummy.com, will be terminating their > business operations on September 30, 2021. > > >>> datetime.date(2021, 9, 30) - datetime.date.today() > datetime.timedelta(days=60) > > > In addition to being a bit personally sad - Tummy has been an avid > supporter of the Twisted community since the very beginning - this means we > have just about 60 days to fully migrate our website, mailing list, and > DNS's hosting from Dornkirk, the machine where it's been running > continuously since 2013, to ... something else. > > There are 2 problems here: > > >1. We need a sponsor to help us find a new hosting or cloud provider >where things can be hosted. We do pay tummy currently, and we can probably >ask the SFC to pay a different hosting provider a similar price for >equivalent services, but ideally we'd find someone willing to donate >something so as not to put a drain on those resources. > >2. I will need significant help with the operational aspects of the >migration. I typically do a bunch of behind-the-scenes system >administration for twistedmatrix.com to keep the whole thing clunking >along, but after my brain has been fried by a year and a half's worth of >pandemic stress, I simply don't have the time, energy, or mental capacity >to do this. I know I've asked for help before and rarely received any, but >if I have to do *this* entirely by myself, the most likely outcome is >that I'll migrate DNS to some free cloud thing so at least my email address >doesn't break (I migrated SMTP, and thereby everything associated with >personal email addresses to Mailgun some years ago, so Tummy is not in that >loop for that), and the website (and this mailing list) will simply stop >working sometime between now and October. So if you'd like to keep >www.twistedmatrix.com, speed.twistedmatrix.com, our IRC bot (which is >offline anyway right now due to Freenode shutting down), >https://twistedmatrix.com/highscores/, this mailing list, or our >dogfooding instance for Twisted Names DNS, please volunteer so we can start >to coordinate. > > > If you've been wanting to just use Github Issues and don't care about > preserving any of the data in Trac, congratulations, you're about to win > that argument by default ;-). > > In the worst case scenario, I will download a backup image before things > get turned off in case someone wants to deal with this > > Hi Glyph Thanks for sending the message. This is bad, but I am confident that we can do the migration in 60 days, as long as we don't expect to have the exact same Twisted infrastructure. We have a monthly credit of 150 USD from MS to spend on Azure services. A quick hack would be to create a new VM in Azure with Ubuntu 18.04 and migrate the existing server to it. Do we have other servers hosted with tummy.com, other than dornkirk.twistedmatrix.com ? I will not have too much time to do the migration, but I can ask a sysadmin from my work to help with that and my company will cover that costs. I think that for now, we should see who is interested with helping with the migration and after 1 week start to work with that team. Maybe we will need to schedule an online meeting, but I hope we can do async work via GitHub Issues (braid) and IRC/Gitter. I saw that you have already considered using containers for Twisted infra https://github.com/twisted-infra/braid/issues/265 In parallel with migrating dornkirk.twistedmatrix.com, we can look at outsourcing some of our services. For example * Move mailing list to the main Python mailing list server * Move DNS to a cloud provider (Azure DNS) * Move Trac Wiki to GitHub Wiki * Move Trac Tickets to GitHub Issues * Move static file hosting to GitHub pages * Move highscore to a container and update it to pull info from GitHub hooks * IRC bot to some sort of integration to Gitter and then IRC bridge * Retire speed tests and buildbot -- >From past experience, I would say that most active Twisted contributors are software developers and not sysadmin. So we don't have the resources to keep custom Twisted infrastructure. And with the limited resources that we have, I would like to see Twisted contributors work at doing code review, fixing bugs rather than doing infrastructure work. So I am +1 for outsourcing as many Twisted services as possible. I think that we have already outsourced a few services: * Automated testing to Azure Pipelines and GitHub Actions * Git Hosting to GitHub * Some Twisted sub-projects use GitHub Issues * Python package downloads to PyPi * Documentation hosting to Read The Docs PS: for quick text messages if you don't like IRC, we are experimenting with Gitter at
[Twisted-Python] Twisted GitHub Organization 3rd Party access
Hi, FYI. I have revoked access to Twisted org for the following applications * Travis CI for Open Source * Landscape.io * bintray * Travis CI You can review the changes at: https://github.com/organizations/twisted/settings/oauth_application_policy -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Twisted 21.7.0 Final Release Announcement
On behalf of the Twisted contributors I announce the final release of Twisted 21.7.0 This is mostly a bugfix release. *Python 3.5 is no longer a supported platform.* *The minimum supported platform is Python 3.6.7.* The notable features are: - Python 3.10 beta is now a supported platform and should be ready for the final 3.10 release. - twisted.web.template.renderElement() now accepts any IRequest implementer instead of only twisted.web.server.Request. Add type hints to twisted.web.template. (#10184) *- *Type hinting was added to twisted.internet.defer, making this the first release of Twisted where you might reasonably be able to use mypy without your own custom stub files. (#10017) The full release notes are available at https://github.com/twisted/twisted/releases/tag/twisted-21.7.0 Documentation is available at https://docs.twistedmatrix.com/en/stable/ Wheels for the release candidate are available on PyPI https://pypi.org/project/Twisted/21.7.0/ python -m pip install Twisted==21.7.0 Many thanks to everyone who had a part in Twisted - the supporters of the Twisted Software Foundation, the developers, and all the people testing and building great things with Twisted! Enjoy the release! -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement
On Mon, 26 Jul 2021 at 06:57, Adi Roiban wrote: > Hi > > Just a quick update. > The release is on hold as I am waiting for Thomas to check a possible > regression. > > I don't know when the results from the investigation will be available. > I hope for the end of today. > > Will be back with more details. > Cheers > > On Fri, 23 Jul 2021, 10:55 Adi Roiban, wrote: > >> Hi, >> >> Thanks for all of you who have sent comments and fixes for RC2. >> >> The third release candidate is available for download and testing >> >> https://pypi.org/project/Twisted/21.7.0rc3/#files >> >> https://github.com/twisted/twisted/releases/tag/twisted-21.7.0rc3 >> >> This contains 2 changes: >> >> The wheel file is now published to PyPi. >> >> >> The following revert was done: >> >> * The ticket in which the regression was reported >> https://twistedmatrix.com/trac/ticket/10235 >> * Reverted ticket https://twistedmatrix.com/trac/ticket/10155 >> * Reverted PR https://github.com/twisted/twisted/pull/1570 >> >> -- >> >> You can send your comments over email or over GitHub >> >> https://github.com/twisted/twisted/pull/1614 >> >> I plan to do the final release by the end of Sunday UTC time. >> >> Regards >> >> On Wed, 21 Jul 2021 at 19:07, John Santos wrote: >> >>> Hi all, >>> >>> I just finally got around to installing and testing the new RC with our >>> primary >>> Twisted App. Everything seems to work fine. >>> >>> I continue to be impressed by what a good job Twisted does with >>> backwards >>> compatibility! >>> >>> -- John Santos >>> >>> >>> On 7/20/2021 7:23 PM, Adi Roiban wrote: >>> > >>> > >>> > On Sun, 18 Jul 2021 at 23:42, Richard van der Hoff >> > <mailto:rich...@matrix.org>> wrote: >>> > >>> > On 16/07/2021 20:27, Glyph wrote: >>> > > >>> > >> One particular problem I came across was the type annotation on >>> > >> inlineCallbacks. I've filed >>> > >> https://twistedmatrix.com/trac/ticket/10231 >>> > <https://twistedmatrix.com/trac/ticket/10231> about it - would >>> > >> appreciate thoughts. >>> > >> >>> > > This definitely looks wrong; there should be a TypeVar in >>> there. Adi, >>> > > I'd go so far as to say that this should be a release blocker, >>> > > although the change should be fairly minimal. >>> > > >>> > > Richard, could you please make a proper PR for this to get CI >>> kicked >>> > > off and make sure the new annotation doesn't cause any failures? >>> > Done, in the form of https://github.com/twisted/twisted/pull/1632 >>> > <https://github.com/twisted/twisted/pull/1632>. >>> > >>> > Hi >>> > >>> > Thanks to Thomas and Richard 21.7.0 RC2 was released. >>> > >>> > Release notes >>> > https://github.com/twisted/twisted/releases/tag/twisted-21.7.0rc2 >>> > <https://github.com/twisted/twisted/releases/tag/twisted-21.7.0rc2> >>> > >>> > PyPy page >>> > https://pypi.org/project/Twisted/21.7.0rc2/ >>> > <https://pypi.org/project/Twisted/21.7.0rc2/> >>> > >>> > Please test and send any issue over email or GitHub >>> > Pull request at https://github.com/twisted/twisted/pull/1614 >>> > <https://github.com/twisted/twisted/pull/1614> >>> > >>> > If all is ok, I plan to do the final release by the end of Friday or >>> during the >>> > weekend. >>> > >>> >> Hi, The final release is ready for review https://github.com/twisted/twisted/pull/1614 Please take a look and leave your comments over GitHub. Once the PR is approved I will create the final release. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement
Hi Just a quick update. The release is on hold as I am waiting for Thomas to check a possible regression. I don't know when the results from the investigation will be available. I hope for the end of today. Will be back with more details. Cheers On Fri, 23 Jul 2021, 10:55 Adi Roiban, wrote: > Hi, > > Thanks for all of you who have sent comments and fixes for RC2. > > The third release candidate is available for download and testing > > https://pypi.org/project/Twisted/21.7.0rc3/#files > > https://github.com/twisted/twisted/releases/tag/twisted-21.7.0rc3 > > This contains 2 changes: > > The wheel file is now published to PyPi. > > > The following revert was done: > > * The ticket in which the regression was reported > https://twistedmatrix.com/trac/ticket/10235 > * Reverted ticket https://twistedmatrix.com/trac/ticket/10155 > * Reverted PR https://github.com/twisted/twisted/pull/1570 > > -- > > You can send your comments over email or over GitHub > > https://github.com/twisted/twisted/pull/1614 > > I plan to do the final release by the end of Sunday UTC time. > > Regards > > On Wed, 21 Jul 2021 at 19:07, John Santos wrote: > >> Hi all, >> >> I just finally got around to installing and testing the new RC with our >> primary >> Twisted App. Everything seems to work fine. >> >> I continue to be impressed by what a good job Twisted does with backwards >> compatibility! >> >> -- John Santos >> >> >> On 7/20/2021 7:23 PM, Adi Roiban wrote: >> > >> > >> > On Sun, 18 Jul 2021 at 23:42, Richard van der Hoff > > <mailto:rich...@matrix.org>> wrote: >> > >> > On 16/07/2021 20:27, Glyph wrote: >> > > >> > >> One particular problem I came across was the type annotation on >> > >> inlineCallbacks. I've filed >> > >> https://twistedmatrix.com/trac/ticket/10231 >> > <https://twistedmatrix.com/trac/ticket/10231> about it - would >> > >> appreciate thoughts. >> > >> >> > > This definitely looks wrong; there should be a TypeVar in >> there. Adi, >> > > I'd go so far as to say that this should be a release blocker, >> > > although the change should be fairly minimal. >> > > >> > > Richard, could you please make a proper PR for this to get CI >> kicked >> > > off and make sure the new annotation doesn't cause any failures? >> > Done, in the form of https://github.com/twisted/twisted/pull/1632 >> > <https://github.com/twisted/twisted/pull/1632>. >> > >> > Hi >> > >> > Thanks to Thomas and Richard 21.7.0 RC2 was released. >> > >> > Release notes >> > https://github.com/twisted/twisted/releases/tag/twisted-21.7.0rc2 >> > <https://github.com/twisted/twisted/releases/tag/twisted-21.7.0rc2> >> > >> > PyPy page >> > https://pypi.org/project/Twisted/21.7.0rc2/ >> > <https://pypi.org/project/Twisted/21.7.0rc2/> >> > >> > Please test and send any issue over email or GitHub >> > Pull request at https://github.com/twisted/twisted/pull/1614 >> > <https://github.com/twisted/twisted/pull/1614> >> > >> > If all is ok, I plan to do the final release by the end of Friday or >> during the >> > weekend. >> > >> > -- >> > Adi Roiban >> > >> > ___ >> > Twisted-Python mailing list >> > Twisted-Python@twistedmatrix.com >> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> > >> >> -- >> John Santos >> Evans Griffiths & Hart, Inc. >> 781-861-0670 ext 539 >> >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> > > > -- > Adi Roiban > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement
Hi, Thanks for all of you who have sent comments and fixes for RC2. The third release candidate is available for download and testing https://pypi.org/project/Twisted/21.7.0rc3/#files https://github.com/twisted/twisted/releases/tag/twisted-21.7.0rc3 This contains 2 changes: The wheel file is now published to PyPi. The following revert was done: * The ticket in which the regression was reported https://twistedmatrix.com/trac/ticket/10235 * Reverted ticket https://twistedmatrix.com/trac/ticket/10155 * Reverted PR https://github.com/twisted/twisted/pull/1570 -- You can send your comments over email or over GitHub https://github.com/twisted/twisted/pull/1614 I plan to do the final release by the end of Sunday UTC time. Regards On Wed, 21 Jul 2021 at 19:07, John Santos wrote: > Hi all, > > I just finally got around to installing and testing the new RC with our > primary > Twisted App. Everything seems to work fine. > > I continue to be impressed by what a good job Twisted does with backwards > compatibility! > > -- John Santos > > > On 7/20/2021 7:23 PM, Adi Roiban wrote: > > > > > > On Sun, 18 Jul 2021 at 23:42, Richard van der Hoff > <mailto:rich...@matrix.org>> wrote: > > > > On 16/07/2021 20:27, Glyph wrote: > > > > > >> One particular problem I came across was the type annotation on > > >> inlineCallbacks. I've filed > > >> https://twistedmatrix.com/trac/ticket/10231 > > <https://twistedmatrix.com/trac/ticket/10231> about it - would > > >> appreciate thoughts. > > >> > > > This definitely looks wrong; there should be a TypeVar in there. > Adi, > > > I'd go so far as to say that this should be a release blocker, > > > although the change should be fairly minimal. > > > > > > Richard, could you please make a proper PR for this to get CI > kicked > > > off and make sure the new annotation doesn't cause any failures? > > Done, in the form of https://github.com/twisted/twisted/pull/1632 > > <https://github.com/twisted/twisted/pull/1632>. > > > > Hi > > > > Thanks to Thomas and Richard 21.7.0 RC2 was released. > > > > Release notes > > https://github.com/twisted/twisted/releases/tag/twisted-21.7.0rc2 > > <https://github.com/twisted/twisted/releases/tag/twisted-21.7.0rc2> > > > > PyPy page > > https://pypi.org/project/Twisted/21.7.0rc2/ > > <https://pypi.org/project/Twisted/21.7.0rc2/> > > > > Please test and send any issue over email or GitHub > > Pull request at https://github.com/twisted/twisted/pull/1614 > > <https://github.com/twisted/twisted/pull/1614> > > > > If all is ok, I plan to do the final release by the end of Friday or > during the > > weekend. > > > > -- > > Adi Roiban > > > > ___ > > Twisted-Python mailing list > > Twisted-Python@twistedmatrix.com > > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > -- > John Santos > Evans Griffiths & Hart, Inc. > 781-861-0670 ext 539 > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] reviews
Hi Thanks Barry for reporting the issue. It should be back online. The twisted-web process was down. I have started it. On Tue, 20 Jul 2021 at 10:56, Barry Scott wrote: > On Tuesday, 20 July 2021 05:38:23 BST Glyph wrote: > > Hello Twisted friends, > > > > This is just a friendly reminder that https://twisted.reviews < > https://twisted.reviews/> is full to bursting - 34 open reviews right > now, some dating back as far as May of last year. If you've got a few free > minutes, can you find a ticket on that list and review the associated PR? > (I recently fixed the report so it's actually in the order you should > review them...) A short review queue is a happy review queue. > > > > $ curl https://twistedmatrix.com/trac/report/25 > curl: (7) Failed to connect to twistedmatrix.com port 443: Connection > refused > > > Remember also that this isn't for nothing, if you do code reviews on > Twisted you get internet points and bragging rights, at > https://twistedmatrix.com/highscores/ < > https://twistedmatrix.com/highscores/>. > > > > This isn't just for project members; if you did not already know, the > rule is that if you're a project member you can review anything, but if > you're an external contributor, you can review submissions by project > members (and it's up to them to decide if your review is adequate before > acting on it). Project members (should) have a "*" next to their name in > the "submitted by" list. (If they don't I think someone needs to go > manually update a list of strings, feel free to report issues if that's > wrong.) > > > > Happy hacking, > > > > -glyph > > > > > > > > > _______ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Enable auto-merge for twisted/twisted
On Sun, 18 Jul 2021 at 01:04, Glyph wrote: > Nope, I'm fully in favor. When we're done reviewing, and waiting for CI, > reviewers should be able to set auto-merge and walk away. Thank you for > raising the issue! > I have enabled auto-merge for twisted/twisted Let me know if you want to be enabled for other repos. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement
On Fri, 16 Jul 2021 at 20:27, Glyph wrote: > > > On Jul 16, 2021, at 2:20 AM, Richard van der Hoff > wrote: > > On 16/07/2021 00:18, Glyph wrote: > > > > On Jul 15, 2021, at 9:00 AM, Richard van der Hoff > wrote: > > We can't just go and add type annotations because we need to maintain > compatibility with older Twisted (to make it possible to package in Debian > et al). > > Any suggestions for keeping mypy happy? > > Are you saying you need it to typecheck against older versions or just run > against them? > > Ah, this gave me the clue I needed. We just need to run against them. > Which means I can put type hints in comments, where they will be ignored at > runtime. It's fiddly, but it will work well enough. > > You can also do 'from __future__ import annotations' to avoid the > annotations getting evaluated, which might be slightly less awkward. > > Thanks Glyph, and thanks to Adi and Barry for your suggestions too. > > One particular problem I came across was the type annotation on > inlineCallbacks. I've filed https://twistedmatrix.com/trac/ticket/10231 > about it - would appreciate thoughts. > > This definitely looks wrong; there should be a TypeVar in there. Adi, I'd > go so far as to say that this should be a release blocker, although the > change should be fairly minimal. > > Richard, could you please make a proper PR for this to get CI kicked off > and make sure the new annotation doesn't cause any failures? > > I think that I will do a RC2 release candidate, since due to this issue the GHA tests on RC1 are red - https://github.com/twisted/twisted/pull/1628 So I plan to cherry-pick those changes and do a RC2. Regarding the release blocker, I will block the release for any ticket from https://twistedmatrix.com/trac/report/26 Right now, I have added https://twistedmatrix.com/trac/ticket/10231 to that report. So, if anyone knows how to fix https://twistedmatrix.com/trac/ticket/10231 please help to unblock the release. Or if you think that https://twistedmatrix.com/trac/ticket/10231 should not be a release blocker, please add your comments. For now, I will delay the release of RC2 to wait to see what is the resolution for #10231 Regards -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Enable pre-commit.ci for twisted/twisted
On Thu, 15 Jul 2021 at 15:36, Jean-Paul Calderone wrote: > On Thu, Jul 15, 2021 at 10:26 AM Adi Roiban wrote: > >> Hi, >> >> As part of this PR https://github.com/twisted/twisted/pull/1619 Thomas >> has requested to enable the https://pre-commit.ci service for >> twisted/twisted repository >> >> The pre-commit.ci is kind of an auto-pilot for our pre-commit >> configuration. >> >> We already do post-commit checks via the pre-commit framework and I think >> that some of the core developers have pre-commit enabled as a git hook. >> >> The convenience if pre-commit.ci is that if you don't have the git-hook >> enable and you commit dirty code, pre-commit.ci will fix it for you. >> >> pre-commit.ci will also automatically create PR to update the pre-commit >> dependencies. >> >> This all comes at the cost of allowing the external pre-commit.ci server >> to write to twisted/twisted repository. >> >> pre-commit.ci was already enabled for twisted/ldaptor >> >> pre-commit.ci is currently enabled for twisted/twisted for testing while >> a final decision is made. >> >> I am +0 on this change due to security reasons...but I do think that it >> will reduce a bit of the frustration for first time contributors. >> > > Can you elaborate on this a little bit? First time contributors won't be > able to push a branch to twisted/twisted. Does giving pre-commit.ci > write access to twisted/twisted also give it write access to forks of > twisted/twisted? Or maybe only to branches in forks with PRs targeting > twisted/twisted? > > External contributors can't push directly to twisted/twisted. Their code will reach twisted/twisted vi PR... just like before I expect that first time contributors will have their fork automatically updated by pre-commit.ci to make the code more likely to be accepted. For example, pre-commit will automatically run black. I don't have any experience with pre-commit.ci ... but this is my expectation, > >> If there are no strong comments against enabling pre-commit.ci, in a >> week, I plan to approve the PR and have the pre-commit.ci enabled for >> the long term. >> > > Thanks for raising this question with the mailing list and for supplying > details about the motivation. Given that there is no justification for the > change written on the ticket or in the PR itself, this is really helpful. > Based on the current feedback, I will approve the PR. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement
On Thu, 15 Jul 2021 at 17:02, Richard van der Hoff wrote: > On 11/07/2021 00:49, Glyph wrote: > > Thank you Adi! Very glad to see us collectively getting back on the > release-management horse again! And it was super encouraging to see how > quickly you were able to get this out, with all the automation that > everyone (not least of all yourself) has been building to make this process > faster and simpler. > > To echo this: thank you for your work, Adi! > > It appears we've buried the lede on one of our biggest features in this > release though - https://github.com/twisted/twisted/pull/1448/files had > no newsfile that shows up in the changelog, but it properly made Deferred > into a generic, so I think this is the first release of Twisted where you > might *reasonably* be able to use mypy without your own custom stub > files! :) > > This looks very cool, but presents us with a problem. > > mypy now complains about instantiations of Deferred without a type > annotation: > > > d = defer.Deferred() > > results in an error: > > > synapse/util/async_helpers.py:124: error: Need type annotation for 'd' > [var-annotated] > > We can't just go and add type annotations because we need to maintain > compatibility with older Twisted (to make it possible to package in Debian > et al). > > Any suggestions for keeping mypy happy? > I also don't have much experience with mypy, but I guess that you can ignore the errors from twisted.internet.defer via mypy.ini [mypy-twisted.internet.defer] allow_untyped_defs = True -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Enable pre-commit.ci for twisted/twisted
Hi, As part of this PR https://github.com/twisted/twisted/pull/1619 Thomas has requested to enable the https://pre-commit.ci service for twisted/twisted repository The pre-commit.ci is kind of an auto-pilot for our pre-commit configuration. We already do post-commit checks via the pre-commit framework and I think that some of the core developers have pre-commit enabled as a git hook. The convenience if pre-commit.ci is that if you don't have the git-hook enable and you commit dirty code, pre-commit.ci will fix it for you. pre-commit.ci will also automatically create PR to update the pre-commit dependencies. This all comes at the cost of allowing the external pre-commit.ci server to write to twisted/twisted repository. pre-commit.ci was already enabled for twisted/ldaptor pre-commit.ci is currently enabled for twisted/twisted for testing while a final decision is made. I am +0 on this change due to security reasons...but I do think that it will reduce a bit of the frustration for first time contributors. If there are no strong comments against enabling pre-commit.ci, in a week, I plan to approve the PR and have the pre-commit.ci enabled for the long term. We can always rewoke to pre-commit.ci write access if needed. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted 21.7.0 Pre-Release Announcement
On Sun, 11 Jul 2021 at 00:49, Glyph wrote: > Thank you Adi! Very glad to see us collectively getting back on the > release-management horse again! And it was super encouraging to see how > quickly you were able to get this out, with all the automation that > everyone (not least of all yourself) has been building to make this process > faster and simpler. > > It appears we've buried the lede on one of our biggest features in this > release though - https://github.com/twisted/twisted/pull/1448/files had > no newsfile that shows up in the changelog, but it properly made Deferred > into a generic, so I think this is the first release of Twisted where you > might *reasonably* be able to use mypy without your own custom stub > files! :) > Thanks Glyph and Thomas and Tom for the release notes feedback. I was thinking that 21.7.0 is ready for the final review. PR at https://github.com/twisted/twisted/pull/1614 I am doing some fast-tracking here (as I now have some time) But while writing this email, the python 3.8 and 3.9 version on GitHub VMs were updates and we now have broken tests in trunk and in the release branch The release blocking ticket was created at https://twistedmatrix.com/trac/ticket/10230 Please see if you can help with unlocking the release. Thomas is already helping. Either fix the issue, or decide if it's ok to release with this issue. There was also another minor idna pypy update release blocker that was already fixed by Thomas. I was planning to push the final release on Saturday 17 of July, to allow for a full 1 week of feedback for the release candidate. - There is also a PR for updating the release docs - https://github.com/twisted/twisted/pull/1616 Regards -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Adding Tom Most's SSH key to root on dornkirk.twistedmatrix.com
Hi team, I would like to give Tom Most SSH access to dornkirk.twistedmatrix.com to help with Twisted web / trac services. Anyone over here think that is a bad idea? Thanks! -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Fwd: Enable auto-merge for twisted/twisted
Hi, Do you see any issues if we enable the auto-merge feature for twisted/twisted? The feature is documented here https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/automatically-merging-a-pull-request With this, in theory you don't have to wait for all the CI to be green before a merge. You got your reviews, you fix some typos that don't require a re-review and then set the merge message and your are done. Or there was a flaky test failure and just re-run the job and the merge will be done once all the jobs are green again. - This can be abused if someone will enable auto-merge before the PR is approved by a reviewer. As most of the time the reviewer will accept the PR but will request or suggest minor changes. If the auto-merge is enabled, the PR is mergesd as soon as it is reviewed. - The config looks like this: You can allow setting pull requests to merge automatically once all required reviews and status checks have passed. - Allow auto-merge Waits for merge requirements to be met and then merges automatically. There is also the option to block a merge only after all the conversations are marked as resolved. I am using this for my project and it works ok... a bit annoying but can help if you accedentally forgot to address a comment. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Twisted-Infra org - twm added as maintainer
Hi, I tried to send this email 9 days ago... but I was not subscribed to the list using this email... and have not seen the bounce message... if any. -- I just wanted to let you know that I have added Tom Most as a "core" team member to the twisted-infra organization. This is more of an information. Let me know if I did something wrong I don't know if there is any process in place for adding new members to twisted-infra org. Tom can merge the code but without access to the actual production server there is not much he can do. I think that we can give him access to the servers to help with the infrastructure. I have not added the keys yet, but if any other member of the team think that this is a good idea please check with Tom what are the preferred key/keys to be added to ~/.ssh/authorized_keys Tom's SSH Keys https://api.github.com/users/twm/keys I can also add the keys once I get some feedback. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] twisted.web HTTPS client certificate
On Mon, 12 Jul 2021 at 08:09, Ian Haywood wrote: > I am trying to work out how to retrieve on the server a X.509 > certificate presented by the HTTPS client. This code tries to tell me > the transport has no peer certificate. > > same error when I use wget as a client so I think my problem is in the > server code. I'm using self-signed certificates > > Any clues as to what I'm doing wrong? > > Ian > > As commented by Glyph you will first need to setup one or a list of accepted CAs. Then setup custom server-side context. Add the CA via https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context.load_verify_locations This is done via https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context.set_verify to define a path or dir or implement a custom one via https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context.get_cert_store Without set_verify, during the TLS/SSL handshake the server will not ask the client to send its own certificate. - I am using X509 authentication as a passwordless authentication for automated transfers, similar to the SSH key authentication. The x509 certificate authentication is used by the Spanish government across many of their services. Taxes, customs, health service ... -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Twisted 21.7.0 Pre-Release Announcement
On behalf of the Twisted contributors I announce the release candidate of Twisted 21.7.0 This is mostly a bugfix release. *Python 3.5 is no longer a supported platform.* *The minimum supported platform is Python 3.6.7.* The notable features are: - Python 3.10 beta is now a supported platform. - twisted.web.template.renderElement() now accepts any IRequest implementer instead of only twisted.web.server.Request. Add type hints to twisted.web.template. (#10184) The release and NEWS file is available for review at https://github.com/twisted/twisted/pull/1614/files Release candidate documentation is available at https://twisted--1614.org.readthedocs.build/en/1614/ Wheels for the release candidate are available on PyPI https://pypi.org/project/Twisted/21.7.0rc1/ python -m pip install Twisted==21.7.0rc1 Please test it and report any issues. If nothing comes up in 1 week, 21.7.0 will be released based on the latest release candidate. Many thanks to everyone who had a part in Twisted - the supporters of the Twisted Software Foundation, the developers, and all the people testing and building great things with Twisted! -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Release Planning
On Fri, 9 Jul 2021 at 08:34, Tom Most wrote: > > > Automating all this is a great project! Please feel free to ping me for > reviews. > > There is one blocker PR for the release as the newsfragment checks are failing on the release branch... since the code is executed on the to-be-merged on a branch named by GitHub Action something like refs/pull/1617/merge so the new fragment check fails to detect it as a release branch...as a release branch should have a name starting with 'release-' The PR is here. Any feedback will help to get this merge https://github.com/twisted/twisted/pull/1617 This PR is important as I want to have a green release branch. For the previous release, we just ignored the failure and did the release anyway But for this release I want it fixed :) This is also a non-blocker PR which contains a minor clarification for the release process https://github.com/twisted/twisted/pull/1616 -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Release Planning
On Tue, 6 Jul 2021 at 14:21, Adi Roiban wrote: > > > On Tue, 6 Jul 2021 at 01:55, Glyph wrote: > >> >> On July 5, 2021 at 3:55:38 PM, Adi Roiban (a...@roiban.ro) wrote: >> >>> Hi, >>> >>> I don't have much time or much "need" but if needed I can help with a >>> new Twisted release. >>> >> Hooray! I’m hype for 21.7! >> > I have created the release ticket and will create the release branch and > RC soon. > I am planning to create the release branch soon. I plan to do the release based on the updated documentation from https://github.com/twisted/twisted/pull/1616 That is, create the release branch, push the packages to PyPI and only after these steps as for a review for the PR and send the release candidate announcement. Let me know if the release pull request should have an approved review before pushing the release candidate files to PyPi. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Release Planning
On Tue, 6 Jul 2021 at 01:55, Glyph wrote: > > On July 5, 2021 at 3:55:38 PM, Adi Roiban (a...@roiban.ro) wrote: > >> Hi, >> >> I don't have much time or much "need" but if needed I can help with a new >> Twisted release. >> > Hooray! I’m hype for 21.7! > I have created the release ticket and will create the release branch and RC soon. There is now a PR with README update for freenode removal that is waiting to be reviewed. https://github.com/twisted/twisted/pull/1611 It would be nice to have it as part of the next release so that the PYPi page is updated, but I don't consider it a blocker. > >> I am still working on py3 migration as an urgent task so I am stuck with >> 20.3.0. >> >> There was a lot of confusion / miscommunication / non-cooperation lately >> with the Twisted release process. >> >> For example, I don't know when a new release is required / appropriate. >> >> Maybe we can look at doing 2 releases per year around December and June, >> or sync with Ubuntu releases? >> What do you think?. >> > My own preference is to just do them as often as we have capacity for. > More releases means peoples’ work gets into the world faster, there’s more > motivation to work on Twisted, more features means more users and more > users means more potential contributors and it’s a virtuous cycle. > > However since I personally have the bandwidth to do zero (0) releases per > year myself, this does mean that others get to dictate the schedule ;-). > If you’d like to do these two, other volunteers can do more (and more than > one planned release manager at a time would be great). If at all possible, > I think one per quarter, so four per year, would be a good lower bound to > aim for. (Among other things, a quarterly cycle makes it easier to align > with a popular style of resource planning in many companies.) > Agree that it would be nice to release as often as possible > >> My plan is to have the release process documented and automated so that >> anyone can take the role of the release manager for a certain release. >> > So say we all! Thanks for all your work towards making this happen so far. > >> >> The current documentation for the release is here >> >> https://docs.twistedmatrix.com/en/twisted-21.2.0/core/development/policy/release-process.html >> >> Happy to recessive feedback and PRs for the current documentation. >> >> >> >> I don't see any tickets flagged as regressions in Trac ( >> https://twistedmatrix.com/trac/report/26) so I guess that we can cut a >> release right away. >> >> -- >> >> I am thinking of a scenario in which person X sends a PR that is merged >> and that person X needs a new release so the same person X can act as >> release manager for that release. >> I don't know if that will work :) >> > Looking forward to crossing that bridge when we come to it. > > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] IRC still alive?
Hi, On Tue, 6 Jul 2021 at 01:48, Glyph wrote: > > On July 5, 2021 at 4:44:26 PM, Adi Roiban (a...@roiban.ro) wrote: > >> Hi >> >> On Tue, 6 Jul 2021 at 00:02, Tom Most wrote: >> >>> Hi Adi, >>> >>> IRC has moved to Libera.Chat <https://libera.chat/> due to the Freenode >>> implosion <https://lwn.net/Articles/857140/>. The channel names are the >>> same --- #twisted, #twisted-dev, #twisted.web. >>> >>> ---Tom >>> >>> >> Thanks for the info. >> >> I guess that the website can be updated to no longer mention freenode - >> https://twistedmatrix.com/trac/ >> > That, and kenaan needs to be reconfigured to point at the new network. > Please scrub any mention of freenode that you find from the docs & website. > I am updating wiki as no review is required. I have updated the wiki to point to #twisted-dev instead of #twisted-admin . I think that we no longer need a separate admin channel and Twisted admins can join #twisted-dev I will look at create a PR for the remaining doc and setup.cfg/pypi Or if anyone wants to do that, I am happy to review :) This looks like a good reason to trigger a new release. > >> Is there a web log? >> > Not yet, but feel free to set one up. > Does anyone know any software as a service IRC logger? Otherwise, I think that `kenaan` can be updated to record the logs to a simple TXT file one per day and server those files via HTTP. It will not be very useful but at least there is some sort of record that can be referenced from a ticket of a PR. > >> Cheers >> >> On Mon, Jul 5, 2021, at 3:45 PM, Adi Roiban wrote: >>> >>> Hi, >>> >>> Only now I had time to check the Twisted IRC channel and it looks like >>> the channels are no longer registered ... and the IRC bot is not recording >>> the logs. >>> >>> Should we continue to use IRC or move to gitter or something else that >>> has history and offline messages by default? >>> >>> Cheers >>> -- >>> Adi Roiban >>> ___ >>> Twisted-Python mailing list >>> Twisted-Python@twistedmatrix.com >>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >>> >>> >>> ___ >>> Twisted-Python mailing list >>> Twisted-Python@twistedmatrix.com >>> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >>> >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] IRC still alive?
Hi On Tue, 6 Jul 2021 at 00:02, Tom Most wrote: > Hi Adi, > > IRC has moved to Libera.Chat <https://libera.chat/> due to the Freenode > implosion <https://lwn.net/Articles/857140/>. The channel names are the > same --- #twisted, #twisted-dev, #twisted.web. > > ---Tom > > Thanks for the info. I guess that the website can be updated to no longer mention freenode - https://twistedmatrix.com/trac/ Is there a web log? Cheers On Mon, Jul 5, 2021, at 3:45 PM, Adi Roiban wrote: > > Hi, > > Only now I had time to check the Twisted IRC channel and it looks like the > channels are no longer registered ... and the IRC bot is not recording the > logs. > > Should we continue to use IRC or move to gitter or something else that has > history and offline messages by default? > > Cheers > -- > Adi Roiban > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Twisted Release Planning
Hi, I don't have much time or much "need" but if needed I can help with a new Twisted release. I am still working on py3 migration as an urgent task so I am stuck with 20.3.0. There was a lot of confusion / miscommunication / non-cooperation lately with the Twisted release process. For example, I don't know when a new release is required / appropriate. Maybe we can look at doing 2 releases per year around December and June, or sync with Ubuntu releases? What do you think?. My plan is to have the release process documented and automated so that anyone can take the role of the release manager for a certain release. The current documentation for the release is here https://docs.twistedmatrix.com/en/twisted-21.2.0/core/development/policy/release-process.html Happy to recessive feedback and PRs for the current documentation. I don't see any tickets flagged as regressions in Trac ( https://twistedmatrix.com/trac/report/26) so I guess that we can cut a release right away. -- I am thinking of a scenario in which person X sends a PR that is merged and that person X needs a new release so the same person X can act as release manager for that release. I don't know if that will work :) Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] IRC still alive?
Hi, Only now I had time to check the Twisted IRC channel and it looks like the channels are no longer registered ... and the IRC bot is not recording the logs. Should we continue to use IRC or move to gitter or something else that has history and offline messages by default? Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] How to get the client ip when it ask a resolution with twisted.names
Hi On Sat, 1 May 2021 at 07:43, cont...@benoit-laviale.fr < cont...@benoit-laviale.fr> wrote: > Dear all, > > I am working on twisted names since some days to make kind of filtering > system (with domains blacklists). > So I made my own resolver that can reply defer.fail(error.DomainError()) > to send the query to the next resolver (and in this case, it will be > resolve) or defer.fail(dns.AuthoritativeDomainError(query.name.name)) to > reply NXDOMAIN that can "block" the domain to the final client. > > That part is working well. > > Next, i would like to "block" the resolution regarding the device (Mac > address/IP/???). > > I would like something like that : > > if self.query.meta.client.ip == b"192.168.0.5" or > self.query.meta.client.mac == "XX:YY:ZZ:": > self.do_something() > > My use case is that device of Kelly, (Kelly is a teenager of 12 years) > cant use youtube.com at all, but other devices are allowed to get the > resolution. > How can I get the IP/Name/mac (an identifier) of the device that's > querying my custom dns? > I read something about datagrams, but in my case i think i must get this > information in my resolver... > > Thx and cheers. > I think you are using DNS over UDP... so to understand how UDP works with Twisted you can check this page https://twistedmatrix.com/documents/current/core/howto/udp.html I have never used Twisted names so I am just trying to help based on my general knowledge of Twisted design ... but I think the entry point is here https://github.com/twisted/twisted/blob/63649469c1fe46d8a713e8034239ac3cc0498ea7/src/twisted/names/dns.py#L3242 then it goes to here https://github.com/twisted/twisted/blob/63649469c1fe46d8a713e8034239ac3cc0498ea7/src/twisted/names/server.py#L538 and then in handleQuery where I see that the query is done without the `address`...but the address is available again in `gotResolverResponse`.. so maybe that help --- Hope it helps Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Does anyone know why trunk is failing CI on PyPy 7.3.4?
On Thu, 29 Apr 2021 at 07:23, Glyph wrote: > It's merged. > > Thanks for the merge. I will try to help Twisted by keeping the trunk green. If there is any other issue with the trunk branch getting red out of the blue, ping me and I will try to help. --- We now have a fix for Twisted, but I think that this is a bug in PYPY 7.3.4 FYI I have reported this bug upstream. https://foss.heptapod.net/pypy/pypy/-/issues/3451 Cheers > On Wed, Apr 28, 2021, at 3:07 AM, Adi Roiban wrote: > > Hi > > PR at https://github.com/twisted/twisted/pull/1592 > > I hope we can get a quick review and make the trunk green again :) > > Cheeers > > On Wed, 28 Apr 2021 at 10:30, Adi Roiban wrote: > > Hi, > > A quick diff to fix it... and a ticket > https://twistedmatrix.com/trac/ticket/10187 > > I will create a PR and we can refine the fix during the review. > > > diff --git a/src/twisted/words/protocols/irc.py > b/src/twisted/words/protocols/irc.py > index 75300019b..a97abce2e 100644 > --- a/src/twisted/words/protocols/irc.py > +++ b/src/twisted/words/protocols/irc.py > @@ -3678,10 +3678,10 @@ def ctcpExtract(message): > normal_messages.append(messages.pop(0)) > odd = not odd > > -extended_messages[:] = filter(None, extended_messages) > -normal_messages[:] = filter(None, normal_messages) > +extended_messages[:] = list(filter(None, extended_messages)) > +normal_messages[:] = list(filter(None, normal_messages)) > > -extended_messages[:] = map(ctcpDequote, extended_messages) > +extended_messages[:] = list(map(ctcpDequote, extended_messages)) > > > On Wed, 28 Apr 2021 at 10:01, Adi Roiban wrote: > > Hi Wim, > > Thanks for the report > > On Tue, 27 Apr 2021 at 08:14, Wim Lewis wrote: > > TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")? > > I don't have time right now to set up a PyPy-capable environment and try > to reproduce this, but perhaps someone does? > > Long version: > > Trunk has been broken since the last merge a week or so ago, but I > don't think the breakage is due to that merge. As an experiment I made > a PR based off the last successful version of trunk, with a whitespace > change, and it now fails CI as well. So I think the failure must be due > to some change that isn't in Twisted or controlled-for by tox. > > The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. > Unlike our usual CI problems it doesn't seem to be a random failure: it > fails all the time in the same place. But the place doesn't make sense > to me. It's in the IRC CTCP tests, and they fail in the same ways each > time (an expected response is not received). > > The pair of CI runs closest to the change are these: > > run 5793: https://github.com/twisted/twisted/runs/2328450554 > run 5809: https://github.com/twisted/twisted/runs/2360415474 > > There are a lot of differences, but sys.version went from 3.7.9 to > 3.7.10 between those runs, so that seems like the most likely culprit. > > Last working: > > sys.version : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52) > > [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] > > sys.prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 > > sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 > > sys.executable : > /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python > > First broken: > > sys.version : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34) > > [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] > > sys.prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 > > sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 > > sys.executable : > /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python > > PyPy's release notes for 7.3.4 don't list anything that jumps out at me: > > https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html > > My guess would be some latent timing bug in Twisted that was uncovered > by pypy execution time changes (I don't imagine that the CTCP code gets > exercised very heavily these days) or perhaps PyPy got a bug. > > > Wim. > > > I have setup a pyp3.7.4 and I can reproduce it. > > I see 3 options: > > * Option A: Skip those tests on pypy and open a separate ticket to fix > the test > * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the > test and unpin it > * Option C: Just fix the tests :) > > I am looking at option C for one hour... if I can't find a fix will look > into option A. > > Cheers > > -- > Adi Roiban > > > > -- > Adi Roiban > > > > -- > Adi Roiban > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Does anyone know why trunk is failing CI on PyPy 7.3.4?
Hi PR at https://github.com/twisted/twisted/pull/1592 I hope we can get a quick review and make the trunk green again :) Cheeers On Wed, 28 Apr 2021 at 10:30, Adi Roiban wrote: > Hi, > > A quick diff to fix it... and a ticket > https://twistedmatrix.com/trac/ticket/10187 > > I will create a PR and we can refine the fix during the review. > > > diff --git a/src/twisted/words/protocols/irc.py > b/src/twisted/words/protocols/irc.py > index 75300019b..a97abce2e 100644 > --- a/src/twisted/words/protocols/irc.py > +++ b/src/twisted/words/protocols/irc.py > @@ -3678,10 +3678,10 @@ def ctcpExtract(message): > normal_messages.append(messages.pop(0)) > odd = not odd > > -extended_messages[:] = filter(None, extended_messages) > -normal_messages[:] = filter(None, normal_messages) > +extended_messages[:] = list(filter(None, extended_messages)) > +normal_messages[:] = list(filter(None, normal_messages)) > > -extended_messages[:] = map(ctcpDequote, extended_messages) > +extended_messages[:] = list(map(ctcpDequote, extended_messages)) > > > On Wed, 28 Apr 2021 at 10:01, Adi Roiban wrote: > >> Hi Wim, >> >> Thanks for the report >> >> On Tue, 27 Apr 2021 at 08:14, Wim Lewis wrote: >> >>> TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")? >>> >>> I don't have time right now to set up a PyPy-capable environment and try >>> to reproduce this, but perhaps someone does? >>> >>> Long version: >>> >>> Trunk has been broken since the last merge a week or so ago, but I >>> don't think the breakage is due to that merge. As an experiment I made >>> a PR based off the last successful version of trunk, with a whitespace >>> change, and it now fails CI as well. So I think the failure must be due >>> to some change that isn't in Twisted or controlled-for by tox. >>> >>> The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. >>> Unlike our usual CI problems it doesn't seem to be a random failure: it >>> fails all the time in the same place. But the place doesn't make sense >>> to me. It's in the IRC CTCP tests, and they fail in the same ways each >>> time (an expected response is not received). >>> >>> The pair of CI runs closest to the change are these: >>> >>> run 5793: https://github.com/twisted/twisted/runs/2328450554 >>> run 5809: https://github.com/twisted/twisted/runs/2360415474 >>> >>> There are a lot of differences, but sys.version went from 3.7.9 to >>> 3.7.10 between those runs, so that seems like the most likely culprit. >>> >>> Last working: >>> > sys.version : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52) >>> > [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] >>> > sys.prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 >>> > sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 >>> > sys.executable : >>> /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python >>> >>> First broken: >>> > sys.version : 3.7.10 (51efa818fd9b, Apr 04 2021, >>> 11:22:34) >>> > [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] >>> > sys.prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 >>> > sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 >>> > sys.executable : >>> /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python >>> >>> PyPy's release notes for 7.3.4 don't list anything that jumps out at me: >>> >>> https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html >>> >>> My guess would be some latent timing bug in Twisted that was uncovered >>> by pypy execution time changes (I don't imagine that the CTCP code gets >>> exercised very heavily these days) or perhaps PyPy got a bug. >>> >>> >>> Wim. >>> >>> >> I have setup a pyp3.7.4 and I can reproduce it. >> >> I see 3 options: >> >> * Option A: Skip those tests on pypy and open a separate ticket to fix >> the test >> * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the >> test and unpin it >> * Option C: Just fix the tests :) >> >> I am looking at option C for one hour... if I can't find a fix will look >> into option A. >> >> Cheers >> >> -- >> Adi Roiban >> > > > -- > Adi Roiban > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Does anyone know why trunk is failing CI on PyPy 7.3.4?
Hi, A quick diff to fix it... and a ticket https://twistedmatrix.com/trac/ticket/10187 I will create a PR and we can refine the fix during the review. diff --git a/src/twisted/words/protocols/irc.py b/src/twisted/words/protocols/irc.py index 75300019b..a97abce2e 100644 --- a/src/twisted/words/protocols/irc.py +++ b/src/twisted/words/protocols/irc.py @@ -3678,10 +3678,10 @@ def ctcpExtract(message): normal_messages.append(messages.pop(0)) odd = not odd -extended_messages[:] = filter(None, extended_messages) -normal_messages[:] = filter(None, normal_messages) +extended_messages[:] = list(filter(None, extended_messages)) +normal_messages[:] = list(filter(None, normal_messages)) -extended_messages[:] = map(ctcpDequote, extended_messages) +extended_messages[:] = list(map(ctcpDequote, extended_messages)) On Wed, 28 Apr 2021 at 10:01, Adi Roiban wrote: > Hi Wim, > > Thanks for the report > > On Tue, 27 Apr 2021 at 08:14, Wim Lewis wrote: > >> TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")? >> >> I don't have time right now to set up a PyPy-capable environment and try >> to reproduce this, but perhaps someone does? >> >> Long version: >> >> Trunk has been broken since the last merge a week or so ago, but I >> don't think the breakage is due to that merge. As an experiment I made >> a PR based off the last successful version of trunk, with a whitespace >> change, and it now fails CI as well. So I think the failure must be due >> to some change that isn't in Twisted or controlled-for by tox. >> >> The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. >> Unlike our usual CI problems it doesn't seem to be a random failure: it >> fails all the time in the same place. But the place doesn't make sense >> to me. It's in the IRC CTCP tests, and they fail in the same ways each >> time (an expected response is not received). >> >> The pair of CI runs closest to the change are these: >> >> run 5793: https://github.com/twisted/twisted/runs/2328450554 >> run 5809: https://github.com/twisted/twisted/runs/2360415474 >> >> There are a lot of differences, but sys.version went from 3.7.9 to >> 3.7.10 between those runs, so that seems like the most likely culprit. >> >> Last working: >> > sys.version : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52) >> > [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] >> > sys.prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 >> > sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 >> > sys.executable : >> /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python >> >> First broken: >> > sys.version : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34) >> > [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] >> > sys.prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 >> > sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 >> > sys.executable : >> /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python >> >> PyPy's release notes for 7.3.4 don't list anything that jumps out at me: >> >> https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html >> >> My guess would be some latent timing bug in Twisted that was uncovered >> by pypy execution time changes (I don't imagine that the CTCP code gets >> exercised very heavily these days) or perhaps PyPy got a bug. >> >> >> Wim. >> >> > I have setup a pyp3.7.4 and I can reproduce it. > > I see 3 options: > > * Option A: Skip those tests on pypy and open a separate ticket to fix > the test > * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the > test and unpin it > * Option C: Just fix the tests :) > > I am looking at option C for one hour... if I can't find a fix will look > into option A. > > Cheers > > -- > Adi Roiban > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Does anyone know why trunk is failing CI on PyPy 7.3.4?
Hi Wim, Thanks for the report On Tue, 27 Apr 2021 at 08:14, Wim Lewis wrote: > TLDR: Twisted appears broken on PyPy 7.3.4 (aka "3.7.10")? > > I don't have time right now to set up a PyPy-capable environment and try > to reproduce this, but perhaps someone does? > > Long version: > > Trunk has been broken since the last merge a week or so ago, but I > don't think the breakage is due to that merge. As an experiment I made > a PR based off the last successful version of trunk, with a whitespace > change, and it now fails CI as well. So I think the failure must be due > to some change that isn't in Twisted or controlled-for by tox. > > The failure in all cases is in the pypy-3.7-alldeps-nocov-posix task. > Unlike our usual CI problems it doesn't seem to be a random failure: it > fails all the time in the same place. But the place doesn't make sense > to me. It's in the IRC CTCP tests, and they fail in the same ways each > time (an expected response is not received). > > The pair of CI runs closest to the change are these: > > run 5793: https://github.com/twisted/twisted/runs/2328450554 > run 5809: https://github.com/twisted/twisted/runs/2360415474 > > There are a lot of differences, but sys.version went from 3.7.9 to > 3.7.10 between those runs, so that seems like the most likely culprit. > > Last working: > > sys.version : 3.7.9 (7e6e2bb30ac5, Nov 18 2020, 10:55:52) > > [PyPy 7.3.3-beta0 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] > > sys.prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 > > sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.9/x64 > > sys.executable : > /opt/hostedtoolcache/PyPy/3.7.9/x64/bin/python > > First broken: > > sys.version : 3.7.10 (51efa818fd9b, Apr 04 2021, 11:22:34) > > [PyPy 7.3.4 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] > > sys.prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 > > sys.exec_prefix : /opt/hostedtoolcache/PyPy/3.7.10/x64 > > sys.executable : > /opt/hostedtoolcache/PyPy/3.7.10/x64/bin/python > > PyPy's release notes for 7.3.4 don't list anything that jumps out at me: > > https://doc.pypy.org/en/latest/whatsnew-pypy3-7.3.4.html > > My guess would be some latent timing bug in Twisted that was uncovered > by pypy execution time changes (I don't imagine that the CTCP code gets > exercised very heavily these days) or perhaps PyPy got a bug. > > > Wim. > > I have setup a pyp3.7.4 and I can reproduce it. I see 3 options: * Option A: Skip those tests on pypy and open a separate ticket to fix the test * Option B: Pin pypy 3.7.9 for GHA and open a separate ticket to fix the test and unpin it * Option C: Just fix the tests :) I am looking at option C for one hour... if I can't find a fix will look into option A. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Codecov.io security incident
On Fri, 16 Apr 2021 at 20:15, Glyph wrote: > > On Apr 16, 2021, at 11:26 AM, Adi Roiban wrote: > > > For twisted/twisted and I think that other repos the main secret available > for GitHub Action is the PYPY upload token. > > > Just to make sure here - you mean PyPI, right? > > Yes. Sorry. PyPi.org. > I guess that what we can do is stop using the codecov.io bash uploaded and > switch back to python uploader. > > Any other ideas ? > > > I think we are actually OK given the constraints on the env vars, but just > to be safe, we should invalidate / rotate the PyPI upload token. Any admins > have a few spare minutes to do that? (And like… check to make sure nobody > uploaded anything surprising on our project page ;-)). > > I don't have access to Twisted or ldaptor or other projects. I only have access to pydoctor, and I saw that someone from NL (most probably Marteen :) has already rotated the token. https://pypi.org/project/Twisted/#history looks ok. Last release l21.2.0 - Feb 28, 2021 Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Codecov.io security incident
Hi. This is a follow up for https://about.codecov.io/security-update/ that was raised by Maarten The security breach is from January 31, 2021, Here you can see the list of Twisted org projects using Codecov.io https://codecov.io/gh/twisted The projects that might be affected are: twisted Latest commit 3 hours ago - using Bash pydoctor Latest commit a day ago - using Python towncrier Latest commit a day ago - using Python axiom Latest commit 2 days ago - using bash via codecov/codecov-action@v1 klein Latest commit 7 days ago - using bash via codecov/codecov-action@v1 incremental Latest commit 25 days ago - using codecov in Travis ldaptor 2 months ago - using Python So the only targets are: twisted , axiom and klein For twisted/twisted we start using the bash uploaded 19 days ago as part of https://github.com/twisted/twisted/pull/1574/ Before that we were using the python uploader. --- Here is my understanding of what the codecov bash uploader can do: * Read all the env variables present at the time the bash codecov.io script is executed. The env might contain secrets * Use the GitHub Token that is automatically generated for each GitHub Action job The GitHub token is valid while the action is executed and is kind of a super token: Actions: write Checks: write Contents: write Deployments: write Issues: write Metadata: read Packages: write PullRequests: write RepositoryProjects: write SecurityEvents: write Statuses: write --- For twisted/twisted and I think that other repos the main secret available for GitHub Action is the PYPY upload token. This is not used as a general env variable, but is only available to the specific step in which twine is used to upload the files. - The GitHub Org audit page can be used to check org administratie changes https://github.com/organizations/twisted/settings/audit-log I took a quick look and didn't notice anything suspicious. - I don't know how we can prevent these types of security issues. We are a public project with limited resources and are always exposed when we are pulling dependencies from codecov or pypy that we don't fully control. I guess that what we can do is stop using the codecov.io bash uploaded and switch back to python uploader. Any other ideas ? Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On Tue, 6 Apr 2021 at 19:58, Kyle Altendorf wrote: > On 2021-04-01 09:36, Adi Roiban wrote: > > Thanks Kyle for the GitHub communication. I have checked Twisted billing and we have GitHub Teams enabled. So yes, 60x concurrent jobs on GitHub own VMs. GitHub billing now shows $124.00 / month to cover our Team plan but we don't have to pay. For reference Twisted has 41 members [0] Upgrading to enterprise will be $903.00 / month :) -- We now use 13x jobs for a build - 10 - python + 3 lint/mypy/docs. Python jobs now take a maximum of 8 minutes (on 2xCPU VMs with 4 parallel jobs) [1] ... from the previous 30+ minutes In the last month there were about 250 jobs so about 8 per day. So I think that we are ok for now. -- We have 12 extra jobs on Azure. I think that as an experiment we can move 1 Windows and 1 macOS job from Azure to GitHub Actions. I would still keep Azure pipelines jobs to use those extra VMs that are available. Azure allows retrying a single job on failure , or all failed jobs. This is a bit nicery for flaky tests. In GitHub, you have to re-run the whole workflow. - We can get more jobs by hosting our own runners... but that is a different story. [snip] Sounds good. Sounds like exactly what I suggested. :] Except for > maybe misunderstanding about what level of separation is needed for the > interjob dependencies. > > Hopefully this clarifies a bit. I think we actually intend the same > thing. Unless I've missed something else? > I guess that is best to discuss directly on the change made for a PR and see how the final matrix will look. Top priority for me is enabling trial parallel runs on Windows (PR waiting for review) to use the extra CPU available on the VM. Next is enabling some kind of PR triage helper. [0] https://github.com/orgs/twisted/people [1] https://github.com/twisted/twisted/actions/runs/735543893 [2] https://dev.azure.com/twistedmatrix/twisted/_build/results?buildId=3717=results -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On Thu, 1 Apr 2021 at 13:46, Kyle Altendorf wrote: > On 2021-03-31 05:45, Adi Roiban wrote: > > > On Tue, 30 Mar 2021, 22:23 Kyle Altendorf, wrote: > > > >> On 2021-03-30 15:24, Glyph wrote: > >>>> On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: > >>>> > >>>> Hi All, > >>>> > >>>> Has anyone contacted GitHub to see if they would be willing to > >>>> increase the parallelism limit in Actions? My understanding is that > >>>> we maintain two CI systems (GitHub Actions and Azure Pipelines) for > >>>> the sake of more parallelism. While perhaps worthwhile, this > >>>> doesn't > >>>> seem fun. Maybe GitHub would be willing to help out. > >>> > >>> Not as far as I know. Do you want to give it a shot? > >> > >> That was the plan. Submitted. I'll let you all know. > > > > Thanks. > > > > On a related note I am working to reduce the general duration of > > Twisted CI jobs. > > > > This should result in keeping the jobs slot busy for less time and > > allowing more total jobs per day. > > > > The first step was to enable parallel trial tests. Use both CPUs > > available to the CI VMs. > > This was already done for Linux and MacOS. > > > > Distributed trial is not yet supported on Windows, but I am working on > > it. > > I already have a working distributed trial for Windows on my system. > > It needs various fixed and I have submitted smaller PR for review. > > > > Just getting distrial to work on Windows is not all...as we need to > > update and clean the Twsited test to reduce the side effects. > > Linux and macOS are more relaxed in terms of file access and for > > example you can delete a file, even if its opened by another process or > > by the same process. > > Windows is not happy about that. > > > > And then there are the 2 pypy jobs taking 30 minutes each - working to > > fix it here https://github.com/twisted/twisted/pull/1543/ > > > > And then there are also general CI jobs improvements. > > > > For example we can save about 40 seconds if we decide to stop sending > > coverage to coveralls and only use codecov.io > > Another example is save another 1 minute by stop using `tox --notests` > > and replace it with something like > > https://github.com/twisted/python-info-action to show the dependencies. > > With `tox --notest` we now create the wheels and install the > > dependencies twice for each job. > > Would `tox --skip-pkg-install` cut it for the second invocation? > > > https://github.com/ericaltendorf/plotman/pull/66/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR145 > > I don't know. I still don't understand the tox install and test process... and tox is not my friend as every week tox hits me with various errors that I don't understand. Most probably, tox is an advanced tool and I am a poor developer who doesn't know much and sticks to the manual labour of calling venv + pip + trial :) For example, I don't understand why you need to create an sdist and from sdist to create a wheel and then install that wheel inside tox for each tox test environment. Why not create the wheel once and install the same wheel in all environments. > > There is an extra 1 minute required for each job to generate the > > coverage XML file. We can download all the coverage artifacts and > > generate the XML only once. > > I did coverage collection from all jobs for pymodbus. Looks like I > still had the XML report generating in each job, but that could > presumably be shifted. Anyways, I'm sure several of us could do this > off the top of our heads but in case it is of any interest here's what I > did. Or maybe you just point out something silly I did there and should > learn from. :] > > https://github.com/riptideio/pymodbus/pull/592/files > That is one big matrix :) ... I don't understand why you need test and check and coverage, each with a separate matrix. I am thinking of something like this: * A matrix to run jobs with coverage on each environment (os + py combination + extra stuff noipv6, nodeps, etc) * Each job from the matrix will run an initial combine to combine sub-process coverage * Each job will upload once python coverage raw file. * A single job will download those coverage file, combine them once again to generate an XML file to be pushed to codecov.io The maximum jobs time will continue to be the same...as you still need to wait for the slowest env, but the total run time can be reduced by more than 20 minutes as each job will no longer h
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On Tue, 30 Mar 2021 at 15:57, Kyle Altendorf wrote: > Hi All, > SNIP > My understanding is that we maintain > two CI systems (GitHub Actions and Azure Pipelines) for the sake of more > parallelism. > SNIP > -kyle > We also have 2 CI systems because migrating the Windows and macOS jobs from Azure Pipeline to GHA takes time. As an experiment I have enabled one Windows job on GitHub and the tests failed... so that needs extra work. To check what happened I got Windows 2019 server on a local VM and tried to execute `trial twisted` ...tests are failing. So for now, I am also not in a hurry to remove Azure Pipelines as it just works. Fixing is not hard but it takes time. For example some Windows tests are skipped only on Azure... and they fail both on Azure and local dev... I guess that a macOS migration is easier as macOS is similar to Linux, and on macOS we only run the default reactor, which I think is the select reactor. But in preparation for a future GitHub to Azure migration, I think that we can have a single Windows and a single macOS job on GitHub Actions, while keeping Azure VMs. In this way we can have them running for a while in trunk and see if there are any surprises . I guess that with GitHub being a MS company GitHub Actions and Azure Pipelines might use the exact same VM images and VM compute nodes. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Enable and use GitHub Discussions for twisted/twisted
On Mon, 29 Mar 2021 at 23:06, Glyph wrote: > > > On Mar 29, 2021, at 7:27 AM, Adi Roiban wrote: > > Hi, > > Should I enable GitHub discussions for twisted/twisted repo? > > > Rather than phrasing this in terms of advantages & disadvantages, I think > it might be more fruitful to talk about what *problem* we would > ostensibly be trying to solve by using it. Right now I don't think the > list is high-traffic enough that the big advantage (you can have a > discussion without blasting out a message to too many people) would really > solve any issues we're currently having. > > Personally I am -0 because I don't think that Github's structure for > notifications are particularly conducive to an Inbox Zero style approach, > i.e. one where everyone involved with he project knows where to go check > for things that they might want to pay attention to. The mailing list > makes this very straightforward; you just check your email, which you're > probably doing anyway. > > Github does have *some* notion of "inboxes", i.e. > https://github.com/twisted/twisted/pulls/review-requested/@me could > probably replace https://twisted.reviews, but even though I am a person > who thinks *way too much* about inboxes > <https://glyph.twistedmatrix.com/2016/04/email-isnt-the-problem.html>, I > find https://github.com/notifications difficult to approach. If we were > going to turn it on we should probably have a recommended way to check in > on messages for contributors. > > Thanks for the feedback. That is ok. There is no need to use GitHub Discussions. Thanks for the nice read about inboxes :) -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On Tue, 30 Mar 2021, 22:23 Kyle Altendorf, wrote: > > > On 2021-03-30 15:24, Glyph wrote: > >> On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: > >> > >> Hi All, > >> > >> Has anyone contacted GitHub to see if they would be willing to > >> increase the parallelism limit in Actions? My understanding is that > >> we maintain two CI systems (GitHub Actions and Azure Pipelines) for > >> the sake of more parallelism. While perhaps worthwhile, this doesn't > >> seem fun. Maybe GitHub would be willing to help out. > > > > Not as far as I know. Do you want to give it a shot? > > That was the plan. Submitted. I'll let you all know. > Thanks. On a related note I am working to reduce the general duration of Twisted CI jobs. This should result in keeping the jobs slot busy for less time and allowing more total jobs per day. The first step was to enable parallel trial tests. Use both CPUs available to the CI VMs. This was already done for Linux and MacOS. Distributed trial is not yet supported on Windows, but I am working on it. I already have a working distributed trial for Windows on my system. It needs various fixed and I have submitted smaller PR for review. Just getting distrial to work on Windows is not all...as we need to update and clean the Twsited test to reduce the side effects. Linux and macOS are more relaxed in terms of file access and for example you can delete a file, even if its opened by another process or by the same process. Windows is not happy about that. And then there are the 2 pypy jobs taking 30 minutes each - working to fix it here https://github.com/twisted/twisted/pull/1543/ And then there are also general CI jobs improvements. For example we can save about 40 seconds if we decide to stop sending coverage to coveralls and only use codecov.io Another example is save another 1 minute by stop using `tox --notests` and replace it with something like https://github.com/twisted/python-info-action to show the dependencies. With `tox --notest` we now create the wheels and install the dependencies twice for each job. There is an extra 1 minute required for each job to generate the coverage XML file. We can download all the coverage artifacts and generate the XML only once. - Now the actual job duration will differ... I see the pip cache download taking 2 seconds or 2 minutes. But on Linux it takes about 4 minutes to run the tests (inside tox with wheel creation)... and then from 2minutes and above is the extra overhead Cheers > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Enable and use GitHub Discussions for twisted/twisted
Hi, Should I enable GitHub discussions for twisted/twisted repo? Advantages: * it's like email but I think that it's easier to include/exclude people interested in the discussion * If an external person / one time contributor wants to participate in a discussion, subscribing to the whole maillinglist is no longer required * Should be easier to find the "right answer" inside a discussion. * You can search the twisted/twisted repo from one place (GitHub top left search bar) and also find discussion for that topic. * Might help with keeping the comments inside the PR focused on the code review as it should be easier to start a discussion Disadvantages * Depend even more on GitHub / MS.. but I hope that even if * Yet another communication channel for the Twisted community. We already have * this mailinglist * the web mailinglist * IRC * stack overflow * GitHub PRs We already depende a lot on Github for git hosting, code review, automated tests, deployment, pypy publishing. I don't know if the discussion feature will make much of a difference. There is an API [0] that ctwisted-pythontwisted-pythonan be used to export the data if needed or import mailman data. What do you think? Yes... no... maybe...stupid idea :) Thanks --- We have an archive for this list [1] but there is no search. [0] https://docs.github.com/en/graphql/guides/using-the-graphql-api-for-discussions [1] https://twistedmatrix.com/pipermail/twisted-python/ -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Probot app to automatically delete merged branches
On Sat, 13 Mar 2021 at 14:29, Kyle Altendorf wrote: > > On 2021-03-13 08:22, Adi Roiban wrote: > > Hi, > > Do you see anything bad in enabling the delete-merged-branch app for > twisted/twisted ? > > https://probot.github.io/apps/delete-merged-branch/ > > > > There's a checkbox for this. Unless I'm missing something? > > > https://docs.github.com/en/github/administering-a-repository/managing-the-automatic-deletion-of-branches > > Great. Thank for the info. So, new question: Should we enable the automatic deletion for twisted/twisted? Regards -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Probot app to automatically delete merged branches
Hi, Do you see anything bad in enabling the delete-merged-branch app for twisted/twisted ? https://probot.github.io/apps/delete-merged-branch/ Right now Twisted has 1877 branches. My hope is that in this way we can keep the main repo clean and there will be less noise in autocompletion. Now, this will only handle future branches/merges ...but at least is a start. We can see later what to do with existing branches. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Post Release updates
On Sun, 28 Feb 2021 at 20:42, Adi Roiban wrote: > Hi, > > [snip] > > -- > > The https://twistedmatrix.com/documents/current/ documentation was not > updated. > But that can be solved after the release. > I will see if I can update it tonight... and I will check over IRC to see > if someone else has fixed it. > First, I am looking at the Read the docs release issues... it looks like > things are not as automated as we though :D > > - > > I have no idea why the latest tag is not visible on Read the docs > https://readthedocs.org/projects/twisted/versions/ > > I have manually activated it based on the f1da tag. > I went to > https://readthedocs.org/projects/twisted/versions/?version_filter=twisted-21.2.0 > and then clicked "Activate" > > Then I went to https://readthedocs.org/dashboard/twisted/advanced/ and > select it as "Default" version > and then click "Save" > > Then I went to https://readthedocs.org/projects/twisted/builds/ to check > the status of the new build. > > I see that we now have - https://docs.twistedmatrix.com/en/twisted-21.2.0/ > > But https://docs.twistedmatrix.com/en/stable/ is still an older build :( > > I have no idea why... looks like stable is not active and what is > origin/8690 > > [image: image.png] > > Maybe Read The Docs is confused by `twisted-21.2.0` as it doesn't look > like a semantic version . > > -- > > Stable was building for > > https://github.com/twisted/twisted/commit/99137f53a8a6bd1d8bffa5202373236162daf94e > > See RTD stable build > https://readthedocs.org/projects/twisted/builds/13125823/ > > So I have wiped the stable version and now it gets 404. > > I (or someone else from the team) will have check with Read The Docs team > to see why we have this issue with the stable version. > > [snip] FYI I have created this ticket to track the documentation migration progress to Read The Docs. https://twistedmatrix.com/trac/ticket/10117 We are close, but not there yet. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Developer docs should be updated on wiki when steps changed in code?
On Wed, 3 Feb 2021 at 18:09, Glyph wrote: > > [snip] > > I can't quickly find the place where we agreed to this, but I think > several years ago at this point we had a discussion about moving *all* these > docs into the source tree. (If they're on the wiki, there's no review > process or even a place where updates can be staged for commentary before > going live and becoming "official".) > > Until this is fully completed, what is the correct thing to do when there > is a mismatch between docs on the wiki, and docs in the tree? > > > Docs in the tree always win. > > I still refer to a lot of docs on the wiki, especially for process related > things, > so I think it would be nice if the wiki docs were kept up to date, until > the day that > they are fully deleted. > > > Let's start deleting them now, and replacing them with links to the > in-tree docs, rather than updating them. They've been skewing out of date > for a long time. When I was looking for information about how to do a > revert, I found wiki docs about linking to revisions in Subversion which > didn't mention Github, which gives a flavor for how outdated some of this > stuff is. > > Thomas, would you mind doing the honors for this document? Links for > process docs should probably be to > https://docs.twistedmatrix.com/en/latest/ since, for process information > (unlike API information), trunk should be authoritative, not the latest > release. > > > I don't know what wiki page we are talking about here. Any link would help. I see this wiki page and it already has a redirection: https://twistedmatrix.com/trac/wiki/ReleaseProcess My suggestion is that next time a dev needs to update a wiki page, she/he/they should consider moving that page to twisted/twisted repo narrative docs and create a PR for that change. >From this wiki page https://twistedmatrix.com/trac/wiki/TwistedDevelopment I see that Security and Review Process are not yet migrated to narrativedocs. And I think that "Contributor Advancement Path" can be removed as we now have an informal processes for giving write access to the repo. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted 21.2.0 Release Announcement
On Sun, 28 Feb 2021 at 10:28, Craig Rodrigues wrote: > On behalf of Twisted Matrix Laboratories, I am honored to announce the > release of Twisted 21.2.0! > > There are two major announcements for this release: > > - *Python 2.7 support has been dropped*. Twisted 21.2.0 supports Python > 3.5.3 and higher only > - *This will be the last Twisted release to support Python 3.5*. > > Twisted 21.2.0 brings the following: > - twist dns --pyzone now works on Python 3 > - twisted.web.twcgi now works on Python 3 > - all tests pass on all platforms on Python 3.8, including asyncio tests > on Windows > - tests pass on Python 3.9 (thanks to Michał Górny) > - type hints, as specified in PEP 484, are now used in functions > throughout the codebase > - mypy (static type checker) is now run over the codebase during CI to > help detect problems, and help improve the use of types > - code in is now formatted with Black (thanks to Tom Most) > - many CI and automation improvements (thanks to Adi Roiban and Thomas > Grainger) > - many more fixes (225 tickets closed!) > > > You can install by running this command: > > python -m pip install Twisted==21.2.0 > > The full tarball is available at: > https://pypi.org/project/Twisted/21.2.0/#files > > The full NEWS file with all changes is at: > https://github.com/twisted/twisted/blob/twisted-21.2.0/NEWS.rst > > Hi, Thanks Craig for managing this release. Happy to finally see a brand new release. Sad to see Py2.7 go... but this is life... all good things have an end :p Looking forward to also removing Py3.5 :) Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Post Release updates
Hi, https://twistedmatrix.com/documents/current/ is now updated. It will continue to be the "official" documentation link for this release. We can add a redirection after getting RTD version up to the same features... maybe for the next release. Regards On Sun, 28 Feb 2021 at 21:16, Adi Roiban wrote: > Another thing that is missing from Read The Docs to get feature parity > with current documentation. > > If I go to https://docs.twistedmatrix.com/en/twisted-18.4.0/ I don't get > any warning that I am not on the latest stable version. > > RTD has something similar... but not quite what we need > [image: image.png] > But I hope that we can find a way to fix it... but first we need to fix > the "stable" version in RTD > > > Cheers > > On Sun, 28 Feb 2021 at 20:42, Adi Roiban wrote: > >> Hi, >> >> Happy to see a new version of Twisted released. >> Thanks to anyone for making it happen. >> >> - >> >> I think that https://github.com/twisted/twisted/pull/1514 should be >> merged to finalize the release process. >> >> -- >> >> And we have not yet decided if the trunk should be using `.dev` or .post` >> :( >> >> https://twistedmatrix.com/trac/ticket/9542 >> >> I am +1 for .post1 ... as it has the semantic of a post release version. >> while .dev0 is a pre-release. >> >> -- >> >> The https://twistedmatrix.com/documents/current/ documentation was not >> updated. >> But that can be solved after the release. >> I will see if I can update it tonight... and I will check over IRC to see >> if someone else has fixed it. >> First, I am looking at the Read the docs release issues... it looks like >> things are not as automated as we though :D >> >> - >> >> I have no idea why the latest tag is not visible on Read the docs >> https://readthedocs.org/projects/twisted/versions/ >> >> I have manually activated it based on the f1da tag. >> I went to >> https://readthedocs.org/projects/twisted/versions/?version_filter=twisted-21.2.0 >> and then clicked "Activate" >> >> Then I went to https://readthedocs.org/dashboard/twisted/advanced/ and >> select it as "Default" version >> and then click "Save" >> >> Then I went to https://readthedocs.org/projects/twisted/builds/ to check >> the status of the new build. >> >> I see that we now have - >> https://docs.twistedmatrix.com/en/twisted-21.2.0/ >> >> But https://docs.twistedmatrix.com/en/stable/ is still an older build :( >> >> I have no idea why... looks like stable is not active and what is >> origin/8690 >> >> [image: image.png] >> >> Maybe Read The Docs is confused by `twisted-21.2.0` as it doesn't look >> like a semantic version . >> >> -- >> >> Stable was building for >> >> https://github.com/twisted/twisted/commit/99137f53a8a6bd1d8bffa5202373236162daf94e >> >> See RTD stable build >> https://readthedocs.org/projects/twisted/builds/13125823/ >> >> So I have wiped the stable version and now it gets 404. >> >> I (or someone else from the team) will have check with Read The Docs team >> to see why we have this issue with the stable version. >> >> -- >> >> And there are a lot of versions as we have many branches on Twisted. >> >> I guess that at least the branches that were merged, could be removed >> from the main repo. >> I don't see the value for keeping them alive, >> >> That is all for now. >> -- >> Adi Roiban >> > > > -- > Adi Roiban > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Post Release updates
Another thing that is missing from Read The Docs to get feature parity with current documentation. If I go to https://docs.twistedmatrix.com/en/twisted-18.4.0/ I don't get any warning that I am not on the latest stable version. RTD has something similar... but not quite what we need [image: image.png] But I hope that we can find a way to fix it... but first we need to fix the "stable" version in RTD Cheers On Sun, 28 Feb 2021 at 20:42, Adi Roiban wrote: > Hi, > > Happy to see a new version of Twisted released. > Thanks to anyone for making it happen. > > - > > I think that https://github.com/twisted/twisted/pull/1514 should be > merged to finalize the release process. > > -- > > And we have not yet decided if the trunk should be using `.dev` or .post` > :( > > https://twistedmatrix.com/trac/ticket/9542 > > I am +1 for .post1 ... as it has the semantic of a post release version. > while .dev0 is a pre-release. > > -- > > The https://twistedmatrix.com/documents/current/ documentation was not > updated. > But that can be solved after the release. > I will see if I can update it tonight... and I will check over IRC to see > if someone else has fixed it. > First, I am looking at the Read the docs release issues... it looks like > things are not as automated as we though :D > > - > > I have no idea why the latest tag is not visible on Read the docs > https://readthedocs.org/projects/twisted/versions/ > > I have manually activated it based on the f1da tag. > I went to > https://readthedocs.org/projects/twisted/versions/?version_filter=twisted-21.2.0 > and then clicked "Activate" > > Then I went to https://readthedocs.org/dashboard/twisted/advanced/ and > select it as "Default" version > and then click "Save" > > Then I went to https://readthedocs.org/projects/twisted/builds/ to check > the status of the new build. > > I see that we now have - https://docs.twistedmatrix.com/en/twisted-21.2.0/ > > But https://docs.twistedmatrix.com/en/stable/ is still an older build :( > > I have no idea why... looks like stable is not active and what is > origin/8690 > > [image: image.png] > > Maybe Read The Docs is confused by `twisted-21.2.0` as it doesn't look > like a semantic version . > > -- > > Stable was building for > > https://github.com/twisted/twisted/commit/99137f53a8a6bd1d8bffa5202373236162daf94e > > See RTD stable build > https://readthedocs.org/projects/twisted/builds/13125823/ > > So I have wiped the stable version and now it gets 404. > > I (or someone else from the team) will have check with Read The Docs team > to see why we have this issue with the stable version. > > ------ > > And there are a lot of versions as we have many branches on Twisted. > > I guess that at least the branches that were merged, could be removed from > the main repo. > I don't see the value for keeping them alive, > > That is all for now. > -- > Adi Roiban > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Post Release updates
Hi, Happy to see a new version of Twisted released. Thanks to anyone for making it happen. - I think that https://github.com/twisted/twisted/pull/1514 should be merged to finalize the release process. -- And we have not yet decided if the trunk should be using `.dev` or .post` :( https://twistedmatrix.com/trac/ticket/9542 I am +1 for .post1 ... as it has the semantic of a post release version. while .dev0 is a pre-release. -- The https://twistedmatrix.com/documents/current/ documentation was not updated. But that can be solved after the release. I will see if I can update it tonight... and I will check over IRC to see if someone else has fixed it. First, I am looking at the Read the docs release issues... it looks like things are not as automated as we though :D - I have no idea why the latest tag is not visible on Read the docs https://readthedocs.org/projects/twisted/versions/ I have manually activated it based on the f1da tag. I went to https://readthedocs.org/projects/twisted/versions/?version_filter=twisted-21.2.0 and then clicked "Activate" Then I went to https://readthedocs.org/dashboard/twisted/advanced/ and select it as "Default" version and then click "Save" Then I went to https://readthedocs.org/projects/twisted/builds/ to check the status of the new build. I see that we now have - https://docs.twistedmatrix.com/en/twisted-21.2.0/ But https://docs.twistedmatrix.com/en/stable/ is still an older build :( I have no idea why... looks like stable is not active and what is origin/8690 [image: image.png] Maybe Read The Docs is confused by `twisted-21.2.0` as it doesn't look like a semantic version . -- Stable was building for https://github.com/twisted/twisted/commit/99137f53a8a6bd1d8bffa5202373236162daf94e See RTD stable build https://readthedocs.org/projects/twisted/builds/13125823/ So I have wiped the stable version and now it gets 404. I (or someone else from the team) will have check with Read The Docs team to see why we have this issue with the stable version. -- And there are a lot of versions as we have many branches on Twisted. I guess that at least the branches that were merged, could be removed from the main repo. I don't see the value for keeping them alive, That is all for now. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Release instructions: `tox -e towncrier` does not work?
Hi, Sorry for the late reply. Here is a patch waiting for review https://github.com/twisted/twisted/pull/1527 Cheers On Mon, 15 Feb 2021 at 01:19, Craig Rodrigues wrote: > In > https://github.com/twisted/twisted/commit/0ad24679b168370bf0d695825170a7532bf0698f#diff-4fd162cb9fc2878b140dcc9d9d364f9f60d404a76ebda2c10518810ffe63852dL115 > > Adi Roiban changed one of the steps the release documentation to run a > command: > > *tox -e towncrier* > > but that tox target does not exist. > > Does Adi or someone else have that tox target in a patch lying > around somewhere? > > For now I can manually run *towncrier *manually outside of tox. > > -- > Craig > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted trunk and klein?
On Thu, 11 Feb 2021 at 19:26, Wilfredo Sánchez Vega wrote: > On Feb 10, 2021, at 3:44 PM, Glyph wrote: > > > This isn't even a bug in Klein, it's an issue with a version pin in its > tox.ini: > https://github.com/twisted/klein/blob/6e7b37158dea2fe73180809803a872ed98143c6d/tox.ini#L36 > > > The constraints from requests (<3,>=2.5) and hyperlink (>=2.5) are > perfectly compatible; one's just a subset of the other. > > > Sure, they are compatible, but Klein doesn’t use requests directly, and > this only looks to be failing in trunk. > > My concern here is that Twisted may have added a dependency on requests, > and that dependency means that if you want IDNA==3.1, as Klein’s tox.ini > does, that you aren’t allowed to. That seems weak; I’d say a bug. No? > > I have created a new virtual env and executed `pip install .` on trunk. It installed the following packages: constantly, zope.interface, attrs, incremental, six, Automat, idna, hyperlink, Twisted `requests` was not installed. I wasn't expecting to see `idna` as it is listed as TLS only deps but it looks like hyperlink depends on idna. Here is the dep tree $ pipdeptree -fl Twisted @ file:///home/adi/dev/twisted attrs==20.3.0 Automat==20.2.0 attrs==20.3.0 six==1.15.0 constantly==15.1.0 hyperlink==21.0.0 idna==3.1 incremental==17.5.0 zope.interface==5.2.0 setuptools==44.0.0 wheel==0.36.2 I guess that we can remove idna from setup.cfg TLS section in Twisted tls = pyopenssl >= 16.0.0 # service_identity 18.1.0 added support for validating IP addresses in # certificate subjectAltNames service_identity >= 18.1.0 idna >= 2.4 Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Upcoming Twisted Release
Hi, So, speaking for myself here and not representing the Twisted community or anyone else. I think that the consensus inside the Twisted contributors is that we should have a new release ASAP. The current release is blocked by the following issues: * A release blocker bug - https://twistedmatrix.com/trac/ticket/10069 * A release process protocol / procedure / communication --- For the blocker bug, there were 2 PR there were closed * https://github.com/twisted/twisted/pull/1499 * https://github.com/twisted/twisted/pull/1501 There is one PR still in work https://github.com/twisted/twisted/pull/1502 We can wait one more week to see if this issue can be fixed and unblock the release in this way. Otherwise, we can revert the change that introduced the regression and have the release unblocked. Note that reverting a merge that introduced a regression as soon as the regression is detected, is the current best practice. The revert was not done as there was hope that this can be fixed. With the release unblock Craig or I or someone else can create the release branch and push the release candidate to PYPI. --- For the release procedure / communication The release documentation and the process to publish the wheels to PyPI was updated with the goal of making it easier for the current release to happen. You just create a tag and you have the documentation and PyPI wheels already published. You can still manually create the wheels and publish them with twine If anyone thinks that the current changes that were made for automatic Docs (narrative and API) and wheel publishing are a regression, they can be reverted at any time, and we can merge them after the release. The ad-hoc / informal Twisted dev communication happens over IRC Freenode #twisted-dev It would be nice if we can have all active developers there for quick feedback and brainstorming. Have I forgotten anything? I hope that in 2 weeks we can have a release candidate and by the end of February a final release. Cheers On Sat, 16 Jan 2021 at 02:56, Craig Rodrigues wrote: > > > On Fri, Jan 15, 2021 at 10:42 AM Glyph wrote: > >> >> >> >> Following up on this, since we are now officially past the "beginning" of >> January: Craig, is a release imminent or can someone else pick up some >> tasks yet? >> >> > I'm close to getting this done, and want to complete this. > > -- > Craig > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Questions about Project Leadership committee
On Sun, 24 Jan 2021 at 09:57, Glyph wrote: > > On Jan 24, 2021, at 1:24 AM, Adi Roiban wrote: > > Hi, > > On Sun, 24 Jan 2021 at 04:29, Glyph wrote: > >> >> On January 23, 2021 at 6:10:04 PM, Craig Rodrigues ( >> rodr...@crodrigues.org) wrote: >> >>> >>> >>> On Sat, Jan 23, 2021 at 4:54 PM Glyph wrote: >>> >>>> At the time the committee was created, I don’t remember if this was >>>> public, so I don’t feel comfortable sharing identities publicly because it >>>> wasn’t part of the deal at the time. This is not the way I’d structure >>>> things now, but given that several members are unresponsive and don’t seem >>>> to want to be associated with the project any more, I can’t ask them if >>>> it’s OK with them. If you can find any old public documentation feel free >>>> to share though; the issue is that I’m not sure it *has* been public. If it >>>> has, it can by all means remain public. >>>> >>>> I can at least share that as I recall there are 6 members and I’m one >>>> of them. But you’d want to confirm this with the conservancy, so please >>>> join Adi’s private thread with them for further discussion. >>>> >>> >>> Thanks for the clarification. >>> Have you recently contacted the 5 other committee members to confirm if >>> they want to remain on the committee or not, >>> and either received confirmation (or feedback timeout)? >>> >> Not recently although I’ve reached out several times in the past. I have >> no plans to reach out again until someone has worked out with the SFC what >> our options are and proposed a concrete plan. >> > I was in contact with SFC over IRC. > > The first thing that someone need to do is send a message to > twis...@sfconservancy.org > > Only after no response is received in time (I don't know how long that > is...maybe 3 weeks) we can contact SFC and they > will allocate extra resources to help solve this issue. > > I have not sent a message to that list. > For now, I don't plan to do it. > I feel there is no consensus across the current active Twisted developers. > > I encourage anyone else who wants to do it, to send the message to > twis...@sfconservancy.org > > Good luck > > > Thanks for the update, Adi; I appreciate your taking the initiative on > this. > > OK. I am a liar :) I have just sent an email to twis...@sfconservancy.org with what I think can be done to activate the committee. I added Glypt and Craig to CC. Let me know if you are also interested in that private discussion. > If we want to have a private deliberation among committers about next > steps, I believe > https://github.com/orgs/twisted/teams/twisted-contributors/discussions makes > it possible to do that, so that might be a tool to use. > > I have created this team private thread on GitHub https://github.com/orgs/twisted/teams/twisted-contributors/discussions/1 > (To be clear, I think any conclusions from this discussions need to be > made public and transparent, but for reasons previously mentioned in this > conversation, we may want to be able to share ideas less publicly before we > pick one to avoid muddling things like future bidding on work.) > > I will try to follow up if I get something. I have also asked the committee to come back with a conclusion over the mailing list. Glyphs, I understand you very well in regard to the conflict to interest. And you are right. Nothing personal. It's just a personal frustration that we need to make extra effort, when we don't have much time and energy for code reviews. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Questions about Project Leadership committee
Hi, On Sun, 24 Jan 2021 at 04:29, Glyph wrote: > > On January 23, 2021 at 6:10:04 PM, Craig Rodrigues (rodr...@crodrigues.org) > wrote: > >> >> >> On Sat, Jan 23, 2021 at 4:54 PM Glyph wrote: >> >>> At the time the committee was created, I don’t remember if this was >>> public, so I don’t feel comfortable sharing identities publicly because it >>> wasn’t part of the deal at the time. This is not the way I’d structure >>> things now, but given that several members are unresponsive and don’t seem >>> to want to be associated with the project any more, I can’t ask them if >>> it’s OK with them. If you can find any old public documentation feel free >>> to share though; the issue is that I’m not sure it *has* been public. If it >>> has, it can by all means remain public. >>> >>> I can at least share that as I recall there are 6 members and I’m one of >>> them. But you’d want to confirm this with the conservancy, so please join >>> Adi’s private thread with them for further discussion. >>> >> >> Thanks for the clarification. >> Have you recently contacted the 5 other committee members to confirm if >> they want to remain on the committee or not, >> and either received confirmation (or feedback timeout)? >> > Not recently although I’ve reached out several times in the past. I have > no plans to reach out again until someone has worked out with the SFC what > our options are and proposed a concrete plan. > I was in contact with SFC over IRC. The first thing that someone need to do is send a message to twis...@sfconservancy.org Only after no response is received in time (I don't know how long that is...maybe 3 weeks) we can contact SFC and they will allocate extra resources to help solve this issue. I have not sent a message to that list. For now, I don't plan to do it. I feel there is no consensus across the current active Twisted developers. I encourage anyone else who wants to do it, to send the message to twis...@sfconservancy.org Good luck -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Ldaptor attributes in Python 3
Hi, On Fri, 22 Jan 2021 at 11:15, Peter Westlake wrote: > I'm using Ldaptor for LDAP access because it fits in with Twisted, and the > results are a bit mystifying. The results that come back from a search are > a list of LDAPEntryWithClient. Indexing this, e.g. entry['cn'], gives an > object of type JournaledLDAPAttributeSet, which is a subclass of set. It > contains the name of the attribute and a list containing the value: > > JournaledLDAPAttributeSet(b'cn', [b'Peter Westlake']) > > Getting the value out of that isn't straightforward: I'd have to check the > elements of the set for a list. Is there a better way? > > I've also found that the items() method gives something more tractable: > > >>> entry.items() > [ > (b'cn', b'Peter Westlake'), > ... > ] > > so I can turn that into a dict and index that. But again, this seems like > an odd design for an API. Is there a better way to extract the value of an > attribute from an entry? > > Can you send a full example of how you would like the API to behave ? Or send a PR ? :) I am doing a client-side search like this and I was happy with API o = LDAPEntry(client, base_dn) results = yield o.search( attributes=[b'objectclass'], scope=LDAP_SCOPE_wholeSubtree, filterText='(cn=test)', ) for result in results: print (result['objectclass']) -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] reactor for Linux io_uring
On Tue, 5 Jan 2021 at 13:44, Jean-Paul Calderone wrote: > On Tue, Jan 5, 2021 at 6:49 AM Barry Scott > wrote: > >> On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote: >> > In investigating async file I/O I came across this. In a nutshell it's >> > the new epoll() >> >> > It's marginally more efficient although this is only apparent at very >> > high loads. >> >> > What's more interesting is that io_uring accepts files as >> > well as network/pipe handles: avoiding the need for threads. >> >> What threads? Why do you call out file FDs different from socket FDs? >> >> The point of io_uring is to avoid transitions between user and kernel >> right? >> Nothing to do with thread. >> >> In current twisted you can run complex network code without threads >> already. >> >> > Here's a good intro: https://unixism.net/loti/index.html >> >> Also there is full coverage of io_uring on lwn.net. >> Its a fast evolving kernel API. >> >> > If people think an IoUringReactor is worthwhile I'll open a ticket and >> > make a start. >> >> I'm guessing that you will need to write a Python extension to get at >> io_uring or use ctypes. Is that what you where expecting? >> >> > However it will need a reviewer... :-) >> >> Yes this is going to be complex code that few people have any experience >> with. >> > > Just so there's a counter-perspective out there, I would like to suggest > that reactors are neither magical nor particular complex in the grand > scheme of software and that if you start with the assumption that a piece > of code is going to be complex and hard to review then you will almost > certainly create a piece of software that is complex and hard to review. > > My recommendation would be to instead start with the assumption that it's > just a bit more mundane code binding a relatively small and straightforward > C API related to copying bytes from one location to another. Yea, maybe > there will be some hassles getting it to compile smoothly in all the > desired environments or maybe there will be a few tricky areas for memory > management or some other kind of low-level, localized bookkeeping - but > those kinds of issues don't have to make for a complex piece of code. > Starting from this assumption, try to produce an implementation that is > straightforward, well-factored, and easily reviewed. What do you have to > lose? > > Jean-Paul > > Hi, There is this wrapper https://pypi.org/project/liburing/ I can give it a try and review a possible implementation. My suggestion is to find a real world / production use case for the new reactor so that we can run more than unit / functional tests. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plan/Goal for GitHub Sponsors
On Mon, 4 Jan 2021 at 00:54, Glyph wrote: > It’s complicated and I’m not a lawyer, so maybe it is indeed not a > problem. But in brief it’s like trademark protection, kind of, in that it > becomes SFC’s problem to be aware that you’ve said these things and tell > you not to say them. The twisted project (which is a bit of an amorphous > concept to begin with) has authorized SFC to be its fiscal sponsor, the SFC > has gone through the rigamarole with the IRS to ensure this is an > exempt-able public benefit activity, and now someone is making claims about > the project hiring, which they’re on the hook for. How does the IRS know > your status of affiliation with the project or the conservancy for sure? > Somebody has to investigate it, investigating means asking a bunch of > questions and sucking up the SFC’s time and energy, even if no enforcement > action is ever formally taken. > > In short: talk to the SFC first about the project’s status, get an actual > official recommendation and not my random opinion about what may or may not > be a problem, before doing anything related to fundraising. I can’t say > anything authoritative about what is allowed, because as far as I > understand it, *nothing* is allowed without untangling the PLC/approval > process first. :-) > > (Except to raise money for the already-authorized expenses related to the > continued hosting of twistedmatrix.com, of course.) > Thanks for the info and sorry for the trouble :) I will try to send a (private) message to SFC these days to untangle the PLC team and I will add you to CC. Until we solve the PLC issue, I would consider this discussion blocked. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Plan/Goal for GitHub Sponsors
On Sun, 3 Jan 2021 at 23:36, Glyph wrote: > You can’t fundraise for a job opening that you haven’t cleared with the > SFC as mission-aligned and properly transparent; this could get them in > trouble with the IRS. You’ll need to clear this by writing a grant proposal > and having it approved first. Please delete this posting from the sponsors > page as soon as you can, since even posting to this list probably > constitutes public advertising. > Hi, I have remove the info. The fundraising was never active. This was still WIP waiting for some format approval. I think that I have mentioned that I have no intention to apply for any of these jobs and only volunteer to bootstrap this process. I hope nobody will get into any trouble by my actions. To me it makes no sense. How can the IRS think that SFC is at fault for a public message posted by me, someone with no affiliation to Twisted or SFC ? Cheers > On January 3, 2021 at 2:46:42 PM, Adi Roiban (a...@roiban.ro) wrote: > >> >> >> On Sun, 3 Jan 2021 at 20:30, Jean-Paul Calderone < >> exar...@twistedmatrix.com> wrote: >> >>> On Wed, Dec 23, 2020 at 4:41 PM Adi Roiban wrote: >>> >>>> Hi, >>>> >>>> I started a separate discussion to confirm the goal for a future >>>> fundraising. >>>> >>>> Top priority - Pay someone to help with the review queue >>>> Nice to have - Move server/services to Azure VM - We have at least >>>> $100 monthly allowance for Azure not sure if we still have the huge >>>> $2000 allowance on rackspace. >>>> Nice to have - Migrate Trac wiki to GitHub Wiki >>>> Nice to have - Migrate Trac Ticket to GitHub Issues >>>> >>> >>> I have a suggestion for another priority to be inserted above that top >>> priority - pay someone to look after the day-to-day logistics of the >>> project. For example: to triage tickets in the issue tracker (identify >>> duplicates to avoid redundant effort, classify issues by feature request vs >>> defect vs regression, etc), keep track of the release process so releases >>> are completed in a timely manner, identify and eliminate friction in the >>> development process, and identify big-picture directions / priorities / >>> roadmap items and track and coordinate efforts to achieve them. I don't >>> think someone could be hired to set the Twisted roadmap but someone could >>> be hired to solicit this information from core developers and the wider >>> community and organize it into a coherent plan. >>> >>> >> Many thanks for your comments. >> I agree. >> >> In this case, I think that for any near-future fundraising effort >> (including GitHub Sponsors) we should have a single goal: >> >> 1. Raise money to hire a project manager. >> >> That will be a part time job and the person will work on other projects. >> >> The job activities will be (non exhaustive list - feel free to suggest): >> >> * triage tickets >> * act as the release manager (the actual release has some automation) so >> this is more about communication >> * identify friction in the development process and document and try to >> get consensus for a propose solution (the actual implementation can be done >> by someone else) >> * identify big-picture directions / priorities / roadmap items and track >> and coordinate efforts to achieve them. >> * help with fundraising / communication to future sponsors >> >> >> >> Do you have any idea of the required effort for a project manager for >> Twisted? >> Maybe we can start with raising money to hire someone for 10 hours per >> week. >> Please suggest a different number of hours if you think that 10 hours are >> not a good start. >> >> I think that at least for the first month, 10 hours per week is not >> much... >> but maybe after the second month, we can have 5 hours per week for >> project management >> and use extra money for the review queue or implement the top priority >> tasks identified by the project manager. >> >> --- >> >> Then, if we raise more than 10 hours per week, we can dedicate that money >> to code review. >> >> Then, if we ran out of reviews in the queue, use the leftover money for >> reducing operation overhead / removing roadblock. >> >> >>> Ideally this person could also look after fundraising efforts to ensure >>> that there are funds to continue to support their other activities. >>> >>> >>
Re: [Twisted-Python] Plan/Goal for GitHub Sponsors
On Sun, 3 Jan 2021 at 20:30, Jean-Paul Calderone wrote: > On Wed, Dec 23, 2020 at 4:41 PM Adi Roiban wrote: > >> Hi, >> >> I started a separate discussion to confirm the goal for a future >> fundraising. >> >> Top priority - Pay someone to help with the review queue >> Nice to have - Move server/services to Azure VM - We have at least $100 >> monthly allowance for Azure not sure if we still have the huge $2000 >> allowance on rackspace. >> Nice to have - Migrate Trac wiki to GitHub Wiki >> Nice to have - Migrate Trac Ticket to GitHub Issues >> > > I have a suggestion for another priority to be inserted above that top > priority - pay someone to look after the day-to-day logistics of the > project. For example: to triage tickets in the issue tracker (identify > duplicates to avoid redundant effort, classify issues by feature request vs > defect vs regression, etc), keep track of the release process so releases > are completed in a timely manner, identify and eliminate friction in the > development process, and identify big-picture directions / priorities / > roadmap items and track and coordinate efforts to achieve them. I don't > think someone could be hired to set the Twisted roadmap but someone could > be hired to solicit this information from core developers and the wider > community and organize it into a coherent plan. > > Many thanks for your comments. I agree. In this case, I think that for any near-future fundraising effort (including GitHub Sponsors) we should have a single goal: 1. Raise money to hire a project manager. That will be a part time job and the person will work on other projects. The job activities will be (non exhaustive list - feel free to suggest): * triage tickets * act as the release manager (the actual release has some automation) so this is more about communication * identify friction in the development process and document and try to get consensus for a propose solution (the actual implementation can be done by someone else) * identify big-picture directions / priorities / roadmap items and track and coordinate efforts to achieve them. * help with fundraising / communication to future sponsors Do you have any idea of the required effort for a project manager for Twisted? Maybe we can start with raising money to hire someone for 10 hours per week. Please suggest a different number of hours if you think that 10 hours are not a good start. I think that at least for the first month, 10 hours per week is not much... but maybe after the second month, we can have 5 hours per week for project management and use extra money for the review queue or implement the top priority tasks identified by the project manager. --- Then, if we raise more than 10 hours per week, we can dedicate that money to code review. Then, if we ran out of reviews in the queue, use the leftover money for reducing operation overhead / removing roadblock. > Ideally this person could also look after fundraising efforts to ensure > that there are funds to continue to support their other activities. > > We have a catch-22 situation here ... we need to hire someone to work on fundraising ... we need to raise funds to hire someone :) I can volunteer to bootstrap this effort and try to raise initial money to find a project manager that can look after future fundraising. I don't have much free time and I am not a good project manager or communication manager :) Right now, I don't know who we could hire and what could be the selection criteria ... But I think that we can focus to see if we can raise 10 hours per week and then worry about finding the right person :) Helping with the review queue is great but it's a purely reactive > activity. This is fine so far as it goes but it leaves the project without > a coherent direction, which in turn makes less productive use of the > resources available. The project should continue to operate reactively to > address issues raised by the community but to really stay relevant, the > core Twisted team itself also needs to identify coherent future goals and > work to achieve them. > > Messing with CI configuration, wikis, issue tracking, etc, may also all be > beneficial but they're not useful goals in themselves - they should all be > in support of a goal like reducing operational overhead to allow resources > to be directed elsewhere or removing roadblocks that stand in the way of > other contributors having an impact. > <https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python> > True. See the updated GitHub Sponsor page https://github.com/sponsors/twisted?preview=true Let me know what you think and if you think that is ready to make it public. We need feedback on everything on the preview page: * Project goal * Support tiers * Proje
Re: [Twisted-Python] Upcoming Twisted Release
On Mon, 28 Dec 2020 at 18:17, Glyph wrote: > > On Dec 28, 2020, at 3:38 AM, Jean-Paul Calderone < > exar...@twistedmatrix.com> wrote: > > I guess there are no comments against removing a ticket from the >> release-blocking list if the ticket is not active for 1 or 2 weeks. >> > > Commenting against this was the main reason for my earlier reply. I've > left my quoted reply above. > > > I'm also against this. Inactivity for a week or two is not enough reason > to allow a known regression to be present in a release. > > Is there a reason we can't identify the commit that caused the problem and > simply revert before doing the release, then fix it afterwards? > > If we're in a position where trunk has drifted so far between the > regression and its identification that it is too labor-intensive to revert > and we need to fix it forward, then the release *is* stuck - that's the > whole point of the "release blocker" category. This is a very unfortunate > situation but less unfortunate than shipping buggy releases that are known > to break big users of Twisted. > > Thanks for your feedback. I have created https://twistedmatrix.com/trac/ticket/10073 to add extra info to our release documentation. So, we will continue to have release candidates and will block a release for as long as it's needed. I can continue to use Twisted from pinned trunk without any issue :) It looks like with enough noise we got 2 PRs for the release blocker ticket https://github.com/twisted/twisted/pull/1499 https://github.com/twisted/twisted/pull/1501 :) So I hope the release will no longer be blocked for much longer :) -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Upcoming Twisted Release
On Mon, 28 Dec 2020 at 01:00, Jean-Paul Calderone wrote: > On Sun, Dec 27, 2020 at 6:59 PM Adi Roiban wrote: > >> Hi Craig, >> >> On Sun, 27 Dec 2020 at 20:10, Craig Rodrigues >> wrote: >> >>> >>> >>> On Sat, Dec 26, 2020 at 3:50 PM Adi Roiban wrote: >>> >>>> Hi, >>>> >>>> I plan to act as a release manager for the next release and follow the >>>> plan documented at >>>> >>>> >>>> https://docs.twistedmatrix.com/en/latest/core/development/policy/release-process.html >>>> >>>> >>> I was previously working on releasing Twisted. I was running into >>> various roadblocks, but was moving forward, >>> and got permission from Glyph to move forward with this. >>> Has this changed? >>> >> >> If you want to do the release, I am more than happy to not have to do the >> release myself :) >> >> >>> Unfortunately, Amber did not respond to any e-mails that I sent to her >>> and Glyph, so I tried to move forward the >>> best that I could. >>> >> >> Is there anything still blocking you ? >> Can I help? >> >> >>>> >>>> So no other tickets are in the blocker queue: >>>> https://twistedmatrix.com/trac/report/26 >>>> >>>> -- >>>> >>>> Do you know any other release blocker issues? >>>> >>>> >>> I filed this: >>> https://twistedmatrix.com/trac/ticket/10070 >>> >>> which I found with Pierre Tardy's help by running buildbot's test suite >>> against Twisted trunk. >>> This looks like a problem on the Twisted side, and should be fixed >>> before a Twisted release is pushed out. >>> >> >> OK. I closed https://twistedmatrix.com/trac/ticket/10069 as I think it's >> a duplicate. >> >> Do you or Pierre plan to fix that ticket? >> >> I think that you should block the release only if someone is committed to >> fixing the release blocker. >> >> I plan to update the release documentation to make it clear that all >> release blocker tichet >> should have an owner and there are plans to fix the ticket in a maximum >> of 2 weeks. >> >> Otherwise we risk to block the release forever... and if we delay forever >> people will start using "trunk" >> and if everybody is using trunk, what is the point of a release :) ? >> > > Part of the point is that when someone runs `pip install ...` they get a > *working* version of Twisted, to the best of the project's ability to > provide one. > > Fortunately many regressions aren't that difficult to resolve. At worst, > find the merge that introduced them and revert it. This works best when > regressions are found in a timely manner, of course. Of course it's also > nice if the problem can be fixed without backing out whatever (presumably > desirable) set of changes it came along with. > > Part of the release managers job is to motivate this kind of work to > happen. A standing policy to revert the cause of a regression can also > serve as good motivation to get the other kind of fix in, too. > > It's better if these known regressions don't linger for months, though. > It looks like the Buildbot PR had a failing CI run in October. I'd suggest > that not waiting until December is a good way to avoid having these kinds > of situations turn into a larger problem. > > Jean-Paul > > Thanks Kyle and Jean-Paul for your feedback. With GitHub Actions is easy to have wheels and full docs published on PyPi and Read The Docs. We can do a release candidate and the release candidate might be a good way to do one more release rehearsal before the final release :) I was wrong to suggest installing based on trunk ... even when use pip to install from git, it should have been installed based on the release branch :) But that issue is solved. - I guess there are no comments against removing a ticket from the release-blocking list if the ticket is not active for 1 or 2 weeks. - Crag, if you have time, can you join the IRC channel ? Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Upcoming Twisted Release
Hi Craig, On Sun, 27 Dec 2020 at 20:10, Craig Rodrigues wrote: > > > On Sat, Dec 26, 2020 at 3:50 PM Adi Roiban wrote: > >> Hi, >> >> I plan to act as a release manager for the next release and follow the >> plan documented at >> >> >> https://docs.twistedmatrix.com/en/latest/core/development/policy/release-process.html >> >> > I was previously working on releasing Twisted. I was running into various > roadblocks, but was moving forward, > and got permission from Glyph to move forward with this. > Has this changed? > If you want to do the release, I am more than happy to not have to do the release myself :) > Unfortunately, Amber did not respond to any e-mails that I sent to her and > Glyph, so I tried to move forward the > best that I could. > Is there anything still blocking you ? Can I help? >> >> So no other tickets are in the blocker queue: >> https://twistedmatrix.com/trac/report/26 >> >> -- >> >> Do you know any other release blocker issues? >> >> > I filed this: > https://twistedmatrix.com/trac/ticket/10070 > > which I found with Pierre Tardy's help by running buildbot's test suite > against Twisted trunk. > This looks like a problem on the Twisted side, and should be fixed before > a Twisted release is pushed out. > OK. I closed https://twistedmatrix.com/trac/ticket/10069 as I think it's a duplicate. Do you or Pierre plan to fix that ticket? I think that you should block the release only if someone is committed to fixing the release blocker. I plan to update the release documentation to make it clear that all release blocker tichet should have an owner and there are plans to fix the ticket in a maximum of 2 weeks. Otherwise we risk to block the release forever... and if we delay forever people will start using "trunk" and if everybody is using trunk, what is the point of a release :) ? -- For the future, maybe we can stop doing a release candidate wheel and instead just announce that a release is coming and people should test trunk and flag any release blocker ticket. I think that with pip, is now very easy to install Twisted from trunk. Cheers -- > Craig > ___________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Upcoming Twisted Release
Hi, I plan to act as a release manager for the next release and follow the plan documented at https://docs.twistedmatrix.com/en/latest/core/development/policy/release-process.html - The plan is to push the release candidate only on PyPI and to host the release candidate documentation (including pydoctor generated API docs) only on Read the Docs site. The Twisted Wiki Download page was updated to inform that PyPI is the only server where we publish the latest versions. So release candidates and final releases will be pushed only to PyPI. https://twistedmatrix.com/trac/wiki/Downloads While the release candidate documentation in only published on Read The Docs, at least for the next release I plan to also publish the documentation to: * https://twistedmatrix.com/documents/current/ * https://twistedmatrix.com/documents/VERSION/ After the release we can see if we can redirect them to Read The Docs. For now, the only release blocker are the missing intersphinx API links on Read The Docs between the narrative docs and apidocs, but that is on track to be solved by next week. So no other tickets are in the blocker queue: https://twistedmatrix.com/trac/report/26 -- Do you know any other release blocker issues? Since we have good continuous testing, maybe we can stop doing release candidates. The only thing blocking a release would be a "release blocket" ticket and truck CI checks not being green. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Updating the project leadership committee
Thanks, Glyph The top posting worthy question is: Why else volunteers to be a member of the refreshed Twisted project leadership committee ? --- I volunteer to be a member of the committee. I can also take care of writing the required text to be forwarded to the existing committee for an up or down vote. I hope we don't need to bypass the current committee. Regards On Sat, 26 Dec 2020 at 22:12, Glyph wrote: > Hi Adi, > > The best next step would be to reach out to Conservancy directly—probably > Karen Sandler, based on what the website says now—and describe the > situation we are in. The problem is that the process we currently have in > place is that the old committee nominates the new one and as far as I’m > aware it doesn’t support any other options. The folks at the conservancy > are probably the only ones who can change this. > > The plans you propose all sound pretty reasonable to me, I just don’t have > the time or energy to do all the planning and coordinating required to make > them happen. > > If you did have a fully fleshed out, written up proposal for what to do > next, volunteers ready to be the next PLC, and all we needed was a PLC > up-or-down vote, I *might* be able to help by tracking down a majority of > the existing PLC and needling them to vote just once; however a better > solution would be to get a more sustainable process in place with the SFC > that takes this situation into account. > > I am *extremely *enthusiastic for your future reign on a new and improved > project leadership committee :-). Thank you for working on this! > > -g > > On Dec 16, 2020, at 3:33 AM, Adi Roiban wrote: > > > Hi, > > Based on a comment from Glyph > > > The current "project leadership committee" is more than half made up of > people who are a. no longer affiliated with the project and b. never answer > their email at this point, > > I started this discussion to see if we can update / refresh the project > leadership committee. > > I remember that in 2015 I read the same comment. > > Glypy, do you know the required number of members for the committee? > > My suggestion is to try to arrange for a committee that is updated every 2 > years... or maybe every year. > I can organize the communication and project management stuff. > > Here is the list of committers for the last to years > > > https://github.com/twisted/twisted/graphs/contributors?from=2018-12-16=2020-12-16=c > > This is a quantitative approach... and does not imply quality or > dedication, but I think that is a good starting point for searching for > candidates for the committee. > > I can also volunteer to be part of the committee. > > Maybe we can create a new list of members for the commitee and then see if > this list will be accepted by theSoftware Freedom Conservancy. > > I can also volunteer to do a bit of noise and see if we can raise more > money. > > I will not apply for the fellowship, so I hope that in this way we will > not have any complaints about my intentions :) > > PS: I have many patches for Twisted that are not commited. I could just > hire another person to "review" them, but I don't think that this is fair. > In fact, all the patches already passed an internal review, done by I > person paid by my company. > So, I hope that we can hire an independent reviewer > > -- > Adi Roiban > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Plan/Goal for GitHub Sponsors
Hi, I started a separate discussion to confirm the goal for a future fundraising. Top priority - Pay someone to help with the review queue Nice to have - Move server/services to Azure VM - We have at least $100 monthly allowance for Azure not sure if we still have the huge $2000 allowance on rackspace. Nice to have - Migrate Trac wiki to GitHub Wiki Nice to have - Migrate Trac Ticket to GitHub Issues That is all..but first we should solve the release queue. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Participating to GitHub Sponsors
Hi Thanks Barry for the feedback, I have pressed the green button... then then I was asked to enter more info and press more green buttons :) I wasn't aware that this is a clone of patreon.com ... I ended up with this https://github.com/sponsors/twisted?preview=true The page is not active as it asks for a Stripe payment API...but I think that is for individuals and not organizations. So we might need to get in touch the the Freedom Software Conservancy to sort this out. More info inline... On Thu, 17 Dec 2020 at 12:00, Barry Scott wrote: > On Wednesday, 16 December 2020 11:17:06 GMT Adi Roiban wrote: > > Hi, > > > > On Wed, 16 Dec 2020 at 02:49, Ian Haywood wrote: > > > > > No objection, but any plans for the money? I'd ask for more reviewers, > I'd > > > like to contribute more to twisted but there's little point without > access > > > to code review in a reasonable timeframe. > > > > > > > First we need funds... I am sure that we will find a way to spend it > well. > > It usually for a sponsor to ask what the sponsorship will be used for. > I'd assume that twisted projects needs a plan for how to spend the money > to share with the sponsor. > > I agree that this is backward. At this point we are not asking for money ... but will do soon. This is just a discussion for a future money raising effort. First question was whether to enable future donations via GitHub and then via Stripe and then via the Software Freedom Conservancy. I think that this is answered. I think that is worth exploring the fundraising option and also to check that we have any luck in getting a plan approved. There are a couple of issues that needs to be solved and the GitHub process can act as a guide... but we need: * 1. creating a plan * 2. finding someone to approve it * 3. finding a way to fund the plan Cheers Adi Barry > > > > > > Trying to see if we can solve the reviewing queue with money is one > > option... but is not that easy. > > > > I am available to review one of your PR in exchange for another review :) > > > > Regards > > > > PS: If you have any conch-ssh PR, send me a message and I will review it > > (for free... no reviewed asked in exchange) > > > > > > > On 16/12/2020 11:56 am, Adi Roiban wrote: > > > > > > Hi, > > > > > > Does anyone have anything against applying so that Twisted can be on > the > > > waitlist of the GitHub Sponsors ? > > > > > > https://github.com/sponsors/twisted/waitlist > > > > > > As far as I know, the money for the Twisted project is managed by the > > > Software Freedom Conservancy organization and they are already approved > > > hosts for GitHub Sponsors. > > > > > > At this point, we only need to press the green button. > > > The bank account and other administrative issues are already sorted > out. > > > > > > > > > > > > If there are no complaints , I plan to press the green button in 1 > week. > > > > > > We will then have to see how to access those funds via Software Freedom > > > Conservancy... but that is part 2. > > > > > > Cheers > > > -- > > > Adi Roiban > > > > > > ___ > > > Twisted-Python mailing listTwisted-Python@twistedmatrix.comhttps:// > twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > > > ___ > > > Twisted-Python mailing list > > > Twisted-Python@twistedmatrix.com > > > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > > > > > > > > > > > > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Updating the project leadership committee
Hi, Based on a comment from Glyph > The current "project leadership committee" is more than half made up of people who are a. no longer affiliated with the project and b. never answer their email at this point, I started this discussion to see if we can update / refresh the project leadership committee. I remember that in 2015 I read the same comment. Glypy, do you know the required number of members for the committee? My suggestion is to try to arrange for a committee that is updated every 2 years... or maybe every year. I can organize the communication and project management stuff. Here is the list of committers for the last to years https://github.com/twisted/twisted/graphs/contributors?from=2018-12-16=2020-12-16=c This is a quantitative approach... and does not imply quality or dedication, but I think that is a good starting point for searching for candidates for the committee. I can also volunteer to be part of the committee. Maybe we can create a new list of members for the commitee and then see if this list will be accepted by theSoftware Freedom Conservancy. I can also volunteer to do a bit of noise and see if we can raise more money. I will not apply for the fellowship, so I hope that in this way we will not have any complaints about my intentions :) PS: I have many patches for Twisted that are not commited. I could just hire another person to "review" them, but I don't think that this is fair. In fact, all the patches already passed an internal review, done by I person paid by my company. So, I hope that we can hire an independent reviewer -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Participating to GitHub Sponsors
Hi, On Wed, 16 Dec 2020 at 02:49, Ian Haywood wrote: > No objection, but any plans for the money? I'd ask for more reviewers, I'd > like to contribute more to twisted but there's little point without access > to code review in a reasonable timeframe. > First we need funds... I am sure that we will find a way to spend it well. Trying to see if we can solve the reviewing queue with money is one option... but is not that easy. I am available to review one of your PR in exchange for another review :) Regards PS: If you have any conch-ssh PR, send me a message and I will review it (for free... no reviewed asked in exchange) > On 16/12/2020 11:56 am, Adi Roiban wrote: > > Hi, > > Does anyone have anything against applying so that Twisted can be on the > waitlist of the GitHub Sponsors ? > > https://github.com/sponsors/twisted/waitlist > > As far as I know, the money for the Twisted project is managed by the > Software Freedom Conservancy organization and they are already approved > hosts for GitHub Sponsors. > > At this point, we only need to press the green button. > The bank account and other administrative issues are already sorted out. > > > > If there are no complaints , I plan to press the green button in 1 week. > > We will then have to see how to access those funds via Software Freedom > Conservancy... but that is part 2. > > Cheers > -- > Adi Roiban > > ___ > Twisted-Python mailing > listTwisted-Python@twistedmatrix.comhttps://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Participating to GitHub Sponsors
Hi, Does anyone have anything against applying so that Twisted can be on the waitlist of the GitHub Sponsors ? https://github.com/sponsors/twisted/waitlist As far as I know, the money for the Twisted project is managed by the Software Freedom Conservancy organization and they are already approved hosts for GitHub Sponsors. At this point, we only need to press the green button. The bank account and other administrative issues are already sorted out. If there are no complaints , I plan to press the green button in 1 week. We will then have to see how to access those funds via Software Freedom Conservancy... but that is part 2. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Injecting 400 errors via HTTP Proxy
Hi Vlad On Fri, 4 Dec 2020 at 02:11, Vlad Petric wrote: > Hi Everyone, > > I'd like to randomly inject 400 errors via the HTTP Proxy. > > I'm testing the robustness of an application that performs a lot of HTTP > GETs. I am using the basic http web proxy, and I need to modify it to > randomly insert some errors. > > Should I be using my own class derived from ProxyRequest, and override > process()? If that's the case, how would I be injecting 400s? > > Thank you so much! > Vlad, a twisted n00b > ___ > Is this a forward or reverse proxy? ProxyRequest is just like any request... you can set headers, response code, write content to it... Here is an example: https://gist.github.com/adiroiban/ef9dfd2cce5c59b70e1bc0600b58adeb -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Re-run Azure tests
Hi, On Mon, 19 Oct 2020 at 05:47, Wilfredo Sánchez Vega wrote: > Yeah, log in with GH has never worked for me. Always results in what > claims to be a transient error. > > They worked for me and I like them more than buildbot logs :) Maybe we can try something to reduce the load for logs. For example the run report into a file to contain only the failed + todo. Use a separate step to show the content...have that step fail on errors. When you click on a build, it should get you to the short summary. Cheers > -wsv > > On Oct 13, 2020, at 9:40 PM, Glyph wrote: > > > > On Oct 13, 2020, at 4:26 PM, Adi Roiban wrote: > > Thanks. So you need to be logged with github account and not your normal > Azure/Devops account. > I guess most of us are connected to Azure using the Azure AD account. > > I was not expecting to see "Login via Github" on the MS login page. > > After the Github login, the github re-run works. It automatically re-runs > all the failed tests...not only a single job. > > > After reading this I tried for like 20 minutes to try and figure out what > microsoft is doing when I "log in"; it seems it's figured out my azure > account *is* my github account, I could have sworn I had 2 but now I only > have 1. > > Not sure how one gets into this situation though, and the fact that large > parts of the MS / live.com auth stack are just broken (right now the > logout button doesn't work at all, I have to delete all my cookies) makes > this tricky to debug. So I'm giving up now :). > > -g > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] revert / rework of "AMPv2" change?
On Mon, 19 Oct 2020 at 18:18, Glyph wrote: > I just glanced at https://github.com/twisted/twisted/pull/1417 after it > was merged and noticed a few problems: > > - It exposes a new top-level name rather than setting the protocol version > as a parameter > - The NEWS files are malformed which is going to lead to some confusing > duplication in the changelog > - AMPv2 itself does not appear to be "standardized" - the page at > https://amp-protocol.net/conversations_v2.html appears to be a > preliminary suggestion for how long values might be handled rather than > something broadly implemented; for one thing, I'd never heard of it before > and for another the site itself doesn't link the document. This would be > fine as a first draft, but I think it needs to be given some more > revisions, given that (for example) there are a number of > backwards-compatible ways to do this. > - The layering is wrong because it puts the protocol-parsing into a leaf > class in the hierarchy, when the parsing logic was deliberately isolated to > a lower level to facilitate different framing mechanisms. > - As JP pointed out, the tests have a potential bug where they can leak > errors between cases. > > I don't normally like to revert folks' work once it's been reviewed and > accepted, particularly when there's no process violation (broken CI, lack > of code coverage etc), but I'm particularly concerned about lending > Twisted's imprimatur to this protocol extension as a whole new version > without much more context on who is implementing it and what other options > were considered, particularly when I personally (nominally the inventor of > AMP) don't like the direction it has taken :). > > What do other folks think? Anyone else have more of a finger on the pulse > of where the "v2" conversation has been happening? > > Just my 2 cents I think that whether this should be reverted or reworked should be between the person raising this issue (Glyph) and the author of the PR (Jonathan). I see that Jonathan has responded to the feedback inside the merged PR.I hope it can be reworked. Since AMP v2 is not standardized maybe is ok to have it as a separate experimental top level name. I have never used AMP ...I tried to use it via ampoule but ended up using the stdlib process pool as I was not able see any performance difference. Since `amp` is used by `trial -j` we need to keep it inside twisted. But since AMP v2 is experimental, I think that is best to have it as a separate txamp2 project. In this way, the twisted AMP v2 project can follow its own rules. Once AMP v2 is "mature" it can be moved to core twisted So my take is to revert it and move it to twisted/txamp2. BTW this is also my suggestion for the PR trying to introduce support for SMB have twisted SMB implementation in a separate project. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Re-run Azure tests
On Mon, 12 Oct 2020 at 11:30, Adi Roiban wrote: > Hi, > > There was a discussion over IRC about re-running failed Azure tests. > > I was confused, as if I go to the Azure build page, I don't see any re-run > job option. > > Maarten told me that the re-run link is on the GitHub page, and not on > Azure. > > See the screenshot below. > > I hope this helps :) > > Cheers > > It looks like the re-run feature is broken. I guess that we need to move all the tests that require re-run to GitHub Actions. Tests that don't require a re-run (documentation build or linters) can be kept or moved to Azure to continue having many parallel builds. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Re-run Azure tests
Hi, There was a discussion over IRC about re-running failed Azure tests. I was confused, as if I go to the Azure build page, I don't see any re-run job option. Maarten told me that the re-run link is on the GitHub page, and not on Azure. See the screenshot below. I hope this helps :) Cheers [image: image.png] -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Request for new Twisted release?
On Sun, 4 Oct 2020 at 01:14, Glyph wrote: > > > On Oct 3, 2020, at 4:15 PM, Craig Rodrigues > wrote: > > > > On Wed, Sep 23, 2020 at 10:11 AM Glyph wrote: > >> >> >> On Sep 22, 2020, at 3:42 PM, Craig Rodrigues >> wrote: >> >> Amber, >> >> Can we have a new Twisted release within the next two months, say in >> Nov./Dec. timeframe, or >> sooner if you'd like? >> >> >> Given that Amber hasn't been super responsive lately, and that I firmly >> believe it should be possible to take breaks from open source (particularly >> amidst the, you know, global collapse of civilization and stuff), I will >> take the opportunity point out yet again that >> http://twisted.readthedocs.org/en/latest/core/development/policy/release-process.html >> documents >> the release process and any sufficiently-motivated volunteer (with at least >> a *bit* of history & trust within the community) can step forward and >> receive all the permissions necessary to do all of these steps. >> >> I will personally volunteer to fill in the permissions gaps for any >> interested parties, although I don't have much bandwidth for the release >> itself. >> >> Amber, if you want to do it, go ahead and get started, I'm just saying we >> don't need to put this on your shoulders :). >> >> > I haven't heard any response from Amber, so I assume that Amber is busy > (and I hope is OK during all the pandemic). > > > I have heard from Amber occasionally and I believe she's healthy and safe, > but if she wants to share more than that it's her own business :). (Are > *any* of us really "OK"?) > > If it is OK with Glyph and Amber, I can focus on pushing out the next > Twisted release, and hopefully get help from Adi. > > > Absolutely OK with me. More than OK. Please do this! I can't wait until > we have the level of automation you've been working on for *anybody* to > cut the release. I take it this means you'll officially be the release > manager for 20.10.0? > > Let me know if you need any more permission bits to make progress. > > I'm working now with Adi to automate more of the release actions via > GitHub Actions, and have gotten pretty far, > such as being able to build wheels of Twisted on GitHub Actions for Linux, > MacOS, and Windows. > > > This is fantastic news! I really appreciate all the effort you're putting > in to simplify and streamline the CI situation. > > > I have a proof of concept PR - https://github.com/twisted/twisted/pull/1422 Read the Docs version of Twisted containing the full documentation - narrative + API. You can check it here - https://twisted--1422.org.readthedocs.build/en/1422/ With API link in the main page and at https://twisted--1422.org.readthedocs.build/en/1422/api/index.html --- I would say that for the first weeks after the release, we can have the documentation only on Read The Docs with the read the docs theme... but it can be confusing if we don't also update https://twistedmatrix.com/documents/current/ docs. Later we can enable the Twisted own theme.. if we want. I think that many seach indexes are using https://twistedmatrix.com/documents/current/api/twisted.html So after the release we should decide if we should mirror the docs from Read The docs to the Twisted own VM... or create a reverse proxy, or create a redirection. -- All this Read the Docs hosting is implemented as it doesn't require any extra permissions and is already integrated with the VCS. We can automatically build the docs via GitHub actions, but the problem is how we get them to the Twisted main server. Allowing GitHub actions to push random stuff can be a security issue... and implementing a restrictive SFTP server ono the Twisted main server requires extra work. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted with pypy3 on Ubuntu 20.4, SSL error: 'ee key too small'
On Tue, 29 Sep 2020 at 23:25, Craig Rodrigues
wrote:
> Hi,
>
> I ran an experiment to run the Twisted tests on Ubuntu 20.4 using this
> version of using pypy3:
>
> *Python 3.6.9 (2ad108f17bdb, Apr 07 2020, 02:29:05*
> *Pypy 7.3.1 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)*
>
>
> A bunch of the conch tests failed in this Ubuntu environment:
> https://github.com/twisted/twisted/runs/1173397508
> like this:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *===Error:
> Traceback (most recent call last): File
> "/home/runner/work/twisted/twisted/build/alldeps-withcov-posix/site-packages/twisted/test/test_sslverify.py",
> line 1780, in test_ellipticCurveDiffieHellmanonData=onData, File
> "/home/runner/work/twisted/twisted/build/alldeps-withcov-posix/site-packages/twisted/test/test_sslverify.py",
> line 716, in loopbackself.serverPort = reactor.listenSSL(0,
> serverFactory, serverCertOpts) File
> "/home/runner/work/twisted/twisted/build/alldeps-withcov-posix/site-packages/twisted/internet/posixbase.py",
> line 593, in listenSSLtlsFactory =
> tls.TLSMemoryBIOFactory(contextFactory, False, factory) File
> "/home/runner/work/twisted/twisted/build/alldeps-withcov-posix/site-packages/twisted/protocols/tls.py",
> line 748, in __init__contextFactory =
> _ContextFactoryToConnectionFactory(contextFactory) File
> "/home/runner/work/twisted/twisted/build/alldeps-withcov-posix/site-packages/twisted/protocols/tls.py",
> line 629, in __init__oldStyleContextFactory.getContext() File
> "/home/runner/work/twisted/twisted/build/alldeps-withcov-posix/site-packages/twisted/internet/_sslverify.py",
> line 1636, in getContextself._context = self._makeContext() File
> "/home/runner/work/twisted/twisted/build/alldeps-withcov-posix/site-packages/twisted/internet/_sslverify.py",
> line 1645, in _makeContextctx.use_certificate(self.certificate) File
> "/home/runner/work/twisted/twisted/build/alldeps-withcov-posix/site-packages/OpenSSL/SSL.py",
> line 960, in use_certificate_raise_current_error() File
> "/opt/hostedtoolcache/PyPy/3.6.9/x64/lib_pypy/_functools.py", line 80, in
> __call__return self._func(*(self._args + fargs), **fkeywords) File
> "/home/runner/work/twisted/twisted/build/alldeps-withcov-posix/site-packages/OpenSSL/_util.py",
> line 54, in exception_from_error_queueraise
> exception_type(errors)OpenSSL.SSL.Error: [('SSL routines',
> 'SSL_CTX_use_certificate', 'ee key too
> small')]twisted.test.test_sslverify.OpenSSLOptionsECDHIntegrationTests.test_ellipticCurveDiffieHellman===*
>
>
>
> I'm not sure what the root problem is, but my suspicion is that the SSL
> library in this particular Ubuntu environment
> does not permit keys smaller than 2048 bits.
>
> Anyone have any other ideas as to the cause of this error?
>
>
Hi,
That is the case.
One work around is append @SECLEVEL=0 to the cipher list but I think
it's better to update the tests to use 2048 bits key.
--
Adi Roiban
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Retry: Conch Issues
Hi
On Fri, 25 Sep 2020 at 00:15, Robert DiFalco
wrote:
> I am having troubles with Conch. I've been able to reproduce them in a
> unit test and all the code is here along with requirements.txt:
>https://github.com/radifalco/conch_issues
>
> I'm trying to reuse my connection, but if it is idle too long the server
> shuts it down. What I don't understand is why the next call to
> `FileTransferClient` does not call back or errback? It will just hang
> forever even though the Factory and Transport know the connection was
> closed (as evidenced by the log messages).
>
> I don't ever want to hang, if I use FileTransferClient I would like it
> always call a callback or errback. I feel like Conch is missing a deferred
> somewhere but damned if I can find it.
>
> Thanks!
>
> Have you checked
https://twistedmatrix.com/documents/current/core/howto/clients.html#reconnection
?
I see that `connector.connect()` is commented in your code.
The theory is that once a client connection is lost, you will need to call
again the whole chain starting with reactor.connectTCP.
-
I know the pain here with SFTP as there is a lot of wrapping: TCP client,
SSH high level protocol, SSH session, SSH channel, SSH subsystem.
One option is to use MySSHClientFactory as a proxy for any SFTP operation.
When the SSH channel is open and you have received the response from the
subsystem request, pass the SFTP client all the way down to the factory.
The factory will know when the connection is lost and then trigger a new
connection... which will update the SFTP client instance on the factory.
That is, instead of getting the SFTP client, the SFTP client will be pushed
to the factory.
I think that the main issue with your code is that it tries to work
directly with the SFTP client, instead of using the factory as a proxy for
SFTP operations.
A protocol represents a single connection that is already made and it can't
reconnect itself.
The factory is responsible for handling new connections, or triggering new
connections.
So, your high-level API is using a very low-level object.
The SFTP client represents an SSH connection that was already made,
authenticated and the SFTP subsystem requested.
For my production code I am doing something like this:
MySSHClientFactory is instantiated with the reactor and the
SFTPClientOptions.
Then, when I want to list a directory I call
sftp_factory.listDir('some/path')
The factory can then check if it needs to get an SFTP client or reuse an
existing one.
A SFTP client is obtained by triggering a new connectTCP to the reactor
with `self` as the factory.
Does it make sense?
--
Adi Roiban
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Request for new Twisted release?
On Wed, 23 Sep 2020 at 18:11, Glyph wrote: > > > On Sep 22, 2020, at 3:42 PM, Craig Rodrigues > wrote: > > Amber, > > Can we have a new Twisted release within the next two months, say in > Nov./Dec. timeframe, or > sooner if you'd like? > > > Given that Amber hasn't been super responsive lately, and that I firmly > believe it should be possible to take breaks from open source (particularly > amidst the, you know, global collapse of civilization and stuff), I will > take the opportunity point out yet again that > http://twisted.readthedocs.org/en/latest/core/development/policy/release-process.html > documents > the release process and any sufficiently-motivated volunteer (with at least > a *bit* of history & trust within the community) can step forward and > receive all the permissions necessary to do all of these steps. > > I will personally volunteer to fill in the permissions gaps for any > interested parties, although I don't have much bandwidth for the release > itself. > > Amber, if you want to do it, go ahead and get started, I'm just saying we > don't need to put this on your shoulders :). > > Hi, To me, the current release process seems complicated and requires manual work and I think that the release document needs some update. The release documentation starts with "Tarballs for Twisted as a whole, and for each of its sub-projects". Is that still the case? Maybe the release can be simplified and automated: * Build the source and binary wheel in GitHub actions as this can be done for Linux (Ubuntu), Windows and macOS. * Host the download source and wheel files only on PyPi and publish them automatically from GitHub action on a new tag is created. * Host the documentation only on the Read The Docs. * Hos the API documentation on Read The Docs - might need some hacking, but at release we can create an API docs package as an artifact which is then pulled when Read The Docs documentation is created and copied as extra HTML files. * Move Twisted blog to GitHub pages... or even read the docs with a separate theme like Crate [1] ... if the blog is still required. -- With the above implemented the release should look like: * Each time the tests for a PR are executed, pydoctor will run and automatically create the API files as an artifact available for download. * *Manually* create a new branch in which the version is updated and the changelog/news/release note is created and all the news fragment files are removed/ . Have the branch reviewed and approved with all the tests passed. When the pydoctor tests are executed, the API docs are created. * The release branches can have a naming convention line 'release-20.0.0`. At first the release branch can have a release candidate version and a GitHub can automatically push the release to PyPi. * *Manually* send an email to Twisted mailing list to announce the pre-release. * Once the branch is merged, *manually* push a new tag * The new tag should trigger the GitHub action for publishing the release on PyPi * The new merge in master should trigger the Read The Docs build... which now will also include the static API docs pages. * Once the release is done, *manually* send a new email to the mailing list. There are still many manual steps, but the only permissions required is commit to the repo. This will no longer use any of the Twisted own infrastructure. [1] https://sphinx-themes.org/html/crate-docs-theme/crate/basic.html -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Conch SFTP Questions
Name, > data=data, > chunk_size=getattr(self.options, "chunkSize", sftp.CHUNK_SIZE))) > > Where sftpClient reuses the existing SSH connection if it is active > (rather than logging in each time). But maybe the sftp service doesn't > multiplex so I have to create a new SSHClientFactory every time I want to > send a distinct file? > > Sorry for all the questions, new to twisted and a bit confused. Thanks! > > Robert > > It would help to have the full code...maybe a gist or repo. I am not sure what `connect` from `yield connect(client_options.host, client_options.port, options, _verify_host_key, auth)` is. You will need to understand the low-level Twisted connection API and implement a reconnecting factory. When a new client-side connection is made, Twisted will use a factory to create the protocol/code used to handle that connection, You will then need to hook into the connectionLost method and do an auto-connection if the connection is lost (when you were not expecting it). - For my project, I am doing in this way: I have my own subclass of FileTransferClient which overwrites the default FileTransferClient,connectionLost method. With that, I am notified when the SFTP subsystem was closed and I can then trigger a new connection - If you want to reuse an SFTP session for multiple operations just reuse the `sftpClient` instance that you got to trigger multiple operations sftpClient = yield conn.getSftpClientDeferred() for file_info in list_of_files_to_send: yield _send_file(sftpClient, file_info) -- Hope it helps -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Only run Azure, Travis, or CircleCI if GitHub actions pass
On Sat, 19 Sep 2020 at 21:30, Craig Rodrigues wrote: > Adi, > > Is it possible to configure the CI so that > Azure, Travis and CircleCI only runs if GitHub actions pass? > > For example here: > > https://github.com/twisted/twisted/runs/1137175269 > > *Azure Pipelines* and *Travis* should only run if the GitHub actions pass. > > I think that is partially possible...but complicated. My question: Why does Twisted need Azure Pipelines (sharing infastructure with GihutActions), Travis-CI and Circle-Ci. Why can't Twisted just use GitHub actions? You can trigger one GitHub action workflow from another workflow, or everything as depende jobs inside one workflow. GitHub Action API is pretty good. I was able to implement a quick hack which sends a diff to GitHub and executes a test without a commit, just like Builbot Try [3]. I don't know how to implement something similar with Travis or Circle-Ci. - Travis-CI builds can be triggered over the HTTP API [1]...but I feel that implementing it is complicated. There should be a dedicated token... which should be created by a person, and then stored as a secret in GttHub Actions. I don't know how to trigger Circle-CI .. here is some info here [2] but looks like it will be deprecated I don't know how to trigger Azure Pipelines. but I guess that at some point MS will kill Azure Pipelines [1] https://docs.travis-ci.com/user/triggering-builds/ [2] https://circleci.com/docs/2.0/api-job-trigger/ [3] https://asciinema.org/a/nn1U7FLtbyWm57b0dT7U5sKD0 -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Change Trac /login link to point to /trac/github/login
Hi, On Sun, 30 Aug 2020 at 17:11, Craig Rodrigues wrote: > Where in https://github.com/twisted-infra/braid/ does this change need to > be done? > > I'm not familiar with how braid works. > > I have created a PR here https://github.com/twisted-infra/braid/pull/263 I think that this is a trac-github plugin/extension issue which was fixed https://github.com/trac-hacks/trac-github/issues/95 The link is created by the extension here: https://github.com/trac-hacks/trac-github/blob/master/tracext/github/__init__.py#L136 I hope someone could review and apply the change in prod. Cheers > -- > Craig > > On Sat, Aug 29, 2020 at 11:30 PM Glyph wrote: > >> If you can express this as a PR to Braid, I can promise you I'll merge & >> apply it, but that's probably the maximum energy I can personally afford to >> dedicate to it :). >> >> On Aug 29, 2020, at 10:56 AM, Craig Rodrigues >> wrote: >> >> If I am not logged in, and click on: >> >> https://twistedmatrix.com/trac/newticket >> >> It takes me to a page: >> >> You are currently not logged in. You may want to do so >> <https://twistedmatrix.com/trac/login> now. >> >> If I click on *do so*, it takes me to: >> >> No handler matched request to /login >> >> Can this be fixed on the Trac side so that */login* redirects to: >> https://twistedmatrix.com/trac/github/login ? >> Or change *do so* to hyperlink to */trac/github/login*? >> >> This is a stumbling block when new contributors are asked to open a >> ticket in Trac. >> >> -- >> Craig >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> >> >> ___ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >> > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] mypy integrated with CI for twisted
On Wed, 24 Jun 2020 at 13:48, Jean-Paul Calderone wrote: > On Wed, Jun 24, 2020 at 12:44 AM Glyph wrote: > >> >> >> On Jun 23, 2020, at 5:34 AM, Adi Roiban wrote: >> >> Hi Craig, >> >> On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues >> wrote: >> >>> I have merged some more fixes for mypy to Twisted trunk branch. >>> >>> In trunk, you can run mypy with: >>> >>> *tox -e mypy* >>> >>> Currently this results in *171* errors, which is way down from >1000 >>> errors >>> a month ago. >>> >>> In addition, if you look at any new PR's there is a *Mypy Ubuntu* job >>> running on Azure pipeline, which runs mypy. Right now errors from this >>> job >>> are ignored and does not block the PR. However, if we can get the mypy >>> errors down to zero, we can make mypy status a blocker for the PR. >>> >>> >> Thanks for working on this. >> >> Looking forward to have a real green mypy build. >> >> A general question: Why Twisted used Azure Devops and not GitHub actions? >> >> >> Azure Pipelines gave us substantially more parallel capacity than is >> available via Github Actions, which means we can make build statuses appear >> much sooner. Plus they support more platforms. >> > > Does Twisted have a special deal with Azure Pipelines? Or is the use of > past-tense in this sentence intentional? :) Or are the docs for the > respective platforms wrong/misleading? > > > https://docs.microsoft.com/en-us/azure/devops/pipelines/licensing/concurrent-jobs?view=azure-devops > says > free-tier public projects get 10 parallel jobs. > > > https://help.github.com/en/actions/getting-started-with-github-actions/about-github-actions#usage-limits > says > free tier projects get 20 parallel jobs. > > (Of course this says nothing about the number of supported platforms.) > > My understanding is that GitHub actions are free for public repositories. My suggestion is to use Azure Pipelines and Travis for the main trial tests and use Circle-CI or GitHub Actions for the other tests. GitHub Actions has a nice integration with a GitHub PR and you can check the results without having to navigate to a different page. And with GitHub actions you can add any new workflow without extra permission to Azure Devop. With GitHub actions for free and available on LInux/Windows/macOS , I am not sure if keeping Circle-Ci makes sense. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] mypy integrated with CI for twisted
Hi Craig, On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues wrote: > I have merged some more fixes for mypy to Twisted trunk branch. > > In trunk, you can run mypy with: > > *tox -e mypy* > > Currently this results in *171* errors, which is way down from >1000 > errors > a month ago. > > In addition, if you look at any new PR's there is a *Mypy Ubuntu* job > running on Azure pipeline, which runs mypy. Right now errors from this job > are ignored and does not block the PR. However, if we can get the mypy > errors down to zero, we can make mypy status a blocker for the PR. > > Thanks for working on this. Looking forward to have a real green mypy build. A general question: Why Twisted used Azure Devops and not GitHub actions? With Github Actions you can add a workload / pipeline with only GitHub access. There is no need for extra Azure permissions. I guess that with Github actions, if we run all the tests on Linux/Windows/MacOS we will very soon run out of time :) But maybe we can have the linters and other quick tests there. --- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] SMB server component for twisted
Hi Ian On Tue, 5 May 2020 at 05:17, Ian Haywood wrote: > I have begun work on a SMB (Server Message Block; Windows filesharing) > server protocol for twisted. > > Work so far is here: https://github.com/ihaywood3/twsmb > > I'm looking for any advice particularly around what I should be doing so > this code is suitable for inclusion in twisted. > > Ian Haywood Not sure if you want the repo included in the Twisted GitHub organization or the twisted/twisted repo? I think is best to have the SMB protocol implementation as a separate repo to be able to release at any time, but it can be added to Twisted organization. This is similar to the LDAP ldaptor repo. I don't think there are any rules for adding repos to the GitHub organization. I would say, as a minimal requirement are: * end user documentation * dev documentation / CONTRIBUTORS file * automated tests / CI I have a long term goal for SMB3 protocol implementation (client and server side). I am currently considering reusing https://github.com/jborean93/smbprotocol The network communication is not ideal... but the design is not that bad. That repo already has the Python code for parsing SMB messages. Regards ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] ldaptor problem with STARTTLS and python 3
On Wed, 29 May 2019 at 08:21, Thomas Westfeld wrote: > [snip] > Dear all, > > it turns out to be an issue with binary vs. string. The __str__ method was > called by the errback only. This has been fixed in pull request #147 already. > All the observed errors have been fixed in the pre-release 19.1 version of > ldaptor. The example is working with STARTTLS when using HEAD. > > Thanks for the great community for fixing things so quickly. > > I am looking forward to the 19.1. release. > Hi Thomas, If you have time, you can check https://github.com/twisted/ldaptor/pull/148 I guess that once that PR is merged we can make a new public release Are you testing this on Python2 or Python3? -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] ldaptor problem with STARTTLS and python 3
On Tue, 28 May 2019 at 11:22, Thomas Westfeld wrote: > > Dear all, > > I am experiencing an error when trying to use the quick usage example posted > on the ldaptor github page https://github.com/twisted/ldaptor . I have > included the starttls command to promote the connection to an encrypted one, > but I am experiencing an deprecation warning: > > DeprecationWarning: LDAPStartTLSInvalidResponseName.__str__ method is > deprecated and will not be used for getting bytes representation in the > future releases, use LDAPStartTLSInvalidResponseName.toWire instead > category=DeprecationWarning) > > This puzzles me as I cannot track this error down. The usual strategy to > introduce debugging print statements or start the debugger like "import pdb; > pdb.set_trace()" does not work. I would appreciate help here how to debug > this deferred code. > > Besides this warning I am receiving no output whatsoever. Also print > statements in the example method do not print anything, which is strange to > me. > > To debug it on a networking level, I have wiresharked the ldap communication > and it seems fine (see below): > > REQUEST > > Transmission Control Protocol, Src Port: 53964, Dst Port: 389, Seq: 1, Ack: > 1, Len: 31 > Lightweight Directory Access Protocol > LDAPMessage extendedReq(1) > messageID: 1 > protocolOp: extendedReq (23) > extendedReq > requestName: 1.3.6.1.4.1.1466.20037 (LDAP_START_TLS_OID) > [Response In: 2587] > > RESPONSE > > Transmission Control Protocol, Src Port: 389, Dst Port: 53964, Seq: 1, Ack: > 32, Len: 46 > Lightweight Directory Access Protocol > LDAPMessage extendedResp(1) > messageID: 1 > protocolOp: extendedResp (24) > extendedResp > resultCode: success (0) > matchedDN: > errorMessage: > responseName: 1.3.6.1.4.1.1466.20037 (LDAP_START_TLS_OID) > [Response To: 2581] > [Time: 0.24000 seconds] > > I have posted the example code with the starttls command in this gist: > > https://gist.github.com/westfeld/bb1d5e8727ce13910623933e041e9782 It looks like a bug. With the Python3 migration the usage of the implicit __str__ for serialization/marshaling of the LDAP messages was removed. Please add a new issue/ticket/bug in Github for ldaptor. I am debugging these things by patching the ldap code and adding a PDB call here https://github.com/twisted/ldaptor/blob/d3c191921bd32c359a3e4974c2251f9af852ab2b/ldaptor/_encoder.py#L56 then you can go up and see where str is called, and consider if it should be repalced with .toWire() Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] A Proposal for reducing the burden of developing on Twisted by dropping Python 2 support
gt; compatible software using Twisted would be able to use the older Twisted > version on Python 2, and the newer version on Python 3, as you would usually > expect. > > > We have 2 big blockers here right now that would prevent doing this as things > stand right now: > > Twisted is not yet fully ported to Python 3, so there's no version where you > can use all of Twisted on Python 3. This picture is way, way smaller than it > has ever been, though - grab a module and start porting: > http://blog.habnab.it/twisted-depgraph/ > We still have Python 2 in production ourselves that requires Twisted; > specifically: > > Trac and several of our ancillary utilities around it. Happily this does > seem to be fairly actively worked on: "opened 4 years ago, last modified 12 > days ago" https://trac.edgewall.org/ticket/12130 Do we really need Trac? Maybe is time to move the tickets and wiki to GitHub. > Dogfood DNS: twisted.names doesn't work on python 3: > https://twistedmatrix.com/trac/ticket/9496 I have never used Twisted.names in production. I tried to help with the review of twisted.names patches, and I remember there was not much interest for twisted.names So maybe we can get rid of it and move it to a separate project. I don't think that we need to have everything in twisted core... if ldaptor is an external project, why we can't have twisted.names > Our front-end webserver might work on pypy3, but we have yet to move it over. > Maybe that would be a good place to start? > > > If this proposal lights a fire under some folks to drive any of these > projects to completion, that would be great! My hope is that this proposal will light a fire and people using Twisted will more about Twisted. I feel that Twisted stability and reduce number of defects is not helping the open source dev model. People will use Twisted, it just work and they forget about it. Or if it doesn't work, they can fix something on their project relatively easy...but takes a lot of effort to send upstream..so they will never put the effort required to have a fix upstream. > You can find the proposal here, in this handy-dandy Google Doc: > https://docs.google.com/document/d/1S4CGgZC09blLIdk3Zo7wBa75A9_JuuH_3akkyjN0lik/edit > > Comments are welcome, as well as which timeline seems reasonable. > > > Personally I feel like Option 1 is the most reasonable. We can't do option 3 > because of the above blockers, and option 2 just seems random to me - why > commit to 4 months of additional maintenance beyond when py2 itself is EOL? > Maybe there's an option like, 1.5, where we de-support py2 at max(2020-01-01, > "the date at which we no longer have any py2 code in production ourselves")? If we keep the current prod infrastructure, I am not confident that we will no longer have py2 code in prod by the end of this year. I don't feel that the current Twisted team have resource to maintain a custom infrastructure. I think that things will be better for everyone if for example of Twisted names we will just use the credit on Azure to keep the DNS and use more Software as a Service so that the team will spend more time on developing new feature than doing sysadmin work. Dogfooding is a good thing, but in this case, I fell that it does not help. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] A Proposal for reducing the burden of developing on Twisted by dropping Python 2 support
On Mon, 25 Mar 2019 at 10:15, Amber Brown wrote: > > Hi everyone, > > Since the Python 2 EOL date is rapidly approaching, I thought it was > time we consider dropping Python 2 support. > [snip] > You can find the proposal here, in this handy-dandy Google Doc: > https://docs.google.com/document/d/1S4CGgZC09blLIdk3Zo7wBa75A9_JuuH_3akkyjN0lik/edit > > Comments are welcome, as well as which timeline seems reasonable. > > - Amber Many thanks Amber for putting this together. I would vote for option 3. ... as it will take some time to clean the code. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] giving a reason when adding a timeout
On Wed, 13 Mar 2019 at 16:07, Chris Withers wrote: > > >> Unless I'm missing something, Deferred.addTimeout is really unhelpful in > >> terms on providing context about *what* timed out. > >> TimeoutError(, 'Deferred') just isn't that useful. > >> > >> How come addTimeout doesn't let you specify a textual reason, or > >> otherwise provide some context about the timeout? > >> > >> Am I missing something obvious here? > > > > I don't know what kind of context do you need :) > > foo.addTimeout(myTimeout, reactor, message='never heard back') > > > You can add your own errback and add your extra error handling there. > > What's the must succinct way to get the above by adding an errback? > That is subjective :) You can try something like this. def eb_timeoutLog(failure, message): failure.trap(TimeoutError) print(message) deferred = someJob() deferred.addTimeout(10, reactor) deferred.addErrback(eb_timeoutLog, message='never heard back') or like this def eb_timeoutLog(failure, message): failure.trap(TimeoutError) print(message) def add_timeout(deferred, timeout, message): deferred.addTimeout(10, reactor) deferred.addErrback(eb_timeoutLog, message) deferred = someJob() add_timeout(deferred, message='never heard back') > > There is also onTimeoutCancel argument. > > Pretty clunky to write a whole function for either this or the errback > just to change the useless 'Deferred' string in the current > implementation with something useful. For production code I add errbacks anyway. The errback catches various error conditions and does more than just printing a text. I am not missing anything here :) > > If there is anything missing from the API, feel free to send a pull > > request in GitHub. > > What's the likely turn around time from me submitting a PR to when it > ends up in a released version of Twisted that I can use? I don't you the answer to that question. If this is something that you want to be improved, my suggestion is to give it a try. The whole Twisted was created by people who had an issue and were willing to effort required to share the solution with the rest of the world :) > Chris -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] giving a reason when adding a timeout
Hi, On Wed, 13 Mar 2019 at 11:21, Chris Withers wrote: > > Hi All, > > Unless I'm missing something, Deferred.addTimeout is really unhelpful in > terms on providing context about *what* timed out. > TimeoutError(, 'Deferred') just isn't that useful. > > How come addTimeout doesn't let you specify a textual reason, or > otherwise provide some context about the timeout? > > Am I missing something obvious here? I don't know what kind of context do you need :) You can add your own errback and add your extra error handling there. There is also onTimeoutCancel argument. If there is anything missing from the API, feel free to send a pull request in GitHub. I guess that it was implemented in this way as the current implementation serves the purpose for the people who have added the support for timeout. Regards, Adi > Chris > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] why can't a callback be called with a deferred?
On Tue, 19 Feb 2019 at 11:01, Chris Withers wrote: > > Hi All, > > There's this assert: > > https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/defer.py#L459 > > ...and I'd like to understand why it's there. [snip] > Unfortunately, the commit that introduced that assert is just a one > liner from Itamar back in 2003 without any context of what problem lead > him to introduce the change. I think it was introduced to catch some common bad usage patterns ... like yours :) If you want to chain the deferreds, use the dedicated helper https://twistedmatrix.com/documents/current/core/howto/defer.html#chaining-deferreds Deferred are not always 100% resolved/called. You might have a deferred called, but the current result might be another deferred... so it has no final result yet. so in your case, instead of `returnValue(result)` use result = yield result returnValue(result) in this way, the result is resolved :) Hope it helps -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Content length of a response in treq
On Sun, 10 Feb 2019 at 18:23, Mehtab Zafar wrote: > > Hi, > I am trying to use treq for a project in which I am also going to generate a > progress bar for the downloading session. But the problem I am facing is that > I can't find a way to get the content length header from the response. > > The "response.length" gives me "unknown size".So I there a way I can get the > length for all types of content? > > Thanks Hi, Which encoding is used for your response? Do you have Chunked encoding https://en.wikipedia.org/wiki/Chunked_transfer_encoding In this case, the content length is not advertised in the response headers. Regards, -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Can transport.write() to hostname instead of IP address?
On Fri, 17 Aug 2018 at 01:25, Sean DiZazzo wrote: > > Hi all! > > After I start a reactor connecting to a specific hostname and port, I do my > thing and then call transport.write() to send the data to the peer. > > From what I can tell, though, the hostname is resolved, and the data is > written back to the ip address itself, instead of the hostname I started the > reactor with. > > This is a problem in my case because we are using nginx's ssl_preread > server_name directive to route several different streams all coming in on the > same ip address. > > So the write() method needs to explicitly use the hostname to route the > packet properly. > > So... Is there any way to have transport.write() use the hostname given > instead of it's resolved IP address? Or am I missing something? > I assume you are using TCP here. I guess that you are missing something. If you want each write to go over its own way / route and have the hostname re-resolved you should open + write + close a connection for each write. But I think that there is something else there and this is now what you want :) Do you use HTTP or have a custom protocol? Cheers, Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twistedmatrix.com: scheduled maintenance July 18th
On Thu, 12 Jul 2018 at 12:57, Jean-Paul Calderone wrote: > > On Wed, Jul 11, 2018 at 11:14 PM Mark Williams wrote: >> >> Note that this is the minimum maintenance necessary to keep >> twistedmatrix.com running. We need money to realize improvements to our >> infrastructure, like newer hardware for the existing server and additional >> hosts that could continue to serve twistedmatrix.com during a maintenance >> window. > > > DNS could be pointed at an S3-backed website to serve a maintenance message, > at least. Not optimal, but better than ECONNREFUSED I would think. I'd be > happy to set this up and foot the bill - if someone provides the content (I > can make web pages but no one really wants that). > > Jean-Paul > Do we need fancy content? I was thinking to just use a plain text message explaining what is going on there. There is also the option to host in Azure. Twisted has $120 each month for Azure services... so there will be no extra payment. Let me know if you want/need access to the Twisted Azure subscription. Azure Blob does not have a "default page"...so you need to set up a CDN to serve a static page out of a container. There is one easy way to create a site from a GitHub repo with Azure App Service: https://github.com/adiroiban/azure-web-app-klein -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] CircleCI for macOS builds
Hi, The guys from Circle-CI have graciously extended our macOS build time from 500 to 1000 minutes. This gives us about 16 hours for macOS builds. For the documentation and static checkers, we have used 21h 33m 31s between Mar 19 - Apr 19 ...and the combine builds runs for about 5 minutes. A single macOS build also runs between 5 and 6 minutes. In the previous configuration, we were running the macOS tests only when the documentation / static checkers and pyflake3 linter were green. With that into consideration, we might be able to fit within the 1000 minutes limit. If 1000 minutes is not enough, I was thinking to configure Circle-CI to run macOS builds on Circle-CI only for PR from external branches. For branches inside the Twisted main repo we have the macOS Buildbot builder, which is triggered automatically. In this way, external contributors should get some feedback for macOS. Cheers, -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Problem with osx10.10-py2.7 builder
On 27 March 2018 at 23:54, Glyph <gl...@twistedmatrix.com> wrote: > > > > On Mon, Mar 26, 2018, at 10:39 PM, Glyph wrote: > > > > On Mar 26, 2018, at 3:32 AM, Jean-Paul Calderone < > exar...@twistedmatrix.com> wrote: > > On Mon, Mar 26, 2018 at 2:58 AM, Glyph <gl...@twistedmatrix.com> wrote: > > Alternately... > > The reason we're doing this at all is that Travis's macOS situation was > pretty dismal when we last tried it. But then, the graphs on > https://www.traviscistatus.com at the time matched that experience: > routinely there were build queues in the hundreds. Right now the "Backlog > macOS Builds for Open Source Projects" shows a completely flat line, zero > for the last 24 hours. > > It might be worth trying out Travis again. > > > FWIW, I moved another project's macOS CI from Travis to CircleCI about a > month ago because the Travis macOS job queue was so long. Counting queue > time, CircleCI finishes those jobs around 6x faster than TravisCI. > > Maybe things have improved on Travis but they're still pretty bad. > > > That's a bummer, but thanks for providing the data point. > > -g > > > > I guess I just needed to look during the PST workday... > > FYI buildbot/osx10.10-py2.7 is broken again - Circle-CI is pretty fast, but we are also burning the time pretty fast. We have used 90% of the time allocated between 19 March of 19 April. Regards, -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Problem with osx10.10-py2.7 builder
Hi, On 25 March 2018 at 05:14, Craig Rodrigues <rodr...@crodrigues.org> wrote: > Hi, > > Can someone check this builder: > > https://buildbot.twistedmatrix.com/builders/osx10.10-py2.7 > > It is failing randomly in weird ways during builds. For example, > in some cases it fails to install tox. In other cases, the tests terminate > in > some random spot. I will try and take a look at it. If someone knows more about the slave running this build, please add the info on wiki at: https://twistedmatrix.com/trac/wiki/ContinuousIntegration/Buildslaves > This builder is marked as "Required", so it is a bit annoying. > My understanding from the previous discussion about the supported platforms is that any build is required. The osx is there as I reminder to run the buildbot tests and not to merge a PR with only Travis and Appveyor. I have reconfigured GitHub to require ubuntu16.04-py2.7-coverage as this is backed by 3 slaves. - We have the circle-ci macOS and for the next 2 days, we can use that :) We are already used 55% of the macOS time and Circle-CI was enalbled in trunk in the last 2 days...so I don't think that we can use Circle-CI for macOS for the long term. I will write them to see if we can get an exception or a discount. Otherwise, I plan to rent a macOS so that it can act as a second slave for osx10.10-py2.7 So far, the support from macincloud was not helpful or friendly, but I guess that we can just try their service for 1 month and see how it goes. -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Buildmaster is down
On 18 March 2018 at 04:56, Glyph <gl...@twistedmatrix.com> wrote: > > >> On Mar 17, 2018, at 11:16 AM, Adi Roiban <a...@roiban.ro> wrote: >> >> Hi, >> >> Today, I tried to reconfigure the Buildbot master from braid/master >> and it failed. [snip] > I don't believe the issue was changes made outside Braid; based on the > failure mode it looks to me like you had an out-of-date or edited > twisted-infra-secret repo. Please be sure to git fetch (git reset --hard > origin/master ; git secret reveal) before doing anything in prod. Thanks. `git secret reveal` was the issue on my side. I knew about git secret, but I was expecting that there will be a warning when the decrypted version is out of sync at pull. There is `fab config.production buildbot.reconfigure` which should just touch the configuration, without updating the virtualenv. Usually this is much faster. Thanks again for the fix, my bad for getting into this! Adi ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] New member of the twisted/twisted-contributors team.
Hi, Just a quick announcement. I made Ilya Skriblovsky (https://github.com/IlyaSkriblovsky) a member of twisted/twisted-contributors team. The main reason is to make it easier to work with Ilya and allow him to run tests on Buildbot. Let me know if you want this change reverted. Regards, -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] List of required builds before a merge
Hi, It is not clear to me what builders need to pass before we can merge something. I expect that all supported "platforms" need to pass, but it is not clear what are the currently supported platforms. We have this info in the wiki but it does not help. https://twistedmatrix.com/trac/wiki/ReviewProcess#Authors:Howtomergethechangetotrunk In GitHub I can see Travis / Appveyor and OSX from Buildot as "Required" Is that all? -- If I check the "supported" group in Buildbot, I see many more builders. The problem is that a significant number of slaves are down and those builders are not available. Is Fedora still supported and required? --- I suggest to use GitHub "Required" marker to document what platforms are supported. We don't have time to maintain the infrastructure, so I suggest to drop support for anything that is not supported by Travis and Appveyor. I know that this might be disruptive. I think that we need it in order to raise awareness that supporting a platform is not easy. If someone (including me) cares about a platform they should find a way to help to project supporting that platform. What do you think? -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Circular references in TLSMemoryBIOProtocol
On 10 March 2018 at 04:57, Glyph <gl...@twistedmatrix.com> wrote: > On Mar 9, 2018, at 4:03 AM, Ilya Skriblovsky <ilyaskriblov...@gmail.com> > wrote: > > > Just wanted to make sure, did I all what I should do for putting this ticket > into review: https://twistedmatrix.com/trac/ticket/9374 ? > Should I just wait for maintainers to review it? > > > You've done all the right things. :) I went for an initial review. > The best thing you can do to accelerate your own change getting reviewed is > to code review others' changes, so that when a reviewer arrives to > stochastically select something to review it is more likely that they'll > select your thing instead of one of the other things :-). +1 I am thinking of adding this to the wiki page for the review process. I think that we can try and see how it goes. Doing a review is not easy, you end up with the responsibility for the approved code, but with the size of the queue I think that we should try. > The second best thing you can do is to donate a lot of money using the form > on the front page of the web site: > https://twistedmatrix.com/trac/#DonatetoTwisted, which, at some point, will > allow us to re-start the Twisted fellowship program > http://labs.twistedmatrix.com/2015/06/twisted-fellowship-2015-call-for.html > and have someone actually keep the review queue clear as their actual paid > responsibility :). > -g +1 on this. I think that it might work even if we get someone to work on this 1 hour per day. Glyph, do you know what is the required money we need to raise to start this program... and are the current money on bank now? Thanks! -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
