Re: [Twisted-Python] Codecov.io security incident
On 2021-04-16 14:26, Adi Roiban wrote: I don't know how we can prevent these types of security issues. We are a public project with limited resources and are always exposed when we are pulling dependencies from codecov or pypy that we don't fully control. I guess that what we can do is stop using the codecov.io bash uploaded and switch back to python uploader. What will this do now? Do you consider the bash uploader a greater future risk than any other thing that codecov, or anyone else, creates? Any other ideas ? In a single CI system (rather than using two) we could do the project coverage absolute limit check and patch coverage check (diff-cover) in-build. Maybe there's even a place we could publish the coverage html output? That said, I've never been much for avoiding services and the proposal for not using a codecov package involves adding another package so... And like you said Adi, it seems pretty implausible to audit all code we use in CI. So, I don't know how there's a solution. But, I'm well aware that I'm not a security person. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On 2021-04-01 09:36, Adi Roiban wrote: For example, I don't understand why you need to create an sdist and from sdist to create a wheel and then install that wheel inside tox for each tox test environment. Why not create the wheel once and install the same wheel in all environments. Are you referring to the use of tox-wheel? Note that the workflow I setup in towncrier has a single build job that creates an sdist and from that a wheel one time, uploads those as artifacts in the workflow, and then every other test job and the publish job use those exact pre-created files. So, I think we agree here. I'll let Grainger weigh in on their present take on tox-wheel if they want. There is an extra 1 minute required for each job to generate the coverage XML file. We can download all the coverage artifacts and generate the XML only once. I did coverage collection from all jobs for pymodbus. Looks like I still had the XML report generating in each job, but that could presumably be shifted. Anyways, I'm sure several of us could do this off the top of our heads but in case it is of any interest here's what I did. Or maybe you just point out something silly I did there and should learn from. :] https://github.com/riptideio/pymodbus/pull/592/files That is one big matrix :) ... I don't understand why you need test and check and coverage, each with a separate matrix. I don't think it really is as big as you think. The test and coverage matrix are separated (and the coverage matrix is kind of not a matrix, what with one job) because that's the level at which GitHub allows you to describe dependencies. If you want to collect the coverage during test runs and then combine all the results together before uploading to codecov etc a single time, I think this is just a mandatory structure. The coverage job must be separated so it can depend on the test matrix. Having checks be a separate matrix is kind of neither here nor there on this topic, I think. And the All job is compensation for a missing GitHub Actions feature. I am thinking of something like this: * A matrix to run jobs with coverage on each environment (os + py combination + extra stuff noipv6, nodeps, etc) * Each job from the matrix will run an initial combine to combine sub-process coverage * Each job will upload once python coverage raw file. * A single job will download those coverage file, combine them once again to generate an XML file to be pushed to codecov.io Sounds good. Sounds like exactly what I suggested. :] Except for maybe misunderstanding about what level of separation is needed for the interjob dependencies. Hopefully this clarifies a bit. I think we actually intend the same thing. Unless I've missed something else? Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On 2021-03-30 17:14, Kyle Altendorf wrote: On 2021-03-30 15:24, Glyph wrote: On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: Hi All, Has anyone contacted GitHub to see if they would be willing to increase the parallelism limit in Actions? My understanding is that we maintain two CI systems (GitHub Actions and Azure Pipelines) for the sake of more parallelism. While perhaps worthwhile, this doesn't seem fun. Maybe GitHub would be willing to help out. Not as far as I know. Do you want to give it a shot? That was the plan. Submitted. I'll let you all know. From GitHub: Hi Kyle, Thanks for taking the time to write in. You would need to upgrade the twisted organization account to Enterprise Cloud for a higher concurrent jobs limit. However, the discount on the twisted organization account is only for GitHub Team, so an upgrade would require payment. https://docs.github.com/en/actions/reference/usage-limits-billing-and-administration#usage-limits All the best, Jimmy https://docs.github.com/en/github/getting-started-with-github/githubs-products#github-enterprise https://docs.github.com/en/github/getting-started-with-github/githubs-products#github-team So it seems we already have 60x parallelism and a discount ($4/month per user if we have Teams for free) and at least a few of us didn't realize it? Admittedly macOS remains at 5x until you get to the enterprise level. So, case closed on that I suppose. I'm not sure where the rest of the discussion about just-GitHub-for-CI stands, but that can be separated from thread. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] towncrier releases 19.9.0 and 21.3.0rc1
Hi all, Towncrier 19.9.0 [0] and 21.3.0 [1] have been published to PyPI. Neither release had significant changes since their corresponding release candidates. None the less, the news is available on GitHub [2]. Thanks to Adi for the relentless help and reviews over the past several months. Cheers, -kyle [0]: https://pypi.org/project/towncrier/19.9.0/ [1]: https://pypi.org/project/towncrier/21.3.0/ [2]: https://github.com/twisted/towncrier/blob/eab34611b93a4ba6e3805cd546a674d88dbd43cf/NEWS.rst On 2021-03-26 09:18, Kyle Altendorf wrote: Hi everyone, I am announcing 2 twisted releases. This is a bit more complicated than normal since 19.9.0rc1 was never followed by a non-RC release. You can see a detailed checklist and discussion of this in #313 [0]. Relative to 19.9.0rc1, 19.9.0 contains a few readme fixes and the news fragments processed into the news file. 19.9.0 artifacts have been created [1] and 19.9.0 publishing is being held until after 21.3.0 to avoid a somewhat pointless intermediate step for developers that are using 'the latest non-pre release on PyPI' (presently 19.2.0). No real need for someone to get 19.9.0 now just to see 21.3.0 available next week. The news for 21.3.0 is available for review [2]. 21.3.0rc1 has been published and is open for a week for feedback before 21.3.0 is planned to be released. There are remaining bugs and important features. But, it took long enough to get to this point that I consider it important to get something out. Several people are waiting patiently [3]. As we find time to design and address the remaining issues, hopefully we can manage more frequent releases. Please do let me know of any issues you find that are not already documented. Or, if any of the documented ones should really be release blockers. I sure hope we can get this out though. :] Cheers, -kyle [0]: https://github.com/twisted/towncrier/issues/313 [1]: https://github.com/twisted/towncrier/pull/331#issuecomment-803491163 [2]: https://github.com/twisted/towncrier/blob/21.3.0.rc1/NEWS.rst [3]: https://github.com/twisted/towncrier/issues/269 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On 2021-03-31 05:45, Adi Roiban wrote: On Tue, 30 Mar 2021, 22:23 Kyle Altendorf, wrote: On 2021-03-30 15:24, Glyph wrote: On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: Hi All, Has anyone contacted GitHub to see if they would be willing to increase the parallelism limit in Actions? My understanding is that we maintain two CI systems (GitHub Actions and Azure Pipelines) for the sake of more parallelism. While perhaps worthwhile, this doesn't seem fun. Maybe GitHub would be willing to help out. Not as far as I know. Do you want to give it a shot? That was the plan. Submitted. I'll let you all know. Thanks. On a related note I am working to reduce the general duration of Twisted CI jobs. This should result in keeping the jobs slot busy for less time and allowing more total jobs per day. The first step was to enable parallel trial tests. Use both CPUs available to the CI VMs. This was already done for Linux and MacOS. Distributed trial is not yet supported on Windows, but I am working on it. I already have a working distributed trial for Windows on my system. It needs various fixed and I have submitted smaller PR for review. Just getting distrial to work on Windows is not all...as we need to update and clean the Twsited test to reduce the side effects. Linux and macOS are more relaxed in terms of file access and for example you can delete a file, even if its opened by another process or by the same process. Windows is not happy about that. And then there are the 2 pypy jobs taking 30 minutes each - working to fix it here https://github.com/twisted/twisted/pull/1543/ And then there are also general CI jobs improvements. For example we can save about 40 seconds if we decide to stop sending coverage to coveralls and only use codecov.io Another example is save another 1 minute by stop using `tox --notests` and replace it with something like https://github.com/twisted/python-info-action to show the dependencies. With `tox --notest` we now create the wheels and install the dependencies twice for each job. Would `tox --skip-pkg-install` cut it for the second invocation? https://github.com/ericaltendorf/plotman/pull/66/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR145 There is an extra 1 minute required for each job to generate the coverage XML file. We can download all the coverage artifacts and generate the XML only once. I did coverage collection from all jobs for pymodbus. Looks like I still had the XML report generating in each job, but that could presumably be shifted. Anyways, I'm sure several of us could do this off the top of our heads but in case it is of any interest here's what I did. Or maybe you just point out something silly I did there and should learn from. :] https://github.com/riptideio/pymodbus/pull/592/files Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On 2021-03-30 15:24, Glyph wrote: On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: Hi All, Has anyone contacted GitHub to see if they would be willing to increase the parallelism limit in Actions? My understanding is that we maintain two CI systems (GitHub Actions and Azure Pipelines) for the sake of more parallelism. While perhaps worthwhile, this doesn't seem fun. Maybe GitHub would be willing to help out. Not as far as I know. Do you want to give it a shot? That was the plan. Submitted. I'll let you all know. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] GitHub Actions parallelism limit increase
Hi All, Has anyone contacted GitHub to see if they would be willing to increase the parallelism limit in Actions? My understanding is that we maintain two CI systems (GitHub Actions and Azure Pipelines) for the sake of more parallelism. While perhaps worthwhile, this doesn't seem fun. Maybe GitHub would be willing to help out. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] towncrier releases 19.9.0 and 21.3.0rc1
Hi everyone, I am announcing 2 twisted releases. This is a bit more complicated than normal since 19.9.0rc1 was never followed by a non-RC release. You can see a detailed checklist and discussion of this in #313 [0]. Relative to 19.9.0rc1, 19.9.0 contains a few readme fixes and the news fragments processed into the news file. 19.9.0 artifacts have been created [1] and 19.9.0 publishing is being held until after 21.3.0 to avoid a somewhat pointless intermediate step for developers that are using 'the latest non-pre release on PyPI' (presently 19.2.0). No real need for someone to get 19.9.0 now just to see 21.3.0 available next week. The news for 21.3.0 is available for review [2]. 21.3.0rc1 has been published and is open for a week for feedback before 21.3.0 is planned to be released. There are remaining bugs and important features. But, it took long enough to get to this point that I consider it important to get something out. Several people are waiting patiently [3]. As we find time to design and address the remaining issues, hopefully we can manage more frequent releases. Please do let me know of any issues you find that are not already documented. Or, if any of the documented ones should really be release blockers. I sure hope we can get this out though. :] Cheers, -kyle [0]: https://github.com/twisted/towncrier/issues/313 [1]: https://github.com/twisted/towncrier/pull/331#issuecomment-803491163 [2]: https://github.com/twisted/towncrier/blob/21.3.0.rc1/NEWS.rst [3]: https://github.com/twisted/towncrier/issues/269 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Probot app to automatically delete merged branches
On 2021-03-13 10:40, Adi Roiban wrote: > On Sat, 13 Mar 2021 at 14:29, Kyle Altendorf wrote: > > On 2021-03-13 08:22, Adi Roiban wrote: > > Hi, > Do you see anything bad in enabling the delete-merged-branch app for > twisted/twisted ? > https://probot.github.io/apps/delete-merged-branch/ > > There's a checkbox for this. Unless I'm missing something? > https://docs.github.com/en/github/administering-a-repository/managing-the-automatic-deletion-of-branches Great. Thank for the info. So, new question: Should we enable the automatic deletion for twisted/twisted? I suppose I could have responded to the underlying question as well... I do have deletion enabled in my repos. I don't really like that you lose the references in git though. As long as you stick with GitHub you do have the PR's as a record and from them you can recover the associated branch. I do it but don't super care. Hopefully someone else does. Cheers, -kyle___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Probot app to automatically delete merged branches
On 2021-03-13 08:22, Adi Roiban wrote: > Hi, > > Do you see anything bad in enabling the delete-merged-branch app for > twisted/twisted ? > > https://probot.github.io/apps/delete-merged-branch/ There's a checkbox for this. Unless I'm missing something? https://docs.github.com/en/github/administering-a-repository/managing-the-automatic-deletion-of-branches Cheers, -kyle___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Need help releasing new version of Twisted incremental
On 2021-03-01 19:25, Craig Rodrigues wrote: On Mon, Mar 1, 2021 at 3:27 PM Kyle Altendorf wrote: On 2021-03-01 14:13, Craig Rodrigues wrote: On Mon, Mar 1, 2021 at 9:13 AM Colin Watson wrote: Isn't this just because incremental hasn't had a release since 17.5.0? I added post= support way back in https://github.com/twisted/incremental/pull/37, but that was after 17.5.0. Oh wow! You are right. Can someone help me release a new version of Twisted incremental? I don't know how to do it. I would have expected the existing PRs could be handled and that the release would have a release branch with a towncrier run and review. https://github.com/twisted/incremental/pull/67 Kyle, Yes, I did PR 67 in incremental just now. I have write access to the twisted/incremental Git repo. My point was that you missed a review and you missed running towncrier and depending what release process you are following you missed a tag in the branch and a post-release-version-change. You asked about how to release but gave no time for response. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Need help releasing new version of Twisted incremental
On 2021-03-01 14:13, Craig Rodrigues wrote: On Mon, Mar 1, 2021 at 9:13 AM Colin Watson wrote: Isn't this just because incremental hasn't had a release since 17.5.0? I added post= support way back in https://github.com/twisted/incremental/pull/37, but that was after 17.5.0. Oh wow! You are right. Can someone help me release a new version of Twisted incremental? I don't know how to do it. I would have expected the existing PRs could be handled and that the release would have a release branch with a towncrier run and review. https://github.com/twisted/incremental/pull/67 Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted 21.2.0rc1 Release Candidate Announcement
On 2021-02-14 21:53, Craig Rodrigues wrote: > Please test it, and let me know how your applications fare, good or bad! Hi Craig, Thanks for all the work to get us into the next Twisted release. All, I went ahead and set pip_pre=true in pytest-twisted's tox.ini to try it out and got warnings/errors with PyPy 3.7 about use of t.i.d.returnValue claiming it should only be used in @inlineCallbacks decorated functions even though the location complained about is such. Only PyPy 3.7, not PyPy 3.6 nor CPython 3.7. Linux, macOS, and Windows all failed. https://github.com/pytest-dev/pytest-twisted/pull/134/checks?check_run_id=1904583062 I also pulled a branch and PR off of the release branch and added PyPy 3.7 to the matrix. CI failed on PyPy 3.7. https://github.com/twisted/twisted/pull/1515/checks?check_run_id=1904809978 NonLocalExitTests test_returnValueNonLocalDeferred ... [FAIL] test_returnValueNonLocalWarning ... [FAIL] Ticketed as https://twistedmatrix.com/trac/ticket/10093. Seems like it might be an easy special-case fix so I'll take a look. I don't know if this qualifies as a releaser blocker or not, or if it can get cherry picked into the release anyways. Others will have to weigh in on that. I guess it's basically just lack of support for a 'new' PyPy version. PyPy 3.7 beta was released in November and alpha in September. I understand choosing not to bother fitting this in, but after blowing past CPython 3.8 and 3.9 released it would be nice to actually be all around caught up. Cheers, -kyle___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] reactor for Linux io_uring
On 2021-01-04 10:15, Barry Scott wrote: On Monday, 4 January 2021 04:01:42 GMT Ian Haywood wrote: If people think an IoUringReactor is worthwhile I'll open a ticket and make a start. I'm guessing that you will need to write a Python extension to get at io_uring or use ctypes. Is that what you where expecting? FYI, there is https://github.com/YoSTEALTH/liburing. I have no basis to say if it is good or not, just that it's there. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Upcoming Twisted Release
On 2020-12-27 18:58, Adi Roiban wrote: > For the future, maybe we can stop doing a release candidate wheel and > instead just announce that a release is coming > and people should test trunk and flag any release blocker ticket. With all the work for automated releasing, isn't it easy to do an RC and a week or two later do a regular release? Admittedly, lots of people (like myself) haven't made most of our CI setups run all of pinned versions, latest release, latest pre-release, and latest trunk... but maybe having more projects actually doing release candidates is one piece of encouraging that? Or maybe, practically speaking, nobody is going this route anyways so you are suggesting a better path. > I think that with pip, is now very easy to install Twisted from trunk. In case anyone isn't super familiar... pip install git+https://github.com/twisted/twisted#egg=twisted The #egg=twisted is useful for misc corner cases but not always required. One not-so-corner case is when you want to specify an extra then it is #egg=twisted[windows_platform], for example. Cheers, -kyle___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] towncrier review policies and etiquette (and maybe releases too)
Where does towncrier stand on review policies and etiquette? I generally don't like to just jump into new projects and start reviewing and merging but I don't see other activity on that front nor do I see any guidance on how it should be done. If I know how to proceed properly then I might dedicate some review time to it. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Request for new Twisted release?
I guess this one is top-posting... Anyways, I would personally default away from CI-provider-specific solutions. It seems like a command line tool ought to suffice? But maybe there's some security something or other I'm missing. I just see that we presently use four CI systems so picking one and expecting to stick with it seems a bit naive. Presumably someone will (helpfully) find an issue with this but I've copied this between a few projects. https://github.com/altendky/pyqt5-tools/blob/b0ecb9bf53ded53c28763d76f1e87d0b8180d2b8/twineontag.py I know there are many other solutions. Since this topic is a bit more general than just PyPI uploads I've also got a few comments about towncrier. Per the link below I added towncrier to my RTD builds so that PRs will render the newsfragments. Also, I started to vaguely think through some option for using towncrier that would allow for creating a release by pushing a tag rather than pulling a PR and doing various manual steps and modifying a generated news file etc. No answers yet, just a thing I'd like to get going at some point. Certainly `release by pushing a tag` is harder in bigger projects but I don't have in mind yet a reason it can't happen. https://github.com/altendky/qtrio/blob/7a268c3fc14afe4cde7cbedd423d1f74f616a299/docs/source/conf.py#L133-L148 Cheers, -kyle On 2020-09-28 13:48, Glyph wrote: > Yes, please, let's do this! I have been trying to fix or enable automated > releases for smaller Twisted-org projects as I have time, but Twisted itself > is a bit too tricky for my level of capacity during the end times. > > Kyle Altendorf has done quite a bit of work on this, so let's be sure to > leverage that work and not to start over: > https://twistedmatrix.com/trac/ticket/9531. > > -g > > On Sep 28, 2020, at 9:25 AM, Craig Rodrigues wrote: > > Adi, > > In this Twisted subprostlject, you implemented something which releases to > Pypi upon creation of a tag: > > https://github.com/twisted/twistedchecker/blob/master/.travis.yml#L23 > > Maybe we can do something similar for the main Twisted project. > > GitHub Actions allows you to run a specific workflow in response to a GitHub > Release event: > > https://github.com/actions/create-release > https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#release > > > So maybe we can implement your suggested steps in a new GitHub Actions > workflow specific to doing a release. > > -- > Craig > > On Thu, Sep 24, 2020 at 4:32 PM Adi Roiban wrote: > > Maybe the release can be simplified and automated: > > * Build the source and binary wheel in GitHub actions as this can be done for > Linux (Ubuntu), Windows and macOS. > * Host the download source and wheel files only on PyPi and publish them > automatically from GitHub action on a new tag is created. > * Host the documentation only on the Read The Docs. > * Hos the API documentation on Read The Docs - might need some hacking, but > at release we can create an API docs package as an artifact which is then > pulled when Read The Docs documentation is created and copied as extra HTML > files. > * Move Twisted blog to GitHub pages... or even read the docs with a separate > theme like Crate [1] ... if the blog is still required. > > -- > > With the above implemented the release should look like: > > * Each time the tests for a PR are executed, pydoctor will run and > automatically create the API files as an artifact available for download. > * Manually create a new branch in which the version is updated and the > changelog/news/release note is created and all the news fragment files are > removed/ . Have the branch reviewed and approved with all the tests passed. > When the pydoctor tests are executed, the API docs are created. > * The release branches can have a naming convention line 'release-20.0.0`. At > first the release branch can have a release candidate version and a GitHub > can automatically push the release to PyPi. > * Manually send an email to Twisted mailing list to announce the pre-release. > * Once the branch is merged, manually push a new tag > * The new tag should trigger the GitHub action for publishing the release on > PyPi > * The new merge in master should trigger the Read The Docs build... which now > will also include the static API docs pages. > * Once the release is done, manually send a new email to the mailing list. > > There are still many manual steps, but the only permissions required is > commit to the repo. > This will no longer use any of the Twisted own infrastructure. > > [1] https://sphinx-themes.org/html/crate-docs-theme/crate/basic.html > -- > Adi Roiban ___
Re: [Twisted-Python] Black enabled in trunk
On 2020-09-14 13:43, Glyph wrote: On Sep 13, 2020, at 11:45 PM, Tom Most wrote: To adjust the formatting: tox -e black-reformat The formatting is checked by a new GitHub Actions lint built. Could we possibly use something like this: https://github.com/cclauss/autoblack to just _do_ the formatting rather than "check" if it's correct? PR templates are all well and good but the best checklist item is the one that's already checked off... It might be worthwhile but I think there is some downside to CI frequently injecting commits (or amending them). Inevitably, changes like this cause conflicts. For small PRs it's easiest to merge forward and then run tox -e black-reformat. For larger ones it can help to apply formatting before merge. To do this: * Run black on the files your branch changes (be sure to use Black 20.8b1, not an older version) * Commit the result, like `git commit -am "Fade to black"` * Add that commit to .git-blame-ignore-revs to avoid polluting git blame. Is .git-blame-ignore-revs going to be the next 'newsfile' what with every PR adding lines to it? Also, for the record, this doesn't address GitHub blame and does require everyone set the previously mentioned git config since it doesn't go with the repo. :[ Though that's mostly an annoyance with git. Oh yeah... thanks! Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Management of PyPI maintainers (as related to qt5reactor)
On 2020-07-30 14:10, Jean-Paul Calderone wrote: On Thu, Jul 30, 2020 at 10:34 AM Kyle Altendorf wrote: Following up on: https://github.com/twisted/qt5reactor/issues/50#issuecomment-658432478 qt5reactor has recently been moved into the Twisted organization on GitHub. The intent is that being in an org will make it less likely that existing maintainers disappear and the project is stranded with nobody having the authority to pass it on to any new maintainers. If we happen to get more people interested in maintenance that's a bonus, but it is not the reason for the move. The question is, how should the Twisted organization manage PyPI access for its projects? Glyph mentioned there is a 1password account that could be relevant. I have not used 1password personally so I don't know any details about how it would fit in here. Twisted itself has six maintainers listed on PyPI: exarkun, glyph, hawkowl, itamarst, jml, and markrwilliams. Any opinions? 1Password vs. just-add-a-couple-maintainers-to-the-qt5reactor-pypi vs. ...? Can you clarify this a bit? PyPI has perfectly serviceable support for multiple maintainers per project. What benefits come from sharing some kind of credentials (and what credentials) via a tool like 1Password? It seems like folks who should be qt5reactor PyPI maintainers can have their personal PyPI accounts configured as maintainers on PyPI and then the problem's solved. So, if I've missed something, maybe you can help clarify. qt5reactor isn't particularly active and and my hope in it moving into the Twisted organization is that if all 'active' maintainers are lost and someone else volunteers later, an organizational maintainer could choose to give the new volunteer the necessary authority. It may well be that this is a silly reason to make the move but I haven't been corrected about it yet. :] I didn't originate the 1password suggestion but if a Twisted PyPI account were created, as Adi mentioned, and the credentials stored in 1password then that would associate control with the Twisted organization rather than individual developers. The presently 'active' individual developers would presumably retain their PyPI maintainership rights as well. Any more clear now? Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Management of PyPI maintainers (as related to qt5reactor)
Following up on: https://github.com/twisted/qt5reactor/issues/50#issuecomment-658432478 qt5reactor has recently been moved into the Twisted organization on GitHub. The intent is that being in an org will make it less likely that existing maintainers disappear and the project is stranded with nobody having the authority to pass it on to any new maintainers. If we happen to get more people interested in maintenance that's a bonus, but it is not the reason for the move. The question is, how should the Twisted organization manage PyPI access for its projects? Glyph mentioned there is a 1password account that could be relevant. I have not used 1password personally so I don't know any details about how it would fit in here. Twisted itself has six maintainers listed on PyPI: exarkun, glyph, hawkowl, itamarst, jml, and markrwilliams. Any opinions? 1Password vs. just-add-a-couple-maintainers-to-the-qt5reactor-pypi vs. ...? Thanks for your consideration. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] mypy integrated with CI for twisted
On 2020-06-24 00:43, Glyph wrote: > On Jun 23, 2020, at 5:34 AM, Adi Roiban wrote: > > Hi Craig, > > On Tue, 23 Jun 2020 at 00:36, Craig Rodrigues wrote: > I have merged some more fixes for mypy to Twisted trunk branch. > > In trunk, you can run mypy with: > > tox -e mypy > > Currently this results in 171 errors, which is way down from >1000 errors > a month ago. > > In addition, if you look at any new PR's there is a Mypy Ubuntu job > running on Azure pipeline, which runs mypy. Right now errors from this job > are ignored and does not block the PR. However, if we can get the mypy > errors down to zero, we can make mypy status a blocker for the PR. > > Thanks for working on this. > > Looking forward to have a real green mypy build. > > A general question: Why Twisted used Azure Devops and not GitHub actions? Azure Pipelines gave us substantially more parallel capacity than is available via Github Actions, which means we can make build statuses appear much sooner. Plus they support more platforms. Just noticed we don't actually link to Azure from the readme. Presently we have 14 builds there so within the tense pickiness :] we don't get any benefit (yet). I'm curious though, what additional platforms does Azure get us? ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Pip's Twisted 19.7.0 package is offered to upgrade my v19.2.1...
On 2019-09-10 15:14, Ant wrote: Are there any updates on this? I still see this minor issue. :( Sorry, it appears I found time for a ticket and a PR... but not an email. They have been submitted for review. Ticket: https://twistedmatrix.com/trac/ticket/9701 PR: https://github.com/twisted/twisted/pull/1183 Cheers, -kyle On Sun, Aug 11, 2019 at 12:31:20PM -0700, Ant wrote: Hello. I cannot figure out how to report this minor bug of a minor upgrade issue (https://github.com/twisted/twisted didn't have an issues section) that I noticed since last week: $ pip list --outdated DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429). Package Version Latest Type --- --- -- - Twisted 19.2.1 19.7.0 sdist $ pip install --upgrade Twisted DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429). Collecting Twisted Using cached https://files.pythonhosted.org/packages/61/31/3855dcacd1d3b2e60c0b4ccc8e727b8cd497bd7087d327d81a9f0cbb580c/Twisted-19.7.0.tar.bz2 ERROR: Complete output from command python setup.py egg_info: ERROR: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-install-2h_su73e/Twisted/setup.py", line 20, in setuptools.setup(**_setup["getSetupArgs"]()) File "", line 257, in getSetupArgs File "", line 208, in _checkPythonVersion ImportError: Twisted on Python 3 requires Python 3.5 or later. ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-2h_su73e/Twisted/ That v19.7.0 shouldn't even be offered for my outdated setups. :( Thank you for reading and hopefully answering soon. :) ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Pip's Twisted 19.7.0 package is offered to upgrade my v19.2.1...
On 2019-08-11 15:31, Ant wrote: Hello. I cannot figure out how to report this minor bug of a minor upgrade issue (https://github.com/twisted/twisted didn't have an issues section) that I noticed since last week: $ pip list --outdated DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429). Package Version Latest Type --- --- -- - Twisted 19.2.1 19.7.0 sdist $ pip install --upgrade Twisted DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429). Collecting Twisted Using cached https://files.pythonhosted.org/packages/61/31/3855dcacd1d3b2e60c0b4ccc8e727b8cd497bd7087d327d81a9f0cbb580c/Twisted-19.7.0.tar.bz2 ERROR: Complete output from command python setup.py egg_info: ERROR: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-install-2h_su73e/Twisted/setup.py", line 20, in setuptools.setup(**_setup["getSetupArgs"]()) File "", line 257, in getSetupArgs File "", line 208, in _checkPythonVersion ImportError: Twisted on Python 3 requires Python 3.5 or later. ERROR: Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-2h_su73e/Twisted/ That v19.7.0 shouldn't even be offered for my outdated setups. :( Perhaps a `python_requires=` would do the trick? https://github.com/twisted/twisted/blob/c55c778e6c7a758cb9cee568a9188ac49e628c27/src/twisted/python/_setup.py#L54-L75 Maybe: python_requires='>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*!=3.4.*' And perhaps also drop the manual checking code? I'm kind of busy but if someone else doesn't get to it first I'll try to get to a ticket/PR later. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Azure Pipelines
On 2019-06-22 14:05, Amber Brown wrote: On 22/6/19 9:59 am, Kyle Altendorf wrote: Hmm... I'm running Python inside a build to generate the matrix for that build (with a yaml file for a pipeline). Now I'm curious what matrixing is being used that isn't supported yet. But perhaps it isn't worth bothering to work through. Cheers, -kyle It now says: #Multi-configuration and multi-agent job options are not exported to YAML. Configure these options using documentation guidance: https://docs.microsoft.com/vsts/pipelines/process/phases So maybe it'd work? Anyone want to volunteer to see? :) I don't know what our setup is now so I can't really compare. Do we have a matrix built from a few axes? Like Windows/macOS and py2.7/3.7 generating 4 builds for us? AFAIK that isn't yet supported. https://github.com/microsoft/azure-pipelines-yaml/issues/20 But, you can certainly write a job and list out the four (or whatever) parameter sets you want it to be run against. For romp I went ahead and implemented a little Python cross matrixer which runs as the first job and outputs JSON. It's output is used as the matrix def for another job which does whatever it's supposed to. So there are options anywhere from hand-coded YAML to build-parameter-dependent on-the-fly-generated JSON to configure a 'matrix', and whatever in between. I'm guessing at present we'd be satisfied with just hand-coded YAML parameter sets. Maybe if we moved all platforms etc to Azure we'd like a generated matrix (either a manual script when the matrix definition is changed or in-build generation though that of course adds a bit of build time). Honestly, I would expect to usually want exceptions to any basic matrix cross-producting system so just defining it with regular Python code from the get-go has some upsides. I need to tweak exttrs a bit and romp has a bit of a specialized purpose but they are the two examples I have. https://github.com/altendky/exttr/blob/60aa5d6e9f04631b8e86552620f2d9cd2a0a3a04/azure_job_template.yml#L15-L43 https://github.com/altendky/romp/blob/0b05a7830ba1b02e503dd0004ffdce5e0ae3164f/azure-pipelines.yml#L43 After all that... is this actually what we are talking about when it comes to matrixing? Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Azure Pipelines
On 2019-06-21 18:40, Amber Brown wrote: The matrix stuff we're using is not yet supported as a pipeline workflow, so, when it is, we will move it :) Hmm... I'm running Python inside a build to generate the matrix for that build (with a yaml file for a pipeline). Now I'm curious what matrixing is being used that isn't supported yet. But perhaps it isn't worth bothering to work through. Cheers, -kyle On 22/6/19 4:51 am, Kyle Altendorf wrote: On 2019-06-21 12:05, Glyph wrote: On Jun 21, 2019, at 6:39 AM, Jean-Paul Calderone wrote: I can't see any Azure Pipeline configuration in the repository, I don't see much information on the ticket that (maybe?) introduced Azure Pipeline usage. Once you're logged in, I think you can also see the build configuration, under "edit pipeline" in the same menu the "retry" item is in. Not sure if there are some permissions to manage here for either the retry or edit buttons. If there are, and you can't see either of them, let's get them set up for anyone who wants them; but I think Github org membership should be sufficient, since from what I can tell in their control panel that's how I had permission. Do we want Azure Pipelines config on their website while Travis, AppVeyor, and Circle are all in our source repository? Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Azure Pipelines
On 2019-06-21 12:05, Glyph wrote: On Jun 21, 2019, at 6:39 AM, Jean-Paul Calderone wrote: I can't see any Azure Pipeline configuration in the repository, I don't see much information on the ticket that (maybe?) introduced Azure Pipeline usage. Once you're logged in, I think you can also see the build configuration, under "edit pipeline" in the same menu the "retry" item is in. Not sure if there are some permissions to manage here for either the retry or edit buttons. If there are, and you can't see either of them, let's get them set up for anyone who wants them; but I think Github org membership should be sufficient, since from what I can tell in their control panel that's how I had permission. Do we want Azure Pipelines config on their website while Travis, AppVeyor, and Circle are all in our source repository? Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] a farewell to buildbot
On 2019-03-03 09:07, Chris Withers wrote: What do you make of CircleCI? I found it mostly pleasant but I'm expecting to be using Azure personally. It's the only one that I've worked with that covers what I think ought to be the basics. Linux/macOS/Windows, CPython 2,7/3.4+ (ok, maybe 3.5+...), and PyPy 2/3 (I had to download it but it wasn't too painful to get it working), and artifact storage. CircleCI was nice but it lacks Windows entirely and only has CPython 2/3 on macOS iirc. They do provide some nice templating features. AppVeyor lacks macOS (I haven't tried their Linux) and just one worker at a time. Travis now has the OSes covered and their matrixing is nice (until you do anything 'special' and it can't be used afaik) but only really supports Python on Linux and doesn't have artifact storage. All that said, I just started with Azure Pipelines this week and have only put one pure Python project on it so who knows what will turn up as I do more. https://dev.azure.com/altendky/exttr/_build/results?buildId=120 Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] trouble installing 18.9.0
On 2019-02-05 16:26, Joe Smongeski wrote: I've been trying to install the latest Twisted 18.9.0 with pip install, (using Oracle LInux 7.4), but I'm getting compilation errors. The first one was: src/twisted/test/raiser.c:4:20: fatal error: Python.h: No such file or directory #include "Python.h" This is usually addressed by installing a python3-dev package, or similar. What did you do? I was able to work around that, but now I"m getting this error: gcc -pthread -shared -Wl,-z,relro -g -L ./python3.6m build/temp.linux-x86_64-3.6/src/twisted/test/raiser.o -L/usr/lib64 -LPYTHON3.6M -o build/lib.linux-x86_64-3.6/twisted/test/raiser.cpython-36m-x86_64-linux-gnu.so /ETC/ALTERNATIVES/LD: CANNOT FIND -LPYTHON3.6M collect2: error: ld returned 1 exit status error: command 'gcc' failed with exit status 1 Since the compile is under the control of all the install scripts, I can't determine where the linker is looking for "python36m". The all caps `PYTHON3.6M` and `/ETC/ALTERNATIVES/LD: CANNOT FIND -LPYTHON3.6M` seem pretty weird to me. It might be helpful to share a full transcript (prompts, commands, output) of the installation (including creating a virtualenv or venv, etc) Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] code coverage when code executed in a twisted worker thread
On November 29, 2018 8:10:41 AM EST, Chris Withers wrote: >I've noticed that code coverage doesn't appear to be recorded when code > >is executed in something that has been deferToThread'ed. > >Is this a known issue? Does anyone have an explanation? Are you using coverage.py? I don't know offhand but does Twisted use `thread` or `threading`? https://coverage.readthedocs.io/en/coverage-4.2/trouble.html#things-that-don-t-work ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] pytest-twisted questions
On 2018-11-13 17:37, Chris Withers wrote: On 13/11/2018 20:31, Kyle Altendorf wrote: but I know that I mostly don't have test classes and, for the class I do have, I didn't inherit. Mostly I just @pytest.inlineCallbacks (I still don't like the namespace squashing into pytest though :] ) and I suppose in the probably-not-too-distant future I'll instead be using more @pytest_twisted.async_await (ala #31/#34). Okay, but twisted.trial.unittest.TestCase does a bunch of reactor management stuff, most notable making you aware when you've left the reactor in a bad state. As far as I can see from the code, pytest-twisted does not do that, correct? I don't believe so. It sounds like I should review twisted.trial.unittest.TestCase and consider implementing a fixture to provide the checks. Perhaps default it to autouse with a cli parameter to disable it. Perhaps #4 is relevant though. https://github.com/pytest-dev/pytest-twisted/issues/4 https://github.com/altendky/stlib/blob/b34796cbba959d9cb2cb843f3cc5fc815c7cb6c6/epyqlib/tests/utils/test_twisted.py#L65-L93 Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] automated tests for a server application
On 2018-11-06 11:28, Chris Withers wrote: On 06/11/2018 12:14, Kyle Altendorf wrote: On November 6, 2018 6:41:23 AM EST, Chris Withers wrote: Cool, do you have any example tests that do this? Interesting, looks like pytest-twisted does away for the need for this by showing how to install a fake reactor globally: https://github.com/pytest-dev/pytest-twisted/blob/master/pytest_twisted.py#L129-L142 What is 'fake' about this globally installed normally-the-default reactor? (Otherwise the qt5reactor if chosen) I guess I'm still not clear on what the point of using a 'fake' reactor over a 'real' one is. Not that I'm an expert here... Nothing, but the technique could be used to install a fake reactor rather than having to change all the existing code to accept an optional reactor parameter. I use @pytest_twisted.inlineCallbacks anyways, yes. Overall I'm not clear what was recommended here. Why fake the reactor? Even if not using a 'real client' wouldn't you just fake the data going through the connections rather than faking the entire reactor? I'd love to see some good example of faking the data going through the connections, can you point me at some? I would assume you would just write a 'client' in the test of whatever complexity (could just write hardcoded byte sequences) which opens a connection to the server and transmits the bytes and then asserts things about the response. But no, I don't have any code. I can't say I have a good test suite myself and I also don't actually use Twisted for internet stuff (canbus and serial). Sorry. I would expect the servers provided by Twisted would have their own tests you could look at though. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Mixing async/await and asyncio
On September 19, 2018 8:50:41 AM EDT, "whalebot.helms...@gmail.com" wrote: >I want to learn how to use twisted together with asyncio. I create >several files to describe my problem >https://gist.github.com/whalebot-helmsman/c400eb66c0bd35e406de3f8f704adf13: > >- I can use pure asyncio for simple case (asyncio_ex.py) > >- I can use pure twisted for simple case (twisted_ex.py) > >- I can await in twisted on twsited defer (twisted_await.py) > >I have a problem in case I try to await in twisted for asyncio >coroutine(twisted_await_asyncio.py). Program never finishes and I got >only started messages > >2018-09-19 12:48:59,633 [WARNING] PID:8238 >/home/nikita/tmp/twisted_await_asyncio.py:12 started >2018-09-19 12:48:59,633 [WARNING] PID:8238 >/home/nikita/tmp/twisted_await_asyncio.py:12 started I can't say I've done this but I think this page is on topic. https://meejah.ca/blog/python3-twisted-and-asyncio Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] T9217 / PR1052: Wheels, wheels, and more wheels
On 2018-09-07 01:58, Glyph wrote: On Sep 5, 2018, at 5:44 AM, Kyle Altendorf wrote: On 2018-09-05 03:25, Glyph wrote: On Aug 29, 2018, at 10:13 AM, Kyle Altendorf wrote: wheel links: https://github.com/twisted/twisted/pull/1051#issuecomment-416743261 (and next comment) Now that I've got the wheel builds happening I figured it'd be good to try them out on 'real' machines. Turns out we get a failure on twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker for at least the two checks I've done so far (Windows and OSX). I haven't done more than a cursory look at that yet, but it's on the list to understand and resolve. More testing would of course be welcome. Real world, just trial Twisted's own tests, whatever would be appreciated if you are interested. failures with wheels: https://github.com/twisted/twisted/pull/1051#issuecomment-416977723 It's not clear to me that we have any test environments that accurately test what happens after you install from PyPI. If we could build these wheels and then actually run tests on installing them that would be a big step forward in test fidelity! But given the current state of the art, I'd be happy if they just got built from the same code that passed our tests in existing configurations. cibuildwheel does have an option to provide a test command. If we did that would we replace the existing tests or supplement? If supplementing that will presumably be a significant increase in build times. Especially if we did it for all wheels. Definitely replace. What would you want to even supplement them with? I was referring to supplementing the existing src tests with the wheel tests. So, how much of a build time hit do we want to take? I was thinking that for starters here the goal would just be build automation, independent of trying to get a more realistic test path; only doing these builds when pushing a relevant tag. If it's easy to just switch over everything to this then... great, I guess? :) But let's not gild the lily here. Alrighty, another PR for swapping over tests from src to wheels. I would tend to always build the wheels. It's easier for people to test intermediate releases for bugs. It makes a release 'less special' so there is less likely to be a latent issue with the process or artifacts. Etc. But, let's see what the build time hit is. Travis: 4 minutes, about half the time of a single version test job AppVeyor: 7 minutes, about the same as a single version test job Circle OS X: 9 minutes, a bit over a single version test job Circle Linux: 5 minutes, a bit under a single (OS X) version test job Ok, so for about the cost of testing a single Python version we can get wheels. Especially if we want to switch over to testing wheels instead of source, I'd plan on building wheels for every commit. but sure, I didn't mean to argue against automation. Could we do a test run by uploading wheels to, say, https://test.pypi.org <https://test.pypi.org/> first? Shall the process be build wheel, upload to test PyPI, install from test PyPI, test... :] But sure, I could see ending up with a test PyPI upload at least for a tag/release. Could technically do it for every build but that seems excessive, even given my comments above about exercising the process more often rather than less. Here's one spot I did automatic deployment off of Travis to PyPI. https://github.com/altendky/gitignoreio/blob/aeeb9546b6f27f411b191699d3625b4d0a08e7cf/.travis.yml#L11-L19 Pretty straightforward though I think it does the wheel build which isn't what we'd really want. That's pretty cool. I assumed you'd have to write a 'twine' command - Travis just has a 'deploy' provider called "pypi"? What's actually happening behind the scenes there, I wonder! (Also, you should probably upload an sdist as well as a bdist_wheel; there are cases where wheels aren't always what you want.) Yeah, it's cool but then you have to know what is behind it... for each CI host... and how to make it use the files you want... So I'd tend towards a twine call. I made this silly little script the other day to handle this in another repo. https://github.com/altendky/pyqt5-tools/blob/master/twineontag.py I'd have to check on how to deploy existing wheels as well as those from OS X and Windows. I'd suggest a separate PR? I mean maybe it'd be easier to get one PR reviewed than two but that doesn't seem like a great reason to mash things together. Smaller PRs are always easier to review - they just take more latency to build :). Ok, I'll make separate tickets for: * Testing wheels vs. src * Automatic deployment to PyPI Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] T9217 / PR1052: Wheels, wheels, and more wheels
On 2018-09-05 03:25, Glyph wrote: On Aug 29, 2018, at 10:13 AM, Kyle Altendorf wrote: wheel links: https://github.com/twisted/twisted/pull/1051#issuecomment-416743261 (and next comment) Now that I've got the wheel builds happening I figured it'd be good to try them out on 'real' machines. Turns out we get a failure on twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker for at least the two checks I've done so far (Windows and OSX). I haven't done more than a cursory look at that yet, but it's on the list to understand and resolve. More testing would of course be welcome. Real world, just trial Twisted's own tests, whatever would be appreciated if you are interested. failures with wheels: https://github.com/twisted/twisted/pull/1051#issuecomment-416977723 It's not clear to me that we have any test environments that accurately test what happens after you install from PyPI. If we could build these wheels and then actually run tests on installing them that would be a big step forward in test fidelity! But given the current state of the art, I'd be happy if they just got built from the same code that passed our tests in existing configurations. cibuildwheel does have an option to provide a test command. If we did that would we replace the existing tests or supplement? If supplementing that will presumably be a significant increase in build times. Especially if we did it for all wheels. So, how much of a build time hit do we want to take? I hear that Travis OSX builds were really slow, but from what I can see Circle isn't doing any OSX (other than what I added). The issue for addressing this is https://twistedmatrix.com/trac/ticket/9445 if you'd like to pick up where Adi left off. It's probably just resolving some conflicts, resubmitting for review, and bugging people to get it through the process :). https://github.com/twisted/twisted/pull/1056/files It looks like the builds I added in my wheel PR, for whatever that is worth. Well, other than picking xcode v9 vs me picking v10 which neither of the PRs use anyways. I added the requires over in the wheel PR as well in case it ends up saving a useful amount of our OS X quota. Now that I've got something rough in place, are there any opinions about how this should work? I don't know the present release workflow so I don't know if we'd want an automatic push to PyPI on tags (probably not) I definitely want this. This is really the whole point, what we're trying to get to :). Why don't you think we want this? Uhm... hmm... *shrug* I figured someone might want to do a final spot check or have manual intervention for a real release. You can't replace a file on PyPI so... but sure, I didn't mean to argue against automation. Here's one spot I did automatic deployment off of Travis to PyPI. https://github.com/altendky/gitignoreio/blob/aeeb9546b6f27f411b191699d3625b4d0a08e7cf/.travis.yml#L11-L19 Pretty straightforward though I think it does the wheel build which isn't what we'd really want. I'd have to check on how to deploy existing wheels as well as those from OS X and Windows. I'd suggest a separate PR? I mean maybe it'd be easier to get one PR reviewed than two but that doesn't seem like a great reason to mash things together. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] T9217 / PR1052: Wheels, wheels, and more wheels
Hi All, I am working on ticket #9217 / PR #1051 to add lots more wheel generation to the Twisted CI. I decided to give the cibuildwheel package a try and it made this process almost too easy (well... sort of :] ). I've got AppVeyor-Windows, Travis-Linux, and Circle-OSX all building a variety of wheels for the supported Python versions and bit depths. Travis doesn't save artifacts 'easily' so I went ahead and doubled up on Linux on Circle for now, though it's having some Docker issues at the moment and hasn't been successful yet. For some reason in this one case the project directory isn't getting mounted into the container as expected. wheel links: https://github.com/twisted/twisted/pull/1051#issuecomment-416743261 (and next comment) Now that I've got the wheel builds happening I figured it'd be good to try them out on 'real' machines. Turns out we get a failure on twisted.cred.test.test_strcred.SSHCheckerTests.test_isChecker for at least the two checks I've done so far (Windows and OSX). I haven't done more than a cursory look at that yet, but it's on the list to understand and resolve. More testing would of course be welcome. Real world, just trial Twisted's own tests, whatever would be appreciated if you are interested. failures with wheels: https://github.com/twisted/twisted/pull/1051#issuecomment-416977723 Overall, it's a bit unclear what the intended use of the various CI hosts are for Twisted. I hear that Travis OSX builds were really slow, but from what I can see Circle isn't doing any OSX (other than what I added). There wasn't any artifact storage being used on Circle either. So, I'm not sure if there was a reason to use Circle that went away, or... But, not having to hook Travis up to S3 or somesuch for storage is quite nice so Circle wins at least in that category. Now that I've got something rough in place, are there any opinions about how this should work? I don't know the present release workflow so I don't know if we'd want an automatic push to PyPI on tags (probably not), or just artifacts on the build server to grab manually (would need some S3 or such for Travis, or Circle for Linux builds as well). Anything else? Do we want automated tests against the wheels? cibuildwheel does have a feature for that though I haven't done anything with it yet. Anyways... hello, thanks for Twisted, and I hope this work ends up saving some people some time. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
