Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
2018-06-04 18:31 GMT+02:00 Victor Stinner : > Quick update a few days later. > (...) > Except Windows 7 which has issues with test_asyncio and > multiprocessing tests because this buildbot is slow, it seems like > most CIs are now stable. The bug wasn't specific to this buildbot, it was a very old race condition in the Windows ProactorEventLoop which only started recently to trigger test_asyncio failures. See https://bugs.python.org/issue33694 for the details. > Known issues: > > * PPC64 Fedora 3.x, PPC64LE Fedora 3.x, s390x RHEL 3.x: > https://bugs.python.org/issue33630 > * AIX: always red > * USBan: experimental buildbot > * Alpine: platform not supported yet (musl issues) Except of these few CIs, the main issue was test__xxsubinterpreters which is known to crash: https://bugs.python.org/issue33615 To fix CIs and give more time to Eric Snow to look at this crash, I decided to skip test__xxsubinterpreters. You may want to offer help to Eric to look into these tricky issues. So again, except of the few known issues listed above, all other CIs (Travis CI, AppVeyor, all other buildbots) should now pass. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
2018-06-07 4:45 GMT+02:00 Mariatta Wijaya : > Are there APIs we can use to check the status of builbots? Buildbots offer different ways to send notifications: emails and IRC bot for example. If you want to *poll* for recent builds, I don't know. I would suggest to use notifications (push) rather than polling. > Maybe we can have the our bots check for the buildbot status in backport > PRs. Right now, I'm not confortable with this idea because there is a risk to scare newcomers with false alarms and real bugs which are not coming from their changes, but may be known or are new but still unrelated to their changes. Moreover, even when a buildbot fails because of a real regression, a build may include multiple changes (I saw builds with more than 25 changes on some buildbots). Buildbot is different than a CI: slow buildbots are able to pack test multiple new changes at once. So again, there is a high risk of false alarms. Maybe I'm too conservative and we can try something with good documentation and proper warnings to explain properly such hypotetical notifications on pull requests. See also my other email which explains this differently: https://mail.python.org/pipermail/python-dev/2018-May/153759.html Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
On Wed, Jun 6, 2018 at 9:45 PM, Mariatta Wijaya wrote: > Are there APIs we can use to check the status of builbots? > Maybe we can have the our bots check for the buildbot status in backport > PRs. There is a REST API for buildbot; I have no idea how usable/useful it is though (but I think the new UI interacts with the master mostly via REST calls). I am planning to eventually get buildbot integration with GitHub set up, possibly in September. I think it should be possible to make only stable bots show up as status checks, or even just a select subset of the stable bots. -- Zach ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
Are there APIs we can use to check the status of builbots? Maybe we can have the our bots check for the buildbot status in backport PRs. On Wed, May 30, 2018, 2:33 AM Victor Stinner wrote: > > Buildbots only send email notifications to buildbot-sta...@python.org > when the state changes from success (green) to failure (red). It's > much simpler for me to spot a regression when most buildbots are > green. > > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
Activating Travis CI GitHub app is being tracked in https://github.com/python/core-workflow/issues/255 Let's not press the button until after the 3.7 release. On Wed, Jun 6, 2018, 3:57 PM INADA Naoki wrote: > > 2018年6月7日(木) 2:44 Brett Cannon : > >> >> On Wed, 6 Jun 2018 at 09:27 INADA Naoki wrote: >> >>> First I was also confused between travis-ci.com and travis-ci.org ... The documentation shows an example with .com, but Python organization uses .org. Victor >>> >>> .org is legacy. >>> >>> Open source projects can migrate to new .com. >>> >> >> ... eventually: "existing user accounts and repositories will be migrated >> over time." I have not seen any announcements or anything regarding how >> when or how to migrate ourselves. >> >> -Brett >> > > Before waiting notice from Travis-CI, we need to activate the repository > on new site. > > > https://docs.travis-ci.com/user/open-source-on-travis-ci-com/#Existing-Open-Source-Repositories-on-travis-ci.org > > However, open source repositories will be migrated to travis-ci.com > > gradually, > beginning at the end of Q2 2018. You will receive an email when the > migration for a repository is complete. This is an opt-in process: to have > a repository migrated over, it must first be activated on travis-ci.com. > > Could someone who is > python org admin > owner try activa > > ting from here? > https://travis-ci.com/profile/python > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mariatta.wijaya%40gmail.com > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
2018年6月7日(木) 2:44 Brett Cannon : > > On Wed, 6 Jun 2018 at 09:27 INADA Naoki wrote: > >> First I was also >>> confused between travis-ci.com and travis-ci.org ... The documentation >>> shows an example with .com, but Python organization uses .org. >>> >>> Victor >>> >> >> .org is legacy. >> >> Open source projects can migrate to new .com. >> > > ... eventually: "existing user accounts and repositories will be migrated > over time." I have not seen any announcements or anything regarding how > when or how to migrate ourselves. > > -Brett > Before waiting notice from Travis-CI, we need to activate the repository on new site. https://docs.travis-ci.com/user/open-source-on-travis-ci-com/#Existing-Open-Source-Repositories-on-travis-ci.org > However, open source repositories will be migrated to travis-ci.com gradually, beginning at the end of Q2 2018. You will receive an email when the migration for a repository is complete. This is an opt-in process: to have a repository migrated over, it must first be activated on travis-ci.com. Could someone who is python org admin owner try activa ting from here? https://travis-ci.com/profile/python ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
On Wed, 6 Jun 2018 at 09:27 INADA Naoki wrote: > First I was also >> confused between travis-ci.com and travis-ci.org ... The documentation >> shows an example with .com, but Python organization uses .org. >> >> Victor >> > > .org is legacy. > > Open source projects can migrate to new .com. > ... eventually: "existing user accounts and repositories will be migrated over time." I have not seen any announcements or anything regarding how when or how to migrate ourselves. -Brett > > Maybe, ssh is .com only feature. > > > https://blog.travis-ci.com/2018-05-02-open-source-projects-on-travis-ci-com-with-github-apps > > https://docs.travis-ci.com/user/open-source-on-travis-ci-com/ > > ___ > Python-Dev mailing list > Python-Dev@python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
> > First I was also > confused between travis-ci.com and travis-ci.org ... The documentation > shows an example with .com, but Python organization uses .org. > > Victor > .org is legacy. Open source projects can migrate to new .com. Maybe, ssh is .com only feature. https://blog.travis-ci.com/2018-05-02-open-source-projects-on-travis-ci-com-with-github-apps https://docs.travis-ci.com/user/open-source-on-travis-ci-com/ ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
On 06.06.2018 18:10, Victor Stinner wrote:
2018-06-04 21:37 GMT+02:00 Ivan Pozdeev :
https://docs.travis-ci.com/user/running-build-in-debug-mode/ is the official
doc on how to debug a Travis CI build via ssh.
Did you already try it? The doc mentions a "[Debug]" button, but I
cannot see it whereas I'm logged in in the Python organization.
Last I checked, they wrote it's only available for paid accounts (on
travis-ci.com) by default and only enabled for others on a case-by-case
basis, but I cannot find this info now.
So suggest you make a support ticket at
https://github.com/travis-ci/travis-ci .
I also tried the curl API call but it fails with:
{
"@type": "error",
"error_type": "wrong_credentials",
"error_message": "access denied"
}
curl -s -X POST \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Travis-API-Version: 3" \
-H "Authorization: token X" \
-d "{\"quiet\": true}" \
https://api.travis-ci.org/job/388706591/debug
where X is my hidden token ;-)
If I use an invalid token ID, I get a different error: just the string
"access denied", instead of a JSON dictionary. First I was also
confused between travis-ci.com and travis-ci.org ... The documentation
shows an example with .com, but Python organization uses .org.
Victor
--
Regards,
Ivan
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
2018-06-04 21:37 GMT+02:00 Ivan Pozdeev :
> https://docs.travis-ci.com/user/running-build-in-debug-mode/ is the official
> doc on how to debug a Travis CI build via ssh.
Did you already try it? The doc mentions a "[Debug]" button, but I
cannot see it whereas I'm logged in in the Python organization.
I also tried the curl API call but it fails with:
{
"@type": "error",
"error_type": "wrong_credentials",
"error_message": "access denied"
}
curl -s -X POST \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "Travis-API-Version: 3" \
-H "Authorization: token X" \
-d "{\"quiet\": true}" \
https://api.travis-ci.org/job/388706591/debug
where X is my hidden token ;-)
If I use an invalid token ID, I get a different error: just the string
"access denied", instead of a JSON dictionary. First I was also
confused between travis-ci.com and travis-ci.org ... The documentation
shows an example with .com, but Python organization uses .org.
Victor
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
No, replying only to you wasn't intended. https://docs.travis-ci.com/user/running-build-in-debug-mode/ is the official doc on how to debug a Travis CI build via ssh. On 04.06.2018 22:31, Victor Stinner wrote: FYI you only replied to me in private. Is it on purpose? I'm interested if I can learn how to get a SSH access to Travis CI! 2018-06-04 21:05 GMT+02:00 Ivan Pozdeev : On 04.06.2018 19:31, Victor Stinner wrote: 2018-05-30 11:33 GMT+02:00 Victor Stinner : I fixed a few tests which failed randomly. There are still a few, but the most annoying have been fixed. Quick update a few days later. For an unknown reason, test_multiprocessing_forkserver.TestIgnoreEINTR.test_ignore() started to fail very frequently but only on Travis CI. I have no explanation why only Travis CI. I failed to reproduce the issue on a Ubuntu Trusty container or in a Ubuntu Trusty VM. After hours of debug, I found the bug and wrote a fix. But the fix didn't work in all cases. A second fix and now it seems like the issue is gone! FYI Travis claim they provide ssh access on request to debug particularly pesky issues. Last time I tried, got no response from them though. https://bugs.python.org/issue33532 if you are curious about the strange multiprocessing send() which must block but it doesn't :-) Except Windows 7 which has issues with test_asyncio and multiprocessing tests because this buildbot is slow, it seems like most CIs are now stable. Known issues: * PPC64 Fedora 3.x, PPC64LE Fedora 3.x, s390x RHEL 3.x: https://bugs.python.org/issue33630 * AIX: always red * USBan: experimental buildbot * Alpine: platform not supported yet (musl issues) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan -- Regards, Ivan ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
2018-05-30 11:33 GMT+02:00 Victor Stinner : > I fixed a few tests which failed randomly. There are still a few, but > the most annoying have been fixed. Quick update a few days later. For an unknown reason, test_multiprocessing_forkserver.TestIgnoreEINTR.test_ignore() started to fail very frequently but only on Travis CI. I have no explanation why only Travis CI. I failed to reproduce the issue on a Ubuntu Trusty container or in a Ubuntu Trusty VM. After hours of debug, I found the bug and wrote a fix. But the fix didn't work in all cases. A second fix and now it seems like the issue is gone! https://bugs.python.org/issue33532 if you are curious about the strange multiprocessing send() which must block but it doesn't :-) Except Windows 7 which has issues with test_asyncio and multiprocessing tests because this buildbot is slow, it seems like most CIs are now stable. Known issues: * PPC64 Fedora 3.x, PPC64LE Fedora 3.x, s390x RHEL 3.x: https://bugs.python.org/issue33630 * AIX: always red * USBan: experimental buildbot * Alpine: platform not supported yet (musl issues) Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Keeping an eye on Travis CI, AppVeyor and buildbots: revert on regression
Hi, I fixed a few tests which failed randomly. There are still a few, but the most annoying have been fixed. I will continue to keep an eye on our CIs: Travis CI, AppVeyor and buildbots. Sometimes, even when I report a regression, the author doesn't fix the bug. But when a test fails on Travis CI or AppVeyor, we cannot merge a pull request, so it blocks our whole workflow. I will restart the policy that I proposed last year: if a change introduces a regression and I'm unable to fix it quickly (say in less than 2 hours and the author isn't available), I will simply revert the change. Please don't take it personally, the purpose is just to unblock our workflow and be able to detect other regressions. It's just an opportunity for the change author to fix the change without the pressure of the broken CI. Buildbots only send email notifications to buildbot-sta...@python.org when the state changes from success (green) to failure (red). It's much simpler for me to spot a regression when most buildbots are green. By the way, while miss-lington is really an amazing tool (thanks Mariatta!!!), most core developers (including me!) forgot that 2 years ago, the policy was to check if a change doesn't break buildbots before backporting a change (well, 2 years ago we forward-ported changes, but it's the same idea ;-)). Please remind that we only run tests on Linux and Windows as pre-commit checks on GitHub pull requests, whereas it's very common to spot bugs on buildbots thanks to the diversity of platforms and architectures (and different performance). All buildbot builders can be watched at: http://buildbot.python.org/all/#/builders But I prefer to watch notifications on IRC (#python-dev on Freenode) and the buildbot-status mailing list. https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/ -- Night gathers, AND NOW MY WATCH BEGINS. IT SHALL NOT END UNTIL MY DEATH. I shall take no wife, hold no lands, father no children. I shall wear no crowns and win no glory. I shall live and die at my post. I am the sword in the darkness. I am the watcher on the walls. I am the fire that burns against cold, the light that brings the dawn, the horn that wakes the sleepers, the shield that guards the realms of men. I pledge my life and honor to the Night's Watch, for this night and all the nights to come. Victor ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
