TL;DR - lets just define an env var to be practical about this, but discourage its use.
All CI systems I've dealt with, whatever they're called, make it possible to officially detect when running within one. Otherwise code that *needs* to do so resorts to non-canonical extreme bespoke measures to try and figure it out (hmm is buildkite-worker in my parent process tree? that's interesting... what is my hostname? what's laying around the filesystem... does my branch name start with `gh-readonly-queue/`...) those are "wrong" ways of doing it when we could provide a right way and just discourage its use. ie: set a "RUNNING_UNDER_BUILDBOT_BUT_EVERYTIME_YOUR_TEST_USES_THIS_FACT_A_PUPPY_DIES=1" in the environment for example. that socializes reserving such things for extreme circumstances well enough. On Tue, Aug 26, 2025 at 4:52 PM Jason R. Coombs <jar...@jaraco.com> wrote: > I agree in general. > > But if you dig into the issue I've encountered, the cause of the failure > is elusive, the tests that are implicated in triggering the failure are > unrelated the the actual cause, and those tests are still providing value > where they're succeeding. I don't want to disable them universally and > reduce the coverage. I also wish to unblock the merging of the changes, > which are stable in other environments (buildbots, cpython github workers, > and upstream in importlib_metadata). > > I've spent hours and hours this past weekend just trying to isolate the > issue by swapping out changes and triggering buildbot runs, but that > approach was slow and tedious and ultimately ineffective (I failed to > isolate the pertinent factors). I'm exhausted and need to find a way to > replicate the failure in some developer environment (a skill I was hoping > to solicit from others) in order to perform more rapid experimentation. > > My intention was to tackle things in this order: > > 1. Suppress test failures on buildbots. Ideally, it would be only the > implicated buildbots, but I was willing to hit all buildbots for this small > exclusion. If the buildbot execution could be detected by buildbot name, > that might provide more fine grained control. > 2. Merge the PR so that those changes could gain soak time in the alpha > builds (and close out this effort rather than making it a dependency on > this emergent issue). > 3. File an issue about the failures in the buildbot to solicit help to > fix the root cause of the buildbot failures. > > I guess I could just replace (1) with a broad exclusion, but it would be > safer to limit the scope of the test skip to help retain coverage and also > continue to demonstrate that the test passes in non-buildbot environments. > > My preference would be to expose functionality in buildbots similar to the > environment indicators exposed by other CI environments like Travis CI, > Appveyor, and GitHub Actions (all of which expose both generic `CI` > indicators and more specific environment details (action name, runner > details). Indeed, when I asked an AI about how to detect a buildbot > environment, it suggested a couple of environment variables I could use for > that purpose (assuming/hallucinating based on what it expected based on > other runners). > > If we instead want to by principle disallow detection of the buildbot > environments, it would make sense to make that an explicit note in the > buildbot documentation, so we can reference the guidance and help future > affected use-cases resolve the approach quickly. > > That said, if we go with the latter approach, this thread could serve as > the pertinent guidance. > > How can we reconcile these tradeoffs? > > > > > On Mon, Aug 25, 2025 at 5:14 PM, Zachary Ware <z...@python.org<mailto: > z...@python.org>> wrote: > > On Mon, Aug 25, 2025 at 11:39 AM Gregory P. Smith <g...@krypto.org<mailto: > g...@krypto.org>> wrote: > > I think an environment variable could be set globally for all workers via > the code in > https://github.com/python/buildmaster-config/tree/main/master/custom > > It could, but I personally really don't think we want to. There should not > be any difference between a buildbot run and a regular test run on a > machine of similar setup, so I don't think we want to be trying to skip > things specifically on buildbots. If it *must* be skipped on buildbots, > skip it everywhere. > > _______________________________________________ > Python-Buildbots mailing list -- python-buildbots@python.org > To unsubscribe send an email to python-buildbots-le...@python.org > https://mail.python.org/mailman3//lists/python-buildbots.python.org > Member address: g...@krypto.org > _______________________________________________ Python-Buildbots mailing list -- python-buildbots@python.org To unsubscribe send an email to python-buildbots-le...@python.org https://mail.python.org/mailman3//lists/python-buildbots.python.org Member address: arch...@mail-archive.com
