[issue37043] Buildbots fail when new files are added

[Jason R. Coombs]

Jason R. Coombs  added the comment:

I encountered this issue again today in issue46118.

I started looking into creating the patchcheck, but I realize it may be a 
little tricky to detect the introduction of a new folder, because `git diff` 
doesn't actually report new folders, and my initial research indicates it's 
[far from straightforward](https://stackoverflow.com/a/36293191/70170).

I thought maybe git diff --dirstat might help, but it doesn't. It reports 100% 
different for a new file in a new folder and same for a new file in an existing 
folder.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37043] Buildbots fail when new files are added

[Jason R. Coombs]

Jason R. Coombs  added the comment:

In GH-13565, @yan12125 provides [this 
reference](https://github.com/python/buildmaster-config/blob/master/master/custom/factories.py)
 to the buildbot code that copies the code and thus imposes the requirement to 
declare source directories.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37043] Buildbots fail when new files are added

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
nosy: +barry

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37043] Buildbots fail when new files are added

[Zachary Ware]

Zachary Ware  added the comment:

The issue here is not with buildbots, but with installation on POSIX platforms. 
 We do now have a couple of buildbots that install Python to a local location 
before running the tests, which is what flushes this out (see 
https://github.com/python/buildmaster-config/blob/master/master/custom/factories.py#L134-L147;
 they simply run `make install` before running the tests).

Adding these buildbots as pre-merge CI is not currently an option due to 
security implications (I don't want unreviewed code running on my home 
network).  I have plans to eventually allow certain builders to be run 
pre-merge iff the `awaiting merge` label is present on the PR, but I haven't 
had time to work on that yet.

It might be possible to adjust one of the Travis builds to install before 
running tests, but that leaves some other tests un-run, which just relocates 
the problem.

Removing reliance on an explicit listing of directories sounds nice, but does 
open up the possibility of installing more than expected if run from a dirty 
checkout.

What about adding a "new directories added to Makefile.pre.in" check to 
Tools/scripts/patchcheck.py?

--
nosy: +zach.ware

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37043] Buildbots fail when new files are added

[Jason R. Coombs]

New submission from Jason R. Coombs :

As [reported here](https://bugs.python.org/issue34632#msg343445), I submitted a 
pull request that passed all tests locally and in CI, but when accepted, build 
bots started to fail as a result of new files having been added to the project. 
It seems it's necessary in Makefile.pre.in to duplicate the git manifest to 
declare those directories... but only for buildbot builds. I don't fully 
understand why that is the case, but it would be nicer if there were 
protections from this footgun, especially since this issue may manifest several 
times in the same PR.

I can think of some ways to prevent this undesirable behavior:

- Include buildbot builds in the GitHub merge request checks.
- Add a GitHub bot that checks merge requests that create directories and alert 
the review of directories created if Makefile.pre.in doesn't include changes 
reflecting those directories.
- Remove the reliance on a separate directory listing in Makefile.pre.in.

--
components: Tests
messages: 343478
nosy: jaraco
priority: normal
severity: normal
status: open
title: Buildbots fail when new files are added
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com