In the short term, I could try to limit precommits to these two machines
following that example, but presumably that would mean longer queues. Who
owns these machines? Could we just wipe them and install fresh, modern,
consistent OS/environments on them? (The container story seems like a great
long-term solution, especially for local reproducibility, but probably not
as easy...)

On Mon, Apr 16, 2018 at 2:33 PM Yifan Zou <> wrote:

> The Jenkins worker configurations is a pain point of beam build and tests,
> and it is indeed difficult to debug. Originally, python tests such as
> beam_PostCommit_Python_Verify only run on one worker due to BEAM-1817
> <>.
> We probably need to do the same thing for
> beam_PreCommit_Python_GradleBuild in short term.
> In order to solve this problem, we did research and experiments on running
> Jenkins tests within a container and organized a short documentation. It is
> being reviewed within Engprod team and will be shared for wider review
> shortly.
> Yifan Zou
> On Mon, Apr 16, 2018 at 1:10 PM Robert Bradshaw <>
> wrote:
>> I've been trying to debug why beam_PreCommit_Python_GradleBuild seems to
>> be
>> failing so often, and it looks like the beam-sdks-python:setupVirtualenv
>> task succeeds on beam2 and beam6, but always fails on beam1, beam3, beam4,
>> and beam8. (I didn't see any runs on beam5 or beam7, I vaguely seem to
>> remember beam5 being blacklisted...) I can't reproduce the failure locally
>> and the remote logs (e.g.
>> ) don't seem to be very enlightening either. This leads to a couple of
>> questions:
>> * How are our jenkins beam workers configured, and why aren't they the
>> same?
>> * How does one go about debugging failures like this?
>> Before too much effort is invested, how far are we from using containers
>> to
>> manage the build environments?

Reply via email to