The Jenkins worker configurations is a pain point of beam build and tests, and it is indeed difficult to debug. Originally, python tests such as beam_PostCommit_Python_Verify only run on one worker due to BEAM-1817 <https://issues.apache.org/jira/projects/BEAM/issues/BEAM-3395?filter=allissues>. We probably need to do the same thing for beam_PreCommit_Python_GradleBuild in short term. In order to solve this problem, we did research and experiments on running Jenkins tests within a container and organized a short documentation. It is being reviewed within Engprod team and will be shared for wider review shortly.
Yifan Zou On Mon, Apr 16, 2018 at 1:10 PM Robert Bradshaw <rober...@google.com> wrote: > I've been trying to debug why beam_PreCommit_Python_GradleBuild seems to be > failing so often, and it looks like the beam-sdks-python:setupVirtualenv > task succeeds on beam2 and beam6, but always fails on beam1, beam3, beam4, > and beam8. (I didn't see any runs on beam5 or beam7, I vaguely seem to > remember beam5 being blacklisted...) I can't reproduce the failure locally > and the remote logs (e.g. > https://builds.apache.org/job/beam_PreCommit_Python_GradleBuild/471/console > ) don't seem to be very enlightening either. This leads to a couple of > questions: > > * How are our jenkins beam workers configured, and why aren't they the > same? > * How does one go about debugging failures like this? > > Before too much effort is invested, how far are we from using containers to > manage the build environments? >
