On Mon, Mar 11, 2019 at 7:03 AM Michael Luckey <[email protected]> wrote:
> > > On Mon, Mar 11, 2019 at 3:51 AM Kenneth Knowles <[email protected]> wrote: > >> I have never actually tried to run a full build recently. It takes a long >> time and usually isn't what is needed for a particular change. FWIW I view >> Beam at this point as a mini-monorepo, so each directory and target can be >> healthy/unhealthy on its own. >> > > Fair Point. Totally agree. > > >> >> But it does seem like we should at least know what is unhealthy and why. >> Have you been filing Jiras about the failures? Are they predictable? Are >> they targets that pass in Jenkins but not in vanilla build? That would mean >> our Jenkins environment is too rich and should be slimmed down probably. >> >> Kenn >> > > Unfortunately those failures are not really predictable. Usually, I start > with plain './gradlew build' and keep adding some '-x > :beam-sdks-python:testPy2Gcp -x :beam-sdks-python:testPython' until build > succeeds. Afterwards it seems to be possible to remove this exclusions step > by step, thereby filling the build cache, which on next reruns might have > some impact on how tasks are executed. > > Most of failures are python related. Had not much success getting into > those. From time to time I see 'seemingly' similar failures on Jenkins, but > tracing on python is more difficult coming from the more java background. > Also using Mac I believe to remember that preinstalled python had some > issues/differences compared with private installs. Others are those > Elasticsearch - which were worked on lately - and ClickHouse tests which > seem to be still flaky. > > So I mainly blamed my setup and did not yet have the time to further track > those failures down. But > > As I did use a vanilla system and was not able to get beam to build, i got > thinking about > > 1. The release process > The release manager has lot of stuff to take care for, but is also > supposed to run a full gradle build on her local machine [1]. Apart from > that being a long lasting task, if it keeps failing this puts additional > burden on the release manager. So I was wondering, why we can not push that > to Jenkins as we do with all these other tests [2]. Here I did not find any > existing Job doing such, so wanted to ask for feedback here. > > If a full build has to be run - and of course it makes some sense on > release - I would suggest to get that running on a regular base on Jenkins > just to ensure not to be surprised during release. And as a sideeffect > enable the release manager to also delegate this to Jenkins to free her > time (and dev box). > +1, this will be great. Quite often we end up catching issues right when we are doing the release. I would one up this request and suggest a Jenkins job running most of the release process as much as possible to avoid last minute surprises. Also, I wonder if we could build our releases in a controlled environment (e.g. container), that way the release would be more reproducible and the release manager would have to spend less time setting their environment (e.g. environment for building python in your case). > > Until now I had not yet have the time to investigate, whether all this > pre/Post/elseJobs cover all our tests on all modules. Hoped someone else > could point to the list of jobs which cover all tests. > > 2. Those newcomers > As a naive newcomer to a project I usually deal with those jars. After > problems arose, I might download corresponding sources and try to > investigate. First thing I usually do here is a make/build/install to get > those private builds in. Also on our contributor site [3] we recommend > ensuring to be able to run all tests with 'gradlew check' which is not to > far away from full build. Probably no one would expect a full build to fail > here, which again makes me think we need an equivalent Job on Jenkins here? > > Of course it is also fully reasonable to not support this full build in > line with that mini-monorepo. > > On the other hand, after filling the build cache - and enable more task > for caching - a full build should not be to expensive in general. Although > I d prefer some more granularity like e.g. 'gradlew -p sdks/java build' to > build all java, which, if I understand correctly, would be a side effect of > fixing [4]. > > [1] > https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh#L142 > [2] > https://github.com/apache/beam/blob/master/release/src/main/scripts/verify_release_build.sh#L168-L190 > [3] https://beam.apache.org/contribute/ > [4] https://issues.apache.org/jira/browse/BEAM-4046 > > >> >> On Sun, Mar 10, 2019 at 7:05 PM Michael Luckey <[email protected]> >> wrote: >> >>> Hi, >>> >>> while fiddling with beams release process and trying to get that working >>> on my machine, I again stumbled about 'gradlew build' >>> >>> Till now I am unable to get a full build run successfully on my machine. >>> Keeps failing. After setting up a clean docker instance and trying to run >>> there, I also failed miserably. Which of kind is expected as part of the >>> tests rely on docker itself which probably does not work out of the box. >>> But apart from these expected failures, there were also tons of other >>> failures. Need to setup a full blown vm and test again, but wanted to ask >>> here first as after looking into Jenkins setup, I was also unable to find a >>> job which actually executes a full build. >>> >>> So I am wondering, whether >>> - anyone is actually able to successfully execute a full build >>> (preferable without build cache) >>> - it is intended we do only run a 'gradlew build' during release on >>> release managers box >>> - if that full build is somehow implicit in Jenkins setup, and I am just >>> not finding my way to the job/list of jobs which execute this full build >>> >>> Any help/insights appreciated here. >>> >>> michel >>> >>> >>>
