We saw similar timeouts of the python precommit and it usually break the Jenkins build workers. I've run the precommit manually several times. It sometimes stuck at :sdks:python:docs and consumes 80G+ memory. Our build VMs eventually ran out of memory (104G memory available in total) then disconnected. Not sure what happened during that job.
On Sat, Jul 13, 2019 at 10:27 PM Tanay Tummalapalli <ttanay...@gmail.com> wrote: > Yes. It passed on the second attempt. > > But, I'm yet to figure out why it hangs for ~1.5 hours. > > On Sun, Jul 14, 2019 at 10:36 AM Rakesh Kumar <rakeshku...@lyft.com> > wrote: > >> >> >> Even I am running into the same issue. Though my test passed but >> <https://builds.apache.org/job/beam_PreCommit_Python_Phrase/656/console>somehow >> the task didn't terminate, eventually the task was aborted. I have already >> tried a couple of times to retrigger tye python precommit but it failed >> every time. >> >> @Tanay did it pass it for you? >> >> >> >> >> >> On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli <ttanay...@gmail.com> >> wrote: >> >>> Thank You Valentyn! >>> >>> I'll retest it. >>> Hopefully, it's a transient issue. >>> >>> Regards, >>> - Tanay Tummalapalli >>> >>> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev <valen...@google.com> >>> wrote: >>> >>>> No, we did not reduce the timeout recently. Looking at console logs, >>>> nothing happened for an hour or so, >>>> >>>> *06:57:50 py27-cython: commands succeeded 06:57:50 congratulations :) >>>> 06:57:50 * >>>> >>>> *06:57:50* >* Task :sdks:python:preCommitPy2**08:22:33* Build timed out >>>> (after 120 minutes). Marking the build as aborted. >>>> >>>> >>>> However, we can also see in the logs that py36-cython suite never >>>> started, not sure way. I assume gradle waited for this suite to finish. >>>> Try "retest this please", hopefully this is a transient gradle issue. I >>>> did not observe it before. >>>> >>>> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli <ttanay...@gmail.com> >>>> wrote: >>>> >>>>> Hi Udi, >>>>> >>>>> I rebased another PR[1] onto the fix mentioned above. The lint error >>>>> is fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is failing >>>>> because of a timeout at 120 minutes[2]. >>>>> The log says "Build timed out (after 120 minutes). Marking the build >>>>> as aborted." >>>>> Another PR's Python PreCommit job aborted with the same error[3]. >>>>> >>>>> I found this issue - "[BEAM-3040] Python precommit timed out after 150 >>>>> minutes"[4]. >>>>> Was the timeout reduced recently? >>>>> >>>>> Regards, >>>>> - Tanay Tummalapalli >>>>> >>>>> [1] https://github.com/apache/beam/pull/8871 >>>>> [2] >>>>> https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull >>>>> >>>>> [3] https://github.com/apache/beam/pull/9050 >>>>> [4] https://issues.apache.org/jira/browse/BEAM-3040 >>>>> >>>>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> wrote: >>>>> >>>>>> This is due to >>>>>> https://github.com/apache/beam/pull/8969 >>>>>> and >>>>>> https://github.com/apache/beam/pull/8934 >>>>>> being merged today. >>>>>> >>>>>> Fix is here: https://github.com/apache/beam/pull/9044 >>>>>> >>>>>
