Thanks @Robert! Python PreCommit is green now! On Mon, Jul 22, 2019 at 5:57 PM Robert Bradshaw <rober...@google.com> wrote:
> This was due to a bad release artifact push. This has now been fixed > upstream. > > On Mon, Jul 22, 2019 at 11:00 AM Robert Bradshaw <rober...@google.com> > wrote: > > > > Looks like https://sourceforge.net/p/docutils/bugs/365/ > > > > On Sun, Jul 21, 2019 at 11:56 PM Tanay Tummalapalli <ttanay...@gmail.com> > wrote: > >> > >> Hi everyone, > >> > >> The Python PreCommit from the Jenkins job "beam_PreCommit_Python_Cron" > is failing[1]. The task :sdks:python:docs is failing with this traceback: > >> > >> Traceback (most recent call last): > >> File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/bin/sphinx-apidoc", > line 8, in <module> > >> from sphinx.apidoc import main > >> File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/apidoc.py", > line 27, in <module> > >> from sphinx.quickstart import EXTENSIONS > >> File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/sphinx/quickstart.py", > line 35, in <module> > >> from docutils.utils import column_width > >> File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/utils/__init__.py", > line 21, in <module> > >> import docutils.io > >> File > "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Cron/src/sdks/python/build/srcs/sdks/python/target/.tox-docs/docs/local/lib/python2.7/site-packages/docutils/io.py", > line 348 > >> (self.destination.mode, mode)), file=self._stderr) > >> ^ > >> SyntaxError: invalid syntax > >> > >> It seems to be due to a bug in docutils which is a dependency of sphinx. > >> Unfortunately, I don't have much context about this. I'd love to help > with fixing this. > >> > >> There is another failure in test_wordcount_it of task > :sdks:python:test-suites:dataflow:preCommitIT, but it seems to be a flaky > test. > >> > >> [1] > https://scans.gradle.com/s/ygctjiqxcz6qa/console-log?task=:sdks:python:docs > >> > >> On Mon, Jul 15, 2019 at 11:16 PM Yifan Zou <yifan...@google.com> wrote: > >>> > >>> We saw similar timeouts of the python precommit and it usually break > the Jenkins build workers. I've run the precommit manually several times. > It sometimes stuck at :sdks:python:docs and consumes 80G+ memory. Our build > VMs eventually ran out of memory (104G memory available in total) then > disconnected. Not sure what happened during that job. > >>> > >>> > >>> > >>> On Sat, Jul 13, 2019 at 10:27 PM Tanay Tummalapalli < > ttanay...@gmail.com> wrote: > >>>> > >>>> Yes. It passed on the second attempt. > >>>> > >>>> But, I'm yet to figure out why it hangs for ~1.5 hours. > >>>> > >>>> On Sun, Jul 14, 2019 at 10:36 AM Rakesh Kumar <rakeshku...@lyft.com> > wrote: > >>>>> > >>>>> > >>>>> > >>>>> Even I am running into the same issue. Though my test passed but > somehow the task didn't terminate, eventually the task was aborted. I have > already tried a couple of times to retrigger tye python precommit but it > failed every time. > >>>>> > >>>>> @Tanay did it pass it for you? > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Fri, Jul 12, 2019 at 2:22 PM Tanay Tummalapalli < > ttanay...@gmail.com> wrote: > >>>>>> > >>>>>> Thank You Valentyn! > >>>>>> > >>>>>> I'll retest it. > >>>>>> Hopefully, it's a transient issue. > >>>>>> > >>>>>> Regards, > >>>>>> - Tanay Tummalapalli > >>>>>> > >>>>>> On Sat, Jul 13, 2019 at 2:39 AM Valentyn Tymofieiev < > valen...@google.com> wrote: > >>>>>>> > >>>>>>> No, we did not reduce the timeout recently. Looking at console > logs, nothing happened for an hour or so, > >>>>>>> > >>>>>>> 06:57:50 py27-cython: commands succeeded > >>>>>>> 06:57:50 congratulations :) > >>>>>>> 06:57:50 > >>>>>>> > >>>>>>> 06:57:50 > Task :sdks:python:preCommitPy2 > >>>>>>> 08:22:33 Build timed out (after 120 minutes). Marking the build as > aborted. > >>>>>>> > >>>>>>> > >>>>>>> However, we can also see in the logs that py36-cython suite never > started, not sure way. I assume gradle waited for this suite to finish. > >>>>>>> Try "retest this please", hopefully this is a transient gradle > issue. I did not observe it before. > >>>>>>> > >>>>>>> On Fri, Jul 12, 2019 at 1:22 PM Tanay Tummalapalli < > ttanay...@gmail.com> wrote: > >>>>>>>> > >>>>>>>> Hi Udi, > >>>>>>>> > >>>>>>>> I rebased another PR[1] onto the fix mentioned above. The lint > error is fixed, but, the "beam_PreCommit_Python_Commit" Jenkins job is > failing because of a timeout at 120 minutes[2]. > >>>>>>>> The log says "Build timed out (after 120 minutes). Marking the > build as aborted." > >>>>>>>> Another PR's Python PreCommit job aborted with the same error[3]. > >>>>>>>> > >>>>>>>> I found this issue - "[BEAM-3040] Python precommit timed out > after 150 minutes"[4]. > >>>>>>>> Was the timeout reduced recently? > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> - Tanay Tummalapalli > >>>>>>>> > >>>>>>>> [1] https://github.com/apache/beam/pull/8871 > >>>>>>>> [2] > https://builds.apache.org/job/beam_PreCommit_Python_Commit/7412/consoleFull > >>>>>>>> [3] https://github.com/apache/beam/pull/9050 > >>>>>>>> [4] https://issues.apache.org/jira/browse/BEAM-3040 > >>>>>>>> > >>>>>>>> On Fri, Jul 12, 2019 at 5:42 AM Udi Meiri <eh...@google.com> > wrote: > >>>>>>>>> > >>>>>>>>> This is due to > >>>>>>>>> https://github.com/apache/beam/pull/8969 > >>>>>>>>> and > >>>>>>>>> https://github.com/apache/beam/pull/8934 > >>>>>>>>> being merged today. > >>>>>>>>> > >>>>>>>>> Fix is here: https://github.com/apache/beam/pull/9044 >