+1 to periodic cleanups of the workers. I do not know what would be a good frequency daily or a different one. Do we have a jira for this?
On Thu, Apr 30, 2020 at 2:22 PM Udi Meiri <eh...@google.com> wrote: > I summarized my idea here: https://issues.apache.org/jira/browse/BEAM-9865 > +1 to this idea as well. > > > On Thu, Apr 30, 2020 at 2:01 PM Maximilian Michels <m...@apache.org> wrote: > >> On 30.04.20 21:48, Hannah Jiang wrote: >> > --info tag was passed to docker image build commands with PythonDocker >> > Precommit to capture more logs. Without the tag, errors from >> > DockerFile step are not printed out to the console. >> >> Thanks for the info (pun intended). >> >> On 30.04.20 21:48, Hannah Jiang wrote: >> > Indeed, I can see the no space left on device in the following but >> > not in the log above: >> > >> > --info tag was passed to docker image build commands with PythonDocker >> > Precommit to capture more logs. Without the tag, errors from DockerFile >> > step are not printed out to the console. >> > >> > On Thu, Apr 30, 2020 at 11:19 AM Udi Meiri <eh...@google.com >> > <mailto:eh...@google.com>> wrote: >> > >> > I checked node 8 and it had over 40GB space available. Does your job >> > require more than that? >> > >> > Long term, I'm thinking we could clean up workspaces for successful >> > jobs. This should free up additional space (I guess at least 100GB). >> > https://plugins.jenkins.io/ws-cleanup/ - we already use this plugin >> > to clean workspaces at job start. >> > >> > >> > On Thu, Apr 30, 2020, 07:33 Maximilian Michels <m...@apache.org >> > <mailto:m...@apache.org>> wrote: >> > >> > *It's working again, probably because it's running on a >> different >> > machine now. >> > >> > Who can check the disk space of the Jenkins hosts? >> > >> > Thanks, >> > Max >> > >> > On 30.04.20 11:55, Maximilian Michels wrote: >> > > Sorry, I meant to include the Jenkins log: >> > > >> > >> https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Streaming_PR/5/console >> > > >> > > Thanks for investigating Hannah! Indeed, I can see the no >> > space left on >> > > device in the following but not in the log above: >> > > >> > >> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/473/console >> > > >> > > I'm going to try running the build again. Do you think we >> > could add more >> > > storage to our Jenkins hosts or delete old build data? >> > > >> > > Thanks, >> > > Max >> > > >> > > On 30.04.20 08:43, Hannah Jiang wrote: >> > >> Max, I found a link from your PR and noticed below errors. >> > This would be >> > >> the true error. >> > >> >> > >> *07:57:03* >*Task :sdks:python:container:py37:docker* >> > >> *07:57:03* [91mERROR: Could not install packages due to an >> > EnvironmentError: [Errno 28] No space left on device >> > >> *07:57:03* >> > >> *07:57:03* [0m >> > >> *07:57:03* >*Task :sdks:python:container:py35:docker* >> > >> *07:57:03* [91mERROR: Could not install packages due to an >> > EnvironmentError: [Errno 28] No space left on device >> > >> >> > >> >> > >> >> > >> On Wed, Apr 29, 2020 at 5:59 PM Hannah Jiang >> > <hannahji...@google.com <mailto:hannahji...@google.com> >> > >> <mailto:hannahji...@google.com >> > <mailto:hannahji...@google.com>>> wrote: >> > >> >> > >> There is a PythonDocker Precommit test running for PRs >> > with Python >> > >> changes. It seems running well.[1] >> > >> Max, can you please give me a link so I can check more >> > details? Do >> > >> other images with different Python versions fail as well? >> > >> >> > >> >> > 1. >> https://builds.apache.org/job/beam_PreCommit_PythonDocker_Commit/ >> > >> >> > >> >> > >> On Wed, Apr 29, 2020 at 2:44 PM Ahmet Altay >> > <al...@google.com <mailto:al...@google.com> >> > >> <mailto:al...@google.com <mailto:al...@google.com>>> >> wrote: >> > >> >> > >> +Valentyn Tymofieiev <mailto:valen...@google.com >> > <mailto:valen...@google.com>> +Hannah Jiang >> > >> <mailto:hannahji...@google.com >> > <mailto:hannahji...@google.com>> -- in case they have relevant >> > >> information. >> > >> >> > >> On Wed, Apr 29, 2020 at 12:35 PM Maximilian Michels >> > >> <m...@apache.org <mailto:m...@apache.org> >> > <mailto:m...@apache.org <mailto:m...@apache.org>>> wrote: >> > >> >> > >> Hi, >> > >> >> > >> has anyone noticed the Python 3.7 Docker >> > container fails to >> > >> build? I >> > >> haven't been able to build the Python 3.7 >> > container, neither >> > >> locally nor >> > >> on Jenkins. >> > >> >> > >> I get: >> > >> >> > >> 17:48:10 > Task >> :sdks:python:container:py37:docker >> > >> 17:49:36 The command '/bin/sh -c pip install -r >> > >> /tmp/base_image_requirements.txt && python -c >> "from >> > >> google.protobuf.internal import >> > api_implementation; assert >> > >> api_implementation._default_implementation_type >> > == 'cpp'; print >> > >> ('Verified fast protobuf used.')" && rm -rf >> > >> /root/.cache/pip' returned a >> > >> non-zero code: 1 >> > >> 17:49:36 >> > >> 17:49:36 > Task >> > :sdks:python:container:py37:docker FAILED >> > >> >> > >> >> > >> Cheers, >> > >> Max >> > >> >> > >> >
