What I invoked in the apache-beam-testing project: gcloud dataflow jobs list --created-before=-P5H --status=active --format="value(JOB_ID)" --region=us-central|xargs gcloud dataflow jobs cancel
wt., 10 gru 2019 o 13:28 Łukasz Gajowy <lgaj...@apache.org> napisał(a): > Hi Kirill, > > We (along with Michał and Kamil) noticed the problem as well in Dataflow > ValidatesRunner suites yesterday. I started investigating the problem and I > noticed that there are jobs running for 5 days and counting. It seems that > those are not stopped by "beam_CancelStaleDataflowJobs" job that runs > randomly each day. After investigating deeper, it seems that lots of the > jobs that are stale are from " > https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/"; job that > is currently being ABORTED due to timeout. > > Some tests (I'm not sure if this is the exhaustive list but they seem to > appear in the dataflow console repeatedly) that seem to not be killed and > eat our resources: > - test_reshuffle_preserves_timestamps > <https://github.com/apache/beam/blob/719b8cc5e51dcd3e98425ecae5ec246657d46eca/sdks/python/apache_beam/transforms/util_test.py#L487> > (spotted multiple times in the dataflow console) (Python SDK) > - test_flatten_same_pcollections > <https://github.com/apache/beam/blob/44d456830442e5f13b7fd3bd684695e2b69e2c0d/sdks/python/apache_beam/transforms/ptransform_test.py#L596> > (Python SDK) > - testPairWithIndexWindowedTimestampedBounded > <https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java#L158> > (Java SDK) > - testPairWithIndexBasicBounded > <https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java#L125> > > I created https://issues.apache.org/jira/browse/BEAM-8938 to track tests > like this. Right now I'm going to kill all jobs that hang like this and > ignore the tests that I tracked down in a pr for the issue I created. > > I think it's good that job_CancelStaleDataflowJobs didn't catch them - I > think that if it did, we would not spot the problem. Is it possible to set > up some alerting on Dataflow instead of automatically cleaning the jobs? > IMO we should fix the tests rather than cancel them. > > Thanks, > Łukasz > > > wt., 10 gru 2019 o 00:09 Kirill Kozlov <kirillkoz...@google.com> > napisał(a): > >> Hello everyone! >> >> It looks like JavaPostCommit Jenkins tests [1] are failing due to CPU >> quota limitations. >> Could someone please look into this? >> >> [1] >> https://builds.apache.org/job/beam_PostCommit_Java/4838/testReport/junit/org.apache.beam.examples.complete/TrafficMaxLaneFlowIT/testE2ETrafficMaxLaneFlow/ >> >> -- >> Kirill >> >
