https://github.com/apache/beam/pull/10342 - pr that skips the tests listed above - looking for reviewers
Thanks! wt., 10 gru 2019 o 13:30 Łukasz Gajowy <lgaj...@apache.org> napisał(a): > What I invoked in the apache-beam-testing project: > > gcloud dataflow jobs list --created-before=-P5H --status=active > --format="value(JOB_ID)" --region=us-central|xargs gcloud dataflow jobs > cancel > > wt., 10 gru 2019 o 13:28 Łukasz Gajowy <lgaj...@apache.org> napisał(a): > >> Hi Kirill, >> >> We (along with Michał and Kamil) noticed the problem as well in Dataflow >> ValidatesRunner suites yesterday. I started investigating the problem and I >> noticed that there are jobs running for 5 days and counting. It seems that >> those are not stopped by "beam_CancelStaleDataflowJobs" job that runs >> randomly each day. After investigating deeper, it seems that lots of the >> jobs that are stale are from " >> https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/"; job that >> is currently being ABORTED due to timeout. >> >> Some tests (I'm not sure if this is the exhaustive list but they seem to >> appear in the dataflow console repeatedly) that seem to not be killed and >> eat our resources: >> - test_reshuffle_preserves_timestamps >> <https://github.com/apache/beam/blob/719b8cc5e51dcd3e98425ecae5ec246657d46eca/sdks/python/apache_beam/transforms/util_test.py#L487> >> (spotted multiple times in the dataflow console) (Python SDK) >> - test_flatten_same_pcollections >> <https://github.com/apache/beam/blob/44d456830442e5f13b7fd3bd684695e2b69e2c0d/sdks/python/apache_beam/transforms/ptransform_test.py#L596> >> (Python SDK) >> - testPairWithIndexWindowedTimestampedBounded >> <https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java#L158> >> (Java SDK) >> - testPairWithIndexBasicBounded >> <https://github.com/apache/beam/blob/c2f0d282337f3ae0196a7717712396a5a41fdde1/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SplittableDoFnTest.java#L125> >> >> I created https://issues.apache.org/jira/browse/BEAM-8938 to track tests >> like this. Right now I'm going to kill all jobs that hang like this and >> ignore the tests that I tracked down in a pr for the issue I created. >> >> I think it's good that job_CancelStaleDataflowJobs didn't catch them - I >> think that if it did, we would not spot the problem. Is it possible to set >> up some alerting on Dataflow instead of automatically cleaning the jobs? >> IMO we should fix the tests rather than cancel them. >> >> Thanks, >> Łukasz >> >> >> wt., 10 gru 2019 o 00:09 Kirill Kozlov <kirillkoz...@google.com> >> napisał(a): >> >>> Hello everyone! >>> >>> It looks like JavaPostCommit Jenkins tests [1] are failing due to CPU >>> quota limitations. >>> Could someone please look into this? >>> >>> [1] >>> https://builds.apache.org/job/beam_PostCommit_Java/4838/testReport/junit/org.apache.beam.examples.complete/TrafficMaxLaneFlowIT/testE2ETrafficMaxLaneFlow/ >>> >>> -- >>> Kirill >>> >>
