[ https://issues.apache.org/jira/browse/BEAM-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16415968#comment-16415968 ]
yifan zou commented on BEAM-3908: --------------------------------- These jobs were started by beam_release_nightlySnapshot, I'll look into why jobs were not terminated after test finished. > Leaderboard / gamestats leaking Dataflow Jobs > --------------------------------------------- > > Key: BEAM-3908 > URL: https://issues.apache.org/jira/browse/BEAM-3908 > Project: Beam > Issue Type: Bug > Components: runner-dataflow, testing > Affects Versions: Not applicable > Reporter: Luke Cwik > Assignee: Alan Myrvold > Priority: Critical > > I found that the leaderboard/gamestats Dataflow streaming jobs weren't being > cleaned up by the test infrastructure which lead to quota issues because all > the VMs/disks/memory being consumed causing other Jenkins jobs to fail. > I manually stopped all the jobs that had been running for more then 12 hrs. > There were about 20 jobs like this. > Example links: > https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_22_19_25-7861256924404398606?project=apache-beam-testing&organizationId=433637338589 > https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_18_49_54-7185486205606862436?project=apache-beam-testing&organizationId=433637338589 > https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_23_14_26-6599347078760080693?project=apache-beam-testing&organizationId=433637338589 > https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-06_18_32_33-7276493109541122240?project=apache-beam-testing&organizationId=433637338589 -- This message was sent by Atlassian JIRA (v7.6.3#76005)