There's been a fairly long-standing discussion regarding merging ValidatesRunner tests in the same class together into one pipeline; this would give us more breathing room in terms of Dataflow job quota and may allow us to run multiple ValidatesRunner suites at the same time.
On Fri, Jun 29, 2018 at 3:49 PM Lukasz Cwik <[email protected]> wrote: > I believe it was increased so we could run batch and streaming tests in > parallel but not enough to run multiple Dataflow VR runs in parallel. > > On Fri, Jun 29, 2018 at 3:43 PM Reuven Lax <[email protected]> wrote: > >> Didn't we recently have the quota increased? >> >> On Fri, Jun 29, 2018, 3:40 PM Lukasz Cwik <[email protected]> wrote: >> >>> Dataflow VR only runs one at a time due to Dataflow job quota capacity. >>> >>> On Fri, Jun 29, 2018 at 3:39 PM Andrew Pilloud <[email protected]> >>> wrote: >>> >>>> Looks like it is only running 1 instance of Dataflow ValidatesRunner at >>>> a time, and the job takes 2+ hours. The queue is a little backed up. >>>> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle_PR/ >>>> >>>> Andrew >>>> >>>> On Fri, Jun 29, 2018 at 3:36 PM Lukasz Cwik <[email protected]> wrote: >>>> >>>>> I think the jobs aren't being scheduled or the status is failing to be >>>>> sent back to Github. >>>>> >>>>> >>>>> >>>>> On Fri, Jun 29, 2018 at 3:22 PM Reuven Lax <[email protected]> wrote: >>>>> >>>>>> Well it's now moving on 2.5 hours, and the tests still haven't even >>>>>> started to execute. Something seems very wrong. >>>>>> >>>>>> On Fri, Jun 29, 2018 at 3:14 PM Andrew Pilloud <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> I don't know what is going on, but I saw the same high latency in >>>>>>> jobs right after filtered pre-commit was turned on for the first time. >>>>>>> (I >>>>>>> don't have anything to suggest it is related, just a memorable time it >>>>>>> happened.) I'm also noticing the Jenkins UI is a bit laggy. >>>>>>> >>>>>>> Andrew >>>>>>> >>>>>>> On Fri, Jun 29, 2018 at 3:09 PM Reuven Lax <[email protected]> wrote: >>>>>>> >>>>>>>> In addition to they delay in the trigger phrases, I'm seeing >>>>>>>> multi-hour (!) delays in running tests. >>>>>>>> >>>>>>>> For example: https://github.com/apache/beam/pull/5545 >>>>>>>> >>>>>>>> I triggered the Dataflow ValidatesRunner tests. The check showed up >>>>>>>> as yellow with no "Details" link, which generally means it had not yet >>>>>>>> been >>>>>>>> scheduled on Jenkins. It stayed like this for two hours, never even >>>>>>>> starting to run. Apparently there were a ton of idle Jenkins executors >>>>>>>> at >>>>>>>> the time, so it's not that all our Jenkins executors were busy. As of >>>>>>>> now, >>>>>>>> it still has not started running. >>>>>>>> >>>>>>>> Does anybody have any idea what's going on here? >>>>>>>> >>>>>>>> Reuven >>>>>>>> >>>>>>> -- ------- Jason Kuster Apache Beam / Google Cloud Dataflow See something? Say something. go/jasonkuster-feedback
