My run spent 5 hours and 40 minutes in the queue. I could easily see enough people trying to run these tests to cause days of waiting time.
On Fri, Jun 29, 2018 at 3:57 PM Jason Kuster <[email protected]> wrote: > There's been a fairly long-standing discussion regarding merging > ValidatesRunner tests in the same class together into one pipeline; this > would give us more breathing room in terms of Dataflow job quota and may > allow us to run multiple ValidatesRunner suites at the same time. > > On Fri, Jun 29, 2018 at 3:49 PM Lukasz Cwik <[email protected]> wrote: > >> I believe it was increased so we could run batch and streaming tests in >> parallel but not enough to run multiple Dataflow VR runs in parallel. >> >> On Fri, Jun 29, 2018 at 3:43 PM Reuven Lax <[email protected]> wrote: >> >>> Didn't we recently have the quota increased? >>> >>> On Fri, Jun 29, 2018, 3:40 PM Lukasz Cwik <[email protected]> wrote: >>> >>>> Dataflow VR only runs one at a time due to Dataflow job quota capacity. >>>> >>>> On Fri, Jun 29, 2018 at 3:39 PM Andrew Pilloud <[email protected]> >>>> wrote: >>>> >>>>> Looks like it is only running 1 instance of Dataflow ValidatesRunner at >>>>> a time, and the job takes 2+ hours. The queue is a little backed up. >>>>> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle_PR/ >>>>> >>>>> Andrew >>>>> >>>>> On Fri, Jun 29, 2018 at 3:36 PM Lukasz Cwik <[email protected]> wrote: >>>>> >>>>>> I think the jobs aren't being scheduled or the status is failing to >>>>>> be sent back to Github. >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jun 29, 2018 at 3:22 PM Reuven Lax <[email protected]> wrote: >>>>>> >>>>>>> Well it's now moving on 2.5 hours, and the tests still haven't even >>>>>>> started to execute. Something seems very wrong. >>>>>>> >>>>>>> On Fri, Jun 29, 2018 at 3:14 PM Andrew Pilloud <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I don't know what is going on, but I saw the same high latency in >>>>>>>> jobs right after filtered pre-commit was turned on for the first time. >>>>>>>> (I >>>>>>>> don't have anything to suggest it is related, just a memorable time it >>>>>>>> happened.) I'm also noticing the Jenkins UI is a bit laggy. >>>>>>>> >>>>>>>> Andrew >>>>>>>> >>>>>>>> On Fri, Jun 29, 2018 at 3:09 PM Reuven Lax <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> In addition to they delay in the trigger phrases, I'm seeing >>>>>>>>> multi-hour (!) delays in running tests. >>>>>>>>> >>>>>>>>> For example: https://github.com/apache/beam/pull/5545 >>>>>>>>> >>>>>>>>> I triggered the Dataflow ValidatesRunner tests. The check showed >>>>>>>>> up as yellow with no "Details" link, which generally means it had not >>>>>>>>> yet >>>>>>>>> been scheduled on Jenkins. It stayed like this for two hours, never >>>>>>>>> even >>>>>>>>> starting to run. Apparently there were a ton of idle Jenkins >>>>>>>>> executors at >>>>>>>>> the time, so it's not that all our Jenkins executors were busy. As of >>>>>>>>> now, >>>>>>>>> it still has not started running. >>>>>>>>> >>>>>>>>> Does anybody have any idea what's going on here? >>>>>>>>> >>>>>>>>> Reuven >>>>>>>>> >>>>>>>> > > -- > ------- > Jason Kuster > Apache Beam / Google Cloud Dataflow > > See something? Say something. go/jasonkuster-feedback > <https://goto.google.com/jasonkuster-feedback> >
