Didn't we recently have the quota increased? On Fri, Jun 29, 2018, 3:40 PM Lukasz Cwik <[email protected]> wrote:
> Dataflow VR only runs one at a time due to Dataflow job quota capacity. > > On Fri, Jun 29, 2018 at 3:39 PM Andrew Pilloud <[email protected]> > wrote: > >> Looks like it is only running 1 instance of Dataflow ValidatesRunner at >> a time, and the job takes 2+ hours. The queue is a little backed up. >> https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle_PR/ >> >> Andrew >> >> On Fri, Jun 29, 2018 at 3:36 PM Lukasz Cwik <[email protected]> wrote: >> >>> I think the jobs aren't being scheduled or the status is failing to be >>> sent back to Github. >>> >>> >>> >>> On Fri, Jun 29, 2018 at 3:22 PM Reuven Lax <[email protected]> wrote: >>> >>>> Well it's now moving on 2.5 hours, and the tests still haven't even >>>> started to execute. Something seems very wrong. >>>> >>>> On Fri, Jun 29, 2018 at 3:14 PM Andrew Pilloud <[email protected]> >>>> wrote: >>>> >>>>> I don't know what is going on, but I saw the same high latency in jobs >>>>> right after filtered pre-commit was turned on for the first time. (I don't >>>>> have anything to suggest it is related, just a memorable time it >>>>> happened.) >>>>> I'm also noticing the Jenkins UI is a bit laggy. >>>>> >>>>> Andrew >>>>> >>>>> On Fri, Jun 29, 2018 at 3:09 PM Reuven Lax <[email protected]> wrote: >>>>> >>>>>> In addition to they delay in the trigger phrases, I'm seeing >>>>>> multi-hour (!) delays in running tests. >>>>>> >>>>>> For example: https://github.com/apache/beam/pull/5545 >>>>>> >>>>>> I triggered the Dataflow ValidatesRunner tests. The check showed up >>>>>> as yellow with no "Details" link, which generally means it had not yet >>>>>> been >>>>>> scheduled on Jenkins. It stayed like this for two hours, never even >>>>>> starting to run. Apparently there were a ton of idle Jenkins executors at >>>>>> the time, so it's not that all our Jenkins executors were busy. As of >>>>>> now, >>>>>> it still has not started running. >>>>>> >>>>>> Does anybody have any idea what's going on here? >>>>>> >>>>>> Reuven >>>>>> >>>>>
