Dataflow VR only runs one at a time due to Dataflow job quota capacity. On Fri, Jun 29, 2018 at 3:39 PM Andrew Pilloud <[email protected]> wrote:
> Looks like it is only running 1 instance of Dataflow ValidatesRunner at a > time, and the job takes 2+ hours. The queue is a little backed up. > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle_PR/ > > Andrew > > On Fri, Jun 29, 2018 at 3:36 PM Lukasz Cwik <[email protected]> wrote: > >> I think the jobs aren't being scheduled or the status is failing to be >> sent back to Github. >> >> >> >> On Fri, Jun 29, 2018 at 3:22 PM Reuven Lax <[email protected]> wrote: >> >>> Well it's now moving on 2.5 hours, and the tests still haven't even >>> started to execute. Something seems very wrong. >>> >>> On Fri, Jun 29, 2018 at 3:14 PM Andrew Pilloud <[email protected]> >>> wrote: >>> >>>> I don't know what is going on, but I saw the same high latency in jobs >>>> right after filtered pre-commit was turned on for the first time. (I don't >>>> have anything to suggest it is related, just a memorable time it happened.) >>>> I'm also noticing the Jenkins UI is a bit laggy. >>>> >>>> Andrew >>>> >>>> On Fri, Jun 29, 2018 at 3:09 PM Reuven Lax <[email protected]> wrote: >>>> >>>>> In addition to they delay in the trigger phrases, I'm seeing >>>>> multi-hour (!) delays in running tests. >>>>> >>>>> For example: https://github.com/apache/beam/pull/5545 >>>>> >>>>> I triggered the Dataflow ValidatesRunner tests. The check showed up as >>>>> yellow with no "Details" link, which generally means it had not yet been >>>>> scheduled on Jenkins. It stayed like this for two hours, never even >>>>> starting to run. Apparently there were a ton of idle Jenkins executors at >>>>> the time, so it's not that all our Jenkins executors were busy. As of now, >>>>> it still has not started running. >>>>> >>>>> Does anybody have any idea what's going on here? >>>>> >>>>> Reuven >>>>> >>>>
