[
https://issues.apache.org/jira/browse/BEAM-6528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Oliveira updated BEAM-6528:
----------------------------------
Description:
* [Jenkins
Job|https://builds.apache.org/job/beam_PostCommit_Python_Verify/7219/]
* [Gradle Build Scan|https://scans.gradle.com/s/cdjxkpm7dgtxm]
Initial investigation:
It seems this test has failed four times just today with flakes in various
integration tests. I've linked to the first instance of this failure above.
In the Dataflow dashboard I see this error:
{noformat}
Startup of the worker pool in zone us-central1-f failed to bring up any of the
desired 1 workers. ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: The zone
'projects/apache-beam-testing/zones/us-central1-f' does not have enough
resources available to fulfill the request. '(resource
type:pd-standard)'.{noformat}
Possibly some quota issue?
Edit:
I'm updating this bug because it seems like this isn't restricted to
beam_PostCommit_Python_Verify after all. I've now seen this happening in
various java postcommits:
[beam_PostCommit_Java|https://builds.apache.org/job/beam_PostCommit_Java/2466/]
[beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow|https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow/617/]
[beam_PostCommit_Java_ValidatesRunner_Dataflow|https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/1959/]
In each case the error looks something like this:
{noformat}
java.lang.RuntimeException: Startup of the worker pool in zone us-central1-f
failed to bring up any of the desired 1 workers.
ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: The zone
'projects/apache-beam-testing/zones/us-central1-f' does not have enough
resources available to fulfill the request. '(resource
type:pd-standard)'.Workflow failed.{noformat}
was:
* [Jenkins
Job|https://builds.apache.org/job/beam_PostCommit_Python_Verify/7219/]
* [Gradle Build Scan|https://scans.gradle.com/s/cdjxkpm7dgtxm]
Initial investigation:
It seems this test has failed four times just today with flakes in various
integration tests. I've linked to the first instance of this failure above.
In the Dataflow dashboard I see this error:
{noformat}
Startup of the worker pool in zone us-central1-f failed to bring up any of the
desired 1 workers. ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: The zone
'projects/apache-beam-testing/zones/us-central1-f' does not have enough
resources available to fulfill the request. '(resource
type:pd-standard)'.{noformat}
Possibly some quota issue?
Summary: Dataflow jobs failing in multiple PostCommits, including Java
PostCommit. (was: [beam_PostCommit_Python_Verify] [postCommitIT] Various
integration tests are very flaky.)
> Dataflow jobs failing in multiple PostCommits, including Java PostCommit.
> -------------------------------------------------------------------------
>
> Key: BEAM-6528
> URL: https://issues.apache.org/jira/browse/BEAM-6528
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, test-failures
> Reporter: Daniel Oliveira
> Assignee: Alan Myrvold
> Priority: Major
> Labels: currently-failing, flake
>
> * [Jenkins
> Job|https://builds.apache.org/job/beam_PostCommit_Python_Verify/7219/]
> * [Gradle Build Scan|https://scans.gradle.com/s/cdjxkpm7dgtxm]
> Initial investigation:
> It seems this test has failed four times just today with flakes in various
> integration tests. I've linked to the first instance of this failure above.
> In the Dataflow dashboard I see this error:
> {noformat}
> Startup of the worker pool in zone us-central1-f failed to bring up any of
> the desired 1 workers. ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: The zone
> 'projects/apache-beam-testing/zones/us-central1-f' does not have enough
> resources available to fulfill the request. '(resource
> type:pd-standard)'.{noformat}
> Possibly some quota issue?
>
> Edit:
> I'm updating this bug because it seems like this isn't restricted to
> beam_PostCommit_Python_Verify after all. I've now seen this happening in
> various java postcommits:
> [beam_PostCommit_Java|https://builds.apache.org/job/beam_PostCommit_Java/2466/]
>
> [beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow|https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow/617/]
>
> [beam_PostCommit_Java_ValidatesRunner_Dataflow|https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/1959/]
> In each case the error looks something like this:
> {noformat}
> java.lang.RuntimeException: Startup of the worker pool in zone us-central1-f
> failed to bring up any of the desired 1 workers.
> ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS: The zone
> 'projects/apache-beam-testing/zones/us-central1-f' does not have enough
> resources available to fulfill the request. '(resource
> type:pd-standard)'.Workflow failed.{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
