[
https://issues.apache.org/jira/browse/BEAM-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yichi Zhang updated BEAM-11793:
-------------------------------
Description:
When using Stateful DoFn in batch pipeline with dataflow runner v2, workflow
will fail with
{code:java}
Workflow failed. Causes:
S08:Reshuffle/ReshufflePerKey/GroupByKey/Read+Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Reshuffle/RemoveRandomKeys+Do/Reshard/Write
failed., The job failed because a work item has failed 4 times. Look in
previous log entries for the cause of each one of the 4 failures. For more
information, see https://cloud.google.com/dataflow/docs/guides/common-errors.
The work item was attempted on these workers:
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 20).,
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 20).,
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 20).,
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 20).
{code}
was:
When using Stateful DoFn in batch pipeline with dataflow runner v2, workflow
will fail with
{code:java}
Workflow failed. Causes:
S08:Reshuffle/ReshufflePerKey/GroupByKey/Read+Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Reshuffle/RemoveRandomKeys+Do/Reshard/Write
failed., The job failed because a work item has failed 4 times. Look in
previous log entries for the cause of each one of the 4 failures. For more
information, see https://cloud.google.com/dataflow/docs/guides/common-errors.
The work item was attempted on these workers:
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 3).,
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 3).,
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 3).,
df2-combinefnlifecycletes-02101100-mv27-local-0
Root cause: Shard number -598290054 is not in range [0, 3).
{code}
> Batch stateful DoFn does not work with runner v2 portable job submission
> ------------------------------------------------------------------------
>
> Key: BEAM-11793
> URL: https://issues.apache.org/jira/browse/BEAM-11793
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Reporter: Yichi Zhang
> Priority: P2
>
> When using Stateful DoFn in batch pipeline with dataflow runner v2, workflow
> will fail with
> {code:java}
> Workflow failed. Causes:
> S08:Reshuffle/ReshufflePerKey/GroupByKey/Read+Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+Reshuffle/RemoveRandomKeys+Do/Reshard/Write
> failed., The job failed because a work item has failed 4 times. Look in
> previous log entries for the cause of each one of the 4 failures. For more
> information, see https://cloud.google.com/dataflow/docs/guides/common-errors.
> The work item was attempted on these workers:
> df2-combinefnlifecycletes-02101100-mv27-local-0
> Root cause: Shard number -598290054 is not in range [0, 20).,
> df2-combinefnlifecycletes-02101100-mv27-local-0
> Root cause: Shard number -598290054 is not in range [0, 20).,
> df2-combinefnlifecycletes-02101100-mv27-local-0
> Root cause: Shard number -598290054 is not in range [0, 20).,
> df2-combinefnlifecycletes-02101100-mv27-local-0
> Root cause: Shard number -598290054 is not in range [0, 20).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
