[
https://issues.apache.org/jira/browse/BEAM-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037346#comment-17037346
]
Liu Wang edited comment on BEAM-9003 at 2/14/20 10:55 PM:
----------------------------------------------------------
Thank you for pointing that out!
I ran the test with 2.20.0 version.
Command:
python setup.py nosetests -test-pipeline-options="runner=TestDataflowRunner
--dataflow_worker_jar='./../../runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.20.0-SNAPSHOT.jar'
--project=google.com:clouddfe --temp_location=gs://clouddfe-test/staging$USER
--output=gs://world-readable-mkcq69tkcu/$USER/result.txt
--sdk_location=./build/apache-beam-2.20.0.dev0.tar.gz --num_workers=1
--sleep_secs=20 --streaming "
--tests=apache_beam.transforms.util_test.ReshuffleTest --attr=ValidatesRunner
--nocapture
Link:
[https://pantheon.corp.google.com/dataflow/jobs/us-central1/2020-02-14_14_38_22-18393797308579978754?project=google.com:clouddfe]
Error message:
Error message from worker: java.lang.IllegalStateException: TimestampCombiner
moved element from 294247-01-10T04:00:54.775Z (TIMESTAMP_MAX_VALUE) to earlier
time 294247-01-09T04:00:54.775Z (end of global window) for window
org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56
org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117)
org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154)
org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98)
org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605)
org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349)
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94)
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134)
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748) java.lang.IllegalStateException:
TimestampCombiner moved element from 294247-01-10T04:00:54.775Z
(TIMESTAMP_MAX_VALUE) to earlier time 294247-01-09T04:00:54.775Z (end of global
window) for window
org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56
org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117)
org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154)
org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98)
org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605)
org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349)
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94)
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134)
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
was (Author: liuwang_jira):
Thank you for pointing that out!
I ran the test with 2.20.0 version.
Command:
python setup.py nosetests --test-pipeline-options="--runner=TestDataflowRunner
--dataflow_worker_jar='./../../runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.20.0-SNAPSHOT.jar'
--project=google.com:clouddfe --temp_location=gs://clouddfe-test/staging-$USER
--output=gs://world-readable-mkcq69tkcu/$USER/result.txt
--sdk_location=./build/apache-beam-2.20.0.dev0.tar.gz --num_workers=1
--sleep_secs=20 --streaming "
--tests=apache_beam.transforms.util_test.ReshuffleTest --attr=ValidatesRunner
--nocapture
Link:
[https://pantheon.corp.google.com/dataflow/jobs/us-central1/2020-02-14_14_38_22-18393797308579978754?project=google.com:clouddfe]
Error message:
Error message from worker: java.lang.IllegalStateException: TimestampCombiner
moved element from 294247-01-10T04:00:54.775Z (TIMESTAMP_MAX_VALUE) to earlier
time 294247-01-09T04:00:54.775Z (end of global window) for window
org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56
org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117)
org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154)
org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98)
org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605)
org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349)
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94)
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134)
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748) java.lang.IllegalStateException:
TimestampCombiner moved element from 294247-01-10T04:00:54.775Z
(TIMESTAMP_MAX_VALUE) to earlier time 294247-01-09T04:00:54.775Z (end of global
window) for window
org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56
org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117)
org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154)
org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98)
org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605)
org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349)
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94)
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80)
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134)
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153)
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:748)
> test_reshuffle_preserves_timestamps
> (apache_beam.transforms.util_test.ReshuffleTest) does not work in Streaming
> VR suite on Dataflow
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-9003
> URL: https://issues.apache.org/jira/browse/BEAM-9003
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Liu Wang
> Priority: Major
>
> Per investigation in https://issues.apache.org/jira/browse/BEAM-8877, the
> test times out and was recently added to VR test suite.
> [~liumomo315], I will sickbay this test for streaming, could you please help
> triage the failure?
> Thank you!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
