[jira] [Commented] (BEAM-5999) Proto error when running test_pardo_timers of Python PortableValidatesRunner
[
https://issues.apache.org/jira/browse/BEAM-5999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679743#comment-16679743
]
Robert Bradshaw commented on BEAM-5999:
---
Yeah, I'll take a look. I suspect it's due to the thread on rethinking timers
as PCollections.
> Proto error when running test_pardo_timers of Python PortableValidatesRunner
>
>
> Key: BEAM-5999
> URL: https://issues.apache.org/jira/browse/BEAM-5999
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Maximilian Michels
>Priority: Major
> Labels: portability, portability-flink
> Fix For: 2.9.0
>
>
> When I run {{FlinkRunnerTest.test_pardo_timers}}, I get the following error
> during fusing the Proto pipeline. Note, the Java ValidatesPortableRunner
> timer tests run fine. Seems to be an issue with the Python translation.
> {noformat}
> [flink-runner-job-server] ERROR
> org.apache.beam.runners.flink.FlinkJobInvocation - Error during job
> invocation
> test_pardo_timers_1541532770.41_45967eac-8089-4a5f-9d2b-7a4a9d01c7f1.
> java.lang.IllegalArgumentException
> at
> org.apache.beam.model.pipeline.v1.RunnerApi$PTransform.getInputsOrThrow(RunnerApi.java:7418)
> at
> org.apache.beam.runners.core.construction.graph.QueryablePipeline.lambda$getTimers$11(QueryablePipeline.java:385)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.runners.core.construction.graph.QueryablePipeline.getTimers(QueryablePipeline.java:392)
> at
> org.apache.beam.runners.core.construction.graph.GreedyStageFuser.forGrpcPortRead(GreedyStageFuser.java:93)
> at
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fuseSiblings(GreedyPipelineFuser.java:340)
> at
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fusePipeline(GreedyPipelineFuser.java:142)
> at
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.(GreedyPipelineFuser.java:75)
> at
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fuse(GreedyPipelineFuser.java:88)
> at
> org.apache.beam.runners.flink.FlinkJobInvocation.runPipeline(FlinkJobInvocation.java:96)
> at
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
> at
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
> at
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> To reproduce, enable the test in {{FlinkRunnerTest}}, then run the following
> from {{sdks/python}}:
> {noformat}
> python -m apache_beam.runners.portability.flink_runner_test
> --flink_job_server_jar=/path/to//beam/runners/flink/job-server/build/libs/beam-runners-flink_2.11-job-server-2.9.0-SNAPSHOT.jar
> --streaming FlinkRunnerTest.test_pardo_timers
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5999) Proto error when running test_pardo_timers of Python PortableValidatesRunner
[
https://issues.apache.org/jira/browse/BEAM-5999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677231#comment-16677231
]
Maximilian Michels commented on BEAM-5999:
--
Probably a low-hanging fruit but haven't looked into it. Do you mind having a
look [~robertwb]?
> Proto error when running test_pardo_timers of Python PortableValidatesRunner
>
>
> Key: BEAM-5999
> URL: https://issues.apache.org/jira/browse/BEAM-5999
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, testing
>Reporter: Maximilian Michels
>Priority: Major
> Labels: portability, portability-flink
> Fix For: 2.9.0
>
>
> When I run {{FlinkRunnerTest.test_pardo_timers}}, I get the following error
> during fusing the Proto pipeline. Note, the Java ValidatesPortableRunner
> timer tests run fine. Seems to be an issue with the Python translation.
> {noformat}
> [flink-runner-job-server] ERROR
> org.apache.beam.runners.flink.FlinkJobInvocation - Error during job
> invocation
> test_pardo_timers_1541532770.41_45967eac-8089-4a5f-9d2b-7a4a9d01c7f1.
> java.lang.IllegalArgumentException
> at
> org.apache.beam.model.pipeline.v1.RunnerApi$PTransform.getInputsOrThrow(RunnerApi.java:7418)
> at
> org.apache.beam.runners.core.construction.graph.QueryablePipeline.lambda$getTimers$11(QueryablePipeline.java:385)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116)
> at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
> at
> org.apache.beam.runners.core.construction.graph.QueryablePipeline.getTimers(QueryablePipeline.java:392)
> at
> org.apache.beam.runners.core.construction.graph.GreedyStageFuser.forGrpcPortRead(GreedyStageFuser.java:93)
> at
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fuseSiblings(GreedyPipelineFuser.java:340)
> at
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fusePipeline(GreedyPipelineFuser.java:142)
> at
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.(GreedyPipelineFuser.java:75)
> at
> org.apache.beam.runners.core.construction.graph.GreedyPipelineFuser.fuse(GreedyPipelineFuser.java:88)
> at
> org.apache.beam.runners.flink.FlinkJobInvocation.runPipeline(FlinkJobInvocation.java:96)
> at
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111)
> at
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58)
> at
> org.apache.beam.repackaged.beam_runners_flink_2.11.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> To reproduce, enable the test in {{FlinkRunnerTest}}, then run the following
> from {{sdks/python}}:
> {noformat}
> python -m apache_beam.runners.portability.flink_runner_test
> --flink_job_server_jar=/path/to//beam/runners/flink/job-server/build/libs/beam-runners-flink_2.11-job-server-2.9.0-SNAPSHOT.jar
> --streaming FlinkRunnerTest.test_pardo_timers
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
