See
<https://builds.apache.org/job/beam_PostCommit_XVR_Flink/1804/display/redirect?page=changes>
Changes:
[lcwik] [BEAM-5605] Honor the bounded source timestamps timestamp.
[chlarsen] Added ThriftIO to list of supported I/O on website and to change log.
[chamikara] Updates DataflowRunner to support multiple SDK environments.
------------------------------------------
[...truncated 785.31 KB...]
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException:
Error received from SDK harness for instruction 18:
java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With
Execution Info
at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
at
org.apache.beam.runners.core.construction.ParDoTranslation.doFnWithExecutionInformationFromProto(ParDoTranslation.java:652)
at
org.apache.beam.runners.core.construction.ParDoTranslation.getDoFn(ParDoTranslation.java:311)
at
org.apache.beam.fn.harness.FnApiDoFnRunner$Context.<init>(FnApiDoFnRunner.java:180)
at
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:314)
at
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:293)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:214)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:182)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:182)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:182)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:410)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:247)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:467)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:242)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:160)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:144)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.InvalidClassException:
org.apache.beam.sdk.io.Read$BoundedSourceAsSDFWrapperFn; local class
incompatible: stream classdesc serialVersionUID = 2423559023477301562, local
class serialVersionUID = -3068287612878615999
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1940)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1806)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2097)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:71)
... 18 more
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57)
at
org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor$ActiveBundle.close(SdkHarnessClient.java:345)
at
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory$1.close(DefaultJobBundleFactory.java:402)
at
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction.$closeResource(FlinkExecutableStageFunction.java:204)
at
org.apache.beam.runners.flink.translation.functions.FlinkExecutableStageFunction.mapPartition(FlinkExecutableStageFunction.java:204)
at
org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:103)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:504)
at
org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
... 1 more
Caused by: java.lang.RuntimeException: Error received from SDK harness for
instruction 18: java.lang.IllegalArgumentException: unable to deserialize
Custom DoFn With Execution Info
at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
at
org.apache.beam.runners.core.construction.ParDoTranslation.doFnWithExecutionInformationFromProto(ParDoTranslation.java:652)
at
org.apache.beam.runners.core.construction.ParDoTranslation.getDoFn(ParDoTranslation.java:311)
at
org.apache.beam.fn.harness.FnApiDoFnRunner$Context.<init>(FnApiDoFnRunner.java:180)
at
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:314)
at
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:293)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:214)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:182)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:182)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:182)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:410)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:247)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:467)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:242)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:160)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:144)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.InvalidClassException:
org.apache.beam.sdk.io.Read$BoundedSourceAsSDFWrapperFn; local class
incompatible: stream classdesc serialVersionUID = 2423559023477301562, local
class serialVersionUID = -3068287612878615999
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1940)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1806)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2097)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:71)
... 18 more
at
org.apache.beam.runners.fnexecution.control.FnApiControlClient$ResponseStreamObserver.onNext(FnApiControlClient.java:160)
at
org.apache.beam.runners.fnexecution.control.FnApiControlClient$ResponseStreamObserver.onNext(FnApiControlClient.java:140)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:251)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:782)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
root: ERROR: java.lang.RuntimeException: Error received from SDK harness for
instruction 18: java.lang.IllegalArgumentException: unable to deserialize
Custom DoFn With Execution Info
at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:74)
at
org.apache.beam.runners.core.construction.ParDoTranslation.doFnWithExecutionInformationFromProto(ParDoTranslation.java:652)
at
org.apache.beam.runners.core.construction.ParDoTranslation.getDoFn(ParDoTranslation.java:311)
at
org.apache.beam.fn.harness.FnApiDoFnRunner$Context.<init>(FnApiDoFnRunner.java:180)
at
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:314)
at
org.apache.beam.fn.harness.FnApiDoFnRunner$Factory.createRunnerForPTransform(FnApiDoFnRunner.java:293)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:214)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:182)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:182)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createRunnerAndConsumersForPTransformRecursively(ProcessBundleHandler.java:182)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.createBundleProcessor(ProcessBundleHandler.java:410)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.lambda$processBundle$0(ProcessBundleHandler.java:247)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler$BundleProcessorCache.get(ProcessBundleHandler.java:467)
at
org.apache.beam.fn.harness.control.ProcessBundleHandler.processBundle(ProcessBundleHandler.java:242)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.delegateOnInstructionRequestType(BeamFnControlClient.java:160)
at
org.apache.beam.fn.harness.control.BeamFnControlClient.lambda$processInstructionRequests$0(BeamFnControlClient.java:144)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.InvalidClassException:
org.apache.beam.sdk.io.Read$BoundedSourceAsSDFWrapperFn; local class
incompatible: stream classdesc serialVersionUID = 2423559023477301562, local
class serialVersionUID = -3068287612878615999
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1940)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1806)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2097)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
at
org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:71)
... 18 more
apache_beam.runners.portability.portable_runner: INFO: Job state changed to
FAILED
--------------------- >> end captured logging << ---------------------
----------------------------------------------------------------------
XML: nosetests-xlangValidateRunner.xml
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 10 tests in 385.428s
FAILED (errors=2)
> Task
> :runners:flink:1.9:job-server:validatesCrossLanguageRunnerPythonUsingJava
> FAILED
> Task
> :runners:flink:1.9:job-server:validatesCrossLanguageRunnerPythonUsingPython
>>> RUNNING integration tests with pipeline options: --runner=PortableRunner
>>> --environment_cache_millis=10000 --job_endpoint=localhost:18091
>>> test options: --attr=UsesCrossLanguageTransforms
setup.py:252: UserWarning: You are using Apache Beam with Python 2. New
releases of Apache Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
<https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src/build/gradleenv/1922375555/local/lib/python2.7/site-packages/setuptools/dist.py>:476:
UserWarning: Normalizing '2.20.0.dev' to '2.20.0.dev0'
normalized_version,
running nosetests
running egg_info
Skipping proto regeneration: all files up to date
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src/sdks/python/apache_beam/__init__.py>:82:
UserWarning: You are using Apache Beam with Python 2. New releases of Apache
Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
<https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src/sdks/python/apache_beam/transforms/trigger_test.py>:541:
YAMLLoadWarning: calling yaml.load_all() without Loader=... is deprecated, as
the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full
details.
for spec in yaml.load_all(open(transcript_filename)):
test_generate_sequence
(apache_beam.io.external.generate_sequence_test.XlangGenerateSequenceTest) ...
ok
test_write (apache_beam.io.external.xlang_parquetio_test.XlangParquetIOTest)
... ok
test_cogroup_by_key
(apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest) ...
ok
test_combine_globally
(apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest) ...
ok
test_combine_per_key
(apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest) ...
ok
test_flatten
(apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest) ...
ok
test_group_by_key
(apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest) ...
ok
test_multi_input_output_with_sideinput
(apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest) ...
ok
test_partition
(apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest) ...
ok
test_prefix
(apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest) ...
ok
----------------------------------------------------------------------
XML: nosetests-xlangValidateRunner.xml
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 10 tests in 165.410s
OK
> Task :runners:flink:1.9:job-server:validatesCrossLanguageRunnerCleanup
Stopping expansion service pid: 2606.
Stopping expansion service pid: 2607.
> Task :runners:flink:1.9:job-server:flinkJobServerCleanup
Stopping job server pid: 27161.
FAILURE: Build completed with 3 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':runners:flink:1.9:job-server:validatesCrossLanguageRunnerJavaUsingJava'.
> There were failing tests. See the report at:
> file://<https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src/runners/flink/1.9/job-server/build/reports/tests/validatesCrossLanguageRunnerJavaUsingJava/index.html>
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':runners:flink:1.9:job-server:validatesCrossLanguageRunnerJavaUsingPython'.
> There were failing tests. See the report at:
> file://<https://builds.apache.org/job/beam_PostCommit_XVR_Flink/ws/src/runners/flink/1.9/job-server/build/reports/tests/validatesCrossLanguageRunnerJavaUsingPython/index.html>
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task
':runners:flink:1.9:job-server:validatesCrossLanguageRunnerPythonUsingJava'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 36m 7s
113 actionable tasks: 96 executed, 15 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/rr7crkg4zmeby
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
