See <https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/2421/display/redirect?page=changes>
Changes: [yoshiki.obata] [BEAM-9980] add groovy functions for python versions [yoshiki.obata] [BEAM-9980] update dataflow test-suites to switch python versions using [yoshiki.obata] [BEAM-9980] version switchable dataflow tasks to be invoked [yoshiki.obata] [BEAM-9980] :sdks:python:test-suites:dataflow included in [Luke Cwik] [BEAM-10670] Make key coder deterministic by using upstream PCollection [Udi Meiri] [BEAM-10697] Remove testPy2Cython from precommit [noreply] [BEAM-9891] Generate query execution summary table after finishing jobs ------------------------------------------ [...truncated 3.04 MB...] ERROR:root:java.lang.RuntimeException: Error received from SDK harness for instruction 4: Traceback (most recent call last): File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,> line 256, in _execute response = task() File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,> line 313, in <lambda> lambda: self.create_worker().do_instruction(request), request) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,> line 483, in do_instruction getattr(request, request_type), request.instruction_id) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/worker/sdk_worker.py";,> line 518, in process_bundle bundle_processor.process_bundle(instruction_id)) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/worker/bundle_processor.py";,> line 983, in process_bundle element.data) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/worker/bundle_processor.py";,> line 219, in process_encoded self.output(decoded_value) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/worker/operations.py";,> line 332, in output cython.cast(Receiver, self.receivers[output_index]).receive(windowed_value) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/worker/operations.py";,> line 195, in receive self.consumer.process(windowed_value) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/worker/operations.py";,> line 779, in process o) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/common.py";,> line 1224, in process_with_sized_restriction watermark_estimator_state=estimator_state) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/common.py";,> line 723, in invoke_process windowed_value, additional_args, additional_kwargs) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/common.py";,> line 872, in _invoke_process_per_window self.threadsafe_restriction_tracker.check_done() File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/sdf_utils.py";,> line 115, in check_done return self._restriction_tracker.check_done() File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/io/restriction_trackers.py";,> line 106, in check_done self._range.stop)) ValueError: OffsetRestrictionTracker is not done since work in range [0, 6) has not been claimed. INFO:apache_beam.runners.portability.portable_runner:Job state changed to FAILED .sssssINFO:apache_beam.runners.worker.worker_pool_main:Listening for workers at localhost:36291 WARNING:root:Make sure that locally built Python SDK docker image has Python 3.5 interpreter. INFO:root:Using Python SDK docker image: apache/beam_python3.5_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com INFO:apache_beam.runners.portability.portable_runner:Environment "LOOPBACK" has started a component necessary for the execution. Be sure to run the pipeline using with Pipeline() as p: p.apply(..) This ensures that the pipeline finishes before this program exits. INFO:apache_beam.runners.portability.portable_runner:Job state changed to STOPPED INFO:apache_beam.runners.portability.portable_runner:Job state changed to STARTING INFO:apache_beam.runners.portability.portable_runner:Job state changed to RUNNING [flink-runner-job-invoker] WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set. [flink-runner-job-invoker] WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (fallback keys: [{key=jobmanager.web.log.path, isDeprecated=true}])'. [[5]{Create, Map(<lambda at fn_runner_test.py:490>), WindowInto(WindowIntoFn), Map(<lambda at fn_runner_test.py:493>)} (1/2)] WARN org.apache.flink.metrics.MetricGroup - The operator name [5]{Create, Map(<lambda at fn_runner_test.py:490>), WindowInto(WindowIntoFn), Map(<lambda at fn_runner_test.py:493>)} exceeded the 80 characters length limit and was truncated. [[5]{Create, Map(<lambda at fn_runner_test.py:490>), WindowInto(WindowIntoFn), Map(<lambda at fn_runner_test.py:493>)} (2/2)] WARN org.apache.flink.metrics.MetricGroup - The operator name [5]{Create, Map(<lambda at fn_runner_test.py:490>), WindowInto(WindowIntoFn), Map(<lambda at fn_runner_test.py:493>)} exceeded the 80 characters length limit and was truncated. INFO:apache_beam.runners.worker.statecache:Creating state cache with size 0 INFO:apache_beam.runners.worker.sdk_worker:Creating insecure control channel for localhost:39205. INFO:apache_beam.runners.worker.sdk_worker:Control channel established. INFO:apache_beam.runners.worker.sdk_worker:Initializing SDKHarness with unbounded number of workers. INFO:apache_beam.runners.worker.sdk_worker:Creating insecure state channel for localhost:37771. INFO:apache_beam.runners.worker.sdk_worker:State channel established. INFO:apache_beam.runners.worker.data_plane:Creating client data channel for localhost:37823 INFO:apache_beam.runners.worker.sdk_worker:No more requests from control plane INFO:apache_beam.runners.worker.sdk_worker:SDK Harness waiting for in-flight requests to complete INFO:apache_beam.runners.worker.data_plane:Closing all cached grpc data channels. INFO:apache_beam.runners.worker.sdk_worker:Closing all cached gRPC state handlers. [assert_that/Group/GroupByKey -> [3]assert_that/{Group, Unkey, Match} (1/2)] WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer - Hanged up for unknown endpoint. INFO:apache_beam.runners.worker.sdk_worker:Done consuming work. Aug 18, 2020 12:27:51 AM org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.SerializingExecutor run SEVERE: Exception while executing runnable org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@4668d3f java.lang.IllegalStateException: call already closed at org.apache.beam.vendor.grpc.v1p26p0.com.google.common.base.Preconditions.checkState(Preconditions.java:511) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerCallImpl.closeInternal(ServerCallImpl.java:209) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerCallImpl.close(ServerCallImpl.java:202) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:371) at org.apache.beam.runners.fnexecution.state.GrpcStateService$Inbound.onCompleted(GrpcStateService.java:150) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onHalfClose(ServerCalls.java:262) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:331) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:817) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.beam.vendor.grpc.v1p26p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) INFO:apache_beam.runners.portability.portable_runner:Job state changed to DONE .INFO:apache_beam.runners.worker.worker_pool_main:Listening for workers at localhost:43717 WARNING:root:Make sure that locally built Python SDK docker image has Python 3.5 interpreter. INFO:root:Using Python SDK docker image: apache/beam_python3.5_sdk:2.25.0.dev. If the image is not available at local, we will try to pull from hub.docker.com INFO:apache_beam.runners.portability.portable_runner:Environment "LOOPBACK" has started a component necessary for the execution. Be sure to run the pipeline using with Pipeline() as p: p.apply(..) This ensures that the pipeline finishes before this program exits. INFO:apache_beam.runners.portability.portable_runner:Job state changed to STOPPED INFO:apache_beam.runners.portability.portable_runner:Job state changed to STARTING INFO:apache_beam.runners.portability.portable_runner:Job state changed to RUNNING [flink-runner-job-invoker] WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set. [flink-runner-job-invoker] WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'Key: 'web.log.path' , default: null (fallback keys: [{key=jobmanager.web.log.path, isDeprecated=true}])'. INFO:apache_beam.runners.worker.statecache:Creating state cache with size 0 INFO:apache_beam.runners.worker.sdk_worker:Creating insecure control channel for localhost:35753. INFO:apache_beam.runners.worker.sdk_worker:Control channel established. INFO:apache_beam.runners.worker.sdk_worker:Initializing SDKHarness with unbounded number of workers. INFO:apache_beam.runners.worker.sdk_worker:Creating insecure state channel for localhost:41855. INFO:apache_beam.runners.worker.sdk_worker:State channel established. INFO:apache_beam.runners.worker.data_plane:Creating client data channel for localhost:45535 INFO:apache_beam.runners.worker.sdk_worker:No more requests from control plane INFO:apache_beam.runners.worker.sdk_worker:SDK Harness waiting for in-flight requests to complete INFO:apache_beam.runners.worker.data_plane:Closing all cached grpc data channels. INFO:apache_beam.runners.worker.sdk_worker:Closing all cached gRPC state handlers. [assert_that/Group/GroupByKey -> [3]assert_that/{Group, Unkey, Match} (1/2)] WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer - Hanged up for unknown endpoint. [grpc-default-executor-3] WARN org.apache.beam.sdk.fn.data.BeamFnDataGrpcMultiplexer - Hanged up for unknown endpoint. INFO:apache_beam.runners.worker.sdk_worker:Done consuming work. INFO:apache_beam.runners.portability.portable_runner:Job state changed to DONE .INFO:__main__:removing conf dir: /tmp/flinktest-conf816waxyi ====================================================================== ERROR: test_expand_kafka_read (__main__.FlinkRunnerTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py";,> line 236, in test_expand_kafka_read expansion_service=self.get_expansion_service())) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/pipeline.py";,> line 556, in __exit__ self.result.wait_until_finish() File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,> line 544, in wait_until_finish self._observe_state(message_thread) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,> line 553, in _observe_state for state_response in self._state_stream: File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/build/gradleenv/2022703439/lib/python3.5/site-packages/grpc/_channel.py";,> line 416, in __next__ return self._next() File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/build/gradleenv/2022703439/lib/python3.5/site-packages/grpc/_channel.py";,> line 697, in _next _common.wait(self._state.condition.wait, _response_ready) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/build/gradleenv/2022703439/lib/python3.5/site-packages/grpc/_common.py";,> line 138, in wait _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/build/gradleenv/2022703439/lib/python3.5/site-packages/grpc/_common.py";,> line 103, in _wait_once wait_fn(timeout=timeout) File "/usr/lib/python3.5/threading.py", line 297, in wait gotit = waiter.acquire(True, timeout) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner_test.py";,> line 82, in handler raise BaseException(msg) BaseException: Timed out after 60 seconds. ====================================================================== ERROR: test_external_transform (__main__.FlinkRunnerTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/portability/flink_runner_test.py";,> line 214, in test_external_transform assert_that(res, equal_to([i for i in range(1, 10)])) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/pipeline.py";,> line 556, in __exit__ self.result.wait_until_finish() File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,> line 544, in wait_until_finish self._observe_state(message_thread) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,> line 553, in _observe_state for state_response in self._state_stream: File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/build/gradleenv/2022703439/lib/python3.5/site-packages/grpc/_channel.py";,> line 416, in __next__ return self._next() File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/build/gradleenv/2022703439/lib/python3.5/site-packages/grpc/_channel.py";,> line 697, in _next _common.wait(self._state.condition.wait, _response_ready) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/build/gradleenv/2022703439/lib/python3.5/site-packages/grpc/_common.py";,> line 138, in wait _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/build/gradleenv/2022703439/lib/python3.5/site-packages/grpc/_common.py";,> line 103, in _wait_once wait_fn(timeout=timeout) File "/usr/lib/python3.5/threading.py", line 297, in wait gotit = waiter.acquire(True, timeout) File "<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner_test.py";,> line 82, in handler raise BaseException(msg) BaseException: Timed out after 60 seconds. ---------------------------------------------------------------------- Ran 100 tests in 421.046s FAILED (errors=2, skipped=26) ==================== Timed out after 60 seconds. ==================== # Thread: <Thread(wait_until_finish_read, started daemon 140082791028480)> # Thread: <Thread(Thread-142, started daemon 140082279335680)> # Thread: <Thread(Thread-135, started daemon 140081767642880)> # Thread: <_Worker(Thread-10, started daemon 140083322816256)> # Thread: <_Worker(Thread-18, started daemon 140082824599296)> # Thread: <_MainThread(MainThread, started 140084140951296)> # Thread: <_Worker(Thread-63, started daemon 140081759250176)> ==================== Timed out after 60 seconds. ==================== # Thread: <Thread(read_grpc_client_inputs, started daemon 140081742464768)> # Thread: <Thread(Thread-176, started daemon 140081750857472)> # Thread: <_MainThread(MainThread, started 140084140951296)> # Thread: <Thread(run_worker_8-1, started daemon 140082807813888)> # Thread: <Thread(wait_until_finish_read, started daemon 140082791028480)> # Thread: <PeriodicThread(Thread-170, started daemon 140082816206592)> # Thread: <Thread(wait_until_finish_read, started daemon 140083914606336)> # Thread: <Thread(Thread-142, started daemon 140082279335680)> # Thread: <Thread(Thread-174, started daemon 140082287728384)> # Thread: <Thread(read_state, started daemon 140082270942976)> # Thread: <Thread(Thread-165, started daemon 140082832992000)> # Thread: <Thread(Thread-175, started daemon 140082262550272)> # Thread: <_Worker(Thread-63, started daemon 140081759250176)> # Thread: <Thread(Thread-172, started daemon 140082304513792)> # Thread: <_Worker(Thread-18, started daemon 140082824599296)> # Thread: <Thread(Thread-171, started daemon 140082799421184)> # Thread: <Thread(Thread-158, started daemon 140082254157568)> # Thread: <_Worker(Thread-10, started daemon 140083322816256)> # Thread: <Thread(Thread-173, started daemon 140082296121088)> > Task > :sdks:python:test-suites:portable:py35:flinkCompatibilityMatrixStreamingPreOptimizeLOOPBACK > FAILED FAILURE: Build failed with an exception. * Where: Script '<https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/ws/src/sdks/python/test-suites/portable/common.gradle'> line: 57 * What went wrong: Execution failed for task ':sdks:python:test-suites:portable:py35:flinkCompatibilityMatrixStreamingPreOptimizeLOOPBACK'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 26m 17s 108 actionable tasks: 90 executed, 17 from cache, 1 up-to-date Publishing build scan... https://gradle.com/s/2ztoq4l7i2zki Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
