See
<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/1508/display/redirect?page=changes>
Changes:
[yifanzou] [BEAM-8621] Fix dependency checking job.
------------------------------------------
[...truncated 1.58 MB...]
19/11/13 02:10:39 INFO ContextCleaner: Cleaned shuffle 68
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2541
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2589
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2758
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2471
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2527
19/11/13 02:10:39 INFO BlockManagerInfo: Removed broadcast_110_piece0 on
localhost:37257 in memory (size: 20.0 KB, free: 13.4 GB)
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2636
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2803
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2625
19/11/13 02:10:39 INFO ContextCleaner: Cleaned shuffle 62
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2828
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2832
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2452
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2683
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2699
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2722
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2812
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2455
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2582
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2709
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2563
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2503
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2659
19/11/13 02:10:39 INFO BlockManagerInfo: Removed broadcast_120_piece0 on
localhost:37257 in memory (size: 11.9 KB, free: 13.4 GB)
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2701
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2754
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2571
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2522
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2830
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2756
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2510
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2552
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2696
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2759
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2600
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2491
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2532
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2736
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2789
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2823
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2688
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2492
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2788
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2559
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2524
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2565
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2814
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2692
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2461
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2501
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2721
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2666
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2551
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2626
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2597
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2498
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2608
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2486
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2599
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2705
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2825
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2457
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2521
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2729
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2578
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2780
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2630
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2693
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2539
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2622
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2714
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2737
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2519
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2525
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2716
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2797
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2508
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2537
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2773
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2691
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2535
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2648
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2764
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2556
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2602
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2528
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2745
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2543
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2581
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2546
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2741
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2766
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2512
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2703
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2495
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2609
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2702
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2767
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2695
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2643
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2811
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2700
19/11/13 02:10:39 INFO BlockManagerInfo: Removed broadcast_108_piece0 on
localhost:37257 in memory (size: 10.3 KB, free: 13.4 GB)
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2765
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2744
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2735
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2651
19/11/13 02:10:39 INFO ContextCleaner: Cleaned accumulator 2545
19/11/13 02:10:39 INFO ContextCleaner: Cleaned shuffle 67
19/11/13 02:10:40 INFO GrpcLoggingService: Beam Fn Logging client connected.
19/11/13 02:10:40 INFO main: Logging handler created.
19/11/13 02:10:40 INFO start: Status HTTP server running at localhost:39619
19/11/13 02:10:40 INFO main: semi_persistent_directory: /tmp
19/11/13 02:10:40 WARN _load_main_session: No session file found:
/tmp/staged/pickled_main_session. Functions defined in __main__ (interactive
session) may fail.
19/11/13 02:10:40 WARN get_all_options: Discarding unparseable args:
[u'--job_server_timeout=60',
u'--app_name=test_windowing_1573611035.32_211aee1e-f0d2-4678-8cc0-a2d15f1ffc99',
u'--direct_runner_use_stacked_bundle', u'--spark_master=local',
u'--options_id=28', u'--enable_spark_metric_sinks', u'--pipeline_type_check']
19/11/13 02:10:40 INFO main: Python sdk harness started with pipeline_options:
{'runner': u'None', 'experiments': [u'beam_fn_api'],
'environment_cache_millis': u'0', 'environment_type': u'PROCESS',
'sdk_location': u'container', 'job_name': u'test_windowing_1573611035.32',
'environment_config': u'{"command":
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh"}',>
'sdk_worker_parallelism': u'1', 'job_endpoint': u'localhost:45951'}
19/11/13 02:10:40 INFO __init__: Creating state cache with size 0
19/11/13 02:10:40 INFO __init__: Creating insecure control channel for
localhost:46759.
19/11/13 02:10:40 INFO __init__: Control channel established.
19/11/13 02:10:40 INFO FnApiControlClientPoolService: Beam Fn Control client
connected with id 258-1
19/11/13 02:10:40 INFO __init__: Initializing SDKHarness with 12 workers.
19/11/13 02:10:40 INFO create_state_handler: Creating insecure state channel
for localhost:42343.
19/11/13 02:10:40 INFO create_state_handler: State channel established.
19/11/13 02:10:40 INFO create_data_channel: Creating client data channel for
localhost:39179
19/11/13 02:10:40 INFO GrpcDataService: Beam Fn Data client connected.
19/11/13 02:10:40 INFO DefaultJobBundleFactory: Closing environment urn:
"beam:env:process:v1"
payload:
"\032\202\001<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build/sdk_worker.sh";>
19/11/13 02:10:40 INFO run: No more requests from control plane
19/11/13 02:10:40 INFO run: SDK Harness waiting for in-flight requests to
complete
19/11/13 02:10:40 INFO close: Closing all cached grpc data channels.
19/11/13 02:10:40 WARN BeamFnDataGrpcMultiplexer: Hanged up for unknown
endpoint.
19/11/13 02:10:40 INFO close: Closing all cached gRPC state handlers.
19/11/13 02:10:40 INFO run: Done consuming work.
19/11/13 02:10:40 INFO main: Python sdk harness exiting.
19/11/13 02:10:40 INFO GrpcLoggingService: Logging client hanged up.
19/11/13 02:10:40 WARN BeamFnDataGrpcMultiplexer: Hanged up for unknown
endpoint.
19/11/13 02:10:40 INFO Executor: Finished task 0.0 in stage 128.0 (TID 156).
12013 bytes result sent to driver
19/11/13 02:10:40 INFO TaskSetManager: Finished task 0.0 in stage 128.0 (TID
156) in 867 ms on localhost (executor driver) (1/1)
19/11/13 02:10:40 INFO TaskSchedulerImpl: Removed TaskSet 128.0, whose tasks
have all completed, from pool
19/11/13 02:10:40 INFO DAGScheduler: ResultStage 128 (foreach at
BoundedDataset.java:124) finished in 0.872 s
19/11/13 02:10:40 INFO DAGScheduler: Job 45 finished: foreach at
BoundedDataset.java:124, took 4.323095 s
19/11/13 02:10:40 INFO SparkPipelineRunner: Job
test_windowing_1573611035.32_211aee1e-f0d2-4678-8cc0-a2d15f1ffc99 finished.
19/11/13 02:10:40 WARN SparkPipelineResult$BatchMode: Collecting monitoring
infos is not implemented yet in Spark portable runner.
19/11/13 02:10:40 INFO AbstractArtifactRetrievalService: Manifest at
/tmp/sparktestmw6vvL/job_6555e300-d5b4-4b2d-9e8a-40cd03cb4767/MANIFEST has 0
artifact locations
19/11/13 02:10:40 INFO BeamFileSystemArtifactStagingService: Removed dir
/tmp/sparktestmw6vvL/job_6555e300-d5b4-4b2d-9e8a-40cd03cb4767/
INFO:root:Job state changed to DONE
.
======================================================================
ERROR: test_pardo_state_with_custom_key_coder (__main__.SparkRunnerTest)
Tests that state requests work correctly when the key coder is an
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/portable_runner_test.py", line 232, in
test_pardo_state_with_custom_key_coder
equal_to(expected))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 435, in
wait_until_finish
for state_response in self._state_stream:
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py";,>
line 395, in next
return self._next()
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py";,>
line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py";,>
line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py";,>
line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
==================== Timed out after 60 seconds. ====================
File "apache_beam/runners/portability/portable_runner_test.py", line 73, in
handler
# Thread: <Thread(wait_until_finish_read, started daemon 139845827606272)>
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
# Thread: <Thread(Thread-119, started daemon 139845477332736)>
======================================================================
# Thread: <_MainThread(MainThread, started 139846606837504)>
==================== Timed out after 60 seconds. ====================
ERROR: test_pardo_timers (__main__.SparkRunnerTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "apache_beam/runners/portability/fn_api_runner_test.py", line 323, in
test_pardo_timers
assert_that(actual, equal_to(expected))
File "apache_beam/pipeline.py", line 436, in __exit__
self.run().wait_until_finish()
File "apache_beam/runners/portability/portable_runner.py", line 435, in
wait_until_finish
for state_response in self._state_stream:
# Thread: <Thread(wait_until_finish_read, started daemon 139845452154624)>
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py";,>
line 395, in next
return self._next()
# Thread: <Thread(Thread-125, started daemon 139845460547328)>
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_channel.py";,>
line 552, in _next
_common.wait(self._state.condition.wait, _response_ready)
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py";,>
line 140, in wait
_wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
# Thread: <_MainThread(MainThread, started 139846606837504)>
File
"<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/grpc/_common.py";,>
line 105, in _wait_once
wait_fn(timeout=timeout)
File "/usr/lib/python2.7/threading.py", line 359, in wait
_sleep(delay)
File "apache_beam/runners/portability/portable_runner_test.py", line 73, in
handler
raise BaseException(msg)
BaseException: Timed out after 60 seconds.
# Thread: <Thread(Thread-119, started daemon 139845477332736)>
----------------------------------------------------------------------
Ran 36 tests in 295.485s
# Thread: <Thread(wait_until_finish_read, started daemon 139845827606272)>
FAILED (errors=2, skipped=9)
> Task :sdks:python:test-suites:portable:py2:sparkValidatesRunner FAILED
FAILURE: Build failed with an exception.
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py2/build.gradle'>
line: 198
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py2:sparkValidatesRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 7m 49s
59 actionable tasks: 46 executed, 13 from cache
Publishing build scan...
https://gradle.com/s/bh5o5n4av2yuw
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
