See
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/5790/display/redirect?page=changes>
Changes:
[benjamin.gonzalez] [BEAM-12572] Add DistinctExample test
[benjamin.gonzalez] [BEAM-12572] Add DistinctExample test
[benjamin.gonzalez] [BEAM-12572] Add java examples tests
[benjamin.gonzalez] [BEAM-12572] Change tests collected by runner
[benjamin.gonzalez] [BEAM-12572] Fix PipelineOptions, skip tests for specific
runners
[benjamin.gonzalez] [BEAM-12572] Skip failing tests
[benjamin.gonzalez] [BEAM-12572] Fix checkstyle warnings
[benjamin.gonzalez] [BEAM-12572] Skip tfidf tests on direct runner
[benjamin.gonzalez] [BEAM-12572] Fix PipelineOptions
[benjamin.gonzalez] [BEAM-12572] Fix checkstyle warning
[benjamin.gonzalez] [BEAM-12572] Enable test with Pipeline serialization error
fixed, link
[benjamin.gonzalez] [BEAM-12572] Sickbay TfIdf test for DirectRunner
[benjamin.gonzalez] [BEAM-12572] Refactor naming for BigQueryClient
[mmack] [adhoc] Update owners for aws modules
[noreply] Merge pull request #17000 from [BEAM-13881] [Playground] Increase test
[noreply] Add deterministic dict coding via key sorting. (#17037)
[noreply] Merge pull request #17054 from [BEAM-14075] [SnowflakeIO] Change a
------------------------------------------
[...truncated 427.22 KB...]
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:411)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:417)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException: Waited 3 minutes for
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture@7f1ab275[status=PENDING]
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:471)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
at
org.apache.beam.runners.fnexecution.data.GrpcDataService.send(GrpcDataService.java:186)
... 24 more
22/03/15 00:08:12 ERROR org.apache.spark.scheduler.TaskSetManager: Task 1 in
stage 127.0 failed 1 times; aborting job
22/03/15 00:08:12 ERROR org.apache.beam.runners.jobsubmission.JobInvocation:
Error during job invocation
test_pardo_state_with_custom_key_coder_1647302700.953042_0c677aab-1f71-452a-a372-47821ceeee60.
org.apache.beam.sdk.Pipeline$PipelineExecutionException:
java.lang.RuntimeException: No client connected within timeout
at
org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:73)
at
org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:104)
at
org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:92)
at
org.apache.beam.runners.spark.SparkPipelineRunner.run(SparkPipelineRunner.java:190)
at
org.apache.beam.runners.jobsubmission.JobInvocation.runPipeline(JobInvocation.java:86)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: No client connected within timeout
at
org.apache.beam.runners.fnexecution.data.GrpcDataService.send(GrpcDataService.java:192)
at
org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor.newBundle(SdkHarnessClient.java:287)
at
org.apache.beam.runners.fnexecution.control.SdkHarnessClient$BundleProcessor.newBundle(SdkHarnessClient.java:197)
at
org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$SimpleStageBundleFactory.getBundle(DefaultJobBundleFactory.java:519)
at
org.apache.beam.runners.fnexecution.control.StageBundleFactory.getBundle(StageBundleFactory.java:87)
at
org.apache.beam.runners.fnexecution.control.StageBundleFactory.getBundle(StageBundleFactory.java:76)
at
org.apache.beam.runners.spark.translation.SparkExecutableStageFunction.processElements(SparkExecutableStageFunction.java:223)
at
org.apache.beam.runners.spark.translation.SparkExecutableStageFunction.call(SparkExecutableStageFunction.java:150)
at
org.apache.beam.runners.spark.translation.SparkExecutableStageFunction.lambda$forPair$55b1e86f$1(SparkExecutableStageFunction.java:125)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:125)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.apply(JavaRDDLike.scala:125)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
at
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:148)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:55)
at org.apache.spark.scheduler.Task.run(Task.scala:123)
at
org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:411)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:417)
... 3 more
Caused by: java.util.concurrent.TimeoutException: Waited 3 minutes for
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture@7f1ab275[status=PENDING]
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:471)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:90)
at
org.apache.beam.runners.fnexecution.data.GrpcDataService.send(GrpcDataService.java:186)
... 24 more
ERROR:root:java.util.concurrent.TimeoutException: Waited 3 minutes for
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture@7f1ab275[status=PENDING]
------------------------------ Captured log call -------------------------------
WARNING root:environments.py:371 Make sure that locally built Python SDK
docker image has Python 3.8 interpreter.
ERROR apache_beam.runners.worker.data_plane:data_plane.py:641 Failed to read
inputs in the data plane.
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/worker/data_plane.py";,>
line 634, in _read_inputs
for elements in elements_iterator:
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/target/.tox-spark-runner-test/spark-runner-test/lib/python3.8/site-packages/grpc/_channel.py";,>
line 426, in __next__
return self._next()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/target/.tox-spark-runner-test/spark-runner-test/lib/python3.8/site-packages/grpc/_channel.py";,>
line 809, in _next
raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that
terminated with:
status = StatusCode.UNAVAILABLE
details = "Broken pipe"
debug_error_string =
"{"created":"@1647302712.178887274","description":"Error received from peer
ipv6:[::1]:43353","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"Broken
pipe","grpc_status":14}"
>
ERROR apache_beam.runners.worker.sdk_worker:sdk_worker.py:271 Error
processing instruction 1. Original traceback is
Traceback
(most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/worker/data_plane.py";,>
line 487, in input_elements
element =
received.get(timeout=1)
File
"/usr/lib/python3.8/queue.py", line 178, in get
raise Empty
_queue.Empty
During
handling of the above exception, another exception occurred:
Traceback
(most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 267, in _execute
response =
task()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 340, in <lambda>
lambda:
self.create_worker().do_instruction(request), request)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 580, in do_instruction
return
getattr(self, request_type)(
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py";,>
line 618, in process_bundle
bundle_processor.process_bundle(instruction_id))
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/worker/bundle_processor.py";,>
line 984, in process_bundle
for
element in data_channel.input_elements(instruction_id,
File
"<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/worker/data_plane.py";,>
line 490, in input_elements
raise
RuntimeError('Channel closed prematurely.')
RuntimeError:
Channel closed prematurely.
ERROR root:portable_runner.py:567 java.util.concurrent.TimeoutException:
Waited 3 minutes for
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.SettableFuture@7f1ab275[status=PENDING]
=============================== warnings summary ===============================
target/.tox-spark-runner-test/spark-runner-test/lib/python3.8/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/target/.tox-spark-runner-test/spark-runner-test/lib/python3.8/site-packages/tenacity/_asyncio.py>:42:
DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use
"async def" instead
def call(self, fn, *args, **kwargs):
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/pytest_spark-runner-test.xml>
-
======== 1 failed, 34 passed, 17 skipped, 1 warnings in 243.19 seconds =========
[31mERROR: InvocationError for command
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/scripts/pytest_validates_runner.sh>
spark-runner-test apache_beam/runners/portability/spark_runner_test.py
'--spark_job_server_jar=<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/runners/spark/2/job-server/build/libs/beam-runners-spark-job-server-2.38.0-SNAPSHOT.jar>
--environment_type=LOOPBACK' (exited with code 1)
[0m[1mspark-runner-test run-test-post: commands[0] | bash
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m___________________________________ summary
____________________________________
[31mERROR: spark-runner-test: commands failed
[0m
> Task :sdks:python:test-suites:portable:py38:sparkCompatibilityMatrixLOOPBACK
> FAILED
> Task :sdks:python:test-suites:portable:py36:sparkCompatibilityMatrixLOOPBACK
[1mspark-runner-test create:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/target/.tox-spark-runner-test/spark-runner-test>
[0m[1mspark-runner-test installdeps: -rbuild-requirements.txt
[0m[1mspark-runner-test inst:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/target/.tox-spark-runner-test/.tmp/package/1/apache-beam.tar.gz>
[0m[1mspark-runner-test installed: apache-beam @
file://<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/target/.tox-spark-runner-test/.tmp/package/1/apache-beam.tar.gz,atomicwrites==1.4.0,attrs==21.4.0,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.12,cloudpickle==2.0.0,crcmod==1.7,cryptography==36.0.1,dataclasses==0.8,deprecation==2.1.0,dill==0.3.1.1,distlib==0.3.1,docker==5.0.3,docopt==0.6.2,execnet==1.9.0,fastavro==1.4.7,freezegun==1.2.0,greenlet==1.1.2,grpcio==1.44.0,grpcio-tools==1.37.0,hdfs==2.6.0,httplib2==0.19.1,idna==3.3,importlib-metadata==4.8.3,mock==2.0.0,more-itertools==8.12.0,mypy-protobuf==1.18,numpy==1.19.5,oauth2client==4.1.3,orjson==3.6.1,packaging==21.3,pandas==1.1.5,parameterized==0.7.5,pbr==5.8.1,pluggy==0.13.1,proto-plus==1.20.3,protobuf==3.19.4,psycopg2-binary==2.9.3,py==1.11.0,pyarrow==6.0.1,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycparser==2.21,pydot==1.4.2,PyHamcrest==1.10.1,pymongo==3.12.3,PyMySQL==1.0.2,pyparsing==2.4.7,pytest==4.6.11,pytest-forked==1.4.0,pytest-timeout==1.4.2,pytest-xdist==1.34.0,python-dateutil==2.8.2,pytz==2021.3,PyYAML==6.0,requests==2.27.1,requests-mock==1.9.3,rsa==4.8,six==1.16.0,SQLAlchemy==1.4.32,tenacity==5.1.5,testcontainers==3.4.2,typing_extensions==4.1.1,urllib3==1.26.8,wcwidth==0.2.5,websocket-client==1.3.1,wrapt==1.14.0,zipp==3.6.0>
[0m[1mspark-runner-test run-test-pre: PYTHONHASHSEED='4265757206'
[0m[1mspark-runner-test run-test-pre: commands[0] | python --version
[0mPython 3.6.15
[1mspark-runner-test run-test-pre: commands[1] | pip --version
[0mpip 21.3.1 from
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/target/.tox-spark-runner-test/spark-runner-test/lib/python3.6/site-packages/pip>
(python 3.6)
[1mspark-runner-test run-test-pre: commands[2] | pip check
[0mNo broken requirements found.
[1mspark-runner-test run-test-pre: commands[3] | bash
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m[1mspark-runner-test run-test: commands[0] |
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/scripts/pytest_validates_runner.sh>
spark-runner-test
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/apache_beam/runners/portability/spark_runner_test.py>
'--spark_job_server_jar=<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/runners/spark/2/job-server/build/libs/beam-runners-spark-job-server-2.38.0-SNAPSHOT.jar>
--environment_type=LOOPBACK'
[0m============================= test session starts
==============================
platform linux -- Python 3.6.15, pytest-4.6.11, py-1.11.0, pluggy-0.13.1
cachedir: target/.tox-spark-runner-test/spark-runner-test/.pytest_cache
rootdir:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python,>
inifile: pytest.ini
plugins: xdist-1.34.0, timeout-1.4.2, forked-1.4.0, requests-mock-1.9.3
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 52 items
apache_beam/runners/portability/spark_runner_test.py .s...ss.......ss... [ 36%]
...s..s...........s.ssss..sssss.. [100%]
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/pytest_spark-runner-test.xml>
-
==================== 35 passed, 17 skipped in 96.32 seconds ====================
[1mspark-runner-test run-test-post: commands[0] | bash
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py36/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m___________________________________ summary
____________________________________
[32m spark-runner-test: commands succeeded
[0m[32m congratulations :)
[0m
> Task :sdks:python:test-suites:portable:py36:sparkValidatesRunner
> Task :sdks:python:test-suites:portable:py37:sparkCompatibilityMatrixLOOPBACK
[1mspark-runner-test create:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/target/.tox-spark-runner-test/spark-runner-test>
[0m[1mspark-runner-test installdeps: -rbuild-requirements.txt
[0m[1mspark-runner-test inst:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/target/.tox-spark-runner-test/.tmp/package/1/apache-beam.tar.gz>
[0m[1mspark-runner-test installed: apache-beam @
file://<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/target/.tox-spark-runner-test/.tmp/package/1/apache-beam.tar.gz,atomicwrites==1.4.0,attrs==21.4.0,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.12,cloudpickle==2.0.0,crcmod==1.7,cryptography==36.0.1,deprecation==2.1.0,dill==0.3.1.1,distlib==0.3.1,docker==5.0.3,docopt==0.6.2,execnet==1.9.0,fastavro==1.4.10,freezegun==1.2.0,greenlet==1.1.2,grpcio==1.44.0,grpcio-tools==1.37.0,hdfs==2.6.0,httplib2==0.19.1,idna==3.3,importlib-metadata==4.11.3,mock==2.0.0,more-itertools==8.12.0,mypy-protobuf==1.18,numpy==1.21.5,oauth2client==4.1.3,orjson==3.6.7,packaging==21.3,pandas==1.3.5,parameterized==0.7.5,pbr==5.8.1,pluggy==0.13.1,proto-plus==1.20.3,protobuf==3.19.4,psycopg2-binary==2.9.3,py==1.11.0,pyarrow==6.0.1,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycparser==2.21,pydot==1.4.2,PyHamcrest==1.10.1,pymongo==3.12.3,PyMySQL==1.0.2,pyparsing==2.4.7,pytest==4.6.11,pytest-forked==1.4.0,pytest-timeout==1.4.2,pytest-xdist==1.34.0,python-dateutil==2.8.2,pytz==2021.3,PyYAML==6.0,requests==2.27.1,requests-mock==1.9.3,rsa==4.8,six==1.16.0,SQLAlchemy==1.4.32,tenacity==5.1.5,testcontainers==3.4.2,typing_extensions==4.1.1,urllib3==1.26.8,wcwidth==0.2.5,websocket-client==1.3.1,wrapt==1.14.0,zipp==3.7.0>
[0m[1mspark-runner-test run-test-pre: PYTHONHASHSEED='2210277081'
[0m[1mspark-runner-test run-test-pre: commands[0] | python --version
[0mPython 3.7.12
[1mspark-runner-test run-test-pre: commands[1] | pip --version
[0mpip 22.0.3 from
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/target/.tox-spark-runner-test/spark-runner-test/lib/python3.7/site-packages/pip>
(python 3.7)
[1mspark-runner-test run-test-pre: commands[2] | pip check
[0mNo broken requirements found.
[1mspark-runner-test run-test-pre: commands[3] | bash
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m[1mspark-runner-test run-test: commands[0] |
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/scripts/pytest_validates_runner.sh>
spark-runner-test
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/apache_beam/runners/portability/spark_runner_test.py>
'--spark_job_server_jar=<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/runners/spark/2/job-server/build/libs/beam-runners-spark-job-server-2.38.0-SNAPSHOT.jar>
--environment_type=LOOPBACK'
[0m============================= test session starts
==============================
platform linux -- Python 3.7.12, pytest-4.6.11, py-1.11.0, pluggy-0.13.1
cachedir: target/.tox-spark-runner-test/spark-runner-test/.pytest_cache
rootdir:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python,>
inifile: pytest.ini
plugins: xdist-1.34.0, timeout-1.4.2, forked-1.4.0, requests-mock-1.9.3
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 52 items
apache_beam/runners/portability/spark_runner_test.py .s...ss.......ss... [ 36%]
...s..s...........s.ssss..sssss.. [100%]
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/pytest_spark-runner-test.xml>
-
==================== 35 passed, 17 skipped in 41.12 seconds ====================
[1mspark-runner-test run-test-post: commands[0] | bash
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m___________________________________ summary
____________________________________
[32m spark-runner-test: commands succeeded
[0m[32m congratulations :)
[0m
> Task :sdks:python:test-suites:portable:py37:sparkValidatesRunner
> Task :sdks:python:test-suites:portable:py39:sparkCompatibilityMatrixLOOPBACK
[1mspark-runner-test create:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python/target/.tox-spark-runner-test/spark-runner-test>
[0m[1mspark-runner-test installdeps: -rbuild-requirements.txt
[0m[1mspark-runner-test inst:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python/target/.tox-spark-runner-test/.tmp/package/1/apache-beam.tar.gz>
[0m[1mspark-runner-test installed: apache-beam @
file://<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python/target/.tox-spark-runner-test/.tmp/package/1/apache-beam.tar.gz,atomicwrites==1.4.0,attrs==21.4.0,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.12,cloudpickle==2.0.0,crcmod==1.7,cryptography==36.0.1,deprecation==2.1.0,dill==0.3.1.1,distlib==0.3.1,docker==5.0.3,docopt==0.6.2,execnet==1.9.0,fastavro==1.4.10,freezegun==1.2.0,greenlet==1.1.2,grpcio==1.44.0,grpcio-tools==1.37.0,hdfs==2.6.0,httplib2==0.19.1,idna==3.3,mock==2.0.0,more-itertools==8.12.0,mypy-protobuf==1.18,numpy==1.22.3,oauth2client==4.1.3,orjson==3.6.7,packaging==21.3,pandas==1.4.1,parameterized==0.7.5,pbr==5.8.1,pluggy==0.13.1,proto-plus==1.20.3,protobuf==3.19.4,psycopg2-binary==2.9.3,py==1.11.0,pyarrow==6.0.1,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycparser==2.21,pydot==1.4.2,PyHamcrest==1.10.1,pymongo==3.12.3,PyMySQL==1.0.2,pyparsing==2.4.7,pytest==4.6.11,pytest-forked==1.4.0,pytest-timeout==1.4.2,pytest-xdist==1.34.0,python-dateutil==2.8.2,pytz==2021.3,PyYAML==6.0,requests==2.27.1,requests-mock==1.9.3,rsa==4.8,six==1.16.0,SQLAlchemy==1.4.32,tenacity==5.1.5,testcontainers==3.4.2,typing_extensions==4.1.1,urllib3==1.26.8,wcwidth==0.2.5,websocket-client==1.3.1,wrapt==1.14.0>
[0m[1mspark-runner-test run-test-pre: PYTHONHASHSEED='346079123'
[0m[1mspark-runner-test run-test-pre: commands[0] | python --version
[0mPython 3.9.10
[1mspark-runner-test run-test-pre: commands[1] | pip --version
[0mpip 22.0.4 from
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python/target/.tox-spark-runner-test/spark-runner-test/lib/python3.9/site-packages/pip>
(python 3.9)
[1mspark-runner-test run-test-pre: commands[2] | pip check
[0mNo broken requirements found.
[1mspark-runner-test run-test-pre: commands[3] | bash
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m[1mspark-runner-test run-test: commands[0] |
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python/scripts/pytest_validates_runner.sh>
spark-runner-test
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python/apache_beam/runners/portability/spark_runner_test.py>
'--spark_job_server_jar=<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/runners/spark/2/job-server/build/libs/beam-runners-spark-job-server-2.38.0-SNAPSHOT.jar>
--environment_type=LOOPBACK'
[0m============================= test session starts
==============================
platform linux -- Python 3.9.10, pytest-4.6.11, py-1.11.0, pluggy-0.13.1
cachedir: target/.tox-spark-runner-test/spark-runner-test/.pytest_cache
rootdir:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python,>
inifile: pytest.ini
plugins: xdist-1.34.0, timeout-1.4.2, forked-1.4.0, requests-mock-1.9.3
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 52 items
apache_beam/runners/portability/spark_runner_test.py .s...ss.......ss... [ 36%]
...s..s...........s.ssss..sssss.. [100%]
=============================== warnings summary ===============================
target/.tox-spark-runner-test/spark-runner-test/lib/python3.9/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python/target/.tox-spark-runner-test/spark-runner-test/lib/python3.9/site-packages/tenacity/_asyncio.py>:42:
DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use
"async def" instead
def call(self, fn, *args, **kwargs):
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python/pytest_spark-runner-test.xml>
-
============== 35 passed, 17 skipped, 1 warnings in 40.52 seconds ==============
[1mspark-runner-test run-test-post: commands[0] | bash
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/ws/src/sdks/python/test-suites/portable/py39/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m___________________________________ summary
____________________________________
[32m spark-runner-test: commands succeeded
[0m[32m congratulations :)
[0m
> Task :sdks:python:test-suites:portable:py39:sparkValidatesRunner
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py38:sparkCompatibilityMatrixLOOPBACK'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 10m 32s
81 actionable tasks: 52 executed, 27 from cache, 2 up-to-date
Publishing build scan...
https://gradle.com/s/diuzpndbw7bss
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
