See
<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/7157/display/redirect?page=changes>
Changes:
[noreply] [RRIO] create RedisClient utility (#29081)
------------------------------------------
[...truncated 776.30 KB...]
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.3/49.3 kB 2.5 MB/s eta 0:00:00
Collecting dill<0.3.2,>=0.3.1.1 (from apache-beam==2.52.0.dev0)
Using cached dill-0.3.1.1-py3-none-any.whl
Collecting cloudpickle~=2.2.1 (from apache-beam==2.52.0.dev0)
Using cached cloudpickle-2.2.1-py3-none-any.whl (25 kB)
Collecting fastavro<2,>=0.23.6 (from apache-beam==2.52.0.dev0)
Using cached
fastavro-1.8.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
(5.5 kB)
Collecting fasteners<1.0,>=0.3 (from apache-beam==2.52.0.dev0)
Using cached fasteners-0.19-py3-none-any.whl.metadata (4.9 kB)
Collecting grpcio!=1.48.0,<2,>=1.33.1 (from apache-beam==2.52.0.dev0)
Using cached
grpcio-1.59.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
(4.0 kB)
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.52.0.dev0)
Using cached hdfs-2.7.3-py3-none-any.whl
Collecting httplib2<0.23.0,>=0.8 (from apache-beam==2.52.0.dev0)
Using cached httplib2-0.22.0-py3-none-any.whl (96 kB)
Collecting js2py<1,>=0.74 (from apache-beam==2.52.0.dev0)
Using cached Js2Py-0.74-py3-none-any.whl (1.0 MB)
Collecting jsonschema<5.0.0,>=4.0.0 (from apache-beam==2.52.0.dev0)
Using cached jsonschema-4.19.1-py3-none-any.whl.metadata (7.9 kB)
Collecting numpy<1.25.0,>=1.14.3 (from apache-beam==2.52.0.dev0)
Using cached
numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
(5.6 kB)
Collecting objsize<0.7.0,>=0.6.1 (from apache-beam==2.52.0.dev0)
Using cached objsize-0.6.1-py3-none-any.whl (9.3 kB)
Collecting packaging>=22.0 (from apache-beam==2.52.0.dev0)
Using cached packaging-23.2-py3-none-any.whl.metadata (3.2 kB)
Collecting pymongo<5.0.0,>=3.8.0 (from apache-beam==2.52.0.dev0)
Using cached
pymongo-4.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
(22 kB)
Collecting proto-plus<2,>=1.7.1 (from apache-beam==2.52.0.dev0)
Using cached proto_plus-1.22.3-py3-none-any.whl.metadata (2.2 kB)
Collecting
protobuf!=4.0.*,!=4.21.*,!=4.22.0,!=4.23.*,!=4.24.0,!=4.24.1,!=4.24.2,<4.25.0,>=3.20.3
(from apache-beam==2.52.0.dev0)
Using cached protobuf-4.24.4-cp37-abi3-manylinux2014_x86_64.whl.metadata (540
bytes)
Collecting pydot<2,>=1.2.0 (from apache-beam==2.52.0.dev0)
Using cached pydot-1.4.2-py2.py3-none-any.whl (21 kB)
Collecting python-dateutil<3,>=2.8.0 (from apache-beam==2.52.0.dev0)
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2018.3 (from apache-beam==2.52.0.dev0)
Using cached pytz-2023.3.post1-py2.py3-none-any.whl.metadata (22 kB)
Collecting regex>=2020.6.8 (from apache-beam==2.52.0.dev0)
Using cached
regex-2023.10.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
(40 kB)
Collecting requests<3.0.0,>=2.24.0 (from apache-beam==2.52.0.dev0)
Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting typing-extensions>=3.7.0 (from apache-beam==2.52.0.dev0)
Using cached typing_extensions-4.8.0-py3-none-any.whl.metadata (3.0 kB)
Collecting zstandard<1,>=0.18.0 (from apache-beam==2.52.0.dev0)
Using cached
zstandard-0.21.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7
MB)
Collecting pyarrow<12.0.0,>=3.0.0 (from apache-beam==2.52.0.dev0)
Using cached
pyarrow-11.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (35.0
MB)
Collecting docopt (from hdfs<3.0.0,>=2.1.0->apache-beam==2.52.0.dev0)
Using cached docopt-0.6.2-py2.py3-none-any.whl
Collecting six>=1.9.0 (from hdfs<3.0.0,>=2.1.0->apache-beam==2.52.0.dev0)
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 (from
httplib2<0.23.0,>=0.8->apache-beam==2.52.0.dev0)
Using cached pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB)
Collecting tzlocal>=1.2 (from js2py<1,>=0.74->apache-beam==2.52.0.dev0)
Using cached tzlocal-5.2-py3-none-any.whl.metadata (7.8 kB)
Collecting pyjsparser>=2.5.1 (from js2py<1,>=0.74->apache-beam==2.52.0.dev0)
Using cached pyjsparser-2.7.1-py3-none-any.whl
Collecting attrs>=22.2.0 (from
jsonschema<5.0.0,>=4.0.0->apache-beam==2.52.0.dev0)
Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
Collecting importlib-resources>=1.4.0 (from
jsonschema<5.0.0,>=4.0.0->apache-beam==2.52.0.dev0)
Using cached importlib_resources-6.1.0-py3-none-any.whl.metadata (4.1 kB)
Collecting jsonschema-specifications>=2023.03.6 (from
jsonschema<5.0.0,>=4.0.0->apache-beam==2.52.0.dev0)
Using cached jsonschema_specifications-2023.7.1-py3-none-any.whl.metadata
(2.8 kB)
Collecting pkgutil-resolve-name>=1.3.10 (from
jsonschema<5.0.0,>=4.0.0->apache-beam==2.52.0.dev0)
Using cached pkgutil_resolve_name-1.3.10-py3-none-any.whl (4.7 kB)
Collecting referencing>=0.28.4 (from
jsonschema<5.0.0,>=4.0.0->apache-beam==2.52.0.dev0)
Using cached referencing-0.30.2-py3-none-any.whl.metadata (2.6 kB)
Collecting rpds-py>=0.7.1 (from
jsonschema<5.0.0,>=4.0.0->apache-beam==2.52.0.dev0)
Using cached
rpds_py-0.10.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
(3.7 kB)
Collecting dnspython<3.0.0,>=1.16.0 (from
pymongo<5.0.0,>=3.8.0->apache-beam==2.52.0.dev0)
Using cached dnspython-2.4.2-py3-none-any.whl.metadata (4.9 kB)
Collecting charset-normalizer<4,>=2 (from
requests<3.0.0,>=2.24.0->apache-beam==2.52.0.dev0)
Using cached
charset_normalizer-3.3.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
(33 kB)
Collecting idna<4,>=2.5 (from requests<3.0.0,>=2.24.0->apache-beam==2.52.0.dev0)
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<3,>=1.21.1 (from
requests<3.0.0,>=2.24.0->apache-beam==2.52.0.dev0)
Using cached urllib3-2.0.7-py3-none-any.whl.metadata (6.6 kB)
Collecting certifi>=2017.4.17 (from
requests<3.0.0,>=2.24.0->apache-beam==2.52.0.dev0)
Using cached certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting zipp>=3.1.0 (from
importlib-resources>=1.4.0->jsonschema<5.0.0,>=4.0.0->apache-beam==2.52.0.dev0)
Using cached zipp-3.17.0-py3-none-any.whl.metadata (3.7 kB)
Collecting backports.zoneinfo (from
tzlocal>=1.2->js2py<1,>=0.74->apache-beam==2.52.0.dev0)
Using cached backports.zoneinfo-0.2.1-cp38-cp38-manylinux1_x86_64.whl (74 kB)
Using cached
fastavro-1.8.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Using cached fasteners-0.19-py3-none-any.whl (18 kB)
Using cached
grpcio-1.59.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB)
Using cached jsonschema-4.19.1-py3-none-any.whl (83 kB)
Using cached
numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Downloading
orjson-3.9.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.5/138.5 kB 6.5 MB/s eta 0:00:00
Using cached packaging-23.2-py3-none-any.whl (53 kB)
Using cached proto_plus-1.22.3-py3-none-any.whl (48 kB)
Using cached protobuf-4.24.4-cp37-abi3-manylinux2014_x86_64.whl (311 kB)
Using cached
pymongo-4.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (687 kB)
Using cached pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)
Using cached
regex-2023.10.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (776
kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Using cached typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Using cached certifi-2023.7.22-py3-none-any.whl (158 kB)
Using cached
charset_normalizer-3.3.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
(138 kB)
Using cached dnspython-2.4.2-py3-none-any.whl (300 kB)
Using cached importlib_resources-6.1.0-py3-none-any.whl (33 kB)
Using cached jsonschema_specifications-2023.7.1-py3-none-any.whl (17 kB)
Using cached pyparsing-3.1.1-py3-none-any.whl (103 kB)
Using cached referencing-0.30.2-py3-none-any.whl (25 kB)
Using cached
rpds_py-0.10.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
Using cached tzlocal-5.2-py3-none-any.whl (17 kB)
Using cached urllib3-2.0.7-py3-none-any.whl (124 kB)
Using cached zipp-3.17.0-py3-none-any.whl (7.4 kB)
Building wheels for collected packages: apache-beam
Building editable for apache-beam (pyproject.toml): started
Building editable for apache-beam (pyproject.toml): still running...
Building editable for apache-beam (pyproject.toml): still running...
Building editable for apache-beam (pyproject.toml): still running...
Building editable for apache-beam (pyproject.toml): finished with status
'done'
Created wheel for apache-beam:
filename=apache_beam-2.52.0.dev0-0.editable-cp38-cp38-linux_x86_64.whl
size=4354
sha256=e414d9a943fdfa89d2f46e4a84e89381fea548b3ea5647410d09f33092113e6f
Stored in directory:
/tmp/pip-ephem-wheel-cache-nhtdlkl7/wheels/ac/52/de/febaa438a45e9de73dbe46cdcae89d0f86d2fa2d8f6196d824
Successfully built apache-beam
Installing collected packages: pytz, pyjsparser, docopt, crcmod, zstandard,
zipp, urllib3, typing-extensions, six, rpds-py, regex, pyparsing, protobuf,
pkgutil-resolve-name, packaging, orjson, objsize, numpy, idna, grpcio,
fasteners, fastavro, dnspython, dill, cloudpickle, charset-normalizer, certifi,
backports.zoneinfo, attrs, tzlocal, requests, referencing, python-dateutil,
pymongo, pydot, pyarrow, proto-plus, importlib-resources, httplib2,
jsonschema-specifications, js2py, hdfs, jsonschema, apache-beam
Successfully installed apache-beam-2.52.0.dev0 attrs-23.1.0
backports.zoneinfo-0.2.1 certifi-2023.7.22 charset-normalizer-3.3.1
cloudpickle-2.2.1 crcmod-1.7 dill-0.3.1.1 dnspython-2.4.2 docopt-0.6.2
fastavro-1.8.4 fasteners-0.19 grpcio-1.59.0 hdfs-2.7.3 httplib2-0.22.0 idna-3.4
importlib-resources-6.1.0 js2py-0.74 jsonschema-4.19.1
jsonschema-specifications-2023.7.1 numpy-1.24.4 objsize-0.6.1 orjson-3.9.10
packaging-23.2 pkgutil-resolve-name-1.3.10 proto-plus-1.22.3 protobuf-4.24.4
pyarrow-11.0.0 pydot-1.4.2 pyjsparser-2.7.1 pymongo-4.5.0 pyparsing-3.1.1
python-dateutil-2.8.2 pytz-2023.3.post1 referencing-0.30.2 regex-2023.10.3
requests-2.31.0 rpds-py-0.10.6 six-1.16.0 typing-extensions-4.8.0 tzlocal-5.2
urllib3-2.0.7 zipp-3.17.0 zstandard-0.21.0
PIPELINE_PY="
import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.transforms import Create
from apache_beam.transforms import Map
# To test that our main session is getting plumbed through artifact staging
# correctly, create a global variable. If the main session is not plumbed
# through properly, global_var will be undefined and the pipeline will fail.
global_var = 1
pipeline_options = PipelineOptions()
pipeline_options.view_as(SetupOptions).save_main_session = True
pipeline = beam.Pipeline(options=pipeline_options)
pcoll = (pipeline
| Create([0, 1, 2])
| Map(lambda x: x + global_var))
assert_that(pcoll, equal_to([1, 2, 3]))
result = pipeline.run()
result.wait_until_finish()
"
if [[ "$RUNNER" = "FlinkRunner" ]]; then
INPUT_JAR_ARG="flink_job_server_jar"
else
INPUT_JAR_ARG="spark_job_server_jar"
fi
# Create the jar
OUTPUT_JAR="test-pipeline-${RUNNER}-$(date +%Y%m%d-%H%M%S).jar"
(python -c "$PIPELINE_PY" \
--runner "$RUNNER" \
--"$INPUT_JAR_ARG" "$JOB_SERVER_JAR" \
--output_executable_path $OUTPUT_JAR \
--parallelism 1 \
--sdk_worker_parallelism 1 \
--environment_type DOCKER \
--environment_options "docker_container_image=$PYTHON_CONTAINER_IMAGE" \
) || TEST_EXIT_CODE=$? # don't fail fast here; clean up before exiting
ERROR:apache_beam.utils.subprocess_server:Starting job service with ['java',
'-jar',
'<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/runners/flink/1.16/job-server/build/libs/beam-runners-flink-1.16-job-server-2.52.0-SNAPSHOT.jar',>
'--flink-master', '[auto]', '--artifacts-dir',
'/tmp/beam-tempd5u1fpxz/artifactsgj5iwbqz', '--job-port', '52319',
'--artifact-port', '0', '--expansion-port', '0']
ERROR:apache_beam.utils.subprocess_server:Error bringing up service
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/utils/subprocess_server.py";,>
line 88, in start
raise RuntimeError(
RuntimeError: Service failed to start up with error 1
Traceback (most recent call last):
File "<string>", line 23, in <module>
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 585, in run
return self.runner.run_pipeline(self, self._options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/runner.py";,>
line 192, in run_pipeline
return self.run_portable_pipeline(
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,>
line 381, in run_portable_pipeline
job_service_handle = self.create_job_service(options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,>
line 296, in create_job_service
return self.create_job_service_handle(server.start(), options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/job_server.py";,>
line 81, in start
self._endpoint = self._job_server.start()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/job_server.py";,>
line 110, in start
return self._server.start()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/utils/subprocess_server.py";,>
line 88, in start
raise RuntimeError(
RuntimeError: Service failed to start up with error 1
if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then
# Execute the jar
java -jar $OUTPUT_JAR || TEST_EXIT_CODE=$?
fi
rm -rf $ENV_DIR
rm -f $OUTPUT_JAR
if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then
echo ">>> SUCCESS"
else
echo ">>> FAILURE"
fi
exit $TEST_EXIT_CODE
>>> FAILURE
> Task :sdks:python:test-suites:portable:py38:testJavaJarCreatorFlinkRunner
> FAILED
FAILURE: Build completed with 2 failures.
1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':runners:flink:1.16:job-server:miniCluster'.
> Cannot expand ZIP
> '<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/runners/flink/1.16/job-server/build/libs/beam-runners-flink-1.16-job-server-2.52.0-SNAPSHOT.jar'.>
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 383
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py38:testJavaJarCreatorFlinkRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Get more help at https://help.gradle.org.
==============================================================================
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
For more on this, please refer to
https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings
in the Gradle documentation.
BUILD FAILED in 11m 53s
146 actionable tasks: 99 executed, 45 from cache, 2 up-to-date
Publishing build scan...
https://ge.apache.org/s/rufx4j2tng4dq
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
