See
<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/6831/display/redirect>
Changes:
------------------------------------------
[...truncated 658.66 KB...]
Using cached urllib3-2.0.4-py3-none-any.whl (123 kB)
Installing collected packages: pytz, docopt, crcmod, zstandard, urllib3,
typing-extensions, six, regex, pyparsing, protobuf, orjson, objsize, numpy,
idna, grpcio, fasteners, fastavro, dnspython, dill, cloudpickle,
charset-normalizer, certifi, requests, python-dateutil, pymongo, pydot,
pyarrow, proto-plus, httplib2, hdfs, apache-beam
Running setup.py develop for apache-beam
Successfully installed apache-beam-2.50.0.dev0 certifi-2023.7.22
charset-normalizer-3.2.0 cloudpickle-2.2.1 crcmod-1.7 dill-0.3.1.1
dnspython-2.4.1 docopt-0.6.2 fastavro-1.8.2 fasteners-0.18 grpcio-1.56.2
hdfs-2.7.0 httplib2-0.22.0 idna-3.4 numpy-1.24.4 objsize-0.6.1 orjson-3.9.2
proto-plus-1.22.3 protobuf-4.23.4 pyarrow-11.0.0 pydot-1.4.2 pymongo-4.4.1
pyparsing-3.1.1 python-dateutil-2.8.2 pytz-2023.3 regex-2023.6.3
requests-2.31.0 six-1.16.0 typing-extensions-4.7.1 urllib3-2.0.4
zstandard-0.21.0
# Hacky python script to find a free port. Note there is a small chance the
chosen port could
# get taken before being claimed.
SOCKET_SCRIPT="
from __future__ import print_function
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('localhost', 0))
print(s.getsockname()[1])
s.close()
"
FLINK_PORT=$(python -c "$SOCKET_SCRIPT")
Starting Flink mini cluster listening on port 51819
echo "Starting Flink mini cluster listening on port $FLINK_PORT"
java -Dorg.slf4j.simpleLogger.defaultLogLevel=warn -jar
"$FLINK_MINI_CLUSTER_JAR" --rest-port "$FLINK_PORT" --rest-bind-address
localhost &
PIPELINE_PY="
import apache_beam as beam
import logging
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.transforms import Create
from apache_beam.transforms import Map
logging.basicConfig(level=logging.INFO)
pipeline_options = PipelineOptions()
pipeline = beam.Pipeline(options=pipeline_options)
if pipeline_options.view_as(SetupOptions).save_main_session:
# To test that our main session is getting plumbed through artifact staging
# correctly, create a global variable. If the main session is not plumbed
# through properly, global_var will be undefined and the pipeline will fail.
global_var = 1
pcoll = (pipeline
| Create([0, 1, 2])
| Map(lambda x: x + global_var))
else:
pcoll = (pipeline
| Create([0, 1, 2])
| Map(lambda x: x + 1))
assert_that(pcoll, equal_to([1, 2, 3]))
result = pipeline.run()
result.wait_until_finish()
"
if [[ "$SAVE_MAIN_SESSION" -eq 0 ]]; then
(python -c "$PIPELINE_PY" \
--runner FlinkRunner \
--flink_job_server_jar "$FLINK_JOB_SERVER_JAR" \
--parallelism 1 \
--environment_type DOCKER \
--environment_options "docker_container_image=$PYTHON_CONTAINER_IMAGE" \
--flink_master "localhost:$FLINK_PORT" \
--flink_submit_uber_jar \
) || TEST_EXIT_CODE=$? # don't fail fast here; clean up before exiting
else
(python -c "$PIPELINE_PY" \
--runner FlinkRunner \
--flink_job_server_jar "$FLINK_JOB_SERVER_JAR" \
--parallelism 1 \
--environment_type DOCKER \
--environment_options "docker_container_image=$PYTHON_CONTAINER_IMAGE" \
--flink_master "localhost:$FLINK_PORT" \
--flink_submit_uber_jar \
--save_main_session
) || TEST_EXIT_CODE=$? # don't fail fast here; clean up before exiting
fi
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError:
org/kohsuke/args4j/CmdLineException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at
sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:670)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:652)
Caused by: java.lang.ClassNotFoundException: org.kohsuke.args4j.CmdLineException
at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more
INFO:root:Using provided Python SDK container image:
apache/beam_python3.8_sdk:2.50.0.dev
INFO:root:Python SDK container image set to
"apache/beam_python3.8_sdk:2.50.0.dev" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function pack_combiners at 0x7f4ba7e831f0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function lift_combiners at 0x7f4ba7e83280> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sort_stages at 0x7f4ba7e839d0> ====================
INFO:apache_beam.runners.portability.flink_runner:Adding HTTP protocol scheme
to flink_master parameter: http://localhost:51819
INFO:apache_beam.runners.portability.abstract_job_service:Artifact server
started on port 33309
INFO:apache_beam.runners.portability.abstract_job_service:Running job
'job-82e8113c-589d-4478-85c4-b45f35c51ce0'
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/urllib3/connection.py";,>
line 203, in _new_conn
sock = connection.create_connection(
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/urllib3/util/connection.py";,>
line 85, in create_connection
raise err
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/urllib3/util/connection.py";,>
line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/urllib3/connectionpool.py";,>
line 790, in urlopen
response = self._make_request(
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/urllib3/connectionpool.py";,>
line 496, in _make_request
conn.request(
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/urllib3/connection.py";,>
line 395, in request
self.endheaders()
File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
self.send(msg)
File "/usr/lib/python3.8/http/client.py", line 951, in send
self.connect()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/urllib3/connection.py";,>
line 243, in connect
self.sock = self._new_conn()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/urllib3/connection.py";,>
line 218, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection
object at 0x7f4ba7dacbe0>: Failed to establish a new connection: [Errno 111]
Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/requests/adapters.py";,>
line 486, in send
resp = conn.urlopen(
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/urllib3/connectionpool.py";,>
line 844, in urlopen
retries = retries.increment(
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/urllib3/util/retry.py";,>
line 515, in increment
raise MaxRetryError(_pool, url, reason) from reason # type:
ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost',
port=51819): Max retries exceeded with url: /v1/jars/upload (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7f4ba7dacbe0>: Failed to establish a new connection: [Errno 111] Connection
refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 31, in <module>
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 577, in run
return self.runner.run_pipeline(self, self._options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/flink_runner.py";,>
line 45, in run_pipeline
return super().run_pipeline(pipeline, options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,>
line 448, in run_pipeline
job_service_handle.submit(proto_pipeline)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,>
line 117, in submit
return self.run(prepare_response.preparation_id)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,>
line 244, in run
run_response = self.job_service.Run(
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/abstract_job_service.py";,>
line 120, in Run
self._jobs[job_id].run()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py";,>
line 152, in run
self._flink_jar_id = self.post(
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py";,>
line 142, in post
return self.request(requests.post, path, **kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py";,>
line 130, in request
response = method(url, **kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/requests/api.py";,>
line 115, in post
return request("post", url, data=data, json=json, **kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/requests/api.py";,>
line 59, in request
return session.request(method=method, url=url, **kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/requests/sessions.py";,>
line 589, in request
resp = self.send(prep, **send_kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/requests/sessions.py";,>
line 703, in send
r = adapter.send(request, **kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703442/lib/python3.8/site-packages/requests/adapters.py";,>
line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost',
port=51819): Max retries exceeded with url: /v1/jars/upload (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7f4ba7dacbe0>: Failed to establish a new connection: [Errno 111] Connection
refused'))
kill %1 || echo "Failed to shut down Flink mini cluster"
rm -rf "$ENV_DIR"
if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then
echo ">>> SUCCESS"
else
echo ">>> FAILURE"
fi
>>> FAILURE
exit $TEST_EXIT_CODE
> Task
> :sdks:python:test-suites:portable:py38:testUberJarFlinkRunnerSaveMainSession
> FAILED
FAILURE: Build completed with 3 failures.
1: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 383
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py38:testJavaJarCreatorFlinkRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 411
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py38:testUberJarFlinkRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================
3: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 411
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py38:testUberJarFlinkRunnerSaveMainSession'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.6.2/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 10m 40s
149 actionable tasks: 99 executed, 46 from cache, 4 up-to-date
Publishing build scan...
https://ge.apache.org/s/zr67ftsmr3fwy
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
