See
<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/4232/display/redirect?page=changes>
Changes:
[noreply] [BEAM-13079] Updates cross-language transform URNs to use the new
------------------------------------------
[...truncated 408.35 KB...]
import apache_beam as beam
import logging
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
from apache_beam.testing.util import assert_that
from apache_beam.testing.util import equal_to
from apache_beam.transforms import Create
from apache_beam.transforms import Map
logging.basicConfig(level=logging.INFO)
pipeline_options = PipelineOptions()
pipeline = beam.Pipeline(options=pipeline_options)
if pipeline_options.view_as(SetupOptions).save_main_session:
# To test that our main session is getting plumbed through artifact staging
# correctly, create a global variable. If the main session is not plumbed
# through properly, global_var will be undefined and the pipeline will fail.
global_var = 1
pcoll = (pipeline
| Create([0, 1, 2])
| Map(lambda x: x + global_var))
else:
pcoll = (pipeline
| Create([0, 1, 2])
| Map(lambda x: x + 1))
assert_that(pcoll, equal_to([1, 2, 3]))
result = pipeline.run()
result.wait_until_finish()
"
if [[ "$SAVE_MAIN_SESSION" -eq 0 ]]; then
(python -c "$PIPELINE_PY" \
--runner FlinkRunner \
--flink_job_server_jar "$FLINK_JOB_SERVER_JAR" \
--parallelism 1 \
--environment_type DOCKER \
--environment_options "docker_container_image=$PYTHON_CONTAINER_IMAGE" \
--flink_master "localhost:$FLINK_PORT" \
--flink_submit_uber_jar \
) || TEST_EXIT_CODE=$? # don't fail fast here; clean up before exiting
else
(python -c "$PIPELINE_PY" \
--runner FlinkRunner \
--flink_job_server_jar "$FLINK_JOB_SERVER_JAR" \
--parallelism 1 \
--environment_type DOCKER \
--environment_options "docker_container_image=$PYTHON_CONTAINER_IMAGE" \
--flink_master "localhost:$FLINK_PORT" \
--flink_submit_uber_jar \
--save_main_session
) || TEST_EXIT_CODE=$? # don't fail fast here; clean up before exiting
fi
Oct 20, 2021 6:42:23 AM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet
INFO: The configuration option taskmanager.cpu.cores required for local
execution is not set, setting it to the maximal possible value.
Oct 20, 2021 6:42:23 AM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet
INFO: The configuration option taskmanager.memory.task.heap.size required for
local execution is not set, setting it to the maximal possible value.
Oct 20, 2021 6:42:23 AM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet
INFO: The configuration option taskmanager.memory.task.off-heap.size required
for local execution is not set, setting it to the maximal possible value.
Oct 20, 2021 6:42:23 AM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet
INFO: The configuration option taskmanager.memory.network.min required for
local execution is not set, setting it to its default value 64 mb.
Oct 20, 2021 6:42:23 AM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet
INFO: The configuration option taskmanager.memory.network.max required for
local execution is not set, setting it to its default value 64 mb.
Oct 20, 2021 6:42:23 AM
org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils
setConfigOptionToDefaultIfNotSet
INFO: The configuration option taskmanager.memory.managed.size required for
local execution is not set, setting it to its default value 128 mb.
Oct 20, 2021 6:42:23 AM org.apache.flink.runtime.minicluster.MiniCluster start
INFO: Starting Flink Mini Cluster
Oct 20, 2021 6:42:23 AM org.apache.flink.runtime.minicluster.MiniCluster start
INFO: Starting Metrics Registry
Oct 20, 2021 6:42:24 AM org.apache.flink.runtime.metrics.MetricRegistryImpl
<init>
INFO: No metrics reporter configured, no metrics will be exposed/reported.
Oct 20, 2021 6:42:24 AM org.apache.flink.runtime.minicluster.MiniCluster start
INFO: Starting RPC Service(s)
Oct 20, 2021 6:42:24 AM
org.apache.flink.runtime.clusterframework.BootstrapTools startLocalActorSystem
INFO: Trying to start local actor system
WARNING:root:Make sure that locally built Python SDK docker image has Python
3.6 interpreter.
INFO:root:Default Python SDK image for environment is
apache/beam_python3.6_sdk:2.35.0.dev
INFO:root:Using provided Python SDK container image:
apache/beam_python3.6_sdk:2.35.0.dev
INFO:root:Python SDK container image set to
"apache/beam_python3.6_sdk:2.35.0.dev" for Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function pack_combiners at 0x7fe0ef755840> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function lift_combiners at 0x7fe0ef7558c8> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sort_stages at 0x7fe0ef756048> ====================
INFO:apache_beam.runners.portability.flink_runner:Adding HTTP protocol scheme
to flink_master parameter: http://localhost:33669
INFO:apache_beam.runners.portability.abstract_job_service:Artifact server
started on port 42025
INFO:apache_beam.runners.portability.abstract_job_service:Running job
'job-9545d642-1236-4cd4-baaa-26cee688d1d9'
Oct 20, 2021 6:42:26 AM akka.event.slf4j.Slf4jLogger$$anonfun$receive$1
applyOrElse
INFO: Slf4jLogger started
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/urllib3/connection.py";,>
line 175, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/urllib3/util/connection.py";,>
line 96, in create_connection
raise err
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/urllib3/util/connection.py";,>
line 86, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/urllib3/connectionpool.py";,>
line 706, in urlopen
chunked=chunked,
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/urllib3/connectionpool.py";,>
line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/urllib3/connection.py";,>
line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/lib/python3.6/http/client.py", line 964, in send
self.connect()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/urllib3/connection.py";,>
line 205, in connect
conn = self._new_conn()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/urllib3/connection.py";,>
line 187, in _new_conn
self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection
object at 0x7fe0ef3c3b38>: Failed to establish a new connection: [Errno 111]
Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/requests/adapters.py";,>
line 449, in send
timeout=timeout
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/urllib3/connectionpool.py";,>
line 756, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/urllib3/util/retry.py";,>
line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost',
port=33669): Max retries exceeded with url: /v1/jars/upload (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7fe0ef3c3b38>: Failed to establish a new connection: [Errno 111] Connection
refused',))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 31, in <module>
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/pipeline.py";,>
line 573, in run
return self.runner.run_pipeline(self, self._options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/flink_runner.py";,>
line 45, in run_pipeline
return super().run_pipeline(pipeline, options)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,>
line 440, in run_pipeline
job_service_handle.submit(proto_pipeline)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,>
line 115, in submit
return self.run(prepare_response.preparation_id)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/portable_runner.py";,>
line 242, in run
beam_job_api_pb2.RunJobRequest(preparation_id=preparation_id))
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/abstract_job_service.py";,>
line 120, in Run
self._jobs[job_id].run()
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py";,>
line 154, in run
files={'jarfile': ('beam.jar', jar_file)})['filename'].split('/')[-1]
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py";,>
line 142, in post
return self.request(requests.post, path, **kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py";,>
line 130, in request
response = method(url, **kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/requests/api.py";,>
line 117, in post
return request('post', url, data=data, json=json, **kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/requests/api.py";,>
line 61, in request
return session.request(method=method, url=url, **kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/requests/sessions.py";,>
line 542, in request
resp = self.send(prep, **send_kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/requests/sessions.py";,>
line 655, in send
r = adapter.send(request, **kwargs)
File
"<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/build/gradleenv/2022703440/lib/python3.6/site-packages/requests/adapters.py";,>
line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost',
port=33669): Max retries exceeded with url: /v1/jars/upload (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7fe0ef3c3b38>: Failed to establish a new connection: [Errno 111] Connection
refused',))
kill %1 || echo "Failed to shut down Flink mini cluster"
rm -rf "$ENV_DIR"
if [[ "$TEST_EXIT_CODE" -eq 0 ]]; then
echo ">>> SUCCESS"
else
echo ">>> FAILURE"
fi
exit $TEST_EXIT_CODE
> Task
> :sdks:python:test-suites:portable:py36:testUberJarFlinkRunnerSaveMainSession
> FAILED
FAILURE: Build completed with 4 failures.
1: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 266
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py37:testUberJarFlinkRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
2: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 266
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py36:testUberJarFlinkRunner'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
3: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 266
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py37:testUberJarFlinkRunnerSaveMainSession'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
4: Task failed with an exception.
-----------
* Where:
Script
'<https://ci-beam.apache.org/job/beam_PostCommit_PortableJar_Flink/ws/src/sdks/python/test-suites/portable/common.gradle'>
line: 266
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py36:testUberJarFlinkRunnerSaveMainSession'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
==============================================================================
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.9.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 41m 57s
150 actionable tasks: 106 executed, 41 from cache, 3 up-to-date
Publishing build scan...
https://gradle.com/s/stibzi4nxnwj6
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
