See
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/1982/display/redirect?page=changes>
Changes:
[Kyle Weaver] [BEAM-12976] Log projection pushdown optimizations.
[benjamin.gonzalez] [BEAM-12572] Change jobs to run as cron jobs
[alexander.zhuravlev] [BEAM-13820] Changed color of delete icon in pipeline
options dropdown,
[noreply] [BEAM-13193] Aggregates fn api outbound data/timers of different
[noreply] [BEAM-13767] Migrate a bundle of grade tasks to use configuration
[noreply] Merge pull request #16653 from [BEAM-12164]: Add integration tests for
[noreply] Merge pull request #16728 from [BEAM-13823] Update docs for
SnowflakeIO
[noreply] Merge pull request #16660 from [BEAM-13771][Playground] Send multifile
[noreply] Merge pull request #16646 from [BEAM-13643][Playground] Setup running
[noreply] [BEAM-13015] Add state caching benchmark and move benchmarks to their
[noreply] [BEAM-13419] Check for initialization in dataflow runner (#16765)
[noreply] Merge pull request #16701 from [BEAM-13786] [Playground] [Bugfix]
Update
[noreply] Merge pull request #16754 from [BEAM-13838][Playground] Add logs in
case
[noreply] [BEAM-13293] consistent naming for expansion service address and flag
[noreply] Merge pull request #16700 from [BEAM-13790][Playground] Change logic
of
[noreply] [BEAM-13830] update dependency for debeziumio expansion service
(#16743)
[noreply] [BEAM-13761] consistent namings for expansion address in Debezium IO
[noreply] [BEAM-13806] Shutting down SchemaIO expansion services from Go VR
[noreply] [release-2.36.0] Update website/changelog for release 2.36.0 (#16627)
------------------------------------------
[...truncated 251.66 KB...]
}, {
"urn": "beam:metric:pardo_execution_time:process_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/read/ref_PCollection_PCollection_5:0"
}
}, {
"urn": "beam:metric:pardo_execution_time:start_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/read/ref_PCollection_PCollection_5:0"
}
}, {
"urn": "beam:metric:ptransform_execution_time:total_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "transform_0"
}
}, {
"urn": "beam:metric:ptransform_execution_time:total_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "transform_3"
}
}, {
"urn": "beam:metric:ptransform_execution_time:total_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/write/ref_PCollection_PCollection_9:0"
}
}, {
"urn": "beam:metric:data_channel:read_index:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "bg==",
"labels": {
"PTRANSFORM": "fn/read/ref_PCollection_PCollection_5:0"
}
}, {
"urn": "beam:metric:element_count:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "bg==",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_5"
}
}, {
"urn": "beam:metric:element_count:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "bg==",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_7"
}
}, {
"urn": "beam:metric:pardo_execution_time:process_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "transform_2"
}
}, {
"urn": "beam:metric:pardo_execution_time:process_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "transform_1"
}
}, {
"urn": "beam:metric:pardo_execution_time:process_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "transform_0"
}
}, {
"urn": "beam:metric:pardo_execution_time:start_bundle_msecs:v1",
"type": "beam:metrics:sum_int64:v1",
"payload": "AA==",
"labels": {
"PTRANSFORM": "fn/write/ref_PCollection_PCollection_9:0"
}
}, {
"urn": "beam:metric:sampled_byte_size:v1",
"type": "beam:metrics:distribution_int64:v1",
"payload": "JNkIHh8=",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_5"
}
}, {
"urn": "beam:metric:sampled_byte_size:v1",
"type": "beam:metrics:distribution_int64:v1",
"payload": "JYYEDg4=",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_7"
}
}, {
"urn": "beam:metric:sampled_byte_size:v1",
"type": "beam:metrics:distribution_int64:v1",
"payload": "JbwFEhM=",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_6"
}
}, {
"urn": "beam:metric:sampled_byte_size:v1",
"type": "beam:metrics:distribution_int64:v1",
"payload": "I9MEERE=",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_9"
}
}, {
"urn": "beam:metric:sampled_byte_size:v1",
"type": "beam:metrics:distribution_int64:v1",
"payload": "JpQEDg4=",
"labels": {
"PCOLLECTION": "ref_PCollection_PCollection_8"
}
}]
}
}
Feb 08, 2022 12:38:13 AM
org.apache.flink.runtime.io.disk.FileChannelManagerImpl lambda$getFileCloser$0
INFO: FileChannelManager removed spill file directory
/tmp/flink-io-b894271f-242f-4111-a3a4-c997c7a1ac40
Feb 08, 2022 12:38:13 AM
org.apache.flink.runtime.io.network.NettyShuffleEnvironment close
INFO: Shutting down the network environment and its components.
Feb 08, 2022 12:38:13 AM
org.apache.flink.runtime.io.disk.FileChannelManagerImpl lambda$getFileCloser$0
INFO: FileChannelManager removed spill file directory
/tmp/flink-netty-shuffle-baa3e938-9b2e-4b85-ba1d-1393061fc555
Feb 08, 2022 12:38:13 AM org.apache.flink.runtime.taskexecutor.KvStateService
shutdown
INFO: Shutting down the kvState service and its components.
Feb 08, 2022 12:38:13 AM
org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService stop
INFO: Stop job leader service.
Feb 08, 2022 12:38:13 AM org.apache.flink.runtime.filecache.FileCache shutdown
INFO: removed file cache directory
/tmp/flink-dist-cache-66b2c406-27ef-468c-9353-d1fedabddd2c
Feb 08, 2022 12:38:13 AM org.apache.flink.runtime.webmonitor.WebMonitorEndpoint
lambda$shutDownInternal$5
INFO: Removing cache directory /tmp/flink-web-ui
Feb 08, 2022 12:38:13 AM org.apache.flink.runtime.rest.RestServerEndpoint
lambda$closeAsync$1
INFO: Shut down complete.
Feb 08, 2022 12:38:13 AM
org.apache.flink.runtime.resourcemanager.ResourceManager deregisterApplication
INFO: Shut down cluster because application is in CANCELED, diagnostics
DispatcherResourceManagerComponent has been closed..
Feb 08, 2022 12:38:13 AM
org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
closeAsyncInternal
INFO: Closing components.
Feb 08, 2022 12:38:13 AM
org.apache.flink.runtime.dispatcher.runner.AbstractDispatcherLeaderProcess
closeInternal
INFO: Stopping SessionDispatcherLeaderProcess.
Feb 08, 2022 12:38:13 AM org.apache.flink.runtime.dispatcher.Dispatcher onStop
INFO: Stopping dispatcher akka://flink/user/rpc/dispatcher_38.
Feb 08, 2022 12:38:13 AM
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
close
INFO: Closing the slot manager.
Feb 08, 2022 12:38:13 AM org.apache.flink.runtime.dispatcher.Dispatcher
terminateRunningJobs
INFO: Stopping all currently running jobs of dispatcher
akka://flink/user/rpc/dispatcher_38.
Feb 08, 2022 12:38:13 AM
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
suspend
INFO: Suspending the slot manager.
Feb 08, 2022 12:38:13 AM
org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool
releaseSlots
INFO: Releasing slot [5f33d50714a1b47e53c711316a9440c5].
Feb 08, 2022 12:38:13 AM org.apache.flink.runtime.jobmaster.JobMaster
dissolveResourceManagerConnection
INFO: Close ResourceManager connection 2a4e6dc60de516e55c08b5cf62c68480:
Stopping JobMaster for job
test_flink_metrics_1644280691.1291225(989c628a9d2f1e0f72b9d4b0facd02bc)..
------------------------------ Captured log call -------------------------------
WARNING root:environments.py:374 Make sure that locally built Python SDK
docker image has Python 3.7 interpreter.
WARNING apache_beam.options.pipeline_options:pipeline_options.py:324
Discarding invalid overrides: {'max_bundle_time_millis': 95130590130}
--------------------------- Captured stderr teardown ---------------------------
Feb 08, 2022 12:38:13 AM org.apache.flink.runtime.taskexecutor.TaskExecutor
handleOnStopException
INFO: Stopped TaskExecutor akka://flink/user/rpc/taskmanager_36.
Feb 08, 2022 12:38:13 AM org.apache.flink.runtime.dispatcher.Dispatcher
lambda$onStop$0
INFO: Stopped dispatcher akka://flink/user/rpc/dispatcher_38.
Feb 08, 2022 12:38:13 AM org.apache.flink.runtime.rpc.akka.AkkaRpcService
stopService
INFO: Stopping Akka RPC service.
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/pytest_flink-runner-test.xml>
-
============== 1 failed, 130 passed, 40 skipped in 531.75 seconds ==============
[31mERROR: InvocationError for command
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/scripts/pytest_validates_runner.sh>
flink-runner-test apache_beam/runners/portability/flink_runner_test.py
'--flink_job_server_jar=<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/runners/flink/1.13/job-server/build/libs/beam-runners-flink-1.13-job-server-2.37.0-SNAPSHOT.jar>
--environment_type=LOOPBACK' (exited with code 1)
[0m[1mflink-runner-test run-test-post: commands[0] | bash
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py37/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m___________________________________ summary
____________________________________
[31mERROR: flink-runner-test: commands failed
[0m
> Task :sdks:python:test-suites:portable:py37:flinkCompatibilityMatrixLOOPBACK
> FAILED
> Task :sdks:python:test-suites:portable:py38:flinkCompatibilityMatrixLOOPBACK
[1mflink-runner-test create:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/target/.tox-flink-runner-test/flink-runner-test>
[0m[1mflink-runner-test installdeps: -rbuild-requirements.txt
[0m[1mflink-runner-test inst:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/target/.tox-flink-runner-test/.tmp/package/1/apache-beam.tar.gz>
[0m[1mflink-runner-test installed: apache-beam @
file://<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/target/.tox-flink-runner-test/.tmp/package/1/apache-beam.tar.gz,atomicwrites==1.4.0,attrs==21.4.0,certifi==2021.10.8,charset-normalizer==2.0.11,cloudpickle==2.0.0,crcmod==1.7,deprecation==2.1.0,dill==0.3.1.1,distlib==0.3.1,docker==5.0.3,docopt==0.6.2,execnet==1.9.0,fastavro==1.4.9,freezegun==1.1.0,greenlet==1.1.2,grpcio==1.43.0,grpcio-tools==1.37.0,hdfs==2.6.0,httplib2==0.19.1,idna==3.3,mock==2.0.0,more-itertools==8.12.0,mypy-protobuf==1.18,numpy==1.21.5,oauth2client==4.1.3,orjson==3.6.6,packaging==21.3,pandas==1.4.0,parameterized==0.7.5,pbr==5.8.1,pluggy==0.13.1,proto-plus==1.20.0,protobuf==3.19.4,psycopg2-binary==2.9.3,py==1.11.0,pyarrow==6.0.1,pyasn1==0.4.8,pyasn1-modules==0.2.8,pydot==1.4.2,PyHamcrest==1.10.1,pymongo==3.12.3,pyparsing==2.4.7,pytest==4.6.11,pytest-forked==1.4.0,pytest-timeout==1.4.2,pytest-xdist==1.34.0,python-dateutil==2.8.2,pytz==2021.3,PyYAML==6.0,requests==2.27.1,requests-mock==1.9.3,rsa==4.8,six==1.16.0,SQLAlchemy==1.4.31,tenacity==5.1.5,testcontainers==3.4.2,typing_extensions==4.0.1,urllib3==1.26.8,wcwidth==0.2.5,websocket-client==1.2.3,wrapt==1.13.3>
[0m[1mflink-runner-test run-test-pre: PYTHONHASHSEED='1506535809'
[0m[1mflink-runner-test run-test-pre: commands[0] | python --version
[0mPython 3.8.10
[1mflink-runner-test run-test-pre: commands[1] | pip --version
[0mpip 21.3.1 from
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/target/.tox-flink-runner-test/flink-runner-test/lib/python3.8/site-packages/pip>
(python 3.8)
[1mflink-runner-test run-test-pre: commands[2] | pip check
[0mNo broken requirements found.
[1mflink-runner-test run-test-pre: commands[3] | bash
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m[1mflink-runner-test run-test: commands[0] |
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/scripts/pytest_validates_runner.sh>
flink-runner-test
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/apache_beam/runners/portability/flink_runner_test.py>
'--flink_job_server_jar=<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/runners/flink/1.13/job-server/build/libs/beam-runners-flink-1.13-job-server-2.37.0-SNAPSHOT.jar>
--environment_type=LOOPBACK'
[0m============================= test session starts
==============================
platform linux -- Python 3.8.10, pytest-4.6.11, py-1.11.0, pluggy-0.13.1
cachedir: target/.tox-flink-runner-test/flink-runner-test/.pytest_cache
rootdir:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python,>
inifile: pytest.ini
plugins: xdist-1.34.0, timeout-1.4.2, forked-1.4.0, requests-mock-1.9.3
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 171 items
apache_beam/runners/portability/flink_runner_test.py .s...ss...........s [ 11%]
....s.....s...........s..ss.....ss.s...s...ss..sss......s....s.....s.... [ 53%]
.......s..ss.....ssss...s...ss...........s....s.....s...........s..ss... [ 95%]
..ss.s.. [100%]
=============================== warnings summary ===============================
target/.tox-flink-runner-test/flink-runner-test/lib/python3.8/site-packages/tenacity/_asyncio.py:42
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/target/.tox-flink-runner-test/flink-runner-test/lib/python3.8/site-packages/tenacity/_asyncio.py>:42:
DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use
"async def" instead
def call(self, fn, *args, **kwargs):
-- Docs: https://docs.pytest.org/en/latest/warnings.html
- generated xml file:
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/pytest_flink-runner-test.xml>
-
============= 131 passed, 40 skipped, 1 warnings in 496.05 seconds =============
[1mflink-runner-test run-test-post: commands[0] | bash
<https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/ws/src/sdks/python/test-suites/portable/py38/build/srcs/sdks/python/scripts/run_tox_cleanup.sh>
[0m___________________________________ summary
____________________________________
[32m flink-runner-test: commands succeeded
[0m[32m congratulations :)
[0m
> Task :sdks:python:test-suites:portable:py38:flinkValidatesRunner
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task
':sdks:python:test-suites:portable:py37:flinkCompatibilityMatrixLOOPBACK'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 8.0.
You can use '--warning-mode all' to show the individual deprecation warnings
and determine if they come from your own scripts or plugins.
See
https://docs.gradle.org/7.3.2/userguide/command_line_interface.html#sec:command_line_warnings
Execution optimizations have been disabled for 1 invalid unit(s) of work during
this build to ensure correctness.
Please consult deprecation warnings for more details.
BUILD FAILED in 29m 42s
151 actionable tasks: 106 executed, 41 from cache, 4 up-to-date
Publishing build scan...
https://gradle.com/s/xjjbpvzh2dpva
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
