See <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/1169/display/redirect?page=changes>
Changes: [heejong] [BEAM-12838] Update artifact local path for DataflowRunner Java [Valentyn Tymofieiev] Disable Kafka perf tests. [heejong] separate into resolveArtifacts method [heejong] add test [Andrew Pilloud] [BEAM-12850] Calcite drops empty Calc now [Andrew Pilloud] [BEAM-12853] VALUES produces a UNION, window can't be set afterwards [Andrew Pilloud] [BEAM-12852] Revert BigTable changes, just cast to bigint [heejong] update [heejong] fix formatting [Andrew Pilloud] [BEAM-12851] Map output table names [Luke Cwik] [BEAM-12802] Define a prefetchable iterator and iterable and utility ------------------------------------------ [...truncated 937.39 KB...] _________________ ValidateRunnerXlangTest.test_combine_per_key _________________ [gw6] linux -- Python 3.8.5 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/bin/python3.8> self = <apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest testMethod=test_combine_per_key> test_pipeline = None @pytest.mark.xlang_transforms def test_combine_per_key(self, test_pipeline=None): > CrossLanguageTestPipelines().run_combine_per_key( test_pipeline or self.create_pipeline()) apache_beam/transforms/validate_runner_xlang_test.py:278: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ apache_beam/transforms/validate_runner_xlang_test.py:201: in run_combine_per_key assert_that(res, equal_to([('a', 3), ('b', 3)])) apache_beam/pipeline.py:586: in __exit__ self.result = self.run() apache_beam/testing/test_pipeline.py:112: in run result = super(TestPipeline, self).run( apache_beam/pipeline.py:565: in run return self.runner.run_pipeline(self, self._options) apache_beam/runners/dataflow/test_dataflow_runner.py:65: in run_pipeline self.result.wait_until_finish(duration=wait_duration) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <DataflowPipelineResult <Job clientRequestId: '20210908003443001807-9101' createTime: '2021-09-08T00:34:49.533188Z' ...021-09-08T00:34:49.533188Z' steps: [] tempFiles: [] type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> at 0x7fabf6003970> duration = None def wait_until_finish(self, duration=None): if not self.is_in_terminal_state(): if not self.has_job: raise IOError('Failed to get the Dataflow job id.') thread = threading.Thread( target=DataflowRunner.poll_for_job_completion, args=(self._runner, self, duration)) # Mark the thread as a daemon thread so a keyboard interrupt on the main # thread will terminate everything. This is also the reason we will not # use thread.join() to wait for the polling thread. thread.daemon = True thread.start() while thread.is_alive(): time.sleep(5.0) # TODO: Merge the termination code in poll_for_job_completion and # is_in_terminal_state. terminated = self.is_in_terminal_state() assert duration or terminated, ( 'Job did not reach to a terminal state after waiting indefinitely.') if terminated and self.state != PipelineState.DONE: # TODO(BEAM-1290): Consider converting this to an error log based on # theresolution of the issue. > raise DataflowRuntimeException( 'Dataflow pipeline failed. State: %s, Error:\n%s' % (self.state, getattr(self._runner, 'last_error_msg', None)), self) E apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error: E Workflow failed. apache_beam/runners/dataflow/dataflow_runner.py:1632: DataflowRuntimeException ------------------------------ Captured log call ------------------------------- INFO apache_beam.runners.portability.stager:stager.py:300 Copying Beam SDK "<https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/build/apache-beam.tar.gz";> to staging location. WARNING root:environments.py:371 Make sure that locally built Python SDK docker image has Python 3.8 interpreter. INFO root:environments.py:380 Default Python SDK image for environment is apache/beam_python3.8_sdk:2.34.0.dev INFO root:environments.py:295 Using provided Python SDK container image: gcr.io/cloud-dataflow/v1beta3/python38-fnapi:beam-master-20210809 INFO root:environments.py:302 Python SDK container image set to "gcr.io/cloud-dataflow/v1beta3/python38-fnapi:beam-master-20210809" for Docker environment INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function pack_combiners at 0x7fabbb21faf0> ==================== INFO apache_beam.runners.portability.fn_api_runner.translations:translations.py:650 ==================== <function sort_stages at 0x7fabbb21e310> ==================== INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:452 Defaulting to the temp_location as staging_location: gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494 INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/icedtea-sound-Bdoi2wYa757-fzq5vconCy4SSQ22ZaOq7yuC98fKPs8.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/jaccess-CMbK-IOdQPLKHEqCuDnE-yBk-VpbtVT-hgjbHRUGO78.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/localedata-ae5Z0L6ak4922fztWeWy7ajiWXdG3ubNrwerJRFoFj0.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/nashorn-XHtz_UehGpYcLTOrATrTnMNVUgEVa_ttoWkPxnVfqTo.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/cldrdata-k07I6K9W3X5KTQbcDIEsqM0LXyM18f0eR6IaJw-P_xk.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/dnsns-RGhCDg3GVOQVC2r6ka2N0hmI4eqQH6VobuoAnQ74MnE.jar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/beam-sdks-java-testing-expansion-service-testExpansionService-2.34.0-SNAPSHOT-88DKifGQHO779sP9hBu0-rb5NtOcRBI5jVLPY-DPRzw.jar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/beam-sdks-java-testing-expansion-service-testExpansionService-2.34.0-SNAPSHOT-88DKifGQHO779sP9hBu0-rb5NtOcRBI5jVLPY-DPRzw.jar in 4 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/dataflow_python_sdk.tar... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/dataflow_python_sdk.tar in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:639 Starting GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/pipeline.pb... INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:655 Completed GCS upload to gs://dataflow-staging-us-central1-77b801c0838aee13391c0d1885860494/beamapp-jenkins-0908003443-000329.1631061283.000754/pipeline.pb in 0 seconds. INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:819 Create job: <Job clientRequestId: '20210908003443001807-9101' createTime: '2021-09-08T00:34:49.533188Z' currentStateTime: '1970-01-01T00:00:00Z' id: '2021-09-07_17_34_48-2353990166838405452' location: 'us-central1' name: 'beamapp-jenkins-0908003443-000329' projectId: 'apache-beam-testing' stageStates: [] startTime: '2021-09-08T00:34:49.533188Z' steps: [] tempFiles: [] type: TypeValueValuesEnum(JOB_TYPE_BATCH, 1)> INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:821 Created job with id: [2021-09-07_17_34_48-2353990166838405452] INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:822 Submitted job: 2021-09-07_17_34_48-2353990166838405452 INFO apache_beam.runners.dataflow.internal.apiclient:apiclient.py:823 To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobs/us-central1/2021-09-07_17_34_48-2353990166838405452?project=apache-beam-testing INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-09-07_17_34_48-2353990166838405452 is in state JOB_STATE_RUNNING INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:52.258Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2021-09-07_17_34_48-2353990166838405452. The number of workers will be between 1 and 1000. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:52.406Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2021-09-07_17_34_48-2353990166838405452. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:56.753Z: JOB_MESSAGE_BASIC: Worker configuration: e2-standard-2 in us-central1-a. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:57.625Z: JOB_MESSAGE_DETAILED: Expanding SplittableParDo operations into optimizable parts. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:57.668Z: JOB_MESSAGE_DETAILED: Expanding CollectionToSingleton operations into optimizable parts. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:57.743Z: JOB_MESSAGE_DETAILED: Expanding CoGroupByKey operations into optimizable parts. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:57.779Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step assert_that/Group/_CoGBKImpl/GroupByKey: GroupByKey not followed by a combiner. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:57.856Z: JOB_MESSAGE_DEBUG: Combiner lifting skipped for step Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey: GroupByKey not followed by a combiner. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:57.919Z: JOB_MESSAGE_DETAILED: Expanding GroupByKey operations into optimizable parts. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:57.951Z: JOB_MESSAGE_DEBUG: Annotating graph with Autotuner information. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.003Z: JOB_MESSAGE_DETAILED: Fusing adjacent ParDo, Read, Write, and Flatten operations INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.048Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_assert_that-Group-_CoGBKImpl-Flatten_27 INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.080Z: JOB_MESSAGE_DEBUG: Inserted coder converter before flatten ref_AppliedPTransform_assert_that-Group-_CoGBKImpl-Flatten_27 INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.114Z: JOB_MESSAGE_DETAILED: Unzipping flatten ref_AppliedPTransform_assert_that-Group-_CoGBKImpl-Flatten_27 for input ref_AppliedPTransform_assert_that-Group-_CoGBKImpl-Tag-0-_25.None-post28 INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.148Z: JOB_MESSAGE_DETAILED: Fusing unzipped copy of assert_that/Group/_CoGBKImpl/GroupByKey/Write, through flatten assert_that/Group/_CoGBKImpl/Flatten, into producer assert_that/Group/_CoGBKImpl/Flatten/InputIdentity INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.194Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/MapTuple(collect_values) into assert_that/Group/_CoGBKImpl/GroupByKey/Read INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.229Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/RestoreTags into assert_that/Group/_CoGBKImpl/MapTuple(collect_values) INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.267Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Unkey into assert_that/Group/RestoreTags INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.302Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Match into assert_that/Unkey INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.354Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/GroupByKey/Write into assert_that/Group/_CoGBKImpl/Flatten/InputIdentity INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.410Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/FlatMap(<lambda at core.py:2965>) into Create/Impulse INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.453Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/AddRandomKeys into Create/FlatMap(<lambda at core.py:2965>) INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.498Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps) into Create/MaybeReshuffle/Reshuffle/AddRandomKeys INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.535Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps) INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.574Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.607Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Read INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.648Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.682Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys into Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps) INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.715Z: JOB_MESSAGE_DETAILED: Fusing consumer Create/Map(decode) into Create/MaybeReshuffle/Reshuffle/RemoveRandomKeys INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.750Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:compk)/GroupByKey+ExternalTransform(beam:transforms:xlang:test:compk)/Combine.GroupedValues/Partial into Create/Map(decode) INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.787Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:compk)/GroupByKey/Write into ExternalTransform(beam:transforms:xlang:test:compk)/GroupByKey+ExternalTransform(beam:transforms:xlang:test:compk)/Combine.GroupedValues/Partial INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.824Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:compk)/Combine.GroupedValues into ExternalTransform(beam:transforms:xlang:test:compk)/GroupByKey/Read INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.875Z: JOB_MESSAGE_DETAILED: Fusing consumer ExternalTransform(beam:transforms:xlang:test:compk)/Combine.GroupedValues/Extract into ExternalTransform(beam:transforms:xlang:test:compk)/Combine.GroupedValues INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.925Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/WindowInto(WindowIntoFn) into ExternalTransform(beam:transforms:xlang:test:compk)/Combine.GroupedValues/Extract INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.965Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/FlatMap(<lambda at core.py:2965>) into assert_that/Create/Impulse INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:58.998Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Create/Map(decode) into assert_that/Create/FlatMap(<lambda at core.py:2965>) INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.053Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/Tag[0] into assert_that/Create/Map(decode) INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.101Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/Flatten/InputIdentity into assert_that/Group/_CoGBKImpl/Tag[0] INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.148Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/ToVoidKey into assert_that/WindowInto(WindowIntoFn) INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.202Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/Tag[1] into assert_that/ToVoidKey INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.254Z: JOB_MESSAGE_DETAILED: Fusing consumer assert_that/Group/_CoGBKImpl/Flatten/InputIdentity into assert_that/Group/_CoGBKImpl/Tag[1] INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.293Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.337Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.382Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.418Z: JOB_MESSAGE_DEBUG: Assigning stage ids. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.576Z: JOB_MESSAGE_DEBUG: Executing wait step start41 INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.666Z: JOB_MESSAGE_BASIC: Executing operation ExternalTransform(beam:transforms:xlang:test:compk)/GroupByKey/Create INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.698Z: JOB_MESSAGE_BASIC: Executing operation Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.720Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.729Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/_CoGBKImpl/GroupByKey/Create INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.769Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-a... INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.812Z: JOB_MESSAGE_BASIC: Finished operation ExternalTransform(beam:transforms:xlang:test:compk)/GroupByKey/Create INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.827Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/_CoGBKImpl/GroupByKey/Create INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.827Z: JOB_MESSAGE_BASIC: Finished operation Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Create INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.905Z: JOB_MESSAGE_DEBUG: Value "ExternalTransform(beam:transforms:xlang:test:compk)/GroupByKey/Session" materialized. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.954Z: JOB_MESSAGE_DEBUG: Value "Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Session" materialized. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:34:59.992Z: JOB_MESSAGE_DEBUG: Value "assert_that/Group/_CoGBKImpl/GroupByKey/Session" materialized. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:35:00.041Z: JOB_MESSAGE_BASIC: Executing operation Create/Impulse+Create/FlatMap(<lambda at core.py:2965>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:35:00.077Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:2965>)+assert_that/Create/Map(decode)+assert_that/Group/_CoGBKImpl/Tag[0]+assert_that/Group/_CoGBKImpl/Flatten/InputIdentity+assert_that/Group/_CoGBKImpl/GroupByKey/Write INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:35:16.848Z: JOB_MESSAGE_BASIC: Your project already contains 100 Dataflow-created metric descriptors, so new user metrics of the form custom.googleapis.com/* will not be created. However, all user metrics are also available in the metric dataflow.googleapis.com/job/user_counter. If you rely on the custom metrics, you can delete old / unused metric descriptors. See https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.list and https://developers.google.com/apis-explorer/#p/monitoring/v3/monitoring.projects.metricDescriptors.delete INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:35:47.044Z: JOB_MESSAGE_ERROR: Startup of the worker pool in zone us-central1-a failed to bring up any of the desired 1 workers. QUOTA_EXCEEDED: Quota 'CPUS' exceeded. Limit: 2000.0 in region us-central1. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:35:47.078Z: JOB_MESSAGE_ERROR: Workflow failed. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:35:47.136Z: JOB_MESSAGE_BASIC: Finished operation Create/Impulse+Create/FlatMap(<lambda at core.py:2965>)+Create/MaybeReshuffle/Reshuffle/AddRandomKeys+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/Map(reify_timestamps)+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Reify+Create/MaybeReshuffle/Reshuffle/ReshufflePerKey/GroupByKey/Write INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:35:47.136Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Create/Impulse+assert_that/Create/FlatMap(<lambda at core.py:2965>)+assert_that/Create/Map(decode)+assert_that/Group/_CoGBKImpl/Tag[0]+assert_that/Group/_CoGBKImpl/Flatten/InputIdentity+assert_that/Group/_CoGBKImpl/GroupByKey/Write INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:35:47.223Z: JOB_MESSAGE_DETAILED: Cleaning up. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:35:47.392Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:35:47.444Z: JOB_MESSAGE_BASIC: Stopping worker pool... INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:36:13.903Z: JOB_MESSAGE_BASIC: Worker pool stopped. INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:236 2021-09-08T00:36:13.945Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... INFO apache_beam.runners.dataflow.dataflow_runner:dataflow_runner.py:191 Job 2021-09-07_17_34_48-2353990166838405452 is in state JOB_STATE_FAILED =============================== warnings summary =============================== apache_beam/io/filesystems_test.py:54 apache_beam/io/filesystems_test.py:54 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:54: DeprecationWarning: invalid escape sequence \c self.assertIsNone(FileSystems.get_scheme('c:\\abc\cdf')) # pylint: disable=anomalous-backslash-in-string apache_beam/io/filesystems_test.py:62 apache_beam/io/filesystems_test.py:62 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/apache_beam/io/filesystems_test.py>:62: DeprecationWarning: invalid escape sequence \d self.assertTrue(isinstance(FileSystems.get_filesystem('c:\\abc\def'), # pylint: disable=anomalous-backslash-in-string <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42 <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/build/gradleenv/1922375555/lib/python3.8/site-packages/tenacity/_asyncio.py>:42: DeprecationWarning: "@coroutine" decorator is deprecated since Python 3.8, use "async def" instead def call(self, fn, *args, **kwargs): -- Docs: https://docs.pytest.org/en/latest/warnings.html - generated xml file: <https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/ws/src/sdks/python/pytest_xlangValidateRunner.xml> - ============== 4 failed, 4 passed, 12 warnings in 670.50 seconds =============== > Task > :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava > FAILED > Task :runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerCleanup Stopping expansion service pid: 27244. Stopping expansion service pid: 27247. > Task :runners:google-cloud-dataflow-java:cleanUpDockerImages Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210908002543 Untagged: us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:7dc20859932e66fadd475d2573ba88c8499a939757f2841d562842598644d5a0 Tag: [us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210908002543] - referencing digest: [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:7dc20859932e66fadd475d2573ba88c8499a939757f2841d562842598644d5a0] Deleted [[us.gcr.io/apache-beam-testing/java-postcommit-it/java:20210908002543] (referencing [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:7dc20859932e66fadd475d2573ba88c8499a939757f2841d562842598644d5a0])]. Removing untagged image us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:7dc20859932e66fadd475d2573ba88c8499a939757f2841d562842598644d5a0 Digests: - us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:7dc20859932e66fadd475d2573ba88c8499a939757f2841d562842598644d5a0 Deleted [us.gcr.io/apache-beam-testing/java-postcommit-it/java@sha256:7dc20859932e66fadd475d2573ba88c8499a939757f2841d562842598644d5a0]. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':runners:google-cloud-dataflow-java:validatesCrossLanguageRunnerPythonUsingJava'. > Process 'command 'sh'' finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights. * Get more help at https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8.3/userguide/command_line_interface.html#sec:command_line_warnings BUILD FAILED in 20m 8s 95 actionable tasks: 23 executed, 72 up-to-date Publishing build scan... https://gradle.com/s/hbgqfds7c5f4g Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
