[ https://issues.apache.org/jira/browse/BEAM-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846280#comment-16846280 ]
Valentyn Tymofieiev edited comment on BEAM-7351 at 5/22/19 11:26 PM: --------------------------------------------------------------------- Another instance: {noformat} 14:52:28 ====================================================================== 14:52:28 FAIL: test_streaming_wordcount_it (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) 14:52:28 ---------------------------------------------------------------------- 14:52:28 Traceback (most recent call last): 14:52:28 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py", line 104, in test_streaming_wordcount_it 14:52:28 self.test_pipeline.get_full_options_as_args(**extra_opts)) 14:52:28 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount.py", line 101, in run 14:52:28 result = p.run() 14:52:28 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py", line 419, in run 14:52:28 return self.runner.run_pipeline(self, self._options) 14:52:28 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 68, in run_pipeline 14:52:28 hc_assert_that(self.result, pickler.loads(on_success_matcher)) 14:52:28 AssertionError: 14:52:28 Expected: (Test pipeline expected terminated in state: RUNNING and Expected 500 messages.) 14:52:28 but: Expected 500 messages. Got 509 messages. Diffs (item, count): 14:52:28 Expected but not in actual: dict_items([]) 14:52:28 Unexpected: dict_items([('402: 1', 1), ('279: 1', 1), ('242: 1', 1), ('25: 1', 1), ('394: 1', 1), ('112: 1', 1), ('341: 1', 1), ('103: 1', 1), ('286: 1', 1)]) {noformat} https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-22_14_02_08-4131257113460672476?project=apache-beam-testing was (Author: tvalentyn): Another instance: {noformat} 14:52:28 ====================================================================== 14:52:28 FAIL: test_streaming_wordcount_it (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) 14:52:28 ---------------------------------------------------------------------- 14:52:28 Traceback (most recent call last): 14:52:28 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py", line 104, in test_streaming_wordcount_it 14:52:28 self.test_pipeline.get_full_options_as_args(**extra_opts)) 14:52:28 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount.py", line 101, in run 14:52:28 result = p.run() 14:52:28 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/pipeline.py", line 419, in run 14:52:28 return self.runner.run_pipeline(self, self._options) 14:52:28 File "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python3_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", line 68, in run_pipeline 14:52:28 hc_assert_that(self.result, pickler.loads(on_success_matcher)) 14:52:28 AssertionError: 14:52:28 Expected: (Test pipeline expected terminated in state: RUNNING and Expected 500 messages.) 14:52:28 but: Expected 500 messages. Got 509 messages. Diffs (item, count): 14:52:28 Expected but not in actual: dict_items([]) 14:52:28 Unexpected: dict_items([('402: 1', 1), ('279: 1', 1), ('242: 1', 1), ('25: 1', 1), ('394: 1', 1), ('112: 1', 1), ('341: 1', 1), ('103: 1', 1), ('286: 1', 1)]) {noformat} https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-22_14_02_08-4131257113460672476?project=apache-beam-testing > Failure in Python streaming wordcount test: unexpected messages received on > output topic. > ----------------------------------------------------------------------------------------- > > Key: BEAM-7351 > URL: https://issues.apache.org/jira/browse/BEAM-7351 > Project: Beam > Issue Type: Bug > Components: runner-dataflow, sdk-py-core > Reporter: Valentyn Tymofieiev > Assignee: Ahmet Altay > Priority: Major > > Saw this in a PostCommit test today. Likely a flake, but it's a strange > failure mode and we may need to investigate this. > {noformat} > 13:32:02 > ====================================================================== > 13:32:02 FAIL: test_streaming_wordcount_it > (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT) > 13:32:02 > ---------------------------------------------------------------------- > 13:32:02 Traceback (most recent call last): > 13:32:02 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py", > line 110, in test_streaming_wordcount_it > 13:32:02 self.test_pipeline.get_full_options_as_args(**extra_opts)) > 13:32:02 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount.py", > line 101, in run > 13:32:02 result = p.run() > 13:32:02 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py", > line 419, in run > 13:32:02 return self.runner.run_pipeline(self, self._options) > 13:32:02 File > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py", > line 68, in run_pipeline > 13:32:02 hc_assert_that(self.result, pickler.loads(on_success_matcher)) > 13:32:02 AssertionError: > 13:32:02 Expected: (Test pipeline expected terminated in state: RUNNING and > Expected 500 messages.) > 13:32:02 but: Expected 500 messages. Got 508 messages. Diffs (item, > count): > 13:32:02 Expected but not in actual: [] > 13:32:02 Unexpected: [(u'476: 1', 1), (u'416: 1', 1), (u'245: 1', 1), > (u'478: 1', 1), (u'58: 1', 1), (u'364: 1', 1), (u'77: 1', 1), (u'283: 1', 1)] > 13:32:02 > 13:32:02 -------------------- >> begin captured logging << > -------------------- > 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET > http://169.254.169.254 > 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET > http://metadata.google.internal/computeMetadata/v1/project/project-id > 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET > http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true > 13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): > metadata.google.internal:80 > 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 > "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true > HTTP/1.1" 200 144 > 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET > http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token > 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 > "GET > /computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token > HTTP/1.1" 200 176 > 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET > http://169.254.169.254 > 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET > http://metadata.google.internal/computeMetadata/v1/project/project-id > 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET > http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true > 13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): > metadata.google.internal:80 > 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 > "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true > HTTP/1.1" 200 144 > 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET > http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token > 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 > "GET > /computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token > HTTP/1.1" 200 176 > 13:32:02 root: DEBUG: Injecting 500 numbers to topic > projects/apache-beam-testing/topics/wc_topic_inputb7a2edca-b896-496f-9ab7-4fdfee001097 > 13:32:02 google.cloud.pubsub_v1.publisher._batch.thread: DEBUG: Monitor is > waking up > 13:32:02 root: DEBUG: Connecting using Google Application Default Credentials. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb... > 13:32:02 oauth2client.transport: INFO: Attempting refresh to obtain initial > access_token > 13:32:02 oauth2client.transport: INFO: Attempting refresh to obtain initial > access_token > 13:32:02 google.cloud.pubsub_v1.publisher._batch.thread: DEBUG: gRPC Publish > took 0.253291845322 seconds. > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt > in 0 seconds. > 13:32:02 root: INFO: Executing command: > ['/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/build/gradleenv/1922375555/bin/python', > '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', > 'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:'] > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session > in 0 seconds. > 13:32:02 root: INFO: Copying Beam SDK "build/apache-beam.tar.gz" to staging > location. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar > in 0 seconds. > 13:32:02 root: INFO: Starting GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar... > 13:32:02 root: INFO: Completed GCS upload to > gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar > in 3 seconds. > 13:32:02 root: WARNING: Discarding unparseable args: ['--sleep_secs=20', > '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test'] > 13:32:02 root: WARNING: Discarding unparseable args: ['--sleep_secs=20', > '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test'] > 13:32:02 root: DEBUG: JOB: { > 13:32:02 "environment": { > 13:32:02 "clusterManagerApiService": "compute.googleapis.com", > 13:32:02 "dataset": "bigquery.googleapis.com/cloud_dataflow", > 13:32:02 "experiments": [ > 13:32:02 "use_staged_dataflow_worker_jar", > 13:32:02 "use_multiple_sdk_containers" > 13:32:02 ], > 13:32:02 "sdkPipelineOptions": { > 13:32:02 "display_data": [ > 13:32:02 { > 13:32:02 "key": "requirements_file", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": "postcommit_requirements.txt" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "sdk_location", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": "build/apache-beam.tar.gz" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "save_main_session", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "BOOLEAN", > 13:32:02 "value": true > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "num_workers", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "INTEGER", > 13:32:02 "value": 1 > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "runner", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": "TestDataflowRunner" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "staging_location", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "dataflow_worker_jar", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.14.0-SNAPSHOT.jar" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "wait_until_finish_duration", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "INTEGER", > 13:32:02 "value": 180000 > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "project", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": "apache-beam-testing" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "streaming", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "BOOLEAN", > 13:32:02 "value": true > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "dataflow_kms_key", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "experiments", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": "['use_staged_dataflow_worker_jar', > 'use_multiple_sdk_containers']" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "temp_location", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "beam_plugins", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": "['apache_beam.io.filesystem.FileSystem', > 'apache_beam.io.hadoopfilesystem.HadoopFileSystem', > 'apache_beam.io.localfilesystem.LocalFileSystem', > 'apache_beam.io.gcp.gcsfilesystem.GCSFileSystem', > 'apache_beam.io.filesystem_test.TestingFileSystem', > 'apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer', > > 'apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer', > > 'apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer', > > 'apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer']" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "job_name", > 13:32:02 "namespace": > "apache_beam.options.pipeline_options.PipelineOptions", > 13:32:02 "type": "STRING", > 13:32:02 "value": "beamapp-jenkins-0517194018-707115" > 13:32:02 } > 13:32:02 ], > 13:32:02 "options": { > 13:32:02 "beam_plugins": [ > 13:32:02 "apache_beam.io.filesystem.FileSystem", > 13:32:02 "apache_beam.io.hadoopfilesystem.HadoopFileSystem", > 13:32:02 "apache_beam.io.localfilesystem.LocalFileSystem", > 13:32:02 "apache_beam.io.gcp.gcsfilesystem.GCSFileSystem", > 13:32:02 "apache_beam.io.filesystem_test.TestingFileSystem", > 13:32:02 > "apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer", > > 13:32:02 > "apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer", > > 13:32:02 > "apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer", > > 13:32:02 > "apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer" > 13:32:02 ], > 13:32:02 "dataflow_endpoint": "https://dataflow.googleapis.com", > 13:32:02 "dataflow_kms_key": > "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test", > > 13:32:02 "dataflow_worker_jar": > "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.14.0-SNAPSHOT.jar", > > 13:32:02 "direct_runner_bundle_repeat": 0, > 13:32:02 "direct_runner_use_stacked_bundle": true, > 13:32:02 "dry_run": false, > 13:32:02 "enable_streaming_engine": false, > 13:32:02 "environment_cache_millis": 0, > 13:32:02 "experiments": [ > 13:32:02 "use_staged_dataflow_worker_jar", > 13:32:02 "use_multiple_sdk_containers" > 13:32:02 ], > 13:32:02 "fake_flag": false, > 13:32:02 "job_name": "beamapp-jenkins-0517194018-707115", > 13:32:02 "m_flag": false, > 13:32:02 "male": false, > 13:32:02 "mock_flag": false, > 13:32:02 "no_auth": false, > 13:32:02 "num_workers": 1, > 13:32:02 "pipelineUrl": > "gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb", > > 13:32:02 "pipeline_type_check": true, > 13:32:02 "profile_cpu": false, > 13:32:02 "profile_memory": false, > 13:32:02 "profile_sample_rate": 1.0, > 13:32:02 "project": "apache-beam-testing", > 13:32:02 "region": "us-central1", > 13:32:02 "requirements_file": "postcommit_requirements.txt", > 13:32:02 "runner": "TestDataflowRunner", > 13:32:02 "runtime_type_check": false, > 13:32:02 "save_main_session": true, > 13:32:02 "sdk_location": "build/apache-beam.tar.gz", > 13:32:02 "sdk_worker_parallelism": 0, > 13:32:02 "staging_location": > "gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255", > > 13:32:02 "streaming": true, > 13:32:02 "style": "scrambled", > 13:32:02 "temp_location": > "gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255", > > 13:32:02 "type_check_strictness": "DEFAULT_TO_ANY", > 13:32:02 "update": false, > 13:32:02 "wait_until_finish_duration": 180000 > 13:32:02 } > 13:32:02 }, > 13:32:02 "serviceKmsKeyName": > "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test", > > 13:32:02 "tempStoragePrefix": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255", > > 13:32:02 "userAgent": { > 13:32:02 "name": "Apache Beam Python 2.7 SDK", > 13:32:02 "version": "2.14.0.dev" > 13:32:02 }, > 13:32:02 "version": { > 13:32:02 "job_type": "FNAPI_STREAMING", > 13:32:02 "major": "7" > 13:32:02 }, > 13:32:02 "workerPools": [ > 13:32:02 { > 13:32:02 "autoscalingSettings": {}, > 13:32:02 "dataDisks": [ > 13:32:02 {} > 13:32:02 ], > 13:32:02 "kind": "harness", > 13:32:02 "numWorkers": 1, > 13:32:02 "packages": [ > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt", > > 13:32:02 "name": "requirements.txt" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz", > > 13:32:02 "name": "PyHamcrest-1.9.0.tar.gz" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz", > > 13:32:02 "name": "mock-3.0.5.tar.gz" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz", > > 13:32:02 "name": "mock-3.0.4.tar.gz" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip", > > 13:32:02 "name": "setuptools-41.0.1.zip" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz", > > 13:32:02 "name": "pbr-5.2.0.tar.gz" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz", > > 13:32:02 "name": "mock-2.0.0.tar.gz" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz", > > 13:32:02 "name": "mock-3.0.3.tar.gz" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz", > > 13:32:02 "name": "six-1.12.0.tar.gz" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz", > > 13:32:02 "name": "funcsigs-1.0.2.tar.gz" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session", > > 13:32:02 "name": "pickled_main_session" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar", > > 13:32:02 "name": "dataflow_python_sdk.tar" > 13:32:02 }, > 13:32:02 { > 13:32:02 "location": > "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar", > > 13:32:02 "name": "dataflow-worker.jar" > 13:32:02 } > 13:32:02 ], > 13:32:02 "taskrunnerSettings": { > 13:32:02 "parallelWorkerSettings": { > 13:32:02 "baseUrl": "https://dataflow.googleapis.com", > 13:32:02 "servicePath": "https://dataflow.googleapis.com" > 13:32:02 } > 13:32:02 }, > 13:32:02 "workerHarnessContainerImage": > "gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190509" > 13:32:02 } > 13:32:02 ] > 13:32:02 }, > 13:32:02 "name": "beamapp-jenkins-0517194018-707115", > 13:32:02 "steps": [ > 13:32:02 { > 13:32:02 "kind": "ParallelRead", > 13:32:02 "name": "s1", > 13:32:02 "properties": { > 13:32:02 "display_data": [ > 13:32:02 { > 13:32:02 "key": "source", > 13:32:02 "label": "Read Source", > 13:32:02 "namespace": "apache_beam.io.iobase.Read", > 13:32:02 "shortValue": "_PubSubSource", > 13:32:02 "type": "STRING", > 13:32:02 "value": "apache_beam.io.gcp.pubsub._PubSubSource" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "with_attributes", > 13:32:02 "label": "With Attributes", > 13:32:02 "namespace": "apache_beam.io.gcp.pubsub._PubSubSource", > 13:32:02 "type": "BOOLEAN", > 13:32:02 "value": false > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "subscription", > 13:32:02 "label": "Pubsub Subscription", > 13:32:02 "namespace": "apache_beam.io.gcp.pubsub._PubSubSource", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "projects/apache-beam-testing/subscriptions/wc_subscription_inputb7a2edca-b896-496f-9ab7-4fdfee001097" > 13:32:02 } > 13:32:02 ], > 13:32:02 "format": "pubsub", > 13:32:02 "output_info": [ > 13:32:02 { > 13:32:02 "encoding": { > 13:32:02 "@type": "kind:windowed_value", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": "kind:bytes" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:global_window" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_wrapper": true > 13:32:02 }, > 13:32:02 "output_name": "out", > 13:32:02 "user_name": "ReadFromPubSub/Read.out" > 13:32:02 } > 13:32:02 ], > 13:32:02 "pubsub_subscription": > "projects/apache-beam-testing/subscriptions/wc_subscription_inputb7a2edca-b896-496f-9ab7-4fdfee001097", > > 13:32:02 "user_name": "ReadFromPubSub/Read" > 13:32:02 } > 13:32:02 }, > 13:32:02 { > 13:32:02 "kind": "ParallelDo", > 13:32:02 "name": "s2", > 13:32:02 "properties": { > 13:32:02 "display_data": [ > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": > "apache_beam.transforms.core.CallableWrapperDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": "<lambda>" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", > 13:32:02 "shortValue": "CallableWrapperDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "apache_beam.transforms.core.CallableWrapperDoFn" > 13:32:02 } > 13:32:02 ], > 13:32:02 "non_parallel_inputs": {}, > 13:32:02 "output_info": [ > 13:32:02 { > 13:32:02 "encoding": { > 13:32:02 "@type": "kind:windowed_value", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_FastPrimitivesCoder_3" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_FastPrimitivesCoder_3" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_pair_like": true, > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_FastPrimitivesCoder_3" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:global_window" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_wrapper": true > 13:32:02 }, > 13:32:02 "output_name": "out", > 13:32:02 "user_name": "decode.out" > 13:32:02 } > 13:32:02 ], > 13:32:02 "parallel_input": { > 13:32:02 "@type": "OutputReference", > 13:32:02 "output_name": "out", > 13:32:02 "step_name": "s1" > 13:32:02 }, > 13:32:02 "serialized_fn": "ref_AppliedPTransform_decode_4", > 13:32:02 "user_name": "decode" > 13:32:02 } > 13:32:02 }, > 13:32:02 { > 13:32:02 "kind": "ParallelDo", > 13:32:02 "name": "s3", > 13:32:02 "properties": { > 13:32:02 "display_data": [ > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", > 13:32:02 "shortValue": "WordExtractingDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "apache_beam.examples.wordcount.WordExtractingDoFn" > 13:32:02 } > 13:32:02 ], > 13:32:02 "non_parallel_inputs": {}, > 13:32:02 "output_info": [ > 13:32:02 { > 13:32:02 "encoding": { > 13:32:02 "@type": "kind:windowed_value", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": > "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_StrUtf8Coder_4" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:global_window" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_wrapper": true > 13:32:02 }, > 13:32:02 "output_name": "out", > 13:32:02 "user_name": "split.out" > 13:32:02 } > 13:32:02 ], > 13:32:02 "parallel_input": { > 13:32:02 "@type": "OutputReference", > 13:32:02 "output_name": "out", > 13:32:02 "step_name": "s2" > 13:32:02 }, > 13:32:02 "serialized_fn": "ref_AppliedPTransform_split_5", > 13:32:02 "user_name": "split" > 13:32:02 } > 13:32:02 }, > 13:32:02 { > 13:32:02 "kind": "ParallelDo", > 13:32:02 "name": "s4", > 13:32:02 "properties": { > 13:32:02 "display_data": [ > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": > "apache_beam.transforms.core.CallableWrapperDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": "<lambda>" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", > 13:32:02 "shortValue": "CallableWrapperDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "apache_beam.transforms.core.CallableWrapperDoFn" > 13:32:02 } > 13:32:02 ], > 13:32:02 "non_parallel_inputs": {}, > 13:32:02 "output_info": [ > 13:32:02 { > 13:32:02 "encoding": { > 13:32:02 "@type": "kind:windowed_value", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": "kind:pair", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": > "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_StrUtf8Coder_4" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:varint" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_pair_like": true > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:global_window" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_wrapper": true > 13:32:02 }, > 13:32:02 "output_name": "out", > 13:32:02 "user_name": "pair_with_one.out" > 13:32:02 } > 13:32:02 ], > 13:32:02 "parallel_input": { > 13:32:02 "@type": "OutputReference", > 13:32:02 "output_name": "out", > 13:32:02 "step_name": "s3" > 13:32:02 }, > 13:32:02 "serialized_fn": "ref_AppliedPTransform_pair_with_one_6", > 13:32:02 "user_name": "pair_with_one" > 13:32:02 } > 13:32:02 }, > 13:32:02 { > 13:32:02 "kind": "ParallelDo", > 13:32:02 "name": "s5", > 13:32:02 "properties": { > 13:32:02 "display_data": [ > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": "apache_beam.transforms.core.WindowInto", > 13:32:02 "shortValue": "WindowIntoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": "apache_beam.transforms.core.WindowIntoFn" > 13:32:02 } > 13:32:02 ], > 13:32:02 "non_parallel_inputs": {}, > 13:32:02 "output_info": [ > 13:32:02 { > 13:32:02 "encoding": { > 13:32:02 "@type": "kind:windowed_value", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": "kind:pair", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": > "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_StrUtf8Coder_4" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:varint" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_pair_like": true > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:interval_window" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_wrapper": true > 13:32:02 }, > 13:32:02 "output_name": "out", > 13:32:02 "user_name": "WindowInto(WindowIntoFn).out" > 13:32:02 } > 13:32:02 ], > 13:32:02 "parallel_input": { > 13:32:02 "@type": "OutputReference", > 13:32:02 "output_name": "out", > 13:32:02 "step_name": "s4" > 13:32:02 }, > 13:32:02 "serialized_fn": "<string of 436 bytes>", > 13:32:02 "user_name": "WindowInto(WindowIntoFn)" > 13:32:02 } > 13:32:02 }, > 13:32:02 { > 13:32:02 "kind": "GroupByKey", > 13:32:02 "name": "s6", > 13:32:02 "properties": { > 13:32:02 "display_data": [], > 13:32:02 "output_info": [ > 13:32:02 { > 13:32:02 "encoding": { > 13:32:02 "@type": "kind:windowed_value", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": "kind:pair", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": > "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_StrUtf8Coder_4" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:stream", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": "kind:varint" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_stream_like": true > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_pair_like": true > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:interval_window" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_wrapper": true > 13:32:02 }, > 13:32:02 "output_name": "out", > 13:32:02 "user_name": "group.out" > 13:32:02 } > 13:32:02 ], > 13:32:02 "parallel_input": { > 13:32:02 "@type": "OutputReference", > 13:32:02 "output_name": "out", > 13:32:02 "step_name": "s5" > 13:32:02 }, > 13:32:02 "serialized_fn": > "%0AH%22F%0A%1Fref_Coder_IntervalWindowCoder_1%12%23%0A%21%0A%1F%0A%1Dbeam%3Acoder%3Ainterval_window%3Av1j%5D%0A%2C%0A%2A%0A%20beam%3Awindowfn%3Afixed_windows%3Av0.1%1A%06%0A%02%08%0F%12%00%10%01%1A%1Fref_Coder_IntervalWindowCoder_1%22%02%3A%00%28%010%018%01H%01", > > 13:32:02 "user_name": "group" > 13:32:02 } > 13:32:02 }, > 13:32:02 { > 13:32:02 "kind": "ParallelDo", > 13:32:02 "name": "s7", > 13:32:02 "properties": { > 13:32:02 "display_data": [ > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": > "apache_beam.transforms.core.CallableWrapperDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": "count_ones" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", > 13:32:02 "shortValue": "CallableWrapperDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "apache_beam.transforms.core.CallableWrapperDoFn" > 13:32:02 } > 13:32:02 ], > 13:32:02 "non_parallel_inputs": {}, > 13:32:02 "output_info": [ > 13:32:02 { > 13:32:02 "encoding": { > 13:32:02 "@type": "kind:windowed_value", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": "kind:pair", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": > "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_StrUtf8Coder_4" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_FastPrimitivesCoder_3" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_FastPrimitivesCoder_3" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_pair_like": true, > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_FastPrimitivesCoder_3" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_pair_like": true > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:interval_window" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_wrapper": true > 13:32:02 }, > 13:32:02 "output_name": "out", > 13:32:02 "user_name": "count.out" > 13:32:02 } > 13:32:02 ], > 13:32:02 "parallel_input": { > 13:32:02 "@type": "OutputReference", > 13:32:02 "output_name": "out", > 13:32:02 "step_name": "s6" > 13:32:02 }, > 13:32:02 "serialized_fn": "ref_AppliedPTransform_count_9", > 13:32:02 "user_name": "count" > 13:32:02 } > 13:32:02 }, > 13:32:02 { > 13:32:02 "kind": "ParallelDo", > 13:32:02 "name": "s8", > 13:32:02 "properties": { > 13:32:02 "display_data": [ > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": > "apache_beam.transforms.core.CallableWrapperDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": "format_result" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", > 13:32:02 "shortValue": "CallableWrapperDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "apache_beam.transforms.core.CallableWrapperDoFn" > 13:32:02 } > 13:32:02 ], > 13:32:02 "non_parallel_inputs": {}, > 13:32:02 "output_info": [ > 13:32:02 { > 13:32:02 "encoding": { > 13:32:02 "@type": "kind:windowed_value", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_FastPrimitivesCoder_3" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": > "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/", > > 13:32:02 "component_encodings": [], > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_FastPrimitivesCoder_3" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_pair_like": true, > 13:32:02 "pipeline_proto_coder_id": > "ref_Coder_FastPrimitivesCoder_3" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:interval_window" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_wrapper": true > 13:32:02 }, > 13:32:02 "output_name": "out", > 13:32:02 "user_name": "format.out" > 13:32:02 } > 13:32:02 ], > 13:32:02 "parallel_input": { > 13:32:02 "@type": "OutputReference", > 13:32:02 "output_name": "out", > 13:32:02 "step_name": "s7" > 13:32:02 }, > 13:32:02 "serialized_fn": "ref_AppliedPTransform_format_10", > 13:32:02 "user_name": "format" > 13:32:02 } > 13:32:02 }, > 13:32:02 { > 13:32:02 "kind": "ParallelDo", > 13:32:02 "name": "s9", > 13:32:02 "properties": { > 13:32:02 "display_data": [ > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": > "apache_beam.transforms.core.CallableWrapperDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": "<lambda>" > 13:32:02 }, > 13:32:02 { > 13:32:02 "key": "fn", > 13:32:02 "label": "Transform Function", > 13:32:02 "namespace": "apache_beam.transforms.core.ParDo", > 13:32:02 "shortValue": "CallableWrapperDoFn", > 13:32:02 "type": "STRING", > 13:32:02 "value": > "apache_beam.transforms.core.CallableWrapperDoFn" > 13:32:02 } > 13:32:02 ], > 13:32:02 "non_parallel_inputs": {}, > 13:32:02 "output_info": [ > 13:32:02 { > 13:32:02 "encoding": { > 13:32:02 "@type": "kind:windowed_value", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": "kind:bytes" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:interval_window" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_wrapper": true > 13:32:02 }, > 13:32:02 "output_name": "out", > 13:32:02 "user_name": "encode.out" > 13:32:02 } > 13:32:02 ], > 13:32:02 "parallel_input": { > 13:32:02 "@type": "OutputReference", > 13:32:02 "output_name": "out", > 13:32:02 "step_name": "s8" > 13:32:02 }, > 13:32:02 "serialized_fn": "ref_AppliedPTransform_encode_11", > 13:32:02 "user_name": "encode" > 13:32:02 } > 13:32:02 }, > 13:32:02 { > 13:32:02 "kind": "ParallelWrite", > 13:32:02 "name": "s10", > 13:32:02 "properties": { > 13:32:02 "display_data": [], > 13:32:02 "encoding": { > 13:32:02 "@type": "kind:windowed_value", > 13:32:02 "component_encodings": [ > 13:32:02 { > 13:32:02 "@type": "kind:bytes" > 13:32:02 }, > 13:32:02 { > 13:32:02 "@type": "kind:global_window" > 13:32:02 } > 13:32:02 ], > 13:32:02 "is_wrapper": true > 13:32:02 }, > 13:32:02 "format": "pubsub", > 13:32:02 "parallel_input": { > 13:32:02 "@type": "OutputReference", > 13:32:02 "output_name": "out", > 13:32:02 "step_name": "s9" > 13:32:02 }, > 13:32:02 "pubsub_topic": > "projects/apache-beam-testing/topics/wc_topic_outputb7a2edca-b896-496f-9ab7-4fdfee001097", > > 13:32:02 "user_name": "WriteToPubSub/Write/NativeWrite" > 13:32:02 } > 13:32:02 } > 13:32:02 ], > 13:32:02 "type": "JOB_TYPE_STREAMING" > 13:32:02 } > 13:32:02 root: INFO: Create job: <Job > 13:32:02 createTime: u'2019-05-17T19:40:28.428825Z' > 13:32:02 currentStateTime: u'1970-01-01T00:00:00Z' > 13:32:02 id: u'2019-05-17_12_40_27-16144646158774694719' > 13:32:02 location: u'us-central1' > 13:32:02 name: u'beamapp-jenkins-0517194018-707115' > 13:32:02 projectId: u'apache-beam-testing' > 13:32:02 stageStates: [] > 13:32:02 startTime: u'2019-05-17T19:40:28.428825Z' > 13:32:02 steps: [] > 13:32:02 tempFiles: [] > 13:32:02 type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)> > 13:32:02 root: INFO: Created job with id: > [2019-05-17_12_40_27-16144646158774694719] > 13:32:02 root: INFO: To access the Dataflow monitoring console, please > navigate to > https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-17_12_40_27-16144646158774694719?project=apache-beam-testing > 13:32:02 root: INFO: Job 2019-05-17_12_40_27-16144646158774694719 is in state > JOB_STATE_RUNNING > 13:32:02 root: INFO: 2019-05-17T19:40:31.054Z: JOB_MESSAGE_DETAILED: Checking > permissions granted to controller Service Account. > 13:32:02 root: INFO: 2019-05-17T19:40:31.699Z: JOB_MESSAGE_BASIC: Worker > configuration: n1-standard-4 in us-central1-a. > 13:32:02 root: INFO: 2019-05-17T19:40:32.189Z: JOB_MESSAGE_DETAILED: > Expanding SplittableParDo operations into optimizable parts. > 13:32:02 root: INFO: 2019-05-17T19:40:32.192Z: JOB_MESSAGE_DETAILED: > Expanding CollectionToSingleton operations into optimizable parts. > 13:32:02 root: INFO: 2019-05-17T19:40:32.202Z: JOB_MESSAGE_DETAILED: > Expanding CoGroupByKey operations into optimizable parts. > 13:32:02 root: INFO: 2019-05-17T19:40:32.208Z: JOB_MESSAGE_DETAILED: > Expanding SplittableProcessKeyed operations into optimizable parts. > 13:32:02 root: INFO: 2019-05-17T19:40:32.211Z: JOB_MESSAGE_DETAILED: > Expanding GroupByKey operations into streaming Read/Write steps > 13:32:02 root: INFO: 2019-05-17T19:40:32.219Z: JOB_MESSAGE_DEBUG: Annotating > graph with Autotuner information. > 13:32:02 root: INFO: 2019-05-17T19:40:32.232Z: JOB_MESSAGE_DETAILED: Fusing > adjacent ParDo, Read, Write, and Flatten operations > 13:32:02 root: INFO: 2019-05-17T19:40:32.235Z: JOB_MESSAGE_DETAILED: Fusing > consumer decode into ReadFromPubSub/Read > 13:32:02 root: INFO: 2019-05-17T19:40:32.238Z: JOB_MESSAGE_DETAILED: Fusing > consumer pair_with_one into split > 13:32:02 root: INFO: 2019-05-17T19:40:32.240Z: JOB_MESSAGE_DETAILED: Fusing > consumer count into group/MergeBuckets > 13:32:02 root: INFO: 2019-05-17T19:40:32.243Z: JOB_MESSAGE_DETAILED: Fusing > consumer WriteToPubSub/Write/NativeWrite into encode > 13:32:02 root: INFO: 2019-05-17T19:40:32.245Z: JOB_MESSAGE_DETAILED: Fusing > consumer encode into format > 13:32:02 root: INFO: 2019-05-17T19:40:32.247Z: JOB_MESSAGE_DETAILED: Fusing > consumer group/MergeBuckets into group/ReadStream > 13:32:02 root: INFO: 2019-05-17T19:40:32.250Z: JOB_MESSAGE_DETAILED: Fusing > consumer format into count > 13:32:02 root: INFO: 2019-05-17T19:40:32.252Z: JOB_MESSAGE_DETAILED: Fusing > consumer group/WriteStream into WindowInto(WindowIntoFn) > 13:32:02 root: INFO: 2019-05-17T19:40:32.254Z: JOB_MESSAGE_DETAILED: Fusing > consumer split into decode > 13:32:02 root: INFO: 2019-05-17T19:40:32.256Z: JOB_MESSAGE_DETAILED: Fusing > consumer WindowInto(WindowIntoFn) into pair_with_one > 13:32:02 root: INFO: 2019-05-17T19:40:32.268Z: JOB_MESSAGE_DEBUG: Adding > StepResource setup and teardown to workflow graph. > 13:32:02 root: INFO: 2019-05-17T19:40:32.283Z: JOB_MESSAGE_DEBUG: Adding > workflow start and stop steps. > 13:32:02 root: INFO: 2019-05-17T19:40:32.296Z: JOB_MESSAGE_DEBUG: Assigning > stage ids. > 13:32:02 root: INFO: 2019-05-17T19:40:32.457Z: JOB_MESSAGE_DEBUG: Executing > wait step start2 > 13:32:02 root: INFO: 2019-05-17T19:40:32.490Z: JOB_MESSAGE_DEBUG: Starting > worker pool setup. > 13:32:02 root: INFO: 2019-05-17T19:40:32.496Z: JOB_MESSAGE_BASIC: Starting 1 > workers... > 13:32:02 root: INFO: 2019-05-17T19:40:34.639Z: JOB_MESSAGE_BASIC: Executing > operation > group/ReadStream+group/MergeBuckets+count+format+encode+WriteToPubSub/Write/NativeWrite > 13:32:02 root: INFO: 2019-05-17T19:40:34.639Z: JOB_MESSAGE_BASIC: Executing > operation > ReadFromPubSub/Read+decode+split+pair_with_one+WindowInto(WindowIntoFn)+group/WriteStream > 13:32:02 root: INFO: 2019-05-17T19:41:03.592Z: JOB_MESSAGE_DETAILED: Workers > have started successfully. > 13:32:02 root: INFO: 2019-05-17T19:41:47.135Z: JOB_MESSAGE_DEBUG: Executing > input step topology_init_attach_disk_input_step > 13:32:02 root: WARNING: Timing out on waiting for job > 2019-05-17_12_40_27-16144646158774694719 after 184 seconds > 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET > http://169.254.169.254 > 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET > http://metadata.google.internal/computeMetadata/v1/project/project-id > 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET > http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true > 13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1): > metadata.google.internal:80 > 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 > "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true > HTTP/1.1" 200 144 > 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET > http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token > 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 > "GET > /computeMetadata/v1/instance/service-accounts/844138762903-comp...@developer.gserviceaccount.com/token > HTTP/1.1" 200 176 > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)