[
https://issues.apache.org/jira/browse/BEAM-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pablo Estrada updated BEAM-7351:
--------------------------------
Status: Open (was: Triage Needed)
> Failure in Python streaming wordcount test: unexpected messages received on
> output topic.
> -----------------------------------------------------------------------------------------
>
> Key: BEAM-7351
> URL: https://issues.apache.org/jira/browse/BEAM-7351
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow, sdk-py-core
> Reporter: Valentyn Tymofieiev
> Assignee: Valentyn Tymofieiev
> Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Saw this in a PostCommit test today. Likely a flake, but it's a strange
> failure mode and we may need to investigate this.
> {noformat}
> 13:32:02
> ======================================================================
> 13:32:02 FAIL: test_streaming_wordcount_it
> (apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT)
> 13:32:02
> ----------------------------------------------------------------------
> 13:32:02 Traceback (most recent call last):
> 13:32:02 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py",
> line 110, in test_streaming_wordcount_it
> 13:32:02 self.test_pipeline.get_full_options_as_args(**extra_opts))
> 13:32:02 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount.py",
> line 101, in run
> 13:32:02 result = p.run()
> 13:32:02 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> 13:32:02 return self.runner.run_pipeline(self, self._options)
> 13:32:02 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
> line 68, in run_pipeline
> 13:32:02 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 13:32:02 AssertionError:
> 13:32:02 Expected: (Test pipeline expected terminated in state: RUNNING and
> Expected 500 messages.)
> 13:32:02 but: Expected 500 messages. Got 508 messages. Diffs (item,
> count):
> 13:32:02 Expected but not in actual: []
> 13:32:02 Unexpected: [(u'476: 1', 1), (u'416: 1', 1), (u'245: 1', 1),
> (u'478: 1', 1), (u'58: 1', 1), (u'364: 1', 1), (u'77: 1', 1), (u'283: 1', 1)]
> 13:32:02
> 13:32:02 -------------------- >> begin captured logging <<
> --------------------
> 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
> http://169.254.169.254
> 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/project/project-id
> 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
> 13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
> metadata.google.internal:80
> 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
> "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true
> HTTP/1.1" 200 144
> 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token
> 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
> "GET
> /computeMetadata/v1/instance/service-accounts/[email protected]/token
> HTTP/1.1" 200 176
> 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
> http://169.254.169.254
> 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/project/project-id
> 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
> 13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
> metadata.google.internal:80
> 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
> "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true
> HTTP/1.1" 200 144
> 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token
> 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
> "GET
> /computeMetadata/v1/instance/service-accounts/[email protected]/token
> HTTP/1.1" 200 176
> 13:32:02 root: DEBUG: Injecting 500 numbers to topic
> projects/apache-beam-testing/topics/wc_topic_inputb7a2edca-b896-496f-9ab7-4fdfee001097
> 13:32:02 google.cloud.pubsub_v1.publisher._batch.thread: DEBUG: Monitor is
> waking up
> 13:32:02 root: DEBUG: Connecting using Google Application Default Credentials.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb...
> 13:32:02 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 13:32:02 oauth2client.transport: INFO: Attempting refresh to obtain initial
> access_token
> 13:32:02 google.cloud.pubsub_v1.publisher._batch.thread: DEBUG: gRPC Publish
> took 0.253291845322 seconds.
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt
> in 0 seconds.
> 13:32:02 root: INFO: Executing command:
> ['/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/build/gradleenv/1922375555/bin/python',
> '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r',
> 'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session
> in 0 seconds.
> 13:32:02 root: INFO: Copying Beam SDK "build/apache-beam.tar.gz" to staging
> location.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar
> in 0 seconds.
> 13:32:02 root: INFO: Starting GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar...
> 13:32:02 root: INFO: Completed GCS upload to
> gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar
> in 3 seconds.
> 13:32:02 root: WARNING: Discarding unparseable args: ['--sleep_secs=20',
> '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
> 13:32:02 root: WARNING: Discarding unparseable args: ['--sleep_secs=20',
> '--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
> 13:32:02 root: DEBUG: JOB: {
> 13:32:02 "environment": {
> 13:32:02 "clusterManagerApiService": "compute.googleapis.com",
> 13:32:02 "dataset": "bigquery.googleapis.com/cloud_dataflow",
> 13:32:02 "experiments": [
> 13:32:02 "use_staged_dataflow_worker_jar",
> 13:32:02 "use_multiple_sdk_containers"
> 13:32:02 ],
> 13:32:02 "sdkPipelineOptions": {
> 13:32:02 "display_data": [
> 13:32:02 {
> 13:32:02 "key": "requirements_file",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "postcommit_requirements.txt"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "sdk_location",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "build/apache-beam.tar.gz"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "save_main_session",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "BOOLEAN",
> 13:32:02 "value": true
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "num_workers",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "INTEGER",
> 13:32:02 "value": 1
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "runner",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "TestDataflowRunner"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "staging_location",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "dataflow_worker_jar",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.14.0-SNAPSHOT.jar"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "wait_until_finish_duration",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "INTEGER",
> 13:32:02 "value": 180000
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "project",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "apache-beam-testing"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "streaming",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "BOOLEAN",
> 13:32:02 "value": true
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "dataflow_kms_key",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "experiments",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "['use_staged_dataflow_worker_jar',
> 'use_multiple_sdk_containers']"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "temp_location",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "beam_plugins",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "['apache_beam.io.filesystem.FileSystem',
> 'apache_beam.io.hadoopfilesystem.HadoopFileSystem',
> 'apache_beam.io.localfilesystem.LocalFileSystem',
> 'apache_beam.io.gcp.gcsfilesystem.GCSFileSystem',
> 'apache_beam.io.filesystem_test.TestingFileSystem',
> 'apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer',
>
> 'apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer',
>
> 'apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer',
>
> 'apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer']"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "job_name",
> 13:32:02 "namespace":
> "apache_beam.options.pipeline_options.PipelineOptions",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "beamapp-jenkins-0517194018-707115"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "options": {
> 13:32:02 "beam_plugins": [
> 13:32:02 "apache_beam.io.filesystem.FileSystem",
> 13:32:02 "apache_beam.io.hadoopfilesystem.HadoopFileSystem",
> 13:32:02 "apache_beam.io.localfilesystem.LocalFileSystem",
> 13:32:02 "apache_beam.io.gcp.gcsfilesystem.GCSFileSystem",
> 13:32:02 "apache_beam.io.filesystem_test.TestingFileSystem",
> 13:32:02
> "apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer",
>
> 13:32:02
> "apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer",
>
> 13:32:02
> "apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer",
>
> 13:32:02
> "apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer"
> 13:32:02 ],
> 13:32:02 "dataflow_endpoint": "https://dataflow.googleapis.com";,
> 13:32:02 "dataflow_kms_key":
> "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test",
>
> 13:32:02 "dataflow_worker_jar":
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.14.0-SNAPSHOT.jar",
>
> 13:32:02 "direct_runner_bundle_repeat": 0,
> 13:32:02 "direct_runner_use_stacked_bundle": true,
> 13:32:02 "dry_run": false,
> 13:32:02 "enable_streaming_engine": false,
> 13:32:02 "environment_cache_millis": 0,
> 13:32:02 "experiments": [
> 13:32:02 "use_staged_dataflow_worker_jar",
> 13:32:02 "use_multiple_sdk_containers"
> 13:32:02 ],
> 13:32:02 "fake_flag": false,
> 13:32:02 "job_name": "beamapp-jenkins-0517194018-707115",
> 13:32:02 "m_flag": false,
> 13:32:02 "male": false,
> 13:32:02 "mock_flag": false,
> 13:32:02 "no_auth": false,
> 13:32:02 "num_workers": 1,
> 13:32:02 "pipelineUrl":
> "gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb",
>
> 13:32:02 "pipeline_type_check": true,
> 13:32:02 "profile_cpu": false,
> 13:32:02 "profile_memory": false,
> 13:32:02 "profile_sample_rate": 1.0,
> 13:32:02 "project": "apache-beam-testing",
> 13:32:02 "region": "us-central1",
> 13:32:02 "requirements_file": "postcommit_requirements.txt",
> 13:32:02 "runner": "TestDataflowRunner",
> 13:32:02 "runtime_type_check": false,
> 13:32:02 "save_main_session": true,
> 13:32:02 "sdk_location": "build/apache-beam.tar.gz",
> 13:32:02 "sdk_worker_parallelism": 0,
> 13:32:02 "staging_location":
> "gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255",
>
> 13:32:02 "streaming": true,
> 13:32:02 "style": "scrambled",
> 13:32:02 "temp_location":
> "gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255",
>
> 13:32:02 "type_check_strictness": "DEFAULT_TO_ANY",
> 13:32:02 "update": false,
> 13:32:02 "wait_until_finish_duration": 180000
> 13:32:02 }
> 13:32:02 },
> 13:32:02 "serviceKmsKeyName":
> "projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test",
>
> 13:32:02 "tempStoragePrefix":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255",
>
> 13:32:02 "userAgent": {
> 13:32:02 "name": "Apache Beam Python 2.7 SDK",
> 13:32:02 "version": "2.14.0.dev"
> 13:32:02 },
> 13:32:02 "version": {
> 13:32:02 "job_type": "FNAPI_STREAMING",
> 13:32:02 "major": "7"
> 13:32:02 },
> 13:32:02 "workerPools": [
> 13:32:02 {
> 13:32:02 "autoscalingSettings": {},
> 13:32:02 "dataDisks": [
> 13:32:02 {}
> 13:32:02 ],
> 13:32:02 "kind": "harness",
> 13:32:02 "numWorkers": 1,
> 13:32:02 "packages": [
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt",
>
> 13:32:02 "name": "requirements.txt"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz",
>
> 13:32:02 "name": "PyHamcrest-1.9.0.tar.gz"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz",
>
> 13:32:02 "name": "mock-3.0.5.tar.gz"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz",
>
> 13:32:02 "name": "mock-3.0.4.tar.gz"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip",
>
> 13:32:02 "name": "setuptools-41.0.1.zip"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz",
>
> 13:32:02 "name": "pbr-5.2.0.tar.gz"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz",
>
> 13:32:02 "name": "mock-2.0.0.tar.gz"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz",
>
> 13:32:02 "name": "mock-3.0.3.tar.gz"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz",
>
> 13:32:02 "name": "six-1.12.0.tar.gz"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz",
>
> 13:32:02 "name": "funcsigs-1.0.2.tar.gz"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session",
>
> 13:32:02 "name": "pickled_main_session"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar",
>
> 13:32:02 "name": "dataflow_python_sdk.tar"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "location":
> "storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar",
>
> 13:32:02 "name": "dataflow-worker.jar"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "taskrunnerSettings": {
> 13:32:02 "parallelWorkerSettings": {
> 13:32:02 "baseUrl": "https://dataflow.googleapis.com";,
> 13:32:02 "servicePath": "https://dataflow.googleapis.com";
> 13:32:02 }
> 13:32:02 },
> 13:32:02 "workerHarnessContainerImage":
> "gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190509"
> 13:32:02 }
> 13:32:02 ]
> 13:32:02 },
> 13:32:02 "name": "beamapp-jenkins-0517194018-707115",
> 13:32:02 "steps": [
> 13:32:02 {
> 13:32:02 "kind": "ParallelRead",
> 13:32:02 "name": "s1",
> 13:32:02 "properties": {
> 13:32:02 "display_data": [
> 13:32:02 {
> 13:32:02 "key": "source",
> 13:32:02 "label": "Read Source",
> 13:32:02 "namespace": "apache_beam.io.iobase.Read",
> 13:32:02 "shortValue": "_PubSubSource",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "apache_beam.io.gcp.pubsub._PubSubSource"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "with_attributes",
> 13:32:02 "label": "With Attributes",
> 13:32:02 "namespace": "apache_beam.io.gcp.pubsub._PubSubSource",
> 13:32:02 "type": "BOOLEAN",
> 13:32:02 "value": false
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "subscription",
> 13:32:02 "label": "Pubsub Subscription",
> 13:32:02 "namespace": "apache_beam.io.gcp.pubsub._PubSubSource",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "projects/apache-beam-testing/subscriptions/wc_subscription_inputb7a2edca-b896-496f-9ab7-4fdfee001097"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "format": "pubsub",
> 13:32:02 "output_info": [
> 13:32:02 {
> 13:32:02 "encoding": {
> 13:32:02 "@type": "kind:windowed_value",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type": "kind:bytes"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:global_window"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_wrapper": true
> 13:32:02 },
> 13:32:02 "output_name": "out",
> 13:32:02 "user_name": "ReadFromPubSub/Read.out"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "pubsub_subscription":
> "projects/apache-beam-testing/subscriptions/wc_subscription_inputb7a2edca-b896-496f-9ab7-4fdfee001097",
>
> 13:32:02 "user_name": "ReadFromPubSub/Read"
> 13:32:02 }
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "kind": "ParallelDo",
> 13:32:02 "name": "s2",
> 13:32:02 "properties": {
> 13:32:02 "display_data": [
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace":
> "apache_beam.transforms.core.CallableWrapperDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "<lambda>"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
> 13:32:02 "shortValue": "CallableWrapperDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "apache_beam.transforms.core.CallableWrapperDoFn"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "non_parallel_inputs": {},
> 13:32:02 "output_info": [
> 13:32:02 {
> 13:32:02 "encoding": {
> 13:32:02 "@type": "kind:windowed_value",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_FastPrimitivesCoder_3"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_FastPrimitivesCoder_3"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_pair_like": true,
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_FastPrimitivesCoder_3"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:global_window"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_wrapper": true
> 13:32:02 },
> 13:32:02 "output_name": "out",
> 13:32:02 "user_name": "decode.out"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "parallel_input": {
> 13:32:02 "@type": "OutputReference",
> 13:32:02 "output_name": "out",
> 13:32:02 "step_name": "s1"
> 13:32:02 },
> 13:32:02 "serialized_fn": "ref_AppliedPTransform_decode_4",
> 13:32:02 "user_name": "decode"
> 13:32:02 }
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "kind": "ParallelDo",
> 13:32:02 "name": "s3",
> 13:32:02 "properties": {
> 13:32:02 "display_data": [
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
> 13:32:02 "shortValue": "WordExtractingDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "apache_beam.examples.wordcount.WordExtractingDoFn"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "non_parallel_inputs": {},
> 13:32:02 "output_info": [
> 13:32:02 {
> 13:32:02 "encoding": {
> 13:32:02 "@type": "kind:windowed_value",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type":
> "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_StrUtf8Coder_4"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:global_window"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_wrapper": true
> 13:32:02 },
> 13:32:02 "output_name": "out",
> 13:32:02 "user_name": "split.out"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "parallel_input": {
> 13:32:02 "@type": "OutputReference",
> 13:32:02 "output_name": "out",
> 13:32:02 "step_name": "s2"
> 13:32:02 },
> 13:32:02 "serialized_fn": "ref_AppliedPTransform_split_5",
> 13:32:02 "user_name": "split"
> 13:32:02 }
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "kind": "ParallelDo",
> 13:32:02 "name": "s4",
> 13:32:02 "properties": {
> 13:32:02 "display_data": [
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace":
> "apache_beam.transforms.core.CallableWrapperDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "<lambda>"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
> 13:32:02 "shortValue": "CallableWrapperDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "apache_beam.transforms.core.CallableWrapperDoFn"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "non_parallel_inputs": {},
> 13:32:02 "output_info": [
> 13:32:02 {
> 13:32:02 "encoding": {
> 13:32:02 "@type": "kind:windowed_value",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type": "kind:pair",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type":
> "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_StrUtf8Coder_4"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:varint"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_pair_like": true
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:global_window"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_wrapper": true
> 13:32:02 },
> 13:32:02 "output_name": "out",
> 13:32:02 "user_name": "pair_with_one.out"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "parallel_input": {
> 13:32:02 "@type": "OutputReference",
> 13:32:02 "output_name": "out",
> 13:32:02 "step_name": "s3"
> 13:32:02 },
> 13:32:02 "serialized_fn": "ref_AppliedPTransform_pair_with_one_6",
> 13:32:02 "user_name": "pair_with_one"
> 13:32:02 }
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "kind": "ParallelDo",
> 13:32:02 "name": "s5",
> 13:32:02 "properties": {
> 13:32:02 "display_data": [
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace": "apache_beam.transforms.core.WindowInto",
> 13:32:02 "shortValue": "WindowIntoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "apache_beam.transforms.core.WindowIntoFn"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "non_parallel_inputs": {},
> 13:32:02 "output_info": [
> 13:32:02 {
> 13:32:02 "encoding": {
> 13:32:02 "@type": "kind:windowed_value",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type": "kind:pair",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type":
> "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_StrUtf8Coder_4"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:varint"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_pair_like": true
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:interval_window"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_wrapper": true
> 13:32:02 },
> 13:32:02 "output_name": "out",
> 13:32:02 "user_name": "WindowInto(WindowIntoFn).out"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "parallel_input": {
> 13:32:02 "@type": "OutputReference",
> 13:32:02 "output_name": "out",
> 13:32:02 "step_name": "s4"
> 13:32:02 },
> 13:32:02 "serialized_fn": "<string of 436 bytes>",
> 13:32:02 "user_name": "WindowInto(WindowIntoFn)"
> 13:32:02 }
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "kind": "GroupByKey",
> 13:32:02 "name": "s6",
> 13:32:02 "properties": {
> 13:32:02 "display_data": [],
> 13:32:02 "output_info": [
> 13:32:02 {
> 13:32:02 "encoding": {
> 13:32:02 "@type": "kind:windowed_value",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type": "kind:pair",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type":
> "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_StrUtf8Coder_4"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:stream",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type": "kind:varint"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_stream_like": true
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_pair_like": true
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:interval_window"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_wrapper": true
> 13:32:02 },
> 13:32:02 "output_name": "out",
> 13:32:02 "user_name": "group.out"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "parallel_input": {
> 13:32:02 "@type": "OutputReference",
> 13:32:02 "output_name": "out",
> 13:32:02 "step_name": "s5"
> 13:32:02 },
> 13:32:02 "serialized_fn":
> "%0AH%22F%0A%1Fref_Coder_IntervalWindowCoder_1%12%23%0A%21%0A%1F%0A%1Dbeam%3Acoder%3Ainterval_window%3Av1j%5D%0A%2C%0A%2A%0A%20beam%3Awindowfn%3Afixed_windows%3Av0.1%1A%06%0A%02%08%0F%12%00%10%01%1A%1Fref_Coder_IntervalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
>
> 13:32:02 "user_name": "group"
> 13:32:02 }
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "kind": "ParallelDo",
> 13:32:02 "name": "s7",
> 13:32:02 "properties": {
> 13:32:02 "display_data": [
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace":
> "apache_beam.transforms.core.CallableWrapperDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "count_ones"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
> 13:32:02 "shortValue": "CallableWrapperDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "apache_beam.transforms.core.CallableWrapperDoFn"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "non_parallel_inputs": {},
> 13:32:02 "output_info": [
> 13:32:02 {
> 13:32:02 "encoding": {
> 13:32:02 "@type": "kind:windowed_value",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type": "kind:pair",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type":
> "StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_StrUtf8Coder_4"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_FastPrimitivesCoder_3"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_FastPrimitivesCoder_3"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_pair_like": true,
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_FastPrimitivesCoder_3"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_pair_like": true
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:interval_window"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_wrapper": true
> 13:32:02 },
> 13:32:02 "output_name": "out",
> 13:32:02 "user_name": "count.out"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "parallel_input": {
> 13:32:02 "@type": "OutputReference",
> 13:32:02 "output_name": "out",
> 13:32:02 "step_name": "s6"
> 13:32:02 },
> 13:32:02 "serialized_fn": "ref_AppliedPTransform_count_9",
> 13:32:02 "user_name": "count"
> 13:32:02 }
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "kind": "ParallelDo",
> 13:32:02 "name": "s8",
> 13:32:02 "properties": {
> 13:32:02 "display_data": [
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace":
> "apache_beam.transforms.core.CallableWrapperDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "format_result"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
> 13:32:02 "shortValue": "CallableWrapperDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "apache_beam.transforms.core.CallableWrapperDoFn"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "non_parallel_inputs": {},
> 13:32:02 "output_info": [
> 13:32:02 {
> 13:32:02 "encoding": {
> 13:32:02 "@type": "kind:windowed_value",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_FastPrimitivesCoder_3"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type":
> "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
>
> 13:32:02 "component_encodings": [],
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_FastPrimitivesCoder_3"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_pair_like": true,
> 13:32:02 "pipeline_proto_coder_id":
> "ref_Coder_FastPrimitivesCoder_3"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:interval_window"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_wrapper": true
> 13:32:02 },
> 13:32:02 "output_name": "out",
> 13:32:02 "user_name": "format.out"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "parallel_input": {
> 13:32:02 "@type": "OutputReference",
> 13:32:02 "output_name": "out",
> 13:32:02 "step_name": "s7"
> 13:32:02 },
> 13:32:02 "serialized_fn": "ref_AppliedPTransform_format_10",
> 13:32:02 "user_name": "format"
> 13:32:02 }
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "kind": "ParallelDo",
> 13:32:02 "name": "s9",
> 13:32:02 "properties": {
> 13:32:02 "display_data": [
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace":
> "apache_beam.transforms.core.CallableWrapperDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value": "<lambda>"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "key": "fn",
> 13:32:02 "label": "Transform Function",
> 13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
> 13:32:02 "shortValue": "CallableWrapperDoFn",
> 13:32:02 "type": "STRING",
> 13:32:02 "value":
> "apache_beam.transforms.core.CallableWrapperDoFn"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "non_parallel_inputs": {},
> 13:32:02 "output_info": [
> 13:32:02 {
> 13:32:02 "encoding": {
> 13:32:02 "@type": "kind:windowed_value",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type": "kind:bytes"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:interval_window"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_wrapper": true
> 13:32:02 },
> 13:32:02 "output_name": "out",
> 13:32:02 "user_name": "encode.out"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "parallel_input": {
> 13:32:02 "@type": "OutputReference",
> 13:32:02 "output_name": "out",
> 13:32:02 "step_name": "s8"
> 13:32:02 },
> 13:32:02 "serialized_fn": "ref_AppliedPTransform_encode_11",
> 13:32:02 "user_name": "encode"
> 13:32:02 }
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "kind": "ParallelWrite",
> 13:32:02 "name": "s10",
> 13:32:02 "properties": {
> 13:32:02 "display_data": [],
> 13:32:02 "encoding": {
> 13:32:02 "@type": "kind:windowed_value",
> 13:32:02 "component_encodings": [
> 13:32:02 {
> 13:32:02 "@type": "kind:bytes"
> 13:32:02 },
> 13:32:02 {
> 13:32:02 "@type": "kind:global_window"
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "is_wrapper": true
> 13:32:02 },
> 13:32:02 "format": "pubsub",
> 13:32:02 "parallel_input": {
> 13:32:02 "@type": "OutputReference",
> 13:32:02 "output_name": "out",
> 13:32:02 "step_name": "s9"
> 13:32:02 },
> 13:32:02 "pubsub_topic":
> "projects/apache-beam-testing/topics/wc_topic_outputb7a2edca-b896-496f-9ab7-4fdfee001097",
>
> 13:32:02 "user_name": "WriteToPubSub/Write/NativeWrite"
> 13:32:02 }
> 13:32:02 }
> 13:32:02 ],
> 13:32:02 "type": "JOB_TYPE_STREAMING"
> 13:32:02 }
> 13:32:02 root: INFO: Create job: <Job
> 13:32:02 createTime: u'2019-05-17T19:40:28.428825Z'
> 13:32:02 currentStateTime: u'1970-01-01T00:00:00Z'
> 13:32:02 id: u'2019-05-17_12_40_27-16144646158774694719'
> 13:32:02 location: u'us-central1'
> 13:32:02 name: u'beamapp-jenkins-0517194018-707115'
> 13:32:02 projectId: u'apache-beam-testing'
> 13:32:02 stageStates: []
> 13:32:02 startTime: u'2019-05-17T19:40:28.428825Z'
> 13:32:02 steps: []
> 13:32:02 tempFiles: []
> 13:32:02 type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
> 13:32:02 root: INFO: Created job with id:
> [2019-05-17_12_40_27-16144646158774694719]
> 13:32:02 root: INFO: To access the Dataflow monitoring console, please
> navigate to
> https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-17_12_40_27-16144646158774694719?project=apache-beam-testing
> 13:32:02 root: INFO: Job 2019-05-17_12_40_27-16144646158774694719 is in state
> JOB_STATE_RUNNING
> 13:32:02 root: INFO: 2019-05-17T19:40:31.054Z: JOB_MESSAGE_DETAILED: Checking
> permissions granted to controller Service Account.
> 13:32:02 root: INFO: 2019-05-17T19:40:31.699Z: JOB_MESSAGE_BASIC: Worker
> configuration: n1-standard-4 in us-central1-a.
> 13:32:02 root: INFO: 2019-05-17T19:40:32.189Z: JOB_MESSAGE_DETAILED:
> Expanding SplittableParDo operations into optimizable parts.
> 13:32:02 root: INFO: 2019-05-17T19:40:32.192Z: JOB_MESSAGE_DETAILED:
> Expanding CollectionToSingleton operations into optimizable parts.
> 13:32:02 root: INFO: 2019-05-17T19:40:32.202Z: JOB_MESSAGE_DETAILED:
> Expanding CoGroupByKey operations into optimizable parts.
> 13:32:02 root: INFO: 2019-05-17T19:40:32.208Z: JOB_MESSAGE_DETAILED:
> Expanding SplittableProcessKeyed operations into optimizable parts.
> 13:32:02 root: INFO: 2019-05-17T19:40:32.211Z: JOB_MESSAGE_DETAILED:
> Expanding GroupByKey operations into streaming Read/Write steps
> 13:32:02 root: INFO: 2019-05-17T19:40:32.219Z: JOB_MESSAGE_DEBUG: Annotating
> graph with Autotuner information.
> 13:32:02 root: INFO: 2019-05-17T19:40:32.232Z: JOB_MESSAGE_DETAILED: Fusing
> adjacent ParDo, Read, Write, and Flatten operations
> 13:32:02 root: INFO: 2019-05-17T19:40:32.235Z: JOB_MESSAGE_DETAILED: Fusing
> consumer decode into ReadFromPubSub/Read
> 13:32:02 root: INFO: 2019-05-17T19:40:32.238Z: JOB_MESSAGE_DETAILED: Fusing
> consumer pair_with_one into split
> 13:32:02 root: INFO: 2019-05-17T19:40:32.240Z: JOB_MESSAGE_DETAILED: Fusing
> consumer count into group/MergeBuckets
> 13:32:02 root: INFO: 2019-05-17T19:40:32.243Z: JOB_MESSAGE_DETAILED: Fusing
> consumer WriteToPubSub/Write/NativeWrite into encode
> 13:32:02 root: INFO: 2019-05-17T19:40:32.245Z: JOB_MESSAGE_DETAILED: Fusing
> consumer encode into format
> 13:32:02 root: INFO: 2019-05-17T19:40:32.247Z: JOB_MESSAGE_DETAILED: Fusing
> consumer group/MergeBuckets into group/ReadStream
> 13:32:02 root: INFO: 2019-05-17T19:40:32.250Z: JOB_MESSAGE_DETAILED: Fusing
> consumer format into count
> 13:32:02 root: INFO: 2019-05-17T19:40:32.252Z: JOB_MESSAGE_DETAILED: Fusing
> consumer group/WriteStream into WindowInto(WindowIntoFn)
> 13:32:02 root: INFO: 2019-05-17T19:40:32.254Z: JOB_MESSAGE_DETAILED: Fusing
> consumer split into decode
> 13:32:02 root: INFO: 2019-05-17T19:40:32.256Z: JOB_MESSAGE_DETAILED: Fusing
> consumer WindowInto(WindowIntoFn) into pair_with_one
> 13:32:02 root: INFO: 2019-05-17T19:40:32.268Z: JOB_MESSAGE_DEBUG: Adding
> StepResource setup and teardown to workflow graph.
> 13:32:02 root: INFO: 2019-05-17T19:40:32.283Z: JOB_MESSAGE_DEBUG: Adding
> workflow start and stop steps.
> 13:32:02 root: INFO: 2019-05-17T19:40:32.296Z: JOB_MESSAGE_DEBUG: Assigning
> stage ids.
> 13:32:02 root: INFO: 2019-05-17T19:40:32.457Z: JOB_MESSAGE_DEBUG: Executing
> wait step start2
> 13:32:02 root: INFO: 2019-05-17T19:40:32.490Z: JOB_MESSAGE_DEBUG: Starting
> worker pool setup.
> 13:32:02 root: INFO: 2019-05-17T19:40:32.496Z: JOB_MESSAGE_BASIC: Starting 1
> workers...
> 13:32:02 root: INFO: 2019-05-17T19:40:34.639Z: JOB_MESSAGE_BASIC: Executing
> operation
> group/ReadStream+group/MergeBuckets+count+format+encode+WriteToPubSub/Write/NativeWrite
> 13:32:02 root: INFO: 2019-05-17T19:40:34.639Z: JOB_MESSAGE_BASIC: Executing
> operation
> ReadFromPubSub/Read+decode+split+pair_with_one+WindowInto(WindowIntoFn)+group/WriteStream
> 13:32:02 root: INFO: 2019-05-17T19:41:03.592Z: JOB_MESSAGE_DETAILED: Workers
> have started successfully.
> 13:32:02 root: INFO: 2019-05-17T19:41:47.135Z: JOB_MESSAGE_DEBUG: Executing
> input step topology_init_attach_disk_input_step
> 13:32:02 root: WARNING: Timing out on waiting for job
> 2019-05-17_12_40_27-16144646158774694719 after 184 seconds
> 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
> http://169.254.169.254
> 13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/project/project-id
> 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
> 13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
> metadata.google.internal:80
> 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
> "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true
> HTTP/1.1" 200 144
> 13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token
> 13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
> "GET
> /computeMetadata/v1/instance/service-accounts/[email protected]/token
> HTTP/1.1" 200 176
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
