Valentyn Tymofieiev created BEAM-7351:
-----------------------------------------
Summary: Failure in Python streaming wordcount test: unexpected
messages received on output topic.
Key: BEAM-7351
URL: https://issues.apache.org/jira/browse/BEAM-7351
Project: Beam
Issue Type: Bug
Components: runner-dataflow, sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Ahmet Altay
Saw this in a PostCommit test today. Likely a flake, but it's a strange failure
mode and we may need to investigate this.
{noformat}
13:32:02 ======================================================================
13:32:02 FAIL: test_streaming_wordcount_it
(apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT)
13:32:02 ----------------------------------------------------------------------
13:32:02 Traceback (most recent call last):
13:32:02 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount_it_test.py",
line 110, in test_streaming_wordcount_it
13:32:02 self.test_pipeline.get_full_options_as_args(**extra_opts))
13:32:02 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/examples/streaming_wordcount.py",
line 101, in run
13:32:02 result = p.run()
13:32:02 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/pipeline.py",
line 419, in run
13:32:02 return self.runner.run_pipeline(self, self._options)
13:32:02 File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
line 68, in run_pipeline
13:32:02 hc_assert_that(self.result, pickler.loads(on_success_matcher))
13:32:02 AssertionError:
13:32:02 Expected: (Test pipeline expected terminated in state: RUNNING and
Expected 500 messages.)
13:32:02 but: Expected 500 messages. Got 508 messages. Diffs (item, count):
13:32:02 Expected but not in actual: []
13:32:02 Unexpected: [(u'476: 1', 1), (u'416: 1', 1), (u'245: 1', 1), (u'478:
1', 1), (u'58: 1', 1), (u'364: 1', 1), (u'77: 1', 1), (u'283: 1', 1)]
13:32:02
13:32:02 -------------------- >> begin captured logging << --------------------
13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
http://169.254.169.254
13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/project/project-id
13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
metadata.google.internal:80
13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET
/computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1"
200 144
13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token
13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET
/computeMetadata/v1/instance/service-accounts/[email protected]/token
HTTP/1.1" 200 176
13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
http://169.254.169.254
13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/project/project-id
13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
metadata.google.internal:80
13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET
/computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1"
200 144
13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token
13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET
/computeMetadata/v1/instance/service-accounts/[email protected]/token
HTTP/1.1" 200 176
13:32:02 root: DEBUG: Injecting 500 numbers to topic
projects/apache-beam-testing/topics/wc_topic_inputb7a2edca-b896-496f-9ab7-4fdfee001097
13:32:02 google.cloud.pubsub_v1.publisher._batch.thread: DEBUG: Monitor is
waking up
13:32:02 root: DEBUG: Connecting using Google Application Default Credentials.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb...
13:32:02 oauth2client.transport: INFO: Attempting refresh to obtain initial
access_token
13:32:02 oauth2client.transport: INFO: Attempting refresh to obtain initial
access_token
13:32:02 google.cloud.pubsub_v1.publisher._batch.thread: DEBUG: gRPC Publish
took 0.253291845322 seconds.
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt
in 0 seconds.
13:32:02 root: INFO: Executing command:
['/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/build/gradleenv/1922375555/bin/python',
'-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r',
'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:']
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session
in 0 seconds.
13:32:02 root: INFO: Copying Beam SDK "build/apache-beam.tar.gz" to staging
location.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar
in 0 seconds.
13:32:02 root: INFO: Starting GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar...
13:32:02 root: INFO: Completed GCS upload to
gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar
in 3 seconds.
13:32:02 root: WARNING: Discarding unparseable args: ['--sleep_secs=20',
'--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
13:32:02 root: WARNING: Discarding unparseable args: ['--sleep_secs=20',
'--kms_key_name=projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test']
13:32:02 root: DEBUG: JOB: {
13:32:02 "environment": {
13:32:02 "clusterManagerApiService": "compute.googleapis.com",
13:32:02 "dataset": "bigquery.googleapis.com/cloud_dataflow",
13:32:02 "experiments": [
13:32:02 "use_staged_dataflow_worker_jar",
13:32:02 "use_multiple_sdk_containers"
13:32:02 ],
13:32:02 "sdkPipelineOptions": {
13:32:02 "display_data": [
13:32:02 {
13:32:02 "key": "requirements_file",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value": "postcommit_requirements.txt"
13:32:02 },
13:32:02 {
13:32:02 "key": "sdk_location",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value": "build/apache-beam.tar.gz"
13:32:02 },
13:32:02 {
13:32:02 "key": "save_main_session",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "BOOLEAN",
13:32:02 "value": true
13:32:02 },
13:32:02 {
13:32:02 "key": "num_workers",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "INTEGER",
13:32:02 "value": 1
13:32:02 },
13:32:02 {
13:32:02 "key": "runner",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value": "TestDataflowRunner"
13:32:02 },
13:32:02 {
13:32:02 "key": "staging_location",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value":
"gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255"
13:32:02 },
13:32:02 {
13:32:02 "key": "dataflow_worker_jar",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value":
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.14.0-SNAPSHOT.jar"
13:32:02 },
13:32:02 {
13:32:02 "key": "wait_until_finish_duration",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "INTEGER",
13:32:02 "value": 180000
13:32:02 },
13:32:02 {
13:32:02 "key": "project",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value": "apache-beam-testing"
13:32:02 },
13:32:02 {
13:32:02 "key": "streaming",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "BOOLEAN",
13:32:02 "value": true
13:32:02 },
13:32:02 {
13:32:02 "key": "dataflow_kms_key",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value":
"projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test"
13:32:02 },
13:32:02 {
13:32:02 "key": "experiments",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value": "['use_staged_dataflow_worker_jar',
'use_multiple_sdk_containers']"
13:32:02 },
13:32:02 {
13:32:02 "key": "temp_location",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value":
"gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255"
13:32:02 },
13:32:02 {
13:32:02 "key": "beam_plugins",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value": "['apache_beam.io.filesystem.FileSystem',
'apache_beam.io.hadoopfilesystem.HadoopFileSystem',
'apache_beam.io.localfilesystem.LocalFileSystem',
'apache_beam.io.gcp.gcsfilesystem.GCSFileSystem',
'apache_beam.io.filesystem_test.TestingFileSystem',
'apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer',
'apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer',
'apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer',
'apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer']"
13:32:02 },
13:32:02 {
13:32:02 "key": "job_name",
13:32:02 "namespace":
"apache_beam.options.pipeline_options.PipelineOptions",
13:32:02 "type": "STRING",
13:32:02 "value": "beamapp-jenkins-0517194018-707115"
13:32:02 }
13:32:02 ],
13:32:02 "options": {
13:32:02 "beam_plugins": [
13:32:02 "apache_beam.io.filesystem.FileSystem",
13:32:02 "apache_beam.io.hadoopfilesystem.HadoopFileSystem",
13:32:02 "apache_beam.io.localfilesystem.LocalFileSystem",
13:32:02 "apache_beam.io.gcp.gcsfilesystem.GCSFileSystem",
13:32:02 "apache_beam.io.filesystem_test.TestingFileSystem",
13:32:02
"apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer",
13:32:02
"apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer",
13:32:02
"apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer",
13:32:02
"apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer"
13:32:02 ],
13:32:02 "dataflow_endpoint": "https://dataflow.googleapis.com";,
13:32:02 "dataflow_kms_key":
"projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test",
13:32:02 "dataflow_worker_jar":
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify_PR/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.14.0-SNAPSHOT.jar",
13:32:02 "direct_runner_bundle_repeat": 0,
13:32:02 "direct_runner_use_stacked_bundle": true,
13:32:02 "dry_run": false,
13:32:02 "enable_streaming_engine": false,
13:32:02 "environment_cache_millis": 0,
13:32:02 "experiments": [
13:32:02 "use_staged_dataflow_worker_jar",
13:32:02 "use_multiple_sdk_containers"
13:32:02 ],
13:32:02 "fake_flag": false,
13:32:02 "job_name": "beamapp-jenkins-0517194018-707115",
13:32:02 "m_flag": false,
13:32:02 "male": false,
13:32:02 "mock_flag": false,
13:32:02 "no_auth": false,
13:32:02 "num_workers": 1,
13:32:02 "pipelineUrl":
"gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pipeline.pb",
13:32:02 "pipeline_type_check": true,
13:32:02 "profile_cpu": false,
13:32:02 "profile_memory": false,
13:32:02 "profile_sample_rate": 1.0,
13:32:02 "project": "apache-beam-testing",
13:32:02 "region": "us-central1",
13:32:02 "requirements_file": "postcommit_requirements.txt",
13:32:02 "runner": "TestDataflowRunner",
13:32:02 "runtime_type_check": false,
13:32:02 "save_main_session": true,
13:32:02 "sdk_location": "build/apache-beam.tar.gz",
13:32:02 "sdk_worker_parallelism": 0,
13:32:02 "staging_location":
"gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255",
13:32:02 "streaming": true,
13:32:02 "style": "scrambled",
13:32:02 "temp_location":
"gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255",
13:32:02 "type_check_strictness": "DEFAULT_TO_ANY",
13:32:02 "update": false,
13:32:02 "wait_until_finish_duration": 180000
13:32:02 }
13:32:02 },
13:32:02 "serviceKmsKeyName":
"projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test",
13:32:02 "tempStoragePrefix":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0517194018-707115.1558122018.707255",
13:32:02 "userAgent": {
13:32:02 "name": "Apache Beam Python 2.7 SDK",
13:32:02 "version": "2.14.0.dev"
13:32:02 },
13:32:02 "version": {
13:32:02 "job_type": "FNAPI_STREAMING",
13:32:02 "major": "7"
13:32:02 },
13:32:02 "workerPools": [
13:32:02 {
13:32:02 "autoscalingSettings": {},
13:32:02 "dataDisks": [
13:32:02 {}
13:32:02 ],
13:32:02 "kind": "harness",
13:32:02 "numWorkers": 1,
13:32:02 "packages": [
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/requirements.txt",
13:32:02 "name": "requirements.txt"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/PyHamcrest-1.9.0.tar.gz",
13:32:02 "name": "PyHamcrest-1.9.0.tar.gz"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.5.tar.gz",
13:32:02 "name": "mock-3.0.5.tar.gz"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.4.tar.gz",
13:32:02 "name": "mock-3.0.4.tar.gz"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/setuptools-41.0.1.zip",
13:32:02 "name": "setuptools-41.0.1.zip"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pbr-5.2.0.tar.gz",
13:32:02 "name": "pbr-5.2.0.tar.gz"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-2.0.0.tar.gz",
13:32:02 "name": "mock-2.0.0.tar.gz"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/mock-3.0.3.tar.gz",
13:32:02 "name": "mock-3.0.3.tar.gz"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/six-1.12.0.tar.gz",
13:32:02 "name": "six-1.12.0.tar.gz"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/funcsigs-1.0.2.tar.gz",
13:32:02 "name": "funcsigs-1.0.2.tar.gz"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/pickled_main_session",
13:32:02 "name": "pickled_main_session"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow_python_sdk.tar",
13:32:02 "name": "dataflow_python_sdk.tar"
13:32:02 },
13:32:02 {
13:32:02 "location":
"storage.googleapis.com/temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0517194018-707115.1558122018.707255/dataflow-worker.jar",
13:32:02 "name": "dataflow-worker.jar"
13:32:02 }
13:32:02 ],
13:32:02 "taskrunnerSettings": {
13:32:02 "parallelWorkerSettings": {
13:32:02 "baseUrl": "https://dataflow.googleapis.com";,
13:32:02 "servicePath": "https://dataflow.googleapis.com";
13:32:02 }
13:32:02 },
13:32:02 "workerHarnessContainerImage":
"gcr.io/cloud-dataflow/v1beta3/python-fnapi:beam-master-20190509"
13:32:02 }
13:32:02 ]
13:32:02 },
13:32:02 "name": "beamapp-jenkins-0517194018-707115",
13:32:02 "steps": [
13:32:02 {
13:32:02 "kind": "ParallelRead",
13:32:02 "name": "s1",
13:32:02 "properties": {
13:32:02 "display_data": [
13:32:02 {
13:32:02 "key": "source",
13:32:02 "label": "Read Source",
13:32:02 "namespace": "apache_beam.io.iobase.Read",
13:32:02 "shortValue": "_PubSubSource",
13:32:02 "type": "STRING",
13:32:02 "value": "apache_beam.io.gcp.pubsub._PubSubSource"
13:32:02 },
13:32:02 {
13:32:02 "key": "with_attributes",
13:32:02 "label": "With Attributes",
13:32:02 "namespace": "apache_beam.io.gcp.pubsub._PubSubSource",
13:32:02 "type": "BOOLEAN",
13:32:02 "value": false
13:32:02 },
13:32:02 {
13:32:02 "key": "subscription",
13:32:02 "label": "Pubsub Subscription",
13:32:02 "namespace": "apache_beam.io.gcp.pubsub._PubSubSource",
13:32:02 "type": "STRING",
13:32:02 "value":
"projects/apache-beam-testing/subscriptions/wc_subscription_inputb7a2edca-b896-496f-9ab7-4fdfee001097"
13:32:02 }
13:32:02 ],
13:32:02 "format": "pubsub",
13:32:02 "output_info": [
13:32:02 {
13:32:02 "encoding": {
13:32:02 "@type": "kind:windowed_value",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type": "kind:bytes"
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:global_window"
13:32:02 }
13:32:02 ],
13:32:02 "is_wrapper": true
13:32:02 },
13:32:02 "output_name": "out",
13:32:02 "user_name": "ReadFromPubSub/Read.out"
13:32:02 }
13:32:02 ],
13:32:02 "pubsub_subscription":
"projects/apache-beam-testing/subscriptions/wc_subscription_inputb7a2edca-b896-496f-9ab7-4fdfee001097",
13:32:02 "user_name": "ReadFromPubSub/Read"
13:32:02 }
13:32:02 },
13:32:02 {
13:32:02 "kind": "ParallelDo",
13:32:02 "name": "s2",
13:32:02 "properties": {
13:32:02 "display_data": [
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace":
"apache_beam.transforms.core.CallableWrapperDoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "<lambda>"
13:32:02 },
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
13:32:02 "shortValue": "CallableWrapperDoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "apache_beam.transforms.core.CallableWrapperDoFn"
13:32:02 }
13:32:02 ],
13:32:02 "non_parallel_inputs": {},
13:32:02 "output_info": [
13:32:02 {
13:32:02 "encoding": {
13:32:02 "@type": "kind:windowed_value",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_3"
13:32:02 },
13:32:02 {
13:32:02 "@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_3"
13:32:02 }
13:32:02 ],
13:32:02 "is_pair_like": true,
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_3"
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:global_window"
13:32:02 }
13:32:02 ],
13:32:02 "is_wrapper": true
13:32:02 },
13:32:02 "output_name": "out",
13:32:02 "user_name": "decode.out"
13:32:02 }
13:32:02 ],
13:32:02 "parallel_input": {
13:32:02 "@type": "OutputReference",
13:32:02 "output_name": "out",
13:32:02 "step_name": "s1"
13:32:02 },
13:32:02 "serialized_fn": "ref_AppliedPTransform_decode_4",
13:32:02 "user_name": "decode"
13:32:02 }
13:32:02 },
13:32:02 {
13:32:02 "kind": "ParallelDo",
13:32:02 "name": "s3",
13:32:02 "properties": {
13:32:02 "display_data": [
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
13:32:02 "shortValue": "WordExtractingDoFn",
13:32:02 "type": "STRING",
13:32:02 "value":
"apache_beam.examples.wordcount.WordExtractingDoFn"
13:32:02 }
13:32:02 ],
13:32:02 "non_parallel_inputs": {},
13:32:02 "output_info": [
13:32:02 {
13:32:02 "encoding": {
13:32:02 "@type": "kind:windowed_value",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type":
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id": "ref_Coder_StrUtf8Coder_4"
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:global_window"
13:32:02 }
13:32:02 ],
13:32:02 "is_wrapper": true
13:32:02 },
13:32:02 "output_name": "out",
13:32:02 "user_name": "split.out"
13:32:02 }
13:32:02 ],
13:32:02 "parallel_input": {
13:32:02 "@type": "OutputReference",
13:32:02 "output_name": "out",
13:32:02 "step_name": "s2"
13:32:02 },
13:32:02 "serialized_fn": "ref_AppliedPTransform_split_5",
13:32:02 "user_name": "split"
13:32:02 }
13:32:02 },
13:32:02 {
13:32:02 "kind": "ParallelDo",
13:32:02 "name": "s4",
13:32:02 "properties": {
13:32:02 "display_data": [
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace":
"apache_beam.transforms.core.CallableWrapperDoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "<lambda>"
13:32:02 },
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
13:32:02 "shortValue": "CallableWrapperDoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "apache_beam.transforms.core.CallableWrapperDoFn"
13:32:02 }
13:32:02 ],
13:32:02 "non_parallel_inputs": {},
13:32:02 "output_info": [
13:32:02 {
13:32:02 "encoding": {
13:32:02 "@type": "kind:windowed_value",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type": "kind:pair",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type":
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_StrUtf8Coder_4"
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:varint"
13:32:02 }
13:32:02 ],
13:32:02 "is_pair_like": true
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:global_window"
13:32:02 }
13:32:02 ],
13:32:02 "is_wrapper": true
13:32:02 },
13:32:02 "output_name": "out",
13:32:02 "user_name": "pair_with_one.out"
13:32:02 }
13:32:02 ],
13:32:02 "parallel_input": {
13:32:02 "@type": "OutputReference",
13:32:02 "output_name": "out",
13:32:02 "step_name": "s3"
13:32:02 },
13:32:02 "serialized_fn": "ref_AppliedPTransform_pair_with_one_6",
13:32:02 "user_name": "pair_with_one"
13:32:02 }
13:32:02 },
13:32:02 {
13:32:02 "kind": "ParallelDo",
13:32:02 "name": "s5",
13:32:02 "properties": {
13:32:02 "display_data": [
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace": "apache_beam.transforms.core.WindowInto",
13:32:02 "shortValue": "WindowIntoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "apache_beam.transforms.core.WindowIntoFn"
13:32:02 }
13:32:02 ],
13:32:02 "non_parallel_inputs": {},
13:32:02 "output_info": [
13:32:02 {
13:32:02 "encoding": {
13:32:02 "@type": "kind:windowed_value",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type": "kind:pair",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type":
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_StrUtf8Coder_4"
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:varint"
13:32:02 }
13:32:02 ],
13:32:02 "is_pair_like": true
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:interval_window"
13:32:02 }
13:32:02 ],
13:32:02 "is_wrapper": true
13:32:02 },
13:32:02 "output_name": "out",
13:32:02 "user_name": "WindowInto(WindowIntoFn).out"
13:32:02 }
13:32:02 ],
13:32:02 "parallel_input": {
13:32:02 "@type": "OutputReference",
13:32:02 "output_name": "out",
13:32:02 "step_name": "s4"
13:32:02 },
13:32:02 "serialized_fn": "<string of 436 bytes>",
13:32:02 "user_name": "WindowInto(WindowIntoFn)"
13:32:02 }
13:32:02 },
13:32:02 {
13:32:02 "kind": "GroupByKey",
13:32:02 "name": "s6",
13:32:02 "properties": {
13:32:02 "display_data": [],
13:32:02 "output_info": [
13:32:02 {
13:32:02 "encoding": {
13:32:02 "@type": "kind:windowed_value",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type": "kind:pair",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type":
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_StrUtf8Coder_4"
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:stream",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type": "kind:varint"
13:32:02 }
13:32:02 ],
13:32:02 "is_stream_like": true
13:32:02 }
13:32:02 ],
13:32:02 "is_pair_like": true
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:interval_window"
13:32:02 }
13:32:02 ],
13:32:02 "is_wrapper": true
13:32:02 },
13:32:02 "output_name": "out",
13:32:02 "user_name": "group.out"
13:32:02 }
13:32:02 ],
13:32:02 "parallel_input": {
13:32:02 "@type": "OutputReference",
13:32:02 "output_name": "out",
13:32:02 "step_name": "s5"
13:32:02 },
13:32:02 "serialized_fn":
"%0AH%22F%0A%1Fref_Coder_IntervalWindowCoder_1%12%23%0A%21%0A%1F%0A%1Dbeam%3Acoder%3Ainterval_window%3Av1j%5D%0A%2C%0A%2A%0A%20beam%3Awindowfn%3Afixed_windows%3Av0.1%1A%06%0A%02%08%0F%12%00%10%01%1A%1Fref_Coder_IntervalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
13:32:02 "user_name": "group"
13:32:02 }
13:32:02 },
13:32:02 {
13:32:02 "kind": "ParallelDo",
13:32:02 "name": "s7",
13:32:02 "properties": {
13:32:02 "display_data": [
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace":
"apache_beam.transforms.core.CallableWrapperDoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "count_ones"
13:32:02 },
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
13:32:02 "shortValue": "CallableWrapperDoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "apache_beam.transforms.core.CallableWrapperDoFn"
13:32:02 }
13:32:02 ],
13:32:02 "non_parallel_inputs": {},
13:32:02 "output_info": [
13:32:02 {
13:32:02 "encoding": {
13:32:02 "@type": "kind:windowed_value",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type": "kind:pair",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type":
"StrUtf8Coder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlzBJUWhJWkWziAeVyGDZmMhY20hU5IeAAajEkY=",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_StrUtf8Coder_4"
13:32:02 },
13:32:02 {
13:32:02 "@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_3"
13:32:02 },
13:32:02 {
13:32:02 "@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_3"
13:32:02 }
13:32:02 ],
13:32:02 "is_pair_like": true,
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_3"
13:32:02 }
13:32:02 ],
13:32:02 "is_pair_like": true
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:interval_window"
13:32:02 }
13:32:02 ],
13:32:02 "is_wrapper": true
13:32:02 },
13:32:02 "output_name": "out",
13:32:02 "user_name": "count.out"
13:32:02 }
13:32:02 ],
13:32:02 "parallel_input": {
13:32:02 "@type": "OutputReference",
13:32:02 "output_name": "out",
13:32:02 "step_name": "s6"
13:32:02 },
13:32:02 "serialized_fn": "ref_AppliedPTransform_count_9",
13:32:02 "user_name": "count"
13:32:02 }
13:32:02 },
13:32:02 {
13:32:02 "kind": "ParallelDo",
13:32:02 "name": "s8",
13:32:02 "properties": {
13:32:02 "display_data": [
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace":
"apache_beam.transforms.core.CallableWrapperDoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "format_result"
13:32:02 },
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
13:32:02 "shortValue": "CallableWrapperDoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "apache_beam.transforms.core.CallableWrapperDoFn"
13:32:02 }
13:32:02 ],
13:32:02 "non_parallel_inputs": {},
13:32:02 "output_info": [
13:32:02 {
13:32:02 "encoding": {
13:32:02 "@type": "kind:windowed_value",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_3"
13:32:02 },
13:32:02 {
13:32:02 "@type":
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
13:32:02 "component_encodings": [],
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_3"
13:32:02 }
13:32:02 ],
13:32:02 "is_pair_like": true,
13:32:02 "pipeline_proto_coder_id":
"ref_Coder_FastPrimitivesCoder_3"
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:interval_window"
13:32:02 }
13:32:02 ],
13:32:02 "is_wrapper": true
13:32:02 },
13:32:02 "output_name": "out",
13:32:02 "user_name": "format.out"
13:32:02 }
13:32:02 ],
13:32:02 "parallel_input": {
13:32:02 "@type": "OutputReference",
13:32:02 "output_name": "out",
13:32:02 "step_name": "s7"
13:32:02 },
13:32:02 "serialized_fn": "ref_AppliedPTransform_format_10",
13:32:02 "user_name": "format"
13:32:02 }
13:32:02 },
13:32:02 {
13:32:02 "kind": "ParallelDo",
13:32:02 "name": "s9",
13:32:02 "properties": {
13:32:02 "display_data": [
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace":
"apache_beam.transforms.core.CallableWrapperDoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "<lambda>"
13:32:02 },
13:32:02 {
13:32:02 "key": "fn",
13:32:02 "label": "Transform Function",
13:32:02 "namespace": "apache_beam.transforms.core.ParDo",
13:32:02 "shortValue": "CallableWrapperDoFn",
13:32:02 "type": "STRING",
13:32:02 "value": "apache_beam.transforms.core.CallableWrapperDoFn"
13:32:02 }
13:32:02 ],
13:32:02 "non_parallel_inputs": {},
13:32:02 "output_info": [
13:32:02 {
13:32:02 "encoding": {
13:32:02 "@type": "kind:windowed_value",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type": "kind:bytes"
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:interval_window"
13:32:02 }
13:32:02 ],
13:32:02 "is_wrapper": true
13:32:02 },
13:32:02 "output_name": "out",
13:32:02 "user_name": "encode.out"
13:32:02 }
13:32:02 ],
13:32:02 "parallel_input": {
13:32:02 "@type": "OutputReference",
13:32:02 "output_name": "out",
13:32:02 "step_name": "s8"
13:32:02 },
13:32:02 "serialized_fn": "ref_AppliedPTransform_encode_11",
13:32:02 "user_name": "encode"
13:32:02 }
13:32:02 },
13:32:02 {
13:32:02 "kind": "ParallelWrite",
13:32:02 "name": "s10",
13:32:02 "properties": {
13:32:02 "display_data": [],
13:32:02 "encoding": {
13:32:02 "@type": "kind:windowed_value",
13:32:02 "component_encodings": [
13:32:02 {
13:32:02 "@type": "kind:bytes"
13:32:02 },
13:32:02 {
13:32:02 "@type": "kind:global_window"
13:32:02 }
13:32:02 ],
13:32:02 "is_wrapper": true
13:32:02 },
13:32:02 "format": "pubsub",
13:32:02 "parallel_input": {
13:32:02 "@type": "OutputReference",
13:32:02 "output_name": "out",
13:32:02 "step_name": "s9"
13:32:02 },
13:32:02 "pubsub_topic":
"projects/apache-beam-testing/topics/wc_topic_outputb7a2edca-b896-496f-9ab7-4fdfee001097",
13:32:02 "user_name": "WriteToPubSub/Write/NativeWrite"
13:32:02 }
13:32:02 }
13:32:02 ],
13:32:02 "type": "JOB_TYPE_STREAMING"
13:32:02 }
13:32:02 root: INFO: Create job: <Job
13:32:02 createTime: u'2019-05-17T19:40:28.428825Z'
13:32:02 currentStateTime: u'1970-01-01T00:00:00Z'
13:32:02 id: u'2019-05-17_12_40_27-16144646158774694719'
13:32:02 location: u'us-central1'
13:32:02 name: u'beamapp-jenkins-0517194018-707115'
13:32:02 projectId: u'apache-beam-testing'
13:32:02 stageStates: []
13:32:02 startTime: u'2019-05-17T19:40:28.428825Z'
13:32:02 steps: []
13:32:02 tempFiles: []
13:32:02 type: TypeValueValuesEnum(JOB_TYPE_STREAMING, 2)>
13:32:02 root: INFO: Created job with id:
[2019-05-17_12_40_27-16144646158774694719]
13:32:02 root: INFO: To access the Dataflow monitoring console, please navigate
to
https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2019-05-17_12_40_27-16144646158774694719?project=apache-beam-testing
13:32:02 root: INFO: Job 2019-05-17_12_40_27-16144646158774694719 is in state
JOB_STATE_RUNNING
13:32:02 root: INFO: 2019-05-17T19:40:31.054Z: JOB_MESSAGE_DETAILED: Checking
permissions granted to controller Service Account.
13:32:02 root: INFO: 2019-05-17T19:40:31.699Z: JOB_MESSAGE_BASIC: Worker
configuration: n1-standard-4 in us-central1-a.
13:32:02 root: INFO: 2019-05-17T19:40:32.189Z: JOB_MESSAGE_DETAILED: Expanding
SplittableParDo operations into optimizable parts.
13:32:02 root: INFO: 2019-05-17T19:40:32.192Z: JOB_MESSAGE_DETAILED: Expanding
CollectionToSingleton operations into optimizable parts.
13:32:02 root: INFO: 2019-05-17T19:40:32.202Z: JOB_MESSAGE_DETAILED: Expanding
CoGroupByKey operations into optimizable parts.
13:32:02 root: INFO: 2019-05-17T19:40:32.208Z: JOB_MESSAGE_DETAILED: Expanding
SplittableProcessKeyed operations into optimizable parts.
13:32:02 root: INFO: 2019-05-17T19:40:32.211Z: JOB_MESSAGE_DETAILED: Expanding
GroupByKey operations into streaming Read/Write steps
13:32:02 root: INFO: 2019-05-17T19:40:32.219Z: JOB_MESSAGE_DEBUG: Annotating
graph with Autotuner information.
13:32:02 root: INFO: 2019-05-17T19:40:32.232Z: JOB_MESSAGE_DETAILED: Fusing
adjacent ParDo, Read, Write, and Flatten operations
13:32:02 root: INFO: 2019-05-17T19:40:32.235Z: JOB_MESSAGE_DETAILED: Fusing
consumer decode into ReadFromPubSub/Read
13:32:02 root: INFO: 2019-05-17T19:40:32.238Z: JOB_MESSAGE_DETAILED: Fusing
consumer pair_with_one into split
13:32:02 root: INFO: 2019-05-17T19:40:32.240Z: JOB_MESSAGE_DETAILED: Fusing
consumer count into group/MergeBuckets
13:32:02 root: INFO: 2019-05-17T19:40:32.243Z: JOB_MESSAGE_DETAILED: Fusing
consumer WriteToPubSub/Write/NativeWrite into encode
13:32:02 root: INFO: 2019-05-17T19:40:32.245Z: JOB_MESSAGE_DETAILED: Fusing
consumer encode into format
13:32:02 root: INFO: 2019-05-17T19:40:32.247Z: JOB_MESSAGE_DETAILED: Fusing
consumer group/MergeBuckets into group/ReadStream
13:32:02 root: INFO: 2019-05-17T19:40:32.250Z: JOB_MESSAGE_DETAILED: Fusing
consumer format into count
13:32:02 root: INFO: 2019-05-17T19:40:32.252Z: JOB_MESSAGE_DETAILED: Fusing
consumer group/WriteStream into WindowInto(WindowIntoFn)
13:32:02 root: INFO: 2019-05-17T19:40:32.254Z: JOB_MESSAGE_DETAILED: Fusing
consumer split into decode
13:32:02 root: INFO: 2019-05-17T19:40:32.256Z: JOB_MESSAGE_DETAILED: Fusing
consumer WindowInto(WindowIntoFn) into pair_with_one
13:32:02 root: INFO: 2019-05-17T19:40:32.268Z: JOB_MESSAGE_DEBUG: Adding
StepResource setup and teardown to workflow graph.
13:32:02 root: INFO: 2019-05-17T19:40:32.283Z: JOB_MESSAGE_DEBUG: Adding
workflow start and stop steps.
13:32:02 root: INFO: 2019-05-17T19:40:32.296Z: JOB_MESSAGE_DEBUG: Assigning
stage ids.
13:32:02 root: INFO: 2019-05-17T19:40:32.457Z: JOB_MESSAGE_DEBUG: Executing
wait step start2
13:32:02 root: INFO: 2019-05-17T19:40:32.490Z: JOB_MESSAGE_DEBUG: Starting
worker pool setup.
13:32:02 root: INFO: 2019-05-17T19:40:32.496Z: JOB_MESSAGE_BASIC: Starting 1
workers...
13:32:02 root: INFO: 2019-05-17T19:40:34.639Z: JOB_MESSAGE_BASIC: Executing
operation
group/ReadStream+group/MergeBuckets+count+format+encode+WriteToPubSub/Write/NativeWrite
13:32:02 root: INFO: 2019-05-17T19:40:34.639Z: JOB_MESSAGE_BASIC: Executing
operation
ReadFromPubSub/Read+decode+split+pair_with_one+WindowInto(WindowIntoFn)+group/WriteStream
13:32:02 root: INFO: 2019-05-17T19:41:03.592Z: JOB_MESSAGE_DETAILED: Workers
have started successfully.
13:32:02 root: INFO: 2019-05-17T19:41:47.135Z: JOB_MESSAGE_DEBUG: Executing
input step topology_init_attach_disk_input_step
13:32:02 root: WARNING: Timing out on waiting for job
2019-05-17_12_40_27-16144646158774694719 after 184 seconds
13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
http://169.254.169.254
13:32:02 google.auth.transport._http_client: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/project/project-id
13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
13:32:02 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
metadata.google.internal:80
13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET
/computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1"
200 144
13:32:02 google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token
13:32:02 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET
/computeMetadata/v1/instance/service-accounts/[email protected]/token
HTTP/1.1" 200 176
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
