See <https://builds.apache.org/job/beam_PostCommit_Python35/2139/display/redirect?page=changes>
Changes: [zyichi] Update BigQuery source in bigquery_tornadoes example [rohde.samuel] Add dependency comment in streaming cache [ehudm] [BEAM-1894] Remove obsolete EagerRunner test ------------------------------------------ [...truncated 8.01 MB...] INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:47.830Z: JOB_MESSAGE_BASIC: Finished operation read from datastore/Reshuffle/ReshufflePerKey/GroupByKey/Read+read from datastore/Reshuffle/ReshufflePerKey/GroupByKey/GroupByWindow+read from datastore/Reshuffle/ReshufflePerKey/FlatMap(restore_timestamps)+read from datastore/Reshuffle/RemoveRandomKeys+read from datastore/Read+Globally/CombineGlobally(CountCombineFn)/KeyWithVoid+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Reify+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:47.905Z: JOB_MESSAGE_BASIC: Executing operation Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:47.965Z: JOB_MESSAGE_BASIC: Finished operation Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.051Z: JOB_MESSAGE_BASIC: Executing operation Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Globally/CombineGlobally(CountCombineFn)/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.347Z: JOB_MESSAGE_BASIC: Finished operation read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+row to string+count/CombineGlobally(CountCombineFn)/KeyWithVoid+count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Reify+count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.432Z: JOB_MESSAGE_DEBUG: Value "read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough).cleanup_signal" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.475Z: JOB_MESSAGE_BASIC: Executing operation count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.519Z: JOB_MESSAGE_BASIC: Executing operation read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0) INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.545Z: JOB_MESSAGE_BASIC: Finished operation count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.565Z: JOB_MESSAGE_BASIC: Finished operation read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0) INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.631Z: JOB_MESSAGE_BASIC: Executing operation count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+count/CombineGlobally(CountCombineFn)/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.664Z: JOB_MESSAGE_DEBUG: Value "read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0).output" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:48.793Z: JOB_MESSAGE_BASIC: Executing operation read/_PassThroughThenCleanup/Create/Read+read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/ParDo(RemoveJsonFiles) INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:59.689Z: JOB_MESSAGE_BASIC: Finished operation Globally/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+Globally/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+Globally/CombineGlobally(CountCombineFn)/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:59.806Z: JOB_MESSAGE_DEBUG: Value "Globally/CombineGlobally(CountCombineFn)/UnKey.out" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:34:59.966Z: JOB_MESSAGE_BASIC: Executing operation Globally/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0) INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:00.008Z: JOB_MESSAGE_BASIC: Finished operation Globally/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0) INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:00.069Z: JOB_MESSAGE_DEBUG: Value "Globally/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0).output" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:00.129Z: JOB_MESSAGE_BASIC: Executing operation Globally/CombineGlobally(CountCombineFn)/DoOnce/Read+Globally/CombineGlobally(CountCombineFn)/InjectDefault/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.754Z: JOB_MESSAGE_BASIC: Finished operation Globally/CombineGlobally(CountCombineFn)/DoOnce/Read+Globally/CombineGlobally(CountCombineFn)/InjectDefault/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.842Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:01.806Z: JOB_MESSAGE_BASIC: Finished operation read/_PassThroughThenCleanup/Create/Read+read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/ParDo(RemoveJsonFiles) INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.851Z: JOB_MESSAGE_BASIC: Finished operation count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Read+count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine+count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Extract+count/CombineGlobally(CountCombineFn)/UnKey INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.960Z: JOB_MESSAGE_DEBUG: Value "count/CombineGlobally(CountCombineFn)/UnKey.out" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:06.046Z: JOB_MESSAGE_BASIC: Executing operation count/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0) INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:06.121Z: JOB_MESSAGE_BASIC: Finished operation count/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0) INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:06.208Z: JOB_MESSAGE_DEBUG: Value "count/CombineGlobally(CountCombineFn)/InjectDefault/_UnpickledSideInput(UnKey.out.0).output" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:06.353Z: JOB_MESSAGE_BASIC: Executing operation count/CombineGlobally(CountCombineFn)/DoOnce/Read+count/CombineGlobally(CountCombineFn)/InjectDefault/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.897Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:05.977Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Read+assert_that/Group/GroupByKey/GroupByWindow+assert_that/Group/Map(_merge_tagged_vals_under_key)+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:15.162Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Read+assert_that/Group/GroupByKey/GroupByWindow+assert_that/Group/Map(_merge_tagged_vals_under_key)+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:15.243Z: JOB_MESSAGE_DEBUG: Executing success step success49 INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:15.377Z: JOB_MESSAGE_DETAILED: Cleaning up. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:15.442Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:15.471Z: JOB_MESSAGE_BASIC: Stopping worker pool... INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:12.061Z: JOB_MESSAGE_BASIC: Finished operation count/CombineGlobally(CountCombineFn)/DoOnce/Read+count/CombineGlobally(CountCombineFn)/InjectDefault/InjectDefault+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:12.176Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:12.258Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:12.349Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Read+assert_that/Group/GroupByKey/GroupByWindow+assert_that/Group/Map(_merge_tagged_vals_under_key)+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:21.514Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Read+assert_that/Group/GroupByKey/GroupByWindow+assert_that/Group/Map(_merge_tagged_vals_under_key)+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:21.619Z: JOB_MESSAGE_DEBUG: Executing success step success40 INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:22.304Z: JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:22.352Z: JOB_MESSAGE_BASIC: Worker pool stopped. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:22.387Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:21.775Z: JOB_MESSAGE_DETAILED: Cleaning up. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:21.874Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:21.918Z: JOB_MESSAGE_BASIC: Stopping worker pool... INFO:apache_beam.runners.dataflow.dataflow_runner:Job 2020-04-01_11_24_57-11894884384059897893 is in state JOB_STATE_DONE INFO:apache_beam.io.gcp.tests.bigquery_matcher:Attempting to perform query SELECT name, language FROM python_bq_file_loads_1585765477684.output_table1 to BQ DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254 DEBUG:google.auth.transport._http_client:Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None) DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal:80 DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144 DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/[email protected]/token HTTP/1.1" 200 192 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bigquery.googleapis.com:443 DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/queries/3ebc1b47-a534-418d-b338-71f32978e101?maxResults=0&location=US HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anonfe808ac11b24b22de19473db82f1fcded616be8e/data HTTP/1.1" 200 None INFO:apache_beam.io.gcp.tests.bigquery_matcher:Result of query is: [('flink', 'java'), ('flink', 'scala'), ('spark', 'scala'), ('spark', 'py'), ('spark', 'scala'), ('beam', 'py'), ('beam', 'java'), ('beam', 'go')] INFO:apache_beam.io.gcp.tests.bigquery_matcher:Attempting to perform query SELECT name, foundation FROM python_bq_file_loads_1585765477684.output_table2 to BQ DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254 DEBUG:google.auth.transport._http_client:Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None) DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal:80 DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144 DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/[email protected]/token HTTP/1.1" 200 192 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bigquery.googleapis.com:443 DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/queries/db638bb8-5aaa-4f01-9458-fb04bde37e43?maxResults=0&location=US HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anon1b43990a7ebf311ab68df50df170d9add0d97d6f/data HTTP/1.1" 200 None INFO:apache_beam.io.gcp.tests.bigquery_matcher:Result of query is: [('flink', 'apache'), ('spark', 'apache'), ('beam', 'apache')] INFO:apache_beam.io.gcp.tests.bigquery_matcher:Attempting to perform query SELECT name, language FROM python_bq_file_loads_1585765477684.output_table3 to BQ DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254 DEBUG:google.auth.transport._http_client:Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None) DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal:80 DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144 DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/[email protected]/token HTTP/1.1" 200 192 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bigquery.googleapis.com:443 DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/queries/1c48bd0a-e153-4c30-b9da-1eff82e8b4aa?maxResults=0&location=US HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anon76d123eec7f074c32c066e6c140bbcfec8c20386/data HTTP/1.1" 200 None INFO:apache_beam.io.gcp.tests.bigquery_matcher:Result of query is: [('beam', 'go'), ('beam', 'py'), ('spark', 'py'), ('beam', 'java'), ('flink', 'java'), ('flink', 'scala'), ('spark', 'scala'), ('spark', 'scala')] INFO:apache_beam.io.gcp.tests.bigquery_matcher:Attempting to perform query SELECT name, foundation FROM python_bq_file_loads_1585765477684.output_table4 to BQ DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254 DEBUG:google.auth.transport._http_client:Making request: GET http://metadata.google.internal/computeMetadata/v1/project/project-id DEBUG:urllib3.util.retry:Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None) DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): metadata.google.internal:80 DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1" 200 144 DEBUG:google.auth.transport.requests:Making request: GET http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token DEBUG:urllib3.connectionpool:http://metadata.google.internal:80 "GET /computeMetadata/v1/instance/service-accounts/[email protected]/token HTTP/1.1" 200 192 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): bigquery.googleapis.com:443 DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "POST /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/queries/43cc7280-6966-4be9-9363-373020682ea7?maxResults=0&location=US HTTP/1.1" 200 None DEBUG:urllib3.connectionpool:https://bigquery.googleapis.com:443 "GET /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anon979d83537faa78c1aa3c158f35540b91c806ce04/data HTTP/1.1" 200 None INFO:apache_beam.io.gcp.tests.bigquery_matcher:Result of query is: [('beam', 'apache'), ('flink', 'apache'), ('spark', 'apache')] INFO:apache_beam.io.gcp.bigquery_file_loads_test:Deleting dataset python_bq_file_loads_1585765477684 in project apache-beam-testing INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:36.430Z: JOB_MESSAGE_DETAILED: Checking permissions granted to controller Service Account. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:37.954Z: JOB_MESSAGE_BASIC: Finished operation read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+assert_that/WindowInto(WindowIntoFn)+assert_that/ToVoidKey+assert_that/Group/pair_with_1+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.022Z: JOB_MESSAGE_DEBUG: Value "read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough).cleanup_signal" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.100Z: JOB_MESSAGE_BASIC: Executing operation read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0) INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.163Z: JOB_MESSAGE_BASIC: Finished operation read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0) INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.242Z: JOB_MESSAGE_DEBUG: Value "read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/_UnpickledSideInput(ParDo(PassThrough).cleanup_signal.0).output" materialized. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.327Z: JOB_MESSAGE_BASIC: Executing operation read/_PassThroughThenCleanup/Create/Read+read/_PassThroughThenCleanup/ParDo(RemoveJsonFiles)/ParDo(RemoveJsonFiles) WARNING:apache_beam.options.pipeline_options:--region not set; will default to us-central1. Future releases of Beam will require the user to set --region explicitly, or else have a default set via the gcloud tool. https://cloud.google.com/compute/docs/regions-zones INFO:apache_beam.io.gcp.bigquery_file_loads_test:Created dataset python_bq_file_loads_1585766138451 in project apache-beam-testing DEBUG:apache_beam.io.gcp.bigquery_tools:Created the table with id output_table1 INFO:apache_beam.io.gcp.bigquery_tools:Created table apache-beam-testing.python_bq_file_loads_1585766138451.output_table1 with schema <TableSchema fields: [<TableFieldSchema fields: [] mode: 'NULLABLE' name: 'name' type: 'STRING'>, <TableFieldSchema fields: [] mode: 'NULLABLE' name: 'language' type: 'STRING'>]>. Result: <Table creationTime: 1585766139493 etag: 'm+zyxtJvlxUouzbvg8jJTw==' id: 'apache-beam-testing:python_bq_file_loads_1585766138451.output_table1' kind: 'bigquery#table' lastModifiedTime: 1585766139638 location: 'US' numBytes: 0 numLongTermBytes: 0 numRows: 0 schema: <TableSchema fields: [<TableFieldSchema fields: [] mode: 'NULLABLE' name: 'name' type: 'STRING'>, <TableFieldSchema fields: [] mode: 'NULLABLE' name: 'language' type: 'STRING'>]> selfLink: 'https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_bq_file_loads_1585766138451/tables/output_table1' tableReference: <TableReference datasetId: 'python_bq_file_loads_1585766138451' projectId: 'apache-beam-testing' tableId: 'output_table1'> type: 'TABLE'>. DEBUG:apache_beam.io.gcp.bigquery_tools:Created the table with id output_table2 INFO:apache_beam.io.gcp.bigquery_tools:Created table apache-beam-testing.python_bq_file_loads_1585766138451.output_table2 with schema <TableSchema fields: [<TableFieldSchema fields: [] mode: 'NULLABLE' name: 'name' type: 'STRING'>, <TableFieldSchema fields: [] mode: 'NULLABLE' name: 'foundation' type: 'STRING'>]>. Result: <Table creationTime: 1585766139899 etag: 'svQ8MWgUNzEInmMgCQ/e/w==' id: 'apache-beam-testing:python_bq_file_loads_1585766138451.output_table2' kind: 'bigquery#table' lastModifiedTime: 1585766140038 location: 'US' numBytes: 0 numLongTermBytes: 0 numRows: 0 schema: <TableSchema fields: [<TableFieldSchema fields: [] mode: 'NULLABLE' name: 'name' type: 'STRING'>, <TableFieldSchema fields: [] mode: 'NULLABLE' name: 'foundation' type: 'STRING'>]> selfLink: 'https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_bq_file_loads_1585766138451/tables/output_table2' tableReference: <TableReference datasetId: 'python_bq_file_loads_1585766138451' projectId: 'apache-beam-testing' tableId: 'output_table2'> type: 'TABLE'>. WARNING:apache_beam.options.pipeline_options:--region not set; will default to us-central1. Future releases of Beam will require the user to set --region explicitly, or else have a default set via the gcloud tool. https://cloud.google.com/compute/docs/regions-zones INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.277Z: JOB_MESSAGE_DETAILED: Workers have started successfully. INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:38.308Z: JOB_MESSAGE_DETAILED: Workers have started successfully. /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35/src/sdks/python/apache_beam/io/gcp/bigquery.py:1416: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported experiments = p.options.view_as(DebugOptions).experiments or [] /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35/src/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:863: BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported temp_location = p.options.view_as(GoogleCloudOptions).temp_location WARNING:apache_beam.runners.dataflow.dataflow_runner:Typical end users should not use this worker jar feature. It can only be used when FnAPI is enabled. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/pipeline.pb... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/pipeline.pb in 0 seconds. INFO:apache_beam.runners.portability.stager:Executing command: ['/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35/src/build/gradleenv/-1734967054/bin/python', '-m', 'pip', 'download', '--dest', '/tmp/dataflow-requirements-cache', '-r', 'postcommit_requirements.txt', '--exists-action', 'i', '--no-binary', ':all:'] INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:40.611Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Create/Read+assert_that/Group/pair_with_0+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:40.701Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:40.758Z: JOB_MESSAGE_BASIC: Finished operation assert_that/Group/GroupByKey/Close INFO:apache_beam.runners.dataflow.dataflow_runner:2020-04-01T18:35:40.840Z: JOB_MESSAGE_BASIC: Executing operation assert_that/Group/GroupByKey/Read+assert_that/Group/GroupByKey/GroupByWindow+assert_that/Group/Map(_merge_tagged_vals_under_key)+assert_that/Unkey+assert_that/Match INFO:apache_beam.runners.portability.stager:Copying Beam SDK "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35/src/sdks/python/build/apache-beam.tar.gz" to staging location. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/requirements.txt... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/requirements.txt in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/six-1.14.0.tar.gz... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/six-1.14.0.tar.gz in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/parameterized-0.7.1.tar.gz... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/parameterized-0.7.1.tar.gz in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/mock-2.0.0.tar.gz... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/mock-2.0.0.tar.gz in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/pbr-5.4.4.tar.gz... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/pbr-5.4.4.tar.gz in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/funcsigs-1.0.2.tar.gz... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/funcsigs-1.0.2.tar.gz in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/PyHamcrest-1.10.1.tar.gz... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/PyHamcrest-1.10.1.tar.gz in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/dataflow_python_sdk.tar... INFO:apache_beam.runners.dataflow.internal.apiclient:Completed GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/dataflow_python_sdk.tar in 0 seconds. INFO:apache_beam.runners.dataflow.internal.apiclient:Starting GCS upload to gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0401183541-731687.1585766141.731865/dataflow-worker.jar... FATAL: command execution failed hudson.remoting.ChannelClosedException: Channel "unknown": Remote call on JNLP4-connect connection from 165.171.154.104.bc.googleusercontent.com/104.154.171.165:40434 failed. The channel is closing down or has closed down at hudson.remoting.Channel.call(Channel.java:950) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:283) at com.sun.proxy.$Proxy140.isAlive(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1150) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1142) at hudson.Launcher$ProcStarter.join(Launcher.java:470) at hudson.plugins.gradle.Gradle.perform(Gradle.java:317) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741) at hudson.model.Build$BuildExecution.build(Build.java:206) at hudson.model.Build$BuildExecution.doRun(Build.java:163) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1815) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused by: java.nio.channels.ClosedChannelException at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.NIONetworkLayer.ready(NIONetworkLayer.java:142) at org.jenkinsci.remoting.protocol.IOHub$OnReady.run(IOHub.java:795) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure ERROR: apache-beam-jenkins-12 is offline; cannot locate JDK 1.8 (latest) --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
