[
https://issues.apache.org/jira/browse/BEAM-9148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-9148:
-------------------------------
Assignee: Boyuan Zhang
> test flakiness: BigQueryQueryToTableIT.test_big_query_standard_sql
> ------------------------------------------------------------------
>
> Key: BEAM-9148
> URL: https://issues.apache.org/jira/browse/BEAM-9148
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp, sdk-py-core, test-failures
> Reporter: Udi Meiri
> Assignee: Boyuan Zhang
> Priority: Major
>
> There might be other flaky test cases from the same class, but I'm focusing
> on test_big_query_standard_sql here.
> {code}
> 19:39:12
> ======================================================================
> 19:39:12 FAIL: test_big_query_standard_sql
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 19:39:12
> ----------------------------------------------------------------------
> 19:39:12 Traceback (most recent call last):
> 19:39:12 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
> line 172, in test_big_query_standard_sql
> 19:39:12 big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 19:39:12 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
> line 84, in run_bq_pipeline
> 19:39:12 result = p.run()
> 19:39:12 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 112, in run
> 19:39:12 else test_runner_api))
> 19:39:12 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 461, in run
> 19:39:12 self._options).run(False)
> 19:39:12 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 474, in run
> 19:39:12 return self.runner.run_pipeline(self, self._options)
> 19:39:12 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 53, in run_pipeline
> 19:39:12 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 19:39:12 AssertionError:
> 19:39:12 Expected: (Test pipeline expected terminated in state: DONE and
> Expected checksum is 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72)
> 19:39:12 but: Expected checksum is
> 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72 Actual checksum is
> da39a3ee5e6b4b0d3255bfef95601890afd80709
> 19:39:12
> 19:39:12 -------------------- >> begin captured logging <<
> --------------------
> 19:39:12 root: DEBUG: Unhandled type_constraint: Union[]
> 19:39:12 root: DEBUG: Unhandled type_constraint: Union[]
> 19:39:12 apache_beam.runners.direct.direct_runner: INFO: Running pipeline
> with DirectRunner.
> 19:39:12 apache_beam.io.gcp.bigquery_tools: DEBUG: Query SELECT * FROM
> (SELECT "apple" as fruit) UNION ALL (SELECT "orange" as fruit) does not
> reference any tables.
> 19:39:12 apache_beam.io.gcp.bigquery_tools: WARNING: Dataset
> apache-beam-testing:temp_dataset_90f5797bdb5f4137af750399f91a8e66 does not
> exist so we will create it as temporary with location=None
> 19:39:12 apache_beam.io.gcp.bigquery: DEBUG: Creating or getting table
> <TableReference
> 19:39:12 datasetId: 'python_query_to_table_15792323245106'
> 19:39:12 projectId: 'apache-beam-testing'
> 19:39:12 tableId: 'output_table'> with schema {'fields': [{'name': 'fruit',
> 'type': 'STRING', 'mode': 'NULLABLE'}]}.
> 19:39:12 apache_beam.io.gcp.bigquery_tools: DEBUG: Created the table with id
> output_table
> 19:39:12 apache_beam.io.gcp.bigquery_tools: INFO: Created table
> apache-beam-testing.python_query_to_table_15792323245106.output_table with
> schema <TableSchema
> 19:39:12 fields: [<TableFieldSchema
> 19:39:12 fields: []
> 19:39:12 mode: 'NULLABLE'
> 19:39:12 name: 'fruit'
> 19:39:12 type: 'STRING'>]>. Result: <Table
> 19:39:12 creationTime: 1579232328576
> 19:39:12 etag: 'WYysl6UIvc8IWMmTiiKhbg=='
> 19:39:12 id:
> 'apache-beam-testing:python_query_to_table_15792323245106.output_table'
> 19:39:12 kind: 'bigquery#table'
> 19:39:12 lastModifiedTime: 1579232328629
> 19:39:12 location: 'US'
> 19:39:12 numBytes: 0
> 19:39:12 numLongTermBytes: 0
> 19:39:12 numRows: 0
> 19:39:12 schema: <TableSchema
> 19:39:12 fields: [<TableFieldSchema
> 19:39:12 fields: []
> 19:39:12 mode: 'NULLABLE'
> 19:39:12 name: 'fruit'
> 19:39:12 type: 'STRING'>]>
> 19:39:12 selfLink:
> 'https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_query_to_table_15792323245106/tables/output_table'
> 19:39:12 tableReference: <TableReference
> 19:39:12 datasetId: 'python_query_to_table_15792323245106'
> 19:39:12 projectId: 'apache-beam-testing'
> 19:39:12 tableId: 'output_table'>
> 19:39:12 type: 'TABLE'>.
> 19:39:12 apache_beam.io.gcp.bigquery: DEBUG: Attempting to flush to all
> destinations. Total buffered: 2
> 19:39:12 apache_beam.io.gcp.bigquery: DEBUG: Flushing data to
> apache-beam-testing:python_query_to_table_15792323245106.output_table. Total
> 2 rows.
> 19:39:12 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Attempting to
> perform query SELECT fruit from
> `python_query_to_table_15792323245106.output_table`; to BQ
> 19:39:12 google.auth.transport._http_client: DEBUG: Making request: GET
> http://169.254.169.254
> 19:39:12 google.auth.transport._http_client: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/project/project-id
> 19:39:12 urllib3.util.retry: DEBUG: Converted retries value: 3 ->
> Retry(total=3, connect=None, read=None, redirect=None, status=None)
> 19:39:12 google.auth.transport.requests: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
> 19:39:12 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
> metadata.google.internal:80
> 19:39:12 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
> "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true
> HTTP/1.1" 200 144
> 19:39:12 google.auth.transport.requests: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token
> 19:39:12 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
> "GET
> /computeMetadata/v1/instance/service-accounts/[email protected]/token
> HTTP/1.1" 200 181
> 19:39:12 urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1):
> www.googleapis.com:443
> 19:39:12 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "POST
> /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None
> 19:39:12 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "GET
> /bigquery/v2/projects/apache-beam-testing/queries/032dabb2-92e3-472b-993e-f0e840ba0104?maxResults=0&location=US
> HTTP/1.1" 200 None
> 19:39:12 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "GET
> /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anon2425bb619e2b0611267244ec96ed403c7aae8d57/data
> HTTP/1.1" 200 None
> 19:39:12 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Read from given
> query (SELECT fruit from
> `python_query_to_table_15792323245106.output_table`;), total rows 0
> 19:39:12 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Generate checksum:
> da39a3ee5e6b4b0d3255bfef95601890afd80709
> 19:39:12 --------------------- >> end captured logging <<
> ---------------------
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1387/timestamps/?time=HH:mm:ss&timeZone=GMT-8&appendLog&locale=en_US
> {code}
> 04:09:22
> ======================================================================
> 04:09:22 FAIL: test_big_query_standard_sql
> (apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
> 04:09:22
> ----------------------------------------------------------------------
> 04:09:22 Traceback (most recent call last):
> 04:09:22 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
> line 172, in test_big_query_standard_sql
> 04:09:22 big_query_query_to_table_pipeline.run_bq_pipeline(options)
> 04:09:22 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
> line 84, in run_bq_pipeline
> 04:09:22 result = p.run()
> 04:09:22 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 113, in run
> 04:09:22 else test_runner_api))
> 04:09:22 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 461, in run
> 04:09:22 self._options).run(False)
> 04:09:22 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 474, in run
> 04:09:22 return self.runner.run_pipeline(self, self._options)
> 04:09:22 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 53, in run_pipeline
> 04:09:22 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 04:09:22 AssertionError:
> 04:09:22 Expected: (Test pipeline expected terminated in state: DONE and
> Expected checksum is 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72)
> 04:09:22 but: Expected checksum is
> 158a8ea1c254fcf40d4ed3e7c0242c3ea0a29e72 Actual checksum is
> da39a3ee5e6b4b0d3255bfef95601890afd80709
> 04:09:22
> 04:09:22 -------------------- >> begin captured logging <<
> --------------------
> 04:09:22 root: DEBUG: Unhandled type_constraint: Union[]
> 04:09:22 root: DEBUG: Unhandled type_constraint: Union[]
> 04:09:22 apache_beam.runners.direct.direct_runner: INFO: Running pipeline
> with DirectRunner.
> 04:09:22 apache_beam.io.gcp.bigquery_tools: DEBUG: Query SELECT * FROM
> (SELECT "apple" as fruit) UNION ALL (SELECT "orange" as fruit) does not
> reference any tables.
> 04:09:22 apache_beam.io.gcp.bigquery_tools: WARNING: Dataset
> apache-beam-testing:temp_dataset_25b82ddd70424425ac110ac6a6de303c does not
> exist so we will create it as temporary with location=None
> 04:09:22 apache_beam.io.gcp.bigquery: DEBUG: Creating or getting table
> <TableReference
> 04:09:22 datasetId: 'python_query_to_table_15791765509862'
> 04:09:22 projectId: 'apache-beam-testing'
> 04:09:22 tableId: 'output_table'> with schema {'fields': [{'name': 'fruit',
> 'type': 'STRING', 'mode': 'NULLABLE'}]}.
> 04:09:22 apache_beam.io.gcp.bigquery_tools: DEBUG: Created the table with id
> output_table
> 04:09:22 apache_beam.io.gcp.bigquery_tools: INFO: Created table
> apache-beam-testing.python_query_to_table_15791765509862.output_table with
> schema <TableSchema
> 04:09:22 fields: [<TableFieldSchema
> 04:09:22 fields: []
> 04:09:22 mode: 'NULLABLE'
> 04:09:22 name: 'fruit'
> 04:09:22 type: 'STRING'>]>. Result: <Table
> 04:09:22 creationTime: 1579176555984
> 04:09:22 etag: 'qPxrZxyNtuoTRqGhw9zi3Q=='
> 04:09:22 id:
> 'apache-beam-testing:python_query_to_table_15791765509862.output_table'
> 04:09:22 kind: 'bigquery#table'
> 04:09:22 lastModifiedTime: 1579176556029
> 04:09:22 location: 'US'
> 04:09:22 numBytes: 0
> 04:09:22 numLongTermBytes: 0
> 04:09:22 numRows: 0
> 04:09:22 schema: <TableSchema
> 04:09:22 fields: [<TableFieldSchema
> 04:09:22 fields: []
> 04:09:22 mode: 'NULLABLE'
> 04:09:22 name: 'fruit'
> 04:09:22 type: 'STRING'>]>
> 04:09:22 selfLink:
> 'https://www.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_query_to_table_15791765509862/tables/output_table'
> 04:09:22 tableReference: <TableReference
> 04:09:22 datasetId: 'python_query_to_table_15791765509862'
> 04:09:22 projectId: 'apache-beam-testing'
> 04:09:22 tableId: 'output_table'>
> 04:09:22 type: 'TABLE'>.
> 04:09:22 apache_beam.io.gcp.bigquery: DEBUG: Attempting to flush to all
> destinations. Total buffered: 2
> 04:09:22 apache_beam.io.gcp.bigquery: DEBUG: Flushing data to
> apache-beam-testing:python_query_to_table_15791765509862.output_table. Total
> 2 rows.
> 04:09:22 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Attempting to
> perform query SELECT fruit from
> `python_query_to_table_15791765509862.output_table`; to BQ
> 04:09:22 google.auth.transport._http_client: DEBUG: Making request: GET
> http://169.254.169.254
> 04:09:22 google.auth.transport._http_client: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/project/project-id
> 04:09:22 urllib3.util.retry: DEBUG: Converted retries value: 3 ->
> Retry(total=3, connect=None, read=None, redirect=None, status=None)
> 04:09:22 google.auth.transport.requests: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
> 04:09:22 urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
> metadata.google.internal:80
> 04:09:22 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
> "GET /computeMetadata/v1/instance/service-accounts/default/?recursive=true
> HTTP/1.1" 200 144
> 04:09:22 google.auth.transport.requests: DEBUG: Making request: GET
> http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token
> 04:09:22 urllib3.connectionpool: DEBUG: http://metadata.google.internal:80
> "GET
> /computeMetadata/v1/instance/service-accounts/[email protected]/token
> HTTP/1.1" 200 181
> 04:09:22 urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1):
> www.googleapis.com:443
> 04:09:22 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "POST
> /bigquery/v2/projects/apache-beam-testing/jobs HTTP/1.1" 200 None
> 04:09:22 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "GET
> /bigquery/v2/projects/apache-beam-testing/queries/1c114200-2ef8-4916-9084-ba6e38cc9e4a?maxResults=0&location=US
> HTTP/1.1" 200 None
> 04:09:22 urllib3.connectionpool: DEBUG: https://www.googleapis.com:443 "GET
> /bigquery/v2/projects/apache-beam-testing/datasets/_7357fab0f784d2a7327ddbe81cdd1f4ca7e429cd/tables/anon046b15f69112e6f2ad0b7c87ec9f1684fd8ab3bf/data
> HTTP/1.1" 200 None
> 04:09:22 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Read from given
> query (SELECT fruit from
> `python_query_to_table_15791765509862.output_table`;), total rows 0
> 04:09:22 apache_beam.io.gcp.tests.bigquery_matcher: INFO: Generate checksum:
> da39a3ee5e6b4b0d3255bfef95601890afd80709
> 04:09:22 --------------------- >> end captured logging <<
> ---------------------
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1379/timestamps/?time=HH:mm:ss&timeZone=GMT-8&appendLog&locale=en_US
> (note there are 2 test failures in this second log)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
