[
https://issues.apache.org/jira/browse/BEAM-12766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434051#comment-17434051
]
Kyle Weaver commented on BEAM-12766:
------------------------------------
This is affecting 2.34.0 release verification as well.
apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT.test_big_query_new_types_native
"Already Exists: Table
apache-beam-testing:python_query_to_table_16352023569588.python_new_types_table"
This may be completely unrelated, but I've recently dealt with a similar
problem where we were seeing a lot of collisions with what should have been a
large random number (BEAM-12694). I wouldn't be surprised if we are misusing
the random library somehow.
If the number isn't really random, then a timestamp with second granularity is
not enough to avoid collisions.
> Already Exists: Dataset apache-beam-testing:python_bq_file_loads_NNN
> --------------------------------------------------------------------
>
> Key: BEAM-12766
> URL: https://issues.apache.org/jira/browse/BEAM-12766
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
> Affects Versions: 2.33.0
> Reporter: Luke Cwik
> Assignee: Pablo Estrada
> Priority: P1
> Labels: currently-failing, flaky
>
> It looks like there are too many collisions in the dataset name.
> There don't seem to be many changes to these tests or the implementation in
> several months except for the [migration to nosetest to
> pytest|https://github.com/apache/beam/pull/14859]
> We seem to be generating a dataset name based upon the [second at which the
> test is run with a relatively large random value
> appended|https://github.com/apache/beam/blob/a1abdf1d16cb6c74fd2460e90e89cb3cb7c81308/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py#L737].
> Note that this same setup code is copied in multiple locations such as [BQ
> Write
> IT|https://github.com/apache/beam/blob/a1abdf1d16cb6c74fd2460e90e89cb3cb7c81308/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py#L63].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
