[
https://issues.apache.org/jira/browse/BEAM-10853?focusedWorklogId=478791&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-478791
]
ASF GitHub Bot logged work on BEAM-10853:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Sep/20 19:57
Start Date: 03/Sep/20 19:57
Worklog Time Spent: 10m
Work Description: pabloem commented on a change in pull request #12761:
URL: https://github.com/apache/beam/pull/12761#discussion_r483220143
##########
File path: sdks/python/apache_beam/io/gcp/bigquery_file_loads.py
##########
@@ -356,8 +359,13 @@ def process(self, element, job_name_prefix=None):
copy_from_reference.projectId = vp.RuntimeValueProvider.get_value(
'project', str, '')
- copy_job_name = '%s_%s' % (
+ copy_job_name = '%s_%s_%s' % (
job_name_prefix,
+ _bq_uuid(
+ '%s:%s.%s' % (
Review comment:
Addressing the q about why this did not work - the BQ job names are
generated by joining a prefix with the hashof a table name. COPY jobs are
necessary when we write to multiple temporary tables, and copy to a single
destination table.
Because we were hashing the destination table name, ALL the copy jobs had
the same destination, so they had the same hash, and thus the same job name -
and this would cause a failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 478791)
Time Spent: 1h (was: 50m)
> Copy jobs for multi-table imports to BQ have duplicated names.
> --------------------------------------------------------------
>
> Key: BEAM-10853
> URL: https://issues.apache.org/jira/browse/BEAM-10853
> Project: Beam
> Issue Type: Improvement
> Components: io-py-gcp
> Reporter: Pablo Estrada
> Assignee: Pablo Estrada
> Priority: P2
> Time Spent: 1h
> Remaining Estimate: 0h
>
> A user has reported this issue, and I've written a fix here:
> https://github.com/apache/beam/pull/12761
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
