[ https://issues.apache.org/jira/browse/BEAM-7822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pablo Estrada resolved BEAM-7822. --------------------------------- Fix Version/s: 2.16.0 Resolution: Fixed > TriggerCopyJobs in BQ file loads is not atmomic in case of failure > ------------------------------------------------------------------ > > Key: BEAM-7822 > URL: https://issues.apache.org/jira/browse/BEAM-7822 > Project: Beam > Issue Type: Test > Components: io-py-gcp > Affects Versions: 2.13.0 > Reporter: Tanay Tummalapalli > Assignee: Tanay Tummalapalli > Priority: Minor > Fix For: 2.16.0 > > > Scenario: > If temp_tables are being used, during copying data from the temp table to the > destination table, if there is a failure in BigQuery mid-way through > execution, it will raise an Exception causing the pipeline to fail. As a > result, some temp_tables will be copied and some will not be. When the > pipeline is rerun, it will cause the same data to be written to new > temp_tables and copy jobs will be triggered to copy this data to the > destination table. > This will result in duplicate data being written to the BigQuery destination > table. -- This message was sent by Atlassian Jira (v8.3.2#803003)