ahmedabu98 opened a new issue, #22920: URL: https://github.com/apache/beam/issues/22920
### What happened? During a large FILE_LOAD BQ write, the WriteRename DoFn currently takes care of both the copy job and the deletion of temp tables (the latter happening at finishBundle). This is risky because if an error occurs in the middle and the whole bundle retries, we may attempt to copy from tables that no longer exist. Example: Say, we have 10 temp tables and we receive a BQ exception when deleting the 5th table. WriteRename will retry and attempt to copy from all 10, but at this point 4 tables no longer exist. This bundle will continue retrying forever. Possible solution would be to separate the copy step and the delete step into two different DoFns so that their retries are separated as well. ### Issue Priority Priority: 2 ### Issue Component Component: io-java-gcp -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
