ahmedabu98 opened a new issue, #28309:
URL: https://github.com/apache/beam/issues/28309
### What happened?
Was testing FILE_LOADS streaming writes and found that when dynamic
destinations are set and copy jobs are used (ie. large data) and
CREATE_IF_NEEDED is set, only the first table is created. For example, if I'm
writing to two tables A and B, it becomes a race condition on which copy job is
seen first in the pipeline. If copy job to table A is performed first, then
table A will be created and all subsequent copy jobs to table B will fail with
an error similar to the following:
```
WARNING: Load job
beam_bq_job_COPY_testpipelineahmedabualsaud0905154941c26eb941_17a94e2694554455aa31cca9f9389b49_4e2479b4160d04b56a8075645f4974e1_00003-0
failed, will retry: {
"errorResult" : {
"message" : "Not found: Table <project>:<dataset>.mytable_B",
"reason" : "notFound"
},
"errors" : [ {
"message" : "Not found: Table <project>:<dataset>.mytable_B",
"reason" : "notFound"
} ],
"state" : "DONE"
}. Next job id
beam_bq_job_COPY_testpipelineahmedabualsaud0905154941c26eb941_17a94e2694554455aa31cca9f9389b49_4e2479b4160d04b56a8075645f4974e1_00003-1
```
What we would expect instead is for all tables to be created.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
