kennknowles opened a new issue, #18706:
URL: https://github.com/apache/beam/issues/18706
My workflow : KAFKA -\> Dataflow streaming -\> BigQuery
Given that having low-latency isn't important in my case, I use FILE_LOADS
to reduce the costs. I'm using _BigQueryIO.Write_ with a _DynamicDestination_,
which is a table with the current hour as a suffix.
This _BigQueryIO.Write_ is configured like this :
```
.withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
.withMethod(Method.FILE_LOADS)
.withTriggeringFrequency(triggeringFrequency)
.withNumFileShards(100)
```
The first table is successfully created and is written to. But then the
following tables are never created and I get these exceptions:
```
(99e5cd8c66414e7a): java.lang.RuntimeException: Failed to create load job
with id prefix
5047f71312a94bf3a42ee5d67feede75_5295fbf25e1a7534f85e25dcaa9f4986_00001_00023,
reached max retries: 3, last failed load job: {
"configuration" : {
"load" : {
"createDisposition"
: "CREATE_NEVER",
"destinationTable" : {
"datasetId" : "dev_mydataset",
"projectId"
: "myproject-id",
"tableId" : "mytable_20180302_16"
},
```
The _CreateDisposition_ used is _CREATE_NEVER_, contrary as
_CREATE_IF_NEEDED_ as specified.
Imported from Jira
[BEAM-3772](https://issues.apache.org/jira/browse/BEAM-3772). Original Jira may
contain additional context.
Reported by: benjben.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
