[
https://issues.apache.org/jira/browse/BEAM-3200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16257715#comment-16257715
]
Eugene Kirpichov commented on BEAM-3200:
----------------------------------------
Hmm, I'd expect that the code you linked would have the pane indices assigned
independently for each destination, but I'm not confident that the code
actually accomplishes that. And there is no test for DynamicDestinations in
combination with FileLoads. If it doesn't, then yes, this is a pretty serious
bug.
[~reuvenlax] wrote this and I reviewed this, but Reuven is currently traveling
and I'm overwhelmed with a lot of other things, and I don't know if there's
anyone else who knows this code well - so help would be appreciated. Some ways
you could help, if you wanted to, would be to try writing a test in
BigQueryIOTest that reproduces the problem, and/or contributing a fix.
I wonder if triggered loads should support only CREATE_IF_NEEDED and
WRITE_APPEND, and unconditionally use them?..
> Streaming Pipeline throws RuntimeException when using DynamicDestinations and
> Method.FILE_LOADS
> -----------------------------------------------------------------------------------------------
>
> Key: BEAM-3200
> URL: https://issues.apache.org/jira/browse/BEAM-3200
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp
> Affects Versions: 2.2.0
> Reporter: AJ
> Assignee: Chamikara Jayalath
> Priority: Critical
>
> I am trying to use Method.FILE_LOADS for loading data into BQ in my streaming
> pipeline using RC3 release of 2.2.0. I am writing to around 500 tables using
> DynamicDestinations and I am also using
> withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED). Everything works
> fine when the first time bigquery load jobs get triggered. But on subsequent
> triggers pipeline throws a RuntimeException about table not found even though
> I created the pipeline with CreateDisposition.CREATE_IF_NEEDED. The exact
> exception is:
> {code}
> java.lang.RuntimeException: Failed to create load job with id prefix
> 717aed9ed1ef4aa7a616e1132f8b7f6d_a0928cae3d670b32f01ab2d9fe5cc0ee_00001_00001,
> reached max retries: 3, last failed load job: {
> "configuration" : {
> "load" : {
> "createDisposition" : "CREATE_NEVER",
> "destinationTable" : {
> "datasetId" : ...,
> "projectId" : ...,
> "tableId" : ....
> },
> "errors" : [ }
> "message" : "Not found: Table ....,
> "reason" : "notFound"
> } ],
> {code}
> My theory is all the subsequent load jobs get trigged using CREATE_NEVER
> disposition and
> this might be due to
> https://github.com/apache/beam/blob/release-2.2.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L140
> When using DynamicDestinations all the destination tables might not be known
> during the first trigger and hence the pipeline's create disposition should
> be respected.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
