[
https://issues.apache.org/jira/browse/BEAM-3200?focusedWorklogId=312612&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-312612
]
ASF GitHub Bot logged work on BEAM-3200:
----------------------------------------
Author: ASF GitHub Bot
Created on: 14/Sep/19 22:26
Start Date: 14/Sep/19 22:26
Worklog Time Spent: 10m
Work Description: bcbradle commented on issue #9556: [BEAM-3200,
BEAM-3772] Fix: Honor create and write disposition with dynamic destinations
URL: https://github.com/apache/beam/pull/9556#issuecomment-531518270
This is surprising then, if the pane is per key this implies that
configuring the write and create disposition should be respected for dynamic
destinations no matter what, but that isn't what is happening and I did a trace
through the code to find where the create disposition could be overridden
without touching the other configuration-- this is the only place that lines up
with what I see in the error.
This could mean that the code is broken somewhere else and that is causing
certain assumptions about the pane being per key to break. From the point of
view I have as a user, setting the write disposition and create disposition as
I have above should produce the expected result-- there shouldn't (can't) be an
error about the load job not being able to be created because the create
disposition was 'Create Never'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 312612)
Time Spent: 1h 50m (was: 1h 40m)
> Streaming Pipeline throws RuntimeException when using DynamicDestinations and
> Method.FILE_LOADS
> -----------------------------------------------------------------------------------------------
>
> Key: BEAM-3200
> URL: https://issues.apache.org/jira/browse/BEAM-3200
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
> Affects Versions: 2.2.0
> Reporter: AJ
> Priority: Critical
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> I am trying to use Method.FILE_LOADS for loading data into BQ in my streaming
> pipeline using RC3 release of 2.2.0. I am writing to around 500 tables using
> DynamicDestinations and I am also using
> withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED). Everything works
> fine when the first time bigquery load jobs get triggered. But on subsequent
> triggers pipeline throws a RuntimeException about table not found even though
> I created the pipeline with CreateDisposition.CREATE_IF_NEEDED. The exact
> exception is:
> {code}
> java.lang.RuntimeException: Failed to create load job with id prefix
> 717aed9ed1ef4aa7a616e1132f8b7f6d_a0928cae3d670b32f01ab2d9fe5cc0ee_00001_00001,
> reached max retries: 3, last failed load job: {
> "configuration" : {
> "load" : {
> "createDisposition" : "CREATE_NEVER",
> "destinationTable" : {
> "datasetId" : ...,
> "projectId" : ...,
> "tableId" : ....
> },
> "errors" : [ }
> "message" : "Not found: Table ....,
> "reason" : "notFound"
> } ],
> {code}
> My theory is all the subsequent load jobs get trigged using CREATE_NEVER
> disposition and
> this might be due to
> https://github.com/apache/beam/blob/release-2.2.0/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java#L140
> When using DynamicDestinations all the destination tables might not be known
> during the first trigger and hence the pipeline's create disposition should
> be respected.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
