[
https://issues.apache.org/jira/browse/BEAM-4257?focusedWorklogId=103757&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-103757
]
ASF GitHub Bot logged work on BEAM-4257:
----------------------------------------
Author: ASF GitHub Bot
Created on: 20/May/18 17:45
Start Date: 20/May/18 17:45
Worklog Time Spent: 10m
Work Description: reuvenlax commented on issue #5341: [BEAM-4257]
Increases BigQuery streaming error information
URL: https://github.com/apache/beam/pull/5341#issuecomment-390498916
I'm not sure how to easily make this backwards compatible. Changing the
type of the output PCollection (from PCollection<TableRow> to
PCollection<SomethingElse>) is not a compatible change.
On Sun, May 20, 2018 at 5:27 AM Carlos Alonso <notificati...@github.com>
wrote:
> Hi @reuvenlax <https://github.com/reuvenlax>. Many thanks for your
> comments. Completely agree on the backwards incompatibility issue, my bad.
> Will fix it ASAP.
>
> On the other comment, regarding extending the information in the retry
> policy I think I'm not following you. My idea is that for every insertion
> error have the details on why it failed and the table it was going towards
> (we use dynamic destinations). All of them in a PCollection so that the
> pipeline can do something with them (store them in a dead-letter as of
> today for further inspection, that's why the error and the table
> destination is so important).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/5341#issuecomment-390477232>, or mute
> the thread
>
<https://github.com/notifications/unsubscribe-auth/AUGE1b21f0QV-gJG58XHtqFgoTefqOXdks5t0WE3gaJpZM4T74Ud>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 103757)
Time Spent: 1h 10m (was: 1h)
> Add error reason and table destination to BigQueryIO streaming failed inserts
> -----------------------------------------------------------------------------
>
> Key: BEAM-4257
> URL: https://issues.apache.org/jira/browse/BEAM-4257
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Carlos Alonso
> Assignee: Carlos Alonso
> Priority: Minor
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When using `BigQueryIO.Write` and getting `WriteResult.getFailedInserts()` we
> get a `PCollection<TableRow>` which is fine, but in order to properly work on
> the errors downstream having extended information such as the `InsertError`
> fields and the `TableReference` it was routed to would be really valuable.
>
> My suggestion is to create a new object that contains all that information
> and return a `PCollection` of those instead.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
