[
https://issues.apache.org/jira/browse/BEAM-11006?focusedWorklogId=497699&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-497699
]
ASF GitHub Bot logged work on BEAM-11006:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Oct/20 01:54
Start Date: 09/Oct/20 01:54
Worklog Time Spent: 10m
Work Description: dhercher commented on pull request #13055:
URL: https://github.com/apache/beam/pull/13055#issuecomment-705920722
:R @pabloem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 497699)
Remaining Estimate: 335h 40m (was: 335h 50m)
Time Spent: 20m (was: 10m)
> Allow Failsafe Handling of BigQuery Streaming Writes
> ----------------------------------------------------
>
> Key: BEAM-11006
> URL: https://issues.apache.org/jira/browse/BEAM-11006
> Project: Beam
> Issue Type: Improvement
> Components: extensions-java-gcp
> Reporter: Dylan Hercher
> Priority: P2
> Labels: bigquery, google-cloud-bigquery
> Original Estimate: 336h
> Time Spent: 20m
> Remaining Estimate: 335h 40m
>
> To allow handling of a generic failsafe (of any type) would allow a dead
> letter queue to retain the original source data rather than the cleaned
> version and could be more easily understood and re-processed.
>
> The BigQueryIO.Write currently supports `withFormatFunction` which allows for
> a serializable function to be applied to each datapoint -> TableRow. Ideally
> that same source value could be converted with a separate function:
> `withFailsafeFormatFunction` taken (InputT -> TableRow) or possibly (InputT
> -> OutputT), though the backwards compatibility of OutputT is more difficult.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
