[
https://issues.apache.org/jira/browse/BEAM-11006?focusedWorklogId=499681&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499681
]
ASF GitHub Bot logged work on BEAM-11006:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 12/Oct/20 22:14
Start Date: 12/Oct/20 22:14
Worklog Time Spent: 10m
Work Description: dhercher commented on a change in pull request #13055:
URL: https://github.com/apache/beam/pull/13055#discussion_r503560850
##########
File path:
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java
##########
@@ -2002,6 +2006,11 @@ static String getExtractDestinationUri(String
extractDestinationDir) {
return toBuilder().setFormatFunction(formatFunction).build();
}
+ /** Formats the user's type into a {@link TableRow} to be written to an
error collector. */
+ public Write<T> withFailsafeFormatFunction(SerializableFunction<T,
TableRow> formatFunction) {
Review comment:
I've been looking into this feature and it will require a number of
additional changes. IMO it makes more sense to merge this first feature, then
add a second issue / PR for the dynamic typing as it appears to be a relatively
complex addon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 499681)
Remaining Estimate: 334.5h (was: 334h 40m)
Time Spent: 1.5h (was: 1h 20m)
> Allow Failsafe Handling of BigQuery Streaming Writes
> ----------------------------------------------------
>
> Key: BEAM-11006
> URL: https://issues.apache.org/jira/browse/BEAM-11006
> Project: Beam
> Issue Type: Improvement
> Components: extensions-java-gcp
> Reporter: Dylan Hercher
> Priority: P2
> Labels: bigquery, google-cloud-bigquery
> Original Estimate: 336h
> Time Spent: 1.5h
> Remaining Estimate: 334.5h
>
> To allow handling of a generic failsafe (of any type) would allow a dead
> letter queue to retain the original source data rather than the cleaned
> version and could be more easily understood and re-processed.
>
> The BigQueryIO.Write currently supports `withFormatFunction` which allows for
> a serializable function to be applied to each datapoint -> TableRow. Ideally
> that same source value could be converted with a separate function:
> `withFailsafeFormatFunction` taken (InputT -> TableRow) or possibly (InputT
> -> OutputT), though the backwards compatibility of OutputT is more difficult.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
