reuvenlax commented on PR #17423: URL: https://github.com/apache/beam/pull/17423#issuecomment-1133459531
We're not pushing all errors, just the SchemaConversionExceptions which are thrown when converting the json to a proto. If a downstream ParDo threw our internal SchemaConversionException, that would be a very weird thing to do (we could make it package private to ensure this can't happen). On Fri, May 20, 2022 at 7:11 PM Chamikara Jayalath ***@***.***> wrote: > ***@***.**** commented on this pull request. > > Thanks. > ------------------------------ > > In > sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiConvertMessages.java > <https://github.com/apache/beam/pull/17423#discussion_r878585977>: > > > throws Exception { > dynamicDestinations.setSideInputAccessorFromProcessContext(c); > MessageConverter<ElementT> messageConverter = > messageConverters.get( > element.getKey(), dynamicDestinations, getDatasetService(pipelineOptions)); > - StorageApiWritePayload payload = messageConverter.toMessage(element.getValue()); > - o.output(KV.of(element.getKey(), payload)); > + try { > + StorageApiWritePayload payload = messageConverter.toMessage(element.getValue()); > + o.get(successfulWritesTag).output(KV.of(element.getKey(), payload)); > + } catch (TableRowToStorageApiProto.SchemaConversionException e) { > + TableRow tableRow = messageConverter.toTableRow(element.getValue()); > > I'm bit worried about just pushing all messages from an exception handler > to a DLQ. > > (1) This could result in errors from downstream fused steps being sent to > DQL instead of being retried. > (2)Messages being send to a DLQ in an unintended way may be perceived as > dataloss by a user of the I/O connector. > > I think we should build a retry policy around this (or use existing BQ > retry policy) so that users explicitly mark messages that should be sent to > a DLQ. > > WDYT ? > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/beam/pull/17423#pullrequestreview-980722306>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFAYJVKKQVNK44CGTVGR2C3VLAL3VANCNFSM5T6AVIFA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
