[
https://issues.apache.org/jira/browse/BEAM-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361127#comment-17361127
]
Zachary Houfek commented on BEAM-12474:
---------------------------------------
My initial thought was to modify
[PubsubIO.Read#expand|https://github.com/apache/beam/blob/815e2c94bee24ef0c15c78876d6266618c238ba5/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java#L800]
to catch errors in the MapElements call. If Read was configured with a
dead-letter topic, then it would add a PubsubIO.Write transform to write to the
dead-letter in the pipeline. If not, errors would simply be logged. Either way,
it would return successful results similar to what it does now.
Or is there a reason for adding it to PayloadSerializer as opposed to PubsubIO
directly?
> Configure PubsubIO with dead-letter topic
> -----------------------------------------
>
> Key: BEAM-12474
> URL: https://issues.apache.org/jira/browse/BEAM-12474
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Zachary Houfek
> Priority: P2
>
> Based on discussion from [PR14971|https://github.com/apache/beam/pull/14971].
> Currently, PubsubIO throws a RuntimeException when it can't parse a payload.
> It would perhaps be helpful if users could configure PubsubIO.Read with a
> dead-letter topic so that parsing errors instead write to the topic rather
> than just throw an exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
