[
https://issues.apache.org/jira/browse/BEAM-12474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17361056#comment-17361056
]
Brian Hulette edited comment on BEAM-12474 at 6/10/21, 4:12 PM:
----------------------------------------------------------------
Note that we've added support for dead-letter pubsub topics in a couple of
different places already. We have support for it in SQL via
[PubsubSchemaIOProvider|https://github.com/apache/beam/blob/815e2c94bee24ef0c15c78876d6266618c238ba5/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubSchemaIOProvider.java#L216]
(see
[docs|https://beam.apache.org/documentation/dsls/sql/extensions/create-external-table/#pubsub]),
and Reza implemented this for JSON over Pubsub
(https://github.com/apache/beam/pull/11929).
I think it makes sense to add a user interface for specifying a deadletter
queue directly in PubsubIO, but we should also think about how all of this work
can be generalized. E.g. maybe we could do something like what Reza did with
JSON, but do it generally with the PayloadSerializer/PayloadSerializerProvider
interface.
was (Author: bhulette):
Note that we've added support for dead-letter pubsub topics in a couple of
different places already. We have support for it in SQL via
[PubsubSchemaIOProvider|https://github.com/apache/beam/blob/815e2c94bee24ef0c15c78876d6266618c238ba5/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubSchemaIOProvider.java#L216]
(see
[docs|https://beam.apache.org/documentation/dsls/sql/extensions/create-external-table/#pubsub],
and Reza implemented this for JSON over Pubsub
(https://github.com/apache/beam/pull/11929).
I think it makes sense to add a user interface for specifying a deadletter
queue directly in PubsubIO, but we should also think about how all of this work
can be generalized. E.g. maybe we could do something like what Reza did with
JSON, but do it generally with the PayloadSerializer/PayloadSerializerProvider
interface.
> Configure PubsubIO with dead-letter topic
> -----------------------------------------
>
> Key: BEAM-12474
> URL: https://issues.apache.org/jira/browse/BEAM-12474
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Zachary Houfek
> Priority: P2
>
> Based on discussion from [PR14971|https://github.com/apache/beam/pull/14971].
> Currently, PubsubIO throws a RuntimeException when it can't parse a payload.
> It would perhaps be helpful if users could configure PubsubIO.Read with a
> dead-letter topic so that parsing errors instead write to the topic rather
> than just throw an exception.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
