[ https://issues.apache.org/jira/browse/BEAM-4199?focusedWorklogId=102664&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-102664 ]
ASF GitHub Bot logged work on BEAM-4199: ---------------------------------------- Author: ASF GitHub Bot Created on: 16/May/18 21:24 Start Date: 16/May/18 21:24 Worklog Time Spent: 10m Work Description: akedin commented on a change in pull request #5380: [BEAM-4199][SQL] Add support for DLQ to PubsubJsonTable URL: https://github.com/apache/beam/pull/5380#discussion_r188777375 ########## File path: sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubIOJsonTable.java ########## @@ -123,12 +141,32 @@ public BeamIOType getSourceType() { @Override public PCollection<Row> buildIOReader(Pipeline pipeline) { - return + PCollectionTuple rowsWithDlq = PBegin .in(pipeline) .apply("readFromPubsub", readMessagesWithAttributes()) - .apply("parseMessageToRow", PubsubMessageToRow.forSchema(getSchema())) - .setCoder(getSchema().getRowCoder()); + .apply("parseMessageToRow", createParserParDo()); Review comment: Is there a good example / guide for display data? I don't see anything obvious in the docs or in intellij ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 102664) Time Spent: 1h 10m (was: 1h) > [SQL] Add a DLQ support for Pubsub tables > ----------------------------------------- > > Key: BEAM-4199 > URL: https://issues.apache.org/jira/browse/BEAM-4199 > Project: Beam > Issue Type: Improvement > Components: dsl-sql > Reporter: Anton Kedin > Assignee: Anton Kedin > Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently we crash the pipeline if there's any error processing the message > from the pubsub, including if it has incorrect JSON format, like missing > fields etc. > Correct solution would be for the user to specify a way to handle the errors, > and ideally point to a dead-letter-queue where Beam should send the messages > it could not process. -- This message was sent by Atlassian JIRA (v7.6.3#76005)