[
https://issues.apache.org/jira/browse/BEAM-12737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17448742#comment-17448742
]
Kyle Weaver commented on BEAM-12737:
------------------------------------
Your change is on the 2.35.0 release branch so it will be included in that
release once it is ready. https://github.com/apache/beam/commits/release-2.35.0
Usually we estimate releases should take around 6 weeks, starting from today.
But there is a lot of variance so subscribe to
[[email protected]|mailto:[email protected]] for updates.
[https://beam.apache.org/contribute/#when-will-my-change-show-up-in-an-apache-beam-release]
> [SQL] add API to retrieve failed events in the collection due to query
> runtime error
> -------------------------------------------------------------------------------------
>
> Key: BEAM-12737
> URL: https://issues.apache.org/jira/browse/BEAM-12737
> Project: Beam
> Issue Type: Improvement
> Components: dsl-sql
> Reporter: Brachi Packter
> Priority: P3
>
> When calling to
> ```
> collection.apply(SqlTransform.query(query))
> ```
> we had cases in our production that query causes some errors for small amount
> of events in the collection.
> For example:
> 1. functions or UDF that produce null pointer exception when it is called it
> with null value
> 2. casting issues in sql failed for some invalid data.
> and more.
> For sure we can fix those issues, but the point it that it can be missed when
> developing the query and failed only in production runtime in some rare cases.
> Current behaviour is that query is failed and records are retried forever.
> We would want an option to get failed records from the query and then we can
> send them to DLQ or totally skip them
> Is it possible to have something similar to what we have in BigQueryIo?
> `getFailedInsertsWithErr` ?
> {code:java}
> .apply(BigQueryIO.write()
>
> .withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())
> .withExtendedErrorInfo())
> .getFailedInsertsWithErr().apply(....);
> {code}
> This is also mentioned in Beam style guide:
> https://beam.apache.org/contribute/ptransform-style-guide/#error-handling
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
