Hi Steven, I've added you as a contributor, so feel free to assign the issue to yourself.
BigQueryIO has a similar pattern for its write transform. You can do BigQueryIO.write(.......).withExtendedErrorInfo()[1], and it will return a WriteResult[2], which contains a PCollection of failed inserts with their errors[3]. You could add a similar option to ElasticSearchIO which determines when to retry errors, and what to do with them. Ideally, the default behavior would not change (to ensure backwards compatibility), but when specifying the new configuration, that would help you address your use case. By the way, this pattern is very common, so I bet it will be a useful addition for other users. If that works for you, we'd be happy to welcome your contribution. Best -P. [1] https://beam.apache.org/releases/javadoc/2.24.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.html#withExtendedErrorInfo-- [2] https://beam.apache.org/releases/javadoc/2.24.0/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.html [3] https://beam.apache.org/releases/javadoc/2.24.0/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.html#getFailedInsertsWithErr-- On Wed, Sep 30, 2020 at 11:23 AM Steven Gaunt <[email protected]> wrote: > Hi guys > > I've created a jira ticket BEAM-10990 > <https://issues.apache.org/jira/browse/BEAM-10990> as we are keen to use > ElasticSearchIO. > > would there be any objections if I was create a branch and propose a fix > for BEAM-10990 <https://issues.apache.org/jira/browse/BEAM-10990> ? > > thanks > > Steve > > This message contains proprietary information from Equifax which may be > confidential. If you are not an intended recipient, please refrain from any > disclosure, copying, distribution or use of this information and note that > such actions are prohibited. If you have received this transmission in > error, please notify by e-mail [email protected]. Equifax® is a > registered trademark of Equifax Inc. All rights reserved. > >
