Sounds like a very good addition. I'd say this can be a single PR since changes are related. Please open a JIRA for tracking.
Have you though about introducing a similar change to Python SDK ? (doesn't have to be the same PR). - Cham On Wed, Oct 3, 2018 at 8:31 AM Jeff Klukas <[email protected]> wrote: > If this looks good for MapElements, I agree that it makes sense to extend > to FlatMapElements and Filter and to keep the API consistent between them. > > Do you have suggestions on how to submit changes with that wider scope? > Would one PR altering MapElements, FlatMapElements, Filter, ParseJsons, and > AsJsons be too large to reasonably review? Should I open an overall JIRA > ticket to track and break this into smaller PRs? > > On Wed, Oct 3, 2018 at 10:31 AM Reuven Lax <[email protected]> wrote: > >> Sounds cool. Why not support this on other transforms as well? >> (FlatMapElements, Filter, etc.) >> >> Reuven >> >> On Tue, Oct 2, 2018 at 4:49 PM Jeff Klukas <[email protected]> wrote: >> >>> I've seen a few Beam users mention the need to handle errors in their >>> transforms by using a try/catch and routing to different outputs based on >>> whether an exception was thrown. This was particularly nicely written up in >>> a post by Vallery Lancey: >>> >>> >>> https://medium.com/@vallerylancey/error-handling-elements-in-apache-beam-pipelines-fffdea91af2a >>> >>> I'd love to see this pattern better supported directly in the Beam API, >>> because it currently requires the user to implement a full DoFn even for >>> the simplest cases. >>> >>> I propose we support for a MapElements-like transform that allows the >>> user to specify a set of exceptions to catch and route to a failure output. >>> Something like: >>> >>> MapElements >>> .via(myFunctionThatThrows) >>> .withSuccessTag(successTag) >>> .withFailureTag(failureTag, JsonParsingException.class) >>> >>> which would output a PCollectionTuple with both the successful outcomes >>> of the map operation and also a collection of the inputs that threw >>> JsonParsingException. >>> >>> To make this more concrete, I put together a proof of concept PR: >>> https://github.com/apache/beam/pull/6518 I'd appreciate feedback about >>> whether this seems like a worthwhile addition and a feasible approach. >>> >>
