[
https://issues.apache.org/jira/browse/BEAM-5638?focusedWorklogId=200356&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-200356
]
ASF GitHub Bot logged work on BEAM-5638:
----------------------------------------
Author: ASF GitHub Bot
Created on: 19/Feb/19 04:45
Start Date: 19/Feb/19 04:45
Worklog Time Spent: 10m
Work Description: tims commented on issue #7736: [BEAM-5638] Exception
handling for Java MapElements and FlatMapElements
URL: https://github.com/apache/beam/pull/7736#issuecomment-464980759
A `PCollection<Either<G,B>>` would mean that to chain transformation you'd
need to inspect a lot of either good or bad elements you likely don't care
about?
There was an interesting post recently about handling pipeline errors in
spark that might provide some inspiration / validation:
https://www.thoughtworks.com/insights/blog/capturing-data-pipeline-errors-functionally-writer-monads
I've found that the pain that most effects readability of my pipelines is
extracting the main and errors PCollections from the tuple, then unioning the
errors tags.
Thinking more on this...
If I have `PCollection<Foo> foos` and a `PTransform<Foo,Bar> fooToBar`,
`PTransform<Bar,Baz> barToBaz`
Normally I'd do this if I'm not doing error handling:
```
PCollection<Baz> bazs = foos.apply(fooToBar).apply(barToBaz)
```
If I had something like: `PCollectionWithErrors<Foo, ErrorT> foos =
foos.apply(WithErrors.<MyError>of((exception -> ...).withCoder(myErrorCoder));`.
Then I would like to be able to do something like this:
```
PCollectionWithErrors<Baz, ErrorT> bazWithErrs = foos
.apply(WithErrors.of(errorHandler).withCoder(MyErrorCoder.of()))
.apply(fooToBar)
.apply(barToBaz);
PCollection<Baz> bazs = bazWithErrs.main();
PCollection<ErrorT> errs = bazWithErrs.errors();
```
So that the PTransforms fooToBaz and barToBaz are automatically configured
with the ErrorT factory / error handler. In the PCollectionWithErrors.apply
function. And wrapped PCollection<ErrorsT> are automatically unioned.
This is assuming each PTransform only returns a single main, they just want
to add errors, and that they are able to be configured with an exception
handler.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 200356)
Time Spent: 6.5h (was: 6h 20m)
Remaining Estimate: 161.5h (was: 161h 40m)
> Add exception handling to single message transforms in Java SDK
> ---------------------------------------------------------------
>
> Key: BEAM-5638
> URL: https://issues.apache.org/jira/browse/BEAM-5638
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Reporter: Jeff Klukas
> Assignee: Jeff Klukas
> Priority: Minor
> Labels: triaged
> Original Estimate: 168h
> Time Spent: 6.5h
> Remaining Estimate: 161.5h
>
> Add methods to MapElements, FlatMapElements, and Filter that allow users to
> specify expected exceptions and tuple tags to associate with the with
> collections of the successfully and unsuccessfully processed elements.
> See discussion on dev list:
> https://lists.apache.org/thread.html/936ed2a5f2c01be066fd903abf70130625e0b8cf4028c11b89b8b23f@%3Cdev.beam.apache.org%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
