[
https://issues.apache.org/jira/browse/BEAM-5413?focusedWorklogId=145307&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-145307
]
ASF GitHub Bot logged work on BEAM-5413:
----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Sep/18 14:40
Start Date: 18/Sep/18 14:40
Worklog Time Spent: 10m
Work Description: jklukas commented on issue #6414: [BEAM-5413] Add
PTransform::compose for lambda-based composite transforms
URL: https://github.com/apache/beam/pull/6414#issuecomment-422421164
I see now that docs are still in a separate repo. I'm backing out the
changes to the getting-started guide, but the content I'd propose adding to the
Composite Transforms section is:
```
{:.language-java}
If your logic only needs to be used once, you may instead define your
composite
transform as an anonymous Java 8 lambda function passed to
`PTransform.compose`:
```java
input.apply(PTransform.compose((PCollection<String> lines) -> {
PCollection<String> words = lines.apply(
ParDo.of(new ExtractWordsFn()));
PCollection<KV<String, Long>> wordCounts =
words.apply(Count.<String>perElement());
return wordCounts;
}))
\```
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 145307)
Time Spent: 20m (was: 10m)
> Add method for defining composite transforms as lambda expressions
> ------------------------------------------------------------------
>
> Key: BEAM-5413
> URL: https://issues.apache.org/jira/browse/BEAM-5413
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
> Reporter: Jeff Klukas
> Assignee: Kenneth Knowles
> Priority: Minor
> Fix For: 2.8.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Defining a composite transform today requires writing a full named subclass
> of PTransform (as [the programming guide
> documents|https://beam.apache.org/documentation/programming-guide/#composite-transforms]
> but there are cases where users may want to define a fairly trivial
> composite transform using a less verbose Java 8 lambda expression.
> Consider an example where the user has defined MyDeserializationTransform
> that attempts to deserialize byte arrays into some object, returning a
> PCollectionTuple with tags for successfully deserialized records (mainTag)
> and for errors (errorTag).
> If we introduce a PTransform::compose method that takes in a
> SerializableFunction, the user can handle errors in a small lambda expression:
>
> {code:java}
> byteArrays
> .apply("attempt to deserialize messages",
> new MyDeserializationTransform())
> .apply("write deserialization errors",
> PTransform.compose((PCollectionTuple input) -> {
> input
> .get(errorTag)
> .apply(new MyErrorOutputTransform());
> return input.get(mainTag);
> })
> .apply("more processing on the deserialized messages",
> new MyOtherTransform())
> {code}
> This style allows a more concise and fluent pipeline definition than is
> currently possible.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
