johnyangk commented on a change in pull request #123: [NEMO-129] Support Beam's
WindowedWordCount example
URL: https://github.com/apache/incubator-nemo/pull/123#discussion_r224953385
##########
File path:
compiler/frontend/beam/src/main/java/org/apache/nemo/compiler/frontend/beam/PipelineTranslator.java
##########
@@ -198,11 +196,29 @@ private static void parDoMultiOutputTranslator(final
TranslationContext ctx,
pValueWithTupleTag.getKey()));
}
+ private static GroupByKeyTransform createGBKTransform(
+ final TranslationContext ctx,
+ final TransformVertex transformVertex) {
+ final AppliedPTransform pTransform =
transformVertex.getNode().toAppliedPTransform(PIPELINE.get());
+ final PCollection<?> mainInput = (PCollection<?>)
+
Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(pTransform));
+ final TupleTag mainOutputTag = new TupleTag<>("main output");
+
+ return new GroupByKeyTransform(
+ getOutputCoders(pTransform),
+ mainOutputTag,
+ Collections.emptyList(),
+ mainInput.getWindowingStrategy(),
+ Collections.emptyList(), /* side inputs */
+ ctx.pipelineOptions,
+ SystemReduceFn.buffering(mainInput.getCoder()));
Review comment:
I skimmed through `SystemReduceFn`, and noticed that we have two options
here.
- `SystemReduceFn.buffering()`: Buffers all input values in persistent state
and produces an iterable over it
- `SystemReduceFn.combining()`: Combines all input values using a `CombineFn`
My guess is that `SystemReduceFn.combining()` provides better performance,
assuming that the operation is combinable and a data reduction happens with
combining. Maybe we should try to use `SystemReduceFn.combining()` as much as
we can?
Also in both cases a buffer with a state id is created and used. Does that
mean our `InMemoryStateInternalsFactory` will be used by `SystemReduceFn`? It'd
be nice to leave a comment(TODO) regarding this as we may want to use different
types of stores for `SystemReduceFn` operations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
