taegeonum commented on a change in pull request #123: [NEMO-129] Support Beam's
WindowedWordCount example
URL: https://github.com/apache/incubator-nemo/pull/123#discussion_r225756889
##########
File path:
compiler/frontend/beam/src/main/java/org/apache/nemo/compiler/frontend/beam/PipelineTranslator.java
##########
@@ -198,11 +196,29 @@ private static void parDoMultiOutputTranslator(final
TranslationContext ctx,
pValueWithTupleTag.getKey()));
}
+ private static GroupByKeyTransform createGBKTransform(
+ final TranslationContext ctx,
+ final TransformVertex transformVertex) {
+ final AppliedPTransform pTransform =
transformVertex.getNode().toAppliedPTransform(PIPELINE.get());
+ final PCollection<?> mainInput = (PCollection<?>)
+
Iterables.getOnlyElement(TransformInputs.nonAdditionalInputs(pTransform));
+ final TupleTag mainOutputTag = new TupleTag<>("main output");
+
+ return new GroupByKeyTransform(
+ getOutputCoders(pTransform),
+ mainOutputTag,
+ Collections.emptyList(),
+ mainInput.getWindowingStrategy(),
+ Collections.emptyList(), /* side inputs */
+ ctx.pipelineOptions,
+ SystemReduceFn.buffering(mainInput.getCoder()));
Review comment:
Nemo currently does not perform incremental combining.
It groups inputs according to the window and key and combine the collected
inputs after grouping is done for all bounded inputs. Using
`System.ReduceFn.combining` can enable incremental combining, and I think this
is a separate issue. I've filed an issue for this
https://issues.apache.org/jira/browse/NEMO-222
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
