kennknowles commented on code in PR #30545:
URL: https://github.com/apache/beam/pull/30545#discussion_r1521279495
##########
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkBatchPortablePipelineTranslator.java:
##########
@@ -290,6 +296,26 @@ public FlinkPortablePipelineTranslator.Executor translate(
return context;
}
+ private static <K, V> void translateRedistributeByKey(
+ PTransformNode transform, RunnerApi.Pipeline pipeline,
BatchTranslationContext context) {
+ DataSet<WindowedValue<KV<K, V>>> inputDataSet =
+ context.getDataSetOrThrow(
+
Iterables.getOnlyElement(transform.getTransform().getInputsMap().values()));
+ context.addDataSet(
+
Iterables.getOnlyElement(transform.getTransform().getOutputsMap().values()),
+ inputDataSet.rebalance());
Review Comment:
I think for FlinkRunner today, all of Reshuffle, RedistributeArbitrarily,
and RedistributeByKey are basically the same semantics. I do think that is a
coincidence rather than a logical necessity, which is why I gave them each
their own method.
(these are basically just copy/paste of Reshuffle translation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
