gemini-code-assist[bot] commented on code in PR #38011:
URL: https://github.com/apache/beam/pull/38011#discussion_r3014810370
##########
runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TransformTranslator.java:
##########
@@ -486,6 +487,8 @@ public void evaluate(
TranslationUtils.getTupleTagCoders(outputs);
all =
all.mapToPair(TranslationUtils.getTupleTagEncodeFunction(coderMap))
+ .filter(Objects::nonNull) // skip nulls to save on
encoding, nulls are tags that
+ // are not read
Review Comment:
For consistency with other translation files updated in this pull request,
consider keeping the comment on a single line.
```suggestion
.filter(Objects::nonNull) // skip nulls to save on
encoding, nulls are tags that are not read
```
##########
runners/spark/src/main/java/org/apache/beam/runners/spark/translation/TranslationUtils.java:
##########
@@ -445,8 +445,13 @@ public static Map<TupleTag<?>, Coder<WindowedValue<?>>>
getTupleTagCoders(
return tuple2 -> {
TupleTag<?> tupleTag = tuple2._1;
WindowedValue<?> windowedValue = tuple2._2;
- return new Tuple2<>(
- tupleTag, ValueAndCoderLazySerializable.of(windowedValue,
coderMap.get(tupleTag)));
+ Coder<WindowedValue<?>> coder = coderMap.get(tupleTag);
+ if (coder == null) {
+ // there is no coder as this step is leaf step and is not read
anywhere, so coder is pruned
+ // from coderMap
+ return null;
+ }
Review Comment:
The comment "this step is leaf step" is slightly inaccurate. In Beam, a
transform (step) might have multiple outputs, and the fact that one output is
unconsumed doesn't necessarily make the transform a leaf in the pipeline graph.
It is more precise to say that the specific output associated with the
`tupleTag` is unconsumed, which is why its coder has been pruned.
```suggestion
if (coder == null) {
// there is no coder as this output is unconsumed and is not read
anywhere, so coder is pruned
// from coderMap
return null;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
