AHeise commented on a change in pull request #13735:
URL: https://github.com/apache/flink/pull/13735#discussion_r518033137
##########
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/partitioner/CustomPartitionerWrapper.java
##########
@@ -55,6 +56,14 @@ public int
selectChannel(SerializationDelegate<StreamRecord<T>> record) {
return partitioner.partition(key, numberOfChannels);
}
+ @Override
+ public ChannelStateRescaler getDownstreamChannelStateRescaler() {
+ // fully rely on filtering downstream
+ // note that custom partitioners are not officially supported -
the user has to force rescaling
+ // in that case, we assume that the custom partitioner is
deterministic
+ return ChannelStateRescaler.BROADCAST;
Review comment:
`BROADCAST` would replicate each channel state to each subtask (very
costly) including all new states. All superfluous replicas are filtered out
downstream by the partitioner.
Note that I will extend your check for cyclic graphs to customer
partitioners for disabling UC. So the user has to force it to make it work.
However, I do not see a reason to then fail on the second level again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
