tillrohrmann commented on a change in pull request #10462:
[FLINK-15031][runtime] Calculate required shuffle memory before allocating
slots if resources are specified
URL: https://github.com/apache/flink/pull/10462#discussion_r354852749
##########
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/shuffle/NettyShuffleMaster.java
##########
@@ -32,12 +35,18 @@
*/
public class NettyShuffleMaster implements
ShuffleMaster<NettyShuffleDescriptor> {
+ private final int networkBuffersPerInputChannel;
+
+ private final int networkBufferSize;
+
@VisibleForTesting
public NettyShuffleMaster() {
this(new Configuration());
}
public NettyShuffleMaster(final Configuration configuration) {
+ networkBuffersPerInputChannel =
configuration.getInteger(NettyShuffleEnvironmentOptions.NETWORK_BUFFERS_PER_CHANNEL);
+ networkBufferSize =
ConfigurationParserUtils.getPageSize(configuration);
Review comment:
We have a bit of duplicated logic here. We do the parsing of these config
options here and in `NettyShuffleEnvironmentConfiguration.fromConfiguration`. I
think it is not important to fix right away but maybe it makes sense to
introduce a common configuration for the `ShuffleMaster` and the
`ShuffleEnvironment`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
