wsry commented on pull request #13595: URL: https://github.com/apache/flink/pull/13595#issuecomment-717347640
> @wsry I have two more questions on the minimum buffers logic and confi > > (1) > > I previously thought that this is to ensure that the sort buffers are not too small. But I saw now that they can actually be smaller, because the minimum is `Math.min(minBuffersPerSortMergePartition , numSubpartitions + 1)`. So if we have a low parallelism, then the sort shuffle might run with something like 5 buffers (assuming a user turns it on for all shuffles). Is that on purpose? > > Does it make sense to just use the `minBuffersPerSortMergePartition` value and not lower it below that? > > BTW: I think I now understand why you called it "max buffers" before - because it is the maximum to which the minimum requirement can grow. > > (2) > > The current value for `taskmanager.network.sort-shuffle.min-buffers` may be high for mini clusters. At least if we end up doing what is proposed in (1) and not lower it for lower parallelism. > > The current min is 512 buffers (16 MB) and the mini cluster has only 64 MB network memory by default. So as soon as there are four shuffles, the mini cluster will break, which is pretty soon. > We should probably set a different min on the mini cluster, like 64 buffers or so. This does not require a new config key, it you can just set this directly in the mini cluster configuration. Using `taskmanager.network.sort-shuffle.min-buffers` as the min required number of buffers directly is also a reasonable choice. Previously, I was thinking that users can switch to sort-merge shuffle directly without changing the config value (may need to increase network buffers for small parallelism jobs). However, to achieve a better performance, the user may have to tune these config values, if so, leaving the config values unchanged seems meaningless. I will update the PR soon as suggested. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
