advancedxy commented on PR #471: URL: https://github.com/apache/incubator-uniffle/pull/471#issuecomment-1380242176
> > So, I think we could reuse the rss.server.single.buffer.flush.threshold settings, which is 64MB by default. We don't have to introduce rss.server.huge-partition.memory.limit.ratio ? > > Memory limit is required. Usually the huge partition writing speed is fast, and the HDFS flushing speed is slower than writing. If missing memory limit, the regular partitions will be affected. > > By the way, this design has been introduced into our internal version, it works well. Before this PR, sometimes one huge partition will make other regular partitions buffer require fail. I'm ok to add `rss.server.huge-partition.size.threshold` setting, which is used to add memory limit. I just think maybe we should reuse `rss.server.single.buffer.flush.threshold` instead of another `rss.server.huge-partition.memory.limit.ratio` settings? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
