advancedxy commented on PR #471: URL: https://github.com/apache/incubator-uniffle/pull/471#issuecomment-1380179474
> > I don't think we should introduce a new configuration to control buffer flush. rss.server.single.buffer.flush.enabled can be used for the huge partition buffer flush purpose. If the huge partition limit is enabled, the single buffer flush could be enabled automatically. > > If single buffer flush could be enabled automatically, how to set the flush threshold size? I don't hope the `rss.server.single.buffer.flush.enabled` is enabled for regular partitions, which could be flushed to ssd/hdd directly instead of cold storage. And huge partition could be flushed to HDFS. > > > I think these two similar configurations will bring more confusion to end user. > > Emm. Yes. Sorry, I forgot to reply this comment. If I understand the code and design correctly, `single.buffer.flushed.enabled` is added to support flushing big buffer to cold storage, such as hdfs directly, which I think it perfectly matches you intention when serving huge partitions. So, I think we could reuse the `rss.server.single.buffer.flush.threshold` settings, which is 64MB by default. We don't have to introduce `rss.server.huge-partition.memory.limit.ratio` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
