tarun11Mavani commented on PR #16344: URL: https://github.com/apache/pinot/pull/16344#issuecomment-3145470161
> > SizeBasedSegmentFlushThresholdComputer uses pre-commit row counts instead of post-commit filtered counts for accurate threshold calculations > > @tarun11Mavani : can you share more details about this? If we are using pre-commit row counts, then I suppose the size based flush threshold computer uses the size of the Mutable segment and not the immutable segment? Yes. Currently, I am using pre-commit row count to calculate and estimate the max docs for next segment. All of this is considering the size of immutable segment which is being built. The flush threshold in the config will control the max size of the mutable segment. IMO, If we try to match the max size with immutable segment, we might end up creating a very large segment in memory and that could lead to other issues related to index file sizes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
