mcvsubbu commented on issue #7192: URL: https://github.com/apache/pinot/issues/7192#issuecomment-885903527
Another thing to do in this area is to avoid multiple partitions getting into the segment completion phase at the same time. This can be done from the controller. For different partitions, it can provide a slightly increased or decreased number of rows as threshold. The variations may have to be significant for high ingestion use cases, so this will. have to be a function of ingestion rate. This will move us towards a system where at any given time, only some partitions are awaiting completion while others are consuming. Presumably a high ingestion system has several partitions so that we see new rows all the time, and for a low ingestion rate, it may not matter too much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
