noob-se7en opened a new pull request, #14811:
URL: https://github.com/apache/pinot/pull/14811

   **Problem Statement**
   The Force Commit API can cause high ingestion lag and slower queries because 
it triggers the simultaneous segment commit for all consuming segments. This 
happens because:
   1. If N is the number of partition groups a server is consuming from, The 
API will cause all of the N consuming segments to commit. Hence N consumer 
threads will rush to acquire segment build semaphore. If the Semaphore allows 
only M permits, Only M consuming segments are in the segment build stage and 
the remaining (N - M) consumer threads are waiting on the Semaphore. Since (N - 
M) consumer threads are waiting, the consumption lag can become substantial.
   2. Since M consuming segments are built in parallel, queries can become 
slower on poorly sized servers due to high memory consumption.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to