jackjlli commented on PR #8694: URL: https://github.com/apache/pinot/pull/8694#issuecomment-1131967348
> For my understanding, does this feature implicitly lower impact on the servers as well? If we set this and the controller is using the semaphore, does this mean the servers will also be throttled in the rate they download? > > For context, I just pushed 4096 segments to a table, and the servers were at 100% CPU for ~30 minutes while it downloaded the segments and finished building indices. @jadami10 Yes, this could be one of the optimizations to lower your CPU usage. While as you mentioned there are two major steps that require servers to handle. For building indices, you can enable the index generation logic during the segment generation so that servers won't be busy doing so. You can enable it in the table config. Take a look at this: https://github.com/apache/pinot/blob/master/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java#L181 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
