mcvsubbu commented on issue #4402: the most appropriate segment size URL: https://github.com/apache/incubator-pinot/issues/4402#issuecomment-524563670 @sunny19930321 sorry, did not see this issue until now. This article can be helpful. https://engineering.linkedin.com/blog/2019/auto-tuning-pinot Segment size is something that is to be determined as per your application requirements, and memory availability (and of course, CPU availability). As you know, pinot queries are processed on a per-segment basis, responses from segments within a server are merged first, and then sent to the broker. The broker merges responses across servers. Each segment is processed in its own thread. Having huge segments reduces the number of threads used for processing, so reduces parallelism. On the other hand, having a large number of small segments causes the per-segment overhead to be high (servers retain per-segment metadata in memory all the time), and potential thread-contention while processing queries. In our experiences on large data pushes, we have found segment size of between 100M to 500M to be a fair size with servers running on 64G machines. On machines with more memory & cpu, the segment size could be increased, if we find benefit (e.g. if there are a large number of segments, then we could have a smaller number of larger segments).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
