kfaraz commented on PR #17732: URL: https://github.com/apache/druid/pull/17732#issuecomment-2661336152
> I am considering setting up a test Druid cluster to evaluate this change. However, it's possible that the results will show that this PR doesn't provide meaningful improvements. @umisan , yes, that's what I fear as well. I had actually tried out a change to reduce the amount of deserialization done but it did not affect the total polling time at all. FYI, we have recently merged a segment caching feature in #17653 . The code in this patch is able to do a delta poll of ~600k segments in just a couple of seconds. Essentially, it does not fetch segments from the metadata store which have not been updated. While this does not currently impact the polling time in `SqlSegmentsMetadataManager`, I intend to plug in the same logic there once I have had time to test out and benchmark the cache changes thoroughly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
