jihoonson commented on issue #7890: Set Test timeout higher for robust performance URL: https://github.com/apache/incubator-druid/pull/7890#issuecomment-502862785 > and also discovered that after the initial load of the segment by each thread, time reduces to ~1 ms, the only first time by each thread is taking ~100ms. Yes, this is what I'm seeing here too. What's happening is `SegmentLoaderLocalCacheManager.loadInLocation()` takes ~ 100ms at this [line](https://github.com/apache/incubator-druid/blob/master/server/src/main/java/org/apache/druid/segment/loading/SegmentLoaderLocalCacheManager.java#L235) only for the very first call. That is, only the thread first called this method takes ~ 100ms and subsequent calls are waiting for the lock to be released after the first call is finished. Once this method is called, I guess some code path is cached (probably generic or code generation part) and subsequent calls are optimized to take less than 1 ms. > Looks like getting lock is significant overhead to the methods since it requires acquiring a lock when entering the method - or at least checking whether the current thread already possesses the lock. These operations are quite expensive and they have to be done every single time one of the methods is entered. Since - due to the recursion - this happens a lot, it has an extreme impact on the program performance. This is what I'm suspecting too. Synchronization here is inevitable since we need to support parallel segment loading. So, if this is the case, we should mitigate the lock overhead somehow. However, I'm still not sure what's the root cause. > changed NUM_THREAD to 1 instead of 4 (default), full test time reduced to ~500ms from ~5000ms with 4 threads This part sounds like all CPU cores are not being used for tests. What was the exact command you used to run those tests?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
