LakshSingla commented on issue #17669: URL: https://github.com/apache/druid/issues/17669#issuecomment-2617776115
I was working on something similar a while back. Maybe these changes will help you https://github.com/apache/druid/pull/14834. What I found was that the speed improvements weren't a lot for MSQ because transfer manager in the SDK v1(?) didn't parallelize if the downloads were chunked. It helped a bit when segments were getting downloaded and uploaded. However, for some reason, segments uploaded without the transfer manager were slowly downloaded when downloaded with the transfer manager. As such, it would be really cool if you can benchmark the performance of the changes, along with speedups/slowdown when downloaded and uploaded pre-post your patch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
