cryptoe opened a new pull request, #13981: URL: https://github.com/apache/druid/pull/13981
When durableStorage is enabled, with #13741 we started eagerly fetching results. Now with https://github.com/apache/druid/blob/master/extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java#L1240 meant we started eagerly fetching all s3 results. This fix will enable us to fetch lazily the s3 commands. The larger fix would be to ensure durableStorage returns a new type of readableFrameChannel which signals the worker to fetch the file remotely in case we are fetching the output from the same worker here which will be a follow up PR. https://github.com/apache/druid/blob/master/extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerImpl.java#L924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
