alamb commented on PR #14286: URL: https://github.com/apache/datafusion/pull/14286#issuecomment-2614037782
> Based on the previous discussions, and draft PRs, I ended up with this Object store wrapper to spawn the io tasks in a different handle: https://github.com/delta-io/delta-rs/blob/main/crates%2Fcore%2Fsrc%2Fstorage%2Fmod.rs#L116-L124 This is neat -- I actually like that it is in terms of just a tokio runtime rather than somehthing like `DedicatedExecutor`. I might try and work on IoObjectStore to follow that pattern instead One thing I noticed is that it doesn't actually shim the get result stream (so while the intial GET request is on the other runtime, when reading data from the stream that will still happen on the same runtime) I handled this by wrapping the stream like this: ```rust /// Wrap the result stream if necessary fn wrap_if_necessary(&self, payload: GetResultPayload) -> GetResultPayload { match payload { GetResultPayload::File(_, _) => payload, GetResultPayload::Stream(stream) => { let new_stream = self.dedicated_executor.run_io_stream(stream).boxed(); GetResultPayload::Stream(new_stream) } } } ``` It might be an interesting thing to try in delta.rs :thin' -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
