alamb commented on issue #13692: URL: https://github.com/apache/datafusion/issues/13692#issuecomment-2529361202
> For my part, I'm going to spend the next few hours seeing if I can boil down an initial reproducer per the ask [here](https://github.com/apache/datafusion/pull/13424#issuecomment-2526272457). Hopefully it will help to show concretely where the current architecture falls over and test prototype approaches against. This would be *super* helpful. There are many open source datasets on https://huggingface.co/datasets -- maybe we can find some suitably large ones to run queries against, and throttle the network bandwidth somehow (blocking IO sleeps as in an ObjectStore wrapper, as @tustvold suggests) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
