adamreeve commented on issue #15191: URL: https://github.com/apache/datafusion/issues/15191#issuecomment-5139378421
Hi @alamb and @xudong963, I've been experimenting with using `ProgressiveEval` for streaming data from Delta Lake tables in order, and found it can be beneficial for improving performance and reducing memory requirements. It looks like this issue has stalled though. I'd like to pick up this work to get this added to DataFusion and work with `ListingTable`. It looks like a lot of the prerequisites needed like per-partition statistics are now in-place, and this just needs someone to provide benchmarks to prove the optimization is worthwhile to move it forward. Does that sound right? Are there any existing benchmarks I should use as a starting point, and is there anyone else currently working in this area? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
