NoahKusaba commented on PR #2638: URL: https://github.com/apache/datafusion-ballista/pull/2638#issuecomment-6882197393
Battery done. I ran 81 queries against plain DataFusion over Iceberg tables (several commits, partitioned, empty, an older snapshot) and a Parquet table: filters, limits, aggregates, joins, windows, CTEs, set operations and time travel. Each ran under both planners, with default sizing and with 1 and 2 partitions per task. - The static planner matched DataFusion on all 81, in every configuration. - Under the adaptive planner, the split-union limitation is the only Iceberg-related failure. `UNION` without `ALL` fails too, so I've reworded the limitation to cover it (50a7dac7). - A `LEFT` or `FULL OUTER JOIN` against an empty side never finishes under the adaptive planner. It isn't Iceberg-specific: it reproduces with Parquet on both sides, and it's the bug #2837 fixes. I've filed it as #2865. `ballista.planner.propagate_empty.enabled = false` avoids it. I've also filed #2866 for the split union over a custom source, which #2852 fixes. Merging on the current pin and doing the DataFusion 56 bump as a follow-up once apache/datafusion-iceberg#99 lands sounds good to me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
