andygrove commented on PR #2565: URL: https://github.com/apache/datafusion-ballista/pull/2565#issuecomment-6100616743
Good catch. At SF1, 130,093 of the 2,880,404 `store_sales` rows have a NULL `ss_sold_date_sk`, and four other fact tables have NULL date keys too. So Spark-written data will always have that directory. I went with option a in bdd253c16. When a partition column has a `__HIVE_DEFAULT_PARTITION__` directory, the runner registers the files as `<table>_raw` with that column as a string. `<table>` is then a view that maps the directory to NULL and casts the other values back to `Int32`. Filters on the key still prune directories, and DataFusion pushes the cast into the scan's projection, so the plan barely changes. Tables without that directory register exactly as before. `null_date_sk_partition_reads_as_null` covers it. I also partitioned SF1 the way Spark does, with the NULL rows in `__HIVE_DEFAULT_PARTITION__`, and all 99 queries return the same results as on the original data. The real fix belongs upstream in apache/datafusion#18083, so the view can go once that lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
