akshat-lakhera opened a new pull request, #26962: URL: https://github.com/apache/datafusion/pull/26962
## Which issue does this PR close? - Closes #26492. ## Rationale for this change COPY ... PARTITIONED BY wrote partition values into directory names without Hive escaping, so a value like x=y ended up as p0=x=y. DataFusion and pyarrow read that back fine, but DuckDB and Polars either fail or silently drop the partition column. pyarrow, DuckDB and Polars all write p0=x%3Dy, and DataFusion already decodes that on read since #23226. ## What changes are included in this PR? The write side now percent-encodes partition values with Hive's escapePathName set, plus space (as pyarrow, DuckDB and Polars do) and the characters object_store already encoded, so existing names for those don't change. The encoded segment is joined with PathPart::parse so the % isn't encoded a second time. I also pointed the listing prefix pruning in catalog-listing at the same encoding function. It had its own smaller set, so a filter like column2 = 'x=y' would only list column2=x=y/ and miss the newly written column2=x%3Dy/, returning no rows. With one shared set, pruning stops before any value that needs escaping, same as it already did for space and %. I left the partition column name as is, since the issue is about values. Hive escapes keys too, so I can follow up on that separately if it's wanted. ## Are these changes tested? Yes. There's a unit test for the directory name in demux.rs (x=y, a b, a/b, non-ASCII, 100%, a:b, plain), and a copy.slt case that writes those values with COPY, reads them back through a partitioned external table, runs filtered queries on x=y and a b, and checks that the column2=x%3Dy directory is encoded once. The filtered x=y query fails without the catalog-listing change. ## Are there any user-facing changes? Partition directories written by COPY and INSERT now use Hive escaping for characters like =, :, ' and space. Datasets written before this change still read correctly, since the reader decodes values. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
