kazantsev-maksim opened a new pull request, #4683: URL: https://github.com/apache/datafusion-comet/pull/4683
## Which issue does this PR close? Part of: https://github.com/apache/datafusion-comet/pull/4670 ## Rationale for this change `Empty2Null` is an internal Spark expression that converts an empty string "" into null. The logic is trivial: if the value is null or a zero-length string, it returns null; otherwise it returns the string unchanged. Purpose The function is applied during partitioned file writes (parquet, orc, etc.) — specifically to the partition columns. The reason is the correctness of Hive-style partitioning. In Hive-style directory naming, an empty string and null are indistinguishable: both would produce a path like col1=, which is ambiguous and breaks reading the data back. To avoid this, Spark runs partition columns through Empty2Null before writing, so empty strings end up in the same default partition as null: `col1=__HIVE_DEFAULT_PARTITION__` ## What changes are included in this PR? ## How are these changes tested? Add rust unit tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
