RussellSpitzer commented on code in PR #17428:
URL: https://github.com/apache/iceberg/pull/17428#discussion_r4075156271
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteManifestsSparkAction.java:
##########
@@ -315,10 +315,17 @@ private Column sortColumn() {
partitionFieldClustering);
// Map the top level partition column names to the column name
referenced within the manifest
- // entry dataframe
+ // entry dataframe. Backtick-quote the partition field name (escaping
any embedded backtick)
+ // because it may itself contain a literal '.' (e.g. when partitioning
on a nested source
+ // column) - the partition struct is always flat, so unquoted, col()
would misparse that dot
+ // as a further level of nesting instead of treating the whole name as
one field.
Column[] partitionColumns =
partitionFieldClustering.stream()
- .map(p -> col(DATA_FILE_PARTITION_COLUMN_NAME + "." + p))
+ .map(
+ p ->
+ col(
+ String.format(
+ "%s.`%s`", DATA_FILE_PARTITION_COLUMN_NAME,
p.replace("`", "``"))))
Review Comment:
Rather than doing this by string manipulation, could we do something like
```java
Column[] partitionColumns =
partitionFieldClustering.stream()
.map(p -> col(DATA_FILE_PARTITION_COLUMN_NAME).getField(p))
.toArray(Column[]::new);
```
That way we avoid any possible other escaping issues and don't rely on a
string replace.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
