sunchao commented on code in PR #5724:
URL: https://github.com/apache/datafusion-comet/pull/5724#discussion_r3954379736
##########
spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala:
##########
@@ -581,6 +582,52 @@ object IcebergReflection extends Logging {
}
}
+ // scalastyle:off line.size.limit
+ /**
+ * Maps Iceberg's logical primitive-column names to their physical Parquet
leaf paths.
+ *
+ * This mirrors the map Iceberg Java builds before applying per-column
writer settings. In
+ * particular, Parquet's canonical three-level encodings insert `list` for
array elements and
+ * `key_value` for map keys/values, so logical names such as `tags.element`
and `attrs.value`
+ * cannot be passed directly to Parquet writer properties.
+ *
+ * See:
+ *
https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L411-L421
+ */
+ // scalastyle:on line.size.limit
+ def getParquetPathByIcebergColumnName(schema: Any): Option[Map[String,
String]] = {
+ import scala.jdk.CollectionConverters._
+ try {
+ val parquetSchemaUtil = loadClass(ClassNames.PARQUET_SCHEMA_UTIL)
+ val parquetSchema = parquetSchemaUtil
+ .getMethod("convert", loadClass(ClassNames.SCHEMA), classOf[String])
+ .invoke(null, schema.asInstanceOf[AnyRef], "table")
Review Comment:
### Correctness
#### [P2] Resolve Bloom paths against the native writer schema
Could we account for field-name sanitization before using this Java schema
as the native path map? Iceberg Java converts a quoted column such as `order
id` to physical name `order_x20id`, but the pinned iceberg-rust
`ToArrowSchemaConverter` and parquet-rs writer preserve `order id`. With
`write.parquet.bloom-filter-enabled.column.order id=true`, this map therefore
configures `ColumnPath(["order_x20id"])` while the writer looks up `["order
id"]`. The native write silently omits the requested Bloom filter. The same
mismatch affects dotted names and names starting with a digit.
Please derive paths from the schema the native writer actually emits, or
fall back when the Java conversion renames a configured field, and add a
quoted-name test that checks the written footer for the filter. A bounded
component using the exact name/path methods reproduced the lookup miss for
three such names, with plain-name and native-path controls passing. This was
not a full Spark/JNI reproduction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
