NikitaMatskevich commented on code in PR #5724:
URL: https://github.com/apache/datafusion-comet/pull/5724#discussion_r3956622413
##########
spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala:
##########
@@ -581,6 +582,52 @@ object IcebergReflection extends Logging {
}
}
+ // scalastyle:off line.size.limit
+ /**
+ * Maps Iceberg's logical primitive-column names to their physical Parquet
leaf paths.
+ *
+ * This mirrors the map Iceberg Java builds before applying per-column
writer settings. In
+ * particular, Parquet's canonical three-level encodings insert `list` for
array elements and
+ * `key_value` for map keys/values, so logical names such as `tags.element`
and `attrs.value`
+ * cannot be passed directly to Parquet writer properties.
+ *
+ * See:
+ *
https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L411-L421
+ */
+ // scalastyle:on line.size.limit
+ def getParquetPathByIcebergColumnName(schema: Any): Option[Map[String,
String]] = {
+ import scala.jdk.CollectionConverters._
+ try {
+ val parquetSchemaUtil = loadClass(ClassNames.PARQUET_SCHEMA_UTIL)
+ val parquetSchema = parquetSchemaUtil
+ .getMethod("convert", loadClass(ClassNames.SCHEMA), classOf[String])
+ .invoke(null, schema.asInstanceOf[AnyRef], "table")
Review Comment:
Thank you for review! Fixed in "fix: fall back for sanitized Iceberg bloom
paths" by falling back to java implementation when those problematic column
names occur. I don't think its a big blocker for the coverage of Comet,
personally I didn't see such naming in production before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
