NikitaMatskevich commented on code in PR #5724:
URL: https://github.com/apache/datafusion-comet/pull/5724#discussion_r3956712165
##########
spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala:
##########
@@ -581,6 +582,52 @@ object IcebergReflection extends Logging {
}
}
+ // scalastyle:off line.size.limit
+ /**
+ * Maps Iceberg's logical primitive-column names to their physical Parquet
leaf paths.
+ *
+ * This mirrors the map Iceberg Java builds before applying per-column
writer settings. In
+ * particular, Parquet's canonical three-level encodings insert `list` for
array elements and
+ * `key_value` for map keys/values, so logical names such as `tags.element`
and `attrs.value`
+ * cannot be passed directly to Parquet writer properties.
+ *
+ * See:
+ *
https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L411-L421
+ */
+ // scalastyle:on line.size.limit
+ def getParquetPathByIcebergColumnName(schema: Any): Option[Map[String,
String]] = {
+ import scala.jdk.CollectionConverters._
+ try {
+ val parquetSchemaUtil = loadClass(ClassNames.PARQUET_SCHEMA_UTIL)
+ val parquetSchema = parquetSchemaUtil
+ .getMethod("convert", loadClass(ClassNames.SCHEMA), classOf[String])
+ .invoke(null, schema.asInstanceOf[AnyRef], "table")
Review Comment:
Added 2 boolean flags `icebergSupportsBloomFpp` and
`icebergSupportsBloomNdv` based on reflection of Fpp and Ndv string prefixes
availability in current iceberg-java. Based on these flags, now tests verify
expected behaviors for each scenario. This was pushed in commit "[fix: match
Iceberg-version bloom property
support](https://github.com/apache/datafusion-comet/pull/5724/commits/fcef2c4f1d2867e02220887c14cec73610ba5e2f)".
- Spark 4.1 / Iceberg 1.11: complete writer suites, 115/115 passed; final
parity test also rerun successfully.
- Spark 3.4 / Iceberg 1.5.2: detection and action suites passed after
correcting the version-aware footer assertion.
- Spark 3.5 / Iceberg 1.8.1: full suites passed; final FPP-only test rerun
successfully.
- Spark 4.0 / Iceberg 1.10.0: full suites passed; final FPP-only test
rerun successfully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
