rich7420 commented on code in PR #7156:
URL: https://github.com/apache/datafusion-comet/pull/7156#discussion_r4403798402
##########
spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala:
##########
@@ -1216,6 +1216,48 @@ object IcebergReflection extends Logging {
}
}
+ /**
+ * Top-level column names of `schema` whose type differs in some schema of
the table's history
+ * (`table.schemas()`), such as an int column promoted to long. Data files
written before the
+ * promotion keep the older type, and iceberg-rust compares literals against
it incorrectly: the
+ * page index skips every page for range and IN predicates, and the row
filter casts the literal
+ * to the older type, so an out-of-range value becomes NULL
(apache/iceberg-rust#3122).
+ *
+ * Expiring snapshots with `cleanExpiredMetadata` removes schemas that no
retained snapshot
+ * uses, even though files written under them can remain. Iceberg numbers
schemas consecutively
+ * from 0, so a gap means the history is incomplete, and every column of a
type that a promotion
+ * can produce (long, double, decimal) is returned. The history cannot
reveal a file whose type
+ * differs without a recorded schema change, such as one imported by
`add_files`, `migrate` or
+ * `snapshot`, or one promoted under metadata from before Iceberg 0.12,
which kept only the
+ * current schema. Called at serialization time, so it throws on reflection
failure.
+ */
+ def promotedColumns(table: Any, schema: Any): Set[String] = {
+ import scala.jdk.CollectionConverters._
+ val findField = getMethod(schema.getClass, "findField", classOf[Int])
+ def typeOf(field: Any): String = getMethod(field.getClass,
"type").invoke(field).toString
Review Comment:
Reused `fieldTypeOf(field).toString`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
