comphead commented on code in PR #7233:
URL: https://github.com/apache/datafusion-comet/pull/7233#discussion_r4579934278
##########
spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala:
##########
@@ -1216,6 +1216,69 @@ object IcebergReflection extends Logging {
}
}
+ /**
+ * Top-level column names of `schema` whose type differs in some schema of
the table's history
+ * (`table.schemas()`), such as an int column promoted to long. Data files
written before the
+ * promotion keep the older type, and iceberg-rust compares literals against
it incorrectly: the
+ * page index skips every page for range and IN predicates, and the row
filter casts the literal
+ * to the older type, so an out-of-range value becomes NULL
(apache/iceberg-rust#3122).
+ *
+ * Expiring snapshots with `cleanExpiredMetadata` removes schemas that no
retained snapshot
+ * uses, even though files written under them can remain. Iceberg numbers
schemas consecutively
+ * from 0, so a gap means the history is incomplete, and every column of a
type that a promotion
+ * can produce (long, double, decimal) is returned. The history cannot
reveal a file whose type
+ * differs without a recorded schema change, such as one imported by
`add_files`, `migrate` or
+ * `snapshot`, or one promoted under metadata from before Iceberg 0.12,
which kept only the
+ * current schema. Called at serialization time, so it throws on reflection
failure.
+ */
+ def promotedColumns(table: Any, schema: Any): Set[String] = {
+ import scala.jdk.CollectionConverters._
+ val findField = getMethod(schema.getClass, "findField", classOf[Int])
+ def typeOf(field: Any): String = getMethod(field.getClass,
"type").invoke(field).toString
+ val schemasById = getMethod(table.getClass, "schemas")
+ .invoke(table)
+ .asInstanceOf[java.util.Map[_, _]]
+ .asScala
+ val history = schemasById.values.toSeq
+ val ids = schemasById.keys.map(_.asInstanceOf[Integer].intValue).toSet
+ val historyComplete = ids.nonEmpty && ids.min == 0 && ids.max == ids.size
- 1
+ getMethod(schema.getClass, "columns")
+ .invoke(schema)
+ .asInstanceOf[java.util.List[_]]
+ .asScala
+ .filter { column =>
+ val id = fieldIdOf(column).asInstanceOf[AnyRef]
+ val current = typeOf(column)
+ val promotable =
+ current == "long" || current == "double" ||
current.startsWith("decimal(")
+ (promotable && !historyComplete) ||
+ history.exists(older => Option(findField.invoke(older,
id)).exists(typeOf(_) != current))
+ }
+ .map(fieldName)
+ .toSet
+ }
+
+ /**
+ * Top-level column names of `schema` whose Parquet files can store a
narrower integer or
+ * floating-point type than the column: int (Hive TINYINT and SMALLINT),
long (int) and double
+ * (float). Files imported by `add_files`, `migrate` or `snapshot` are not
checked against the
+ * schema, so the schema history cannot reveal them, but those procedures
set a name mapping.
+ * Callers treat these columns as promoted for a table that has one.
+ */
+ def narrowableColumns(schema: Any): Set[String] = {
+ import scala.jdk.CollectionConverters._
+ getMethod(schema.getClass, "columns")
+ .invoke(schema)
+ .asInstanceOf[java.util.List[_]]
+ .asScala
+ .filter { column =>
+ Set("int", "long", "double").contains(
+ getMethod(column.getClass, "type").invoke(column).toString)
+ }
+ .map(fieldName)
+ .toSet
Review Comment:
`narrowableColumns` walks `columns` and reads each `type` the same way
`promotedColumns` does, but it keeps a separate type list. The latest commit on
#7156 (e1afd07abe) adds `timestamp` and `timestamp_ns` to the `promotable` list
in `promotedColumns`, so the two lists already differ after the rebase. Could
this become a parameter of `promotedColumns`, for example `filesMayBeNarrower:
Boolean`? It would add `int` to the promotable types and treat them as
`!historyComplete` does. That keeps one list and also stops building
`Set("int", "long", "double")` once per column. Decimal columns would join the
set too. That is harmless, because `pageIndexUnsupportedColumns` already keeps
every decimal residual out of the scan.
##########
spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala:
##########
@@ -1009,6 +1086,19 @@ object CometIcebergNativeScan extends
CometOperatorSerde[CometBatchScanExec] wit
// schema so a filter column projected out of the scan output is still
recognized.
val pageIndexUnsupportedColumns =
IcebergReflection.pageIndexUnsupportedColumns(metadata.tableSchema)
+ // Columns whose type changed over the table's history; residual
predicates comparing them
+ // with a literal are dropped, except equality. The scan schema also
covers a column that
+ // VERSION AS OF reads after it was dropped from the table.
+ // add_files, migrate and snapshot import files without checking their
types against the
+ // schema, so the history records nothing for them, but they set a name
mapping. A table with
+ // one is treated as if every column that can hold a narrower type were
promoted.
+ val schemas = Seq(metadata.tableSchema, metadata.scanSchema)
+ val recordedPromotions =
+ schemas.flatMap(IcebergReflection.promotedColumns(metadata.table, _))
+ val importedNarrowings =
+ if (metadata.nameMapping.isDefined)
schemas.flatMap(IcebergReflection.narrowableColumns)
+ else Nil
+ val promotedColumns = (recordedPromotions ++ importedNarrowings).toSet
Review Comment:
The comment block above still describes only the schema history, and the new
block repeats the reason that the `narrowableColumns` scaladoc gives. Could
they be merged into one comment? It would also help to say here that this is a
workaround for apache/iceberg-rust#3122. apache/iceberg-rust#3540 (row filter)
and apache/iceberg-rust#3507 (page index) are open, and once the pin includes
both, this guard and the one for recorded promotions can be removed. Until
then, `IN` lookups on `BIGINT` keys of migrated tables lose row group pruning
in the native scan, which is a common pattern.
##########
spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala:
##########
@@ -4306,6 +4306,307 @@ class CometIcebergNativeSuite
}
}
+ /**
+ * Compares `SELECT id FROM table WHERE predicate` with Spark for each
predicate, alone and
+ * after `l IS NOT NULL AND`. That array null check is a conjunct the serde
never pushes, as in
+ * the queries that found these bugs, so each predicate is also checked in a
residual left
+ * unpushed.
+ */
+ private def checkResidualPredicates(table: String, predicates: Seq[String]):
Unit =
+ for (predicate <- predicates; prefix <- Seq("", "l IS NOT NULL AND ")) {
+ checkIcebergNativeScan(s"SELECT id FROM $table WHERE
$prefix($predicate)")
+ }
+
+ test("floating-point residuals preserve signed zeros in native scans") {
+ assume(icebergAvailable, "Iceberg not available in classpath")
+
+ withHadoopCatalog("test_cat") {
+ withSQLConf(
+ CometConf.COMET_ENABLED.key -> "true",
+ CometConf.COMET_EXEC_ENABLED.key -> "true",
+ CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+ val table = "test_cat.db.floating_residual_test"
+ spark.sql(s"CREATE TABLE $table (id INT, f FLOAT, d DOUBLE, l
ARRAY<INT>) USING iceberg")
+ try {
+ withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+ // One file containing both zeros keeps file pruning from deciding
the comparison.
+ // Distinct ids make a missing signed-zero row observable without
comparing floats.
+ spark
+ .sql("""
+ SELECT id, CAST(v AS FLOAT) AS f, CAST(v AS DOUBLE) AS d, l
+ FROM VALUES
+ (1, '-0.0', array(1)),
+ (2, '0.0', array(1)),
+ (3, '1.0', CAST(NULL AS ARRAY<INT>)),
+ (4, '-1.0', array(1)),
+ (5, NULL, array(1)),
+ (6, 'NaN', array(1))
+ AS input(id, v, l)
+ """)
+ .coalesce(1)
+ .write
+ .format("iceberg")
+ .mode("append")
+ .saveAsTable(table)
+ assert(spark.sql(s"SELECT file_path FROM
$table.files").collect().length == 1)
+ val rows = spark.sql(s"SELECT id, f, d FROM
$table").collect().sortBy(_.getInt(0))
+ assert(java.lang.Float.floatToRawIntBits(rows(0).getFloat(1)) ==
Int.MinValue)
+ assert(java.lang.Double.doubleToRawLongBits(rows(0).getDouble(2))
== Long.MinValue)
+ assert(java.lang.Float.floatToRawIntBits(rows(1).getFloat(1)) == 0)
+ assert(java.lang.Double.doubleToRawLongBits(rows(1).getDouble(2))
== 0L)
+ }
+
+ for ((column, dataType) <- Seq("f" -> "FLOAT", "d" -> "DOUBLE")) {
+ val zero = s"CAST(0 AS $dataType)"
+ val negativeZero = s"CAST('-0.0' AS $dataType)"
+ val one = s"CAST(1 AS $dataType)"
+ val minusOne = s"CAST(-1 AS $dataType)"
+ val predicates = Seq(
+ s"$column = $zero",
+ s"$column = $negativeZero",
+ s"$column >= $zero",
+ s"$column <= $negativeZero",
+ s"$column IN ($zero, $one)",
+ s"NOT ($column < $zero)",
+ // Nonzero literals push equality and IN; range comparisons stay
out.
+ s"$column = $one",
+ s"$column IN ($one, $minusOne)",
+ s"NOT ($column IN ($one, $minusOne) AND id > 1)",
+ s"$column < $one",
+ s"$column > $minusOne",
+ s"NOT ($column = $one)",
+ s"$column = $zero OR id = 3",
+ s"NOT ($column = $zero AND id > 1)",
+ s"$column IS NULL",
+ s"$column IS NOT NULL")
+ checkResidualPredicates(table, predicates)
+ }
+ } finally {
+ spark.sql(s"DROP TABLE $table")
+ }
+ }
+ }
+ }
+
+ test("negated null-safe equality keeps NULL rows in native scans") {
+ assume(icebergAvailable, "Iceberg not available in classpath")
+
+ withHadoopCatalog("test_cat") {
+ withSQLConf(
+ CometConf.COMET_ENABLED.key -> "true",
+ CometConf.COMET_EXEC_ENABLED.key -> "true",
+ CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+ val table = "test_cat.db.null_safe_residual_test"
+ spark.sql(s"CREATE TABLE $table (id INT, b BOOLEAN, l ARRAY<INT>)
USING iceberg")
+ try {
+ withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+ // One file keeps file pruning from deciding the comparison.
+ spark
+ .sql("""
+ SELECT * FROM VALUES
+ (NULL, NULL, array(1)), (1, true, array(1)), (2, false,
array(1))
+ AS t(id, b, l)
+ """)
+ .coalesce(1)
+ .write
+ .format("iceberg")
+ .mode("append")
+ .saveAsTable(table)
+ assert(spark.sql(s"SELECT file_path FROM
$table.files").collect().length == 1)
+ }
+ // Iceberg converts `id <=> 1` to an equality, which is false for a
NULL id in Spark but
+ // NULL in the native reader, so only its negation can drop the NULL
row. Spark also
+ // writes that negation as IS DISTINCT FROM and, for booleans, IS
NOT TRUE. A null
+ // check ANDed with it rejects the NULL row either way.
+ val predicates = Seq(
+ "NOT (id <=> 1)",
+ "NOT (id <=> 1 AND id < 5)",
+ "NOT (id <=> 1) OR id = 2",
+ "id IS DISTINCT FROM 1",
+ "b IS NOT TRUE",
+ "NOT b",
+ "b = false",
+ "id IS NOT NULL AND NOT (id <=> 1)",
+ "NOT (id = 1)",
+ "id != 1")
+ checkResidualPredicates(table, predicates)
+ // `id != 1` reaches Iceberg as a not-equal, and `b = false` as `b
IS NOT NULL AND NOT b`,
+ // whose null check covers the negated equality, so both are still
pushed. A negated
+ // `<=>` has no such check and stays out.
+ // Planning commonData leaks manifest streams on Iceberg versions
before 1.8.0.
+ if (icebergVersionAtLeast(1, 8)) {
+ for ((predicate, pushed) <- Seq(
+ "b = false" -> true,
+ "id != 1" -> true,
+ "id IS NOT NULL AND NOT (id <=> 1)" -> true,
+ "NOT (id <=> 1)" -> false,
+ "b IS NOT TRUE" -> false)) {
+ val (_, cometPlan) = checkSparkAnswer(s"SELECT id FROM $table
WHERE $predicate")
+ val scans = collectIcebergNativeScans(cometPlan)
+ assert(scans.length == 1, s"$cometPlan")
+ val common =
OperatorOuterClass.IcebergScanCommon.parseFrom(scans.head.commonData)
+ assert(common.getResidualPoolCount == (if (pushed) 1 else 0),
s"$predicate")
+ }
+ }
+ } finally {
+ spark.sql(s"DROP TABLE $table")
+ }
+ }
+ }
+ }
+
+ test("residuals on tables with a name mapping keep rows from imported
files") {
+ assume(icebergAvailable, "Iceberg not available in classpath")
+
+ withHadoopCatalog("test_cat") {
+ withSQLConf(
+ CometConf.COMET_ENABLED.key -> "true",
+ CometConf.COMET_EXEC_ENABLED.key -> "true",
+ CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+ val table = "test_cat.db.imported_residual_test"
+ spark.sql(s"CREATE TABLE $table (id BIGINT, t INT, d DOUBLE) USING
iceberg")
+ try {
+ withTempDir { dir =>
+ // The file stores id as INT, t as TINYINT and d as FLOAT.
Appending it the way
+ // add_files does adds a name mapping but records no schema change.
+ val source = new File(dir, "source").getAbsolutePath
+ withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+ spark
+ .sql("SELECT CAST(id AS INT) AS id, CAST(id AS TINYINT) AS t,
" +
+ "CAST(id AS FLOAT) AS d FROM range(1, 4)")
+ .coalesce(1)
+ .write
+ .parquet(source)
+ }
+ val icebergTable = loadIcebergTable(spark, "test_cat", "db",
"imported_residual_test")
+ .asInstanceOf[org.apache.iceberg.Table]
+ icebergTable
+ .updateProperties()
+ .set(
+ org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING,
+ org.apache.iceberg.mapping.NameMappingParser
+
.toJson(org.apache.iceberg.mapping.MappingUtil.create(icebergTable.schema())))
+ .commit()
+ val file = new
File(source).listFiles().find(_.getName.endsWith(".parquet")).get
+ icebergTable
+ .newAppend()
+ .appendFile(
+ org.apache.iceberg.DataFiles
+ .builder(icebergTable.spec())
+ .withPath(file.getAbsolutePath)
+ .withFileSizeInBytes(file.length())
+ .withRecordCount(3L)
+ .withFormat(org.apache.iceberg.FileFormat.PARQUET)
+ .build())
+ .commit()
Review Comment:
The guard depends on `add_files`, `migrate` and `snapshot` setting
`schema.name-mapping.default`, but the test sets the mapping itself. Would
`SparkActions.get().snapshotTable(...)`, as used in `migration - nested name
mapping falls back to Spark`, work with a source Parquet table declared as `id
BIGINT, t INT, d DOUBLE` over this file? Then the test would also catch Iceberg
no longer setting the mapping. I have not tried whether the import accepts the
type mismatch.
##########
spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala:
##########
@@ -4306,6 +4306,307 @@ class CometIcebergNativeSuite
}
}
+ /**
+ * Compares `SELECT id FROM table WHERE predicate` with Spark for each
predicate, alone and
+ * after `l IS NOT NULL AND`. That array null check is a conjunct the serde
never pushes, as in
+ * the queries that found these bugs, so each predicate is also checked in a
residual left
+ * unpushed.
+ */
+ private def checkResidualPredicates(table: String, predicates: Seq[String]):
Unit =
+ for (predicate <- predicates; prefix <- Seq("", "l IS NOT NULL AND ")) {
+ checkIcebergNativeScan(s"SELECT id FROM $table WHERE
$prefix($predicate)")
+ }
+
+ test("floating-point residuals preserve signed zeros in native scans") {
+ assume(icebergAvailable, "Iceberg not available in classpath")
+
+ withHadoopCatalog("test_cat") {
+ withSQLConf(
+ CometConf.COMET_ENABLED.key -> "true",
+ CometConf.COMET_EXEC_ENABLED.key -> "true",
+ CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+ val table = "test_cat.db.floating_residual_test"
+ spark.sql(s"CREATE TABLE $table (id INT, f FLOAT, d DOUBLE, l
ARRAY<INT>) USING iceberg")
+ try {
+ withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+ // One file containing both zeros keeps file pruning from deciding
the comparison.
+ // Distinct ids make a missing signed-zero row observable without
comparing floats.
+ spark
+ .sql("""
+ SELECT id, CAST(v AS FLOAT) AS f, CAST(v AS DOUBLE) AS d, l
+ FROM VALUES
+ (1, '-0.0', array(1)),
+ (2, '0.0', array(1)),
+ (3, '1.0', CAST(NULL AS ARRAY<INT>)),
+ (4, '-1.0', array(1)),
+ (5, NULL, array(1)),
+ (6, 'NaN', array(1))
+ AS input(id, v, l)
+ """)
+ .coalesce(1)
+ .write
+ .format("iceberg")
+ .mode("append")
+ .saveAsTable(table)
+ assert(spark.sql(s"SELECT file_path FROM
$table.files").collect().length == 1)
+ val rows = spark.sql(s"SELECT id, f, d FROM
$table").collect().sortBy(_.getInt(0))
+ assert(java.lang.Float.floatToRawIntBits(rows(0).getFloat(1)) ==
Int.MinValue)
+ assert(java.lang.Double.doubleToRawLongBits(rows(0).getDouble(2))
== Long.MinValue)
+ assert(java.lang.Float.floatToRawIntBits(rows(1).getFloat(1)) == 0)
+ assert(java.lang.Double.doubleToRawLongBits(rows(1).getDouble(2))
== 0L)
+ }
+
+ for ((column, dataType) <- Seq("f" -> "FLOAT", "d" -> "DOUBLE")) {
+ val zero = s"CAST(0 AS $dataType)"
+ val negativeZero = s"CAST('-0.0' AS $dataType)"
+ val one = s"CAST(1 AS $dataType)"
+ val minusOne = s"CAST(-1 AS $dataType)"
+ val predicates = Seq(
+ s"$column = $zero",
+ s"$column = $negativeZero",
+ s"$column >= $zero",
+ s"$column <= $negativeZero",
+ s"$column IN ($zero, $one)",
+ s"NOT ($column < $zero)",
+ // Nonzero literals push equality and IN; range comparisons stay
out.
+ s"$column = $one",
+ s"$column IN ($one, $minusOne)",
+ s"NOT ($column IN ($one, $minusOne) AND id > 1)",
+ s"$column < $one",
+ s"$column > $minusOne",
+ s"NOT ($column = $one)",
+ s"$column = $zero OR id = 3",
+ s"NOT ($column = $zero AND id > 1)",
+ s"$column IS NULL",
+ s"$column IS NOT NULL")
+ checkResidualPredicates(table, predicates)
+ }
+ } finally {
+ spark.sql(s"DROP TABLE $table")
+ }
+ }
+ }
+ }
+
+ test("negated null-safe equality keeps NULL rows in native scans") {
+ assume(icebergAvailable, "Iceberg not available in classpath")
+
+ withHadoopCatalog("test_cat") {
+ withSQLConf(
+ CometConf.COMET_ENABLED.key -> "true",
+ CometConf.COMET_EXEC_ENABLED.key -> "true",
+ CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+ val table = "test_cat.db.null_safe_residual_test"
+ spark.sql(s"CREATE TABLE $table (id INT, b BOOLEAN, l ARRAY<INT>)
USING iceberg")
+ try {
+ withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+ // One file keeps file pruning from deciding the comparison.
+ spark
+ .sql("""
+ SELECT * FROM VALUES
+ (NULL, NULL, array(1)), (1, true, array(1)), (2, false,
array(1))
+ AS t(id, b, l)
+ """)
+ .coalesce(1)
+ .write
+ .format("iceberg")
+ .mode("append")
+ .saveAsTable(table)
+ assert(spark.sql(s"SELECT file_path FROM
$table.files").collect().length == 1)
+ }
+ // Iceberg converts `id <=> 1` to an equality, which is false for a
NULL id in Spark but
+ // NULL in the native reader, so only its negation can drop the NULL
row. Spark also
+ // writes that negation as IS DISTINCT FROM and, for booleans, IS
NOT TRUE. A null
+ // check ANDed with it rejects the NULL row either way.
+ val predicates = Seq(
+ "NOT (id <=> 1)",
+ "NOT (id <=> 1 AND id < 5)",
+ "NOT (id <=> 1) OR id = 2",
+ "id IS DISTINCT FROM 1",
+ "b IS NOT TRUE",
+ "NOT b",
+ "b = false",
+ "id IS NOT NULL AND NOT (id <=> 1)",
+ "NOT (id = 1)",
+ "id != 1")
+ checkResidualPredicates(table, predicates)
+ // `id != 1` reaches Iceberg as a not-equal, and `b = false` as `b
IS NOT NULL AND NOT b`,
+ // whose null check covers the negated equality, so both are still
pushed. A negated
+ // `<=>` has no such check and stays out.
+ // Planning commonData leaks manifest streams on Iceberg versions
before 1.8.0.
+ if (icebergVersionAtLeast(1, 8)) {
+ for ((predicate, pushed) <- Seq(
+ "b = false" -> true,
+ "id != 1" -> true,
+ "id IS NOT NULL AND NOT (id <=> 1)" -> true,
+ "NOT (id <=> 1)" -> false,
+ "b IS NOT TRUE" -> false)) {
+ val (_, cometPlan) = checkSparkAnswer(s"SELECT id FROM $table
WHERE $predicate")
+ val scans = collectIcebergNativeScans(cometPlan)
+ assert(scans.length == 1, s"$cometPlan")
+ val common =
OperatorOuterClass.IcebergScanCommon.parseFrom(scans.head.commonData)
+ assert(common.getResidualPoolCount == (if (pushed) 1 else 0),
s"$predicate")
+ }
+ }
+ } finally {
+ spark.sql(s"DROP TABLE $table")
+ }
+ }
+ }
+ }
+
+ test("residuals on tables with a name mapping keep rows from imported
files") {
+ assume(icebergAvailable, "Iceberg not available in classpath")
+
+ withHadoopCatalog("test_cat") {
+ withSQLConf(
+ CometConf.COMET_ENABLED.key -> "true",
+ CometConf.COMET_EXEC_ENABLED.key -> "true",
+ CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+ val table = "test_cat.db.imported_residual_test"
+ spark.sql(s"CREATE TABLE $table (id BIGINT, t INT, d DOUBLE) USING
iceberg")
+ try {
+ withTempDir { dir =>
+ // The file stores id as INT, t as TINYINT and d as FLOAT.
Appending it the way
+ // add_files does adds a name mapping but records no schema change.
+ val source = new File(dir, "source").getAbsolutePath
+ withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+ spark
+ .sql("SELECT CAST(id AS INT) AS id, CAST(id AS TINYINT) AS t,
" +
+ "CAST(id AS FLOAT) AS d FROM range(1, 4)")
+ .coalesce(1)
+ .write
+ .parquet(source)
+ }
+ val icebergTable = loadIcebergTable(spark, "test_cat", "db",
"imported_residual_test")
+ .asInstanceOf[org.apache.iceberg.Table]
+ icebergTable
+ .updateProperties()
+ .set(
+ org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING,
+ org.apache.iceberg.mapping.NameMappingParser
+
.toJson(org.apache.iceberg.mapping.MappingUtil.create(icebergTable.schema())))
+ .commit()
+ val file = new
File(source).listFiles().find(_.getName.endsWith(".parquet")).get
+ icebergTable
+ .newAppend()
+ .appendFile(
+ org.apache.iceberg.DataFiles
+ .builder(icebergTable.spec())
+ .withPath(file.getAbsolutePath)
+ .withFileSizeInBytes(file.length())
+ .withRecordCount(3L)
+ .withFormat(org.apache.iceberg.FileFormat.PARQUET)
+ .build())
+ .commit()
+ spark.sql(s"REFRESH TABLE $table")
+
+ // iceberg-rust skips the imported file's pages for these, or
casts 200 and
+ // 3000000000 to a NULL TINYINT and INT.
+ val predicates = Seq(
+ "id > 0",
+ "id < 3000000000",
+ "id IN (1, 2)",
+ "t < 200",
+ "t IN (1, 200)",
+ "d > 0.5D",
Review Comment:
`d > 0.5D` never reaches the native scan because of the `FLOAT`/`DOUBLE`
range rule, whether or not the table has a name mapping, so it passes with this
guard disabled. Is it there for when #7588 restores range pruning on `DOUBLE`?
If so, a short comment would say so. Otherwise `d IN (1.0D, 2.0D)` is the case
that exercises `d`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]