comphead commented on code in PR #7233:
URL: https://github.com/apache/datafusion-comet/pull/7233#discussion_r4579934278


##########
spark/src/main/scala/org/apache/comet/iceberg/IcebergReflection.scala:
##########
@@ -1216,6 +1216,69 @@ object IcebergReflection extends Logging {
     }
   }
 
+  /**
+   * Top-level column names of `schema` whose type differs in some schema of 
the table's history
+   * (`table.schemas()`), such as an int column promoted to long. Data files 
written before the
+   * promotion keep the older type, and iceberg-rust compares literals against 
it incorrectly: the
+   * page index skips every page for range and IN predicates, and the row 
filter casts the literal
+   * to the older type, so an out-of-range value becomes NULL 
(apache/iceberg-rust#3122).
+   *
+   * Expiring snapshots with `cleanExpiredMetadata` removes schemas that no 
retained snapshot
+   * uses, even though files written under them can remain. Iceberg numbers 
schemas consecutively
+   * from 0, so a gap means the history is incomplete, and every column of a 
type that a promotion
+   * can produce (long, double, decimal) is returned. The history cannot 
reveal a file whose type
+   * differs without a recorded schema change, such as one imported by 
`add_files`, `migrate` or
+   * `snapshot`, or one promoted under metadata from before Iceberg 0.12, 
which kept only the
+   * current schema. Called at serialization time, so it throws on reflection 
failure.
+   */
+  def promotedColumns(table: Any, schema: Any): Set[String] = {
+    import scala.jdk.CollectionConverters._
+    val findField = getMethod(schema.getClass, "findField", classOf[Int])
+    def typeOf(field: Any): String = getMethod(field.getClass, 
"type").invoke(field).toString
+    val schemasById = getMethod(table.getClass, "schemas")
+      .invoke(table)
+      .asInstanceOf[java.util.Map[_, _]]
+      .asScala
+    val history = schemasById.values.toSeq
+    val ids = schemasById.keys.map(_.asInstanceOf[Integer].intValue).toSet
+    val historyComplete = ids.nonEmpty && ids.min == 0 && ids.max == ids.size 
- 1
+    getMethod(schema.getClass, "columns")
+      .invoke(schema)
+      .asInstanceOf[java.util.List[_]]
+      .asScala
+      .filter { column =>
+        val id = fieldIdOf(column).asInstanceOf[AnyRef]
+        val current = typeOf(column)
+        val promotable =
+          current == "long" || current == "double" || 
current.startsWith("decimal(")
+        (promotable && !historyComplete) ||
+        history.exists(older => Option(findField.invoke(older, 
id)).exists(typeOf(_) != current))
+      }
+      .map(fieldName)
+      .toSet
+  }
+
+  /**
+   * Top-level column names of `schema` whose Parquet files can store a 
narrower integer or
+   * floating-point type than the column: int (Hive TINYINT and SMALLINT), 
long (int) and double
+   * (float). Files imported by `add_files`, `migrate` or `snapshot` are not 
checked against the
+   * schema, so the schema history cannot reveal them, but those procedures 
set a name mapping.
+   * Callers treat these columns as promoted for a table that has one.
+   */
+  def narrowableColumns(schema: Any): Set[String] = {
+    import scala.jdk.CollectionConverters._
+    getMethod(schema.getClass, "columns")
+      .invoke(schema)
+      .asInstanceOf[java.util.List[_]]
+      .asScala
+      .filter { column =>
+        Set("int", "long", "double").contains(
+          getMethod(column.getClass, "type").invoke(column).toString)
+      }
+      .map(fieldName)
+      .toSet

Review Comment:
   `narrowableColumns` walks `columns` and reads each `type` the same way 
`promotedColumns` does, but it keeps a separate type list. The latest commit on 
#7156 (e1afd07abe) adds `timestamp` and `timestamp_ns` to the `promotable` list 
in `promotedColumns`, so the two lists already differ after the rebase. Could 
this become a parameter of `promotedColumns`, for example `filesMayBeNarrower: 
Boolean`? It would add `int` to the promotable types and treat them as 
`!historyComplete` does. That keeps one list and also stops building 
`Set("int", "long", "double")` once per column. Decimal columns would join the 
set too. That is harmless, because `pageIndexUnsupportedColumns` already keeps 
every decimal residual out of the scan.



##########
spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala:
##########
@@ -1009,6 +1086,19 @@ object CometIcebergNativeScan extends 
CometOperatorSerde[CometBatchScanExec] wit
     // schema so a filter column projected out of the scan output is still 
recognized.
     val pageIndexUnsupportedColumns =
       IcebergReflection.pageIndexUnsupportedColumns(metadata.tableSchema)
+    // Columns whose type changed over the table's history; residual 
predicates comparing them
+    // with a literal are dropped, except equality. The scan schema also 
covers a column that
+    // VERSION AS OF reads after it was dropped from the table.
+    // add_files, migrate and snapshot import files without checking their 
types against the
+    // schema, so the history records nothing for them, but they set a name 
mapping. A table with
+    // one is treated as if every column that can hold a narrower type were 
promoted.
+    val schemas = Seq(metadata.tableSchema, metadata.scanSchema)
+    val recordedPromotions =
+      schemas.flatMap(IcebergReflection.promotedColumns(metadata.table, _))
+    val importedNarrowings =
+      if (metadata.nameMapping.isDefined) 
schemas.flatMap(IcebergReflection.narrowableColumns)
+      else Nil
+    val promotedColumns = (recordedPromotions ++ importedNarrowings).toSet

Review Comment:
   The comment block above still describes only the schema history, and the new 
block repeats the reason that the `narrowableColumns` scaladoc gives. Could 
they be merged into one comment? It would also help to say here that this is a 
workaround for apache/iceberg-rust#3122. apache/iceberg-rust#3540 (row filter) 
and apache/iceberg-rust#3507 (page index) are open, and once the pin includes 
both, this guard and the one for recorded promotions can be removed. Until 
then, `IN` lookups on `BIGINT` keys of migrated tables lose row group pruning 
in the native scan, which is a common pattern.



##########
spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala:
##########
@@ -4306,6 +4306,307 @@ class CometIcebergNativeSuite
     }
   }
 
+  /**
+   * Compares `SELECT id FROM table WHERE predicate` with Spark for each 
predicate, alone and
+   * after `l IS NOT NULL AND`. That array null check is a conjunct the serde 
never pushes, as in
+   * the queries that found these bugs, so each predicate is also checked in a 
residual left
+   * unpushed.
+   */
+  private def checkResidualPredicates(table: String, predicates: Seq[String]): 
Unit =
+    for (predicate <- predicates; prefix <- Seq("", "l IS NOT NULL AND ")) {
+      checkIcebergNativeScan(s"SELECT id FROM $table WHERE 
$prefix($predicate)")
+    }
+
+  test("floating-point residuals preserve signed zeros in native scans") {
+    assume(icebergAvailable, "Iceberg not available in classpath")
+
+    withHadoopCatalog("test_cat") {
+      withSQLConf(
+        CometConf.COMET_ENABLED.key -> "true",
+        CometConf.COMET_EXEC_ENABLED.key -> "true",
+        CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+        val table = "test_cat.db.floating_residual_test"
+        spark.sql(s"CREATE TABLE $table (id INT, f FLOAT, d DOUBLE, l 
ARRAY<INT>) USING iceberg")
+        try {
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            // One file containing both zeros keeps file pruning from deciding 
the comparison.
+            // Distinct ids make a missing signed-zero row observable without 
comparing floats.
+            spark
+              .sql("""
+                SELECT id, CAST(v AS FLOAT) AS f, CAST(v AS DOUBLE) AS d, l
+                FROM VALUES
+                  (1, '-0.0', array(1)),
+                  (2, '0.0', array(1)),
+                  (3, '1.0', CAST(NULL AS ARRAY<INT>)),
+                  (4, '-1.0', array(1)),
+                  (5, NULL, array(1)),
+                  (6, 'NaN', array(1))
+                AS input(id, v, l)
+              """)
+              .coalesce(1)
+              .write
+              .format("iceberg")
+              .mode("append")
+              .saveAsTable(table)
+            assert(spark.sql(s"SELECT file_path FROM 
$table.files").collect().length == 1)
+            val rows = spark.sql(s"SELECT id, f, d FROM 
$table").collect().sortBy(_.getInt(0))
+            assert(java.lang.Float.floatToRawIntBits(rows(0).getFloat(1)) == 
Int.MinValue)
+            assert(java.lang.Double.doubleToRawLongBits(rows(0).getDouble(2)) 
== Long.MinValue)
+            assert(java.lang.Float.floatToRawIntBits(rows(1).getFloat(1)) == 0)
+            assert(java.lang.Double.doubleToRawLongBits(rows(1).getDouble(2)) 
== 0L)
+          }
+
+          for ((column, dataType) <- Seq("f" -> "FLOAT", "d" -> "DOUBLE")) {
+            val zero = s"CAST(0 AS $dataType)"
+            val negativeZero = s"CAST('-0.0' AS $dataType)"
+            val one = s"CAST(1 AS $dataType)"
+            val minusOne = s"CAST(-1 AS $dataType)"
+            val predicates = Seq(
+              s"$column = $zero",
+              s"$column = $negativeZero",
+              s"$column >= $zero",
+              s"$column <= $negativeZero",
+              s"$column IN ($zero, $one)",
+              s"NOT ($column < $zero)",
+              // Nonzero literals push equality and IN; range comparisons stay 
out.
+              s"$column = $one",
+              s"$column IN ($one, $minusOne)",
+              s"NOT ($column IN ($one, $minusOne) AND id > 1)",
+              s"$column < $one",
+              s"$column > $minusOne",
+              s"NOT ($column = $one)",
+              s"$column = $zero OR id = 3",
+              s"NOT ($column = $zero AND id > 1)",
+              s"$column IS NULL",
+              s"$column IS NOT NULL")
+            checkResidualPredicates(table, predicates)
+          }
+        } finally {
+          spark.sql(s"DROP TABLE $table")
+        }
+      }
+    }
+  }
+
+  test("negated null-safe equality keeps NULL rows in native scans") {
+    assume(icebergAvailable, "Iceberg not available in classpath")
+
+    withHadoopCatalog("test_cat") {
+      withSQLConf(
+        CometConf.COMET_ENABLED.key -> "true",
+        CometConf.COMET_EXEC_ENABLED.key -> "true",
+        CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+        val table = "test_cat.db.null_safe_residual_test"
+        spark.sql(s"CREATE TABLE $table (id INT, b BOOLEAN, l ARRAY<INT>) 
USING iceberg")
+        try {
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            // One file keeps file pruning from deciding the comparison.
+            spark
+              .sql("""
+                SELECT * FROM VALUES
+                  (NULL, NULL, array(1)), (1, true, array(1)), (2, false, 
array(1))
+                AS t(id, b, l)
+              """)
+              .coalesce(1)
+              .write
+              .format("iceberg")
+              .mode("append")
+              .saveAsTable(table)
+            assert(spark.sql(s"SELECT file_path FROM 
$table.files").collect().length == 1)
+          }
+          // Iceberg converts `id <=> 1` to an equality, which is false for a 
NULL id in Spark but
+          // NULL in the native reader, so only its negation can drop the NULL 
row. Spark also
+          // writes that negation as IS DISTINCT FROM and, for booleans, IS 
NOT TRUE. A null
+          // check ANDed with it rejects the NULL row either way.
+          val predicates = Seq(
+            "NOT (id <=> 1)",
+            "NOT (id <=> 1 AND id < 5)",
+            "NOT (id <=> 1) OR id = 2",
+            "id IS DISTINCT FROM 1",
+            "b IS NOT TRUE",
+            "NOT b",
+            "b = false",
+            "id IS NOT NULL AND NOT (id <=> 1)",
+            "NOT (id = 1)",
+            "id != 1")
+          checkResidualPredicates(table, predicates)
+          // `id != 1` reaches Iceberg as a not-equal, and `b = false` as `b 
IS NOT NULL AND NOT b`,
+          // whose null check covers the negated equality, so both are still 
pushed. A negated
+          // `<=>` has no such check and stays out.
+          // Planning commonData leaks manifest streams on Iceberg versions 
before 1.8.0.
+          if (icebergVersionAtLeast(1, 8)) {
+            for ((predicate, pushed) <- Seq(
+                "b = false" -> true,
+                "id != 1" -> true,
+                "id IS NOT NULL AND NOT (id <=> 1)" -> true,
+                "NOT (id <=> 1)" -> false,
+                "b IS NOT TRUE" -> false)) {
+              val (_, cometPlan) = checkSparkAnswer(s"SELECT id FROM $table 
WHERE $predicate")
+              val scans = collectIcebergNativeScans(cometPlan)
+              assert(scans.length == 1, s"$cometPlan")
+              val common = 
OperatorOuterClass.IcebergScanCommon.parseFrom(scans.head.commonData)
+              assert(common.getResidualPoolCount == (if (pushed) 1 else 0), 
s"$predicate")
+            }
+          }
+        } finally {
+          spark.sql(s"DROP TABLE $table")
+        }
+      }
+    }
+  }
+
+  test("residuals on tables with a name mapping keep rows from imported 
files") {
+    assume(icebergAvailable, "Iceberg not available in classpath")
+
+    withHadoopCatalog("test_cat") {
+      withSQLConf(
+        CometConf.COMET_ENABLED.key -> "true",
+        CometConf.COMET_EXEC_ENABLED.key -> "true",
+        CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+        val table = "test_cat.db.imported_residual_test"
+        spark.sql(s"CREATE TABLE $table (id BIGINT, t INT, d DOUBLE) USING 
iceberg")
+        try {
+          withTempDir { dir =>
+            // The file stores id as INT, t as TINYINT and d as FLOAT. 
Appending it the way
+            // add_files does adds a name mapping but records no schema change.
+            val source = new File(dir, "source").getAbsolutePath
+            withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+              spark
+                .sql("SELECT CAST(id AS INT) AS id, CAST(id AS TINYINT) AS t, 
" +
+                  "CAST(id AS FLOAT) AS d FROM range(1, 4)")
+                .coalesce(1)
+                .write
+                .parquet(source)
+            }
+            val icebergTable = loadIcebergTable(spark, "test_cat", "db", 
"imported_residual_test")
+              .asInstanceOf[org.apache.iceberg.Table]
+            icebergTable
+              .updateProperties()
+              .set(
+                org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING,
+                org.apache.iceberg.mapping.NameMappingParser
+                  
.toJson(org.apache.iceberg.mapping.MappingUtil.create(icebergTable.schema())))
+              .commit()
+            val file = new 
File(source).listFiles().find(_.getName.endsWith(".parquet")).get
+            icebergTable
+              .newAppend()
+              .appendFile(
+                org.apache.iceberg.DataFiles
+                  .builder(icebergTable.spec())
+                  .withPath(file.getAbsolutePath)
+                  .withFileSizeInBytes(file.length())
+                  .withRecordCount(3L)
+                  .withFormat(org.apache.iceberg.FileFormat.PARQUET)
+                  .build())
+              .commit()

Review Comment:
   The guard depends on `add_files`, `migrate` and `snapshot` setting 
`schema.name-mapping.default`, but the test sets the mapping itself. Would 
`SparkActions.get().snapshotTable(...)`, as used in `migration - nested name 
mapping falls back to Spark`, work with a source Parquet table declared as `id 
BIGINT, t INT, d DOUBLE` over this file? Then the test would also catch Iceberg 
no longer setting the mapping. I have not tried whether the import accepts the 
type mismatch.



##########
spark/src/test/scala/org/apache/comet/CometIcebergNativeSuite.scala:
##########
@@ -4306,6 +4306,307 @@ class CometIcebergNativeSuite
     }
   }
 
+  /**
+   * Compares `SELECT id FROM table WHERE predicate` with Spark for each 
predicate, alone and
+   * after `l IS NOT NULL AND`. That array null check is a conjunct the serde 
never pushes, as in
+   * the queries that found these bugs, so each predicate is also checked in a 
residual left
+   * unpushed.
+   */
+  private def checkResidualPredicates(table: String, predicates: Seq[String]): 
Unit =
+    for (predicate <- predicates; prefix <- Seq("", "l IS NOT NULL AND ")) {
+      checkIcebergNativeScan(s"SELECT id FROM $table WHERE 
$prefix($predicate)")
+    }
+
+  test("floating-point residuals preserve signed zeros in native scans") {
+    assume(icebergAvailable, "Iceberg not available in classpath")
+
+    withHadoopCatalog("test_cat") {
+      withSQLConf(
+        CometConf.COMET_ENABLED.key -> "true",
+        CometConf.COMET_EXEC_ENABLED.key -> "true",
+        CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+        val table = "test_cat.db.floating_residual_test"
+        spark.sql(s"CREATE TABLE $table (id INT, f FLOAT, d DOUBLE, l 
ARRAY<INT>) USING iceberg")
+        try {
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            // One file containing both zeros keeps file pruning from deciding 
the comparison.
+            // Distinct ids make a missing signed-zero row observable without 
comparing floats.
+            spark
+              .sql("""
+                SELECT id, CAST(v AS FLOAT) AS f, CAST(v AS DOUBLE) AS d, l
+                FROM VALUES
+                  (1, '-0.0', array(1)),
+                  (2, '0.0', array(1)),
+                  (3, '1.0', CAST(NULL AS ARRAY<INT>)),
+                  (4, '-1.0', array(1)),
+                  (5, NULL, array(1)),
+                  (6, 'NaN', array(1))
+                AS input(id, v, l)
+              """)
+              .coalesce(1)
+              .write
+              .format("iceberg")
+              .mode("append")
+              .saveAsTable(table)
+            assert(spark.sql(s"SELECT file_path FROM 
$table.files").collect().length == 1)
+            val rows = spark.sql(s"SELECT id, f, d FROM 
$table").collect().sortBy(_.getInt(0))
+            assert(java.lang.Float.floatToRawIntBits(rows(0).getFloat(1)) == 
Int.MinValue)
+            assert(java.lang.Double.doubleToRawLongBits(rows(0).getDouble(2)) 
== Long.MinValue)
+            assert(java.lang.Float.floatToRawIntBits(rows(1).getFloat(1)) == 0)
+            assert(java.lang.Double.doubleToRawLongBits(rows(1).getDouble(2)) 
== 0L)
+          }
+
+          for ((column, dataType) <- Seq("f" -> "FLOAT", "d" -> "DOUBLE")) {
+            val zero = s"CAST(0 AS $dataType)"
+            val negativeZero = s"CAST('-0.0' AS $dataType)"
+            val one = s"CAST(1 AS $dataType)"
+            val minusOne = s"CAST(-1 AS $dataType)"
+            val predicates = Seq(
+              s"$column = $zero",
+              s"$column = $negativeZero",
+              s"$column >= $zero",
+              s"$column <= $negativeZero",
+              s"$column IN ($zero, $one)",
+              s"NOT ($column < $zero)",
+              // Nonzero literals push equality and IN; range comparisons stay 
out.
+              s"$column = $one",
+              s"$column IN ($one, $minusOne)",
+              s"NOT ($column IN ($one, $minusOne) AND id > 1)",
+              s"$column < $one",
+              s"$column > $minusOne",
+              s"NOT ($column = $one)",
+              s"$column = $zero OR id = 3",
+              s"NOT ($column = $zero AND id > 1)",
+              s"$column IS NULL",
+              s"$column IS NOT NULL")
+            checkResidualPredicates(table, predicates)
+          }
+        } finally {
+          spark.sql(s"DROP TABLE $table")
+        }
+      }
+    }
+  }
+
+  test("negated null-safe equality keeps NULL rows in native scans") {
+    assume(icebergAvailable, "Iceberg not available in classpath")
+
+    withHadoopCatalog("test_cat") {
+      withSQLConf(
+        CometConf.COMET_ENABLED.key -> "true",
+        CometConf.COMET_EXEC_ENABLED.key -> "true",
+        CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+        val table = "test_cat.db.null_safe_residual_test"
+        spark.sql(s"CREATE TABLE $table (id INT, b BOOLEAN, l ARRAY<INT>) 
USING iceberg")
+        try {
+          withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+            // One file keeps file pruning from deciding the comparison.
+            spark
+              .sql("""
+                SELECT * FROM VALUES
+                  (NULL, NULL, array(1)), (1, true, array(1)), (2, false, 
array(1))
+                AS t(id, b, l)
+              """)
+              .coalesce(1)
+              .write
+              .format("iceberg")
+              .mode("append")
+              .saveAsTable(table)
+            assert(spark.sql(s"SELECT file_path FROM 
$table.files").collect().length == 1)
+          }
+          // Iceberg converts `id <=> 1` to an equality, which is false for a 
NULL id in Spark but
+          // NULL in the native reader, so only its negation can drop the NULL 
row. Spark also
+          // writes that negation as IS DISTINCT FROM and, for booleans, IS 
NOT TRUE. A null
+          // check ANDed with it rejects the NULL row either way.
+          val predicates = Seq(
+            "NOT (id <=> 1)",
+            "NOT (id <=> 1 AND id < 5)",
+            "NOT (id <=> 1) OR id = 2",
+            "id IS DISTINCT FROM 1",
+            "b IS NOT TRUE",
+            "NOT b",
+            "b = false",
+            "id IS NOT NULL AND NOT (id <=> 1)",
+            "NOT (id = 1)",
+            "id != 1")
+          checkResidualPredicates(table, predicates)
+          // `id != 1` reaches Iceberg as a not-equal, and `b = false` as `b 
IS NOT NULL AND NOT b`,
+          // whose null check covers the negated equality, so both are still 
pushed. A negated
+          // `<=>` has no such check and stays out.
+          // Planning commonData leaks manifest streams on Iceberg versions 
before 1.8.0.
+          if (icebergVersionAtLeast(1, 8)) {
+            for ((predicate, pushed) <- Seq(
+                "b = false" -> true,
+                "id != 1" -> true,
+                "id IS NOT NULL AND NOT (id <=> 1)" -> true,
+                "NOT (id <=> 1)" -> false,
+                "b IS NOT TRUE" -> false)) {
+              val (_, cometPlan) = checkSparkAnswer(s"SELECT id FROM $table 
WHERE $predicate")
+              val scans = collectIcebergNativeScans(cometPlan)
+              assert(scans.length == 1, s"$cometPlan")
+              val common = 
OperatorOuterClass.IcebergScanCommon.parseFrom(scans.head.commonData)
+              assert(common.getResidualPoolCount == (if (pushed) 1 else 0), 
s"$predicate")
+            }
+          }
+        } finally {
+          spark.sql(s"DROP TABLE $table")
+        }
+      }
+    }
+  }
+
+  test("residuals on tables with a name mapping keep rows from imported 
files") {
+    assume(icebergAvailable, "Iceberg not available in classpath")
+
+    withHadoopCatalog("test_cat") {
+      withSQLConf(
+        CometConf.COMET_ENABLED.key -> "true",
+        CometConf.COMET_EXEC_ENABLED.key -> "true",
+        CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true") {
+        val table = "test_cat.db.imported_residual_test"
+        spark.sql(s"CREATE TABLE $table (id BIGINT, t INT, d DOUBLE) USING 
iceberg")
+        try {
+          withTempDir { dir =>
+            // The file stores id as INT, t as TINYINT and d as FLOAT. 
Appending it the way
+            // add_files does adds a name mapping but records no schema change.
+            val source = new File(dir, "source").getAbsolutePath
+            withSQLConf(CometConf.COMET_ENABLED.key -> "false") {
+              spark
+                .sql("SELECT CAST(id AS INT) AS id, CAST(id AS TINYINT) AS t, 
" +
+                  "CAST(id AS FLOAT) AS d FROM range(1, 4)")
+                .coalesce(1)
+                .write
+                .parquet(source)
+            }
+            val icebergTable = loadIcebergTable(spark, "test_cat", "db", 
"imported_residual_test")
+              .asInstanceOf[org.apache.iceberg.Table]
+            icebergTable
+              .updateProperties()
+              .set(
+                org.apache.iceberg.TableProperties.DEFAULT_NAME_MAPPING,
+                org.apache.iceberg.mapping.NameMappingParser
+                  
.toJson(org.apache.iceberg.mapping.MappingUtil.create(icebergTable.schema())))
+              .commit()
+            val file = new 
File(source).listFiles().find(_.getName.endsWith(".parquet")).get
+            icebergTable
+              .newAppend()
+              .appendFile(
+                org.apache.iceberg.DataFiles
+                  .builder(icebergTable.spec())
+                  .withPath(file.getAbsolutePath)
+                  .withFileSizeInBytes(file.length())
+                  .withRecordCount(3L)
+                  .withFormat(org.apache.iceberg.FileFormat.PARQUET)
+                  .build())
+              .commit()
+            spark.sql(s"REFRESH TABLE $table")
+
+            // iceberg-rust skips the imported file's pages for these, or 
casts 200 and
+            // 3000000000 to a NULL TINYINT and INT.
+            val predicates = Seq(
+              "id > 0",
+              "id < 3000000000",
+              "id IN (1, 2)",
+              "t < 200",
+              "t IN (1, 200)",
+              "d > 0.5D",

Review Comment:
   `d > 0.5D` never reaches the native scan because of the `FLOAT`/`DOUBLE` 
range rule, whether or not the table has a name mapping, so it passes with this 
guard disabled. Is it there for when #7588 restores range pruning on `DOUBLE`? 
If so, a short comment would say so. Otherwise `d IN (1.0D, 2.0D)` is the case 
that exercises `d`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to