aquwikimedia opened a new issue, #18927:
URL: https://github.com/apache/iceberg/issues/18927

   ### Apache Iceberg version
   
   1.6.1
   
   ### Query engine
   
   Spark
   
   ### Please describe the bug 🐞
   
   ### Summary
   
   With **Spark 3.5.4 or later** and the **Iceberg 1.6.x** Spark 3.5 runtime, 
Iceberg's vectorized Parquet reader returns **wrong values without any error** 
for every row after the first batch (5,000 rows by default) of each data file. 
Sometimes the executor crashes instead (SIGSEGV, "double free", 
`IncompatibleClassChangeError: vtable stub`). Files on disk are correct; only 
the read is wrong.
   
   This was fixed on `main` by #11731 (`Bump Apache Spark to 3.5.4`) and 
backported to `1.7.x` (released in 1.7.2). It was never backported to `1.6.x`.
   
   ### Cause
   
   Spark 3.5.4 (SPARK-50235, SPARK-50463) changed the code generated for 
`ColumnarToRow`: it calls `ColumnarBatch.closeIfFreeable()` after each batch. 
`ColumnVector.closeIfFreeable()` calls `close()` by default; vectors that reuse 
their memory must override it with a no-op, as Spark's own vectors do.
   
   `IcebergArrowColumnVector` in 1.6.x reuses its Arrow vectors for every batch 
of a file but does not override `closeIfFreeable()`. After the first batch, 
Spark closes the vectors; Iceberg then writes the next batches into released 
buffers. Iceberg sets `arrow.enable_unsafe_memory_access=true`, so there is no 
bounds check and no error: Spark reads zeros or stale values (e.g. `DATE` 
columns become `1970-01-01`), and the stray writes can corrupt other memory.
   
   It only happens with whole-stage codegen on (the default), and only when 
Iceberg uses the vectorized reader (all projected columns are primitive).
   
   ### Affected versions
   
   | Iceberg | Spark 3.5.0 - 3.5.3 | Spark 3.5.4+ |
   |---|---|---|
   | ≤ 1.7.1 (`iceberg-spark-runtime-3.5_2.12`) | OK | **affected** |
   | 1.7.2, ≥ 1.8.0 | OK | OK (fixed by #11731) |
   
   ### Reproduction
   
   Spark 3.5.8 (any 3.5.4+), Java 8, 
`iceberg-spark-runtime-3.5_2.12-1.6.1.jar`. One data file of 1,000 rows, read 
with a batch size of 100:
   
   ```python
   from pyspark.sql import SparkSession
   
   spark = (
       SparkSession.builder.master("local[1]")
       .config("spark.jars", "iceberg-spark-runtime-3.5_2.12-1.6.1.jar")
       .config("spark.sql.catalog.local", 
"org.apache.iceberg.spark.SparkCatalog")
       .config("spark.sql.catalog.local.type", "hadoop")
       .config("spark.sql.catalog.local.warehouse", "/tmp/iceberg_repro")
       .getOrCreate()
   )
   
   expected = spark.range(0, 1000).selectExpr(
       "id", "date_add(date'2025-01-01', cast(id % 5 AS INT)) AS d", "id * 10 
AS v", "concat('row_', id) AS s"
   )
   
expected.coalesce(1).writeTo("local.db.t").tableProperty("write.distribution-mode",
 "none").create()
   
   actual = spark.read.option("vectorization-enabled", 
"true").option("batch-size", "100").table("local.db.t")
   print(actual.exceptAll(expected).count())
   ```
   
   Expected: `0`. Actual: `900` on Linux (every row after the first batch); on 
macOS the JVM crashed with SIGSEGV. With `spark.sql.codegen.wholeStage=false`, 
`vectorization-enabled=false`, or Spark 3.5.3, the result is `0`.
   
   We first hit it in production: a table with `years()` partitions (files > 
5,000 rows) came back with wrong dates after a Spark 3.5.1 → 3.5.8 upgrade; 
smaller `days()` partitions hid it.
   
   ### Why a backport to 1.6.x
   
   1.6.1 is the **last Iceberg release that runs on Java 8**: 1.7.0 and later 
need Java 11. Spark 3.5.x still supports Java 8, so Java 8 Spark 3.5 users have 
no fixed release:
   
   * upgrading Iceberg to 1.7.2 is not possible on Java 8;
   * staying on Spark ≤ 3.5.3 means giving up the Spark 3.5.4 - 3.5.8 fixes, 
including security fixes;
   * turning off vectorized reads 
(`spark.sql.iceberg.vectorization.enabled=false`) works, but costs about 25 - 
30% on scans in our tests.
   
   And the failure mode is silent wrong results, which users of 1.6.1 with a 
recent Spark 3.5 may not have noticed.
   
   The fix is the same 5-line override as in #11731, plus the `spark-hive35` 
bump to 3.5.4 so that 1.6.x is built and tested against a Spark version that 
has `closeIfFreeable()`. I have a PR ready against `1.6.x`.
   
   I understand that 1.6.x is not actively maintained. Even if a 1.6.2 release 
is not planned, having the fix on the `1.6.x` branch would help Java 8 users 
who build from it, and this issue documents the incompatibility for people 
searching for it.
   
   
   ### Willingness to contribute
   
   - [x] I can contribute a fix for this bug independently
   - [ ] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to