aquwikimedia opened a new issue, #18927:
URL: https://github.com/apache/iceberg/issues/18927
### Apache Iceberg version
1.6.1
### Query engine
Spark
### Please describe the bug 🐞
### Summary
With **Spark 3.5.4 or later** and the **Iceberg 1.6.x** Spark 3.5 runtime,
Iceberg's vectorized Parquet reader returns **wrong values without any error**
for every row after the first batch (5,000 rows by default) of each data file.
Sometimes the executor crashes instead (SIGSEGV, "double free",
`IncompatibleClassChangeError: vtable stub`). Files on disk are correct; only
the read is wrong.
This was fixed on `main` by #11731 (`Bump Apache Spark to 3.5.4`) and
backported to `1.7.x` (released in 1.7.2). It was never backported to `1.6.x`.
### Cause
Spark 3.5.4 (SPARK-50235, SPARK-50463) changed the code generated for
`ColumnarToRow`: it calls `ColumnarBatch.closeIfFreeable()` after each batch.
`ColumnVector.closeIfFreeable()` calls `close()` by default; vectors that reuse
their memory must override it with a no-op, as Spark's own vectors do.
`IcebergArrowColumnVector` in 1.6.x reuses its Arrow vectors for every batch
of a file but does not override `closeIfFreeable()`. After the first batch,
Spark closes the vectors; Iceberg then writes the next batches into released
buffers. Iceberg sets `arrow.enable_unsafe_memory_access=true`, so there is no
bounds check and no error: Spark reads zeros or stale values (e.g. `DATE`
columns become `1970-01-01`), and the stray writes can corrupt other memory.
It only happens with whole-stage codegen on (the default), and only when
Iceberg uses the vectorized reader (all projected columns are primitive).
### Affected versions
| Iceberg | Spark 3.5.0 - 3.5.3 | Spark 3.5.4+ |
|---|---|---|
| ≤ 1.7.1 (`iceberg-spark-runtime-3.5_2.12`) | OK | **affected** |
| 1.7.2, ≥ 1.8.0 | OK | OK (fixed by #11731) |
### Reproduction
Spark 3.5.8 (any 3.5.4+), Java 8,
`iceberg-spark-runtime-3.5_2.12-1.6.1.jar`. One data file of 1,000 rows, read
with a batch size of 100:
```python
from pyspark.sql import SparkSession
spark = (
SparkSession.builder.master("local[1]")
.config("spark.jars", "iceberg-spark-runtime-3.5_2.12-1.6.1.jar")
.config("spark.sql.catalog.local",
"org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.local.type", "hadoop")
.config("spark.sql.catalog.local.warehouse", "/tmp/iceberg_repro")
.getOrCreate()
)
expected = spark.range(0, 1000).selectExpr(
"id", "date_add(date'2025-01-01', cast(id % 5 AS INT)) AS d", "id * 10
AS v", "concat('row_', id) AS s"
)
expected.coalesce(1).writeTo("local.db.t").tableProperty("write.distribution-mode",
"none").create()
actual = spark.read.option("vectorization-enabled",
"true").option("batch-size", "100").table("local.db.t")
print(actual.exceptAll(expected).count())
```
Expected: `0`. Actual: `900` on Linux (every row after the first batch); on
macOS the JVM crashed with SIGSEGV. With `spark.sql.codegen.wholeStage=false`,
`vectorization-enabled=false`, or Spark 3.5.3, the result is `0`.
We first hit it in production: a table with `years()` partitions (files >
5,000 rows) came back with wrong dates after a Spark 3.5.1 → 3.5.8 upgrade;
smaller `days()` partitions hid it.
### Why a backport to 1.6.x
1.6.1 is the **last Iceberg release that runs on Java 8**: 1.7.0 and later
need Java 11. Spark 3.5.x still supports Java 8, so Java 8 Spark 3.5 users have
no fixed release:
* upgrading Iceberg to 1.7.2 is not possible on Java 8;
* staying on Spark ≤ 3.5.3 means giving up the Spark 3.5.4 - 3.5.8 fixes,
including security fixes;
* turning off vectorized reads
(`spark.sql.iceberg.vectorization.enabled=false`) works, but costs about 25 -
30% on scans in our tests.
And the failure mode is silent wrong results, which users of 1.6.1 with a
recent Spark 3.5 may not have noticed.
The fix is the same 5-line override as in #11731, plus the `spark-hive35`
bump to 3.5.4 so that 1.6.x is built and tested against a Spark version that
has `closeIfFreeable()`. I have a PR ready against `1.6.x`.
I understand that 1.6.x is not actively maintained. Even if a 1.6.2 release
is not planned, having the fix on the `1.6.x` branch would help Java 8 users
who build from it, and this issue documents the incompatibility for people
searching for it.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]