fabientra opened a new issue, #13570:
URL: https://github.com/apache/gluten/issues/13570

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   With the native Iceberg scan (`IcebergScanTransformer`), a simple count over 
one partition of an Iceberg table returns fewer rows than vanilla Spark in most 
runs. The number of missing rows changes from run to run (0 to 543 here). No 
task fails, no task is retried, and the logs have no error or warning.
   
   In the same engine, with the same query:
   - `spark.gluten.sql.columnar.batchscan=false` (vanilla Iceberg reader, rest 
of the plan in Gluten) is exact every time.
   - The same 15 data files, read as plain Parquet with the native 
`FileSourceScanExecTransformer` (``parquet.`<dir>/{file1,...,file15}` ``), are 
exact every time. The result is identical to vanilla Spark on the same path, 5 
runs out of 5.
   
   So the rows seem to be lost in the Iceberg split path (Gluten's Iceberg 
split info or Velox's Iceberg split reader). Parquet decoding, S3 reads, 
filters and aggregation return correct results on the plain Parquet path.
   
   We found it in production-like tests: an Iceberg overwrite of about 8.2 
billion rows committed 49 to 398 fewer rows than vanilla Spark in 4 runs out of 
5, with all tasks successful.
   
   **Query** (table and value names changed):
   
   ```sql
   SELECT count(*) AS n, count(DISTINCT id) AS ids, bit_xor(xxhash64(id)) AS 
id_hash
   FROM db.events
   WHERE name IN (<27 event names>)
     AND server_created_at >= CAST('2026-11-04' AS DATE) AND server_created_at 
< CAST('2026-11-05' AS DATE)
     AND app_id = '<one app>' AND platform = 'android'
   ```
   
   Expected (vanilla Spark, and Gluten with `batchscan=false`): `n = 
39,125,071`.
   
   **Table**
   - Iceberg 1.10.0, Glue catalog, `S3FileIO`, data on S3. Partition spec: 
`day(server_created_at)`, `bucket(app_id)`.
   - No delete files in the partition (position or equality).
   - The Velox Iceberg doc marks partitioned tables as "Fallback mostly". Here 
the scan of a table partitioned by `day` and `bucket` is offloaded 
(`IcebergScanTransformer`), and it returns wrong results without a fallback or 
an error.
   - The matching rows are in 15 data files of one partition (the partition has 
83 data files, 292 M rows, 19.4 GB).
   - In our first analysis, all lost rows came from one file: 3,668,268 rows, 
520 MB, 5 row groups, `split_offsets` = `[4, 132840796, 259353067, 386848326, 
513486252]`. With the default 128 MB split size, Iceberg plans this file as 
several file scan tasks. But a split size of 1 GB, so that no file is cut into 
several tasks, does not stop the loss (see below).
   
   **Results** (same query, executors with 3 cores, 5g heap, 10g off-heap)
   
   | Setup | Runs | Missing rows per run |
   |---|---|---|
   | 1.8.0-SNAPSHOT nightly (2026-10-27), native Iceberg scan | 11 | −76, −211, 
−18, −433, −32, −38, −1, 0, −80, −64, −38 |
   | same, `prefetchRowGroups=0` | 5 | −66, −64, 0, −543, −85 |
   | same, `maxCoalescedDistance=0B` | 5 | 0, −122, −9, −57, −320 |
   | same, `loadQuantum=8MB` | 5 | −62, −270, −203, −112, 0 |
   | same, `fileHandleCacheEnabled=false` | 5 | −61, −152, −402, −9, −245 |
   | same, `IOThreads=0` and `SplitPreloadPerDriver=0` (the 8.2 B row job) | 3 
| −398, −146, −49 |
   | same, read option `split-size` = 1 GB (no file is cut: 4 tasks instead of 
39) | 5 | −420, −416, −63, −423, −142 (control in the same application, default 
split size: −104, 0, −324, −70, −195) |
   | 1.7.0-SNAPSHOT nightly (2026-06-18), Graviton3 nodes only | 5 | 0, −212, 
−323, −38, −18 |
   | 1.8.0-SNAPSHOT, `batchscan=false` | 3 | 0, 0, 0 |
   | 1.8.0-SNAPSHOT, the same 15 files as plain Parquet, native scan | 5 | 0, 
0, 0, 0, 0 (identical to vanilla Spark) |
   
   (The Velox settings are static configs, so each of them ran in its own Spark 
application.)
   
   **Other observations**
   - The lost rows come in fixed groups: two runs on two different applications 
lost the same 9 rows (same count, same `id_hash`). A 1.7.0-SNAPSHOT run and a 
1.8.0-SNAPSHOT run lost the same 38 rows. Only which groups go missing changes 
between runs.
   - The lost rows are close together: most of them are in a narrow range of 
`user_id` values.
   - The issue is in 1.7.0-SNAPSHOT (June 2026) too, so it is not a recent 
regression. We did not find a related fix on `main` up to the 2026-11-06 
nightly.
   - On the plain Parquet path, the count differs from the Iceberg count by 
7,083 rows also with Gluten off. We did not investigate this difference: the 
comparison above is always Gluten against vanilla Spark on the same path.
   - Possibly related: #12545 (silent data loss when preload threads fail). But 
`IOThreads=0` does not fix our case.
   
   ### Gluten version
   
   main branch
   
   (Nightly jars from nightlies.apache.org/gluten/nightly-release-jdk17: 
`gluten-velox-bundle-spark3.5_2.12-linux_aarch64-1.8.0-SNAPSHOT.jar` built 
2026-10-27T02:24:53Z, sha256 
`f7b1b71a849a56662284e5f5670baaa75c0a1c4a64dbac20415e6e6908b1e166`, and 
`...-1.7.0-SNAPSHOT.jar` built 2026-06-18T01:03:52Z, sha256 
`05795d60495b901b3888561240440d528ffecee8dc57af4439fac8cd5b22008a`.)
   
   ### Spark version
   
   Spark-3.5.x
   
   ### Spark configurations
   
   ```
   spark.plugins=org.apache.gluten.GlutenPlugin
   spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
   spark.memory.offHeap.enabled=true
   spark.memory.offHeap.size=10g
   spark.executor.memory=5g
   spark.executor.memoryOverhead=6g
   spark.executor.cores=3
   spark.driver.defaultJavaOptions=-Dio.netty.tryReflectionSetAccessible=true
   spark.executor.defaultJavaOptions=-XX:+UseTransparentHugePages 
-Dio.netty.tryReflectionSetAccessible=true -XX:TrimNativeHeapInterval=5000
   
spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions
   spark.sql.catalog.<cat>=org.apache.iceberg.spark.SparkCatalog
   spark.sql.catalog.<cat>.type=glue
   spark.sql.catalog.<cat>.io-impl=org.apache.iceberg.aws.s3.S3FileIO
   spark.sql.shuffle.partitions=1000
   spark.dynamicAllocation.enabled=true
   spark.dynamicAllocation.minExecutors=80
   spark.dynamicAllocation.maxExecutors=200
   spark.decommission.enabled=true
   spark.storage.decommission.enabled=true
   ```
   
   ### System information
   
   - Apache Spark 3.5.6 on Kubernetes (EKS), Iceberg 1.10.0 
(`iceberg-spark-runtime-3.5_2.12`), Java 17 (Temurin 17.0.15), Ubuntu 22.04 
base image.
   - linux/aarch64: AWS Graviton executors (r6gd, r7gd, r8gd). The 
1.7.0-SNAPSHOT runs used r7gd only, because that arm64 jar needs SVE.
   - All runs are on linux/aarch64, with the official nightly jars (the 
releases have no aarch64 jar). We could not test x86. The plain Parquet read on 
the same aarch64 jar is exact.
   
   ### Relevant logs
   
   No error or warning. All tasks succeed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to