fabientra commented on issue #13570:
URL: https://github.com/apache/gluten/issues/13570#issuecomment-6849667798
**Root cause: the delete bitmap is left uninitialised when it grows**
`IcebergSplitReader::next` clears the delete bitmap, then calls
`dwio::common::ensureCapacity<int8_t>(deleteBitmap_, numBytes, pool,
/*preserveOldData=*/false, /*clearBits=*/true)`:
```cpp
if (deleteBitmap_) {
std::memset(deleteBitmap_->asMutable<int8_t>(), 0L,
deleteBitmap_->size()); // clears the OLD buffer
}
...
dwio::common::ensureCapacity<int8_t>(deleteBitmap_, numBytes, pool, false,
true);
```
When the batch is larger than any earlier batch of the split,
`ensureCapacity` (velox/dwio/common/BufferUtil.h) allocates a new buffer, does
not copy the old one (`preserveOldData=false`), and clears only the bytes
beyond the old size:
```cpp
oldSize = data->size();
if (!data->isMutable() || data->capacity() < newCapacity) {
auto newData = AlignedBuffer::allocate<T>(numElements, pool); //
contents not initialised in release builds
if (preserveOldData) { memcpy(...); }
data = newData;
}
...
if (clearBits && newCapacity > oldSize) {
std::memset(data->asMutable<int8_t>() + oldSize, 0L, newCapacity -
oldSize); // bytes [0, oldSize) keep whatever the allocator returned
}
```
So in the batch where the bitmap grows, its first `oldSize` bytes hold stale
bits, `mutation.deletedRows` carries them to the reader, and every stale bit
drops a row. No error, random between runs (it depends on what the allocator
recycles), and the same rows can disappear again when the recycled memory is
the same. The same code is on `facebookincubator/velox` `main`.
`DeletionVectorReader` writes into the same bitmap, so V3 deletion vectors are
exposed as well.
**Why the batch grows within a split.** `TableScan` sizes the first batch
from the Parquet row-size estimate, which
`ParquetRowReader::estimatedRowSize()` computes over the *whole file schema*
(`schemaWithId()`), not the projected columns. For our table (48 leaf columns,
wide strings) that estimate is ~4,100 B/row in 4 of the 5 row groups, so the
first batch is 10 MB / 4,100 ≈ 2,500 rows. After that batch,
`maxFilteringRatio_` (clamped to 1/4) raises the batch to `min(maxBatchSize,
4x)` = 4,096 rows: the bitmap grows from ~315 to 512 bytes, and ~2,500 rows of
that batch read stale bits. Row group 0 estimates at 223 B/row, so splits that
start there get a 4,096-row first batch and never grow, which is why only some
splits lose rows.
**Test: same engine, same query, same snapshot with position deletes,
settings changed with `SET` between arms** (expected 39,125,071 rows):
| Arm | Spark setting | Effect on the batch | Rows lost per run |
|---|---|---|---|
| A | defaults (`maxBatchSize=4096`, `preferredBatchBytes=10MB`) | ~2,500
then 4,096: the bitmap grows | −1, −1, 0, −540 |
| B |
`spark.gluten.sql.columnar.backend.velox.preferredBatchBytes=1073741824` |
first batch already 4,096: never grows | **0, 0, 0, 0, 0, 0** |
| C | `preferredBatchBytes=102400` | ~25 then ~100: grows, but only ~4 stale
bytes | 0, 0, 0, 0 |
| D | `spark.gluten.sql.columnar.maxBatchSize=1024` | 1,024 constant: never
grows | **0, 0, 0, 0** |
| E | `maxBatchSize=16384` | ~2,500 then ~10,000: grows | −170, −38, 0, −407
|
| F | defaults again | grows | −85, −230 |
The arms where the bitmap never grows are exact; the arms where it grows
lose rows. (Arm C grows by so little that almost no rows are exposed.) Earlier
results fit too: the plain Parquet read of the same files has no delete bitmap,
and the snapshot without delete files never allocates one.
**Workaround for users:** make the first scan batch the maximum one, e.g.
`spark.gluten.sql.columnar.backend.velox.preferredBatchBytes=1073741824`. It
leaves a residual risk when a split's first row group has fewer rows than
`maxBatchSize` and a later row group has more.
**Proposed fix** (velox/dwio/common/BufferUtil.h): when a new buffer is
allocated without preserving the old data, clear all of it:
```diff
if (preserveOldData) {
std::memcpy(
newData->template asMutable<uint8_t>(),
data->as<uint8_t>(),
oldSize);
+ } else {
+ // The new buffer holds no old data: with 'clearBits', clear all of
+ // it, not only the bytes beyond the old size.
+ oldSize = 0;
}
data = newData;
```
Alternatively `IcebergSplitReader::next` can clear `numBytes` after
`ensureCapacity` instead of `deleteBitmap_->size()` before it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]