prashantwason opened a new issue, #20308:
URL: https://github.com/apache/hudi/issues/20308
### Bug Description
**What happened:**
Record-level index lookups fail with a backward-seek `IllegalStateException`
when the metadata table's record index HFiles are larger than 2 GB:
```
java.lang.IllegalStateException: The current lookup key is less than the
current position of the cursor, i.e., backward seekTo, which is not supported
and should be avoided. key=... cursor=HFilePosition{offset=-2002322330, ...}
at
org.apache.hudi.io.hfile.HFileReaderImpl.seekTo(HFileReaderImpl.java:196)
at
org.apache.hudi.io.storage.HoodieNativeAvroHFileReader$RecordByKeyIterator.hasNext(HoodieNativeAvroHFileReader.java:458)
...
at
org.apache.hudi.metadata.HoodieBackedTableMetadata.readRecordIndexLocationsWithKeys(HoodieBackedTableMetadata.java:354)
at
org.apache.hudi.index.SparkMetadataTableGlobalRecordLevelIndex$RecordIndexFileGroupLookupFunction.call(SparkMetadataTableGlobalRecordLevelIndex.java:242)
```
The cursor offset in the message is negative. The real offset is
`-2002322330 + 2^32 = 2,292,644,966`, inside a 2,367,011,987-byte record index
HFile.
**Root cause:**
The native HFile reader stores the cursor's position in the file as an `int`:
- `HFileCursor.offset` is an `int`.
- `HFileReaderImpl` casts `BlockIndexEntry.getOffset()` (a `long`) to `int`.
- `HFileDataBlock.seekTo/next` take an `int blockStartOffsetInFile`.
Past 2 GB the cursor offset wraps to a negative value. Most code paths still
work, because the cursor offset and the block start are both truncated the same
way and only their difference is used. One path breaks:
1. HFiles written by HBase store shortened ("fake") first keys in the block
index. Each one sorts below its block's real first key.
2. A missing key in `[fake first key, actual first key)` returns
`SEEK_TO_BEFORE_BLOCK_FIRST_KEY` and leaves the cursor on the block's actual
first key, ahead of the lookup key. This is valid (HUDI-9325 / #13179).
3. If the next sorted lookup key falls in the same range, `seekTo` checks
`isAtFirstKeyOfBlock()`, which compares `cursor.getOffset() ==
indexEntry.getOffset() + HFILEBLOCK_HEADER_SIZE`. The `int` cursor offset is
negative and the `long` block offset is not, so they never match. The lookup is
rejected as a backward seek.
Separately, `HFileReaderImpl.seekTo()` positions the cursor at
`dataBlockIndexEntryMap.firstKey().getOffset() + HFILEBLOCK_HEADER_SIZE`. That
uses the offset of the first key inside its own byte array, not the first data
block's offset in the file. It only works because the first data block always
starts at offset 0.
**What you expected:**
Lookups on record index HFiles larger than 2 GB return the same results as
on smaller files.
**Steps to reproduce:**
1. Build a record index whose file group HFiles exceed 2 GB (about 2.36 GB
per file group in our case), written with shortened block index keys.
2. Upsert into the data table so the record index is looked up with keys
that do not exist in the index (new inserts).
3. When two consecutive sorted lookup keys fall between a block's fake first
key and its actual first key, and the block lies beyond 2 GB in the file, the
lookup fails with the exception above. Retries hit the same keys and fail the
same way.
We reproduced this against the failing 2.37 GB HFile, using the native
reader directly. For each data block we looked up two missing keys inside
`[fake first key, actual first key)`, then the actual first key:
- Blocks below 2 GB: 0 of 10,346 failed.
- Blocks at or beyond 2 GB: 1,056 of 1,056 failed.
- With the cursor offset changed to `long`: 0 failures in both groups.
### Environment
**Hudi version:** 1.2.x. The same code is on current master.
**Query engine:** Spark
**Relevant configs:** Metadata table with record-level index enabled
(`hoodie.metadata.record.index.enable=true`), large record index file groups.
### Logs and Stack Trace
See the stack trace above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
