voonhous opened a new issue, #19232:
URL: https://github.com/apache/hudi/issues/19232
**Describe the problem you faced**
On a MOR table with Lance base files and INLINE blobs, compaction rewrites
untouched rows with `inline_data=NULL`, silently losing the blob bytes. Rows
updated in the same compaction window keep their bytes (they come from the log
records); only rows carried over from the base file lose data.
Existing coverage masks this: in `testBlobInlineCompactionRoundTrip` the
untouched rows land in file groups without log files, which compaction never
rewrites, so no test ever depended on an untouched inline-blob base row
surviving a compaction rewrite.
**To Reproduce**
In `testBlobInlineCompactionRoundTrip` (`TestLanceDataSource`), force all
rows into one file group so untouched rows go through the compaction rewrite:
1. `coalesce(1)` on the input dataframe and
`hoodie.bulkinsert.shuffle.parallelism=1` (single base file)
2. run the test: after the upsert of ids 0..2 triggers inline compaction,
CONTENT-mode `read_blob()` on the untouched ids throws:
```
java.lang.IllegalStateException: read_blob() cannot materialize bytes for an
INLINE blob under DESCRIPTOR mode. Under
hoodie.read.blob.inline.mode=DESCRIPTOR, INLINE blobs are returned as
metadata-only (inline_data=NULL, synthesized reference). To read bytes, set
hoodie.read.blob.inline.mode=CONTENT
at
org.apache.spark.sql.hudi.blob.BatchedBlobReader$$anon$1.collectBatch(BatchedBlobReader.scala:234)
```
Fails at 6 rows; no batch-boundary or scale needed. Reproduces on current
master (9a8edd6e6495) -- predates and is unrelated to the chunked-read change
in #19181.
**Expected behavior**
Untouched INLINE blob rows keep their bytes across compaction.
**Suspected cause (unverified)**
The CONTENT pin lives in `HoodieSparkLanceReader`, but the file-group-reader
compaction path appears to read the base file through the datasource reader
(`SparkLanceReaderBase`), where `resolveBlobReadMode(storageConf)` falls back
to the DESCRIPTOR default because compaction sets no reader option -- the
compactor reads `inline_data=NULL` and rewrites it into the new base file. The
`widenBlobSubtreeNullability` handling that the internal path lacks is a second
suspect.
**Environment Description**
* Hudi version: master (1.3.0-SNAPSHOT)
* Table type: MOR, Lance base file format, INLINE blobs
* Spark version: 3.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
