vinothchandar opened a new pull request, #19181: URL: https://github.com/apache/hudi/pull/19181
### Describe the issue this Pull Request addresses lance-core 4.0.0 aborts the JVM in its Arrow C-stream export (arrow_array::ffi_stream::get_next: "range end index N out of range for slice of length 0") whenever a single readAll stream crosses Lance's internal BLOB page boundary (512 rows); the requested batchSize does not help because Lance re-chunks BLOB columns at 512 internally. As a result CoW Lance tables with BLOB columns could not be read past 512 rows: OUT_OF_LINE reads threw an Arrow "should have as many children as in the schema" error and INLINE CONTENT reads crashed the JVM (SIGABRT). Read BLOB-containing Lance files in <=512-row row-range chunks, issuing a fresh readAll per chunk so each FFI stream stays within a single BLOB page and the buggy second-page export is never reached. LanceRecordIterator gains a chunkedBlobReader(...) factory driven by an ArrowReaderSupplier; SparkLanceReaderBase (SQL/DataFrame read) and HoodieSparkLanceReader (internal CONTENT path) route to it when the projection contains a BLOB field. Non-BLOB reads keep the single streamed reader and are unchanged; the per-row hot path is untouched, with only small fixed per-chunk allocations. Also document hoodie.read.blob.inline.mode option placement (read option for both DataFrame and SQL) and add batch-scale regression tests: INLINE CONTENT, VECTOR + OUT_OF_LINE BLOB at n=100/1000 (crossing the 512-row boundary), with vector top-k, projection/filter, and read_blob byte + SHA-256 checks. ### Summary and Changelog lance reads work reliably without crashes for base files with larger number of records. ### Impact none ### Risk Level none ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
