voonhous commented on code in PR #19181:
URL: https://github.com/apache/hudi/pull/19181#discussion_r3549673096
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceReader.java:
##########
@@ -208,8 +208,17 @@ private ClosableIterator<UnsafeRow>
getUnsafeRowIterator(HoodieSchema requestedS
// Pinned to CONTENT: compaction/merge/log-replay need actual bytes to
rewrite.
// The user-facing `hoodie.read.blob.inline.mode` is honored by
SparkLanceReaderBase.
FileReadOptions readOpts =
FileReadOptions.builder().blobReadMode(BlobReadMode.CONTENT).build();
- ArrowReader arrowReader = lanceReader.readAll(columnNames, null,
DEFAULT_BATCH_SIZE, readOpts);
+ // BLOB reads must be chunked to dodge a lance-core FFI abort (see
LanceRecordIterator).
+ // Unwrap a nullable UNION wrapper before checking whether the
projection has a BLOB field.
+ HoodieSchema recordSchema = requestedSchema.getNonNullType();
+ boolean hasBlob = recordSchema.hasFields()
+ && recordSchema.getFields().stream().anyMatch(f ->
f.schema().isBlobField());
+ if (hasBlob) {
+ return LanceRecordIterator.chunkedBlobReader(allocator, lanceReader,
columnNames, readOpts,
Review Comment:
tried adding this and it surfaced a pre-existing bug: compaction nulls
inline blob bytes of untouched rows in the rewritten file group (repros on
master without this PR, even at 6 rows, once the base file holds untouched rows
-- our existing test masks it because untouched rows never sat in a compacted
file group). filed #19232 with the repro; the >512 compaction test belongs with
that fix. not blocking this PR on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
