voonhous commented on code in PR #19913:
URL: https://github.com/apache/hudi/pull/19913#discussion_r3994801564
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/blob/BatchedBlobReader.scala:
##########
@@ -331,9 +339,16 @@ class BatchedBlobReader(
currentStartOffset = row.offset
currentEndOffset = row.offset + row.length
currentRows = ArrayBuffer(row)
+ } else if (row.offset == currentRows.last.offset && row.length ==
currentRows.last.length) {
+ // Same descriptor as the previous row: one blob referenced by more
than one row (join
+ // fan-out, duplicate records). It is served from the current read and
the range does
+ // not grow.
+ currentRows += row
} else {
val gap = row.offset - currentEndOffset
- // Check for overlap
+ // A blob is a distinct entity, so two blobs never share bytes. Rows
are sorted by
+ // (offset, length) and identical descriptors were handled above, so a
start inside the
+ // current range means two different blobs overlap, which indicates
corruption.
if (row.offset < currentEndOffset) {
Review Comment:
Right, that fixture was the bug on the Lance side. It is fixed in #19916
(merged as b3dfb2a125f6): blob_file_2.bin is now 1536 bytes and row 4 reads
[1024, 1536), so the two rows are disjoint on any partitioning. This PR keeps
the throw for real overlaps on purpose; it only stops rejecting a repeated
identical descriptor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
