voonhous commented on code in PR #19236:
URL: https://github.com/apache/hudi/pull/19236#discussion_r3557526947
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -446,12 +467,43 @@ protected void updateRecordMetadata(InternalRow row,
@AllArgsConstructor(staticName = "of")
private static class SparkArrowWriter implements ArrowWriter<InternalRow> {
private final LanceArrowWriter lanceArrowWriter;
+ // Parallel arrays of top-level BLOB column ordinals and names for the
per-row guard.
+ private final int[] blobFieldOrdinals;
+ private final String[] blobFieldNames;
@Override
public void write(InternalRow row) {
+ validateBlobRow(row);
lanceArrowWriter.write(row);
}
+ /**
+ * Reject descriptor-shaped BLOB rows before they are persisted. A row is
descriptor-shaped when
+ * a top-level BLOB struct is non-null, its type is INLINE, its data is
null, and its reference
+ * is non-null; this only arises when a DESCRIPTOR-mode read leaks onto a
write path, and writing
+ * it would silently drop the inline bytes. The legitimate empty-inline
shape {INLINE, null, null}
+ * stays writable.
+ */
+ private void validateBlobRow(InternalRow row) {
+ for (int i = 0; i < blobFieldOrdinals.length; i++) {
+ int ordinal = blobFieldOrdinals[i];
+ if (row.isNullAt(ordinal)) {
+ continue;
+ }
+ InternalRow blob = row.getStruct(ordinal, 3);
+ if (blob.isNullAt(0) ||
!INLINE_TYPE_TOKEN.equals(blob.getUTF8String(0))) {
+ continue;
+ }
+ if (blob.isNullAt(1) && !blob.isNullAt(2)) {
+ throw new HoodieException(
+ "BLOB column '" + blobFieldNames[i] + "' has an INLINE row with
null data but a "
+ + "populated reference: a DESCRIPTOR-mode read leaked into
the write path, and "
+ + "persisting it would silently drop the blob bytes.
Internal rewrites "
+ + "(compaction, clustering, merge) must read with
hoodie.read.blob.inline.mode=CONTENT.");
Review Comment:
Widened the parenthetical as suggested, and called the INSERT INTO ...
SELECT hard failure out under Impact in the description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
