voonhous commented on code in PR #19236:
URL: https://github.com/apache/hudi/pull/19236#discussion_r3557432868
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -446,12 +467,43 @@ protected void updateRecordMetadata(InternalRow row,
@AllArgsConstructor(staticName = "of")
private static class SparkArrowWriter implements ArrowWriter<InternalRow> {
private final LanceArrowWriter lanceArrowWriter;
+ // Parallel arrays of top-level BLOB column ordinals and names for the
per-row guard.
+ private final int[] blobFieldOrdinals;
+ private final String[] blobFieldNames;
@Override
public void write(InternalRow row) {
+ validateBlobRow(row);
lanceArrowWriter.write(row);
}
+ /**
+ * Reject descriptor-shaped BLOB rows before they are persisted. A row is
descriptor-shaped when
+ * a top-level BLOB struct is non-null, its type is INLINE, its data is
null, and its reference
+ * is non-null; this only arises when a DESCRIPTOR-mode read leaks onto a
write path, and writing
+ * it would silently drop the inline bytes. The legitimate empty-inline
shape {INLINE, null, null}
+ * stays writable.
+ */
+ private void validateBlobRow(InternalRow row) {
+ for (int i = 0; i < blobFieldOrdinals.length; i++) {
+ int ordinal = blobFieldOrdinals[i];
+ if (row.isNullAt(ordinal)) {
+ continue;
+ }
+ InternalRow blob = row.getStruct(ordinal, 3);
Review Comment:
Hoisted them into a package-private BlobStructLayout holder (ordinals,
arities sourced from HoodieSchema.Blob, and the precomputed type tokens); both
classes read the layout from there now. The literal 4 for the reference arity
is gone too.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestLanceDataSource.scala:
##########
@@ -1303,12 +1303,21 @@ class TestLanceDataSource extends
HoodieSparkClientTestBase {
/**
* Compaction must preserve INLINE blob bytes under the DESCRIPTOR default.
MOR compaction reads
- * the base file via {@link HoodieSparkLanceReader}, which hard-pins CONTENT
regardless of the
- * user-facing {@code hoodie.read.blob.inline.mode}. If that pin were to
honor the default
- * (DESCRIPTOR), compaction would read null {@code data} and rewrite a base
file without bytes,
- * silently corrupting untouched rows. This test inserts INLINE blobs,
upserts a subset to force
- * compaction, and asserts that touched rows carry the new bytes while
untouched rows retain the
- * originals.
+ * the base file through the internal write-side reader stack
+ * SparkReaderContextFactory -> SparkFileFormatInternalRowReaderContext ->
SparkLanceReaderBase,
+ * not through HoodieSparkLanceReader (that reader only serves LanceUtils
stats/key reads).
Review Comment:
Fixed all three spots: the deltaCommits message now points at
SparkReaderContextFactory, the docstring parenthetical lists the reader's
actual callers (stats/key reads, bloom-index lookups, legacy
HoodieWriteMergeHandle merges), and the pin comment in HoodieSparkLanceReader
credits those paths instead of compaction/merge/log-replay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
