wwj6591812 commented on code in PR #9301:
URL: https://github.com/apache/paimon/pull/9301#discussion_r3818940095
##########
paimon-format/src/main/java/org/apache/paimon/format/blob/AbstractBlobElementWriter.java:
##########
@@ -181,55 +188,202 @@ private SeekableInputStream openStream(Blob blob,
StreamOpener opener) throws IO
}
}
- protected final BlobDescriptor writeBlobData(BlobCopySource source) throws
IOException {
- long blobPosition = out.getPos();
- if (source.reused()) {
+ /**
+ * Fully stages {@code source} when fetch failures may be converted to
NULL. The returned source
+ * contains only bytes which were fetched successfully, so copying it can
never expose a partial
+ * source payload to the final BLOB output.
+ */
+ protected final @Nullable BlobCopySource
prepareBlobForWrite(BlobCopySource source)
+ throws IOException {
+ if (!writeNullOnFetchFailure) {
+ return source;
+ }
+
+ final BlobStaging staging;
+ try {
+ staging = stagingFactory.create();
Review Comment:
Thanks, fixed. Exact `BlobData` now bypasses staging, including payloads
above the 1 MiB threshold; subclasses remain staged because they may override
stream behavior. Descriptor/stream staging now uses the writer task's
`IOManager` temp directory when available, falling back to the process temp
directory only when no engine directory exists. I added >1 MiB inline-bypass
coverage plus a core-level test that observes a real spill through
`AppendOnlyWriter -> Dedicated -> Multiple -> BlobFileFormat` and verifies
cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
