JingsongLi commented on code in PR #10891:
URL: https://github.com/apache/paimon/pull/10891#discussion_r4430148706
##########
paimon-format/src/main/java/org/apache/paimon/format/blob/VideoFormatWriter.java:
##########
@@ -258,25 +280,61 @@ private void flushRun() {
/** Copies raw video bytes without the ordinary BLOB record header and
trailer. */
private static class RawVideoPayloadWriter extends
AbstractBlobElementWriter {
- private byte[] readIndex(Blob index, long expectedLength) throws
IOException {
+ private boolean readIndex(
+ Blob index, long expectedLength, RandomAccessFile spool, long
payloadLength)
+ throws IOException {
if (index == null) {
- return new byte[0];
+ return true;
}
- byte[] bytes;
+ long start = spool.getFilePointer();
+ boolean complete = false;
try {
- bytes = index.toData();
- if (bytes.length != expectedLength) {
- throw new java.io.EOFException(
- "Video keyframe index length changed while
reading.");
+ try (InputStream in = index.newInputStream()) {
+ byte[] buffer = new byte[64 * 1024];
+ long remaining = expectedLength;
+ while (remaining > 0) {
+ int n = in.read(buffer, 0, (int) Math.min(remaining,
buffer.length));
+ if (n <= 0) {
+ throw new java.io.EOFException(
+ "Video keyframe index length changed while
reading.");
+ }
+ spool.write(buffer, 0, n);
Review Comment:
**[P2] Propagate local index spool write failures**
`spool.write(...)` is inside the catch that calls `handleFetchFailure`. With
`blob-write-null-on-fetch-failure=true`, a local temporary-disk failure such as
ENOSPC is therefore treated as a source fetch failure: `readIndex` returns
`false`, and `addElement` appends `NULL_REFERENCE` even though the referenced
video and index are readable. The write can then complete successfully with the
video cell replaced by NULL.
I reproduced this with a valid index and a `RandomAccessFile` whose `write`
throws `IOException("No space left on device")` while `setLength` and `seek`
work normally: both `addElement` and `close` succeed with fallback enabled;
disabling fallback propagates the exception.
Please restrict the fetch-failure fallback to source stream operations and
let local spool write failures propagate, with a regression test for this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
