wwj6591812 opened a new pull request, #9301:
URL: https://github.com/apache/paimon/pull/9301
### Purpose
`blob-write-null-on-fetch-failure=true` can safely turn a BLOB fetch/open
failure into NULL only while no bytes for that BLOB have reached the managed
BLOB output. An HTTP-backed BLOB write can also fail later, while the response
body is being consumed. One observed failure was:
```text
org.apache.paimon.shade.hc.core5.http.ConnectionClosedException:
Premature end of Content-Length delimited message body
(expected: 990434; received: 89075)
```
At that point a raw BLOB, ARRAY element, or MAP value may already have
appended a record header and a partial payload to the shared BLOB file.
Catching the exception and returning NULL directly would leave unindexed bytes
behind and could corrupt subsequent offsets, CRCs, and records.
`PositionOutputStream` also has no portable record-level rollback across local,
distributed, and object-store implementations.
This PR adds a per-BLOB-element staging boundary so the existing opt-in
option can safely cover terminal response-body fetch failures.
### Changes
When `blob-write-null-on-fetch-failure=true`:
- Read each individual BLOB payload into a private staging buffer before
appending any of its payload bytes to the final managed BLOB output.
- Keep up to 1 MiB in memory and spill larger payloads to a task-local
temporary file.
- Append the staged payload to the final output only after the source body
completes and its declared length is satisfied.
- If the source body ultimately fails while being read, close the source,
discard the stage, and encode that scalar/ARRAY/MAP element as NULL.
- Keep staging creation/write/read/close/delete failures, source cleanup
failures, final output failures, footer/CRC failures, and consumer failures
fatal. These are not remote fetch failures and must not be hidden as NULL.
- Preserve task cancellation: an interrupted task is never converted to a
successful NULL row.
- Discard a failed reusable source so the next descriptor reopens it instead
of continuing from an unknown offset.
When the option is false, the existing direct-streaming fast path and
failure behavior are unchanged.
### Relationship to #9271
This PR and #9271 address different layers and are intentionally independent:
| Case | #9271: HTTP body recovery | This PR: atomic NULL fallback |
| --- | --- | --- |
| A truncated body is recoverable | Resume/replay and return the complete
body | Commit the complete staged body, not NULL |
| Recovery is exhausted | Propagate the terminal read failure | If the
existing option is enabled, discard the stage and write NULL |
| The option is disabled | Recovery is still attempted | Terminal failure
still fails the write |
| This PR is used without #9271 | No transport-level recovery | The first
terminal body-read failure can safely become NULL when opted in |
#9271 recovers transport interruptions: it uses validated `Range + If-Range`
with a strong ETag, or a complete HTTP 200 replay with SHA-256 prefix
verification when no strong ETag is available. It deliberately does not change
NULL semantics.
This PR supplies the output atomicity needed for the separate terminal
policy. With both changes, Paimon first tries bounded body recovery; only after
recovery is exhausted does the existing opt-in setting write NULL. The PRs can
be reviewed and merged in either order.
### Scope and performance
- The new staging path is enabled only by
`blob-write-null-on-fetch-failure=true`; other writers keep the existing direct
path.
- The change applies to the append-only managed-BLOB writer path.
Primary-key BLOB externalization currently keeps both NULL-on-fetch options
disabled and is not expanded by this PR.
- Each active BLOB writer stages one element at a time. It uses at most
approximately 1 MiB of heap before spilling, then performs one local write/read
before the final output write.
- Normal success, handled fetch failure, abort, and close paths delete spill
files. A process hard kill can leave an unreferenced temporary file for
host-level cleanup.
- There is no public API or managed BLOB on-disk format change. Magic bytes,
length/index encoding, CRC handling, descriptors, and NULL encoding remain
unchanged.
### Tests
Format-layer unit tests cover scalar, ARRAY, and MAP layouts; good/fail/good
sequencing; known-length EOF and unknown-length read errors; option-disabled
behavior; reusable-source reopening; exact metrics; cancellation versus socket
timeout; in-memory and spilled staging; cleanup on success/failure/abort/close;
and fatal staging/final-output/consumer failures.
```text
mvn -pl paimon-format -am \
-DskipITs -DwildcardSuites=none \
-Dtest=BlobFormatWriterTest \
-Dsurefire.failIfNoSpecifiedTests=false test
Tests run: 57, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS
```
Flink integration tests use deterministic truncated HTTP responses and cover
scalar, ARRAY, and MAP NULL fallback, a following BLOB in the same writer,
disabled fallback with no committed row/snapshot, and separation from the 404
option.
```text
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS
```
A local synthetic merge with #9271 also passed:
- `HttpClientUtilsTest`: 32 tests
- HTTP recovery plus terminal-NULL Flink integration tests: 7 tests (2
recovery + 5 fallback)
- All failures/errors/skips: 0
The combined integration coverage verifies both successful recovery staying
non-NULL and recovery exhaustion falling back to NULL only when explicitly
enabled.
### API and format
No public API or storage-format change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
