wwj6591812 opened a new pull request, #9271: URL: https://github.com/apache/paimon/pull/9271
### Purpose Paimon BLOB descriptors can refer to HTTP resources. A batch job writing an HTTP-backed BLOB observed the following failure after the server had returned HTTP 200: ```text org.apache.paimon.shade.hc.core5.http.ConnectionClosedException: Premature end of Content-Length delimited message body (expected: 990434; received: 89075) ``` The response body was closed while it was being consumed, after 89,075 of 990,434 declared bytes had been returned. At that point the caller may already have copied the prefix into the BLOB output. Replaying from byte zero without verification could duplicate data, while swallowing the exception or changing the value to NULL would be unsafe after output has started. `HttpClientUtils#getAsInputStream` previously executed the GET and returned the raw entity stream. The existing request retry strategy covers request execution and retryable statuses, but the entity is consumed later. A `ConnectionClosedException`, `TruncatedChunkException`, or premature EOF before a known `Content-Length` can therefore occur after request execution has completed and after bytes have already been delivered. This PR wraps HTTP GET bodies in a bounded, integrity-checked resumable stream: - With a stable validator (a strong ETag or Last-Modified), it sends `Range: bytes=<position>-` with `If-Range`, accepts only HTTP 206, and validates the `Content-Range` start and total length, the response length when known, identity encoding, and any returned validator. - Without a validator, it performs a complete HTTP 200 replay from byte zero, hashes exactly the already-delivered prefix with SHA-256, and continues from that same response only after the prefix matches. A changed prefix or changed known length fails the read. - It follows bounded partial ranges and can recover from another truncation, with at most five body-recovery attempts. - Recovery requests use identity encoding with transparent decompression disabled so HTTP byte offsets stay exact. If an origin ignores the identity request and returns encoded content, the stream preserves the previous transparent decoding behavior but does not attempt byte-offset recovery for that representation. - Discarded responses are closed, terminal failures remain sticky, and failure diagnostics include the sanitized URI, delivered position, declared length, and recovery-attempt count without exposing URL query parameters. This does not change `blob-write-null-on-fetch-failure`, public APIs, table options, or the BLOB storage format. Recovery is limited to the two body-truncation exceptions above and premature EOF detectable from a known response length; unrelated body-read failures retain their existing behavior. Related work: #8412 handles configured NULL values for fetch/open failures, and #9181 reduces HTTP requests while writing BLOBs. This PR addresses the separate case where an accepted response body is truncated during lazy consumption. ### Tests ```text mvn -pl paimon-api -DwildcardSuites=none -Dtest=HttpClientUtilsTest test Tests run: 27, Failures: 0, Errors: 0, Skipped: 0 mvn -pl paimon-api test Tests run: 103, Failures: 0, Errors: 0, Skipped: 0 ``` Coverage includes validator-based Range/If-Range recovery, no-validator full replay and prefix verification, changed-prefix rejection, multiple truncations, bounded ranges, ignored/mismatched ranges, attempt exhaustion with sticky failure, signed-URI sanitization, gzip compatibility, restart-status failure, and close semantics. Checkstyle, Spotless, and `git diff --check` also pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
