wwj6591812 opened a new pull request, #9181: URL: https://github.com/apache/paimon/pull/9181
### Purpose [#8219](https://github.com/apache/paimon/pull/8219) introduced an HTTP existence preflight for `blob-write-null-on-missing-file`. Its final design performs the check in `FlinkRowWrapper`, because the writer-side fallback considered at that time could not safely turn an open failure into NULL after record bytes had been emitted. [#8412](https://github.com/apache/paimon/pull/8412) later changed the BLOB writer to open the source before writing record bytes and added safe open-time classification for HTTP 404 and other fetch failures. The Flink preflight remained, so a materialized HTTP descriptor currently performs: 1. `HEAD` 2. if HEAD is not 200, a range `GET` 3. the full `GET` required to copy the payload into a managed blob file Each operation has its own 429/503 retry loop. This causes redundant requests for successful resources and can amplify load while an HTTP origin is throttling. ### Changes - Compute which root BLOB fields are actually materialized into managed blob files. - For HTTP(S) descriptors of those fields, defer existence and open-failure handling directly to the writer's required full GET. - Preserve the preflight for inline `blob-descriptor-field` / `blob-view-field` values, because those fields have no later writer fetch. - Preserve the preflight for non-HTTP descriptors. - Match HTTP schemes case-insensitively, consistent with the URI reader and writer. The writer remains the single final classifier for materialized HTTP descriptors: - HTTP 404 becomes NULL only when `blob-write-null-on-missing-file=true`. - Other open/fetch failures become NULL only when `blob-write-null-on-fetch-failure=true`. - A body read failure after output has started still fails the write. - `HttpClientUtils.exists`, retry status codes, retry counts, timeout defaults, and the HTTP 416 zero-length-resource contract are unchanged. ### Request reduction With both write-null options enabled: | Final result | Before | After | | --- | --- | --- | | Successful HTTP resource | HEAD + full GET | full GET | | GET-only HTTP resource | HEAD + range GET + full GET | full GET | | HTTP 404 | HEAD + range GET | full GET | | Non-retryable HTTP error | up to 3 request chains | 1 request chain | | Persistent HTTP 429/503 | up to 3 retry chains | 1 retry chain | This reduces the persistent 429/503 upper bound from 18 attempts to 6 with the current default of five additional retries, without adding configuration or changing final row semantics. ### Implementation - `FlinkSinkBuilder` derives materialized BLOB positions from the table row type and excludes `CoreOptions.blobInlineField()`. - The positions are passed to `FlinkRowWrapper`. - `FlinkRowWrapper` skips only the redundant HTTP(S) preflight for those positions. - Existing overloads default to an empty materialized set, preserving their previous behavior. ### Tests Added unit and end-to-end coverage for: - materialized HTTP 200: one full GET and payload round-trip; - materialized HTTP 404: one full GET and NULL with the missing-file option; - materialized HTTP 400: one full GET and NULL with missing-file plus fetch-failure options; - HTTP 429 and 503 followed by 200: two full GET attempts, no HEAD or range request, and payload round-trip; - inline descriptor HTTP 404: HEAD plus range GET is retained and NULL is written; - uppercase HTTP scheme; - a non-BLOB column before a BLOB column, to verify field-position handling. Focused verification completed with 19 tests passing (13 wrapper tests and 6 HTTP BLOB integration cases), together with Checkstyle, Spotless, and `git diff --check`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
