wwj6591812 opened a new pull request, #11327:
URL: https://github.com/apache/paimon/pull/11327
### Purpose
Existing Blob fetch counters report fetch outcomes and successful byte
totals, but do not distinguish a received HTTP response from a failed body
read, or a copied payload from a prepared file or published snapshot. Add
opt-in observations for the native Data Evolution append path at those actual
boundaries.
- Report bounded source-kind, HTTP attempt/response, completed/incomplete
payload, file close/prepare, logical commit and atomic publication counters
through the existing metric registry and Flink bridge.
- Keep the current master SnapshotCommit API and bound-lock semantics.
Preserve unknown publication outcomes, identity-matched reconciliation and
filtered replay without inventing a second publication.
- Preserve existing blobFetch metrics, HTTP/write/commit operations, NULL
fallback and exception behavior. No additional business GET, HEAD, commit or
destination readback is introduced. Recoverable reporter failures do not
replace processing outcomes.
- Keep observation disabled by default with the table option
`blob-ingest-observation.enabled`. Disabled writers retain their original copy
loops; small scope checks remain. Enabled observations add counters/timing; no
performance-improvement claim is made.
- Document application-payload units, raw versus ARRAY/MAP operation
boundaries, reset/aggregation rules and exporter limitations, including
same-JVM Spark writer identity. These metrics do not establish physical network
bytes, IDC/QoS, or business visibility.
Related: #10922 adds the separate blobTransfer payload-copy metrics. This PR
is independent: it does not include the blobTransfer option, transfer histogram
or reporter changes, and does not require #10922 to merge first.
### Tests
JDK 8. 278 distinct targeted local tests passed:
| Scope | Tests |
| --- | ---: |
| Observation scope, HTTP client, source classification and Blob format
writer | 79 |
| Core options, native ingest, lost-publication receipt/retry/replay, and
existing commit regressions | 145 |
| Existing Blob and video file-format regressions | 53 |
| Real Flink metric-registry bridge | 1 |
Tests use loopback HTTP and local files, with actual native
write/prepare/commit/readback and the bound publication lock. They cover
partial reads/writes, received responses followed by body failures, default-off
and nested-disabled scopes, telemetry exceptions, replay, and resource cleanup.
This is local verification, not production or remote-storage acceptance.
Scoped Spotless, Checkstyle and `git diff --check` passed. The option row
was generated with the current configuration documentation generator. The full
documentation site was not built.
The targeted core/Flink Maven reactor completed successfully with the
`flink1` profile, including the normal Flink bridge test. `package` is used so
the existing codegen loader reaches its normal resource-packaging phase. No
fast-build profile, dependency-version change, POM workaround or stub artifact
was used.
```shell
mvn -B -pl paimon-api,paimon-common,paimon-format -am test \
-Dtest=BlobIngestObservationsTest,HttpClientUtilsTest,BlobIngestSourceTest,BlobFormatWriterTest
\
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
-Dmaven.test.failure.ignore=false
mvn -B -Pflink1 -pl paimon-core,paimon-flink/paimon-flink-common -am package
\
-Dtest=CoreOptionsTest,BlobIngestNativeWorkloadTest,BlobIngestCommitTest,FileStoreCommitTest,BlobIngestFlinkMetricsTest
\
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
-Dmaven.test.failure.ignore=false
mvn -B -pl paimon-format -am test \
-Dtest=VideoFileFormatTest,BlobFileFormatTest \
-DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false
-Dmaven.test.failure.ignore=false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]