wwj6591812 opened a new pull request, #11327:
URL: https://github.com/apache/paimon/pull/11327

   ### Purpose
   
   Existing Blob fetch counters report fetch outcomes and successful byte 
totals, but do not distinguish a received HTTP response from a failed body 
read, or a copied payload from a prepared file or published snapshot. Add 
opt-in observations for the native Data Evolution append path at those actual 
boundaries.
   
   - Report bounded source-kind, HTTP attempt/response, completed/incomplete 
payload, file close/prepare, logical commit and atomic publication counters 
through the existing metric registry and Flink bridge.
   - Keep the current master SnapshotCommit API and bound-lock semantics. 
Preserve unknown publication outcomes, identity-matched reconciliation and 
filtered replay without inventing a second publication.
   - Preserve existing blobFetch metrics, HTTP/write/commit operations, NULL 
fallback and exception behavior. No additional business GET, HEAD, commit or 
destination readback is introduced. Recoverable reporter failures do not 
replace processing outcomes.
   - Keep observation disabled by default with the table option 
`blob-ingest-observation.enabled`. Disabled writers retain their original copy 
loops; small scope checks remain. Enabled observations add counters/timing; no 
performance-improvement claim is made.
   - Document application-payload units, raw versus ARRAY/MAP operation 
boundaries, reset/aggregation rules and exporter limitations, including 
same-JVM Spark writer identity. These metrics do not establish physical network 
bytes, IDC/QoS, or business visibility.
   
   Related: #10922 adds the separate blobTransfer payload-copy metrics. This PR 
is independent: it does not include the blobTransfer option, transfer histogram 
or reporter changes, and does not require #10922 to merge first.
   
   ### Tests
   
   JDK 8. 278 distinct targeted local tests passed:
   
   | Scope | Tests |
   | --- | ---: |
   | Observation scope, HTTP client, source classification and Blob format 
writer | 79 |
   | Core options, native ingest, lost-publication receipt/retry/replay, and 
existing commit regressions | 145 |
   | Existing Blob and video file-format regressions | 53 |
   | Real Flink metric-registry bridge | 1 |
   
   Tests use loopback HTTP and local files, with actual native 
write/prepare/commit/readback and the bound publication lock. They cover 
partial reads/writes, received responses followed by body failures, default-off 
and nested-disabled scopes, telemetry exceptions, replay, and resource cleanup. 
This is local verification, not production or remote-storage acceptance.
   
   Scoped Spotless, Checkstyle and `git diff --check` passed. The option row 
was generated with the current configuration documentation generator. The full 
documentation site was not built.
   
   The targeted core/Flink Maven reactor completed successfully with the 
`flink1` profile, including the normal Flink bridge test. `package` is used so 
the existing codegen loader reaches its normal resource-packaging phase. No 
fast-build profile, dependency-version change, POM workaround or stub artifact 
was used.
   
   ```shell
   mvn -B -pl paimon-api,paimon-common,paimon-format -am test \
     
-Dtest=BlobIngestObservationsTest,HttpClientUtilsTest,BlobIngestSourceTest,BlobFormatWriterTest
 \
     -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false 
-Dmaven.test.failure.ignore=false
   mvn -B -Pflink1 -pl paimon-core,paimon-flink/paimon-flink-common -am package 
\
     
-Dtest=CoreOptionsTest,BlobIngestNativeWorkloadTest,BlobIngestCommitTest,FileStoreCommitTest,BlobIngestFlinkMetricsTest
 \
     -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false 
-Dmaven.test.failure.ignore=false
   mvn -B -pl paimon-format -am test \
     -Dtest=VideoFileFormatTest,BlobFileFormatTest \
     -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false 
-Dmaven.test.failure.ignore=false
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to