wwj6591812 commented on PR #10922:
URL: https://github.com/apache/paimon/pull/10922#issuecomment-6701515026

   @JingsongLi Yes, I have pushed an update and completed a local benchmark 
rerun. The additional instrumentation has material overhead in the 
small-payload hot path, so the new `blobTransfer` group is now **opt-in and 
disabled by default**. The revised enabled path has not demonstrated a clear 
speedup over the original PR. Details and comparison data follow.
   
   ### Purpose and metric semantics
   
   The existing `blobFetch` metrics report outcomes and successful payload-byte 
totals. The added group is intended to help diagnose application-level 
payload-copy cost and failures, including confirmed bytes from attempts that 
later fail:
   
   | Metric | Type and meaning |
   | --- | --- |
   | `blobTransferReadBytes` | Counter: payload bytes returned by source reads |
   | `blobTransferWrittenBytes` | Counter: payload bytes from destination 
writes that returned successfully |
   | `blobTransferDurationNanos` | Counter: cumulative elapsed nanoseconds for 
reported outcomes |
   | `blobTransferCompleted` / `blobTransferFailed` | Counters: completed 
payload copies / reported failed attempts |
   | `blobTransferLatencyMillis` | Histogram: elapsed integer milliseconds for 
reported completed and failed attempts |
   
   The metrics are scoped by `table` and a bounded reader-API `sourceKind` 
(`inline`, `http`, `file`, `custom`, `unknown`). They describe the instrumented 
raw/array/map BLOB append-copy paths and shared video-payload path, including 
Data Evolution append writes. Primary-key externalization, BLOB compaction and 
separate video-index reads are outside this instrumentation's coverage. They do 
not identify physical storage locations or IDC/network routes. No URI, 
hostname, or user-supplied path becomes a metric label.
   
   The byte counters exclude file framing/checksums, protocol overhead and 
unobserved underlying transport-retry traffic. Read bytes include bytes 
returned before a later failure; written bytes include write calls that 
returned successfully. Bytes written internally by a write call that 
subsequently throws cannot be fully accounted for. Elapsed time covers 
obtaining the BLOB value through copy completion/failure, including source 
opening, intervening prefix framing and applicable per-payload stream close. A 
completed copy is recorded before record-footer writing, callbacks, flushing, 
or snapshot commit. The histogram retains the latest 1024 observations per 
reporter/source-kind group; its quantiles are not globally mergeable by 
averaging.
   
   ### Implementation and changes in the update
   
   The configuration path is `CoreOptions -> BaseAppendFileStoreWrite -> 
BlobFetchMetrics -> AbstractBlobElementWriter`:
   
   1. Added the table option `blob-transfer-metrics.enabled`, default `false`. 
It controls only the additional group; existing `blobFetch` metrics remain 
active. The option/capability is cached when the writer is constructed, so 
changing it requires constructing a new writer.
   2. Added a default-disabled reporter capability. Existing/custom reporter 
implementations remain opted out unless they explicitly enable it.
   3. Split the enabled and disabled copy loops at their entry. With transfer 
metrics disabled, the writer skips the new timer, source classification, 
per-chunk transfer-byte accounting, transfer callback, and transfer-group 
registration. The disabled reporter returns before taking the transfer-update 
monitor. Existing copying, CRC, framing, I/O, reporter initialization and small 
per-payload enablement checks remain; this is not a zero-overhead claim.
   4. For enabled copies, accumulate read/written bytes in local variables and 
publish the totals once in `finally`, preserving accounting on partial 
failures. Skip zero counter increments and normalize the elapsed duration once.
   5. Retained exact event counting, bounded histogram updates, and 
synchronization with group creation/close. No sampling or event batching was 
introduced. Existing NULL fallback, exception propagation, and BLOB file-format 
behavior are preserved.
   
   To enable the diagnostic group for a newly created writer:
   
   ```sql
   -- Table option
   'blob-transfer-metrics.enabled' = 'true'
   ```
   
   The main implementation update is 
[cae3dc2](https://github.com/apache/paimon/pull/10922/commits/cae3dc2491f92111fb81a66bfe032abf68bfda1d).
 This is a structural optimization and an opt-in policy change; the 
measurements below do not establish an enabled-path performance improvement.
   
   ### Benchmark setup
   
   - Local JMH 1.35 experiment on a shared arm64/macOS development machine, JDK 
8u302; `-Xms512m -Xmx768m -XX:+UseParallelGC`.
   - Each invocation writes 64 BLOBs through the real writer to an in-memory 
destination, including payload copying, CRC, framing, footer, and close. 
Results are normalized per BLOB. Payload/source preparation and reporter setup 
are outside the timed operation.
   - Four warmup iterations of 1 s and five measurement iterations of 1 s, with 
three independent JVM forks per arm/configuration. Each fork contributes one 
mean; the sample size is **3 pairs, not 15 independent iterations**.
   - Cases were run serially with randomized case order and rotating arm order. 
All data below comes from the same November 6 rerun; no earlier-run samples 
were pooled.
   - `inline` is an in-memory source; `file` is warmed local-file I/O; 
`file_reused` reuses the referenced input stream; `http` uses a loopback HTTP 
server. These are not cold-disk or CDN/WAN measurements.
   - `F1/F4` means Flink metric objects/adapter with one/four workers; `P1` 
means Paimon's metric backend with one worker. All cases use the Paimon BLOB 
writer. Four-worker cases have independent writers/reporters, so they do not 
measure shared-reporter lock contention. No periodic production metrics 
exporter runs in the test.
   - The harness constructs the reporter directly to isolate writer cost. 
Public table-option propagation is covered separately by absent/false/true 
integration tests through real append `TableWrite` creation, write, commit, and 
readback.
   
   The four arms are:
   
   | Arm | Revision | Collection behavior |
   | --- | --- | --- |
   | Baseline | `b4342332eafd` | Existing `blobFetch` metrics; no transfer 
instrumentation |
   | Old PR On | `4643d3905cfd` | Original PR with transfer collection enabled |
   | Revised Off | `cae3dc2491f9` | Real default-disabled reporter; existing 
`blobFetch` metrics retained |
   | Revised On | `cae3dc2491f9` | Same revised binaries with transfer 
collection explicitly enabled |
   
   There are **186 valid real JVM forks**: 16 instrumented-source 
configurations x 3 arms x 3 repeats, 6 additional old-PR configurations x 3 
repeats, and 4 NOOP configurations x 2 revisions x 3 repeats. All passed the 
harness's payload/CRC/footer, class-origin/capability, and metric-semantic 
checks. NOOP aliases do not add samples.
   
   Absolute times below are arithmetic means of fork means. Percentage changes 
are geometric means of the three paired time ratios, with Student-t 95% 
intervals on log-ratios (`df=2`), then transformed back. Consequently, 
percentages need not equal division of the displayed arithmetic means. Positive 
changes mean slower. Intervals crossing zero do not demonstrate zero overhead 
or equivalence. The contrasts reuse observations and have no 
multiple-comparison correction; there is no pooled "overall improvement" claim.
   
   ### Four-arm comparison: all six remeasured 1 KiB old-PR cases
   
   Times are **ns/BLOB**, lower is better. The last column is percentage change 
**[95% CI]** of Revised On relative to Old PR On.
   
   | Source / size / backend-workers | Baseline | Old PR On | Revised Off | 
Revised On | On vs old PR, % [95% CI] |
   | --- | ---: | ---: | ---: | ---: | ---: |
   | inline / 1 KiB / F1 | 208.49 | 264.52 | 205.25 | 239.65 | -9.11 [-28.35, 
+15.30] |
   | file / 1 KiB / F1 | 19820.55 | 19920.54 | 20117.45 | 19345.05 | -2.90 
[-9.22, +3.86] |
   | file_reused / 1 KiB / F1 | 1292.09 | 1273.00 | 1296.44 | 1348.97 | +5.70 
[-9.94, +24.06] |
   | http / 1 KiB / F1 | 156044.35 | 178528.66 | 167391.16 | 167213.83 | -5.15 
[-19.14, +11.26] |
   | inline / 1 KiB / P1 | 208.40 | 246.36 | 205.01 | 251.32 | +1.64 [-44.82, 
+87.20] |
   | inline / 1 KiB / F4 | 208.37 | 476.80 | 222.18 | 483.60 | +1.41 [-8.36, 
+12.22] |
   
   All six revised-on/old-PR intervals include zero, so these data do not 
demonstrate an enabled-path speedup. The old PR was remeasured only for these 
six cases; other old-PR cells were not measured.
   
   <details>
   <summary><strong>Full 16-configuration matrix: baseline, revised Off, 
revised On</strong></summary>
   
   Times in this table are **microseconds/BLOB** (`1 us = 1000 ns`). Each 
change column is **percent [95% CI]**. Here `Off/Base`, `On/Off`, and `On/Base` 
compare the indicated numerator's elapsed time with the denominator.
   
   | Source / size / backend-workers | Baseline, us | Off, us | On, us | 
Off/Base, % [95% CI] | On/Off, % [95% CI] | On/Base, % [95% CI] |
   | --- | ---: | ---: | ---: | ---: | ---: | ---: |
   | inline / 1 KiB / F1 | 0.2085 | 0.2052 | 0.2396 | -1.54 [-12.31, +10.56] | 
+16.37 [-10.68, +51.60] | +14.58 [-14.12, +52.87] |
   | inline / 64 KiB / F1 | 12.093 | 12.301 | 12.992 | +1.62 [-1.39, +4.72] | 
+5.59 [-7.44, +20.44] | +7.30 [-4.89, +21.05] |
   | inline / 1 MiB / F1 | 185.684 | 183.379 | 181.759 | -1.21 [-6.72, +4.62] | 
-0.88 [-2.28, +0.54] | -2.08 [-8.20, +4.44] |
   | file / 1 KiB / F1 | 19.821 | 20.117 | 19.345 | +1.49 [-3.69, +6.95] | 
-3.85 [-15.31, +9.16] | -2.42 [-12.57, +8.91] |
   | file / 64 KiB / F1 | 48.589 | 45.845 | 50.584 | -5.41 [-21.22, +13.57] | 
+9.08 [-32.30, +75.77] | +3.18 [-46.65, +99.56] |
   | file / 1 MiB / F1 | 455.333 | 418.266 | 415.607 | -7.71 [-35.96, +32.99] | 
-0.63 [-6.21, +5.28] | -8.30 [-34.55, +28.47] |
   | file_reused / 1 KiB / F1 | 1.292 | 1.296 | 1.349 | +0.23 [-6.44, +7.37] | 
+3.98 [+0.91, +7.14] | +4.21 [-5.35, +14.74] |
   | file_reused / 64 KiB / F1 | 26.252 | 27.122 | 25.929 | +3.22 [-5.15, 
+12.34] | -4.26 [-15.23, +8.13] | -1.18 [-8.03, +6.19] |
   | file_reused / 1 MiB / F1 | 430.389 | 439.346 | 426.362 | +2.10 [+0.38, 
+3.84] | -2.97 [-11.42, +6.28] | -0.94 [-10.59, +9.76] |
   | http / 1 KiB / F1 | 156.044 | 167.391 | 167.214 | +6.91 [+0.58, +13.64] | 
-0.05 [-5.64, +5.87] | +6.85 [+2.53, +11.36] |
   | http / 64 KiB / F1 | 778.338 | 565.084 | 551.655 | -22.00 [-75.72, 
+150.60] | -2.39 [-20.88, +20.41] | -23.87 [-75.03, +132.10] |
   | http / 1 MiB / F1 | 5663.291 | 6017.630 | 5846.251 | +6.12 [-11.46, 
+27.18] | -2.77 [-22.46, +21.93] | +3.18 [-4.80, +11.82] |
   | inline / 1 KiB / P1 | 0.2084 | 0.2050 | 0.2513 | -1.60 [-8.78, +6.14] | 
+21.46 [-19.94, +84.28] | +19.51 [-15.24, +68.52] |
   | inline / 1 KiB / F4 | 0.2084 | 0.2222 | 0.4836 | +6.27 [-22.71, +46.12] | 
+118.36 [+77.77, +168.22] | +132.06 [+103.59, +164.51] |
   | inline / 64 KiB / P1 | 11.824 | 12.214 | 11.737 | +2.81 [-15.91, +25.69] | 
-3.40 [-24.09, +22.93] | -0.69 [-4.94, +3.75] |
   | inline / 64 KiB / F4 | 11.768 | 11.897 | 12.285 | +1.04 [-1.40, +3.54] | 
+3.40 [-34.21, +62.49] | +4.48 [-31.96, +60.42] |
   
   </details>
   
   ### NOOP controls
   
   These controls use an empty collector and construct no real reporter. **They 
are different from Revised Off**, which retains the existing metrics. Times are 
**microseconds/BLOB**; changes are **percent [95% CI]**, revised binaries 
relative to baseline binaries.
   
   | Source / size / backend-workers | Baseline NOOP, us | Revised NOOP, us | 
Change, % [95% CI] |
   | --- | ---: | ---: | ---: |
   | inline / 1 KiB / F1 | 0.2124 | 0.2110 | -0.61 [-9.72, +9.42] |
   | inline / 64 KiB / F1 | 11.295 | 11.525 | +2.00 [-6.64, +11.44] |
   | inline / 1 MiB / F1 | 180.332 | 204.102 | +13.10 [+3.52, +23.55] |
   | inline / 1 KiB / F4 | 0.2822 | 0.2210 | -19.72 [-57.09, +50.18] |
   
   ### Interpretation and current policy
   
   - The clearest enabled-group cost is the 1 KiB inline/F4 case: **222.18 -> 
483.60 ns/BLOB**, about **261 ns/BLOB** additional time; the paired increase is 
**118.36% [77.77%, 168.22%]**. Relative to the original baseline, the enabled 
revision is **132.06% [103.59%, 164.51%]** slower in this case.
   - The enabled-loop restructuring has not established an improvement over the 
original PR; all six measured new-on/old-on intervals cross zero.
   - In the default-off comparisons, the paired percentage intervals are above 
zero for `file_reused / 1 MiB / F1` (**+2.10% [0.38%, 3.84%]**) and `http / 1 
KiB / F1` (**+6.91% [0.58%, 13.64%]**). These are log-ratio intervals; the 
separately computed absolute-time-difference interval for the latter crosses 
zero. They should not be described as proof that every statistical formulation 
agrees.
   - The 1 MiB NOOP/F1 control also differs (**+13.10% [3.52%, 23.55%]**). 
These measurements do not isolate the cause of all residual differences: 
neither zero disabled-path overhead nor attribution of every difference 
exclusively to active telemetry is established.
   - Larger-payload and loopback-I/O intervals are often wide. A small point 
estimate or an interval crossing zero does not establish negligible overhead. 
These are exploratory local measurements on a shared developer machine, not 
production throughput/SLO evidence, a shared-lock-contention test, or a 
measurement of cluster checkpoints/exporter overhead.
   
   I therefore propose keeping the **whole additional group disabled by default 
and enabling it for diagnostics when needed**. This experiment toggles the 
group as a whole; it does not identify an individual counter or histogram that 
is safe to enable by default. That decision would need separate measurements 
and an agreed overhead budget.
   
   ### Validation and current CI
   
   - The targeted local BLOB/options/table-option integration reactor passed 
**171 tests**, with no failures, errors, or skips. Generated configuration 
documentation was updated; documentation verification passed **14 tests on each 
of JDK 8 and JDK 11**.
   - The follow-up commits after the measured `cae3dc2` revision only update 
generated documentation and a Spark 4 test-scoped embedded-Hive dependency; 
they do not change the measured production instrumentation code.
   - The current head is `fa9b4edc723e0b694a64a897d4a7b96dead1832e`. [CI run 
42178275426](https://github.com/apache/paimon/actions/runs/42178275426) is 
successful, with 17 jobs successful and Python skipped by selection, including 
the previously failing Spark 4/Scala 2.13 job and the final CI result.
   
   The code and documentation now make the opt-in behavior and payload-level 
scope explicit. I would welcome feedback on whether this opt-in diagnostic 
group is an acceptable direction for the PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to