wwj6591812 commented on PR #10922: URL: https://github.com/apache/paimon/pull/10922#issuecomment-6701515026
@JingsongLi Yes, I have pushed an update and completed a local benchmark rerun. The additional instrumentation has material overhead in the small-payload hot path, so the new `blobTransfer` group is now **opt-in and disabled by default**. The revised enabled path has not demonstrated a clear speedup over the original PR. Details and comparison data follow. ### Purpose and metric semantics The existing `blobFetch` metrics report outcomes and successful payload-byte totals. The added group is intended to help diagnose application-level payload-copy cost and failures, including confirmed bytes from attempts that later fail: | Metric | Type and meaning | | --- | --- | | `blobTransferReadBytes` | Counter: payload bytes returned by source reads | | `blobTransferWrittenBytes` | Counter: payload bytes from destination writes that returned successfully | | `blobTransferDurationNanos` | Counter: cumulative elapsed nanoseconds for reported outcomes | | `blobTransferCompleted` / `blobTransferFailed` | Counters: completed payload copies / reported failed attempts | | `blobTransferLatencyMillis` | Histogram: elapsed integer milliseconds for reported completed and failed attempts | The metrics are scoped by `table` and a bounded reader-API `sourceKind` (`inline`, `http`, `file`, `custom`, `unknown`). They describe the instrumented raw/array/map BLOB append-copy paths and shared video-payload path, including Data Evolution append writes. Primary-key externalization, BLOB compaction and separate video-index reads are outside this instrumentation's coverage. They do not identify physical storage locations or IDC/network routes. No URI, hostname, or user-supplied path becomes a metric label. The byte counters exclude file framing/checksums, protocol overhead and unobserved underlying transport-retry traffic. Read bytes include bytes returned before a later failure; written bytes include write calls that returned successfully. Bytes written internally by a write call that subsequently throws cannot be fully accounted for. Elapsed time covers obtaining the BLOB value through copy completion/failure, including source opening, intervening prefix framing and applicable per-payload stream close. A completed copy is recorded before record-footer writing, callbacks, flushing, or snapshot commit. The histogram retains the latest 1024 observations per reporter/source-kind group; its quantiles are not globally mergeable by averaging. ### Implementation and changes in the update The configuration path is `CoreOptions -> BaseAppendFileStoreWrite -> BlobFetchMetrics -> AbstractBlobElementWriter`: 1. Added the table option `blob-transfer-metrics.enabled`, default `false`. It controls only the additional group; existing `blobFetch` metrics remain active. The option/capability is cached when the writer is constructed, so changing it requires constructing a new writer. 2. Added a default-disabled reporter capability. Existing/custom reporter implementations remain opted out unless they explicitly enable it. 3. Split the enabled and disabled copy loops at their entry. With transfer metrics disabled, the writer skips the new timer, source classification, per-chunk transfer-byte accounting, transfer callback, and transfer-group registration. The disabled reporter returns before taking the transfer-update monitor. Existing copying, CRC, framing, I/O, reporter initialization and small per-payload enablement checks remain; this is not a zero-overhead claim. 4. For enabled copies, accumulate read/written bytes in local variables and publish the totals once in `finally`, preserving accounting on partial failures. Skip zero counter increments and normalize the elapsed duration once. 5. Retained exact event counting, bounded histogram updates, and synchronization with group creation/close. No sampling or event batching was introduced. Existing NULL fallback, exception propagation, and BLOB file-format behavior are preserved. To enable the diagnostic group for a newly created writer: ```sql -- Table option 'blob-transfer-metrics.enabled' = 'true' ``` The main implementation update is [cae3dc2](https://github.com/apache/paimon/pull/10922/commits/cae3dc2491f92111fb81a66bfe032abf68bfda1d). This is a structural optimization and an opt-in policy change; the measurements below do not establish an enabled-path performance improvement. ### Benchmark setup - Local JMH 1.35 experiment on a shared arm64/macOS development machine, JDK 8u302; `-Xms512m -Xmx768m -XX:+UseParallelGC`. - Each invocation writes 64 BLOBs through the real writer to an in-memory destination, including payload copying, CRC, framing, footer, and close. Results are normalized per BLOB. Payload/source preparation and reporter setup are outside the timed operation. - Four warmup iterations of 1 s and five measurement iterations of 1 s, with three independent JVM forks per arm/configuration. Each fork contributes one mean; the sample size is **3 pairs, not 15 independent iterations**. - Cases were run serially with randomized case order and rotating arm order. All data below comes from the same November 6 rerun; no earlier-run samples were pooled. - `inline` is an in-memory source; `file` is warmed local-file I/O; `file_reused` reuses the referenced input stream; `http` uses a loopback HTTP server. These are not cold-disk or CDN/WAN measurements. - `F1/F4` means Flink metric objects/adapter with one/four workers; `P1` means Paimon's metric backend with one worker. All cases use the Paimon BLOB writer. Four-worker cases have independent writers/reporters, so they do not measure shared-reporter lock contention. No periodic production metrics exporter runs in the test. - The harness constructs the reporter directly to isolate writer cost. Public table-option propagation is covered separately by absent/false/true integration tests through real append `TableWrite` creation, write, commit, and readback. The four arms are: | Arm | Revision | Collection behavior | | --- | --- | --- | | Baseline | `b4342332eafd` | Existing `blobFetch` metrics; no transfer instrumentation | | Old PR On | `4643d3905cfd` | Original PR with transfer collection enabled | | Revised Off | `cae3dc2491f9` | Real default-disabled reporter; existing `blobFetch` metrics retained | | Revised On | `cae3dc2491f9` | Same revised binaries with transfer collection explicitly enabled | There are **186 valid real JVM forks**: 16 instrumented-source configurations x 3 arms x 3 repeats, 6 additional old-PR configurations x 3 repeats, and 4 NOOP configurations x 2 revisions x 3 repeats. All passed the harness's payload/CRC/footer, class-origin/capability, and metric-semantic checks. NOOP aliases do not add samples. Absolute times below are arithmetic means of fork means. Percentage changes are geometric means of the three paired time ratios, with Student-t 95% intervals on log-ratios (`df=2`), then transformed back. Consequently, percentages need not equal division of the displayed arithmetic means. Positive changes mean slower. Intervals crossing zero do not demonstrate zero overhead or equivalence. The contrasts reuse observations and have no multiple-comparison correction; there is no pooled "overall improvement" claim. ### Four-arm comparison: all six remeasured 1 KiB old-PR cases Times are **ns/BLOB**, lower is better. The last column is percentage change **[95% CI]** of Revised On relative to Old PR On. | Source / size / backend-workers | Baseline | Old PR On | Revised Off | Revised On | On vs old PR, % [95% CI] | | --- | ---: | ---: | ---: | ---: | ---: | | inline / 1 KiB / F1 | 208.49 | 264.52 | 205.25 | 239.65 | -9.11 [-28.35, +15.30] | | file / 1 KiB / F1 | 19820.55 | 19920.54 | 20117.45 | 19345.05 | -2.90 [-9.22, +3.86] | | file_reused / 1 KiB / F1 | 1292.09 | 1273.00 | 1296.44 | 1348.97 | +5.70 [-9.94, +24.06] | | http / 1 KiB / F1 | 156044.35 | 178528.66 | 167391.16 | 167213.83 | -5.15 [-19.14, +11.26] | | inline / 1 KiB / P1 | 208.40 | 246.36 | 205.01 | 251.32 | +1.64 [-44.82, +87.20] | | inline / 1 KiB / F4 | 208.37 | 476.80 | 222.18 | 483.60 | +1.41 [-8.36, +12.22] | All six revised-on/old-PR intervals include zero, so these data do not demonstrate an enabled-path speedup. The old PR was remeasured only for these six cases; other old-PR cells were not measured. <details> <summary><strong>Full 16-configuration matrix: baseline, revised Off, revised On</strong></summary> Times in this table are **microseconds/BLOB** (`1 us = 1000 ns`). Each change column is **percent [95% CI]**. Here `Off/Base`, `On/Off`, and `On/Base` compare the indicated numerator's elapsed time with the denominator. | Source / size / backend-workers | Baseline, us | Off, us | On, us | Off/Base, % [95% CI] | On/Off, % [95% CI] | On/Base, % [95% CI] | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | inline / 1 KiB / F1 | 0.2085 | 0.2052 | 0.2396 | -1.54 [-12.31, +10.56] | +16.37 [-10.68, +51.60] | +14.58 [-14.12, +52.87] | | inline / 64 KiB / F1 | 12.093 | 12.301 | 12.992 | +1.62 [-1.39, +4.72] | +5.59 [-7.44, +20.44] | +7.30 [-4.89, +21.05] | | inline / 1 MiB / F1 | 185.684 | 183.379 | 181.759 | -1.21 [-6.72, +4.62] | -0.88 [-2.28, +0.54] | -2.08 [-8.20, +4.44] | | file / 1 KiB / F1 | 19.821 | 20.117 | 19.345 | +1.49 [-3.69, +6.95] | -3.85 [-15.31, +9.16] | -2.42 [-12.57, +8.91] | | file / 64 KiB / F1 | 48.589 | 45.845 | 50.584 | -5.41 [-21.22, +13.57] | +9.08 [-32.30, +75.77] | +3.18 [-46.65, +99.56] | | file / 1 MiB / F1 | 455.333 | 418.266 | 415.607 | -7.71 [-35.96, +32.99] | -0.63 [-6.21, +5.28] | -8.30 [-34.55, +28.47] | | file_reused / 1 KiB / F1 | 1.292 | 1.296 | 1.349 | +0.23 [-6.44, +7.37] | +3.98 [+0.91, +7.14] | +4.21 [-5.35, +14.74] | | file_reused / 64 KiB / F1 | 26.252 | 27.122 | 25.929 | +3.22 [-5.15, +12.34] | -4.26 [-15.23, +8.13] | -1.18 [-8.03, +6.19] | | file_reused / 1 MiB / F1 | 430.389 | 439.346 | 426.362 | +2.10 [+0.38, +3.84] | -2.97 [-11.42, +6.28] | -0.94 [-10.59, +9.76] | | http / 1 KiB / F1 | 156.044 | 167.391 | 167.214 | +6.91 [+0.58, +13.64] | -0.05 [-5.64, +5.87] | +6.85 [+2.53, +11.36] | | http / 64 KiB / F1 | 778.338 | 565.084 | 551.655 | -22.00 [-75.72, +150.60] | -2.39 [-20.88, +20.41] | -23.87 [-75.03, +132.10] | | http / 1 MiB / F1 | 5663.291 | 6017.630 | 5846.251 | +6.12 [-11.46, +27.18] | -2.77 [-22.46, +21.93] | +3.18 [-4.80, +11.82] | | inline / 1 KiB / P1 | 0.2084 | 0.2050 | 0.2513 | -1.60 [-8.78, +6.14] | +21.46 [-19.94, +84.28] | +19.51 [-15.24, +68.52] | | inline / 1 KiB / F4 | 0.2084 | 0.2222 | 0.4836 | +6.27 [-22.71, +46.12] | +118.36 [+77.77, +168.22] | +132.06 [+103.59, +164.51] | | inline / 64 KiB / P1 | 11.824 | 12.214 | 11.737 | +2.81 [-15.91, +25.69] | -3.40 [-24.09, +22.93] | -0.69 [-4.94, +3.75] | | inline / 64 KiB / F4 | 11.768 | 11.897 | 12.285 | +1.04 [-1.40, +3.54] | +3.40 [-34.21, +62.49] | +4.48 [-31.96, +60.42] | </details> ### NOOP controls These controls use an empty collector and construct no real reporter. **They are different from Revised Off**, which retains the existing metrics. Times are **microseconds/BLOB**; changes are **percent [95% CI]**, revised binaries relative to baseline binaries. | Source / size / backend-workers | Baseline NOOP, us | Revised NOOP, us | Change, % [95% CI] | | --- | ---: | ---: | ---: | | inline / 1 KiB / F1 | 0.2124 | 0.2110 | -0.61 [-9.72, +9.42] | | inline / 64 KiB / F1 | 11.295 | 11.525 | +2.00 [-6.64, +11.44] | | inline / 1 MiB / F1 | 180.332 | 204.102 | +13.10 [+3.52, +23.55] | | inline / 1 KiB / F4 | 0.2822 | 0.2210 | -19.72 [-57.09, +50.18] | ### Interpretation and current policy - The clearest enabled-group cost is the 1 KiB inline/F4 case: **222.18 -> 483.60 ns/BLOB**, about **261 ns/BLOB** additional time; the paired increase is **118.36% [77.77%, 168.22%]**. Relative to the original baseline, the enabled revision is **132.06% [103.59%, 164.51%]** slower in this case. - The enabled-loop restructuring has not established an improvement over the original PR; all six measured new-on/old-on intervals cross zero. - In the default-off comparisons, the paired percentage intervals are above zero for `file_reused / 1 MiB / F1` (**+2.10% [0.38%, 3.84%]**) and `http / 1 KiB / F1` (**+6.91% [0.58%, 13.64%]**). These are log-ratio intervals; the separately computed absolute-time-difference interval for the latter crosses zero. They should not be described as proof that every statistical formulation agrees. - The 1 MiB NOOP/F1 control also differs (**+13.10% [3.52%, 23.55%]**). These measurements do not isolate the cause of all residual differences: neither zero disabled-path overhead nor attribution of every difference exclusively to active telemetry is established. - Larger-payload and loopback-I/O intervals are often wide. A small point estimate or an interval crossing zero does not establish negligible overhead. These are exploratory local measurements on a shared developer machine, not production throughput/SLO evidence, a shared-lock-contention test, or a measurement of cluster checkpoints/exporter overhead. I therefore propose keeping the **whole additional group disabled by default and enabling it for diagnostics when needed**. This experiment toggles the group as a whole; it does not identify an individual counter or histogram that is safe to enable by default. That decision would need separate measurements and an agreed overhead budget. ### Validation and current CI - The targeted local BLOB/options/table-option integration reactor passed **171 tests**, with no failures, errors, or skips. Generated configuration documentation was updated; documentation verification passed **14 tests on each of JDK 8 and JDK 11**. - The follow-up commits after the measured `cae3dc2` revision only update generated documentation and a Spark 4 test-scoped embedded-Hive dependency; they do not change the measured production instrumentation code. - The current head is `fa9b4edc723e0b694a64a897d4a7b96dead1832e`. [CI run 42178275426](https://github.com/apache/paimon/actions/runs/42178275426) is successful, with 17 jobs successful and Python skipped by selection, including the previously failing Spark 4/Scala 2.13 job and the final CI result. The code and documentation now make the opt-in behavior and payload-level scope explicit. I would welcome feedback on whether this opt-in diagnostic group is an acceptable direction for the PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
