JingsongLi commented on code in PR #10922:
URL: https://github.com/apache/paimon/pull/10922#discussion_r4523262277


##########
paimon-format/src/main/java/org/apache/paimon/format/blob/AbstractBlobElementWriter.java:
##########
@@ -115,15 +122,23 @@ protected final long writePlaceholderElement() {
     }
 
     protected final BlobFetchResult getBlob(BlobGetter getter) {
+        if (transferMetricsEnabled) {
+            transferStartNanos = System.nanoTime();

Review Comment:
   **[P2] Exclude the previous source's close from the next source's latency**
   
   With a real BLOB writer, I copied a local `FILE` payload, a bounded `CUSTOM` 
payload whose stream takes 350 ms to close, and the same `FILE` payload again. 
The first file copy reported about 2 ms and the second about 356 ms. The timer 
has already started when `reuseSource.prepareFor` closes the previous source, 
so the latency histogram attributes that custom stream's close to `file`. Could 
we exclude the previous source's cleanup time from the next sample, or expose 
handoff time separately? That would keep the source-kind latency meaningful 
when one writer alternates sources.



##########
paimon-core/src/main/java/org/apache/paimon/operation/metrics/BlobFetchMetrics.java:
##########
@@ -117,8 +151,93 @@ public void recordFetchFailure(Throwable throwable) {
         recordFailureReason(throwable);
     }
 
-    public void close() {
-        metricGroup.close();
+    @Override
+    public boolean isTransferMetricsEnabled() {
+        return transferMetricsEnabled;
+    }
+
+    @Override
+    public void recordTransfer(
+            BlobTransferSource source,
+            long readBytes,
+            long writtenBytes,
+            long durationNanos,
+            boolean completed) {
+        if (!transferMetricsEnabled) {
+            return;
+        }
+        synchronized (this) {
+            if (closed) {
+                return;
+            }
+            BlobTransferSource kind = source == null ? 
BlobTransferSource.UNKNOWN : source;
+            TransferMetrics transfer = transferGroups.get(kind);
+            if (transfer == null) {
+                Map<String, String> variables = new LinkedHashMap<>();
+                variables.put("table", tableName);
+                variables.put("sourceKind", kind.metricValue());

Review Comment:
   **[P2] Account for concurrent Spark writers in the exported metrics**
   
   I used two actual writers for the same local Data Evolution table, each with 
the production `SparkMetricRegistry`, in one JVM. The `blobTransferCompleted` 
JMX series read 3 after writer A copied three blobs, then 1 after writer B 
copied one, and stayed 1 after A copied another. `SparkMetricGroup` keys the 
shared registry by group/table/source kind and replaces an existing gauge on 
registration. The Spark adapter also exports only the histogram mean. Could we 
give Spark a writer-scoped or properly aggregated export, or explicitly 
document its last-registered-writer/mean-only behavior and scope the per-writer 
rate and quantile guidance to backends that expose those series?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to