wwj6591812 commented on code in PR #10922:
URL: https://github.com/apache/paimon/pull/10922#discussion_r4531371834
##########
paimon-core/src/main/java/org/apache/paimon/operation/metrics/BlobFetchMetrics.java:
##########
@@ -117,8 +151,93 @@ public void recordFetchFailure(Throwable throwable) {
recordFailureReason(throwable);
}
- public void close() {
- metricGroup.close();
+ @Override
+ public boolean isTransferMetricsEnabled() {
+ return transferMetricsEnabled;
+ }
+
+ @Override
+ public void recordTransfer(
+ BlobTransferSource source,
+ long readBytes,
+ long writtenBytes,
+ long durationNanos,
+ boolean completed) {
+ if (!transferMetricsEnabled) {
+ return;
+ }
+ synchronized (this) {
+ if (closed) {
+ return;
+ }
+ BlobTransferSource kind = source == null ?
BlobTransferSource.UNKNOWN : source;
+ TransferMetrics transfer = transferGroups.get(kind);
+ if (transfer == null) {
+ Map<String, String> variables = new LinkedHashMap<>();
+ variables.put("table", tableName);
+ variables.put("sourceKind", kind.metricValue());
Review Comment:
Thanks, @JingsongLi, for checking this with two real writers and pointing
out the Spark export boundary.
I took the explicit-documentation option you suggested. The metrics
reference now explains that the shared Spark registry has no writer identity in
the key: a later registration replaces the same named gauge, so it reflects the
reporter that most recently registered that metric, rather than a
concurrent-writer sum. It also states that Spark exports only the histogram
mean over its retained samples (up to the latest 1024 per reporter/source
kind), not P95/P99 or a mergeable distribution. The per-writer rate and
quantile guidance is now conditional on the backend actually exposing those
writer identities and statistics.
Spark's existing adapter behavior is unchanged. I also added coverage using
two actual Data Evolution writers with separate production SparkMetricRegistry
instances on the same table, plus a check of the mean-only histogram export.
Validation: all 15 tests in PaimonMetricTest passed with Spark 4.1.2 / Scala
2.13 / JDK 17.
Addressed in c75ea4c956fe34019203ce34d9c2df8cc05718f7.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]