leonardBang commented on code in PR #2933: URL: https://github.com/apache/fluss/pull/2933#discussion_r3007251381
########## fluss-flink/fluss-flink-common/src/main/java/org/apache/fluss/flink/tiering/source/metrics/TieringMetrics.java: ########## @@ -0,0 +1,68 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.fluss.flink.tiering.source.metrics; + +import org.apache.fluss.annotation.Internal; +import org.apache.fluss.metrics.MetricNames; + +import org.apache.flink.metrics.Counter; +import org.apache.flink.metrics.MeterView; +import org.apache.flink.metrics.MetricGroup; +import org.apache.flink.metrics.groups.SourceReaderMetricGroup; + +/** + * A collection class for handling metrics in Tiering source reader. + * + * <p>All metrics are registered under group "fluss.tieringService", which is a child group of + * {@link org.apache.flink.metrics.groups.OperatorMetricGroup}. + * + * <p>The following metrics are available: + * + * <ul> + * <li>{@code fluss.tieringService.writtenBytes} - Counter: cumulative bytes written to the lake + * since the job started (physical file sizes). + * <li>{@code fluss.tieringService.writtenBytesPerSecond} - Meter: write bytes-per-second rate + * derived from the counter using a 60-second sliding window. + * </ul> + */ +@Internal +public class TieringMetrics { + + // Metric group names + public static final String FLUSS_METRIC_GROUP = "fluss"; + public static final String TIERING_SERVICE_GROUP = "tieringService"; + + private final Counter writtenBytesCounter; + + public TieringMetrics(SourceReaderMetricGroup sourceReaderMetricGroup) { + MetricGroup tieringServiceGroup = + sourceReaderMetricGroup + .addGroup(FLUSS_METRIC_GROUP) + .addGroup(TIERING_SERVICE_GROUP); + + this.writtenBytesCounter = + tieringServiceGroup.counter(MetricNames.TIERING_SERVICE_WRITTEN_BYTES); + tieringServiceGroup.meter( + MetricNames.TIERING_SERVICE_WRITTEN_BYTES_RATE, new MeterView(writtenBytesCounter)); + } + + /** Records bytes written to the lake. Called once per bucket completion. */ Review Comment: Called once per bucket completion. -> we can clarify more e.g. the calculation time is triggered when commit to lake. ########## website/docs/maintenance/observability/monitor-metrics.md: ########## @@ -1125,6 +1125,36 @@ How to Use Flink Metrics, you can see [Flink Metrics](https://nightlies.apache.o <td>Table</td> <td>The output records per second.</td> <td>Meter</td> - </tr> + </tr> + </tbody> +</table> + +### Tiering Service Metrics + +These metrics are exposed by the Flink-based tiering source reader when running the Lake Tiering Service. +All metrics are registered under the `fluss.tieringService` metric group, which is a child of the Flink `SourceReaderMetricGroup`. + +<table class="table table-bordered"> + <thead> + <tr> + <th class="text-left" style={{width: '225pt'}}>Metrics Name</th> + <th class="text-left" style={{width: '165pt'}}>Level</th> + <th class="text-left" style={{width: '300pt'}}>Description</th> + <th class="text-left" style={{width: '70pt'}}>Type</th> + </tr> + </thead> + <tbody> + <tr> + <td>writtenBytes</td> + <td>Flink Source Operator</td> Review Comment: Here look like strange when ehe word `written` appears with `Source Operator`, our Source Operator is a special Operator which is responsible for both read and write, could you add some notes to the description? ########## fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/PaimonLakeWriter.java: ########## @@ -84,9 +87,21 @@ public PaimonWriteResult complete() throws IOException { } catch (Exception e) { throw new IOException("Failed to complete Paimon write.", e); } + if (commitMessage instanceof CommitMessageImpl) { + bytesWritten = Review Comment: +1 to use bytes of fluss record, the file size of Paimon on disk reflects the data after processing during the commit phase, and therefore no longer represents the actual data volume during transmission. The tieringService metrics should capture the total data volume across the entire transfer process. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
