DerGut commented on code in PR #3709: URL: https://github.com/apache/iceberg-rust/pull/3709#discussion_r4644890003
########## crates/iceberg/src/metrics.rs: ########## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Metrics reporting API for Iceberg table operations. + +use std::collections::HashMap; +use std::fmt::Debug; +use std::time::Duration; + +use async_trait::async_trait; + +use crate::TableIdent; +use crate::expr::Predicate; +use crate::spec::SchemaId; + +/// Receives metrics reports for completed table operations. +/// +/// Implementations must handle reporting failures internally so they do not affect the operation +/// that produced the report. +#[async_trait] +pub trait MetricsReporter: Debug + Send + Sync { + /// Reports the metrics for a completed table operation. + async fn report(&self, report: &MetricsReport); +} + +/// A metrics report for an Iceberg table operation. +#[derive(Debug)] +#[non_exhaustive] +pub enum MetricsReport { + /// A report produced by planning a table scan. + Scan(ScanReport), +} + +/// Information and metrics collected while planning a table scan. +#[derive(Debug)] +pub struct ScanReport { + pub(crate) table_ident: TableIdent, + pub(crate) snapshot_id: i64, + pub(crate) schema_id: SchemaId, + pub(crate) filter: Option<Predicate>, + pub(crate) projected_field_names: Vec<String>, + pub(crate) projected_field_ids: Vec<i32>, + pub(crate) metrics: ScanMetrics, + pub(crate) metadata: HashMap<String, String>, +} + +impl ScanReport { + /// Returns the identifier of the scanned table. + pub fn table_ident(&self) -> &TableIdent { + &self.table_ident + } + + /// Returns the ID of the scanned snapshot. + pub fn snapshot_id(&self) -> i64 { + self.snapshot_id + } + + /// Returns the ID of the schema used for the scan. + pub fn schema_id(&self) -> SchemaId { + self.schema_id + } + + /// Returns the scan filter, or `None` for an unfiltered scan. + pub fn filter(&self) -> Option<&Predicate> { + self.filter.as_ref() + } + + /// Returns the names of the projected fields. + pub fn projected_field_names(&self) -> &[String] { + &self.projected_field_names + } + + /// Returns the IDs of the projected fields. + pub fn projected_field_ids(&self) -> &[i32] { + &self.projected_field_ids + } + + /// Returns the metrics collected while planning the scan. + pub fn metrics(&self) -> &ScanMetrics { + &self.metrics + } + + /// Returns additional metadata attached to the report. + pub fn metadata(&self) -> &HashMap<String, String> { + &self.metadata + } +} + +/// Metrics collected while planning a table scan. +#[derive(Debug)] +pub struct ScanMetrics { Review Comment: I could see two reasons to make them nullable/ optional that both don't apply to the Rust code base (and it seems the nullability isn't exercised in Java either): ### An individual metric not always being available The currently implemented metrics are always available when a scan is completed, so no need to make individual ones optional (same in Java). Maybe this will be relevant for future metrics, but we can then make only those optional. ### Overall metric collection disabled When a user doesn't want to report metrics, we could skip the collection and not provide a report at all. In the Rust implementation from https://github.com/apache/iceberg-rust/pull/3732 we're always counting, but then discarding the report when it's not needed. An optional report could be an improvement, but I don't see it necessary at the metric level. I tried to run [a benchmark](https://github.com/DerGut/iceberg-rust/tree/bench/scan-metrics-overhead/dev/scan-metrics-overhead) of the counting overhead to assess whether it's worth implementing a skip option. This isn't really realistic because in practice there's a bunch of IO going on to spread the actual counter increments apart, but in the worst case where multiple workers (8 in the benchmark) are contending to update the atomics counters, 200ns per file/ 200ms per million files. That's not a small amount, but small enough given the unlikelyness of this setup that we can defer such a feature IMO. Java btw doesn't do this either. It [always counts](https://github.com/apache/iceberg/blob/a96e7d28c0a77bd4f944ba8f477421be94800460/core/src/main/java/org/apache/iceberg/SnapshotScan.java#L75-L80) and I don't really understand why it uses nullable counters 🤷 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
