rdblue commented on code in PR #5268: URL: https://github.com/apache/iceberg/pull/5268#discussion_r926037826
########## api/src/main/java/org/apache/iceberg/metrics/ScanReport.java: ########## @@ -0,0 +1,251 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iceberg.metrics; + +import java.io.Serializable; +import java.time.Duration; +import java.util.concurrent.TimeUnit; +import org.apache.iceberg.Schema; +import org.apache.iceberg.expressions.Expression; +import org.apache.iceberg.metrics.MetricsContext.Counter; +import org.apache.iceberg.relocated.com.google.common.base.MoreObjects; +import org.apache.iceberg.relocated.com.google.common.base.Preconditions; + +/** + * A Table Scan report that contains all relevant information from a Table Scan. + */ +public class ScanReport implements Serializable { + + private final String tableName; + private final long snapshotId; + private final Expression filter; + private final Schema projection; + private final int matchingDataFiles; + private final int matchingDataManifests; + private final int totalDataManifestsRead; + private final int addedDataFiles; + private final int deletedDataFiles; + private final Duration totalPlanningDuration; + private final long totalFileSizeBytes; Review Comment: I think we should change some of these metrics to be more easily understood by people that see them. To do that, I'd start by establishing some consistent terms that we should use. Here's what I'm thinking: * Use "manifest" to refer to a manifest file * Use "file" to refer to a content file * Use "data" and "delete" to specify whether a manifest/file contains data or deletes * Use "filtered" to count what was rejected by a filter * Use "scanned" to count what is actually read * Use "result" to count the data files that are turned into tasks for the scan (alternatively, we could count `fileTasks`?) I'm trying to avoid using the term "matching" for a couple of reasons. First, it isn't entirely clear what that means when talking about a data file because the filter evaluation results in data files that need to be read, but don't necessarily match the filter. Second, using "filtered" instead gives a clearer picture of what happened during planning, since "matching" isn't always a result. For example, delete files that match the filter might be used or might not match based on sequence number. Using those terms gives some additional consistency. * `dataManifests` / `deleteManifests` - the number of data/delete manifests in the snapshot's manifest list * `filteredDataManifests` / `filteredDeleteManifests` - the number of data/delete manifests rejected by the filter * `scannedDataManifests` / `scannedDeleteManifests` - the number of data/delete manifests that were actually read * `totalDataFiles` / `totalDeleteFiles` - the number of data/delete files in manifests that would be scanned (not actually scanned) * `filteredDataFiles` / `filteredDeleteFiles` - the number of data/delete files rejected by the filter * `indexedDeleteFiles` - the number of delete files that were loaded into the `DeleteFileIndex` * `resultDataFiles` - the number of data files that are selected to be read Does that sound reasonable? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
