andygrove commented on code in PR #7094:
URL: https://github.com/apache/datafusion-comet/pull/7094#discussion_r4311991688
##########
docs/source/user-guide/latest/metrics.md:
##########
@@ -230,12 +230,16 @@ execution metrics. Counters accumulate per scan operator;
they do not instrument
| `scan_io_object_store_response_bytes_read` | Response bytes actually
consumed at that API, including bytes fetched between coalesced ranges. Not
HTTP wire bytes.
|
| `scan_io_metadata_cache_hits` | Successful, cache-eligible
metadata opens requiring no storage reads.
|
| `scan_io_metadata_cache_misses` | Successful, cache-eligible
metadata opens requiring storage reads. Failed opens and encrypted opens, which
bypass this shared cache, increment neither cache counter.
|
+| `scan_io_metadata_prefetches` | Metadata opens that used a load
started when the previous file opened
(`spark.comet.parquet.prefetchNextFileMetadata.enabled`). Each also counts as a
cache miss. |
| `row_groups_pruned_dynamic_filter` | Row groups skipped by a runtime
predicate whose value changes during execution, such as a local TopK threshold.
|
Reader-level and object-store bytes are two views of the same reads; do not
add them together.
Likewise, footer bytes are a subset of metadata bytes, not a third
reader-level category. A warm
-metadata-cache hit contributes no new metadata or footer I/O. A valid
plaintext footer followed by
-a page-index failure still contributes footer bytes; an invalid plaintext
footer does not.
+metadata-cache hit contributes no new metadata or footer I/O. A prefetched
load's footer and
+metadata bytes count when the prefetch reads them, so a prefetch that no open
uses, because the
+scan stopped early or pruned the file before opening it, adds bytes without a
matching cache hit.
Review Comment:
Fixed. The `scan_io_metadata_prefetches` row, the paragraph under the table
and the last sentence of the new `parquet.md` section now say that an unused
prefetch adds bytes with no matching cache miss and no prefetch count. The row
and the section also say the counter covers opens that used a load which read
storage ahead of them. (37ba40d1d6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
