[
https://issues.apache.org/jira/browse/PARQUET-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17850686#comment-17850686
]
Steve Loughran commented on PARQUET-2374:
-----------------------------------------
we have standard names in a constants file, but components can add more, and
there are separate ".failed" counters/min/mean/max/ values for every duration
tracked by name.
here's the result of a big bulk log delete where 117 page list requests where
issued, 86 bulk delete calls. This is my audit log collection repo, doing a
monthly cleanup helps me validate big bulk list and delete perfomance at a
scale normal test runs could never do.
{code}
2024-05-21 11:15:00,912 [shutdown-hook-0] INFO statistics.IOStatisticsLogging
(IOStatisticsLogging.java:logIOStatisticsAtLevel(269)) - IOStatistics:
counters=((action_http_head_request=3)
(audit_request_execution=117)
(audit_span_creation=5)
(directories_created=1)
(files_deleted=21495)
(object_bulk_delete_request=86)
(object_continue_list_request=21)
(object_delete_objects=21495)
(object_list_request=5)
(object_metadata_request=3)
(object_multipart_list=1)
(object_put_request=1)
(object_put_request_completed=1)
(op_delete=1)
(op_get_file_status=2)
(op_glob_status=1)
(store_io_request=117));
gauges=();
minimums=((action_http_head_request.min=74)
(object_bulk_delete_request.min=869)
(object_continue_list_request.min=117)
(object_list_request.min=70)
(object_multipart_list.min=195)
(object_put_request.min=123)
(op_delete.min=95848)
(op_get_file_status.min=151)
(op_glob_status.min=746));
maximums=((action_http_head_request.max=624)
(object_bulk_delete_request.max=1568)
(object_continue_list_request.max=918)
(object_list_request.max=260)
(object_multipart_list.max=195)
(object_put_request.max=123)
(op_delete.max=95848)
(op_get_file_status.max=716)
(op_glob_status.max=746));
means=((action_http_head_request.mean=(samples=3, sum=832, mean=277.3333))
(object_bulk_delete_request.mean=(samples=86, sum=95423, mean=1109.5698))
(object_continue_list_request.mean=(samples=21, sum=9528, mean=453.7143))
(object_list_request.mean=(samples=5, sum=690, mean=138.0000))
(object_multipart_list.mean=(samples=1, sum=195, mean=195.0000))
(object_put_request.mean=(samples=1, sum=123, mean=123.0000))
(op_delete.mean=(samples=1, sum=95848, mean=95848.0000))
(op_get_file_status.mean=(samples=2, sum=867, mean=433.5000))
(op_glob_status.mean=(samples=1, sum=746, mean=746.0000)));
{code}
Note, for anyone who wants the filesystem (not stream) level iostats for abfs
and s3a in filesystem.close() calls, you can turn this on yourself
{code}
<property>
<name>fs.iostatistics.logging.level</name>
<value>info</value>
</property>
{code}
> Add metrics support for parquet file reader
> -------------------------------------------
>
> Key: PARQUET-2374
> URL: https://issues.apache.org/jira/browse/PARQUET-2374
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Affects Versions: 1.13.1
> Reporter: Parth Chandra
> Assignee: Parth Chandra
> Priority: Major
> Fix For: 1.14.0
>
>
> ParquetFileReader is used by many engines - Hadoop, Spark among them. These
> engines report various metrics to measure performance in different
> environments and it is usually useful to be able to get low level metrics out
> of the file reader and writers.
> It would be very useful to allow a simple interface to report the metrics.
> Callers can then implement the interface to record the metrics in any
> subsystem they choose.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
