[
https://issues.apache.org/jira/browse/PARQUET-2374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17838306#comment-17838306
]
Steve Loughran commented on PARQUET-2374:
-----------------------------------------
looking at this, the stats you really want to pick up on are actually the
thread-level context IOStats, which the s3a connector collects inside the
stream, exports through its getIOStatistics() call but also, after close(),
adds to a thread local aggregator (one which should be reset at the beginning
of work).
Really it should be the query engine collecting and aggregating this across the
entire job. Now that spark always builds on versions 3.4.0 this can be
done....the previous ways to add handlers to executors didn't quite line up.
(s3a and manifest committers do collect this stuff, but even if told to call
reset() during committer task attempt init, the read stages have generally done
most of their work. we need to reset at the beginning of a task and at the end
collect the stats and send up to the driver
> Add metrics support for parquet file reader
> -------------------------------------------
>
> Key: PARQUET-2374
> URL: https://issues.apache.org/jira/browse/PARQUET-2374
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Affects Versions: 1.13.1
> Reporter: Parth Chandra
> Assignee: Parth Chandra
> Priority: Major
> Fix For: 1.14.0
>
>
> ParquetFileReader is used by many engines - Hadoop, Spark among them. These
> engines report various metrics to measure performance in different
> environments and it is usually useful to be able to get low level metrics out
> of the file reader and writers.
> It would be very useful to allow a simple interface to report the metrics.
> Callers can then implement the interface to record the metrics in any
> subsystem they choose.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
