[
https://issues.apache.org/jira/browse/HADOOP-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291552#comment-16291552
]
Steve Loughran commented on HADOOP-13171:
-----------------------------------------
I should add I've had a mixed experience using the Storage Stats for some work
where I'm trying to estimate cost of jobs, because its shared across FS
instances and threads. So on any query engine running >1 query in a thread,
even if you have separate FS instances, all their stats get aggregated. Stops
me producing useful values in the committers (which publish stats to their
.pending files): they overcount everything as each task done on the same
process duplicates the values.
I'd really like a storage stats counter which is ThreadLocal, and every metric
updating a counter doing it also for the thread. That could be expensive, so
very much something you'd only want to enable when you planned to use it. At
the same time, for the query engines to explicitly collect this stuff
themselves, it'd be good as some public FS option, "enable threadlocal stats"
> Add StorageStatistics to S3A; instrument some more operations
> -------------------------------------------------------------
>
> Key: HADOOP-13171
> URL: https://issues.apache.org/jira/browse/HADOOP-13171
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 2.8.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: HADOOP-13171-014.patch, HADOOP-13171-016.patch,
> HADOOP-13171-branch-2-001.patch, HADOOP-13171-branch-2-002.patch,
> HADOOP-13171-branch-2-003.patch, HADOOP-13171-branch-2-004.patch,
> HADOOP-13171-branch-2-005.patch, HADOOP-13171-branch-2-006.patch,
> HADOOP-13171-branch-2-007.patch, HADOOP-13171-branch-2-008.patch,
> HADOOP-13171-branch-2-009.patch, HADOOP-13171-branch-2-010.patch,
> HADOOP-13171-branch-2-011.patch, HADOOP-13171-branch-2-012.patch,
> HADOOP-13171-branch-2-013.patch, HADOOP-13171-branch-2-015.patch,
> HADOOP-13171-branch-2.8-017.patch
>
>
> Add {{StorageStatistics}} support to S3A, collecting the same metrics as the
> instrumentation, but sharing across all instances.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
