[
https://issues.apache.org/jira/browse/HADOOP-17461?focusedWorklogId=794121&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-794121
]
ASF GitHub Bot logged work on HADOOP-17461:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 22/Jul/22 08:25
Start Date: 22/Jul/22 08:25
Worklog Time Spent: 10m
Work Description: mehakmeet commented on PR #4352:
URL: https://github.com/apache/hadoop/pull/4352#issuecomment-1192316937
Pushed the changes, makes sense in case of null IOStatisticsContext. There
is one more issue, in case we have `fs.thread.level.iostatistics.enabled=false`
which means empty counters, hence the test getting failed in the test in this
PR only, since it requires thread-level IOstatsitics to be enabled, now we do
the `removeBaseAndBucketOverrides(configuration,
THREAD_LEVEL_IOSTATISTICS_ENABLED);
configuration.setBoolean(THREAD_LEVEL_IOSTATISTICS_ENABLED, true);`
But, since the way this is toggled is in a static initializer for the
`IOStatisticsContextIntegration` utility class, it does it's calculation before
this, and setting it afterwards doesn't have an affect, how do we ensure that
this property is enabled for this test always? Reload the class using
classLoaders(not sure if this works since I tried using the
Thread.currentThread.getClassLoaderContext)?
Also
> Contributed by Mehakmeet Singh
*and Steve Loughran
Issue Time Tracking
-------------------
Worklog Id: (was: 794121)
Time Spent: 8h 10m (was: 8h)
> Add thread-level IOStatistics Context
> -------------------------------------
>
> Key: HADOOP-17461
> URL: https://issues.apache.org/jira/browse/HADOOP-17461
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/azure, fs/s3
> Affects Versions: 3.3.1
> Reporter: Steve Loughran
> Assignee: Mehakmeet Singh
> Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 10m
> Remaining Estimate: 0h
>
> For effective reporting of the iostatistics of individual worker threads, we
> need a thread-level context which IO components update.
> * this contact needs to be passed in two background thread forming work on
> behalf of a task.
> * IO Components (streams, iterators, filesystems) need to update this context
> statistics as they perform work
> * Without double counting anything.
> I imagine a ThreadLocal IOStatisticContext which will be updated in the
> FileSystem API Calls. This context MUST be passed into the background threads
> used by a task, so that IO is correctly aggregated.
> I don't want streams, listIterators &c to do the updating as there is more
> risk of double counting. However, we need to see their statistics if we want
> to know things like "bytes discarded in backwards seeks". And I don't want to
> be updating a shared context object on every read() call.
> If all we want is store IO (HEAD, GET, DELETE, list performance etc) then the
> FS is sufficient.
> If we do want the stream-specific detail, then I propose
> * caching the context in the constructor
> * updating it only in close() or unbuffer() (as we do from S3AInputStream to
> S3AInstrumenation)
> * excluding those we know the FS already collects.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
