abstractdog commented on code in PR #379:
URL: https://github.com/apache/tez/pull/379#discussion_r1855299374
##########
tez-runtime-internals/src/main/java/org/apache/tez/runtime/metrics/TaskCounterUpdater.java:
##########
@@ -69,32 +75,16 @@ public TaskCounterUpdater(TezCounters counters,
Configuration conf, String pid)
public void updateCounters() {
- // FileSystemStatistics are reset each time a new task is seen by the
- // container.
- // This doesn't remove the fileSystem, and does not clear all statistics -
- // so there is a potential of an unused FileSystem showing up for a
- // Container, and strange values for READ_OPS etc.
- Map<String, List<FileSystem.Statistics>> map = new
- HashMap<String, List<FileSystem.Statistics>>();
- for(Statistics stat: FileSystem.getAllStatistics()) {
- String uriScheme = stat.getScheme();
- if (map.containsKey(uriScheme)) {
- List<FileSystem.Statistics> list = map.get(uriScheme);
- list.add(stat);
- } else {
- List<FileSystem.Statistics> list = new
ArrayList<FileSystem.Statistics>();
- list.add(stat);
- map.put(uriScheme, list);
- }
- }
- for (Map.Entry<String, List<FileSystem.Statistics>> entry: map.entrySet())
{
- FileSystemStatisticUpdater updater =
statisticUpdaters.get(entry.getKey());
- if(updater==null) {//new FileSystem has been found in the cache
- updater =
- new FileSystemStatisticUpdater(tezCounters, entry.getValue(),
- entry.getKey());
- statisticUpdaters.put(entry.getKey(), updater);
+ GlobalStorageStatistics globalStorageStatistics =
FileSystem.getGlobalStorageStatistics();
+ Iterator<StorageStatistics> iter = globalStorageStatistics.iterator();
+ while (iter.hasNext()) {
+ StorageStatistics stats = iter.next();
+ if (!statisticUpdaters.containsKey(stats.getScheme())) {
+ Map<String, FileSystemStatisticUpdater> updaterSet = new TreeMap<>();
+ statisticUpdaters.put(stats.getScheme(), updaterSet);
}
+ FileSystemStatisticUpdater updater =
statisticUpdaters.get(stats.getScheme())
+ .computeIfAbsent(stats.getName(), k -> new
FileSystemStatisticUpdater(tezCounters, stats));
Review Comment:
absolutely, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
