abstractdog commented on code in PR #379:
URL: https://github.com/apache/tez/pull/379#discussion_r1881823528
##########
tez-runtime-internals/src/main/java/org/apache/tez/runtime/metrics/TaskCounterUpdater.java:
##########
@@ -69,32 +75,16 @@ public TaskCounterUpdater(TezCounters counters,
Configuration conf, String pid)
public void updateCounters() {
- // FileSystemStatistics are reset each time a new task is seen by the
- // container.
- // This doesn't remove the fileSystem, and does not clear all statistics -
- // so there is a potential of an unused FileSystem showing up for a
- // Container, and strange values for READ_OPS etc.
- Map<String, List<FileSystem.Statistics>> map = new
- HashMap<String, List<FileSystem.Statistics>>();
- for(Statistics stat: FileSystem.getAllStatistics()) {
- String uriScheme = stat.getScheme();
- if (map.containsKey(uriScheme)) {
- List<FileSystem.Statistics> list = map.get(uriScheme);
- list.add(stat);
- } else {
- List<FileSystem.Statistics> list = new
ArrayList<FileSystem.Statistics>();
- list.add(stat);
- map.put(uriScheme, list);
- }
- }
- for (Map.Entry<String, List<FileSystem.Statistics>> entry: map.entrySet())
{
- FileSystemStatisticUpdater updater =
statisticUpdaters.get(entry.getKey());
- if(updater==null) {//new FileSystem has been found in the cache
- updater =
- new FileSystemStatisticUpdater(tezCounters, entry.getValue(),
- entry.getKey());
- statisticUpdaters.put(entry.getKey(), updater);
- }
+ GlobalStorageStatistics globalStorageStatistics =
FileSystem.getGlobalStorageStatistics();
+ Iterator<StorageStatistics> iter = globalStorageStatistics.iterator();
+ while (iter.hasNext()) {
+ StorageStatistics stats = iter.next();
+ // Fetch or initialize the updater set for the scheme
+ Map<String, FileSystemStatisticUpdater> updaterSet = statisticUpdaters
+ .computeIfAbsent(stats.getScheme(), k -> new TreeMap<>());
Review Comment:
hm, I cannot see the point of ordering while adding different statistic
updaters for the same scheme, let me revert back to HashMap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
