[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090762#comment-17090762
 ] 

Eric Payne commented on MAPREDUCE-7277:
---------------------------------------

bq. In the following code, if ever mapId is in queue but not in cache, I think 
totalMemoryUsed could still be out of sync
I think I've convinced myself that this risk is almost null since the 
{{mapId}}'s {{IndexInfo}} will not be in the cache if {{mapId}} is not in the 
queue.

I have an additional question, however.
- {{IndexCache#readIndexFileToCache}} / {{IndexCache#removeMap}}:
-- Is it possible for {{removeMap(mapId)}} to get called after 
{{readIndexFileToCache}} adds {{mapId}}'s IndexInfo to the cache but before it 
adds {{mapId}} to the queue? If so, the following code will leave its IndexInfo 
in the cache (and in the queue after {{readIndexFileToCache}} switches back in):
{code:title=IndexCache#removeMap}
    if (!queue.remove(mapId)) {
      LOG.debug("Map ID {} not found in queue", mapId);
      return;
    }
{code}


> IndexCache totalMemoryUsed differs from cache contents.
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-7277
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7277
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Jonathan Turner Eagles
>            Assignee: Jonathan Turner Eagles
>            Priority: Major
>         Attachments: IndexCacheActualSize.png, MAPREDUCE-7277.001.patch
>
>
> It was observed recently in a nodemanager OOM that the memory was filled with 
> SpillRecords. However, the IndexCache was only 15% full (1.5MB used on a 10MB 
> configured cache size). In particular was noted that the booking variable 
> totalMemoryUsed, was out of sync with the contents of the cache showing 96% 
> full, thereby drastically reducing the effectiveness of the cache.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to