[ 
https://issues.apache.org/jira/browse/CRUNCH-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15575855#comment-15575855
 ] 

Dmitry Goldin commented on CRUNCH-602:
--------------------------------------

I don't fully understand why another layer of indirection would be necessary. 
Imho the sole issue here is that the Distributed Cache is just used wrongly - 
instead of retrieving the location of files localised by YARN, which are kept 
in the property {{mapreduce.job.cache.local.files}} and linked into the 
containers {{CWD}}, Crunch's DistCache just reads the paths from 
{{mapreduce.job.cache.files}} which points to the original location of the 
files (for instance {{/tmp/crunch-1068439135/p10/COMBINE}}) and then uses a 
client for the configured default-filesystem to retrieve them - which just 
leads to a regular, non cached, HDFS call (assuming that defaultFS is hdfs, as 
in most cases). So imho just a wrong use of the distributed cache.

> Combiner initialization repeatedly retrieves RT nodes from DistCache, leading 
> to high NN load
> ---------------------------------------------------------------------------------------------
>
>                 Key: CRUNCH-602
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-602
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 0.12.0, 0.13.0, 0.14.0
>         Environment: Crunch 0.14-SNAPSHOT, CDH5.6.0
>            Reporter: Michael Rose
>            Assignee: Josh Wills
>              Labels: performance
>         Attachments: crunch-602.patch, quickfix_crunch_distcache.patch
>
>
> When running one of our Crunch pipelines, we noticed our NameNode under very 
> heavy load. We run our masters on pretty light hardware, so our NN was 
> sitting at 100% CPU.
> Crunch reads the RTNodes during creation of a CrunchTaskContext. These are 
> created when Mappers and Reducers are created. Importantly, a CrunchCombiner 
> is a subclass of a Reducer, so each mapper will create R combiners where R is 
> the number of reducers and thus R CrunchTaskContexts. Consequently in highly 
> parallel jobs, this means M*R semi-expensive calls to the NameNode.
> In the constructor for CrunchTaskContext, this is the read to the DistCache:
> this.nodes = (List<RTNode>) DistCache.read(conf, path);
> Which then leads to a read into the NN + deserialization.
> For now, we took the overly simplistic approach of caching the results of the 
> DistCache read in a Guava cache. The cache ensures combiners reuse RTNodes 
> with only the overhead of deserialization which is somewhat unavoidable as 
> RTNodes are stateful and not reusable. However, it's not configurable except 
> by modifying code.
> I'll attach the patch, but given that it's not yet configurable I wouldn't 
> call it a "fix available." There may be much better ways of fixing this issue 
> as well -- if you have some guidance I'd be happy to do the legwork on a 
> patch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to