[
https://issues.apache.org/jira/browse/MAPREDUCE-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898342#action_12898342
]
Arun C Murthy commented on MAPREDUCE-1901:
------------------------------------------
bq. sorry for the endless confusion - i will try to write up a detailed doc
tomorrow covering use cases and design/gaps etc.
Can you please attach one?
bq. the changes to distributed cache (of which there are little - i think most
changes are in jobclient and taskrunner) are concerned with making the
assumption that the shared objects are immutable (in which case mtime checks
can be bypassed).
This seems just wrong - *assuming* immutability is just not useful - the cost
of the mtime is trivial compared to doing the I/O which we are anyway saving.
It will essentially introduce randomness *if* the jar changed resulting in
different tasks of the same job getting different data rendering debugging
impossible!
-1
> Jobs should not submit the same jar files over and over again
> -------------------------------------------------------------
>
> Key: MAPREDUCE-1901
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1901
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Joydeep Sen Sarma
> Attachments: 1901.PATCH
>
>
> Currently each Hadoop job uploads the required resources
> (jars/files/archives) to a new location in HDFS. Map-reduce nodes involved in
> executing this job would then download these resources into local disk.
> In an environment where most of the users are using a standard set of jars
> and files (because they are using a framework like Hive/Pig) - the same jars
> keep getting uploaded and downloaded repeatedly. The overhead of this
> protocol (primarily in terms of end-user latency) is significant when:
> - the jobs are small (and conversantly - large in number)
> - Namenode is under load (meaning hdfs latencies are high and made worse, in
> part, by this protocol)
> Hadoop should provide a way for jobs in a cooperative environment to not
> submit the same files over and again. Identifying and caching execution
> resources by a content signature (md5/sha) would be a good alternative to
> have available.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
