[
https://issues.apache.org/jira/browse/MAPREDUCE-6128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-6128:
-------------------------------------
Attachment: MAPREDUCE-6128.v08.patch
Thanks for review, [~jlowe]!
Good point regarding DO_TESTMRJOBS_HACK. Leaving it this way so far, but
CLASSPATH setup in TestMRJobs is worth tackling.
Regarding jars vs non-jars, I think my main point it's our most common case,
and I definitely wanted to exclude directories to avoid some an accidental pick
up of a large directory. But sure we can consider other plain files.
bq. If the manifest asks for two different jars with the same basename then I
think it will silently skip the latter entry. Intentional?
Good observation. It was not truly intentional. I think the last-entry-wins
policy is more intuitive. Thus changing it and documenting it. Added a test.
bq. Theoretically distributed cache archives could also conflict with the
manifest, so I'm thinking the manifest should be processed after archives and
the conflict check should also check the archive list.
Good point, I agree. Added a test for that.
bq. We may want an info (debug?) log message when manifest entries are
overridden by other distributed cache entries.
done as well.
> Automatic addition of bundled jars to distributed cache
> --------------------------------------------------------
>
> Key: MAPREDUCE-6128
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6128
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
> Affects Versions: 2.5.1
> Reporter: Gera Shegalov
> Assignee: Gera Shegalov
> Attachments: MAPREDUCE-6128.v01.patch, MAPREDUCE-6128.v02.patch,
> MAPREDUCE-6128.v03.patch, MAPREDUCE-6128.v04.patch, MAPREDUCE-6128.v05.patch,
> MAPREDUCE-6128.v06.patch, MAPREDUCE-6128.v07.patch, MAPREDUCE-6128.v08.patch
>
>
> On the client side, JDK adds Class-Path elements from the job jar manifest
> on the classpath. In theory there could be many bundled jars in many
> directories such that adding them manually via libjars or similar means to
> task classpaths is cumbersome. If this property is enabled, the same jars are
> added
> to the task classpaths automatically.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
