[
https://issues.apache.org/jira/browse/MAPREDUCE-3693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189428#comment-13189428
]
Roman Shaposhnik commented on MAPREDUCE-3693:
---------------------------------------------
Now that I've looked at the actual implementation (Arun, thanks a lot for
pointing out where it is!) the only concern that I have left is that it is
*extremely* difficult to realize that mapreduce.admin.user.env is what actually
adds
LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native to the YarnChild's environment.
In other words, if the end user wants to tweak mapreduce.admin.user.env it is
a hard requirement to NOT forget adding
LD_LIBRARY_PATH=$HADOOP_COMMON_HOME/lib/native back to whatever user wants to
pass via that property. This strikes me as a pretty brittle implementation,
unless of course mapreduce.admin.user.env is meant to be one of those hidden
options that
you're better off not tweaking.
In summary, I really do think that one of the following needs to happen:
* $HADOOP_COMMON_HOME/lib/native needs to be appended to whatever value of
LD_LIBRARY_PATH there is in all cases
* mapreduce.admin.user.env needs to be documents as a do not touch option
* mapreduce.admin.user.env needs to be documents with a NOTE saying that
setting it will invalidate the default setting of LD_LIBRARY_PATH (which will
break loading of your native libraries)
> Job's shouldn't have to configure the location of compression libraries on
> the server in mapred-site.xml
> --------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3693
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3693
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Roman Shaposhnik
>
> I have noticed that org.apache.hadoop.mapred.MapReduceChildJVM doesn't
> forward the value of -Djava.library.path= from the parent JVM to the child
> JVM. Thus if one wants to use native libraries for compression the only
> option seems to be to manually include relevant java.library.path settings
> into the mapred-site.xml (as mapred.[map|reduce].child.java.opts).
> This seems to be a change in behavior compared to MR1 where TaskRunner.java
> used to do that:
> {noformat}
> String libraryPath = System.getProperty("java.library.path");
> if (libraryPath == null) {
> libraryPath = workDir.getAbsolutePath();
> } else {
> libraryPath += SYSTEM_PATH_SEPARATOR + workDir;
> }
> boolean hasUserLDPath = false;
> for(int i=0; i<javaOptsSplit.length ;i++) {
> if(javaOptsSplit[i].startsWith("-Djava.library.path=")) {
> javaOptsSplit[i] += SYSTEM_PATH_SEPARATOR + libraryPath;
> hasUserLDPath = true;
> break;
> }
> }
> if(!hasUserLDPath) {
> vargs.add("-Djava.library.path=" + libraryPath);
> }
> for (int i = 0; i < javaOptsSplit.length; i++) {
> vargs.add(javaOptsSplit[i]);
> }
> {noformat}
> Is this a regression or a deliberate choice?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
