Ivan Veselovsky created IGNITE-372:
--------------------------------------
Summary: improve Ignite node Hadoop classpath composition
reliability
Key: IGNITE-372
URL: https://issues.apache.org/jira/browse/IGNITE-372
Project: Ignite
Issue Type: Improvement
Affects Versions: sprint-2
Reporter: Ivan Veselovsky
Assignee: Ivan Veselovsky
Priority: Minor
User encountered a problem with Hadoop mapred jobs execution when incorrectly
set in environment
export HADOOP_MAPRED_HOME=${HADOOP_HOME}
export HADOOP_COMMON_HOME=${HADOOP_HOME}
export HADOOP_HDFS_HOME=${HADOOP_HOME}
. That caused incorrect GRIDGAIN_HADOOP_CLASSPATH and , as a consequence, job
failure with NoClassDefFoundError.
Suggestions to improve reliability and tracability of the Hadoop classpath
composition:
1) Simplify the logic of the classpath composition: do not take into account
HADOOP_COMMON_HOME, HADOOP_MAPRED_HOME, HADOOP_HDFS_HOME, also do not read
/etc/default/hadoop .
Compose the classpath basing only on the value of $HADOOP_HOME.
2) when GRIDGAIN_HADOOP_CLASSPATH composed, print out its value if ignite.sh as
run with "-v" option.
3) add the Hadoop classpath validation as a part of Node startup sequence: try
to find (load?) at least one class from each of the key Hadoop APIs (FS,
MapRed, Yarn API).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
