Ivan Veselovsky created IGNITE-483:
--------------------------------------
Summary: Simplify Hadoop "ignition"
Key: IGNITE-483
URL: https://issues.apache.org/jira/browse/IGNITE-483
Project: Ignite
Issue Type: Task
Reporter: Ivan Veselovsky
Currently in Ignite we have setup-hadoop script + java class to replace configs
and create symlinks to Ignite libraries.
This setup script and Ignite launcher script have many dependencies on the
Hadoop distribution layout (files, configs, etc.)
I suggest to simplify this:
1) It seems to me that *no* symlink and/or library copying needed at all.
Instead, to make hadoop client work via Ignite, we can use the following
wrapper around the default hadoop client launcher script:
file "hadoop-ignited" :
-----------------------------------------------
# Ignite home is needed to allow Ignite libraries to find the logger config (it
is resolved relative to the Ignite home):
export IGNITE_HOME=/home/ignite/ignite-hadoop-1.0.0-RC3-SNAPSHOT
# Add necessary Ignite libraries to the Hadoop client classpath:
export
HADOOP_CLASSPATH=${IGNITE_HOME}/libs/ignite-core-1.0.0-RC3-SNAPSHOT.jar:${IGNITE_HOME}/libs/ignite-hadoop/ignite-hadoop-1.0.0-RC3-SNAPSHOT.jar
hadoop --config ${IGNITE_HOME}/ignite-conf "${@}"
-----------------------------------------------
where ${IGNITE_HOME}/ignite-conf is folder inside IGNITE distribution where
the 2 custom configs are located: core-site.xml, mapred-site.xml .
(If the user wants only Ignite MapRed engine, but does not need Ignite
filesystem, he symlinks the default mapred-site.xml in this directory.)
This way, in order to use fully pre-configured Hadoop cluster the user only
needs to add "hadoop-ignited" script into PATH.
2) The only thing that is still needs attention is how the Ignite node should
find the Hadoop libraries. My suggestion is to provide 3 versions of config
scripts depending on the supported layouts, like "layout-apache",
"layout-cloudera", and "layout-bigtop-hortonworks" (I suppose, the latter 2 are
identical) . After that ask use after Ignite archive unzipping write 1 line
into startup script, like LAYOUT=cloudera, and that's it -- the corresponding
env script will be picked up to set up env variables needed by the user's
Hadoop distribution.
TODO: will this approach work with Hive, etc.?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
