[
https://issues.apache.org/jira/browse/IGNITE-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Ozerov updated IGNITE-483:
-----------------------------------
Component/s: IGFS
> Simplify Hadoop "ignition"
> ---------------------------
>
> Key: IGNITE-483
> URL: https://issues.apache.org/jira/browse/IGNITE-483
> Project: Ignite
> Issue Type: Task
> Components: hadoop
> Affects Versions: sprint-1
> Reporter: Ivan Veselovsky
> Assignee: Ivan Veselovsky
>
> Currently in Ignite we have setup-hadoop script + java class to replace
> configs and create symlinks to Ignite libraries.
> This setup script and Ignite launcher script have many dependencies on the
> Hadoop distribution layout (files, configs, etc.)
> I suggest to simplify this:
> 1) It seems to me that *no* symlink and/or library copying needed at all.
> Instead, to make hadoop client work via Ignite, we can use the following
> wrapper around the default hadoop client launcher script:
> file "hadoop-ignited" :
> {code}
> # Ignite home is needed to allow Ignite libraries to find the logger config
> (it is resolved relative to the Ignite home):
> export IGNITE_HOME=/home/ignite/ignite-hadoop-1.0.0-RC3-SNAPSHOT
> # Add necessary Ignite libraries to the Hadoop client classpath:
> export
> HADOOP_CLASSPATH=${IGNITE_HOME}/libs/ignite-core-1.0.0-RC3-SNAPSHOT.jar:${IGNITE_HOME}/libs/ignite-hadoop/ignite-hadoop-1.0.0-RC3-SNAPSHOT.jar:${IGNITE_HOME}/libs/ignite-shmem-1.0.0.jar
> hadoop --config ${IGNITE_HOME}/ignite-conf "${@}"
> {code}
> where {code}${IGNITE_HOME}/ignite-conf{code} is folder inside IGNITE
> distribution where the 2 custom configs are located: core-site.xml,
> mapred-site.xml .
> (If the user wants only Ignite MapRed engine, but does not need Ignite
> filesystem, he symlinks the default mapred-site.xml in this directory.)
> This way, in order to use fully pre-configured Hadoop cluster the user only
> needs to add "hadoop-ignited" script into PATH.
> After that instead of running hadoop jar examples.jar pi 10 10 , use can run
> hadoop-ignited jar examples.jar pi 10 10 to run the sample on fully ignited
> Hadoop.
> 2) The only thing that is still needs attention is how the Ignite node should
> find the Hadoop libraries. My suggestion is to provide 3 versions of config
> scripts depending on the supported layouts, like "layout-apache",
> "layout-cloudera", and "layout-bigtop-hortonworks" (I suppose, the latter 2
> are identical) . After that ask use after Ignite archive unzipping write 1
> line into startup script, like LAYOUT=cloudera, and that's it -- the
> corresponding env script will be picked up to set up env variables needed by
> the user's Hadoop distribution.
> (Earlier attempt to solve this problem described in
> https://issues.apache.org/jira/browse/IGNITE-372 .)
> This approach works with Hive as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
