[
https://issues.apache.org/jira/browse/IGNITE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14365836#comment-14365836
]
Ivan Veselovsky edited comment on IGNITE-218 at 3/17/15 8:47 PM:
-----------------------------------------------------------------
Installing BigTop 0.8.0 on top of Ubuntu in general I followed these
instructions:
http://www.thecloudavenue.com/2014/01/InstallationAndConfigurationOfApacheBigtop.html
.
Here the scripts that initialize HDFS (create folders with appropriate owners
and permissions) may be an important item.
Installing of Ignite node is just several steps (unless otherwise stated we
work under "bigtop" non-root account ):
{code}
cd ~/
unzip -q ..../ignite-hadoop-{version}.zip
cd ignite-hadoop-{version}/
mkdir cfg0/
cp ./config/hadoop/core-site.ignite.xml ./cfg0/core-site.xml
cp ./config/hadoop/mapred-site.ignite.xml ./cfg0/mapred-site.xml
IGH=`pwd -P`
# Create custom hadoop clinet script for Ignite:
cat >~/bin/hadoop-ignited <<<EOF
# This is needed to resolve ignite logging configuration:
export IGNITE_HOME=${IGH}
export
HADOOP_CLASSPATH=\${IGNITE_HOME}/libs/ignite-core-1.0.0-RC3-SNAPSHOT.jar:\${IGNITE_HOME}/libs/ignite-hadoop/ignite-hadoop-1.0.0-RC3-SNAPSHOT.jar
hadoop --config \${IGNITE_HOME}/cfg0 "\${@}"
EOF
chmod +x ~/bin/hadoop-ignited
# now start the Ignite node:
cd bin/
./ignite.sh -v &
# run a Hadoop sample (assume ~/bin is in the PATH):
hadoop-ignited jar ..../hadoop-mapreduce-examples.jar pi 5 5
{code}
Note : if you configure secondary Fs (IGFS over HDFS), you don't need
{code}<constructor-arg name="cfgPath" value="/home/ivan/empty.xml"/> {code} any
more -- now it is optional.
was (Author: iveselovskiy):
Installing BigTop 0.8.0 on top of Ubuntu in general I followed these
instructions:
http://www.thecloudavenue.com/2014/01/InstallationAndConfigurationOfApacheBigtop.html
.
Here the scripts that initialize HDFS (create folders with appropriate owners
and permissions) may be an important item.
Installing of Ignite node is just several steps (unless otherwise stated we
work under "bigtop" non-root account ):
{code}
cd ~/
unzip -q ..../ignite-hadoop-{version}.zip
cd ignite-hadoop-{version}/
mkdir cfg0/
cp ./config/hadoop/core-site.ignite.xml ./cfg0/core-site.xml
cp ./config/hadoop/mapred-site.ignite.xml ./cfg0/mapred-site.xml
IGH=`pwd -P`
# Create custom hadoop clinet script for Ignite:
echo >~/bin/hadoop-ignited <<<EOF
# This is needed to resolve ignite logging configuration:
export IGNITE_HOME=${IGH}
export
HADOOP_CLASSPATH=\${IGNITE_HOME}/libs/ignite-core-1.0.0-RC3-SNAPSHOT.jar:\${IGNITE_HOME}/libs/ignite-hadoop/ignite-hadoop-1.0.0-RC3-SNAPSHOT.jar
hadoop --config \${IGNITE_HOME}/cfg0 "\${@}"
EOF
chmod +x ~/bin/hadoop-ignited
# now start the Ignite node:
cd bin/
./ignite.sh -v &
# run a Hadoop sample (assume ~/bin is in the PATH):
hadoop-ignited jar ..../hadoop-mapreduce-examples.jar pi 5 5
{code}
Note : if you configure secondary Fs (IGFS over HDFS), you don't need
{code}<constructor-arg name="cfgPath" value="/home/ivan/empty.xml"/> {code} any
more -- now it is optional.
> Wrong staging permissions while running MR job under hadoop accelerator
> -----------------------------------------------------------------------
>
> Key: IGNITE-218
> URL: https://issues.apache.org/jira/browse/IGNITE-218
> Project: Ignite
> Issue Type: Bug
> Components: hadoop
> Affects Versions: sprint-1
> Environment: Hadoop 2.4.1
> gridgain-hadoop-6.5.6 (Bigtop trunk) no later than
> d90b89dcc4ac7895d902fed034793c5fcb332e8b
> Reporter: Konstantin Boudnik
> Assignee: Ivan Veselovsky
> Fix For: sprint-3
>
>
> I am trying to run an MR example on gridgain-hadoop setup in Bigtop. The ggfs
> is configured to be backed by HDFS with
> {code}
> <property name="secondaryFileSystem">
> <bean
> class="org.gridgain.grid.kernal.ggfs.hadoop.GridGgfsHadoopFileSystemWrapper">
> <constructor-arg
> value="hdfs://vmhost05-hbase1.bdva.wandisco.com:8020"/>
> <constructor-arg value="/etc/hadoop/conf/hdfs-site.xml"/>
> </bean>
> </property>
> {code}
> set for {{ggfsConfiguration}}
> While all usual filesystem operations work properly, running an MR job
> produces the following exception in the beginning of the run
> {noformat}
> HADOOP_CONF_DIR=/etc/hadoop/gridgain.client.conf hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 2 2
> Number of Maps = 2
> Samples per Map = 2
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> java.io.IOException: The ownership on the staging directory
> /tmp/hadoop-yarn/staging/root/.staging is not as expected. It is owned by
> gridgain. The directory must be owned by the submitter root or by root
> at
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:112)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:348)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {noformat}
> However the permissions seem to be correct.
> {noformat}
> [root@vmhost05-hbase2 gridgain.client.conf]# sudo -u hdfs hadoop fs -ls
> /tmp/hadoop-yarn/staging/root
> Found 1 items
> drwx------ - root root 0 2015-02-10 19:31
> /tmp/hadoop-yarn/staging/root/.staging
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
