[
https://issues.apache.org/jira/browse/IGNITE-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338096#comment-14338096
]
Ivan Veselovsky edited comment on IGNITE-218 at 2/26/15 9:03 AM:
-----------------------------------------------------------------
Hi, Konstantin ,
1) the configuration issue you're mentioning above really takes place. It is
described in https://issues.apache.org/jira/browse/IGNITE-179 . And the
configuration you have written in default-config.xml from that viewpoint seems
to be correct.
2) I'm not sure about {code}<constructor-arg
value="/etc/hadoop/conf/hdfs-site.xml"/>{code} -- all the documentation
mentions core-site.xml. See also
https://issues.apache.org/jira/browse/IGNITE-345 . If you don't need to use any
specific dfs parameters, you may use reference to an empty config there:
{code}<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
</configuration>{code}
3) About {code}HDFS version you are connecting to differs from local
version.{code} -- GG constructs this message heuristically in
GridGgfsHadoopFileSystemWrapper , and it is supposed to mean a problem with
communication to the remote secondary HDFS filesystem. This seems to be very
close to the actual cause of the permission problem you're facing. If the
problem is reproducible, can you please see logs of namenode/datanode -- some
exceptions there may clarify the situation.
4) AFAIK, BigTop will be upgraded, but instead of the latest GridGain , Apache
Ignite release will be used.
was (Author: iveselovskiy):
Hi, Konstantin ,
1) the configuration issue you're mentioning above really takes place. It is
described in https://issues.apache.org/jira/browse/IGNITE-179 . And the
configuration you have written in default-config.xml from that viewpoint seems
to be correct.
2) I'm not sure about {code}<constructor-arg
value="/etc/hadoop/conf/hdfs-site.xml"/>{code} -- all the documentation
mentions core-site.xml.
3) About {code}HDFS version you are connecting to differs from local
version.{code} -- GG constructs this message heuristically in
GridGgfsHadoopFileSystemWrapper , and it is supposed to mean a problem with
communication to the remote secondary HDFS filesystem. This seems to be very
close to the actual cause of the permission problem you're facing. If the
problem is reproducible, can you please see logs of namenode/datanode -- some
exceptions there may clarify the situation.
4) AFAIK, BigTop will be upgraded, but instead of the latest GridGain , Apache
Ignite release will be used.
> Wrong staging permissions while running MR job under hadoop accelerator
> -----------------------------------------------------------------------
>
> Key: IGNITE-218
> URL: https://issues.apache.org/jira/browse/IGNITE-218
> Project: Ignite
> Issue Type: Bug
> Components: hadoop
> Affects Versions: sprint-1
> Environment: Hadoop 2.4.1
> gridgain-hadoop-6.5.6 (Bigtop trunk)
> Reporter: Konstantin Boudnik
> Assignee: Ivan Veselovsky
> Fix For: sprint-2
>
>
> I am trying to run an MR example on gridgain-hadoop setup in Bigtop. The ggfs
> is configured to be backed by HDFS with
> {code}
> <property name="secondaryFileSystem">
> <bean
> class="org.gridgain.grid.kernal.ggfs.hadoop.GridGgfsHadoopFileSystemWrapper">
> <constructor-arg
> value="hdfs://vmhost05-hbase1.bdva.wandisco.com:8020"/>
> <constructor-arg value="/etc/hadoop/conf/hdfs-site.xml"/>
> </bean>
> </property>
> {code}
> set for {{ggfsConfiguration}}
> While all usual filesystem operations work properly, running an MR job
> produces the following exception in the beginning of the run
> {noformat}
> HADOOP_CONF_DIR=/etc/hadoop/gridgain.client.conf hadoop jar
> /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 2 2
> Number of Maps = 2
> Samples per Map = 2
> Wrote input for Map #0
> Wrote input for Map #1
> Starting Job
> java.io.IOException: The ownership on the staging directory
> /tmp/hadoop-yarn/staging/root/.staging is not as expected. It is owned by
> gridgain. The directory must be owned by the submitter root or by root
> at
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:112)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:348)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:306)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {noformat}
> However the permissions seem to be correct.
> {noformat}
> [root@vmhost05-hbase2 gridgain.client.conf]# sudo -u hdfs hadoop fs -ls
> /tmp/hadoop-yarn/staging/root
> Found 1 items
> drwx------ - root root 0 2015-02-10 19:31
> /tmp/hadoop-yarn/staging/root/.staging
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
