Hi All,
I have successfully setup Ignite as a FileSystem (HDFS replacement, in
memory) and this seems to work.
I am now trying to test our the MapReduce feature, i have struggled to get
any MapReduce job i have taken from our traditional Hadoop cluster to run on
IGFS, every time i get and NPE (stack trace to follow).
As such i have made a real simple job that simply moves contents of files
from one dir to another, to make sure we can get the basic running, yet we
are still getting an NPE.
Stack Trace:
Dec 06, 2015 9:09:49 AM
org.apache.ignite.internal.client.impl.connection.GridClientNioTcpConnection
<init>
INFO: Client TCP connection established: /127.0.0.1:11211
Dec 06, 2015 9:09:49 AM
org.apache.ignite.internal.client.impl.GridClientImpl <init>
INFO: Client started [id=e73cf2d5-b099-4eff-85a2-fff0cf2a43b3, protocol=TCP]
Exception in thread "main" java.lang.NullPointerException
at
org.apache.ignite.internal.processors.hadoop.proto.HadoopClientProtocol.getNewJobID(HadoopClientProtocol.java:106)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:357)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at HDFSToolR.main(HDFSToolR.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Map Reduce Job:
Configuration configuration = new Configuration();
configuration.addResource(HDFSToolR.class.getClassLoader().getResource("core-site.xml"));
configuration.addResource(HDFSToolR.class.getClassLoader().getResource("mapred-site.xml"));
Job job = Job.getInstance(configuration, "test");
job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
job.setMapperClass(Mapper.class);
job.setReducerClass(Reducer.class);
FileInputFormat.setInputPaths(job, new Path("/input/"));
FileOutputFormat.setOutputPath(job, new Path("/output" +
System.currentTimeMillis() + "/"));
job.submit();
core-site:
<configuration>
<property>
<name>fs.default.name</name>
<value>igfs://[email protected]:10500</value>
</property>
<property>
<name>fs.igfs.impl</name>
<value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value>
</property>
<property>
<name>fs.AbstractFileSystem.igfs.impl</name>
<value>org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>ignite</value>
</property>
<property>
<name>mapreduce.jobtracker.address</name>
<value>127.0.0.1:11211</value>
</property>
</configuration>
I have looked at the test cases
https://github.com/apache/ignite/tree/master/modules/hadoop/src/test/java/org/apache/ignite/internal/processors/hadoop
But they all seem to use some Ignite job submitter classes, and manually
create the JobID, none seem to use the MR Job interface only.
Eg. Test cases seem to submit the job by creating the JobId:
HadoopJobId jobId = new HadoopJobId(UUID.randomUUID(), 1);
IgniteInternalFuture<?> fut = grid(0).hadoop().submit(jobId,
createJobInfo(job.getConfiguration()));
Where as an end user, just wanting to port their MR jobs without issues, i
would expect to work as normal:
job.submit();
Am i right in my assumption it should be seamless and able to run Hadoop MR
jobs without change?
Is there something wrong in my setup?
What can i do to possible help debug or correct the issue?
How can i get more useful information from the NPE? (i have tried running
debugger in intellij btw, but connection seems to time out as breakpoints
make it reliant on me the human who isn't fast :) )
Any help would be very much appreciated.
Thanks in advance.
Mike
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/NPE-issue-with-trying-to-submit-Hadoop-MapReduce-tp2146.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
