Hi again, I changed the String objects with org.apache.hadoop.io.Text objects
(why is String not accepted?), and now I get another exception, so I don´t
really know if I solved something or I broke something:
java.lang.Exception: java.lang.NullPointerException
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:988)
at
org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
If I could debug it in my IDE, I think I could work faster, but I have the
problems already exposed. How am I testing now? Building a jar, copying it on
the server and running a Hadoop jar command (not very performance approach...).
Could you give me a hand on this? Any Windows + IntelliJ IDEa there? Maaaany
thanks!
Von: Blanca Hernandez [mailto:[email protected]]
Gesendet: Mittwoch, 17. September 2014 15:27
An: [email protected]
Betreff: ClassCastException on running map-reduce jobs + tests on Windows
(mongo-hadoop)
Hi!
I am getting some CCE and don´t really understand why...
Here my mapper:
public class AveragePriceMapper extends Mapper<String, BSONObject, String,
BSONObject>{
@Override
public void map(final String key, final BSONObject val, final Context
context) throws IOException, InterruptedException {
String id = "result_of_making_some_operations";
context.write(id, val);
}
}
And in my configuration:
config.setMapperOutputKey(String.class);
config.setMapperOutputValue(BSONObject.class);
On running my generated jar on the server, seems to work everything ok until:
14/09/17 15:20:36 INFO mapred.MapTask: Processing split:
MongoInputSplit{URI=mongodb://user:pass@host:27017/my_db.my_collection,
authURI=null, min={ "_id" : { "$oid" : "541666d8e4b07265e257a42e"}}, max={ },
query={ }, sort={ }, fields={ }, notimeout=false}
14/09/17 15:20:36 INFO mapred.MapTask: Map output collector class =
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/09/17 15:20:36 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/09/17 15:20:36 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/09/17 15:20:36 INFO mapred.MapTask: soft limit at 83886080
14/09/17 15:20:36 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/09/17 15:20:36 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/09/17 15:20:36 INFO mapred.LocalJobRunner: map task executor complete.
14/09/17 15:20:36 WARN mapred.LocalJobRunner:
job_local1701078621_0001java.lang.Exception: java.lang.ClassCastException:
class java.lang.String
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.ClassCastException: class java.lang.String
at java.lang.Class.asSubclass(Class.java:3126)
at
org.apache.hadoop.mapred.JobConf.getOutputKeyComparator(JobConf.java:885)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:981)
at
org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:391)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:80)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:675)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Did I miss something??
Another issue I am worry about: working on a Windows system makes everything
quite complicated with Hadoop. I have it installed and running, the same as my
mongoDB database (I am using the connector provided by them). Running the same
main class I am using in the hadooop jar call on the server (in the example
before), but from my IDE, I get this exception:
PriviledgedActionException as:hernanbl cause:java.io.IOException: Failed to set
permissions of path:
\tmp\hadoop-hernanbl\mapred\staging\hernanbl1600842219\.staging to 0700
How could I make it run?
Many thanks!!
Best regards,
Blanca
