Hi, Check your kernel's overcommit settings. This will prevent the JVM from allocating memory even when there's free RAM.
Brian On Mar 4, 2011, at 3:55 PM, Ratner, Alan S (IS) wrote: > Aaron, > > Thanks for the rapid responses. > > > * "ulimit -u unlimited" is in .bashrc. > > > * HADOOP_HEAPSIZE is set to 4000 MB in hadoop-env.sh > > > * Mapred.child.ulimit is set to 2048000 in mapred-site.xml > > > * Mapred.child.java.opts is set to -Xmx1536m in mapred-site.xml > > I take it you are suggesting that I change the java.opts command to: > > Mapred.child.java.opts is <value> -Xmx1536m > -Djava.library.path=/path/to/native/libs </value> > > > Alan Ratner > Northrop Grumman Information Systems > Manager of Large-Scale Computing > 9020 Junction Drive > Annapolis Junction, MD 20701 > (410) 707-8605 (cell) > > From: Aaron Kimball [mailto:[email protected]] > Sent: Friday, March 04, 2011 4:30 PM > To: [email protected] > Cc: Ratner, Alan S (IS) > Subject: EXT :Re: Problem running a Hadoop program with external libraries > > Actually, I just misread your email and missed the difference between your > 2nd and 3rd attempts. > > Are you enforcing min/max JVM heap sizes on your tasks? Are you enforcing a > ulimit (either through your shell configuration, or through Hadoop itself)? I > don't know where these "cannot allocate memory" errors are coming from. If > they're from the OS, could it be because it needs to fork() and momentarily > exceed the ulimit before loading the native libs? > > - Aaron > > On Fri, Mar 4, 2011 at 1:26 PM, Aaron Kimball > <[email protected]<mailto:[email protected]>> wrote: > I don't know if putting native-code .so files inside a jar works. A > native-code .so is not "classloaded" in the same way .class files are. > > So the correct .so files probably need to exist in some physical directory on > the worker machines. You may want to doublecheck that the correct directory > on the worker machines is identified in the JVM property 'java.library.path' > (instead of / in addition to $LD_LIBRARY_PATH). This can be manipulated in > the Hadoop configuration setting mapred.child.java.opts (include > '-Djava.library.path=/path/to/native/libs' in the string there.) > > Also, if you added your .so files to a directory that is already used by the > tasktracker (like hadoop-0.21.0/lib/native/Linux-amd64-64/), you may need to > restart the tasktracker instance for it to take effect. (This is true of .jar > files in the $HADOOP_HOME/lib directory; I don't know if it is true for > native libs as well.) > > - Aaron > > On Fri, Mar 4, 2011 at 12:53 PM, Ratner, Alan S (IS) > <[email protected]<mailto:[email protected]>> wrote: > We are having difficulties running a Hadoop program making calls to external > libraries - but this occurs only when we run the program on our cluster and > not from within Eclipse where we are apparently running in Hadoop's > standalone mode. This program invokes the Open Computer Vision libraries > (OpenCV and JavaCV). (I don't think there is a problem with our cluster - > we've run many Hadoop jobs on it without difficulty.) > > 1. I normally use Eclipse to create jar files for our Hadoop programs > but I inadvertently hit the "run as Java application" button and the program > ran fine, reading the input file from the eclipse workspace rather than HDFS > and writing the output file to the same place. Hadoop's output appears > below. (This occurred on the master Hadoop server.) > > 2. I then "exported" from Eclipse a "runnable jar" which "extracted > required libraries" into the generated jar - presumably producing a jar file > that incorporated all the required library functions. (The plain jar file for > this program is 17 kB while the runnable jar is 30MB.) When I try to run > this on my Hadoop cluster (including my master and slave servers) the program > reports that it is unable to locate "libopencv_highgui.so.2.2: cannot open > shared object file: No such file or directory". Now, in addition to this > library being incorporated inside the runnable jar file it is also present on > each of my servers at hadoop-0.21.0/lib/native/Linux-amd64-64/ where we have > loaded the same libraries (to give Hadoop 2 shots at finding them). These > include: > ... > libopencv_highgui_pch_dephelp.a > libopencv_highgui.so > libopencv_highgui.so.2.2 > libopencv_highgui.so.2.2.0 > ... > > When I poke around inside the runnable jar I find javacv_linux-x86_64.jar > which contains: > com/googlecode/javacv/cpp/linux-x86_64/libjniopencv_highgui.so > > 3. I then tried adding the following to mapred-site.xml as suggested in > Patch 2838 that's supposed to be included in hadoop 0.21 > https://issues.apache.org/jira/browse/HADOOP-2838 > <property> > <name>mapred.child.env</name> > > <value>LD_LIBRARY_PATH=/home/ngc/hadoop-0.21.0/lib/native/Linux-amd64-64</value> > </property> > The log is included at the bottom of this email with Hadoop now > complaining about a different missing library with an out-of-memory error. > > Does anyone have any ideas as to what is going wrong here? Any help would be > appreciated. Thanks. > > Alan > > > BTW: Each of our servers has 4 hard drives and many of the errors below refer > to the 3 drives (/media/hd2 or hd3 or hd4) reserved exclusively for HDFS and > thus perhaps not a good place for Hadoop to be looking for a library file. > My slaves have 24 GB RAM, the jar file is 30 MB, and the sequence file being > read is 400 KB - so I hope I am not running out of memory. > > > 1. RUNNING DIRECTLY FROM ECLIPSE IN HADOOP'S STANDALONE MODE - SUCCESS > >>>>> Running Face Program > 11/03/04 12:44:10 INFO security.Groups: Group mapping > impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; > cacheTimeout=300000 > 11/03/04 12:44:10 INFO jvm.JvmMetrics: Initializing JVM Metrics with > processName=JobTracker, sessionId= > 11/03/04 12:44:10 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the same. > 11/03/04 12:44:10 WARN mapreduce.JobSubmitter: No job jar file set. User > classes may not be found. See Job or Job#setJar(String). > 11/03/04 12:44:10 INFO mapred.FileInputFormat: Total input paths to process : > 1 > 11/03/04 12:44:10 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > 11/03/04 12:44:10 INFO mapreduce.JobSubmitter: number of splits:1 > 11/03/04 12:44:10 INFO mapreduce.JobSubmitter: adding the following > namenodes' delegation tokens:null > 11/03/04 12:44:10 WARN security.TokenCache: Overwriting existing token > storage with # keys=0 > 11/03/04 12:44:10 INFO mapreduce.Job: Running job: job_local_0001 > 11/03/04 12:44:10 INFO mapred.LocalJobRunner: Waiting for map tasks > 11/03/04 12:44:10 INFO mapred.LocalJobRunner: Starting task: > attempt_local_0001_m_000000_0 > 11/03/04 12:44:10 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 11/03/04 12:44:10 INFO compress.CodecPool: Got brand-new decompressor > 11/03/04 12:44:10 INFO mapred.MapTask: numReduceTasks: 1 > 11/03/04 12:44:10 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) > 11/03/04 12:44:10 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 > 11/03/04 12:44:10 INFO mapred.MapTask: soft limit at 83886080 > 11/03/04 12:44:10 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600 > 11/03/04 12:44:10 INFO mapred.MapTask: kvstart = 26214396; length = 6553600 > 11/03/04 12:44:11 INFO mapreduce.Job: map 0% reduce 0% > 11/03/04 12:44:16 INFO mapred.LocalJobRunner: > file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > > map > 11/03/04 12:44:17 INFO mapreduce.Job: map 57% reduce 0% > 11/03/04 12:44:18 INFO mapred.LocalJobRunner: > file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > > map > 11/03/04 12:44:18 INFO mapred.MapTask: Starting flush of map output > 11/03/04 12:44:18 INFO mapred.MapTask: Spilling map output > 11/03/04 12:44:18 INFO mapred.MapTask: bufstart = 0; bufend = 1454; bufvoid = > 104857600 > 11/03/04 12:44:18 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = > 26214324(104857296); length = 73/6553600 > 11/03/04 12:44:18 INFO mapred.MapTask: Finished spill 0 > 11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is > done. And is in the process of commiting > 11/03/04 12:44:18 INFO mapred.LocalJobRunner: > file:/home/ngc/eclipse_workspace/HadoopPrograms/Images2/JPGSequenceFile.001:0+411569 > > sort > 11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done. > 11/03/04 12:44:18 INFO mapred.LocalJobRunner: Finishing task: > attempt_local_0001_m_000000_0 > 11/03/04 12:44:18 INFO mapred.LocalJobRunner: Map task executor complete. > 11/03/04 12:44:18 INFO mapred.Merger: Merging 1 sorted segments > 11/03/04 12:44:18 INFO mapred.Merger: Down to the last merge-pass, with 1 > segments left of total size: 1481 bytes > 11/03/04 12:44:18 INFO mapred.LocalJobRunner: > 11/03/04 12:44:18 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is > done. And is in the process of commiting > 11/03/04 12:44:18 INFO mapred.LocalJobRunner: > 11/03/04 12:44:18 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is > allowed to commit now > 11/03/04 12:44:18 INFO mapred.FileOutputCommitter: Saved output of task > 'attempt_local_0001_r_000000_0' to > file:/home/ngc/eclipse_workspace/HadoopPrograms/FaceOutput > 11/03/04 12:44:18 INFO mapred.LocalJobRunner: reduce > sort > 11/03/04 12:44:18 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done. > 11/03/04 12:44:18 INFO mapreduce.Job: map 100% reduce 100% > 11/03/04 12:44:18 INFO mapreduce.Job: Job complete: job_local_0001 > 11/03/04 12:44:18 INFO mapreduce.Job: Counters: 18 > FileInputFormatCounters > BYTES_READ=411439 > FileSystemCounters > FILE_BYTES_READ=825005 > FILE_BYTES_WRITTEN=127557 > Map-Reduce Framework > Combine input records=0 > Combine output records=0 > Failed Shuffles=0 > GC time elapsed (ms)=88 > Map input records=20 > Map output bytes=1454 > Map output records=19 > Merged Map outputs=0 > Reduce input groups=19 > Reduce input records=19 > Reduce output records=19 > Reduce shuffle bytes=0 > Shuffled Maps =0 > Spilled Records=38 > SPLIT_RAW_BYTES=127 >>>>> 0.036993828 img_9619.jpg 2 found at [ 35, 201, 37 ], [ 84, 41, 68 >>>>> ], > ... >>>>> 0.41283935 img_538.jpg 3 found at [ 265, 44, 80 ], [ 132, 32, 101 ], [ >>>>> 210, 119, 228 ], >>>>> Job Finished in 8.679 seconds > > 2. RUNNING THE SAME PROGRAM IN HADOOP'S DISTRIBUTED MODE - FAILURE > > ngc@hadoop1:~$ cd hadoop-0.21.0/ > ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop fs -ls Imag* > 11/03/04 12:58:18 INFO security.Groups: Group mapping > impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; > cacheTimeout=300000 > 11/03/04 12:58:18 WARN conf.Configuration: > mapred.task.id<http://mapred.task.id> is deprecated. Instead, use > mapreduce.task.attempt.id<http://mapreduce.task.attempt.id> > Found 1 items > -rw-r--r-- 2 ngc supergroup 411569 2011-03-02 13:21 > /user/ngc/Images2/JPGSequenceFile.001 > ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar > progs/HannahFace >>>>> Running Face Program > 11/03/04 12:59:39 INFO security.Groups: Group mapping > impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; > cacheTimeout=300000 > 11/03/04 12:59:40 WARN conf.Configuration: > mapred.task.id<http://mapred.task.id> is deprecated. Instead, use > mapreduce.task.attempt.id<http://mapreduce.task.attempt.id> > 11/03/04 12:59:40 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the same. > 11/03/04 12:59:40 INFO mapred.FileInputFormat: Total input paths to process : > 1 > 11/03/04 12:59:40 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > 11/03/04 12:59:40 INFO mapreduce.JobSubmitter: number of splits:100 > 11/03/04 12:59:40 INFO mapreduce.JobSubmitter: adding the following > namenodes' delegation tokens:null > 11/03/04 12:59:41 INFO mapreduce.Job: Running job: job_201103021428_0051 > 11/03/04 12:59:42 INFO mapreduce.Job: map 0% reduce 0% > 11/03/04 12:59:49 INFO mapreduce.Job: map 7% reduce 0% > 11/03/04 12:59:51 INFO mapreduce.Job: map 10% reduce 0% > 11/03/04 13:00:04 INFO mapreduce.Job: map 12% reduce 0% > 11/03/04 13:00:05 INFO mapreduce.Job: map 16% reduce 0% > 11/03/04 13:00:06 INFO mapreduce.Job: map 28% reduce 0% > 11/03/04 13:00:07 INFO mapreduce.Job: map 37% reduce 0% > 11/03/04 13:00:07 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000016_0, Status : FAILED > Error: > /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000016_0/work/tmp/libjniopencv_highgui9051044227445275266.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:00:08 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000044_0, Status : FAILED > Error: > /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000044_0/work/tmp/libjniopencv_highgui6446098204420446112.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:00:09 INFO mapreduce.Job: map 47% reduce 0% > 11/03/04 13:00:09 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000048_0, Status : FAILED > Error: > /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: > > /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_0/work/tmp/libjniopencv_objdetect3671939282732993726.so: > failed to map segment from shared object: Cannot allocate memory > 11/03/04 13:00:09 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000052_0, Status : FAILED > Error: > /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000052_0/work/tmp/libjniopencv_highgui1579426900682939358.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:00:10 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000001_0, Status : FAILED > Error: > /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: > > /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000001_0/work/tmp/libjniopencv_objdetect3457632677367330581.so: > failed to map segment from shared object: Cannot allocate memory > attempt_201103021428_0051_m_000001_0: Java HotSpot(TM) 64-Bit Server VM > warning: Exception java.lang.OutOfMemoryError occurred dispatching signal > SIGTERM to handler- the VM may need to be forcibly terminated > 11/03/04 13:00:11 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000022_0, Status : FAILED > 11/03/04 13:00:12 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000026_0, Status : FAILED > 11/03/04 13:00:13 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000006_0, Status : FAILED > 11/03/04 13:00:14 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000011_0, Status : FAILED > 11/03/04 13:00:16 INFO mapreduce.Job: map 48% reduce 0% > 11/03/04 13:00:17 INFO mapreduce.Job: map 57% reduce 0% > 11/03/04 13:00:18 INFO mapreduce.Job: map 59% reduce 0% > 11/03/04 13:00:18 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000061_0, Status : FAILED > Error: > /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000061_0/work/tmp/libjniopencv_highgui3743962684984755257.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:00:19 INFO mapreduce.Job: map 68% reduce 0% > 11/03/04 13:00:19 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000034_0, Status : FAILED > 11/03/04 13:00:20 INFO mapreduce.Job: map 68% reduce 15% > 11/03/04 13:00:20 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000039_0, Status : FAILED > 11/03/04 13:00:21 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000076_0, Status : FAILED > Error: > /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000076_0/work/tmp/libjniopencv_highgui3438076786756619584.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:00:22 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000057_0, Status : FAILED > 11/03/04 13:00:23 INFO mapreduce.Job: map 68% reduce 23% > 11/03/04 13:00:23 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000065_0, Status : FAILED > 11/03/04 13:00:24 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000069_0, Status : FAILED > 11/03/04 13:00:25 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000082_0, Status : FAILED > 11/03/04 13:00:36 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000082_1, Status : FAILED > Error: > /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000082_1/work/tmp/libjniopencv_highgui7180733690064994995.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:00:39 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000011_1, Status : FAILED > Error: > /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_1/work/tmp/libjniopencv_highgui8978195121954363506.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:00:41 INFO mapreduce.Job: map 73% reduce 23% > 11/03/04 13:00:42 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000069_1, Status : FAILED > 11/03/04 13:00:43 INFO mapreduce.Job: map 73% reduce 24% > 11/03/04 13:00:43 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000048_1, Status : FAILED > Error: > /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000048_1/work/tmp/libjniopencv_highgui7269142293373011624.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:00:44 INFO mapreduce.Job: map 74% reduce 24% > 11/03/04 13:00:46 INFO mapreduce.Job: map 74% reduce 25% > 11/03/04 13:00:46 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000001_1, Status : FAILED > 11/03/04 13:00:47 INFO mapreduce.Job: map 75% reduce 25% > 11/03/04 13:00:48 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000052_1, Status : FAILED > 11/03/04 13:00:49 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000044_1, Status : FAILED > 11/03/04 13:00:49 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000087_0, Status : FAILED > 11/03/04 13:00:49 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000011_2, Status : FAILED > Error: > /tmp/hadoop-ngc/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000011_2/work/tmp/libjniopencv_highgui6941559715123481707.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:00:50 INFO mapreduce.Job: map 79% reduce 25% > 11/03/04 13:00:51 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000006_1, Status : FAILED > Error: > /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000006_1/work/tmp/libjniopencv_highgui72992299570368055.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:00:52 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000026_1, Status : FAILED > 11/03/04 13:00:54 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000022_1, Status : FAILED > 11/03/04 13:00:55 INFO mapreduce.Job: map 79% reduce 26% > 11/03/04 13:00:55 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000016_1, Status : FAILED > 11/03/04 13:00:57 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000091_0, Status : FAILED > 11/03/04 13:00:58 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000096_0, Status : FAILED > 11/03/04 13:00:58 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000034_1, Status : FAILED > Error: > /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000034_1/work/tmp/libjniopencv_highgui3721618225395918920.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:01:01 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000065_1, Status : FAILED > 11/03/04 13:01:03 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000057_1, Status : FAILED > 11/03/04 13:01:04 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000076_1, Status : FAILED > 11/03/04 13:01:06 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000039_1, Status : FAILED > 11/03/04 13:01:07 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000061_1, Status : FAILED > 11/03/04 13:01:09 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000069_2, Status : FAILED > Error: > /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0051/attempt_201103021428_0051_m_000069_2/work/tmp/libjniopencv_highgui8910946496817753039.so: > libopencv_highgui.so.2.2: cannot open shared object file: No such file or > directory > 11/03/04 13:01:10 INFO mapreduce.Job: Task Id : > attempt_201103021428_0051_m_000082_2, Status : FAILED > 11/03/04 13:01:15 INFO mapreduce.Job: Job complete: job_201103021428_0051 > 11/03/04 13:01:15 INFO mapreduce.Job: Counters: 21 > FileInputFormatCounters > BYTES_READ=0 > FileSystemCounters > FILE_BYTES_WRITTEN=3040 > HDFS_BYTES_READ=1048281 > Job Counters > Data-local map tasks=51 > Total time spent by all maps waiting after reserving slots > (ms)=0 > Total time spent by all reduces waiting after reserving slots > (ms)=0 > Failed map tasks=1 > Rack-local map tasks=86 > SLOTS_MILLIS_MAPS=1091359 > SLOTS_MILLIS_REDUCES=0 > Launched map tasks=137 > Launched reduce tasks=2 > Map-Reduce Framework > Combine input records=0 > Failed Shuffles=0 > GC time elapsed (ms)=0 > Map input records=0 > Map output bytes=0 > Map output records=0 > Merged Map outputs=0 > Spilled Records=0 > SPLIT_RAW_BYTES=8960 > Exception in thread "main" java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:782) > at progs.HannahFace.run(HannahFace.java:137) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at progs.HannahFace.main(HannahFace.java:162) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:192) > > 3. SAME COMMAND BUT AFTER I MODIFIED MAPRED-SITE.XML - FAILURE > > ngc@hadoop1:~/hadoop-0.21.0$ bin/hadoop jar ../eclipse/HannahFace.jar > progs/HannahFace >>>>> Running Face Program > 11/03/04 15:07:11 INFO security.Groups: Group mapping > impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; > cacheTimeout=300000 > 11/03/04 15:07:11 WARN conf.Configuration: > mapred.task.id<http://mapred.task.id> is deprecated. Instead, use > mapreduce.task.attempt.id<http://mapreduce.task.attempt.id> > 11/03/04 15:07:11 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the same. > 11/03/04 15:07:12 INFO mapred.FileInputFormat: Total input paths to process : > 1 > 11/03/04 15:07:12 WARN conf.Configuration: mapred.map.tasks is deprecated. > Instead, use mapreduce.job.maps > 11/03/04 15:07:12 INFO mapreduce.JobSubmitter: number of splits:100 > 11/03/04 15:07:12 INFO mapreduce.JobSubmitter: adding the following > namenodes' delegation tokens:null > 11/03/04 15:07:12 INFO mapreduce.Job: Running job: job_201103021428_0069 > 11/03/04 15:07:13 INFO mapreduce.Job: map 0% reduce 0% > 11/03/04 15:07:20 INFO mapreduce.Job: map 6% reduce 0% > 11/03/04 15:07:21 INFO mapreduce.Job: map 10% reduce 0% > 11/03/04 15:07:36 INFO mapreduce.Job: map 18% reduce 0% > 11/03/04 15:07:38 INFO mapreduce.Job: map 28% reduce 0% > 11/03/04 15:07:38 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000016_0, Status : FAILED > Error: > /media/hd4/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000016_0/work/tmp/libjniopencv_highgui4138482228584845301.so: > libxcb.so.1: failed to map segment from shared object: Cannot allocate memory > 11/03/04 15:07:39 INFO mapreduce.Job: map 35% reduce 0% > 11/03/04 15:07:40 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000001_0, Status : FAILED > Error: > /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000001_0/work/tmp/libjniopencv_highgui2385564746644347958.so: > libXau.so.6: failed to map segment from shared object: Cannot allocate memory > 11/03/04 15:07:42 INFO mapreduce.Job: map 39% reduce 0% > 11/03/04 15:07:42 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000022_0, Status : FAILED > 11/03/04 15:07:44 INFO mapreduce.Job: map 50% reduce 0% > 11/03/04 15:07:44 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000026_0, Status : FAILED > 11/03/04 15:07:45 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000011_0, Status : FAILED > 11/03/04 15:07:46 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000034_0, Status : FAILED > 11/03/04 15:07:47 INFO mapreduce.Job: map 50% reduce 13% > 11/03/04 15:07:47 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000039_0, Status : FAILED > 11/03/04 15:07:48 INFO mapreduce.Job: map 59% reduce 13% > 11/03/04 15:07:48 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000082_0, Status : FAILED > Error: > /media/hd2/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000082_0/work/tmp/libjniopencv_highgui2586824844718343743.so: > libxcb-render.so.0: failed to map segment from shared object: Cannot > allocate memory > 11/03/04 15:07:50 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000006_0, Status : FAILED > 11/03/04 15:07:51 INFO mapreduce.Job: map 67% reduce 13% > 11/03/04 15:07:51 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000044_0, Status : FAILED > 11/03/04 15:07:51 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000048_0, Status : FAILED > 11/03/04 15:07:53 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000052_0, Status : FAILED > 11/03/04 15:07:53 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000076_0, Status : FAILED > Error: > /media/hd3/mapred/local/taskTracker/ngc/jobcache/job_201103021428_0069/attempt_201103021428_0069_m_000076_0/work/tmp/libjniopencv_highgui6607923753832414434.so: > libfontconfig.so.1: failed to map segment from shared object: Cannot > allocate memory > 11/03/04 15:07:54 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000057_0, Status : FAILED > 11/03/04 15:07:56 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000061_0, Status : FAILED > 11/03/04 15:07:57 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000065_0, Status : FAILED > 11/03/04 15:07:59 INFO mapreduce.Job: Task Id : > attempt_201103021428_0069_m_000069_0, Status : FAILED > attempt_201103021428_0069_m_000069_0: Java HotSpot(TM) 64-Bit Server VM > warning: Exception java.lang.OutOfMemoryError occurred dispatching signal > SIGTERM to handler- the VM may need to be forcibly terminated > > > >
smime.p7s
Description: S/MIME cryptographic signature
