Hi,
I am trying to iterate over 4.5m rows with about 1000 regions and an
average row size of at least .5MB. Here is the end screen after the map run:
Hadoop
job_200801101958_0001 on
lv1-xen-pdc-2
User:
root
Job Name: building
index
Job File:
/tmp/hadoop-root/mapred/system/job_200801101958_0001/job.xml
Status:
Running
Started at: Thu Jan 10 20:05:11 PST
2008
Running for: 59mins,
8sec
Black-listed TaskTrackers:
1
________________________________________________________________________________________________________________________________________________________
Kind % Complete Num Tasks Pending Running Complete Killed
Failed/Killed
Task
Attempts
map
100.00%
995 0 0 995 0 7
/
36
reduce
1.57%
1 0 1 0 0 1
/
0
Counter Map
Reduce
Total
Job Counters Launched map tasks 0
0
1,038
Launched reduce tasks 0
0
2
Map-Reduce Framework Map input records 4,454,969
0
4,454,969
Map output records 4,454,969
0
4,454,969
Map input bytes 0
0
0
Map output bytes 246,332,525,224 0
246,332,525,224
Combine input records 0
0
0
Combine output records 0
0
0
Reduce input groups 0
0
0
Reduce input records 0
0
0
Reduce output records 0
0
0
________________________________________________________________________________________________________________________________________________________
Map Completion Graph -
close
[EMBED]
________________________________________________________________________________________________________________________________________________________
Reduce Completion Graph -
close
[EMBED]
__________________________________________________________
Then it sort of ends. Here are the errors I got on the name node:
# /usr/local/hadoop/bin/hadoop jar /webapps/worldlingo/maindex.jar
com.worldlingo.hadoop.mapred.index.BuildSearchIndex -m 1024 -r 1
-indexDir /pdcindex
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: 2472756350024972361
at
org.apache.hadoop.hbase.HRegionServer.close(HRegionServer.java:1494)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:401)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at
org.apache.hadoop.hbase.HTable$ClientScanner.close(HTable.java:1099)
at
com.worldlingo.hadoop.mapred.index.TableInputFormat$TableRecordReader.close(TableInputFormat.java:65)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:155)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
any valid local directory for task_200801101958_0001_m_000345_0/spill0.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:88)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(MapTask.java:405)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:387)
at com.worldlingo.hadoop.mapred.index.TableMap.map(TableMap.java:54)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
any valid local directory for task_200801101958_0001_m_000358_0/spill0.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:88)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(MapTask.java:405)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:387)
at com.worldlingo.hadoop.mapred.index.TableMap.map(TableMap.java:54)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
any valid local directory for task_200801101958_0001_m_000373_0/spill0.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:88)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(MapTask.java:405)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:387)
at com.worldlingo.hadoop.mapred.index.TableMap.map(TableMap.java:54)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
Error initializing task_200801101958_0001_m_000390_0:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
any valid local directory for
taskTracker/jobcache/job_200801101958_0001/job.xml
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:635)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1264)
at
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:920)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1300)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2163)
08/01/10 20:26:49 WARN mapred.JobClient: Error reading task
outputhttp://lv1-xen-pdc-60.worldlingo.com:50060/tasklog?plaintext=true&taskid=task_200801101958_0001_m_000390_0&filter=stdout
08/01/10 20:26:49 WARN mapred.JobClient: Error reading task
outputhttp://lv1-xen-pdc-60.worldlingo.com:50060/tasklog?plaintext=true&taskid=task_200801101958_0001_m_000390_0&filter=stderr
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
any valid local directory for task_200801101958_0001_m_000371_0/spill3.out
at
org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:313)
at
org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
at
org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:88)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(MapTask.java:405)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:627)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:209)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)
Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out.
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: 3314387006816878921
at
org.apache.hadoop.hbase.HRegionServer.close(HRegionServer.java:1494)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:401)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at
org.apache.hadoop.hbase.HTable$ClientScanner.close(HTable.java:1099)
at
com.worldlingo.hadoop.mapred.index.TableInputFormat$TableRecordReader.close(TableInputFormat.java:65)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:155)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2043)
I am an absolute noobie at all of this. Could anyone help me understand
what went wrong, and what I have to do to get around it?
Thanks,
Lars
---
Lars George, CTO
WorldLingo
