This is a multiplication test of 2.5 thousand by 2.5 thousand two
matrices on pretty large cluster of hbase. The Multiplication requires
(n + 1) table full scan irrespective of the number of mapper. And Each
map processor requires O(n2) for the communication and O(n3/mappers)
the computation.
I couldn't fix UnknownScannerException yet. How can i debug it?
----
[EMAIL PROTECTED] hama]$ bin/hadoop jar ./lib/hama-0.0.1-dev.jar multiply
2500 2500 100
Matrix Space initializing on Hbase.... Please wait.
Create 250 by 250 Random Matrix A
Matrix Space initializing on Hbase.... Please wait.
Create 250 by 250 Random Matrix B
Matrix Space initializing on Hbase.... Please wait.
A * B multiplication start.
08/02/20 09:46:38 INFO mapred.JobClient: Running job: job_200802181655_0017
08/02/20 09:46:39 INFO mapred.JobClient: map 0% reduce 0%
08/02/20 09:46:43 INFO mapred.JobClient: map 3% reduce 0%
08/02/20 09:46:54 INFO mapred.JobClient: map 3% reduce 1%
08/02/20 09:47:28 INFO mapred.JobClient: map 7% reduce 1%
08/02/20 09:47:29 INFO mapred.JobClient: map 11% reduce 1%
08/02/20 09:47:32 INFO mapred.JobClient: map 20% reduce 1%
08/02/20 09:47:33 INFO mapred.JobClient: map 24% reduce 1%
08/02/20 09:47:35 INFO mapred.JobClient: map 51% reduce 1%
08/02/20 09:47:36 INFO mapred.JobClient: map 68% reduce 1%
08/02/20 09:47:37 INFO mapred.JobClient: map 72% reduce 1%
08/02/20 09:47:39 INFO mapred.JobClient: map 72% reduce 7%
08/02/20 09:47:49 INFO mapred.JobClient: map 72% reduce 18%
08/02/20 09:47:50 INFO mapred.JobClient: map 76% reduce 18%
08/02/20 09:47:51 INFO mapred.JobClient: map 80% reduce 18%
08/02/20 09:47:54 INFO mapred.JobClient: map 84% reduce 23%
08/02/20 09:47:59 INFO mapred.JobClient: Task Id :
task_200802181655_0017_m_000000_0, Status : FAILED
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: 3386299785052400805
at org.apache.hadoop.hbase.HRegionServer.close(HRegionServer.java:1543)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HTable$ClientScanner.close(HTable.java:1169)
at
org.apache.hadoop.hama.mapred.MatrixInputFormat$TableRecordReader.close(Unknown
Source)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:155)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2065)
08/02/20 09:48:00 INFO mapred.JobClient: Task Id :
task_200802181655_0017_m_000001_0, Status : FAILED
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: 5742520330160923642
at org.apache.hadoop.hbase.HRegionServer.close(HRegionServer.java:1543)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HTable$ClientScanner.close(HTable.java:1169)
at
org.apache.hadoop.hama.mapred.MatrixInputFormat$TableRecordReader.close(Unknown
Source)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:155)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2065)
08/02/20 09:48:04 INFO mapred.JobClient: map 84% reduce 28%
08/02/20 09:48:59 INFO mapred.JobClient: map 88% reduce 28%
08/02/20 09:49:03 INFO mapred.JobClient: map 92% reduce 28%
08/02/20 09:49:09 INFO mapred.JobClient: map 92% reduce 30%
08/02/20 10:03:21 INFO mapred.JobClient: Task Id :
task_200802181655_0017_m_000002_0, Status : FAILED
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: -7494576223555737321
at org.apache.hadoop.hbase.HRegionServer.close(HRegionServer.java:1543)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HTable$ClientScanner.close(HTable.java:1169)
at
org.apache.hadoop.hama.mapred.MatrixInputFormat$TableRecordReader.close(Unknown
Source)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:155)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2065)
08/02/20 10:04:10 INFO mapred.JobClient: Task Id :
task_200802181655_0017_m_000001_1, Status : FAILED
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: -6171818892366286315
at org.apache.hadoop.hbase.HRegionServer.close(HRegionServer.java:1543)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HTable$ClientScanner.close(HTable.java:1169)
at
org.apache.hadoop.hama.mapred.MatrixInputFormat$TableRecordReader.close(Unknown
Source)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:155)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2065)
08/02/20 10:04:28 INFO mapred.JobClient: Task Id :
task_200802181655_0017_m_000001_2, Status : FAILED
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: 8467827222433201540
at org.apache.hadoop.hbase.HRegionServer.close(HRegionServer.java:1543)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HTable$ClientScanner.close(HTable.java:1169)
at
org.apache.hadoop.hama.mapred.MatrixInputFormat$TableRecordReader.close(Unknown
Source)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:155)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2065)
08/02/20 10:04:41 INFO mapred.JobClient: Task Id :
task_200802181655_0017_m_000002_1, Status : FAILED
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: 5341583878020310354
at org.apache.hadoop.hbase.HRegionServer.close(HRegionServer.java:1543)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HTable$ClientScanner.close(HTable.java:1169)
at
org.apache.hadoop.hama.mapred.MatrixInputFormat$TableRecordReader.close(Unknown
Source)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:155)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2065)
08/02/20 10:19:40 INFO mapred.JobClient: Task Id :
task_200802181655_0017_m_000002_2, Status : FAILED
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: -1065322572792289707
at org.apache.hadoop.hbase.HRegionServer.close(HRegionServer.java:1543)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HTable$ClientScanner.close(HTable.java:1169)
at
org.apache.hadoop.hama.mapred.MatrixInputFormat$TableRecordReader.close(Unknown
Source)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:155)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2065)
08/02/20 10:20:18 INFO mapred.JobClient: map 100% reduce 100%
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:894)
at org.apache.hadoop.hama.MatrixMultiplication.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.hadoop.hama.ExampleDriver.main(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
----
JobTracker logs :
2008-02-20 10:04:09,914 INFO org.apache.hadoop.mapred.JobTracker:
Removed completed task 'task_200802181655_0017_m_000001_1' from
'tracker_a41003:localhost/127.0.0.1:35439'
2008-02-20 10:04:10,201 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing speculative task tip_200802181655_0017_m_000001
2008-02-20 10:04:10,202 INFO org.apache.hadoop.mapred.JobTracker:
Adding task 'task_200802181655_0017_m_000001_3' to tip
tip_200802181655_0017_m_000001, for tracker
'tracker_a41005:localhost/127.0.0.1:32892'
2008-02-20 10:04:26,691 INFO org.apache.hadoop.mapred.TaskInProgress:
Error from task_200802181655_0017_m_000001_2:
org.apache.hadoop.hbase.UnknownScannerException:
org.apache.hadoop.hbase.UnknownScannerException: 8467827222433201540
at org.apache.hadoop.hbase.HRegionServer.close(HRegionServer.java:1543)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HTable$ClientScanner.close(HTable.java:1169)
at
org.apache.hadoop.hama.mapred.MatrixInputFormat$TableRecordReader.close(Unknown
Source)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:155)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:212)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2065)
2008-02-20 10:04:27,708 INFO org.apache.hadoop.mapred.JobTracker:
Removed completed task 'task_200802181655_0017_m_000001_2' from
'tracker_a41007:localhost/127.0.0.1:43259'
2008-02-20 10:04:28,832 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing speculative task tip_200802181655_0017_m_000001
2008-02-20 10:04:28,832 INFO org.apache.hadoop.mapred.JobTracker:
Adding task 'task_200802181655_0017_m_000001_4' to tip
tip_200802181655_0017_m_000001, for tracker
'tracker_a41009:localhost/127.0.0.1:33534'
--
B. Regards,
Edward yoon @ NHN, corp.
