it seems that a 200 * 200 matrix is not very huge matrix. maybe the reason is not about matrix spliting/moving.
On Tue, Nov 4, 2008 at 12:04 PM, Edward J. Yoon <[EMAIL PROTECTED]>wrote: > This is my full log. (5 node test cluster) > > ---- > [EMAIL PROTECTED] ~/hadoop $ bin/hadoop jar hama-0.1.0-dev-examples.jar mult > -m 50 -r 20 200 200 > 08/11/04 12:57:28 INFO hama.AbstractMatrix: Initializing the matrix > storage. > 08/11/04 12:57:39 INFO hama.AbstractMatrix: Create Matrix > DenseMatrix_randnxkdh > 08/11/04 12:57:43 INFO hama.AbstractMatrix: Create the 200 * 200 > random matrix : DenseMatrix_randnxkdh > 08/11/04 12:57:43 INFO hama.AbstractMatrix: Initializing the matrix > storage. > 08/11/04 12:57:48 INFO hama.AbstractMatrix: Create Matrix > DenseMatrix_randgufzg > 08/11/04 12:57:49 INFO hama.AbstractMatrix: Create the 200 * 200 > random matrix : DenseMatrix_randgufzg > 08/11/04 12:57:49 INFO hama.AbstractMatrix: Initializing the matrix > storage. > 08/11/04 12:57:54 INFO hama.AbstractMatrix: Create Matrix > DenseMatrix_randetuuu > 08/11/04 12:57:54 WARN mapred.JobClient: Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the > same. > 08/11/04 12:57:54 WARN mapred.JobClient: No job jar file set. User > classes may not be found. See JobConf(Class) or > JobConf#setJar(String). > 08/11/04 12:57:55 INFO mapred.JobClient: Running job: job_200811041234_0002 > 08/11/04 12:57:56 INFO mapred.JobClient: map 0% reduce 0% > 08/11/04 12:58:03 INFO mapred.JobClient: map 1% reduce 0% > 08/11/04 12:58:10 INFO mapred.JobClient: map 3% reduce 0% > 08/11/04 12:58:12 INFO mapred.JobClient: map 7% reduce 0% > 08/11/04 12:58:14 INFO mapred.JobClient: map 10% reduce 0% > 08/11/04 12:58:19 INFO mapred.JobClient: map 12% reduce 0% > 08/11/04 12:58:21 INFO mapred.JobClient: map 12% reduce 1% > 08/11/04 12:58:25 INFO mapred.JobClient: map 15% reduce 1% > 08/11/04 12:58:26 INFO mapred.JobClient: map 17% reduce 2% > 08/11/04 12:58:28 INFO mapred.JobClient: map 19% reduce 2% > 08/11/04 12:58:33 INFO mapred.JobClient: map 21% reduce 2% > 08/11/04 12:58:36 INFO mapred.JobClient: map 21% reduce 3% > 08/11/04 12:58:38 INFO mapred.JobClient: map 23% reduce 3% > 08/11/04 12:58:40 INFO mapred.JobClient: map 28% reduce 3% > 08/11/04 12:58:41 INFO mapred.JobClient: map 32% reduce 4% > 08/11/04 12:58:46 INFO mapred.JobClient: map 32% reduce 5% > 08/11/04 12:58:49 INFO mapred.JobClient: map 34% reduce 5% > 08/11/04 12:58:51 INFO mapred.JobClient: map 36% reduce 6% > 08/11/04 12:58:53 INFO mapred.JobClient: map 38% reduce 6% > 08/11/04 12:58:54 INFO mapred.JobClient: map 42% reduce 6% > 08/11/04 12:58:55 INFO mapred.JobClient: map 42% reduce 7% > 08/11/04 12:59:03 INFO mapred.JobClient: map 42% reduce 8% > 08/11/04 12:59:05 INFO mapred.JobClient: map 46% reduce 8% > 08/11/04 12:59:06 INFO mapred.JobClient: map 46% reduce 9% > 08/11/04 12:59:07 INFO mapred.JobClient: map 48% reduce 9% > 08/11/04 12:59:08 INFO mapred.JobClient: map 54% reduce 9% > 08/11/04 12:59:12 INFO mapred.JobClient: map 58% reduce 9% > 08/11/04 12:59:16 INFO mapred.JobClient: map 58% reduce 10% > 08/11/04 12:59:18 INFO mapred.JobClient: map 58% reduce 11% > 08/11/04 12:59:21 INFO mapred.JobClient: map 63% reduce 12% > 08/11/04 12:59:23 INFO mapred.JobClient: map 65% reduce 12% > 08/11/04 12:59:26 INFO mapred.JobClient: map 65% reduce 13% > 08/11/04 12:59:27 INFO mapred.JobClient: map 69% reduce 13% > 08/11/04 12:59:28 INFO mapred.JobClient: map 73% reduce 13% > 08/11/04 12:59:31 INFO mapred.JobClient: map 75% reduce 13% > 08/11/04 12:59:33 INFO mapred.JobClient: map 77% reduce 14% > 08/11/04 12:59:34 INFO mapred.JobClient: map 79% reduce 14% > 08/11/04 12:59:35 INFO mapred.JobClient: map 81% reduce 15% > 08/11/04 12:59:38 INFO mapred.JobClient: map 81% reduce 16% > 08/11/04 12:59:42 INFO mapred.JobClient: map 81% reduce 17% > 08/11/04 12:59:43 INFO mapred.JobClient: map 85% reduce 17% > 08/11/04 12:59:44 INFO mapred.JobClient: map 91% reduce 17% > 08/11/04 12:59:45 INFO mapred.JobClient: map 93% reduce 18% > 08/11/04 12:59:54 INFO mapred.JobClient: map 95% reduce 19% > 08/11/04 12:59:57 INFO mapred.JobClient: map 95% reduce 20% > 08/11/04 13:00:00 INFO mapred.JobClient: map 95% reduce 21% > 08/11/04 13:00:59 INFO mapred.JobClient: Task Id : > attempt_200811041234_0002_m_000000_0, Status : FAILED > org.apache.hadoop.hbase.UnknownScannerException: > org.apache.hadoop.hbase.UnknownScannerException: Name: > -4870261170067331403 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1077) > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:856) > at > org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1155) > at > org.apache.hama.mapred.MatrixInputFormatBase$TableRecordReader.next(MatrixInputFormatBase.java:174) > at > org.apache.hama.mapred.MatrixInputFormatBase$TableRecordReader.next(MatrixInputFormatBase.java:56) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:165) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) > > 08/11/04 13:01:22 INFO mapred.JobClient: map 97% reduce 21% > 08/11/04 13:01:37 INFO mapred.JobClient: map 97% reduce 22% > 08/11/04 13:02:00 INFO mapred.JobClient: map 100% reduce 22% > 08/11/04 13:02:08 INFO mapred.JobClient: map 100% reduce 28% > 08/11/04 13:02:09 INFO mapred.JobClient: map 100% reduce 32% > 08/11/04 13:02:10 INFO mapred.JobClient: map 100% reduce 33% > 08/11/04 13:02:11 INFO mapred.JobClient: map 100% reduce 37% > 08/11/04 13:02:13 INFO mapred.JobClient: map 100% reduce 45% > 08/11/04 13:02:14 INFO mapred.JobClient: map 100% reduce 47% > 08/11/04 13:02:16 INFO mapred.JobClient: map 100% reduce 50% > 08/11/04 13:02:17 INFO mapred.JobClient: map 100% reduce 63% > 08/11/04 13:02:21 INFO mapred.JobClient: map 100% reduce 65% > 08/11/04 13:02:23 INFO mapred.JobClient: map 100% reduce 66% > 08/11/04 13:02:26 INFO mapred.JobClient: map 100% reduce 67% > 08/11/04 13:02:28 INFO mapred.JobClient: map 100% reduce 68% > 08/11/04 13:02:31 INFO mapred.JobClient: map 100% reduce 69% > 08/11/04 13:02:36 INFO mapred.JobClient: map 100% reduce 70% > 08/11/04 13:02:38 INFO mapred.JobClient: map 100% reduce 71% > 08/11/04 13:02:42 INFO mapred.JobClient: map 100% reduce 72% > 08/11/04 13:02:46 INFO mapred.JobClient: map 100% reduce 74% > 08/11/04 13:02:47 INFO mapred.JobClient: map 100% reduce 75% > 08/11/04 13:02:48 INFO mapred.JobClient: map 100% reduce 79% > 08/11/04 13:02:51 INFO mapred.JobClient: map 100% reduce 86% > 08/11/04 13:02:52 INFO mapred.JobClient: map 100% reduce 89% > 08/11/04 13:02:55 INFO mapred.JobClient: map 100% reduce 93% > 08/11/04 13:02:56 INFO mapred.JobClient: map 100% reduce 97% > 08/11/04 13:02:57 INFO mapred.JobClient: map 100% reduce 98% > 08/11/04 13:02:59 INFO mapred.JobClient: Job complete: > job_200811041234_0002 > 08/11/04 13:02:59 INFO mapred.JobClient: Counters: 13 > 08/11/04 13:02:59 INFO mapred.JobClient: File Systems > 08/11/04 13:02:59 INFO mapred.JobClient: Local bytes read=1006933217 > 08/11/04 13:02:59 INFO mapred.JobClient: Local bytes written=1730888348 > 08/11/04 13:02:59 INFO mapred.JobClient: Job Counters > 08/11/04 13:02:59 INFO mapred.JobClient: Launched reduce tasks=28 > 08/11/04 13:02:59 INFO mapred.JobClient: Launched map tasks=58 > 08/11/04 13:02:59 INFO mapred.JobClient: Map-Reduce Framework > 08/11/04 13:02:59 INFO mapred.JobClient: Reduce input groups=200 > 08/11/04 13:02:59 INFO mapred.JobClient: Combine output records=0 > 08/11/04 13:02:59 INFO mapred.JobClient: Map input records=547 > 08/11/04 13:02:59 INFO mapred.JobClient: Reduce output records=200 > 08/11/04 13:02:59 INFO mapred.JobClient: Map output bytes=711260600 > 08/11/04 13:02:59 INFO mapred.JobClient: Map input bytes=0 > 08/11/04 13:02:59 INFO mapred.JobClient: Combine input records=0 > 08/11/04 13:02:59 INFO mapred.JobClient: Map output records=109400 > 08/11/04 13:02:59 INFO mapred.JobClient: Reduce input records=109400 > 08/11/04 13:03:05 INFO client.HBaseAdmin: Disabled DenseMatrix_randnxkdh > 08/11/04 13:03:05 INFO client.HBaseAdmin: Deleted DenseMatrix_randnxkdh > 08/11/04 13:03:10 INFO client.HBaseAdmin: Disabled DenseMatrix_randgufzg > 08/11/04 13:03:10 INFO client.HBaseAdmin: Deleted DenseMatrix_randgufzg > 08/11/04 13:03:16 INFO client.HBaseAdmin: Disabled DenseMatrix_randetuuu > 08/11/04 13:03:17 INFO client.HBaseAdmin: Deleted DenseMatrix_randetuuu > > > On Tue, Nov 4, 2008 at 12:33 PM, Samuel Guo <[EMAIL PROTECTED]> wrote: > > what size of matrix you used to run matrix-matrix mult? > > > > On Tue, Nov 4, 2008 at 11:21 AM, Edward J. Yoon <[EMAIL PROTECTED] > >wrote: > > > >> Hmm... Yes, probably. > >> Current mult algorithm needs one full scan by each rows (n). If it > >> replaced to 2D blocking algorithm, it'll be solved. > >> > >> /Ed > >> > >> On Tue, Nov 4, 2008 at 12:07 PM, Samuel Guo <[EMAIL PROTECTED]> wrote: > >> > On Tue, Nov 4, 2008 at 10:30 AM, Edward J. Yoon < > [EMAIL PROTECTED] > >> >wrote: > >> > > >> >> While matrix-matrix mult, I received a message as below. Does anyone > >> >> know about UnknownScannerException? > >> >> > >> >> ---- > >> >> 08/11/04 11:24:57 INFO mapred.JobClient: map 90% reduce 28% > >> >> 08/11/04 11:24:59 INFO mapred.JobClient: map 90% reduce 30% > >> >> 08/11/04 11:26:49 INFO mapred.JobClient: Task Id : > >> >> attempt_200811041040_0003_m_000002_0, Status : FAILED > >> >> org.apache.hadoop.hbase.UnknownScannerException: > >> >> org.apache.hadoop.hbase.UnknownScannerException: Name: > >> >> -1151932123796109789 > >> >> at > >> >> > >> > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1077) > >> >> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) > >> >> at > >> >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> >> at java.lang.reflect.Method.invoke(Method.java:597) > >> >> at > >> >> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554) > >> >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888) > >> >> > >> >> at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > >> >> Method) > >> >> at > >> >> > >> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > >> >> at > >> >> > >> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > >> >> at > >> java.lang.reflect.Constructor.newInstance(Constructor.java:513) > >> >> at > >> >> > >> > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82) > >> >> at > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:856) > >> > > >> > > >> > I doubt that there may be some region splitting/moving during > scanning, > >> so > >> > we get the RemoteException while getRegionServerWithRetires. > >> > > >> > > >> >> > >> >> at > >> >> > >> > org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1155) > >> >> at > >> >> > >> > org.apache.hama.mapred.MatrixInputFormatBase$TableRecordReader.next(MatrixInputFormatBase.java:174) > >> >> at > >> >> > >> > org.apache.hama.mapred.MatrixInputFormatBase$TableRecordReader.next(MatrixInputFormatBase.java:56) > >> >> at > >> >> > >> > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:165) > >> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45) > >> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) > >> >> at > >> >> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) > >> >> > >> >> 08/11/04 11:27:11 INFO mapred.JobClient: map 100% reduce 30% > >> >> 08/11/04 11:27:17 INFO mapred.JobClient: map 100% reduce 64% > >> >> 08/11/04 11:27:23 INFO mapred.JobClient: Job complete: > >> >> job_200811041040_0003 > >> >> 08/11/04 11:27:23 INFO mapred.JobClient: Counters: 13 > >> >> 08/11/04 11:27:23 INFO mapred.JobClient: File Systems > >> >> 08/11/04 11:27:23 INFO mapred.JobClient: Local bytes > read=32057196 > >> >> 08/11/04 11:27:23 INFO mapred.JobClient: Local bytes > >> written=64599524 > >> >> > >> >> -- > >> >> Best regards, Edward J. Yoon @ NHN, corp. > >> >> [EMAIL PROTECTED] > >> >> http://blog.udanax.org > >> >> > >> > > >> > >> > >> > >> -- > >> Best regards, Edward J. Yoon @ NHN, corp. > >> [EMAIL PROTECTED] > >> http://blog.udanax.org > >> > > > > > > -- > Best regards, Edward J. Yoon @ NHN, corp. > [EMAIL PROTECTED] > http://blog.udanax.org >
