Hmm... Current mult operation algorithm seems not good. On Tue, Nov 4, 2008 at 1:04 PM, Edward J. Yoon <[EMAIL PROTECTED]> wrote: > This is my full log. (5 node test cluster) > > ---- > [EMAIL PROTECTED] ~/hadoop $ bin/hadoop jar hama-0.1.0-dev-examples.jar mult > -m 50 -r 20 200 200 > 08/11/04 12:57:28 INFO hama.AbstractMatrix: Initializing the matrix storage. > 08/11/04 12:57:39 INFO hama.AbstractMatrix: Create Matrix > DenseMatrix_randnxkdh > 08/11/04 12:57:43 INFO hama.AbstractMatrix: Create the 200 * 200 > random matrix : DenseMatrix_randnxkdh > 08/11/04 12:57:43 INFO hama.AbstractMatrix: Initializing the matrix storage. > 08/11/04 12:57:48 INFO hama.AbstractMatrix: Create Matrix > DenseMatrix_randgufzg > 08/11/04 12:57:49 INFO hama.AbstractMatrix: Create the 200 * 200 > random matrix : DenseMatrix_randgufzg > 08/11/04 12:57:49 INFO hama.AbstractMatrix: Initializing the matrix storage. > 08/11/04 12:57:54 INFO hama.AbstractMatrix: Create Matrix > DenseMatrix_randetuuu > 08/11/04 12:57:54 WARN mapred.JobClient: Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the > same. > 08/11/04 12:57:54 WARN mapred.JobClient: No job jar file set. User > classes may not be found. See JobConf(Class) or > JobConf#setJar(String). > 08/11/04 12:57:55 INFO mapred.JobClient: Running job: job_200811041234_0002 > 08/11/04 12:57:56 INFO mapred.JobClient: map 0% reduce 0% > 08/11/04 12:58:03 INFO mapred.JobClient: map 1% reduce 0% > 08/11/04 12:58:10 INFO mapred.JobClient: map 3% reduce 0% > 08/11/04 12:58:12 INFO mapred.JobClient: map 7% reduce 0% > 08/11/04 12:58:14 INFO mapred.JobClient: map 10% reduce 0% > 08/11/04 12:58:19 INFO mapred.JobClient: map 12% reduce 0% > 08/11/04 12:58:21 INFO mapred.JobClient: map 12% reduce 1% > 08/11/04 12:58:25 INFO mapred.JobClient: map 15% reduce 1% > 08/11/04 12:58:26 INFO mapred.JobClient: map 17% reduce 2% > 08/11/04 12:58:28 INFO mapred.JobClient: map 19% reduce 2% > 08/11/04 12:58:33 INFO mapred.JobClient: map 21% reduce 2% > 08/11/04 12:58:36 INFO mapred.JobClient: map 21% reduce 3% > 08/11/04 12:58:38 INFO mapred.JobClient: map 23% reduce 3% > 08/11/04 12:58:40 INFO mapred.JobClient: map 28% reduce 3% > 08/11/04 12:58:41 INFO mapred.JobClient: map 32% reduce 4% > 08/11/04 12:58:46 INFO mapred.JobClient: map 32% reduce 5% > 08/11/04 12:58:49 INFO mapred.JobClient: map 34% reduce 5% > 08/11/04 12:58:51 INFO mapred.JobClient: map 36% reduce 6% > 08/11/04 12:58:53 INFO mapred.JobClient: map 38% reduce 6% > 08/11/04 12:58:54 INFO mapred.JobClient: map 42% reduce 6% > 08/11/04 12:58:55 INFO mapred.JobClient: map 42% reduce 7% > 08/11/04 12:59:03 INFO mapred.JobClient: map 42% reduce 8% > 08/11/04 12:59:05 INFO mapred.JobClient: map 46% reduce 8% > 08/11/04 12:59:06 INFO mapred.JobClient: map 46% reduce 9% > 08/11/04 12:59:07 INFO mapred.JobClient: map 48% reduce 9% > 08/11/04 12:59:08 INFO mapred.JobClient: map 54% reduce 9% > 08/11/04 12:59:12 INFO mapred.JobClient: map 58% reduce 9% > 08/11/04 12:59:16 INFO mapred.JobClient: map 58% reduce 10% > 08/11/04 12:59:18 INFO mapred.JobClient: map 58% reduce 11% > 08/11/04 12:59:21 INFO mapred.JobClient: map 63% reduce 12% > 08/11/04 12:59:23 INFO mapred.JobClient: map 65% reduce 12% > 08/11/04 12:59:26 INFO mapred.JobClient: map 65% reduce 13% > 08/11/04 12:59:27 INFO mapred.JobClient: map 69% reduce 13% > 08/11/04 12:59:28 INFO mapred.JobClient: map 73% reduce 13% > 08/11/04 12:59:31 INFO mapred.JobClient: map 75% reduce 13% > 08/11/04 12:59:33 INFO mapred.JobClient: map 77% reduce 14% > 08/11/04 12:59:34 INFO mapred.JobClient: map 79% reduce 14% > 08/11/04 12:59:35 INFO mapred.JobClient: map 81% reduce 15% > 08/11/04 12:59:38 INFO mapred.JobClient: map 81% reduce 16% > 08/11/04 12:59:42 INFO mapred.JobClient: map 81% reduce 17% > 08/11/04 12:59:43 INFO mapred.JobClient: map 85% reduce 17% > 08/11/04 12:59:44 INFO mapred.JobClient: map 91% reduce 17% > 08/11/04 12:59:45 INFO mapred.JobClient: map 93% reduce 18% > 08/11/04 12:59:54 INFO mapred.JobClient: map 95% reduce 19% > 08/11/04 12:59:57 INFO mapred.JobClient: map 95% reduce 20% > 08/11/04 13:00:00 INFO mapred.JobClient: map 95% reduce 21% > 08/11/04 13:00:59 INFO mapred.JobClient: Task Id : > attempt_200811041234_0002_m_000000_0, Status : FAILED > org.apache.hadoop.hbase.UnknownScannerException: > org.apache.hadoop.hbase.UnknownScannerException: Name: > -4870261170067331403 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1077) > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888) > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at > org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:856) > at > org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1155) > at > org.apache.hama.mapred.MatrixInputFormatBase$TableRecordReader.next(MatrixInputFormatBase.java:174) > at > org.apache.hama.mapred.MatrixInputFormatBase$TableRecordReader.next(MatrixInputFormatBase.java:56) > at > org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:165) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) > at > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) > > 08/11/04 13:01:22 INFO mapred.JobClient: map 97% reduce 21% > 08/11/04 13:01:37 INFO mapred.JobClient: map 97% reduce 22% > 08/11/04 13:02:00 INFO mapred.JobClient: map 100% reduce 22% > 08/11/04 13:02:08 INFO mapred.JobClient: map 100% reduce 28% > 08/11/04 13:02:09 INFO mapred.JobClient: map 100% reduce 32% > 08/11/04 13:02:10 INFO mapred.JobClient: map 100% reduce 33% > 08/11/04 13:02:11 INFO mapred.JobClient: map 100% reduce 37% > 08/11/04 13:02:13 INFO mapred.JobClient: map 100% reduce 45% > 08/11/04 13:02:14 INFO mapred.JobClient: map 100% reduce 47% > 08/11/04 13:02:16 INFO mapred.JobClient: map 100% reduce 50% > 08/11/04 13:02:17 INFO mapred.JobClient: map 100% reduce 63% > 08/11/04 13:02:21 INFO mapred.JobClient: map 100% reduce 65% > 08/11/04 13:02:23 INFO mapred.JobClient: map 100% reduce 66% > 08/11/04 13:02:26 INFO mapred.JobClient: map 100% reduce 67% > 08/11/04 13:02:28 INFO mapred.JobClient: map 100% reduce 68% > 08/11/04 13:02:31 INFO mapred.JobClient: map 100% reduce 69% > 08/11/04 13:02:36 INFO mapred.JobClient: map 100% reduce 70% > 08/11/04 13:02:38 INFO mapred.JobClient: map 100% reduce 71% > 08/11/04 13:02:42 INFO mapred.JobClient: map 100% reduce 72% > 08/11/04 13:02:46 INFO mapred.JobClient: map 100% reduce 74% > 08/11/04 13:02:47 INFO mapred.JobClient: map 100% reduce 75% > 08/11/04 13:02:48 INFO mapred.JobClient: map 100% reduce 79% > 08/11/04 13:02:51 INFO mapred.JobClient: map 100% reduce 86% > 08/11/04 13:02:52 INFO mapred.JobClient: map 100% reduce 89% > 08/11/04 13:02:55 INFO mapred.JobClient: map 100% reduce 93% > 08/11/04 13:02:56 INFO mapred.JobClient: map 100% reduce 97% > 08/11/04 13:02:57 INFO mapred.JobClient: map 100% reduce 98% > 08/11/04 13:02:59 INFO mapred.JobClient: Job complete: job_200811041234_0002 > 08/11/04 13:02:59 INFO mapred.JobClient: Counters: 13 > 08/11/04 13:02:59 INFO mapred.JobClient: File Systems > 08/11/04 13:02:59 INFO mapred.JobClient: Local bytes read=1006933217 > 08/11/04 13:02:59 INFO mapred.JobClient: Local bytes written=1730888348 > 08/11/04 13:02:59 INFO mapred.JobClient: Job Counters > 08/11/04 13:02:59 INFO mapred.JobClient: Launched reduce tasks=28 > 08/11/04 13:02:59 INFO mapred.JobClient: Launched map tasks=58 > 08/11/04 13:02:59 INFO mapred.JobClient: Map-Reduce Framework > 08/11/04 13:02:59 INFO mapred.JobClient: Reduce input groups=200 > 08/11/04 13:02:59 INFO mapred.JobClient: Combine output records=0 > 08/11/04 13:02:59 INFO mapred.JobClient: Map input records=547 > 08/11/04 13:02:59 INFO mapred.JobClient: Reduce output records=200 > 08/11/04 13:02:59 INFO mapred.JobClient: Map output bytes=711260600 > 08/11/04 13:02:59 INFO mapred.JobClient: Map input bytes=0 > 08/11/04 13:02:59 INFO mapred.JobClient: Combine input records=0 > 08/11/04 13:02:59 INFO mapred.JobClient: Map output records=109400 > 08/11/04 13:02:59 INFO mapred.JobClient: Reduce input records=109400 > 08/11/04 13:03:05 INFO client.HBaseAdmin: Disabled DenseMatrix_randnxkdh > 08/11/04 13:03:05 INFO client.HBaseAdmin: Deleted DenseMatrix_randnxkdh > 08/11/04 13:03:10 INFO client.HBaseAdmin: Disabled DenseMatrix_randgufzg > 08/11/04 13:03:10 INFO client.HBaseAdmin: Deleted DenseMatrix_randgufzg > 08/11/04 13:03:16 INFO client.HBaseAdmin: Disabled DenseMatrix_randetuuu > 08/11/04 13:03:17 INFO client.HBaseAdmin: Deleted DenseMatrix_randetuuu > > > On Tue, Nov 4, 2008 at 12:33 PM, Samuel Guo <[EMAIL PROTECTED]> wrote: >> what size of matrix you used to run matrix-matrix mult? >> >> On Tue, Nov 4, 2008 at 11:21 AM, Edward J. Yoon <[EMAIL PROTECTED]>wrote: >> >>> Hmm... Yes, probably. >>> Current mult algorithm needs one full scan by each rows (n). If it >>> replaced to 2D blocking algorithm, it'll be solved. >>> >>> /Ed >>> >>> On Tue, Nov 4, 2008 at 12:07 PM, Samuel Guo <[EMAIL PROTECTED]> wrote: >>> > On Tue, Nov 4, 2008 at 10:30 AM, Edward J. Yoon <[EMAIL PROTECTED] >>> >wrote: >>> > >>> >> While matrix-matrix mult, I received a message as below. Does anyone >>> >> know about UnknownScannerException? >>> >> >>> >> ---- >>> >> 08/11/04 11:24:57 INFO mapred.JobClient: map 90% reduce 28% >>> >> 08/11/04 11:24:59 INFO mapred.JobClient: map 90% reduce 30% >>> >> 08/11/04 11:26:49 INFO mapred.JobClient: Task Id : >>> >> attempt_200811041040_0003_m_000002_0, Status : FAILED >>> >> org.apache.hadoop.hbase.UnknownScannerException: >>> >> org.apache.hadoop.hbase.UnknownScannerException: Name: >>> >> -1151932123796109789 >>> >> at >>> >> >>> org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1077) >>> >> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) >>> >> at >>> >> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>> >> at java.lang.reflect.Method.invoke(Method.java:597) >>> >> at >>> >> org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:554) >>> >> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888) >>> >> >>> >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >>> >> Method) >>> >> at >>> >> >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) >>> >> at >>> >> >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) >>> >> at >>> java.lang.reflect.Constructor.newInstance(Constructor.java:513) >>> >> at >>> >> >>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82) >>> >> at >>> >> >>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:856) >>> > >>> > >>> > I doubt that there may be some region splitting/moving during scanning, >>> so >>> > we get the RemoteException while getRegionServerWithRetires. >>> > >>> > >>> >> >>> >> at >>> >> >>> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1155) >>> >> at >>> >> >>> org.apache.hama.mapred.MatrixInputFormatBase$TableRecordReader.next(MatrixInputFormatBase.java:174) >>> >> at >>> >> >>> org.apache.hama.mapred.MatrixInputFormatBase$TableRecordReader.next(MatrixInputFormatBase.java:56) >>> >> at >>> >> >>> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:165) >>> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:45) >>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227) >>> >> at >>> >> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207) >>> >> >>> >> 08/11/04 11:27:11 INFO mapred.JobClient: map 100% reduce 30% >>> >> 08/11/04 11:27:17 INFO mapred.JobClient: map 100% reduce 64% >>> >> 08/11/04 11:27:23 INFO mapred.JobClient: Job complete: >>> >> job_200811041040_0003 >>> >> 08/11/04 11:27:23 INFO mapred.JobClient: Counters: 13 >>> >> 08/11/04 11:27:23 INFO mapred.JobClient: File Systems >>> >> 08/11/04 11:27:23 INFO mapred.JobClient: Local bytes read=32057196 >>> >> 08/11/04 11:27:23 INFO mapred.JobClient: Local bytes >>> written=64599524 >>> >> >>> >> -- >>> >> Best regards, Edward J. Yoon @ NHN, corp. >>> >> [EMAIL PROTECTED] >>> >> http://blog.udanax.org >>> >> >>> > >>> >>> >>> >>> -- >>> Best regards, Edward J. Yoon @ NHN, corp. >>> [EMAIL PROTECTED] >>> http://blog.udanax.org >>> >> > > > > -- > Best regards, Edward J. Yoon @ NHN, corp. > [EMAIL PROTECTED] > http://blog.udanax.org >
-- Best regards, Edward J. Yoon @ NHN, corp. [EMAIL PROTECTED] http://blog.udanax.org
