Hi, Andrew >> java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23)
Eh, It seems an un-expected exception. Let me check it out. >> Another question is about lack of "breadth first search" sample, >> is this a sub-project of next release? Using M/R iterations, you can simply implement it -- http://blog.udanax.org/2009/02/breadth-first-search-mapreduce.html But, an M/R iterative job is *awfully* slow, so I think it's not worth to implement. Instead, we trying to implement a BSP-based graph computing framework to improve the performance of graph algorithms on Hadoop. On Tue, Feb 16, 2010 at 10:41 PM, Andrew Milkowski <[email protected]> wrote: > Hi there > > Have following working setup: > > hadoop 0.20.1 > hbase 0.20.3 > hama latest SVN trunk version > > Notes: > > had to replace and rebuild hama with new hbase lib (upgraded to version > 0.20.3) > > 1. successfully rand hama example to generate random matrix > > $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixA > > Note: verified hbase table was generated (via shell list and scan) > > 2. moved on to other examples, generated 2 random matrices: > > $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixA > $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixB > > then an attempt at distributed addition: > > $HAMA_HOME/bin/hama examples add1 matrixA matrixB > > > 3. in similar venue tried multi, norms and and similarity (all generated > exceptions) > > Exceptions appear to be hama code related (and not environmentally specific) > > Culprit could be in improper usage of input matrix data to add,multi, norms > and similarity. > > Another question is about lack of "breadth first search" sample, is this a > sub-project of next release? > > thanks > > below attached console stack trace of bash-3.2$ ./run_examples.sh (Note: > ignore INFO zookeeper.ClientCnxn: Exception while closing send thread for > session... exception that is a known "INFO" level exception present in > zookeeper client/server socket code fragment) > > > -------------------------------------------- > > run_examples.sh: > > #!/bin/bash > > export HAMA_JAR="${HAMA_HOME}/hama-0.2.0-dev-examples.jar" > > # cygwin path translation > #if $cygwin; then > # HAMA_CLASSPATH=`cygpath -p -w "$HAMA_CLASSPATH"` > #fi > > $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixA > > $HAMA_HOME/bin/hama examples rand -m 10 -r 10 100 100 30.5% matrixB > > $HAMA_HOME/bin/hama examples add matrixA matrixB > > exit $? > > > > > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:zookeeper.version=3.2.1-808558, built on 08/27/2009 18:48 GMT > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client environment:host.name > =192.168.0.11 > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:java.version=1.6.0_17 > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:java.vendor=Apple Inc. > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:java.class.path=/opt/local/src/hama/current/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/opt/local/src/hama/current/bin/../build/classes:/opt/local/src/hama/current/bin/../build/test:/opt/local/src/hama/current/bin/../build:/opt/local/src/hama/current/bin/../hama-0.2.0-dev-examples.jar:/opt/local/src/hama/current/bin/../hama-0.2.0-dev.jar:/opt/local/src/hama/current/bin/../lib/AgileJSON-2009-03-30.jar:/opt/local/src/hama/current/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/local/src/hama/current/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/local/src/hama/current/bin/../lib/commons-httpclient-3.0.1.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-api-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-math-1.1.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-core.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3.jar:/opt/local/src/hama/current/bin/../lib/jasper-compiler-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/jasper-runtime-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/javacc.jar:/opt/local/src/hama/current/bin/../lib/jetty-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jetty-util-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jruby-complete-1.2.0.jar:/opt/local/src/hama/current/bin/../lib/json.jar:/opt/local/src/hama/current/bin/../lib/junit-3.8.1.jar:/opt/local/src/hama/current/bin/../lib/libthrift-r771587.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.13.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.15.jar:/opt/local/src/hama/current/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/xmlenc-0.52.jar:/opt/local/src/hama/current/bin/../lib/zookeeper-3.2.1.jar:/opt/local/src/hama/current/bin/../lib/jetty-ext/*.jar:/opt/local/src/hama/current/bin/../lib/findbugs/annotations.jar:/opt/local/src/hama/current/bin/../lib/findbugs/ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-analysis-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-commons-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-tree-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-util-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-xml-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/bcel.jar:/opt/local/src/hama/current/bin/../lib/findbugs/dom4j-full.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs-ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugsGUI.jar:/opt/local/src/hama/current/bin/../lib/findbugs/jsr305.jar:/opt/local/src/hama/current/bin/../lib/findbugs/plugin/coreplugin.jar:/opt/local/src/hadoop/current/conf:/opt/local/src/hadoop/current/conf > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:java.io.tmpdir=/var/folders/pM/pMTPzO70Hc4Gd2FB24T-dU+++TM/-Tmp-/ > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:java.compiler=<NA> > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac > OS X > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:os.arch=x86_64 > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:os.version=10.6.2 > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client environment:user.name > =hadoop > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:user.home=/Users/hadoop > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Client > environment:user.dir=/opt/local/hadoop/jobs/hama > 10/02/16 08:28:29 INFO zookeeper.ZooKeeper: Initiating client connection, > connectString=localhost:2181 sessionTimeout=60000 > watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@2efb56b1 > 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset > is false > 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: Attempting connection to server > localhost/127.0.0.1:2181 > 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: Priming connection to > java.nio.channels.SocketChannel[connected > local=/127.0.0.1:58238remote=localhost/ > 127.0.0.1:2181] > 10/02/16 08:28:29 INFO zookeeper.ClientCnxn: Server connection successful > 10/02/16 08:28:30 INFO matrix.AbstractMatrix: Initializing the matrix > storage. > 10/02/16 08:28:32 INFO matrix.AbstractMatrix: Create Matrix > SparseMatrix_randliqns > 10/02/16 08:28:32 INFO matrix.AbstractMatrix: Create the 100 * 100 random > matrix : SparseMatrix_randliqns > Wrote input for Map #0 > Wrote input for Map #1 > Wrote input for Map #2 > Wrote input for Map #3 > Wrote input for Map #4 > Wrote input for Map #5 > Wrote input for Map #6 > Wrote input for Map #7 > Wrote input for Map #8 > Wrote input for Map #9 > 10/02/16 08:28:33 WARN mapred.JobClient: Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the same. > 10/02/16 08:28:33 WARN mapred.JobClient: No job jar file set. User classes > may not be found. See JobConf(Class) or JobConf#setJar(String). > 10/02/16 08:28:33 INFO input.FileInputFormat: Total input paths to process : > 10 > 10/02/16 08:28:34 INFO mapred.JobClient: Running job: job_201002160823_0001 > 10/02/16 08:28:35 INFO mapred.JobClient: map 0% reduce 0% > 10/02/16 08:28:51 INFO mapred.JobClient: map 20% reduce 0% > 10/02/16 08:29:00 INFO mapred.JobClient: map 40% reduce 0% > 10/02/16 08:29:06 INFO mapred.JobClient: map 40% reduce 1% > 10/02/16 08:29:09 INFO mapred.JobClient: map 60% reduce 2% > 10/02/16 08:29:15 INFO mapred.JobClient: map 80% reduce 2% > 10/02/16 08:29:18 INFO mapred.JobClient: map 80% reduce 3% > 10/02/16 08:29:21 INFO mapred.JobClient: map 100% reduce 4% > 10/02/16 08:29:24 INFO mapred.JobClient: map 100% reduce 5% > 10/02/16 08:29:27 INFO mapred.JobClient: map 100% reduce 12% > 10/02/16 08:29:30 INFO mapred.JobClient: map 100% reduce 20% > 10/02/16 08:29:36 INFO mapred.JobClient: map 100% reduce 23% > 10/02/16 08:29:39 INFO mapred.JobClient: map 100% reduce 33% > 10/02/16 08:29:42 INFO mapred.JobClient: map 100% reduce 40% > 10/02/16 08:29:48 INFO mapred.JobClient: map 100% reduce 43% > 10/02/16 08:29:51 INFO mapred.JobClient: map 100% reduce 60% > 10/02/16 08:30:01 INFO mapred.JobClient: map 100% reduce 70% > 10/02/16 08:30:04 INFO mapred.JobClient: map 100% reduce 73% > 10/02/16 08:30:07 INFO mapred.JobClient: map 100% reduce 80% > 10/02/16 08:30:12 INFO mapred.JobClient: map 100% reduce 100% > 10/02/16 08:30:14 INFO mapred.JobClient: Job complete: job_201002160823_0001 > 10/02/16 08:30:14 INFO mapred.JobClient: Counters: 16 > 10/02/16 08:30:14 INFO mapred.JobClient: Job Counters > 10/02/16 08:30:14 INFO mapred.JobClient: Launched reduce tasks=10 > 10/02/16 08:30:14 INFO mapred.JobClient: Launched map tasks=10 > 10/02/16 08:30:14 INFO mapred.JobClient: Data-local map tasks=10 > 10/02/16 08:30:14 INFO mapred.JobClient: FileSystemCounters > 10/02/16 08:30:14 INFO mapred.JobClient: FILE_BYTES_READ=42372 > 10/02/16 08:30:14 INFO mapred.JobClient: HDFS_BYTES_READ=1080 > 10/02/16 08:30:14 INFO mapred.JobClient: FILE_BYTES_WRITTEN=87764 > 10/02/16 08:30:14 INFO mapred.JobClient: Map-Reduce Framework > 10/02/16 08:30:14 INFO mapred.JobClient: Reduce input groups=100 > 10/02/16 08:30:14 INFO mapred.JobClient: Combine output records=0 > 10/02/16 08:30:14 INFO mapred.JobClient: Map input records=10 > 10/02/16 08:30:14 INFO mapred.JobClient: Reduce shuffle bytes=42463 > 10/02/16 08:30:14 INFO mapred.JobClient: Reduce output records=100 > 10/02/16 08:30:14 INFO mapred.JobClient: Spilled Records=200 > 10/02/16 08:30:14 INFO mapred.JobClient: Map output bytes=41912 > 10/02/16 08:30:14 INFO mapred.JobClient: Combine input records=0 > 10/02/16 08:30:14 INFO mapred.JobClient: Map output records=100 > 10/02/16 08:30:14 INFO mapred.JobClient: Reduce input records=100 > 10/02/16 08:30:14 INFO zookeeper.ZooKeeper: Closing session: > 0x126d6f5b8a90003 > 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: Closing ClientCnxn for session: > 0x126d6f5b8a90003 > 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: Exception while closing send > thread for session 0x126d6f5b8a90003 : Read error rc = -1 > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > 10/02/16 08:30:14 WARN zookeeper.ClientCnxn: Ignoring exception during > shutdown input > java.net.SocketException: Socket is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984) > 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: Disconnecting ClientCnxn for > session: 0x126d6f5b8a90003 > 10/02/16 08:30:14 INFO zookeeper.ZooKeeper: Session: 0x126d6f5b8a90003 > closed > 10/02/16 08:30:14 INFO zookeeper.ClientCnxn: EventThread shut down > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:zookeeper.version=3.2.1-808558, built on 08/27/2009 18:48 GMT > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client environment:host.name > =192.168.0.11 > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:java.version=1.6.0_17 > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:java.vendor=Apple Inc. > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:java.class.path=/opt/local/src/hama/current/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/opt/local/src/hama/current/bin/../build/classes:/opt/local/src/hama/current/bin/../build/test:/opt/local/src/hama/current/bin/../build:/opt/local/src/hama/current/bin/../hama-0.2.0-dev-examples.jar:/opt/local/src/hama/current/bin/../hama-0.2.0-dev.jar:/opt/local/src/hama/current/bin/../lib/AgileJSON-2009-03-30.jar:/opt/local/src/hama/current/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/local/src/hama/current/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/local/src/hama/current/bin/../lib/commons-httpclient-3.0.1.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-api-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-math-1.1.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-core.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3.jar:/opt/local/src/hama/current/bin/../lib/jasper-compiler-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/jasper-runtime-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/javacc.jar:/opt/local/src/hama/current/bin/../lib/jetty-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jetty-util-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jruby-complete-1.2.0.jar:/opt/local/src/hama/current/bin/../lib/json.jar:/opt/local/src/hama/current/bin/../lib/junit-3.8.1.jar:/opt/local/src/hama/current/bin/../lib/libthrift-r771587.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.13.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.15.jar:/opt/local/src/hama/current/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/xmlenc-0.52.jar:/opt/local/src/hama/current/bin/../lib/zookeeper-3.2.1.jar:/opt/local/src/hama/current/bin/../lib/jetty-ext/*.jar:/opt/local/src/hama/current/bin/../lib/findbugs/annotations.jar:/opt/local/src/hama/current/bin/../lib/findbugs/ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-analysis-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-commons-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-tree-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-util-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-xml-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/bcel.jar:/opt/local/src/hama/current/bin/../lib/findbugs/dom4j-full.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs-ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugsGUI.jar:/opt/local/src/hama/current/bin/../lib/findbugs/jsr305.jar:/opt/local/src/hama/current/bin/../lib/findbugs/plugin/coreplugin.jar:/opt/local/src/hadoop/current/conf:/opt/local/src/hadoop/current/conf > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:java.io.tmpdir=/var/folders/pM/pMTPzO70Hc4Gd2FB24T-dU+++TM/-Tmp-/ > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:java.compiler=<NA> > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac > OS X > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:os.arch=x86_64 > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:os.version=10.6.2 > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client environment:user.name > =hadoop > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:user.home=/Users/hadoop > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Client > environment:user.dir=/opt/local/hadoop/jobs/hama > 10/02/16 08:30:16 INFO zookeeper.ZooKeeper: Initiating client connection, > connectString=localhost:2181 sessionTimeout=60000 > watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@2efb56b1 > 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset > is false > 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: Attempting connection to server > localhost/fe80:0:0:0:0:0:0:1%1:2181 > 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: Priming connection to > java.nio.channels.SocketChannel[connected local=/fe80:0:0:0:0:0:0:1%1:58457 > remote=localhost/fe80:0:0:0:0:0:0:1%1:2181] > 10/02/16 08:30:16 INFO zookeeper.ClientCnxn: Server connection successful > 10/02/16 08:30:17 INFO matrix.AbstractMatrix: Initializing the matrix > storage. > 10/02/16 08:30:19 INFO matrix.AbstractMatrix: Create Matrix > SparseMatrix_randflbeh > 10/02/16 08:30:19 INFO matrix.AbstractMatrix: Create the 100 * 100 random > matrix : SparseMatrix_randflbeh > Wrote input for Map #0 > Wrote input for Map #1 > Wrote input for Map #2 > Wrote input for Map #3 > Wrote input for Map #4 > Wrote input for Map #5 > Wrote input for Map #6 > Wrote input for Map #7 > Wrote input for Map #8 > Wrote input for Map #9 > 10/02/16 08:30:20 WARN mapred.JobClient: Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the same. > 10/02/16 08:30:20 WARN mapred.JobClient: No job jar file set. User classes > may not be found. See JobConf(Class) or JobConf#setJar(String). > 10/02/16 08:30:20 INFO input.FileInputFormat: Total input paths to process : > 10 > 10/02/16 08:30:21 INFO mapred.JobClient: Running job: job_201002160823_0002 > 10/02/16 08:30:22 INFO mapred.JobClient: map 0% reduce 0% > 10/02/16 08:30:33 INFO mapred.JobClient: map 20% reduce 0% > 10/02/16 08:30:42 INFO mapred.JobClient: map 40% reduce 0% > 10/02/16 08:30:45 INFO mapred.JobClient: map 40% reduce 1% > 10/02/16 08:30:51 INFO mapred.JobClient: map 60% reduce 2% > 10/02/16 08:30:57 INFO mapred.JobClient: map 80% reduce 2% > 10/02/16 08:31:00 INFO mapred.JobClient: map 80% reduce 4% > 10/02/16 08:31:03 INFO mapred.JobClient: map 100% reduce 4% > 10/02/16 08:31:06 INFO mapred.JobClient: map 100% reduce 12% > 10/02/16 08:31:12 INFO mapred.JobClient: map 100% reduce 30% > 10/02/16 08:31:22 INFO mapred.JobClient: map 100% reduce 43% > 10/02/16 08:31:25 INFO mapred.JobClient: map 100% reduce 50% > 10/02/16 08:31:31 INFO mapred.JobClient: map 100% reduce 53% > 10/02/16 08:31:34 INFO mapred.JobClient: map 100% reduce 63% > 10/02/16 08:31:37 INFO mapred.JobClient: map 100% reduce 70% > 10/02/16 08:31:43 INFO mapred.JobClient: map 100% reduce 73% > 10/02/16 08:31:46 INFO mapred.JobClient: map 100% reduce 83% > 10/02/16 08:31:49 INFO mapred.JobClient: map 100% reduce 90% > 10/02/16 08:31:55 INFO mapred.JobClient: map 100% reduce 93% > 10/02/16 08:31:58 INFO mapred.JobClient: map 100% reduce 100% > 10/02/16 08:32:00 INFO mapred.JobClient: Job complete: job_201002160823_0002 > 10/02/16 08:32:00 INFO mapred.JobClient: Counters: 16 > 10/02/16 08:32:00 INFO mapred.JobClient: Job Counters > 10/02/16 08:32:00 INFO mapred.JobClient: Launched reduce tasks=10 > 10/02/16 08:32:00 INFO mapred.JobClient: Launched map tasks=10 > 10/02/16 08:32:00 INFO mapred.JobClient: Data-local map tasks=10 > 10/02/16 08:32:00 INFO mapred.JobClient: FileSystemCounters > 10/02/16 08:32:00 INFO mapred.JobClient: FILE_BYTES_READ=42512 > 10/02/16 08:32:00 INFO mapred.JobClient: HDFS_BYTES_READ=1080 > 10/02/16 08:32:00 INFO mapred.JobClient: FILE_BYTES_WRITTEN=88044 > 10/02/16 08:32:00 INFO mapred.JobClient: Map-Reduce Framework > 10/02/16 08:32:00 INFO mapred.JobClient: Reduce input groups=100 > 10/02/16 08:32:00 INFO mapred.JobClient: Combine output records=0 > 10/02/16 08:32:00 INFO mapred.JobClient: Map input records=10 > 10/02/16 08:32:00 INFO mapred.JobClient: Reduce shuffle bytes=42603 > 10/02/16 08:32:00 INFO mapred.JobClient: Reduce output records=100 > 10/02/16 08:32:00 INFO mapred.JobClient: Spilled Records=200 > 10/02/16 08:32:00 INFO mapred.JobClient: Map output bytes=42052 > 10/02/16 08:32:00 INFO mapred.JobClient: Combine input records=0 > 10/02/16 08:32:00 INFO mapred.JobClient: Map output records=100 > 10/02/16 08:32:00 INFO mapred.JobClient: Reduce input records=100 > 10/02/16 08:32:00 INFO zookeeper.ZooKeeper: Closing session: > 0x126d6f5b8a9000e > 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: Closing ClientCnxn for session: > 0x126d6f5b8a9000e > 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: Exception while closing send > thread for session 0x126d6f5b8a9000e : Read error rc = -1 > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > 10/02/16 08:32:00 WARN zookeeper.ClientCnxn: Ignoring exception during > shutdown input > java.net.SocketException: Socket is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984) > 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: Disconnecting ClientCnxn for > session: 0x126d6f5b8a9000e > 10/02/16 08:32:00 INFO zookeeper.ZooKeeper: Session: 0x126d6f5b8a9000e > closed > 10/02/16 08:32:00 INFO zookeeper.ClientCnxn: EventThread shut down > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:zookeeper.version=3.2.1-808558, built on 08/27/2009 18:48 GMT > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client environment:host.name > =192.168.0.11 > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:java.version=1.6.0_17 > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:java.vendor=Apple Inc. > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:java.class.path=/opt/local/src/hama/current/bin/../conf:/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/lib/tools.jar:/opt/local/src/hama/current/bin/../build/classes:/opt/local/src/hama/current/bin/../build/test:/opt/local/src/hama/current/bin/../build:/opt/local/src/hama/current/bin/../hama-0.2.0-dev-examples.jar:/opt/local/src/hama/current/bin/../hama-0.2.0-dev.jar:/opt/local/src/hama/current/bin/../lib/AgileJSON-2009-03-30.jar:/opt/local/src/hama/current/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/opt/local/src/hama/current/bin/../lib/commons-el-from-jetty-5.1.4.jar:/opt/local/src/hama/current/bin/../lib/commons-httpclient-3.0.1.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-logging-api-1.0.4.jar:/opt/local/src/hama/current/bin/../lib/commons-math-1.1.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-core.jar:/opt/local/src/hama/current/bin/../lib/hadoop-0.20.1-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3-test.jar:/opt/local/src/hama/current/bin/../lib/hbase-0.20.3.jar:/opt/local/src/hama/current/bin/../lib/jasper-compiler-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/jasper-runtime-5.5.12.jar:/opt/local/src/hama/current/bin/../lib/javacc.jar:/opt/local/src/hama/current/bin/../lib/jetty-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jetty-util-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/jruby-complete-1.2.0.jar:/opt/local/src/hama/current/bin/../lib/json.jar:/opt/local/src/hama/current/bin/../lib/junit-3.8.1.jar:/opt/local/src/hama/current/bin/../lib/libthrift-r771587.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.13.jar:/opt/local/src/hama/current/bin/../lib/log4j-1.2.15.jar:/opt/local/src/hama/current/bin/../lib/servlet-api-2.5-6.1.14.jar:/opt/local/src/hama/current/bin/../lib/xmlenc-0.52.jar:/opt/local/src/hama/current/bin/../lib/zookeeper-3.2.1.jar:/opt/local/src/hama/current/bin/../lib/jetty-ext/*.jar:/opt/local/src/hama/current/bin/../lib/findbugs/annotations.jar:/opt/local/src/hama/current/bin/../lib/findbugs/ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-analysis-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-commons-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-tree-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-util-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/asm-xml-3.0.jar:/opt/local/src/hama/current/bin/../lib/findbugs/bcel.jar:/opt/local/src/hama/current/bin/../lib/findbugs/dom4j-full.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs-ant.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugs.jar:/opt/local/src/hama/current/bin/../lib/findbugs/findbugsGUI.jar:/opt/local/src/hama/current/bin/../lib/findbugs/jsr305.jar:/opt/local/src/hama/current/bin/../lib/findbugs/plugin/coreplugin.jar:/opt/local/src/hadoop/current/conf:/opt/local/src/hadoop/current/conf > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:java.io.tmpdir=/var/folders/pM/pMTPzO70Hc4Gd2FB24T-dU+++TM/-Tmp-/ > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:java.compiler=<NA> > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client environment:os.name=Mac > OS X > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:os.arch=x86_64 > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:os.version=10.6.2 > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client environment:user.name > =hadoop > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:user.home=/Users/hadoop > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Client > environment:user.dir=/opt/local/hadoop/jobs/hama > 10/02/16 08:32:02 INFO zookeeper.ZooKeeper: Initiating client connection, > connectString=localhost:2181 sessionTimeout=60000 > watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@76f8968f > 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset > is false > 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: Attempting connection to server > localhost/0:0:0:0:0:0:0:1:2181 > 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: Priming connection to > java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:58679 > remote=localhost/0:0:0:0:0:0:0:1:2181] > 10/02/16 08:32:02 INFO zookeeper.ClientCnxn: Server connection successful > C = A + B > 10/02/16 08:32:05 INFO matrix.AbstractMatrix: Initializing the matrix > storage. > 10/02/16 08:32:07 INFO matrix.AbstractMatrix: Create Matrix > DenseMatrix_randwhqtl > 10/02/16 08:32:07 WARN mapred.JobClient: Use GenericOptionsParser for > parsing the arguments. Applications should implement Tool for the same. > 10/02/16 08:32:07 WARN mapred.JobClient: No job jar file set. User classes > may not be found. See JobConf(Class) or JobConf#setJar(String). > 10/02/16 08:32:08 INFO mapred.JobClient: Running job: job_201002160823_0003 > 10/02/16 08:32:09 INFO mapred.JobClient: map 0% reduce 0% > 10/02/16 08:32:23 INFO mapred.JobClient: Task Id : > attempt_201002160823_0003_m_000000_0, Status : FAILED > java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23) > at > org.apache.hama.matrix.AbstractVector.checkComformantSize(AbstractVector.java:92) > at org.apache.hama.matrix.DenseVector.add(DenseVector.java:150) > at > org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:35) > at > org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:18) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > 10/02/16 08:32:33 INFO mapred.JobClient: Task Id : > attempt_201002160823_0003_m_000000_1, Status : FAILED > java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23) > at > org.apache.hama.matrix.AbstractVector.checkComformantSize(AbstractVector.java:92) > at org.apache.hama.matrix.DenseVector.add(DenseVector.java:150) > at > org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:35) > at > org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:18) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > 10/02/16 08:32:42 INFO mapred.JobClient: Task Id : > attempt_201002160823_0003_m_000000_2, Status : FAILED > java.lang.IndexOutOfBoundsException: v1.size != v2.size (25 != 23) > at > org.apache.hama.matrix.AbstractVector.checkComformantSize(AbstractVector.java:92) > at org.apache.hama.matrix.DenseVector.add(DenseVector.java:150) > at > org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:35) > at > org.apache.hama.matrix.algebra.MatrixAdditionMap.map(MatrixAdditionMap.java:18) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > 10/02/16 08:32:54 INFO mapred.JobClient: Job complete: job_201002160823_0003 > 10/02/16 08:32:54 INFO mapred.JobClient: Counters: 3 > 10/02/16 08:32:54 INFO mapred.JobClient: Job Counters > 10/02/16 08:32:54 INFO mapred.JobClient: Launched map tasks=4 > 10/02/16 08:32:54 INFO mapred.JobClient: Data-local map tasks=4 > 10/02/16 08:32:54 INFO mapred.JobClient: Failed map tasks=1 > 10/02/16 08:32:56 INFO client.HBaseAdmin: Disabled DenseMatrix_randwhqtl > 10/02/16 08:32:56 INFO client.HBaseAdmin: Deleted DenseMatrix_randwhqtl > 10/02/16 08:32:56 INFO zookeeper.ZooKeeper: Closing session: > 0x126d6f5b8a90019 > 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: Closing ClientCnxn for session: > 0x126d6f5b8a90019 > 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: Exception while closing send > thread for session 0x126d6f5b8a90019 : Read error rc = -1 > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > 10/02/16 08:32:56 WARN zookeeper.ClientCnxn: Ignoring exception during > shutdown input > java.net.SocketException: Socket is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984) > 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: Disconnecting ClientCnxn for > session: 0x126d6f5b8a90019 > 10/02/16 08:32:56 INFO zookeeper.ZooKeeper: Session: 0x126d6f5b8a90019 > closed > 10/02/16 08:32:56 INFO zookeeper.ClientCnxn: EventThread shut down > run (rand, rand, add) > -- Best Regards, Edward J. Yoon @ NHN, corp. [email protected] http://blog.udanax.org
