Francisco, there's a problem that requires huge memory and the number of BSP tasks can't be increased manually in Hama 0.6.0 release.
We will release 0.6.1 soon. Please wait or use the TRUNK version. On Fri, Dec 28, 2012 at 10:11 PM, Francisco Sanches <[email protected]> wrote: > Thanks for the reply Edward J. Yoon. > > I made the changes in the configuration suggested that Edward Yoon > continues with more problems. What could observe is that for smaller > graphs, with 2000 vertices My application works perfectly. But when I put > larger graphs got the following error: > > hduser@mestre:/usr/local/hama$ bin/hama jar aplicmenosmsg.jar 8719 > /hdfs/disco/grafonewc.txt /hdfs/disco/grafoout 36 > 12/12/28 10:00:54 INFO bsp.FileInputFormat: Total input paths to process : 1 > 12/12/28 10:00:54 INFO bsp.FileInputFormat: Total # of splits: 36 > 12/12/28 10:00:57 INFO bsp.BSPJobClient: Running job: job_201212280959_0001 > 12/12/28 10:01:00 INFO bsp.BSPJobClient: Current supersteps number: 0 > 12/12/28 10:01:30 INFO bsp.BSPJobClient: Current supersteps number: 2 > 12/12/28 10:01:33 INFO bsp.BSPJobClient: Current supersteps number: 4 > 12/12/28 10:01:39 INFO bsp.BSPJobClient: Current supersteps number: 5 > 12/12/28 10:02:03 INFO bsp.BSPJobClient: Current supersteps number: 6 > 12/12/28 10:02:09 INFO bsp.BSPJobClient: Current supersteps number: 0 > attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO > zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969, > built on 02/23/2011 22:27 GMT > attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO > zookeeper.ZooKeeper: Client environment:host.name=node01 > attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO > zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 > attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO > zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. > attempt_201212280959_0001_000027_0: 12/12/28 10:01:21 INFO > zookeeper.ZooKeeper: Client > environment:java.home=/usr/lib/jvm/java-6-openjdk-i386/jre > . > . > . > attempt_201212280959_0001_000027_0: 12/12/28 10:01:50 INFO > ipc.NettyTransceiver: [id: 0x0129c445, /10.16.1.201:46065 => node08/ > 10.16.1.208:61001] INTEREST_CHANGED > 12/12/28 10:02:12 INFO bsp.BSPJobClient: Job failed. > > > Has anyone had this problem? Got Resolve? If I increase the number of > cluster nodes is possible to solve this problem? Is this is a limitation of > the Graph API Hama? or my application? > > > 2012/12/27 Edward J. Yoon <[email protected]> > >> You should increase max heap size of child JVMs, instead of HAMA_HEAPSIZE. >> >> Add below property to hama-site.xml. >> >> <property> >> <name>bsp.child.java.opts</name> >> <value>-Xmx2048m</value> >> <description>Java opts for the groom server child processes. >> The following symbol, if present, will be interpolated: @taskid@ >> is replaced >> by current TaskID. Any other occurrences of '@' will go unchanged. >> For example, to enable verbose gc logging to a file named for the >> taskid in >> /tmp and to set the heap maximum to be a gigabyte, pass a 'value' of: >> -Xmx1024m -verbose:gc -Xloggc:/tmp/@[email protected] >> The configuration variable bsp.child.ulimit can be used to control the >> maximum virtual memory of the child processes. >> </description> >> </property> >> >> On Thu, Dec 27, 2012 at 5:11 AM, Francisco Sanches >> <[email protected]> wrote: >> > Thanks for the replies: >> > >> > Excuse me by the size of the log. But after I changed the parameter of >> > hama-env.sh to "export HAMA_HEAPSIZE = 2000" My program stopped giving >> the >> > memory error and displays the error below. Remembering that I am running >> > for a graph with just over 65 000 vertices. What is it like to understand >> > is that if I increase the number of nodes in the cluster can run my >> > application with this graph of 65 000 vertices? >> > >> > LOG: >> > >> > hduser@mestre:/usr/local/hama$ bin/hama jar aplicpronto.jar 65537 >> > /hdfs/disco/grafogrande /hdfs/disco/grafoout 36 >> > 12/12/26 17:00:19 INFO bsp.FileInputFormat: Total input paths to process >> : 1 >> > 12/12/26 17:00:19 INFO bsp.FileInputFormat: Total # of splits: 36 >> > 12/12/26 17:00:23 INFO bsp.BSPJobClient: Running job: >> job_201212261658_0001 >> > 12/12/26 17:00:26 INFO bsp.BSPJobClient: Current supersteps number: 0 >> > 12/12/26 17:00:56 INFO bsp.BSPJobClient: Current supersteps number: 1 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969, >> > built on 02/23/2011 22:27 GMT >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:host.name=mestre >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:java.version=1.6.0_24 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:java.vendor=Sun Microsystems Inc. >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client >> > environment:java.home=/usr/lib/jvm/java-6-openjdk-i386/jre >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client >> > >> environment:java.class.path=/usr/local/hama/bin/../conf:/usr/lib/jvm/java-6-openjdk-i386/lib/tools.jar:/usr/local/hama/bin/../hama-core-0.5.0.jar:/usr/local/hama/bin/../hama-examples-0.5.0.jar:/usr/local/hama/bin/../hama-graph-0.5.0.jar:/usr/local/hama/bin/../lib/ant-1.7.1.jar:/usr/local/hama/bin/../lib/ant-launcher-1.7.1.jar:/usr/local/hama/bin/../lib/avro-1.6.0.jar:/usr/local/hama/bin/../lib/avro-ipc-1.6.0.jar:/usr/local/hama/bin/../lib/commons-cli-1.2.jar:/usr/local/hama/bin/../lib/commons-configuration-1.7.jar:/usr/local/hama/bin/../lib/commons-lang-2.6.jar:/usr/local/hama/bin/../lib/commons-logging-1.1.1.jar:/usr/local/hama/bin/../lib/commons-math3-3.0.jar:/usr/local/hama/bin/../lib/guava-10.0.1.jar:/usr/local/hama/bin/../lib/hadoop-core-1.0.0.jar:/usr/local/hama/bin/../lib/hadoop-test-1.0.0.jar:/usr/local/hama/bin/../lib/jackson-core-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jackson-mapper-asl-1.9.2.jar:/usr/local/hama/bin/../lib/jetty-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-annotations-6.1.14.jar:/usr/local/hama/bin/../lib/jetty-util-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/local/hama/bin/../lib/junit-4.8.1.jar:/usr/local/hama/bin/../lib/log4j-1.2.16.jar:/usr/local/hama/bin/../lib/netty-3.2.6.Final.jar:/usr/local/hama/bin/../lib/servlet-api-6.0.32.jar:/usr/local/hama/bin/../lib/slf4j-api-1.5.8.jar:/usr/local/hama/bin/../lib/slf4j-log4j12-1.5.8.jar:/usr/local/hama/bin/../lib/snappy-java-1.0.4.1.jar:/usr/local/hama/bin/../lib/zookeeper-3.3.3.jar::/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/classes:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client >> > >> environment:java.library.path=/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386/server:/usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386:/usr/lib/jvm/java-6-openjdk-i386/jre/../lib/i386:/usr/java/packages/lib/i386:/usr/lib/i386-linux-gnu/jni:/lib/i386-linux-gnu:/usr/lib/i386-linux-gnu:/usr/lib/jni:/lib:/usr/lib >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:java.compiler=<NA> >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:os.name=Linux >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:os.arch=i386 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:os.version=3.2.0-32-generic-pae >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:user.name=hduser >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client environment:user.home=/home/hduser >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Client >> > >> environment:user.dir=/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ZooKeeper: Initiating client connection, >> > connectString=node02:21810,node01:21810,mestre:21810 >> sessionTimeout=1200000 >> > watcher=org.apache.hama.bsp.sync.ZooKeeperSyncClientImpl@1deeb40 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ClientCnxn: Opening socket connection to server mestre/ >> > 10.16.1.200:21810 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > zookeeper.ClientCnxn: Socket connection established to mestre/ >> > 10.16.1.200:21810, initiating session >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:31 INFO >> > sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At mestre/ >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:32 INFO >> > zookeeper.ClientCnxn: Session establishment complete on server mestre/ >> > 10.16.1.200:21810, sessionid = 0x3bd8cbb48a0006, negotiated timeout = >> > 1200000 >> > attempt_201212261658_0001_000000_0: SLF4J: Class path contains multiple >> > SLF4J bindings. >> > attempt_201212261658_0001_000000_0: SLF4J: Found binding in >> > >> [jar:file:/usr/local/hama/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class] >> > attempt_201212261658_0001_000000_0: SLF4J: Found binding in >> > >> [file:/hdfs/disco/hama-hduser/bsp/local/groomServer/attempt_201212261658_0001_000000_0/work/org/slf4j/impl/StaticLoggerBinder.class] >> > attempt_201212261658_0001_000000_0: SLF4J: See >> > http://www.slf4j.org/codes.html#multiple_bindings for an explanation. >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x002f8116] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/ >> > 10.16.1.203:61003] BOUND: /10.16.1.200:47685 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x002f8116, /10.16.1.200:47685 => node03/ >> > 10.16.1.203:61003] CONNECTED: node03/10.16.1.203:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00d8ca48] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/ >> > 10.16.1.205:61001] BOUND: /10.16.1.200:52329 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00d8ca48, /10.16.1.200:52329 => node05/ >> > 10.16.1.205:61001] CONNECTED: node05/10.16.1.205:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x018adae2] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/ >> > 10.16.1.204:61002] BOUND: /10.16.1.200:59179 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x018adae2, /10.16.1.200:59179 => node04/ >> > 10.16.1.204:61002] CONNECTED: node04/10.16.1.204:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x003a835d] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/ >> > 10.16.1.206:61001] BOUND: /10.16.1.200:45011 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x003a835d, /10.16.1.200:45011 => node06/ >> > 10.16.1.206:61001] CONNECTED: node06/10.16.1.206:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00061cd2] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/ >> > 10.16.1.204:61003] BOUND: /10.16.1.200:54578 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00061cd2, /10.16.1.200:54578 => node04/ >> > 10.16.1.204:61003] CONNECTED: node04/10.16.1.204:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00fb6c5f] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/ >> > 10.16.1.205:61002] BOUND: /10.16.1.200:52964 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00fb6c5f, /10.16.1.200:52964 => node05/ >> > 10.16.1.205:61002] CONNECTED: node05/10.16.1.205:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00cb42cf] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/ >> > 10.16.1.203:61001] BOUND: /10.16.1.200:36765 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00cb42cf, /10.16.1.200:36765 => node03/ >> > 10.16.1.203:61001] CONNECTED: node03/10.16.1.203:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x018bbf55] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/ >> > 10.16.1.201:61003] BOUND: /10.16.1.200:45734 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x018bbf55, /10.16.1.200:45734 => node01/ >> > 10.16.1.201:61003] CONNECTED: node01/10.16.1.201:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00ed5d9d] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/ >> > 10.16.1.202:61002] BOUND: /10.16.1.200:60724 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00ed5d9d, /10.16.1.200:60724 => node02/ >> > 10.16.1.202:61002] CONNECTED: node02/10.16.1.202:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node03/10.16.1.203:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x014b84c7] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/ >> > 10.16.1.203:61002] BOUND: /10.16.1.200:45839 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x014b84c7, /10.16.1.200:45839 => node03/ >> > 10.16.1.203:61002] CONNECTED: node03/10.16.1.203:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x0160c4b0] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/ >> > 10.16.1.202:61003] BOUND: /10.16.1.200:44757 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x0160c4b0, /10.16.1.200:44757 => node02/ >> > 10.16.1.202:61003] CONNECTED: node02/10.16.1.202:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node04/10.16.1.204:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x0029ea31] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/ >> > 10.16.1.204:61001] BOUND: /10.16.1.200:40846 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x0029ea31, /10.16.1.200:40846 => node04/ >> > 10.16.1.204:61001] CONNECTED: node04/10.16.1.204:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x00158cc6, /10.16.1.210:36396 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.210:36396 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x016f5261, /10.16.1.200:41941 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.200:41941 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x008ddb93, /10.16.1.206:44493 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.206:44493 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01a9f87c, /10.16.1.202:48243 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.202:48243 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01f217ec, /10.16.1.200:41945 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.200:41945 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00a86d12] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/ >> > 10.16.1.201:61001] BOUND: /10.16.1.200:35763 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00a86d12, /10.16.1.200:35763 => node01/ >> > 10.16.1.201:61001] CONNECTED: node01/10.16.1.201:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x008932e8] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x016d3536, /10.16.1.200:41949 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.200:41949 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/ >> > 10.16.1.200:61002] BOUND: /10.16.1.200:41949 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x008932e8, /10.16.1.200:41949 => mestre/ >> > 10.16.1.200:61002] CONNECTED: mestre/10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: Connecting to node01/10.16.1.201:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00f2164f] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01aa0a15, /10.16.1.209:43193 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.209:43193 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/ >> > 10.16.1.201:61002] BOUND: /10.16.1.200:44908 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> > ipc.NettyTransceiver: [id: 0x00f2164f, /10.16.1.200:44908 => node01/ >> > 10.16.1.201:61002] CONNECTED: node01/10.16.1.201:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01cf662f, /10.16.1.207:57476 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.207:57476 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x003eec1a, /10.16.1.207:57478 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.207:57478 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01de6817, /10.16.1.210:36402 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.210:36402 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01115152, /10.16.1.201:37455 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.201:37455 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x00a56214, /10.16.1.207:57480 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.207:57480 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x004bb369, /10.16.1.205:56188 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.205:56188 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01abdac9, /10.16.1.206:44506 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.206:44506 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01f68272, /10.16.1.205:56189 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.205:56189 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x0103c29b, /10.16.1.202:48251 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.202:48251 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01e8c585, /10.16.1.205:56191 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.205:56191 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x003cbb4b, /10.16.1.208:50918 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.208:50918 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x00ab835a, /10.16.1.208:50917 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.208:50917 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x01443800, /10.16.1.209:43200 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.209:43200 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:51 INFO >> ipc.NettyServer: >> > [id: 0x011ff451, /10.16.1.209:43201 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.209:43201 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01a73d30, /10.16.1.208:50924 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.208:50924 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01108727, /10.16.1.211:44099 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.211:44099 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01a220e6, /10.16.1.206:44511 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.206:44511 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x011415c8, /10.16.1.211:44100 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.211:44100 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x012f41a5, /10.16.1.211:44101 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.211:44101 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node02/10.16.1.202:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x019c4091] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x000e7d53, /10.16.1.201:37460 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.201:37460 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/ >> > 10.16.1.202:61001] BOUND: /10.16.1.200:54333 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x019c4091, /10.16.1.200:54333 => node02/ >> > 10.16.1.202:61001] CONNECTED: node02/10.16.1.202:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x0053abbb, /10.16.1.203:46568 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.203:46568 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x009db992, /10.16.1.203:46569 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.203:46569 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01e80761] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/ >> > 10.16.1.200:61003] BOUND: /10.16.1.200:56327 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01e80761, /10.16.1.200:56327 => mestre/ >> > 10.16.1.200:61003] CONNECTED: mestre/10.16.1.200:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to mestre/10.16.1.200:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01dbb27d] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x0093c4f1, /10.16.1.203:46573 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.203:46573 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/ >> > 10.16.1.200:61001] BOUND: /10.16.1.200:51740 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01dbb27d, /10.16.1.200:51740 => mestre/ >> > 10.16.1.200:61001] CONNECTED: mestre/10.16.1.200:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00b9b618, /10.16.1.204:35237 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.204:35237 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00dfc362, /10.16.1.204:35239 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.204:35239 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01971eb3] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x01408a75, /10.16.1.202:48260 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.202:48260 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/ >> > 10.16.1.211:61003] BOUND: /10.16.1.200:57399 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01971eb3, /10.16.1.200:57399 => node11/ >> > 10.16.1.211:61003] CONNECTED: node11/10.16.1.211:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00c713d2, /10.16.1.201:37463 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.201:37463 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x005b78cf] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/ >> > 10.16.1.211:61001] BOUND: /10.16.1.200:41092 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x005b78cf, /10.16.1.200:41092 => node11/ >> > 10.16.1.211:61001] CONNECTED: node11/10.16.1.211:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] BOUND: / >> > 10.16.1.200:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> ipc.NettyServer: >> > [id: 0x00c28cb7, /10.16.1.204:35245 => /10.16.1.200:61002] CONNECTED: / >> > 10.16.1.204:35245 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01f51e5c] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/ >> > 10.16.1.210:61002] BOUND: /10.16.1.200:48018 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01f51e5c, /10.16.1.200:48018 => node10/ >> > 10.16.1.210:61002] CONNECTED: node10/10.16.1.210:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00d36ff3] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/ >> > 10.16.1.209:61003] BOUND: /10.16.1.200:34916 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00d36ff3, /10.16.1.200:34916 => node09/ >> > 10.16.1.209:61003] CONNECTED: node09/10.16.1.209:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node11/10.16.1.211:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x018e899c] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/ >> > 10.16.1.211:61002] BOUND: /10.16.1.200:58261 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x018e899c, /10.16.1.200:58261 => node11/ >> > 10.16.1.211:61002] CONNECTED: node11/10.16.1.211:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x014bc4e6] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/ >> > 10.16.1.210:61003] BOUND: /10.16.1.200:39073 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x014bc4e6, /10.16.1.200:39073 => node10/ >> > 10.16.1.210:61003] CONNECTED: node10/10.16.1.210:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01be1041] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/ >> > 10.16.1.208:61002] BOUND: /10.16.1.200:39758 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01be1041, /10.16.1.200:39758 => node08/ >> > 10.16.1.208:61002] CONNECTED: node08/10.16.1.208:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01e090ee] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/ >> > 10.16.1.209:61001] BOUND: /10.16.1.200:56029 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01e090ee, /10.16.1.200:56029 => node09/ >> > 10.16.1.209:61001] CONNECTED: node09/10.16.1.209:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x0187f9f1] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/ >> > 10.16.1.207:61003] BOUND: /10.16.1.200:35151 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x0187f9f1, /10.16.1.200:35151 => node07/ >> > 10.16.1.207:61003] CONNECTED: node07/10.16.1.207:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00c3358b] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/ >> > 10.16.1.208:61003] BOUND: /10.16.1.200:35301 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00c3358b, /10.16.1.200:35301 => node08/ >> > 10.16.1.208:61003] CONNECTED: node08/10.16.1.208:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node10/10.16.1.210:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01e82b72] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/ >> > 10.16.1.210:61001] BOUND: /10.16.1.200:34819 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01e82b72, /10.16.1.200:34819 => node10/ >> > 10.16.1.210:61001] CONNECTED: node10/10.16.1.210:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node09/10.16.1.209:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00d0deb5] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/ >> > 10.16.1.209:61002] BOUND: /10.16.1.200:34684 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00d0deb5, /10.16.1.200:34684 => node09/ >> > 10.16.1.209:61002] CONNECTED: node09/10.16.1.209:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01505258] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/ >> > 10.16.1.206:61002] BOUND: /10.16.1.200:35962 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01505258, /10.16.1.200:35962 => node06/ >> > 10.16.1.206:61002] CONNECTED: node06/10.16.1.206:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01d8417c] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/ >> > 10.16.1.207:61001] BOUND: /10.16.1.200:41009 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01d8417c, /10.16.1.200:41009 => node07/ >> > 10.16.1.207:61001] CONNECTED: node07/10.16.1.207:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node05/10.16.1.205:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x017c398e] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/ >> > 10.16.1.205:61003] BOUND: /10.16.1.200:53654 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x017c398e, /10.16.1.200:53654 => node05/ >> > 10.16.1.205:61003] CONNECTED: node05/10.16.1.205:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node08/10.16.1.208:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00e13e7b] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/ >> > 10.16.1.208:61001] BOUND: /10.16.1.200:55138 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x00e13e7b, /10.16.1.200:55138 => node08/ >> > 10.16.1.208:61001] CONNECTED: node08/10.16.1.208:61001 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node06/10.16.1.206:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01503458] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/ >> > 10.16.1.206:61003] BOUND: /10.16.1.200:40540 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01503458, /10.16.1.200:40540 => node06/ >> > 10.16.1.206:61003] CONNECTED: node06/10.16.1.206:61003 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: Connecting to node07/10.16.1.207:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01b963c4] OPEN >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/ >> > 10.16.1.207:61002] BOUND: /10.16.1.200:42816 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:52 INFO >> > ipc.NettyTransceiver: [id: 0x01b963c4, /10.16.1.200:42816 => node07/ >> > 10.16.1.207:61002] CONNECTED: node07/10.16.1.207:61002 >> > attempt_201212261658_0001_000000_0: 12/12/26 17:00:53 INFO >> > graph.GraphJobRunner: Loading finished at 1 steps. >> > 12/12/26 17:03:17 INFO bsp.BSPJobClient: Job failed. >> > >> > >> > >> > 2012/12/26 Edward J. Yoon <[email protected]> >> > >> >> BTW, This log looks like you ran out of memory while job was running >> >> as a local mode. >> >> >> >> You can check the cluster status via web UI. >> >> >> >> 1. http://wiki.apache.org/hama/GettingStarted#Hama_Web_Interfaces >> >> >> >> On Wed, Dec 26, 2012 at 10:00 PM, Francisco Sanches >> >> <[email protected]> wrote: >> >> > attempt_201212260904_0005_000031_0: Exception in thread "Thread-1" >> >> > java.lang.OutOfMemoryError: GC overhead limit exceeded >> >> >> >> >> >> >> >> -- >> >> Best Regards, Edward J. Yoon >> >> @eddieyoon >> >> >> > >> > >> > >> > -- >> > Francisco Sanches >> >> >> >> -- >> Best Regards, Edward J. Yoon >> @eddieyoon >> > > > > -- > Francisco Sanches -- Best Regards, Edward J. Yoon @eddieyoon
