D'oh! My mistake... I had set pull_interval not ha.pull_interval so updates were not propagated to other slaves.
Changing my spring context to be <entry key="ha.pull_interval" value="1" /> Made everything work. Cue serious contributions to the curse box ;) Cheers, Toby. On 29/09/2011 10:07, "Toby O'Rourke" <[email protected]> wrote: >Hi, > >Logs are below. The Exception at the start might be a factor? I made a >change on zoo3 which appeared to be propagated back to zoo1 (the master), >but was not propagated on to zoo2. > >Thanks for all your help! > >Toby. > >---- zoo1 (started before the others so should be master?) ---- >Thu Sep 29 08:50:48 UTC 2011: newMaster(null) called null >java.lang.Exception > at >org.neo4j.kernel.HighlyAvailableGraphDatabase.startUp(HighlyAvailableGraph >D >atabase.java:239) > at >org.neo4j.kernel.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphD >a >tabase.java:135) > at >org.neo4j.kernel.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphD >a >tabase.java:111) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >Method) > at >sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcc >e >ssorImpl.java:39) > at >sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstr >u >ctorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > at >org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:126) > at >org.springframework.beans.factory.support.SimpleInstantiationStrategy.inst >a >ntiate(SimpleInstantiationStrategy.java:108) > at >org.springframework.beans.factory.support.ConstructorResolver.autowireCons >t >ructor(ConstructorResolver.java:280) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.createBean(AbstractAutowireCapableBeanFactory.java:456) > at >org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject( >A >bstractBeanFactory.java:291) > at >org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.get >S >ingleton(DefaultSingletonBeanRegistry.java:222) > at >org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(Ab >s >tractBeanFactory.java:288) > at >org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abst >r >actBeanFactory.java:190) > at >org.springframework.beans.factory.support.DefaultListableBeanFactory.findA >u >towireCandidates(DefaultListableBeanFactory.java:844) > at >org.springframework.beans.factory.support.DefaultListableBeanFactory.doRes >o >lveDependency(DefaultListableBeanFactory.java:786) > at >org.springframework.beans.factory.support.DefaultListableBeanFactory.resol >v >eDependency(DefaultListableBeanFactory.java:703) > at >org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostPr >o >cessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor. >j >ava:547) > at >org.springframework.beans.factory.annotation.InjectionMetadata.inject(Inje >c >tionMetadata.java:84) > at >org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostPr >o >cessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java >: >282) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.populateBean(AbstractAutowireCapableBeanFactory.java:1074) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.doCreateBean(AbstractAutowireCapableBeanFactory.java:517) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.createBean(AbstractAutowireCapableBeanFactory.java:456) > at >org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject( >A >bstractBeanFactory.java:291) > at >org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.get >S >ingleton(DefaultSingletonBeanRegistry.java:222) > at >org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(Ab >s >tractBeanFactory.java:288) > at >org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abst >r >actBeanFactory.java:190) > at >org.springframework.beans.factory.support.ConstructorResolver.instantiateU >s >ingFactoryMethod(ConstructorResolver.java:353) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:98 >3 >) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.createBeanInstance(AbstractAutowireCapableBeanFactory.java:879) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) > at >org.springframework.beans.factory.support.AbstractAutowireCapableBeanFacto >r >y.createBean(AbstractAutowireCapableBeanFactory.java:456) > at >org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject( >A >bstractBeanFactory.java:291) > at >org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.get >S >ingleton(DefaultSingletonBeanRegistry.java:222) > at >org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(Ab >s >tractBeanFactory.java:288) > at >org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abst >r >actBeanFactory.java:194) > at >org.springframework.context.support.AbstractApplicationContext.finishBeanF >a >ctoryInitialization(AbstractApplicationContext.java:885) > at >org.springframework.context.support.AbstractApplicationContext.refresh(Abs >t >ractApplicationContext.java:425) > at >org.springframework.web.context.ContextLoader.createWebApplicationContext( >C >ontextLoader.java:276) > at >org.springframework.web.context.ContextLoader.initWebApplicationContext(Co >n >textLoader.java:197) > at >org.springframework.web.context.ContextLoaderListener.contextInitialized(C >o >ntextLoaderListener.java:47) > at >org.apache.catalina.core.StandardContext.listenerStart(StandardContext.jav >a >:3972) > at >org.apache.catalina.core.StandardContext.start(StandardContext.java:4467) > at >org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java >: >791) > at >org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771) > at >org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526) > at >org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:905) > at >org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:740) > at >org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:500) > at >org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277) > at >org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321) > at >org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupp >o >rt.java:119) > at >org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053) > at >org.apache.catalina.core.StandardHost.start(StandardHost.java:722) > at >org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045) > at >org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443) >at >org.apache.catalina.core.StandardService.start(StandardService.java:516) > at >org.apache.catalina.core.StandardServer.start(StandardServer.java:710) > at org.apache.catalina.startup.Catalina.start(Catalina.java:593) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at >sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: >3 >9) > at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm >p >l.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) >Thu Sep 29 08:50:48 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:50:48 UTC >2011 Got event: WatchedEvent state:SyncConnected type:None >path:null(path=null) >Thu Sep 29 08:50:48 UTC 2011: getMaster none based on [] >Thu Sep 29 08:50:48 UTC 2011: Get master before write:(null, >MachineInfo[ID:-1, sequence:-1, last committed tx id:1, server:null]) >Thu Sep 29 08:50:48 UTC 2011: Wrote HA server zoo1:6001 to zoo keeper >Thu Sep 29 08:50:48 UTC 2011: Did setup, seq=0000000023 new >sessionId=230440023487610881 >Thu Sep 29 08:50:48 UTC 2011: getMaster 1 based on [(1, 1, 23)] >Thu Sep 29 08:50:48 UTC 2011: Get master after write:(null, >MachineInfo[ID:1, sequence:23, last committed tx id:1, server:(zoo1, >6001)]) >Thu Sep 29 08:50:48 UTC 2011: Setting 'master-notify' to 1 >Thu Sep 29 08:50:48 UTC 2011: master-notify set to 1 >Thu Sep 29 08:50:48 UTC 2011: Did set 'master-notify' to 1 >Thu Sep 29 08:50:48 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:50:48 UTC >2011 Got event: WatchedEvent state:SyncConnected type:NodeDataChanged >path:/1317204894718_5015401063079066885/master-notify(path=/1317204894718_ >5 >015401063079066885/master-notify) >Thu Sep 29 08:50:48 UTC 2011: getMaster 1 based on [(1, 1, 23)] >Thu Sep 29 08:50:48 UTC 2011: getMaster 1 based on [(1, 1, 23)] >Thu Sep 29 08:50:48 UTC 2011: ReevaluateMyself: machineId=1 with >master[(null, MachineInfo[ID:1, sequence:23, last committed tx id:1, >server:(zoo1, 6001)])] (I am master=false) >Thu Sep 29 08:50:48 UTC 2011: Internal shutdown of HA db[1] >reference=HighlyAvailableGraphDatabase [/home/ubuntu/db], >masterServer=null >Thu Sep 29 08:50:48 UTC 2011: Opened >[/home/ubuntu/db/nioneo_logical.log.1] clean empty log, version=0 >Thu Sep 29 08:50:48 UTC 2011: Opened [/home/ubuntu/db/index/lucene.log.1] >clean empty log, version=0 >Thu Sep 29 08:50:48 UTC 2011: TM opening log: /home/ubuntu/db/tm_tx_log.1 >Thu Sep 29 08:50:48 UTC 2011: --- CONFIGURATION START --- >Thu Sep 29 08:50:48 UTC 2011: Physical mem: 496MB, Heap size: 121MB >Thu Sep 29 08:50:48 UTC 2011: Kernel version: Neo4j - Graph Database >Kernel 1.4.1 >Thu Sep 29 08:50:48 UTC 2011: Neo4j - Graph Database Kernel 1.4.1 >Thu Sep 29 08:50:48 UTC 2011: Operating System: Linux; version: >2.6.32-30-generic-pae; arch: i386; cpus: 1 >Thu Sep 29 08:50:48 UTC 2011: VM Name: Java HotSpot(TM) Client VM >Thu Sep 29 08:50:48 UTC 2011: VM Vendor: Sun Microsystems Inc. >Thu Sep 29 08:50:48 UTC 2011: VM Version: 20.1-b02 >Thu Sep 29 08:50:48 UTC 2011: Boot Class Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/resources.jar:/usr/lib/jvm/java-6 >- >sun-1.6.0.26/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/sunrs >a >sign.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jsse.jar:/usr/lib/jvm/ja >v >a-6-sun-1.6.0.26/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/ >c >harsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/modules/jdk.boot.jar: >/ >usr/lib/jvm/java-6-sun-1.6.0.26/jre/classes >Thu Sep 29 08:50:48 UTC 2011: Class Path: >/usr/share/tomcat6/bin/bootstrap.jar >Thu Sep 29 08:50:48 UTC 2011: Library Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/client:/usr/lib/jvm/java-6-s >u >n-1.6.0.26/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/i386:/ >u >sr/java/packages/lib/i386:/lib:/usr/lib >Thu Sep 29 08:50:48 UTC 2011: Garbage Collector: Copy: [Eden Space, >Survivor Space] >Thu Sep 29 08:50:48 UTC 2011: Garbage Collector: MarkSweepCompact: [Eden >Space, Survivor Space, Tenured Gen, Perm Gen, Perm Gen [shared-ro], Perm >Gen [shared-rw]] >Thu Sep 29 08:50:48 UTC 2011: VM Arguments: >[-Djava.util.logging.config.file=/home/ubuntu/tomcat/conf/logging.properti >e >s, -Dserver.id=1, -Dha.server.port=6001, -Ddatabase.path=/home/ubuntu/db, >-Dzookeeper.servers=zoo1:2181,zoo2:2181,zoo3:2181, >-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager, >-Djava.endorsed.dirs=/usr/share/tomcat6/endorsed, >-Dcatalina.base=/home/ubuntu/tomcat, -Dcatalina.home=/usr/share/tomcat6, >-Djava.io.tmpdir=/home/ubuntu/tomcat/temp] >Thu Sep 29 08:50:48 UTC 2011: >Thu Sep 29 08:50:48 UTC 2011: create=true >Thu Sep 29 08:50:48 UTC 2011: enable_remote_shell=port=1331 >Thu Sep 29 08:50:48 UTC 2011: ha.machine_id=1 >Thu Sep 29 08:50:48 UTC 2011: ha.server=zoo1:6001 >Thu Sep 29 08:50:48 UTC 2011: >ha.zoo_keeper_servers=zoo1:2181,zoo2:2181,zoo3:2181 >Thu Sep 29 08:50:48 UTC 2011: keep_logical_logs=true >Thu Sep 29 08:50:48 UTC 2011: >logical_log=/home/ubuntu/db/nioneo_logical.log >Thu Sep 29 08:50:48 UTC 2011: neo_store=/home/ubuntu/db/neostore >Thu Sep 29 08:50:48 UTC 2011: neostore.nodestore.db.mapped_memory=20M >Thu Sep 29 08:50:48 UTC 2011: >neostore.propertystore.db.arrays.mapped_memory=130M >Thu Sep 29 08:50:48 UTC 2011: >neostore.propertystore.db.index.keys.mapped_memory=1M >Thu Sep 29 08:50:48 UTC 2011: >neostore.propertystore.db.index.mapped_memory=1M >Thu Sep 29 08:50:48 UTC 2011: neostore.propertystore.db.mapped_memory=90M >Thu Sep 29 08:50:48 UTC 2011: >neostore.propertystore.db.strings.mapped_memory=130M >Thu Sep 29 08:50:48 UTC 2011: >neostore.relationshipstore.db.mapped_memory=100M >Thu Sep 29 08:50:48 UTC 2011: node_auto_indexing=false >Thu Sep 29 08:50:48 UTC 2011: Opened [/home/ubuntu/db/index/lucene.log.1] >clean empty log, version=0 >Thu Sep 29 08:50:48 UTC 2011: TM opening log: /home/ubuntu/db/tm_tx_log.1 >Thu Sep 29 08:50:48 UTC 2011: --- CONFIGURATION START --- >Thu Sep 29 08:50:48 UTC 2011: Physical mem: 496MB, Heap size: 121MB >Thu Sep 29 08:50:48 UTC 2011: Kernel version: Neo4j - Graph Database >Kernel 1.4.1 >Thu Sep 29 08:50:48 UTC 2011: Neo4j - Graph Database Kernel 1.4.1 >Thu Sep 29 08:50:48 UTC 2011: Operating System: Linux; version: >2.6.32-30-generic-pae; arch: i386; cpus: 1 >Thu Sep 29 08:50:48 UTC 2011: VM Name: Java HotSpot(TM) Client VM >Thu Sep 29 08:50:48 UTC 2011: VM Vendor: Sun Microsystems Inc. >Thu Sep 29 08:50:48 UTC 2011: VM Version: 20.1-b02 >Thu Sep 29 08:50:48 UTC 2011: Boot Class Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/resources.jar:/usr/lib/jvm/java-6 >- >sun-1.6.0.26/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/sunrs >a >sign.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jsse.jar:/usr/lib/jvm/ja >v >a-6-sun-1.6.0.26/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/ >c >harsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/modules/jdk.boot.jar: >/ >usr/lib/jvm/java-6-sun-1.6.0.26/jre/classes >Thu Sep 29 08:50:48 UTC 2011: Class Path: >/usr/share/tomcat6/bin/bootstrap.jar >Thu Sep 29 08:50:48 UTC 2011: Library Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/client:/usr/lib/jvm/java-6-s >u >n-1.6.0.26/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/i386:/ >u >sr/java/packages/lib/i386:/lib:/usr/lib >Thu Sep 29 08:50:48 UTC 2011: Garbage Collector: Copy: [Eden Space, >Survivor Space] >Thu Sep 29 08:50:48 UTC 2011: Garbage Collector: MarkSweepCompact: [Eden >Space, Survivor Space, Tenured Gen, Perm Gen, Perm Gen [shared-ro], Perm >Gen [shared-rw]] >Thu Sep 29 08:50:48 UTC 2011: VM Arguments: >[-Djava.util.logging.config.file=/home/ubuntu/tomcat/conf/logging.properti >e >s, -Dserver.id=1, -Dha.server.port=6001, -Ddatabase.path=/home/ubuntu/db, >-Dzookeeper.servers=zoo1:2181,zoo2:2181,zoo3:2181, >-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager, >-Djava.endorsed.dirs=/usr/share/tomcat6/endorsed, >-Dcatalina.base=/home/ubuntu/tomcat, -Dcatalina.home=/usr/share/tomcat6, >-Djava.io.tmpdir=/home/ubuntu/tomcat/temp] >Thu Sep 29 08:50:48 UTC 2011: >Thu Sep 29 08:50:48 UTC 2011: create=true >Thu Sep 29 08:50:48 UTC 2011: enable_remote_shell=port=1331 >Thu Sep 29 08:50:48 UTC 2011: ha.machine_id=1 >Thu Sep 29 08:50:48 UTC 2011: ha.server=zoo1:6001 >Thu Sep 29 08:50:48 UTC 2011: >ha.zoo_keeper_servers=zoo1:2181,zoo2:2181,zoo3:2181 >Thu Sep 29 08:50:48 UTC 2011: keep_logical_logs=true >Thu Sep 29 08:50:48 UTC 2011: >logical_log=/home/ubuntu/db/nioneo_logical.log >Thu Sep 29 08:50:48 UTC 2011: neo_store=/home/ubuntu/db/neostore >Thu Sep 29 08:50:48 UTC 2011: neostore.nodestore.db.mapped_memory=20M >Thu Sep 29 08:50:48 UTC 2011: >neostore.propertystore.db.arrays.mapped_memory=130M >Thu Sep 29 08:50:48 UTC 2011: >neostore.propertystore.db.index.keys.mapped_memory=1M >Thu Sep 29 08:50:48 UTC 2011: >neostore.propertystore.db.index.mapped_memory=1M >Thu Sep 29 08:50:48 UTC 2011: neostore.propertystore.db.mapped_memory=90M >Thu Sep 29 08:50:48 UTC 2011: >neostore.propertystore.db.strings.mapped_memory=130M >Thu Sep 29 08:50:48 UTC 2011: >neostore.relationshipstore.db.mapped_memory=100M >Thu Sep 29 08:50:48 UTC 2011: node_auto_indexing=false >Thu Sep 29 08:50:48 UTC 2011: pull_interval=1 >Thu Sep 29 08:50:48 UTC 2011: rebuild_idgenerators_fast=true >Thu Sep 29 08:50:48 UTC 2011: relationship_auto_indexing=false >Thu Sep 29 08:50:48 UTC 2011: store_dir=/home/ubuntu/db >Thu Sep 29 08:50:48 UTC 2011: use_memory_mapped_buffers=true >Thu Sep 29 08:50:48 UTC 2011: --- CONFIGURATION END --- >Thu Sep 29 08:50:48 UTC 2011: Extension >org.neo4j.kernel.KernelExtension[online backup] loaded ok >Thu Sep 29 08:50:48 UTC 2011: Extension >org.neo4j.kernel.KernelExtension[kernel jmx] loaded ok >Thu Sep 29 08:50:48 UTC 2011: Extension >org.neo4j.kernel.KernelExtension[kernel udc] loaded ok >Thu Sep 29 08:50:48 UTC 2011: MasterServer communication server started >and bound to 6001 >Thu Sep 29 08:52:15 UTC 2011: applyTxWithoutTxId log version: 0, >committing tx=4) @ pos 367 >Thu Sep 29 08:52:15 UTC 2011: Applied external tx and generated tx id=4 >Thu Sep 29 08:52:15 UTC 2011: applyTxWithoutTxId log version: 0, >committing tx=5) @ pos 490 >Thu Sep 29 08:52:15 UTC 2011: Applied external tx and generated tx id=5 >Thu Sep 29 08:52:23 UTC 2011: applyTxWithoutTxId log version: 0, >committing tx=6) @ pos 613 >Thu Sep 29 08:52:23 UTC 2011: Applied external tx and generated tx id=6 >Thu Sep 29 08:52:23 UTC 2011: applyTxWithoutTxId log version: 0, >committing tx=7) @ pos 736 >Thu Sep 29 08:52:23 UTC 2011: Applied external tx and generated tx id=7 >Thu Sep 29 08:54:45 UTC 2011: applyTxWithoutTxId log version: 0, >committing tx=3) @ pos 169 >Thu Sep 29 08:54:45 UTC 2011: Applied external tx and generated tx id=3 >Thu Sep 29 08:54:45 UTC 2011: applyTxWithoutTxId log version: 0, >committing tx=4) @ pos 328 >Thu Sep 29 08:54:45 UTC 2011: Applied external tx and generated tx id=4 >Thu Sep 29 08:54:45 UTC 2011: applyTxWithoutTxId log version: 0, >committing tx=10) @ pos 1113 >Thu Sep 29 08:54:45 UTC 2011: Applied external tx and generated tx id=10 >Thu Sep 29 08:54:45 UTC 2011: applyTxWithoutTxId log version: 0, >committing tx=5) @ pos 487 >Thu Sep 29 08:54:46 UTC 2011: Applied external tx and generated tx id=5 > > > >---- zoo2 ---- >Thu Sep 29 08:52:13 UTC 2011: newMaster(null) called null >java.lang.Exception > at >org.neo4j.kernel.HighlyAvailableGraphDatabase.startUp(HighlyAvailableGraph >D >atabase.java:239) > at >org.neo4j.kernel.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphD >a >tabase.java:135) > at >org.neo4j.kernel.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphD >a >tabase.java:111) > ><snip - more stack trace - same as zoo1> > > >Thu Sep 29 08:52:13 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:52:13 UTC >2011 Got event: WatchedEvent state:SyncConnected type:None >path:null(path=null) >Thu Sep 29 08:52:13 UTC 2011: Read HA server:zoo1:6001 (for machineID 1) >from zoo keeper >Thu Sep 29 08:52:13 UTC 2011: getMaster 1 based on [(1, 3, 23)] >Thu Sep 29 08:52:13 UTC 2011: Client connected to zoo1:6001 >Thu Sep 29 08:52:13 UTC 2011: Get master before >write:(org.neo4j.kernel.ha.MasterClient@eb67e8, MachineInfo[ID:1, >sequence:23, last committed tx id:3, server:(zoo1, 6001)]) >Thu Sep 29 08:52:13 UTC 2011: Wrote HA server zoo1:6001 to zoo keeper >Thu Sep 29 08:52:13 UTC 2011: Did setup, seq=0000000024 new >sessionId=230440023487610882 >Thu Sep 29 08:52:13 UTC 2011: getMaster 1 based on [(1, 3, 23), (2, 1, >24)] >Thu Sep 29 08:52:13 UTC 2011: Get master after >write:(org.neo4j.kernel.ha.MasterClient@eb67e8, MachineInfo[ID:1, >sequence:23, last committed tx id:3, server:(zoo1, 6001)]) >Thu Sep 29 08:52:13 UTC 2011: Setting 'master-notify' to 1 >Thu Sep 29 08:52:13 UTC 2011: master-notify set to 1 >Thu Sep 29 08:52:13 UTC 2011: Did set 'master-notify' to 1 >Thu Sep 29 08:52:13 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:52:13 UTC >2011 Got event: WatchedEvent state:SyncConnected type:NodeDataChanged >path:/1317204894718_5015401063079066885/master-notify(path=/1317204894718_ >5 >015401063079066885/master-notify) >Thu Sep 29 08:52:13 UTC 2011: getMaster 1 based on [(1, 3, 23), (2, 1, >24)] >Thu Sep 29 08:52:13 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:52:13 UTC >2011 Got event: WatchedEvent state:SyncConnected type:NodeDataChanged >path:/1317204894718_5015401063079066885/master-rebound(path=/1317204894718 >_ >5015401063079066885/master-rebound) >Thu Sep 29 08:52:13 UTC 2011: getMaster 1 based on [(1, 3, 23), (2, 1, >24)] >Thu Sep 29 08:52:13 UTC 2011: getMaster 1 based on [(1, 3, 23), (2, 1, >24)] >Thu Sep 29 08:52:13 UTC 2011: ReevaluateMyself: machineId=2 with >master[(org.neo4j.kernel.ha.MasterClient@eb67e8, MachineInfo[ID:1, >sequence:23, last committed tx id:3, server:(zoo1, 6001)])] (I am >master=false) >Thu Sep 29 08:52:13 UTC 2011: master-notify not set, is already 1 >Thu Sep 29 08:52:13 UTC 2011: Internal shutdown of HA db[2] >reference=HighlyAvailableGraphDatabase [/home/ubuntu/db], >masterServer=null >Thu Sep 29 08:52:14 UTC 2011: Opened >[/home/ubuntu/db/nioneo_logical.log.1] clean empty log, version=0 >Thu Sep 29 08:52:14 UTC 2011: Opened [/home/ubuntu/db/index/lucene.log.1] >clean empty log, version=0 >Thu Sep 29 08:52:14 UTC 2011: TM opening log: /home/ubuntu/db/tm_tx_log.1 >Thu Sep 29 08:52:14 UTC 2011: --- CONFIGURATION START --- >Thu Sep 29 08:52:14 UTC 2011: Physical mem: 496MB, Heap size: 121MB >Thu Sep 29 08:52:14 UTC 2011: Kernel version: Neo4j - Graph Database >Kernel 1.4.1 >Thu Sep 29 08:52:14 UTC 2011: Neo4j - Graph Database Kernel 1.4.1 >Thu Sep 29 08:52:14 UTC 2011: Operating System: Linux; version: >2.6.32-30-generic-pae; arch: i386; cpus: 1 >Thu Sep 29 08:52:14 UTC 2011: VM Name: Java HotSpot(TM) Client VM >Thu Sep 29 08:52:14 UTC 2011: VM Vendor: Sun Microsystems Inc. >Thu Sep 29 08:52:14 UTC 2011: VM Version: 20.1-b02 >Thu Sep 29 08:52:14 UTC 2011: Boot Class Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/resources.jar:/usr/lib/jvm/java-6 >- >sun-1.6.0.26/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/sunrs >a >sign.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jsse.jar:/usr/lib/jvm/ja >v >a-6-sun-1.6.0.26/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/ >c >harsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/modules/jdk.boot.jar: >/ >usr/lib/jvm/java-6-sun-1.6.0.26/jre/classes >Thu Sep 29 08:52:14 UTC 2011: Class Path: >/usr/share/tomcat6/bin/bootstrap.jar >Thu Sep 29 08:52:14 UTC 2011: Library Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/client:/usr/lib/jvm/java-6-s >u >n-1.6.0.26/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/i386:/ >u >sr/java/packages/lib/i386:/lib:/usr/lib >Thu Sep 29 08:52:14 UTC 2011: Garbage Collector: Copy: [Eden Space, >Survivor Space] >Thu Sep 29 08:52:14 UTC 2011: Garbage Collector: MarkSweepCompact: [Eden >Space, Survivor Space, Tenured Gen, Perm Gen, Perm Gen [shared-ro], Perm >Gen [shared-rw]] >Thu Sep 29 08:52:14 UTC 2011: VM Arguments: >[-Djava.util.logging.config.file=/home/ubuntu/tomcat/conf/logging.properti >e >s, -Dserver.id=2, -Dha.server.port=6001, -Ddatabase.path=/home/ubuntu/db, >-Dzookeeper.servers=zoo1:2181,zoo2:2181,zoo3:2181, >-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager, >-Djava.endorsed.dirs=/usr/share/tomcat6/endorsed, >-Dcatalina.base=/home/ubuntu/tomcat, -Dcatalina.home=/usr/share/tomcat6, >-Djava.io.tmpdir=/home/ubuntu/tomcat/temp] >Thu Sep 29 08:52:14 UTC 2011: >Thu Sep 29 08:52:14 UTC 2011: create=true >Thu Sep 29 08:52:14 UTC 2011: enable_remote_shell=port=1331 >Thu Sep 29 08:52:14 UTC 2011: ha.machine_id=2 >Thu Sep 29 08:52:14 UTC 2011: ha.server=zoo1:6001 >Thu Sep 29 08:52:14 UTC 2011: >ha.zoo_keeper_servers=zoo1:2181,zoo2:2181,zoo3:2181 >Thu Sep 29 08:52:14 UTC 2011: keep_logical_logs=true >Thu Sep 29 08:52:14 UTC 2011: >logical_log=/home/ubuntu/db/nioneo_logical.log >Thu Sep 29 08:52:14 UTC 2011: neo_store=/home/ubuntu/db/neostore >Thu Sep 29 08:52:14 UTC 2011: neostore.nodestore.db.mapped_memory=20M >Thu Sep 29 08:52:14 UTC 2011: >neostore.propertystore.db.arrays.mapped_memory=130M >Thu Sep 29 08:52:14 UTC 2011: >neostore.propertystore.db.index.keys.mapped_memory=1M >Thu Sep 29 08:52:14 UTC 2011: >neostore.propertystore.db.index.mapped_memory=1M >Thu Sep 29 08:52:13 UTC 2011: Wrote HA server zoo1:6001 to zoo keeper >Thu Sep 29 08:52:13 UTC 2011: Did setup, seq=0000000024 new >sessionId=230440023487610882 >Thu Sep 29 08:52:13 UTC 2011: getMaster 1 based on [(1, 3, 23), (2, 1, >24)] >Thu Sep 29 08:52:13 UTC 2011: Get master after >write:(org.neo4j.kernel.ha.MasterClient@eb67e8, MachineInfo[ID:1, >sequence:23, last committed tx id:3, server:(zoo1, 6001)]) >Thu Sep 29 08:52:13 UTC 2011: Setting 'master-notify' to 1 >Thu Sep 29 08:52:13 UTC 2011: master-notify set to 1 >Thu Sep 29 08:52:13 UTC 2011: Did set 'master-notify' to 1 >Thu Sep 29 08:52:13 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:52:13 UTC >2011 Got event: WatchedEvent state:SyncConnected type:NodeDataChanged >path:/1317204894718_5015401063079066885/master-notify(path=/1317204894718_ >5 >015401063079066885/master-notify) >Thu Sep 29 08:52:13 UTC 2011: getMaster 1 based on [(1, 3, 23), (2, 1, >24)] >Thu Sep 29 08:52:13 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:52:13 UTC >2011 Got event: WatchedEvent state:SyncConnected type:NodeDataChanged >path:/1317204894718_5015401063079066885/master-rebound(path=/1317204894718 >_ >5015401063079066885/master-rebound) >Thu Sep 29 08:52:13 UTC 2011: getMaster 1 based on [(1, 3, 23), (2, 1, >24)] >Thu Sep 29 08:52:13 UTC 2011: getMaster 1 based on [(1, 3, 23), (2, 1, >24)] >Thu Sep 29 08:52:13 UTC 2011: ReevaluateMyself: machineId=2 with >master[(org.neo4j.kernel.ha.MasterClient@eb67e8, MachineInfo[ID:1, >sequence:23, last committed tx id:3, server:(zoo1, 6001)])] (I am >master=false) >Thu Sep 29 08:52:13 UTC 2011: master-notify not set, is already 1 >Thu Sep 29 08:52:13 UTC 2011: Internal shutdown of HA db[2] >reference=HighlyAvailableGraphDatabase [/home/ubuntu/db], >masterServer=null >Thu Sep 29 08:52:14 UTC 2011: Opened >[/home/ubuntu/db/nioneo_logical.log.1] clean empty log, version=0 >Thu Sep 29 08:52:14 UTC 2011: Opened [/home/ubuntu/db/index/lucene.log.1] >clean empty log, version=0 >Thu Sep 29 08:52:14 UTC 2011: TM opening log: /home/ubuntu/db/tm_tx_log.1 >Thu Sep 29 08:52:14 UTC 2011: --- CONFIGURATION START --- >Thu Sep 29 08:52:14 UTC 2011: Physical mem: 496MB, Heap size: 121MB >Thu Sep 29 08:52:14 UTC 2011: Kernel version: Neo4j - Graph Database >Kernel 1.4.1 >Thu Sep 29 08:52:14 UTC 2011: Neo4j - Graph Database Kernel 1.4.1 >Thu Sep 29 08:52:14 UTC 2011: Operating System: Linux; version: >2.6.32-30-generic-pae; arch: i386; cpus: 1 >Thu Sep 29 08:52:14 UTC 2011: VM Name: Java HotSpot(TM) Client VM >Thu Sep 29 08:52:14 UTC 2011: VM Vendor: Sun Microsystems Inc. >Thu Sep 29 08:52:14 UTC 2011: VM Version: 20.1-b02 >Thu Sep 29 08:52:14 UTC 2011: Boot Class Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/resources.jar:/usr/lib/jvm/java-6 >- >sun-1.6.0.26/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/sunrs >a >sign.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jsse.jar:/usr/lib/jvm/ja >v >a-6-sun-1.6.0.26/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/ >c >harsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/modules/jdk.boot.jar: >/ >usr/lib/jvm/java-6-sun-1.6.0.26/jre/classes >Thu Sep 29 08:52:14 UTC 2011: Class Path: >/usr/share/tomcat6/bin/bootstrap.jar >Thu Sep 29 08:52:14 UTC 2011: Library Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/client:/usr/lib/jvm/java-6-s >u >n-1.6.0.26/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/i386:/ >u >sr/java/packages/lib/i386:/lib:/usr/lib >Thu Sep 29 08:52:14 UTC 2011: Garbage Collector: Copy: [Eden Space, >Survivor Space] >Thu Sep 29 08:52:14 UTC 2011: Garbage Collector: MarkSweepCompact: [Eden >Space, Survivor Space, Tenured Gen, Perm Gen, Perm Gen [shared-ro], Perm >Gen [shared-rw]] >Thu Sep 29 08:52:14 UTC 2011: VM Arguments: >[-Djava.util.logging.config.file=/home/ubuntu/tomcat/conf/logging.properti >e >s, -Dserver.id=2, -Dha.server.port=6001, -Ddatabase.path=/home/ubuntu/db, >-Dzookeeper.servers=zoo1:2181,zoo2:2181,zoo3:2181, >-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager, >-Djava.endorsed.dirs=/usr/share/tomcat6/endorsed, >-Dcatalina.base=/home/ubuntu/tomcat, -Dcatalina.home=/usr/share/tomcat6, >-Djava.io.tmpdir=/home/ubuntu/tomcat/temp] >Thu Sep 29 08:52:14 UTC 2011: >Thu Sep 29 08:52:14 UTC 2011: create=true >Thu Sep 29 08:52:14 UTC 2011: enable_remote_shell=port=1331 >Thu Sep 29 08:52:14 UTC 2011: ha.machine_id=2 >Thu Sep 29 08:52:14 UTC 2011: ha.server=zoo1:6001 >Thu Sep 29 08:52:14 UTC 2011: >ha.zoo_keeper_servers=zoo1:2181,zoo2:2181,zoo3:2181 >Thu Sep 29 08:52:14 UTC 2011: keep_logical_logs=true >Thu Sep 29 08:52:14 UTC 2011: >logical_log=/home/ubuntu/db/nioneo_logical.log >Thu Sep 29 08:52:14 UTC 2011: neo_store=/home/ubuntu/db/neostore >Thu Sep 29 08:52:14 UTC 2011: neostore.nodestore.db.mapped_memory=20M >Thu Sep 29 08:52:14 UTC 2011: >neostore.propertystore.db.arrays.mapped_memory=130M >Thu Sep 29 08:52:14 UTC 2011: >neostore.propertystore.db.index.keys.mapped_memory=1M >Thu Sep 29 08:52:14 UTC 2011: >neostore.propertystore.db.index.mapped_memory=1M >Thu Sep 29 08:52:14 UTC 2011: neostore.propertystore.db.mapped_memory=90M >Thu Sep 29 08:52:14 UTC 2011: >neostore.propertystore.db.strings.mapped_memory=130M >Thu Sep 29 08:52:14 UTC 2011: >neostore.relationshipstore.db.mapped_memory=100M >Thu Sep 29 08:52:14 UTC 2011: node_auto_indexing=false >Thu Sep 29 08:52:14 UTC 2011: pull_interval=1 >Thu Sep 29 08:52:14 UTC 2011: rebuild_idgenerators_fast=true >Thu Sep 29 08:52:14 UTC 2011: relationship_auto_indexing=false >Thu Sep 29 08:52:14 UTC 2011: store_dir=/home/ubuntu/db >Thu Sep 29 08:52:14 UTC 2011: use_memory_mapped_buffers=true >Thu Sep 29 08:52:14 UTC 2011: --- CONFIGURATION END --- >Thu Sep 29 08:52:14 UTC 2011: Extension >org.neo4j.kernel.KernelExtension[online backup] loaded ok >Thu Sep 29 08:52:14 UTC 2011: Extension >org.neo4j.kernel.KernelExtension[kernel jmx] loaded ok >Thu Sep 29 08:52:14 UTC 2011: Extension >org.neo4j.kernel.KernelExtension[kernel udc] loaded ok > > > >---- zoo3 ---- > >Thu Sep 29 08:52:20 UTC 2011: newMaster(null) called null >java.lang.Exception > at >org.neo4j.kernel.HighlyAvailableGraphDatabase.startUp(HighlyAvailableGraph >D >atabase.java:239) > at >org.neo4j.kernel.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphD >a >tabase.java:135) > at >org.neo4j.kernel.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphD >a >tabase.java:111) > ><snip - more stack trace - same as zoo1> > > at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) > at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) >Thu Sep 29 08:52:20 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:52:20 UTC >2011 Got event: WatchedEvent state:SyncConnected type:None >path:null(path=null) >Thu Sep 29 08:52:20 UTC 2011: Read HA server:zoo1:6001 (for machineID 2) >from zoo keeper >Thu Sep 29 08:52:20 UTC 2011: Read HA server:zoo1:6001 (for machineID 1) >from zoo keeper >Thu Sep 29 08:52:20 UTC 2011: getMaster 1 based on [(1, 5, 23), (2, 5, >24)] >Thu Sep 29 08:52:20 UTC 2011: Client connected to zoo1:6001 >Thu Sep 29 08:52:20 UTC 2011: Get master before >write:(org.neo4j.kernel.ha.MasterClient@f2ea42, MachineInfo[ID:1, >sequence:23, last committed tx id:5, server:(zoo1, 6001)]) >Thu Sep 29 08:52:20 UTC 2011: Wrote HA server zoo1:6001 to zoo keeper >Thu Sep 29 08:52:21 UTC 2011: Did setup, seq=0000000025 new >sessionId=158382429485989892 >Thu Sep 29 08:52:21 UTC 2011: getMaster 1 based on [(1, 5, 23), (2, 5, >24), (3, 1, 25)] >Thu Sep 29 08:52:21 UTC 2011: Get master after >write:(org.neo4j.kernel.ha.MasterClient@f2ea42, MachineInfo[ID:1, >sequence:23, last committed tx id:5, server:(zoo1, 6001)]) >Thu Sep 29 08:52:21 UTC 2011: Setting 'master-notify' to 1 >Thu Sep 29 08:52:21 UTC 2011: master-notify set to 1 >Thu Sep 29 08:52:21 UTC 2011: Did set 'master-notify' to 1 >Thu Sep 29 08:52:21 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:52:21 UTC >2011 Got event: WatchedEvent state:SyncConnected type:NodeDataChanged >path:/1317204894718_5015401063079066885/master-notify(path=/1317204894718_ >5 >015401063079066885/master-notify) >Thu Sep 29 08:52:21 UTC 2011: getMaster 1 based on [(1, 5, 23), (2, 5, >24), (3, 1, 25)] >Thu Sep 29 08:52:21 UTC 2011: getMaster 1 based on [(1, 5, 23), (2, 5, >24), (3, 1, 25)] >Thu Sep 29 08:52:21 UTC 2011: ReevaluateMyself: machineId=3 with >master[(org.neo4j.kernel.ha.MasterClient@f2ea42, MachineInfo[ID:1, >sequence:23, last committed tx id:5, server:(zoo1, 6001)])] (I am >master=false) >Thu Sep 29 08:52:21 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:52:21 UTC >2011 Got event: WatchedEvent state:SyncConnected type:NodeDataChanged >path:/1317204894718_5015401063079066885/master-rebound(path=/1317204894718 >_ >5015401063079066885/master-rebound) >Thu Sep 29 08:52:21 UTC 2011: master-notify not set, is already 1 >Thu Sep 29 08:52:21 UTC 2011: Internal shutdown of HA db[3] >reference=HighlyAvailableGraphDatabase [/home/ubuntu/db], >masterServer=null >Thu Sep 29 08:52:21 UTC 2011: Opened >[/home/ubuntu/db/nioneo_logical.log.1] clean empty log, version=0 >Thu Sep 29 08:52:21 UTC 2011: Opened [/home/ubuntu/db/index/lucene.log.1] >clean empty log, version=0 >Thu Sep 29 08:52:21 UTC 2011: TM opening log: /home/ubuntu/db/tm_tx_log.1 >Thu Sep 29 08:52:21 UTC 2011: --- CONFIGURATION START --- >Thu Sep 29 08:52:21 UTC 2011: Physical mem: 496MB, Heap size: 121MB >Thu Sep 29 08:52:21 UTC 2011: Kernel version: Neo4j - Graph Database >Kernel 1.4.1 >Thu Sep 29 08:52:21 UTC 2011: Neo4j - Graph Database Kernel 1.4.1 >Thu Sep 29 08:52:21 UTC 2011: Operating System: Linux; version: >2.6.32-30-generic-pae; arch: i386; cpus: 1 >Thu Sep 29 08:52:21 UTC 2011: VM Name: Java HotSpot(TM) Client VM >Thu Sep 29 08:52:21 UTC 2011: VM Vendor: Sun Microsystems Inc. >Thu Sep 29 08:52:21 UTC 2011: VM Version: 20.1-b02 >Thu Sep 29 08:52:21 UTC 2011: Boot Class Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/resources.jar:/usr/lib/jvm/java-6 >- >sun-1.6.0.26/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/sunrs >a >sign.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jsse.jar:/usr/lib/jvm/ja >v >a-6-sun-1.6.0.26/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/ >c >harsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/modules/jdk.boot.jar: >/ >usr/lib/jvm/java-6-sun-1.6.0.26/jre/classes >Thu Sep 29 08:52:21 UTC 2011: Class Path: >/usr/share/tomcat6/bin/bootstrap.jar >Thu Sep 29 08:52:21 UTC 2011: Library Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/client:/usr/lib/jvm/java-6-s >u >n-1.6.0.26/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/i386:/ >u >sr/java/packages/lib/i386:/lib:/usr/lib >Thu Sep 29 08:52:21 UTC 2011: Garbage Collector: Copy: [Eden Space, >Survivor Space] >Thu Sep 29 08:52:21 UTC 2011: Garbage Collector: MarkSweepCompact: [Eden >Space, Survivor Space, Tenured Gen, Perm Gen, Perm Gen [shared-ro], Perm >Gen [shared-rw]] >Thu Sep 29 08:52:21 UTC 2011: VM Arguments: >[-Djava.util.logging.config.file=/home/ubuntu/tomcat/conf/logging.properti >e >s, -Dserver.id=3, -Dha.server.port=6001, -Ddatabase.path=/home/ubuntu/db, >-Dzookeeper.servers=zoo1:2181,zoo2:2181,zoo3:2181, >-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager, >-Djava.endorsed.dirs=/usr/share/tomcat6/endorsed, >-Dcatalina.base=/home/ubuntu/tomcat, -Dcatalina.home=/usr/share/tomcat6, >-Djava.io.tmpdir=/home/ubuntu/tomcat/temp] >Thu Sep 29 08:52:21 UTC 2011: >Thu Sep 29 08:52:21 UTC 2011: create=true >Thu Sep 29 08:52:21 UTC 2011: enable_remote_shell=port=1331 >Thu Sep 29 08:52:21 UTC 2011: ha.machine_id=3 >Thu Sep 29 08:52:21 UTC 2011: ha.server=zoo1:6001 >Thu Sep 29 08:52:21 UTC 2011: >ha.zoo_keeper_servers=zoo1:2181,zoo2:2181,zoo3:2181 >Thu Sep 29 08:52:21 UTC 2011: keep_logical_logs=true >Thu Sep 29 08:52:21 UTC 2011: >logical_log=/home/ubuntu/db/nioneo_logical.log >Thu Sep 29 08:52:21 UTC 2011: neo_store=/home/ubuntu/db/neostore >Thu Sep 29 08:52:21 UTC 2011: neostore.nodestore.db.mapped_memory=20M >Thu Sep 29 08:52:21 UTC 2011: >neostore.propertystore.db.arrays.mapped_memory=130M >Thu Sep 29 08:52:21 UTC 2011: >neostore.propertystore.db.index.keys.mapped_memory=1M >Thu Sep 29 08:52:20 UTC 2011: Get master before >write:(org.neo4j.kernel.ha.MasterClient@f2ea42, MachineInfo[ID:1, >sequence:23, last committed tx id:5, server:(zoo1, 6001)]) >Thu Sep 29 08:52:20 UTC 2011: Wrote HA server zoo1:6001 to zoo keeper >Thu Sep 29 08:52:21 UTC 2011: Did setup, seq=0000000025 new >sessionId=158382429485989892 >Thu Sep 29 08:52:21 UTC 2011: getMaster 1 based on [(1, 5, 23), (2, 5, >24), (3, 1, 25)] >Thu Sep 29 08:52:21 UTC 2011: Get master after >write:(org.neo4j.kernel.ha.MasterClient@f2ea42, MachineInfo[ID:1, >sequence:23, last committed tx id:5, server:(zoo1, 6001)]) >Thu Sep 29 08:52:21 UTC 2011: Setting 'master-notify' to 1 >Thu Sep 29 08:52:21 UTC 2011: master-notify set to 1 >Thu Sep 29 08:52:21 UTC 2011: Did set 'master-notify' to 1 >Thu Sep 29 08:52:21 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:52:21 UTC >2011 Got event: WatchedEvent state:SyncConnected type:NodeDataChanged >path:/1317204894718_5015401063079066885/master-notify(path=/1317204894718_ >5 >015401063079066885/master-notify) >Thu Sep 29 08:52:21 UTC 2011: getMaster 1 based on [(1, 5, 23), (2, 5, >24), (3, 1, 25)] >Thu Sep 29 08:52:21 UTC 2011: getMaster 1 based on [(1, 5, 23), (2, 5, >24), (3, 1, 25)] >Thu Sep 29 08:52:21 UTC 2011: ReevaluateMyself: machineId=3 with >master[(org.neo4j.kernel.ha.MasterClient@f2ea42, MachineInfo[ID:1, >sequence:23, last committed tx id:5, server:(zoo1, 6001)])] (I am >master=false) >Thu Sep 29 08:52:21 UTC 2011: >org.neo4j.kernel.ha.zookeeper.ZooClient@970c0e, Thu Sep 29 08:52:21 UTC >2011 Got event: WatchedEvent state:SyncConnected type:NodeDataChanged >path:/1317204894718_5015401063079066885/master-rebound(path=/1317204894718 >_ >5015401063079066885/master-rebound) >Thu Sep 29 08:52:21 UTC 2011: master-notify not set, is already 1 >Thu Sep 29 08:52:21 UTC 2011: Internal shutdown of HA db[3] >reference=HighlyAvailableGraphDatabase [/home/ubuntu/db], >masterServer=null >Thu Sep 29 08:52:21 UTC 2011: Opened >[/home/ubuntu/db/nioneo_logical.log.1] clean empty log, version=0 >Thu Sep 29 08:52:21 UTC 2011: Opened [/home/ubuntu/db/index/lucene.log.1] >clean empty log, version=0 >Thu Sep 29 08:52:21 UTC 2011: TM opening log: /home/ubuntu/db/tm_tx_log.1 >Thu Sep 29 08:52:21 UTC 2011: --- CONFIGURATION START --- >Thu Sep 29 08:52:21 UTC 2011: Physical mem: 496MB, Heap size: 121MB >Thu Sep 29 08:52:21 UTC 2011: Kernel version: Neo4j - Graph Database >Kernel 1.4.1 >Thu Sep 29 08:52:21 UTC 2011: Neo4j - Graph Database Kernel 1.4.1 >Thu Sep 29 08:52:21 UTC 2011: Operating System: Linux; version: >2.6.32-30-generic-pae; arch: i386; cpus: 1 >Thu Sep 29 08:52:21 UTC 2011: VM Name: Java HotSpot(TM) Client VM >Thu Sep 29 08:52:21 UTC 2011: VM Vendor: Sun Microsystems Inc. >Thu Sep 29 08:52:21 UTC 2011: VM Version: 20.1-b02 >Thu Sep 29 08:52:21 UTC 2011: Boot Class Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/resources.jar:/usr/lib/jvm/java-6 >- >sun-1.6.0.26/jre/lib/rt.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/sunrs >a >sign.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/jsse.jar:/usr/lib/jvm/ja >v >a-6-sun-1.6.0.26/jre/lib/jce.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/ >c >harsets.jar:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/modules/jdk.boot.jar: >/ >usr/lib/jvm/java-6-sun-1.6.0.26/jre/classes >Thu Sep 29 08:52:21 UTC 2011: Class Path: >/usr/share/tomcat6/bin/bootstrap.jar >Thu Sep 29 08:52:21 UTC 2011: Library Path: >/usr/lib/jvm/java-6-sun-1.6.0.26/jre/lib/i386/client:/usr/lib/jvm/java-6-s >u >n-1.6.0.26/jre/lib/i386:/usr/lib/jvm/java-6-sun-1.6.0.26/jre/../lib/i386:/ >u >sr/java/packages/lib/i386:/lib:/usr/lib >Thu Sep 29 08:52:21 UTC 2011: Garbage Collector: Copy: [Eden Space, >Survivor Space] >Thu Sep 29 08:52:21 UTC 2011: Garbage Collector: MarkSweepCompact: [Eden >Space, Survivor Space, Tenured Gen, Perm Gen, Perm Gen [shared-ro], Perm >Gen [shared-rw]] >Thu Sep 29 08:52:21 UTC 2011: VM Arguments: >[-Djava.util.logging.config.file=/home/ubuntu/tomcat/conf/logging.properti >e >s, -Dserver.id=3, -Dha.server.port=6001, -Ddatabase.path=/home/ubuntu/db, >-Dzookeeper.servers=zoo1:2181,zoo2:2181,zoo3:2181, >-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager, >-Djava.endorsed.dirs=/usr/share/tomcat6/endorsed, >-Dcatalina.base=/home/ubuntu/tomcat, -Dcatalina.home=/usr/share/tomcat6, >-Djava.io.tmpdir=/home/ubuntu/tomcat/temp] >Thu Sep 29 08:52:21 UTC 2011: >Thu Sep 29 08:52:21 UTC 2011: create=true >Thu Sep 29 08:52:21 UTC 2011: enable_remote_shell=port=1331 >Thu Sep 29 08:52:21 UTC 2011: ha.machine_id=3 >Thu Sep 29 08:52:21 UTC 2011: ha.server=zoo1:6001 >Thu Sep 29 08:52:21 UTC 2011: >ha.zoo_keeper_servers=zoo1:2181,zoo2:2181,zoo3:2181 >Thu Sep 29 08:52:21 UTC 2011: keep_logical_logs=true >Thu Sep 29 08:52:21 UTC 2011: >logical_log=/home/ubuntu/db/nioneo_logical.log >Thu Sep 29 08:52:21 UTC 2011: neo_store=/home/ubuntu/db/neostore >Thu Sep 29 08:52:21 UTC 2011: neostore.nodestore.db.mapped_memory=20M >Thu Sep 29 08:52:21 UTC 2011: >neostore.propertystore.db.arrays.mapped_memory=130M >Thu Sep 29 08:52:21 UTC 2011: >neostore.propertystore.db.index.keys.mapped_memory=1M >Thu Sep 29 08:52:21 UTC 2011: >neostore.propertystore.db.index.mapped_memory=1M >Thu Sep 29 08:52:21 UTC 2011: neostore.propertystore.db.mapped_memory=90M >Thu Sep 29 08:52:21 UTC 2011: >neostore.propertystore.db.strings.mapped_memory=130M >Thu Sep 29 08:52:21 UTC 2011: >neostore.relationshipstore.db.mapped_memory=100M >Thu Sep 29 08:52:21 UTC 2011: node_auto_indexing=false >Thu Sep 29 08:52:21 UTC 2011: pull_interval=1 >Thu Sep 29 08:52:21 UTC 2011: rebuild_idgenerators_fast=true >Thu Sep 29 08:52:21 UTC 2011: relationship_auto_indexing=false >Thu Sep 29 08:52:21 UTC 2011: store_dir=/home/ubuntu/db >Thu Sep 29 08:52:21 UTC 2011: use_memory_mapped_buffers=true >Thu Sep 29 08:52:21 UTC 2011: --- CONFIGURATION END --- >Thu Sep 29 08:52:21 UTC 2011: Extension >org.neo4j.kernel.KernelExtension[online backup] loaded ok >Thu Sep 29 08:52:21 UTC 2011: Extension >org.neo4j.kernel.KernelExtension[kernel jmx] loaded ok >Thu Sep 29 08:52:21 UTC 2011: Extension >org.neo4j.kernel.KernelExtension[kernel udc] loaded ok > > > > > > > > > > > > > > > > > > > > > >On 28/09/2011 18:39, "Michael Hunger" <[email protected]> >wrote: > >>Could you also get the /home/ubuntu/db/messages.log of each of the 3 >>instances to see if there are any issues with the HA communication? >> >>Thanks so much, >> >>Michael >> >>Am 28.09.2011 um 18:11 schrieb Toby O'Rourke: >> >>> Hi, >>> >>> I'm not quite sure which property files you're talking about. The >>> configuration passed to HighlyAvailableGraphDatabase comes from the map >>> specified as a constructor argument directly in the spring Context. >>> database.path, server.id, ha.server.port, and zookeeper.servers are >>>set >>>by >>> system properties when tomcat is started so for each server: >>> >>> # VM1 >>> export JAVA_OPTS="-Dserver.id=1 -Dha.server.port=6001 >>> -Ddatabase.path=/home/ubuntu/db >>> -Dzookeeper.servers=zoo1:2181,zoo2:2181,zoo3:2181" >>> # VM2 >>> export JAVA_OPTS="-Dserver.id=2 -Dha.server.port=6001 >>> -Ddatabase.path=/home/ubuntu/db >>> -Dzookeeper.servers=zoo1:2181,zoo2:2181,zoo3:2181" >>> # VM3 >>> export JAVA_OPTS="-Dserver.id=3 -Dha.server.port=6001 >>> -Ddatabase.path=/home/ubuntu/db >>> -Dzookeeper.servers=zoo1:2181,zoo2:2181,zoo3:2181" >>> >>> Written as a properties file, for one server, they would look like >>>this: >>> >>> ha.machine_id=1 >>> ha.server=zoo1:6001 >>> ha.zoo_keeper_servers=zoo1:2181,zoo2:2181,zoo3:2181 >>> enable_remote_shell=port=1331 >>> pull_interval=1 >>> >>> But looking at the sample code on the wiki, I'm just short cutting it a >>> bit by doing it in the Spring Context rather than parsing a properties >>> file in a main method? In terms of the zookeeper config, zoo.cfg looks >>> like this: >>> >>> tickTime=2000 >>> dataDir=/var/lib/zookeeper >>> clientPort=2181 >>> initLimit=5 >>> syncLimit=2 >>> server.1=zoo1:2888:3888 >>> server.2=zoo2:2888:3888 >>> server.3=zoo3:2888:3888 >>> >>> >>> Should I be including some other config? >>> >>> Thanks, >>> >>> Toby. >>> >>> >>> On 28/09/2011 15:53, "Michael Hunger" >>><[email protected]> >>> wrote: >>> >>>> Could you also share the property files of the 3 vm's ? >>>> >>>> Thanks >>>> >>>> Michael >>>> >>>> Am 28.09.2011 um 15:59 schrieb Toby O'Rourke: >>>> >>>>> Hi, >>>>> >>>>> I'm trying to do a POC using neo4j and spring-data-graph. I have >>>>> configured 3 VMs running zookeeper and deployed a simple webapp to >>>>> tomcat on each instance. I've run into a bit of a problem with >>>>> replication though. I can only see data being replicated when I >>>>>restart >>>>> each tomcat instance which is obviously no good. >>>>> >>>>> I have the following in my application context: >>>>> >>>>> <bean id="graphDatabaseService" >>>>> class="org.neo4j.kernel.HighlyAvailableGraphDatabase" >>>>> destroy-method="shutdown" >>>>> scope="singleton"> >>>>> <constructor-arg index="0" value="${database.path}"/> >>>>> <constructor-arg index="1"> >>>>> <map> >>>>> <entry key="ha.machine_id" value="${server.id}" /> >>>>> <entry key="ha.server" value="zoo1:${ha.server.port}" >>>>>/> >>>>> <entry key="ha.zoo_keeper_servers" >>>>> value="${zookeeper.servers}" /> >>>>> <entry key="enable_remote_shell" value="port=1331" /> >>>>> <entry key="pull_interval" value="1" /> >>>>> </map> >>>>> </constructor-arg> >>>>> </bean> >>>>> >>>>> I am starting "zoo1" first so that it becomes the master. Server.id >>>>>is >>>>> allocated at runtime from a system property set on each machine >>>>> Dserver.id=x in the tomcat startup script. From what I understand the >>>>> pull_interval=1 setting means that data should be synchronized once >>>>>per >>>>> second, yet I am only seeing it happen when the servers are bounced. >>>>>Am >>>>> I missing something? >>>>> >>>>> Thanks, >>>>> >>>>> Toby. >>>>> >>>>> _______________________________________________ >>>>> Neo4j mailing list >>>>> [email protected] >>>>> https://lists.neo4j.org/mailman/listinfo/user >>>> >>>> _______________________________________________ >>>> Neo4j mailing list >>>> [email protected] >>>> https://lists.neo4j.org/mailman/listinfo/user >>> >>> _______________________________________________ >>> Neo4j mailing list >>> [email protected] >>> https://lists.neo4j.org/mailman/listinfo/user >> >>_______________________________________________ >>Neo4j mailing list >>[email protected] >>https://lists.neo4j.org/mailman/listinfo/user > >_______________________________________________ >Neo4j mailing list >[email protected] >https://lists.neo4j.org/mailman/listinfo/user _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
