[Neo4j] Neo4j High Availability
Hi, I am trying to configure Neo4J High availability. But I am getting some exceptions. The following are the configuration details. We have 3 Linux servers (Amazon cloud instances). Each one has Zookeeper 3.3.2 and Jetty 7. Deployed web application in Jetty by giving corresponding ha-server.conf as follows: ha.machine_id=1 ha.server=localhost:6001 ha.zoo_keeper_servers=internalip:2180, internalip:2180, internalip:2180 ha.pull_interval=5 enable_remote_shell = port=1331 Similary in linux server 2 and 3. Configured Zookeepr as follows: #server1.cfg tickTime=1 initLimit=50 syncLimit=10 dataDir=/usr/share/zookeeper-3.3.2/data/zookeeper1 clientPort=2183 server.1=ip-10-102-53-219.ec2.internal:2888:3888 server.2=ip-10-84-29-207.ec2.internal:2888:3888 server.3=ip-10-102-53-219.ec2.internal:2888:3888 Similary in linux server 2 and 3. Now started Jetty servers and Zookeeper servers. But I am getting the following exception at Web server Jetty servers. [main-SendThread(ipaddtress:2180)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_18] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592) ~[na:1.6.0_18] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.2.jar:3.3.2-1031432] The following exceptions at Zookeeper servers: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:601) 2011-08-23 09:19:30,111 - WARN [Thread-48:QuorumCnxManager$SendWorker@621] - Send worker leaving thread 2011-08-23 09:19:30,113 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967353 (n.zxid), 1 (n.round), FOLLOWING (n.state), 2 (n.sid), LEADING (my state) 2011-08-23 09:20:30,127 - WARN [Thread-51:QuorumCnxManager$RecvWorker@702] - Connection broken for id 2, my id = 3, error = java.nio.channels.AsynchronousCloseException 2011-08-23 09:20:30,127 - WARN [Thread-50:QuorumCnxManager$SendWorker@612] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:601) 2011-08-23 09:20:30,128 - WARN [Thread-50:QuorumCnxManager$SendWorker@621] - Send worker leaving thread 2011-08-23 09:20:30,130 - WARN [Thread-52:QuorumCnxManager$SendWorker@612] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:601) 2011-08-23 09:20:30,130 - WARN [Thread-52:QuorumCnxManager$SendWorker@621] - Send worker leaving thread 2011-08-23 09:20:30,134 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967353 (n.zxid), 1 (n.round), FOLLOWING (n.state), 2 (n.sid), LEADING (my state) 2011-08-23 09:21:30,145 - WARN [Thread-54:QuorumCnxManager$SendWorker@612] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:601) 2011-08-23 09:21:30,147 - WARN [Thread-54:QuorumCnxManager$SendWorker@621] - Send worker leaving thread 2011-08-23 09:21:30,149 - WARN [Thread-55:QuorumCnxManager$RecvWorker@702] - Connection broken
Re: [Neo4j] Neo4j High Availability
Madan, just talked to Mattias on this. Zookeeper is a strange beast and casting exceptions often - what is the actual effect you are seeing - is the cluster not starting up or are there multiple masters? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Tue, Aug 23, 2011 at 11:26 AM, Madan Gubbala mgubb...@allianceglobalservices.com wrote: Hi, I am trying to configure Neo4J High availability. But I am getting some exceptions. The following are the configuration details. We have 3 Linux servers (Amazon cloud instances). Each one has Zookeeper 3.3.2 and Jetty 7. Deployed web application in Jetty by giving corresponding ha-server.conf as follows: ha.machine_id=1 ha.server=localhost:6001 ha.zoo_keeper_servers=internalip:2180, internalip:2180, internalip:2180 ha.pull_interval=5 enable_remote_shell = port=1331 Similary in linux server 2 and 3. Configured Zookeepr as follows: #server1.cfg tickTime=1 initLimit=50 syncLimit=10 dataDir=/usr/share/zookeeper-3.3.2/data/zookeeper1 clientPort=2183 server.1=ip-10-102-53-219.ec2.internal:2888:3888 server.2=ip-10-84-29-207.ec2.internal:2888:3888 server.3=ip-10-102-53-219.ec2.internal:2888:3888 Similary in linux server 2 and 3. Now started Jetty servers and Zookeeper servers. But I am getting the following exception at Web server Jetty servers. [main-SendThread(ipaddtress:2180)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.6.0_18] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592) ~[na:1.6.0_18] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.2.jar:3.3.2-1031432] The following exceptions at Zookeeper servers: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:601) 2011-08-23 09:19:30,111 - WARN [Thread-48:QuorumCnxManager$SendWorker@621] - Send worker leaving thread 2011-08-23 09:19:30,113 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967353 (n.zxid), 1 (n.round), FOLLOWING (n.state), 2 (n.sid), LEADING (my state) 2011-08-23 09:20:30,127 - WARN [Thread-51:QuorumCnxManager$RecvWorker@702] - Connection broken for id 2, my id = 3, error = java.nio.channels.AsynchronousCloseException 2011-08-23 09:20:30,127 - WARN [Thread-50:QuorumCnxManager$SendWorker@612] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:601) 2011-08-23 09:20:30,128 - WARN [Thread-50:QuorumCnxManager$SendWorker@621] - Send worker leaving thread 2011-08-23 09:20:30,130 - WARN [Thread-52:QuorumCnxManager$SendWorker@612] - Interrupted while waiting for message on queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094) at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370) at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:601) 2011-08-23 09:20:30,130 - WARN [Thread-52:QuorumCnxManager$SendWorker@621] - Send worker leaving thread 2011-08-23 09:20:30,134 - INFO [WorkerReceiver Thread:FastLeaderElection@496] - Notification: 3 (n.leader), 4294967353 (n.zxid), 1 (n.round), FOLLOWING (n.state), 2 (n.sid), LEADING (my state) 2011-08-23 09:21:30,145 - WARN
Re: [Neo4j] Neo4J High Availability : embedded or server mode ?
Yes, that was my understanding too (probably because of other embedded dbs like JavaDB etc). But from the responses of Jim and others it looks like Neo4j can handle multiple connections. That's great and NOW HA in embedded makes sense :) On Fri, Mar 25, 2011 at 1:26 PM, Clement Honore honor...@gmail.com wrote: If I've well understood, you can have only one connection to the database for each VM. So, the service must be encapsulated in a singleton I guess. How many transactions each connection can handle ? 2011/3/25 Guru GV guru...@gmail.com Though I don't see a reason not to support it, but I did not understand the point of HA in a embedded mode. Would be interested in hearing couple of examples of what that would be... Embedded would mean - same VM - so replication and concurrency - do they really mean much here ? Also, that brings me to another question - is the Embedded version of Neo4j concurrent for multiple threads ? Meaning can it have multiple simultaneous connections/transactions ongoing ? Thanks Guru On Thu, Mar 24, 2011 at 4:27 PM, Jim Webber j...@neotechnology.com wrote: Hi Kiiv, Both the embedded database *and* the server database can run in HA mode. We tend to think of the server as simply a wrapper around the database that provides a RESTful API to augment the local API. So go right ahead and pick the configuration that's right for you, and you'll be able to use HA whatever configuration you decide. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4J High Availability : embedded or server mode ?
If I've well understood, you can have only one connection to the database for each VM. So, the service must be encapsulated in a singleton I guess. How many transactions each connection can handle ? 2011/3/25 Guru GV guru...@gmail.com Though I don't see a reason not to support it, but I did not understand the point of HA in a embedded mode. Would be interested in hearing couple of examples of what that would be... Embedded would mean - same VM - so replication and concurrency - do they really mean much here ? Also, that brings me to another question - is the Embedded version of Neo4j concurrent for multiple threads ? Meaning can it have multiple simultaneous connections/transactions ongoing ? Thanks Guru On Thu, Mar 24, 2011 at 4:27 PM, Jim Webber j...@neotechnology.com wrote: Hi Kiiv, Both the embedded database *and* the server database can run in HA mode. We tend to think of the server as simply a wrapper around the database that provides a RESTful API to augment the local API. So go right ahead and pick the configuration that's right for you, and you'll be able to use HA whatever configuration you decide. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4J High Availability : embedded or server mode ?
Hi folks, Embedded simply means running in your choice of JVM process. Most embedded Neo4j instances aren't embedded in the hardware sense of embedded systems, they're usually server applications that happen to use Neo4j as their database (sensible choice!). So imagine I have a Web app set to run over several Web servers, each of which has an embedded Neo4j instance. On failure of any of those Web servers, I want both the Web app and the underlying database to continue running. That's when the HA stuff kicks in. It doesn't matter whether you live in a process and are wrapped in your own code, or if you're wrapped in Neo4j-server code, you still want continuity. So to summarise, most embedded database use cases have the database embedded on a server. There's also neo4j server if you want to wrap your embedded database in a remote-able API. Easy, right? :-) Jim On 25 Mar 2011, at 07:56, Clement Honore wrote: If I've well understood, you can have only one connection to the database for each VM. So, the service must be encapsulated in a singleton I guess. How many transactions each connection can handle ? 2011/3/25 Guru GV guru...@gmail.com Though I don't see a reason not to support it, but I did not understand the point of HA in a embedded mode. Would be interested in hearing couple of examples of what that would be... Embedded would mean - same VM - so replication and concurrency - do they really mean much here ? Also, that brings me to another question - is the Embedded version of Neo4j concurrent for multiple threads ? Meaning can it have multiple simultaneous connections/transactions ongoing ? Thanks Guru On Thu, Mar 24, 2011 at 4:27 PM, Jim Webber j...@neotechnology.com wrote: Hi Kiiv, Both the embedded database *and* the server database can run in HA mode. We tend to think of the server as simply a wrapper around the database that provides a RESTful API to augment the local API. So go right ahead and pick the configuration that's right for you, and you'll be able to use HA whatever configuration you decide. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4J High Availability : embedded or server mode ?
Also, that brings me to another question - is the Embedded version of Neo4j concurrent for multiple threads ? Meaning can it have multiple simultaneous connections/transactions ongoing ? The embedded version of neo4j typically works *better* when there are multiple threads interacting with it. That's how you get high aggregate throughput. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4J High Availability : embedded or server mode ?
2011/3/25 Guru GV guru...@gmail.com Though I don't see a reason not to support it, but I did not understand the point of HA in a embedded mode. Would be interested in hearing couple of examples of what that would be... Embedded would mean - same VM - so replication and concurrency - do they really mean much here ? Also, that brings me to another question - is the Embedded version of Neo4j concurrent for multiple threads ? Meaning can it have multiple simultaneous connections/transactions ongoing ? HA in embedded mode doesn't mean multiple database instances in one JVM. It means running the embedded mode (without the server/REST parts) for each database instance in the cluster, each instance will usually reside on different machines. Using embedded mode of neo4j will get you higher throughput (the REST layer is stripped off) since your application runs in the same JVM as the database. It's of course highly concurrenct also, equally to or more than going through REST. Thanks Guru On Thu, Mar 24, 2011 at 4:27 PM, Jim Webber j...@neotechnology.com wrote: Hi Kiiv, Both the embedded database *and* the server database can run in HA mode. We tend to think of the server as simply a wrapper around the database that provides a RESTful API to augment the local API. So go right ahead and pick the configuration that's right for you, and you'll be able to use HA whatever configuration you decide. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Neo4J High Availability : embedded or server mode ?
Hi, I'm new to Neo4J and I'm a little bit confused (maybe totally lost...) about the use of the Neo4J server against an embedded database in high availability mode... I need a database with high availability, replication and high concurrency management accessed as a remote server. If I have well understood, Neo4J server only give an access to the database through a REST API, which didn't implement all of the capabilities of the embedded database with full Java access. Neo4J server obviously could be run in high availability mode. BUT embedded database should to ! But for me, the principle of clustering and replication is inherent to a server mode ! So, if I make an embedded database in HA mode, I'll have a database that can be used like a remote server but with full Java access (with HighlyAvailableGraphDatabase class) ? So what's the advantage of using Neo4J server anymore ? :/ Sorry if I didn't understand anything but after some days of reading and reading again all the documentation and wiki pages, I don't know what to do... Thanks for explanations ! Kiiv PS : sorry if my english is not very understandable, I can try to explain things differently if needed ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4J High Availability : embedded or server mode ?
Hi Kiiv, Both the embedded database *and* the server database can run in HA mode. We tend to think of the server as simply a wrapper around the database that provides a RESTful API to augment the local API. So go right ahead and pick the configuration that's right for you, and you'll be able to use HA whatever configuration you decide. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4J High Availability : embedded or server mode ?
Hi Jim, thanks for your answer. So the Neo4J server and REST API are principaly interesting for non supported langages but with Java you confirm that the embedded database in high availability mode act like a standalone server. Nice ;) Kiiv 2011/3/24 Jim Webber j...@neotechnology.com Hi Kiiv, Both the embedded database *and* the server database can run in HA mode. We tend to think of the server as simply a wrapper around the database that provides a RESTful API to augment the local API. So go right ahead and pick the configuration that's right for you, and you'll be able to use HA whatever configuration you decide. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Neo4J High Availability : embedded or server mode ?
Though I don't see a reason not to support it, but I did not understand the point of HA in a embedded mode. Would be interested in hearing couple of examples of what that would be... Embedded would mean - same VM - so replication and concurrency - do they really mean much here ? Also, that brings me to another question - is the Embedded version of Neo4j concurrent for multiple threads ? Meaning can it have multiple simultaneous connections/transactions ongoing ? Thanks Guru On Thu, Mar 24, 2011 at 4:27 PM, Jim Webber j...@neotechnology.com wrote: Hi Kiiv, Both the embedded database *and* the server database can run in HA mode. We tend to think of the server as simply a wrapper around the database that provides a RESTful API to augment the local API. So go right ahead and pick the configuration that's right for you, and you'll be able to use HA whatever configuration you decide. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
