On 04/08/2010 03:20 AM, Johan Svensson wrote: > Hi Patrick, > > Thanks for the feedback. I will have a look at this and implement > handling for disconnection and expiration of sessions. >
No problem. We'll be psyched to see you roll this out. > Regarding the GC issues we are well aware of these (hopefully the new > "garbage first" or G1 GC will solve these problems). As you say the > concurrent mark sweep GC helps a lot but more important to avoid "GC > trashing" is to make sure there is more (10-15%) available heap than > the application ever consumes at any given moment. > Agree. HBase has tested G1 in 1.6.x but so far it is not stable enough for production use. > I do have a question regarding ZooKeeper. Is there a reason why there > is no embedded version? > > Now I have to start two JVMs on each machine when I really just want to: > > // start a zookeeper server on this machine > ZooKeeperServer server = new ZooKeeperServer( 2181 ); > // start a client and pass in some zookeeper servers > ZooKeeper zoo = new ZooKeeper( "localhost:2181, otherhost:2181, ...", ... > ); I'm not sure what you mean by "embedded". In production you typically want to have dedicated hosts for the servers. See this page for some insight http://wiki.apache.org/hadoop/ZooKeeper/Troubleshooting HBase wraps the zk server for their "quickstart" type use cases (in their startup scripts). But in large online production serving environments you typically run a ZK cluster separately from the client application. Patrick > Regards, > -Johan > > On Wed, Apr 7, 2010 at 12:08 AM, Patrick Hunt<ph...@apache.org> wrote: >> Hi, I'm Patrick (http://twitter.com/phunt) from the ZooKeeper team. >> Peter Neubauer brought to my attention today that you are considering >> use of ZooKeeper in Neo4j, that's great! I took a quick look at the code >> you currently have in SVN and wanted to provide a bit of feedback. >> >> I don't know your domain requirements but in general the mechanics of >> ZooClient use look fine. >> >> The use of 5second timeout is fine. This allows you to detect a client >> (zk client) failure after just 5 seconds. So if the node/process crashes >> you'd identify this after 5 seconds, same if a network connection fails, >> etc... One thing you may not have considered though, is that anything >> that causes the client to not be able to heartbeat to the server would >> also cause the session to be expired (sessions are expired when the zk >> cluster fails to hear from the client w/in the "timeout" time) - so long >> GC pauses could trigger this as well. In 1.6.x jvms we've seen that the >> GC can pause all threads for very long periods (in some cases with hbase >> we saw 4 minute pauses for gc). HBase was the first to see this, we >> worked with Solr early on to help them understand this issue as well. >> The problem can be alleviated somewhat by using the CMS/incremental GC >> options in the JVM, however it cannot be eliminated entirely (in some >> cases the Gc will still drop back to parallel). You need to consider the >> impact of GC on your domain and how to best handle it. >> >> See this JIRA for details on our discussion with Solr, you might gain >> some good insight:http://bit.ly/d7OSQ1 >> https://issues.apache.org/jira/browse/SOLR-1277 >> >> I did notice that ZooClient is not handling disconnection and expiration >> of the session in the "process" method. At the very least you need to >> handle the expiration, you may need to do something for disconnection, >> but this depends on whether you have active or passive actors (masters). >> Here's a good link on session lifecycle: >> http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions >> >> >> You might also want to setup a wiki page similar to these at some point, >> it would help us with future discussion, feedback and provide insight >> for devs/users: >> http://wiki.apache.org/hadoop/ZooKeeper/HBaseAndZooKeeper >> http://wiki.apache.org/solr/ZooKeeperIntegration >> >> Regards and good luck, >> >> Patrick > _______________________________________________ > Neo mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user _______________________________________________ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
