Re: Embedding ZK in another application
Bjorn Borud writes: > > a good reason to want to embed ZooKeeper is for unit tests. I had a > look at the tests that came with ZooKeeper and I am not overly fond of > having to embed a ZooKeeper ensemble by fiddling with the implementation > classes. also, I think it is a bad idea to base a project's test on > extending test base-classes. this was okay in JUnit 3, but if you are > used to JUnit 4 it is a bit ugly. clarification: when I say "a project's test" I didn't mean ZooKeeper but whichever project I need to write tests for with an embedded ensemble of ZK servers. -Bjørn
Re: Embedding ZK in another application
Patrick Hunt writes: > See the quorum tests for examples of how to do this, in particular > take a look at org.apache.zookeeper.test.QuorumBase which is how we > start/stop quorum servers in some of our tests. Our tests can run in > "fork once" mode in junit. Meaning that all the tests > (starting/stopping servers frequently) run in the same jvm, your > embedded case. a good reason to want to embed ZooKeeper is for unit tests. I had a look at the tests that came with ZooKeeper and I am not overly fond of having to embed a ZooKeeper ensemble by fiddling with the implementation classes. also, I think it is a bad idea to base a project's test on extending test base-classes. this was okay in JUnit 3, but if you are used to JUnit 4 it is a bit ugly. when I run embedded servers in unit tests I allocate both temporary directories and unique ports as part of the test setup. for the most(*) part I use the @Rule TemporaryFolder for creating temporary directories that are cleaned up when the test is done. ports are allocated by a utility I've written that reduces the chance of grabbing ports that are already in use. it would be beneficial if it was possible to configure ZooKeeperServer instances and have them work together in an ensemble (within a JVM) without having to resort to hacking the implementation classes or extending test base-classes (yech!). (*) for versions of JUnit I've found in Maven Central @Rule TemporaryFolder doesn't work properly with @BeforeClass, so it isn't possible to use @Rule TemporaryFolder to manage temporary folders that are needed by broken server classes (some servers have lots of badly designed singletons that requires you to run tests in separate threads). In these cases I use a somewhat inferior test utility that basically does the same as TemporaryFolder. -Bjørn
Re: Embedding ZK in another application
On 04/29/2010 11:22 AM, Chad Harrington wrote: On Thu, Apr 29, 2010 at 8:49 AM, Patrick Hunt wrote: This is not foolproof however. We found that in general this would work, however there were some infrequent cases where a restarted server would fail to initialize due to the following issue: "it is possible for the process to complete before the kernel has released the associated network resource, and this port cannot be bound to another process until the kernel has decided that it is done." more detail here: http://hea-www.harvard.edu/~fine/Tech/addrinuse.html as a result we ended up changing the test code to start each test with new quorum/election port numbers. This fixed the problem for us but would not be a solution in your case. Patrick I am not an expert at all on this, but I have used SO_REUSEADDR in other situations to avoid the address in use problem. Would that help here? Hi Chad, seems like it should, but according to the link "Oddly, using SO_REUSEADDR can actually lead to more difficult "address already in use" errors." See the "strategies for avoidance" section. I'm no expert on this particular issue either, so if someone knows for sure speak up. We already do this btw, this is from the server... (not sure if we cover all cases though): ss.socket().setReuseAddress(true); Patrick Chad Harrington chad.harring...@gmail.com On 04/29/2010 07:13 AM, Vishal K wrote: Hi Ted, We want the application that embeds the ZK server to be running even after the ZK server is shutdown. So we don't want to restart the application. Also, we prefer not to use zkServer.sh/zkServer.cmd because these are OS dependent (our application will run on Win as well as Linux). Instead, we thought that calling QuorumPeerMain.initializeAndRun() and QuorumPeerMain.shutdown() will suffice to start and shutdown a ZK server and we won't have to worry about checking the OS. Is there way to cleanly shutdown the ZK server (by invoking ZK server API) when it is embedded in the application without actually restarting the application process? Thanks. On Thu, Apr 29, 2010 at 1:54 AM, Ted Dunning wrote: Hmmm it isn't quite clear what you mean by restart without restarting. Why is killing the server and restarting it not an option? It is common to do a rolling restart on a ZK cluster. Just restart one server at a time. This is often used during system upgrades. On Wed, Apr 28, 2010 at 8:22 PM, Vishal K wrote: What is a good way to restart a ZK server (standalone and quorum) without having to restart it? Currently, I have ZK server embedded in another java application.
Re: Embedding ZK in another application
We do set that Chad but it doesn't seem to help on some systems (especially bsd)... Thanks mahadev On 4/29/10 11:22 AM, "Chad Harrington" wrote: > On Thu, Apr 29, 2010 at 8:49 AM, Patrick Hunt wrote: > >> This is not foolproof however. We found that in general this would work, >> however there were some infrequent cases where a restarted server would fail >> to initialize due to the following issue: >> "it is possible for the process to complete before the kernel has released >> the associated network resource, and this port cannot be bound to another >> process until the kernel has decided that it is done." >> >> more detail here: >> http://hea-www.harvard.edu/~fine/Tech/addrinuse.html >> >> as a result we ended up changing the test code to start each test with new >> quorum/election port numbers. This fixed the problem for us but would not be >> a solution in your case. >> >> Patrick > > > I am not an expert at all on this, but I have used SO_REUSEADDR in other > situations to avoid the address in use problem. Would that help here? > > Chad Harrington > chad.harring...@gmail.com > >> >> >> On 04/29/2010 07:13 AM, Vishal K wrote: >> >>> Hi Ted, >>> >>> We want the application that embeds the ZK server to be running even after >>> the ZK server is shutdown. So we don't want to restart the application. >>> Also, we prefer not to use zkServer.sh/zkServer.cmd because these are OS >>> dependent (our application will run on Win as well as Linux). Instead, we >>> thought that calling QuorumPeerMain.initializeAndRun() and >>> QuorumPeerMain.shutdown() will suffice to start and shutdown a ZK server >>> and >>> we won't have to worry about checking the OS. >>> >>> Is there way to cleanly shutdown the ZK server (by invoking ZK server API) >>> when it is embedded in the application without actually restarting the >>> application process? >>> Thanks. >>> On Thu, Apr 29, 2010 at 1:54 AM, Ted Dunning >>> wrote: >>> >>> Hmmm it isn't quite clear what you mean by restart without restarting. Why is killing the server and restarting it not an option? It is common to do a rolling restart on a ZK cluster. Just restart one server at a time. This is often used during system upgrades. On Wed, Apr 28, 2010 at 8:22 PM, Vishal K wrote: > What is a good way to restart a ZK server (standalone and quorum) > without > having to restart it? > > Currently, I have ZK server embedded in another java application. > >>>
Re: Embedding ZK in another application
On Thu, Apr 29, 2010 at 8:49 AM, Patrick Hunt wrote: > This is not foolproof however. We found that in general this would work, > however there were some infrequent cases where a restarted server would fail > to initialize due to the following issue: > "it is possible for the process to complete before the kernel has released > the associated network resource, and this port cannot be bound to another > process until the kernel has decided that it is done." > > more detail here: > http://hea-www.harvard.edu/~fine/Tech/addrinuse.html > > as a result we ended up changing the test code to start each test with new > quorum/election port numbers. This fixed the problem for us but would not be > a solution in your case. > > Patrick I am not an expert at all on this, but I have used SO_REUSEADDR in other situations to avoid the address in use problem. Would that help here? Chad Harrington chad.harring...@gmail.com > > > On 04/29/2010 07:13 AM, Vishal K wrote: > >> Hi Ted, >> >> We want the application that embeds the ZK server to be running even after >> the ZK server is shutdown. So we don't want to restart the application. >> Also, we prefer not to use zkServer.sh/zkServer.cmd because these are OS >> dependent (our application will run on Win as well as Linux). Instead, we >> thought that calling QuorumPeerMain.initializeAndRun() and >> QuorumPeerMain.shutdown() will suffice to start and shutdown a ZK server >> and >> we won't have to worry about checking the OS. >> >> Is there way to cleanly shutdown the ZK server (by invoking ZK server API) >> when it is embedded in the application without actually restarting the >> application process? >> Thanks. >> On Thu, Apr 29, 2010 at 1:54 AM, Ted Dunning >> wrote: >> >> Hmmm it isn't quite clear what you mean by restart without >>> restarting. >>> >>> Why is killing the server and restarting it not an option? >>> >>> It is common to do a rolling restart on a ZK cluster. Just restart one >>> server at a time. This is often used during system upgrades. >>> >>> On Wed, Apr 28, 2010 at 8:22 PM, Vishal K wrote: >>> >>> What is a good way to restart a ZK server (standalone and quorum) without having to restart it? Currently, I have ZK server embedded in another java application. >>> >>> >>
Re: Embedding ZK in another application
See the quorum tests for examples of how to do this, in particular take a look at org.apache.zookeeper.test.QuorumBase which is how we start/stop quorum servers in some of our tests. Our tests can run in "fork once" mode in junit. Meaning that all the tests (starting/stopping servers frequently) run in the same jvm, your embedded case. This is not foolproof however. We found that in general this would work, however there were some infrequent cases where a restarted server would fail to initialize due to the following issue: "it is possible for the process to complete before the kernel has released the associated network resource, and this port cannot be bound to another process until the kernel has decided that it is done." more detail here: http://hea-www.harvard.edu/~fine/Tech/addrinuse.html as a result we ended up changing the test code to start each test with new quorum/election port numbers. This fixed the problem for us but would not be a solution in your case. Patrick On 04/29/2010 07:13 AM, Vishal K wrote: Hi Ted, We want the application that embeds the ZK server to be running even after the ZK server is shutdown. So we don't want to restart the application. Also, we prefer not to use zkServer.sh/zkServer.cmd because these are OS dependent (our application will run on Win as well as Linux). Instead, we thought that calling QuorumPeerMain.initializeAndRun() and QuorumPeerMain.shutdown() will suffice to start and shutdown a ZK server and we won't have to worry about checking the OS. Is there way to cleanly shutdown the ZK server (by invoking ZK server API) when it is embedded in the application without actually restarting the application process? Thanks. On Thu, Apr 29, 2010 at 1:54 AM, Ted Dunning wrote: Hmmm it isn't quite clear what you mean by restart without restarting. Why is killing the server and restarting it not an option? It is common to do a rolling restart on a ZK cluster. Just restart one server at a time. This is often used during system upgrades. On Wed, Apr 28, 2010 at 8:22 PM, Vishal K wrote: What is a good way to restart a ZK server (standalone and quorum) without having to restart it? Currently, I have ZK server embedded in another java application.
Re: Embedding ZK in another application
Hi, Well looks like FastLeaderElection.shutdown() is not invoked. This has been in 3.3.0. Should have checked on that earlier :-) On Thu, Apr 29, 2010 at 10:13 AM, Vishal K wrote: > Hi Ted, > > We want the application that embeds the ZK server to be running even after > the ZK server is shutdown. So we don't want to restart the application. > Also, we prefer not to use zkServer.sh/zkServer.cmd because these are OS > dependent (our application will run on Win as well as Linux). Instead, we > thought that calling QuorumPeerMain.initializeAndRun() and > QuorumPeerMain.shutdown() will suffice to start and shutdown a ZK server and > we won't have to worry about checking the OS. > > Is there way to cleanly shutdown the ZK server (by invoking ZK server API) > when it is embedded in the application without actually restarting the > application process? > Thanks. > On Thu, Apr 29, 2010 at 1:54 AM, Ted Dunning wrote: > >> Hmmm it isn't quite clear what you mean by restart without restarting. >> >> Why is killing the server and restarting it not an option? >> >> It is common to do a rolling restart on a ZK cluster. Just restart one >> server at a time. This is often used during system upgrades. >> >> On Wed, Apr 28, 2010 at 8:22 PM, Vishal K wrote: >> >> > >> > What is a good way to restart a ZK server (standalone and quorum) >> without >> > having to restart it? >> > >> > Currently, I have ZK server embedded in another java application. >> > >
Re: Embedding ZK in another application
Hi Ted, We want the application that embeds the ZK server to be running even after the ZK server is shutdown. So we don't want to restart the application. Also, we prefer not to use zkServer.sh/zkServer.cmd because these are OS dependent (our application will run on Win as well as Linux). Instead, we thought that calling QuorumPeerMain.initializeAndRun() and QuorumPeerMain.shutdown() will suffice to start and shutdown a ZK server and we won't have to worry about checking the OS. Is there way to cleanly shutdown the ZK server (by invoking ZK server API) when it is embedded in the application without actually restarting the application process? Thanks. On Thu, Apr 29, 2010 at 1:54 AM, Ted Dunning wrote: > Hmmm it isn't quite clear what you mean by restart without restarting. > > Why is killing the server and restarting it not an option? > > It is common to do a rolling restart on a ZK cluster. Just restart one > server at a time. This is often used during system upgrades. > > On Wed, Apr 28, 2010 at 8:22 PM, Vishal K wrote: > > > > > What is a good way to restart a ZK server (standalone and quorum) without > > having to restart it? > > > > Currently, I have ZK server embedded in another java application. >
Re: Embedding ZK in another application
Hmmm it isn't quite clear what you mean by restart without restarting. Why is killing the server and restarting it not an option? It is common to do a rolling restart on a ZK cluster. Just restart one server at a time. This is often used during system upgrades. On Wed, Apr 28, 2010 at 8:22 PM, Vishal K wrote: > > What is a good way to restart a ZK server (standalone and quorum) without > having to restart it? > > Currently, I have ZK server embedded in another java application.
Re: Embedding ZK in another application
Hi, What is a good way to restart a ZK server (standalone and quorum) without having to restart it? Currently, I have ZK server embedded in another java application. The application is supposed to handle (among other things) start/stop of ZK server. I am using ZK 3.2.2, but can certainly move to 3.3.0. The application extends QuorumPeerMain and ZooKeeperServerMain and invokes initializeAndRun() and shutdown() methods for start and stop, respectively. The problem is that shutdown() does clean up everything. I still see all the listener threads active waiting in accept(). Any suggestions how I can restart without restarting the application? Thanks. On Sun, Apr 25, 2010 at 2:52 PM, Vishal K wrote: > Hi Mahadev, Ted, > > Thanks for the feedback. > > > On Fri, Apr 23, 2010 at 3:02 PM, Ted Dunning wrote: > >> It is, of course, your decision, but a key coordination function is to >> determine whether your application is up or not. That is very hard to do >> if >> Zookeeper is inside your application. >> >> On Fri, Apr 23, 2010 at 10:28 AM, Asankha C. Perera > >wrote: >> >> > However, I believe that both the above are fine to live with for the >> > application under consideration, as ZK will be used only to coordinate >> > the larger application. Is there anything else that needs to be >> > considered - and can I safely shutdown the clientPort since the >> > application is always in the same JVM - but, if I do that how would I >> > "connect" to ZK thereafter ? >> > >> > >
Re: Embedding ZK in another application
Hi Mahadev, Ted, Thanks for the feedback. On Fri, Apr 23, 2010 at 3:02 PM, Ted Dunning wrote: > It is, of course, your decision, but a key coordination function is to > determine whether your application is up or not. That is very hard to do > if > Zookeeper is inside your application. > > On Fri, Apr 23, 2010 at 10:28 AM, Asankha C. Perera >wrote: > > > However, I believe that both the above are fine to live with for the > > application under consideration, as ZK will be used only to coordinate > > the larger application. Is there anything else that needs to be > > considered - and can I safely shutdown the clientPort since the > > application is always in the same JVM - but, if I do that how would I > > "connect" to ZK thereafter ? > > >
Re: Embedding ZK in another application
It is, of course, your decision, but a key coordination function is to determine whether your application is up or not. That is very hard to do if Zookeeper is inside your application. On Fri, Apr 23, 2010 at 10:28 AM, Asankha C. Perera wrote: > However, I believe that both the above are fine to live with for the > application under consideration, as ZK will be used only to coordinate > the larger application. Is there anything else that needs to be > considered - and can I safely shutdown the clientPort since the > application is always in the same JVM - but, if I do that how would I > "connect" to ZK thereafter ? >
Re: Embedding ZK in another application
That's true! Thanks mahadev On 4/23/10 11:41 AM, "Asankha C. Perera" wrote: > Hi Mahadev >> I think Ted and Pat had somewhat comentted on this before. >> >> Reiterating these comments below. If you are ok with these points I see no >> concern in ZooKeeper as an embedded application... > > Thanks, I missed this on the archives, and it helps!.. > > I guess if we still decide to embed, the only way to connect to ZK is > still with the normal TCP client.. > > cheers > asankha
Re: Embedding ZK in another application
Hi Mahadev > I think Ted and Pat had somewhat comentted on this before. > > Reiterating these comments below. If you are ok with these points I see no > concern in ZooKeeper as an embedded application... Thanks, I missed this on the archives, and it helps!.. I guess if we still decide to embed, the only way to connect to ZK is still with the normal TCP client.. cheers asankha
Re: Embedding ZK in another application
Hi Vishal and Ashanka, I think Ted and Pat had somewhat comentted on this before. Reiterating these comments below. If you are ok with these points I see no concern in ZooKeeper as an embedded application. Also, as Pat mentioned earlier there are some cases where the server code will "system.exit". This is typically only if quorum communication fails in some weird, unrecoverable way. We have removed most of these but there are a few still remaining. --- Comments by Ted I can't comment on the details of your code (but I have run in-process ZK's in the past without problem) Operationally, however, this isn't a great idea. The problem is two-fold: a) firstly, somebody would probably like to look at Zookeeper to understand the state of your service. If the service is down, then ZK will go away. That means that Zookeeper can't be used that way and is mild to moderate on the logarithmic international suckitude scale. b) secondly, if you want to upgrade your server without upgrading Zookeeper then you still have to bounce Zookeeper. This is probably not a problem, but it can be a slight pain. c) thirdly, you can't scale your service independently of how you scale Zookeeper. This may or may not bother you, but it would bother me. d) fourthly, you will be synchronizing your server restarts with ZK's service restarts. Moving these events away from each other is likely to make them slightly more reliable. There is no failure mode that I know of that would be tickled here, but your service code will be slightly more complex since it has to make sure that ZK is up before it does stuff. If you could make the assumption that ZK is up or exit, that would be simpler. e) yes, I know that is more than two issues. That is itself an issue since any design where the number of worries is increasing so fast is suspect on larger grounds. If there are small problems cropping up at that rate, the likelihood of there being a large problem that comes up seems higher. On 4/23/10 11:04 AM, "Vishal K" wrote: > Hi, > > Good question. We are planning to do something similar as well and it will > great to know if there are any issues with embedding ZK server into an app. > We simply use QourumPeerMain and QourumPeer from our app to start/stop the > ZK server. Is this not a good way to do it? > > On Fri, Apr 23, 2010 at 1:28 PM, Asankha C. Perera wrote: > >> Hi All >> >> I'm very new to ZK, and am looking at embeding ZK into an app that needs >> cluster management - and the objective is to use ZK to notify >> application cluster control operations (e.g. shutdown etc) across nodes. >> >> I came across this post [1] from the user list by Ted Dunning from some >> months back : >> "My experience with Katta has led me to believe that embedding a ZK in a >> product is almost always a bad idea. - The problems are that you can't >> administer the Zookeeper cluster independently and that the cluster >> typically goes down when the associated service goes down." >> >> However, I believe that both the above are fine to live with for the >> application under consideration, as ZK will be used only to coordinate >> the larger application. Is there anything else that needs to be >> considered - and can I safely shutdown the clientPort since the >> application is always in the same JVM - but, if I do that how would I >> "connect" to ZK thereafter ? >> >> thanks and regards >> asankha >> >> [1] http://markmail.org/message/tjonwec7p7dhfpms >>
Re: Embedding ZK in another application
Hi, Good question. We are planning to do something similar as well and it will great to know if there are any issues with embedding ZK server into an app. We simply use QourumPeerMain and QourumPeer from our app to start/stop the ZK server. Is this not a good way to do it? On Fri, Apr 23, 2010 at 1:28 PM, Asankha C. Perera wrote: > Hi All > > I'm very new to ZK, and am looking at embeding ZK into an app that needs > cluster management - and the objective is to use ZK to notify > application cluster control operations (e.g. shutdown etc) across nodes. > > I came across this post [1] from the user list by Ted Dunning from some > months back : > "My experience with Katta has led me to believe that embedding a ZK in a > product is almost always a bad idea. - The problems are that you can't > administer the Zookeeper cluster independently and that the cluster > typically goes down when the associated service goes down." > > However, I believe that both the above are fine to live with for the > application under consideration, as ZK will be used only to coordinate > the larger application. Is there anything else that needs to be > considered - and can I safely shutdown the clientPort since the > application is always in the same JVM - but, if I do that how would I > "connect" to ZK thereafter ? > > thanks and regards > asankha > > [1] http://markmail.org/message/tjonwec7p7dhfpms >
Embedding ZK in another application
Hi All I'm very new to ZK, and am looking at embeding ZK into an app that needs cluster management - and the objective is to use ZK to notify application cluster control operations (e.g. shutdown etc) across nodes. I came across this post [1] from the user list by Ted Dunning from some months back : "My experience with Katta has led me to believe that embedding a ZK in a product is almost always a bad idea. - The problems are that you can't administer the Zookeeper cluster independently and that the cluster typically goes down when the associated service goes down." However, I believe that both the above are fine to live with for the application under consideration, as ZK will be used only to coordinate the larger application. Is there anything else that needs to be considered - and can I safely shutdown the clientPort since the application is always in the same JVM - but, if I do that how would I "connect" to ZK thereafter ? thanks and regards asankha [1] http://markmail.org/message/tjonwec7p7dhfpms
