Re: Embedding ZK in another application

2010-11-02 Thread Bjorn Borud
Patrick Hunt ph...@apache.org writes:

 See the quorum tests for examples of how to do this, in particular
 take a look at org.apache.zookeeper.test.QuorumBase which is how we
 start/stop quorum servers in some of our tests. Our tests can run in
 fork once mode in junit. Meaning that all the tests
 (starting/stopping servers frequently) run in the same jvm, your
 embedded case.

a good reason to want to embed ZooKeeper is for unit tests.  I had a
look at the tests that came with ZooKeeper and I am not overly fond of
having to embed a ZooKeeper ensemble by fiddling with the implementation
classes.  also, I think it is a bad idea to base a project's test on
extending test base-classes.  this was okay in JUnit 3, but if you are
used to JUnit 4 it is a bit ugly.

when I run embedded servers in unit tests I allocate both temporary
directories and unique ports as part of the test setup.  for the most(*)
part I use the @Rule TemporaryFolder for creating temporary directories
that are cleaned up when the test is done.  ports are allocated by a
utility I've written that reduces the chance of grabbing ports that are
already in use.

it would be beneficial if it was possible to configure ZooKeeperServer
instances and have them work together in an ensemble (within a JVM)
without having to resort to hacking the implementation classes or
extending test base-classes (yech!).

(*) for versions of JUnit I've found in Maven Central 
@Rule TemporaryFolder doesn't work properly with @BeforeClass, so it
isn't possible to use @Rule TemporaryFolder to manage temporary folders
that are needed by broken server classes (some servers have lots of
badly designed singletons that requires you to run tests in separate
threads).  In these cases I use a somewhat inferior test utility that
basically does the same as TemporaryFolder.

-Bjørn



Re: Embedding ZK in another application

2010-11-02 Thread Bjorn Borud
Bjorn Borud bbo...@gmail.com writes:

 a good reason to want to embed ZooKeeper is for unit tests.  I had a
 look at the tests that came with ZooKeeper and I am not overly fond of
 having to embed a ZooKeeper ensemble by fiddling with the implementation
 classes.  also, I think it is a bad idea to base a project's test on
 extending test base-classes.  this was okay in JUnit 3, but if you are
 used to JUnit 4 it is a bit ugly.

clarification: when I say a project's test I didn't mean ZooKeeper but
whichever project I need to write tests for with an embedded ensemble of
ZK servers.

-Bjørn



Re: Embedding ZK in another application

2010-04-29 Thread Vishal K
Hi,

Well looks like FastLeaderElection.shutdown() is not invoked. This has been
in 3.3.0. Should have checked on that earlier :-)

On Thu, Apr 29, 2010 at 10:13 AM, Vishal K vishalm...@gmail.com wrote:

 Hi Ted,

 We want the application that embeds the ZK server to be running even after
 the ZK server is shutdown. So we don't want to restart the application.
 Also, we prefer not to use zkServer.sh/zkServer.cmd because these are OS
 dependent (our application will run on Win as well as Linux). Instead, we
 thought that calling QuorumPeerMain.initializeAndRun() and
 QuorumPeerMain.shutdown() will suffice to start and shutdown a ZK server and
 we won't have to worry about checking the OS.

 Is there way to cleanly shutdown the ZK server (by invoking ZK server API)
 when it is embedded in the application without actually restarting the
 application process?
 Thanks.
 On Thu, Apr 29, 2010 at 1:54 AM, Ted Dunning ted.dunn...@gmail.comwrote:

 Hmmm it isn't quite clear what you mean by restart without restarting.

 Why is killing the server and restarting it not an option?

 It is common to do a rolling restart on a ZK cluster.  Just restart one
 server at a time.  This is often used during system upgrades.

 On Wed, Apr 28, 2010 at 8:22 PM, Vishal K vishalm...@gmail.com wrote:

 
  What is a good way to restart a ZK server (standalone and quorum)
 without
  having to restart it?
 
  Currently, I have ZK server embedded in another java application.





Re: Embedding ZK in another application

2010-04-29 Thread Mahadev Konar
We do set that Chad but it doesn't seem to help on some systems (especially
bsd)...

Thanks
mahadev


On 4/29/10 11:22 AM, Chad Harrington chad.harring...@gmail.com wrote:

 On Thu, Apr 29, 2010 at 8:49 AM, Patrick Hunt ph...@apache.org wrote:
 
 This is not foolproof however. We found that in general this would work,
 however there were some infrequent cases where a restarted server would fail
 to initialize due to the following issue:
 it is possible for the process to complete before the kernel has released
 the associated network resource, and this port cannot be bound to another
 process until the kernel has decided that it is done.
 
 more detail here:
 http://hea-www.harvard.edu/~fine/Tech/addrinuse.html
 
 as a result we ended up changing the test code to start each test with new
 quorum/election port numbers. This fixed the problem for us but would not be
 a solution in your case.
 
 Patrick
 
 
 I am not an expert at all on this, but I have used SO_REUSEADDR in other
 situations to avoid the address in use problem.  Would that help here?
 
 Chad Harrington
 chad.harring...@gmail.com
 
 
 
 On 04/29/2010 07:13 AM, Vishal K wrote:
 
 Hi Ted,
 
 We want the application that embeds the ZK server to be running even after
 the ZK server is shutdown. So we don't want to restart the application.
 Also, we prefer not to use zkServer.sh/zkServer.cmd because these are OS
 dependent (our application will run on Win as well as Linux). Instead, we
 thought that calling QuorumPeerMain.initializeAndRun() and
 QuorumPeerMain.shutdown() will suffice to start and shutdown a ZK server
 and
 we won't have to worry about checking the OS.
 
 Is there way to cleanly shutdown the ZK server (by invoking ZK server API)
 when it is embedded in the application without actually restarting the
 application process?
 Thanks.
 On Thu, Apr 29, 2010 at 1:54 AM, Ted Dunningted.dunn...@gmail.com
  wrote:
 
  Hmmm it isn't quite clear what you mean by restart without
 restarting.
 
 Why is killing the server and restarting it not an option?
 
 It is common to do a rolling restart on a ZK cluster.  Just restart one
 server at a time.  This is often used during system upgrades.
 
 On Wed, Apr 28, 2010 at 8:22 PM, Vishal Kvishalm...@gmail.com  wrote:
 
 
 What is a good way to restart a ZK server (standalone and quorum)
 without
 having to restart it?
 
 Currently, I have ZK server embedded in another java application.
 
 
 
 



Re: Embedding ZK in another application

2010-04-29 Thread Patrick Hunt

On 04/29/2010 11:22 AM, Chad Harrington wrote:

On Thu, Apr 29, 2010 at 8:49 AM, Patrick Huntph...@apache.org  wrote:


This is not foolproof however. We found that in general this would work,
however there were some infrequent cases where a restarted server would fail
to initialize due to the following issue:
it is possible for the process to complete before the kernel has released
the associated network resource, and this port cannot be bound to another
process until the kernel has decided that it is done.

more detail here:
http://hea-www.harvard.edu/~fine/Tech/addrinuse.html

as a result we ended up changing the test code to start each test with new
quorum/election port numbers. This fixed the problem for us but would not be
a solution in your case.

Patrick



I am not an expert at all on this, but I have used SO_REUSEADDR in other
situations to avoid the address in use problem.  Would that help here?



Hi Chad, seems like it should, but according to the link

Oddly, using SO_REUSEADDR can actually lead to more difficult address 
already in use errors.


See the strategies for avoidance section. I'm no expert on this 
particular issue either, so if someone knows for sure speak up.


We already do this btw, this is from the server... (not sure if we cover 
all cases though):

  ss.socket().setReuseAddress(true);

Patrick


Chad Harrington
chad.harring...@gmail.com




On 04/29/2010 07:13 AM, Vishal K wrote:


Hi Ted,

We want the application that embeds the ZK server to be running even after
the ZK server is shutdown. So we don't want to restart the application.
Also, we prefer not to use zkServer.sh/zkServer.cmd because these are OS
dependent (our application will run on Win as well as Linux). Instead, we
thought that calling QuorumPeerMain.initializeAndRun() and
QuorumPeerMain.shutdown() will suffice to start and shutdown a ZK server
and
we won't have to worry about checking the OS.

Is there way to cleanly shutdown the ZK server (by invoking ZK server API)
when it is embedded in the application without actually restarting the
application process?
Thanks.
On Thu, Apr 29, 2010 at 1:54 AM, Ted Dunningted.dunn...@gmail.com
  wrote:

  Hmmm it isn't quite clear what you mean by restart without

restarting.

Why is killing the server and restarting it not an option?

It is common to do a rolling restart on a ZK cluster.  Just restart one
server at a time.  This is often used during system upgrades.

On Wed, Apr 28, 2010 at 8:22 PM, Vishal Kvishalm...@gmail.com   wrote:



What is a good way to restart a ZK server (standalone and quorum)
without
having to restart it?

Currently, I have ZK server embedded in another java application.










Re: Embedding ZK in another application

2010-04-25 Thread Vishal K
Hi Mahadev, Ted,

Thanks for the feedback.

On Fri, Apr 23, 2010 at 3:02 PM, Ted Dunning ted.dunn...@gmail.com wrote:

 It is, of course, your decision, but a key coordination function is to
 determine whether your application is up or not.  That is very hard to do
 if
 Zookeeper is inside your application.

 On Fri, Apr 23, 2010 at 10:28 AM, Asankha C. Perera asan...@apache.org
 wrote:

  However, I believe that both the above are fine to live with for the
  application under consideration, as ZK will be used only to coordinate
  the larger application. Is there anything else that needs to be
  considered - and can I safely shutdown the clientPort since the
  application is always in the same JVM - but, if I do that how would I
  connect to ZK thereafter ?
 



Embedding ZK in another application

2010-04-23 Thread Asankha C. Perera
Hi All

I'm very new to ZK, and am looking at embeding ZK into an app that needs
cluster management - and the objective is to use ZK to notify
application cluster control operations (e.g. shutdown etc) across nodes.

I came across this post [1] from the user list by Ted Dunning from some
months back :
My experience with Katta has led me to believe that embedding a ZK in a
product is almost always a bad idea. - The problems are that you can't
administer the Zookeeper cluster independently and that the cluster
typically goes down when the associated service goes down.

However, I believe that both the above are fine to live with for the
application under consideration, as ZK will be used only to coordinate
the larger application. Is there anything else that needs to be
considered - and can I safely shutdown the clientPort since the
application is always in the same JVM - but, if I do that how would I
connect to ZK thereafter ?

thanks and regards
asankha

[1] http://markmail.org/message/tjonwec7p7dhfpms


Re: Embedding ZK in another application

2010-04-23 Thread Vishal K
Hi,

Good question. We are planning to do something similar as well and it will
great to know if there are any issues with embedding ZK server into an app.
We simply use QourumPeerMain and QourumPeer from our app to start/stop the
ZK server. Is this not a good way to do it?

On Fri, Apr 23, 2010 at 1:28 PM, Asankha C. Perera asan...@apache.orgwrote:

 Hi All

 I'm very new to ZK, and am looking at embeding ZK into an app that needs
 cluster management - and the objective is to use ZK to notify
 application cluster control operations (e.g. shutdown etc) across nodes.

 I came across this post [1] from the user list by Ted Dunning from some
 months back :
 My experience with Katta has led me to believe that embedding a ZK in a
 product is almost always a bad idea. - The problems are that you can't
 administer the Zookeeper cluster independently and that the cluster
 typically goes down when the associated service goes down.

 However, I believe that both the above are fine to live with for the
 application under consideration, as ZK will be used only to coordinate
 the larger application. Is there anything else that needs to be
 considered - and can I safely shutdown the clientPort since the
 application is always in the same JVM - but, if I do that how would I
 connect to ZK thereafter ?

 thanks and regards
 asankha

 [1] http://markmail.org/message/tjonwec7p7dhfpms



Re: Embedding ZK in another application

2010-04-23 Thread Mahadev Konar
Hi Vishal and Ashanka,

  I think Ted and Pat had somewhat comentted on this before.

Reiterating these comments below. If you are ok with these points I see no
concern in ZooKeeper as an embedded application.

Also, as Pat mentioned earlier  there are some cases where the server code
will system.exit. This is typically only if quorum communication fails in
some weird, unrecoverable way. We have removed most of these but there are a
few still remaining.


--- Comments by Ted 

I can't comment on the details of your code (but I have run in-process ZK's
in the past without problem)

Operationally, however, this isn't a great idea.  The problem is two-fold:

a) firstly, somebody would probably like to look at Zookeeper to understand
the state of your service.  If the service is
down, then ZK will go away.  That means that Zookeeper can't be used that
way and is mild to moderate
on the logarithmic international suckitude scale.

b) secondly, if you want to upgrade your server without upgrading Zookeeper
then you still have to bounce
Zookeeper.  This is probably not a problem, but it can be a slight pain.

c) thirdly, you can't scale your service independently of how you scale
Zookeeper.  This may or may
not bother you, but it would bother me.

d) fourthly, you will be synchronizing your server restarts with ZK's
service restarts.  Moving these events
away from each other is likely to make them slightly more reliable.  There
is no failure mode that I know
of that would be tickled here, but your service code will be slightly more
complex since it has to make sure
that ZK is up before it does stuff.  If you could make the assumption that
ZK is up or exit, that would be
simpler.

e) yes, I know that is more than two issues.  That is itself an issue since
any design where the number of worries
is increasing so fast is suspect on larger grounds.  If there are small
problems cropping up at that rate, the likelihood
of there being a large problem that comes up seems higher.


On 4/23/10 11:04 AM, Vishal K vishalm...@gmail.com wrote:

 Hi,
 
 Good question. We are planning to do something similar as well and it will
 great to know if there are any issues with embedding ZK server into an app.
 We simply use QourumPeerMain and QourumPeer from our app to start/stop the
 ZK server. Is this not a good way to do it?
 
 On Fri, Apr 23, 2010 at 1:28 PM, Asankha C. Perera asan...@apache.orgwrote:
 
 Hi All
 
 I'm very new to ZK, and am looking at embeding ZK into an app that needs
 cluster management - and the objective is to use ZK to notify
 application cluster control operations (e.g. shutdown etc) across nodes.
 
 I came across this post [1] from the user list by Ted Dunning from some
 months back :
 My experience with Katta has led me to believe that embedding a ZK in a
 product is almost always a bad idea. - The problems are that you can't
 administer the Zookeeper cluster independently and that the cluster
 typically goes down when the associated service goes down.
 
 However, I believe that both the above are fine to live with for the
 application under consideration, as ZK will be used only to coordinate
 the larger application. Is there anything else that needs to be
 considered - and can I safely shutdown the clientPort since the
 application is always in the same JVM - but, if I do that how would I
 connect to ZK thereafter ?
 
 thanks and regards
 asankha
 
 [1] http://markmail.org/message/tjonwec7p7dhfpms
 



Re: Embedding ZK in another application

2010-04-23 Thread Asankha C. Perera
Hi Mahadev
   I think Ted and Pat had somewhat comentted on this before.

 Reiterating these comments below. If you are ok with these points I see no
 concern in ZooKeeper as an embedded application...

Thanks, I missed this on the archives, and it helps!..

I guess if we still decide to embed, the only way to connect to ZK is
still with the normal TCP client..

cheers
asankha


Re: Embedding ZK in another application

2010-04-23 Thread Mahadev Konar
That's true!

Thanks
mahadev


On 4/23/10 11:41 AM, Asankha C. Perera asan...@apache.org wrote:

 Hi Mahadev
   I think Ted and Pat had somewhat comentted on this before.
 
 Reiterating these comments below. If you are ok with these points I see no
 concern in ZooKeeper as an embedded application...
 
 Thanks, I missed this on the archives, and it helps!..
 
 I guess if we still decide to embed, the only way to connect to ZK is
 still with the normal TCP client..
 
 cheers
 asankha



Re: Embedding ZK in another application

2010-04-23 Thread Ted Dunning
It is, of course, your decision, but a key coordination function is to
determine whether your application is up or not.  That is very hard to do if
Zookeeper is inside your application.

On Fri, Apr 23, 2010 at 10:28 AM, Asankha C. Perera asan...@apache.orgwrote:

 However, I believe that both the above are fine to live with for the
 application under consideration, as ZK will be used only to coordinate
 the larger application. Is there anything else that needs to be
 considered - and can I safely shutdown the clientPort since the
 application is always in the same JVM - but, if I do that how would I
 connect to ZK thereafter ?