Hi Lukasz,
  Comments in line:

On 8/28/09 8:24 AM, "Łukasz Osipiuk" <luk...@osipiuk.net> wrote:

> Hi!
> 
> I my name is Łukasz Osipiuk. I am working for one of major Polish
> Internet companies.
> In one of our projects we are intensively using Zookeeper as
> distributed locking system. We implemented slightly modified locking
> algorithm
> from zookeeper docs page.
> 
(http://hadoop.apache.org/zookeeper/docs/current/recipes.html#sc_recipes_Locks>
)
> 
> Unfortunately we experience some problems with deadlocks. As I
> examined the problem it appears that either we misuse zookeeper in
> some way
> or it is buggy. Our app is written in C++ and we are using
> zookeeper_mt C library.
> 
> Tests below are done using server version 3.1.1 and client library
> version 3.2.0, but on production we have both client and server in
> 3.1.1. and experience same problems.
> 
> I attach the code snippet i wrote to isolate our problems. As I run it
> and while it is running randomly kill zookeeper nodes I (from time to
> time) get one of following behaviors:
> 
> 1. the zoo_create() call returns error but still node is created in zookeeper.
>     If such problem happens in locking protocol we get a hanging lock
> without owner which will never disapear. Closing client zookeeper
> session is
>     needed to remove such hanging ephemeral node.
This could happen. If you get a CONNECTIONLOSS error on a create then the
create might or might not have happened. Please take a look at
CONNECTIONLOSS handling on our wiki
http://wiki.apache.org/hadoop/ZooKeeper (I cant get to the direct link since
wiki is down)

Also, take a look at

http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperProgrammers.html

For handling CONNECTIONLOSS. We have an open jira where in we want to avoid
throwing CONNECTIONLOSS error but that will be fixed probably in 3.3.
http://issues.apache.org/jira/browse/ZOOKEEPER-22
> 
> 2. application thread just hangs. From what i observed in gdb it is
> waiting for synchronous operation completion (function
> wait_sync_completion)
> 
Are you accessing the zoookeper handle via 2 different threads? Thoguht the
handle is thread safe but you should make sure that you do not call zoo
api's after you have called zoo_close() on the handle. We have seen this
kind of hanging problem wherein one thread was closing the handle and the
other thread was calling something like zoo_exists().


> Is there a way to avoid this problems? Are we doing something wrong or
> should we create a bug report?
> Is anyone of you using zookeeper as distributed locking service with
> more success?
> 
> Help is really appreciate.
> 
> PS. to compile code snippet use:
> g++ credel.cc -o credel -pedantic -lzookeeper_mt

Hope this helps.

Thanks
mahadev

Reply via email to