Re: how to lock one-of-many ?
You have to be careful there of race conditions. ZK's slightly surprising API makes it pretty easy to get this right, however. The correct way to do what you suggest is to read the list of children in the locks directory and put a watch on the directory at the same time. If the number of locks equals the number of resources, you wait. If it is less, you can randomly pick one of the apparently unlocked resources at random. If you fail, start again by checking the number of resources. On Tue, Feb 23, 2010 at 9:09 PM, Martin Waite wrote: > I guess another optimisation might be to count the number of locks held > first: if the count equals the number of resources, try again later. But > I > suppose that might require a sync call first to ensure that zk instance my > client is connected to is up to date. > -- Ted Dunning, CTO DeepDyve
Re: how to lock one-of-many ?
Thanks Ted. The randomization should work well in this case. regards, Martin On 23 February 2010 18:27, Ted Dunning wrote: > I think that the crux of Mahadev's suggestion is that you do as you say, > but > you should try the resources in randomized order. > > That will have very robust properties, especially with more than a handful > of resources and is easy to code and to analyze. > > It won't work if you really mean "lock first available from this sequence". > > On Tue, Feb 23, 2010 at 4:05 AM, Martin Waite >wrote: > > > For locking, I could loop through > > the available ids, attempting to create a lock for that in the locked > > directory. However this seems a bit clumsy and slow. Also, the locks > are > > held for a relatively short time (1 second on average), and by time I > have > > blundered through all the possible locks, ids that were locked at the > start > > might be available by time I finished. > > > > > > -- > Ted Dunning, CTO > DeepDyve >
Re: how to lock one-of-many ?
Thanks Mahadev. Randomising the look-up is a good fit for what I am trying
to do, and should minimise the number of look-ups performed.
I guess another optimisation might be to count the number of locks held
first: if the count equals the number of resources, try again later. But I
suppose that might require a sync call first to ensure that zk instance my
client is connected to is up to date.
regards,
Martin
On 23 February 2010 17:50, Mahadev Konar wrote:
> Hi Martin,
> How about this-
>
> you have resources in the a directory (say /locks)
>
> each process which needs to lock, lists all the children of this directory
> and then creates an ephemeral node called /locks/resource1/lock depending
> on
> which resource it wants to lock.
>
> This ephemeral node will be deleted by the process as soon as its done
> using
> the resource. A process should only use to resource_{i} if its been able to
> create /locks/resource_{i}/locks.
>
> Would that work?
>
> Thanks
> mahadev
>
> On 2/23/10 4:05 AM, "Martin Waite" wrote:
>
> > Hi,
> >
> > I have a set of resources each of which has a unique identifier. Each
> > resource element must be locked before it is used, and unlocked
> afterwards.
> >
> > The logic of the application is something like:
> >
> > lock any one element;
> > if (none locked) then
> >exit with error;
> > else
> >get resource-id from lock
> >use resource
> >unlock resource
> > end
> >
> > Zookeeper looks like a good candidate for managing these locks, being
> fast
> > and resilient, and it seems quite simple to recover from client failure.
> >
> > However, I cannot think of a good way to implement this sort of
> one-of-many
> > locking.
> >
> > I could create a directory called "available" and another called
> "locked".
> > Available would have one entry for each resource id ( or one entry
> > containing a list of the resource-ids). For locking, I could loop
> through
> > the available ids, attempting to create a lock for that in the locked
> > directory. However this seems a bit clumsy and slow. Also, the locks
> are
> > held for a relatively short time (1 second on average), and by time I
> have
> > blundered through all the possible locks, ids that were locked at the
> start
> > might be available by time I finished.
> >
> > Can anyone think of a more elegant and efficient way of doing this ?
> >
> > regards,
> > Martin
>
>
Re: Bit of help debugging a TIMED OUT session please
Hard to say based on the bits/pieces of the log we have access to. I'd have to see the full log, preferably from both the server and client, to gain more insight. re low numbers, this is the received count for the server, this should always increase never decrease. The fact that it is so low either indicates that the server recently restarted, or clients are not attaching to it. Seems like it should be near the other servers but again, hard to tell based on the small aperture we have via mail. Patrick Stack wrote: Thanks Patrick. See below. On Tue, Feb 23, 2010 at 1:19 PM, Patrick Hunt wrote: Stack you might look at the following: 1) why does server 14 have such a low recv count? Received: 194 while the other servers are at 3.7k + received. Did server 14 fail at some point? Or it's network? This may have caused the timeout seen by the client: Ok. Will check into this the next time. I did take the dump after the observed TIMED_OUT, a good while after. Could this be why the numbers are low? --snippet- 2010-02-21 18:23:55,583 [main-SendThread] INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server 14.u.XXX.com/X.X.X.141:2181 2010-02-21 18:24:00,423 [regionserver/208.76.44.140:60020.compactor-SendThread] WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x226ed968a270003 to sun.nio.ch.selectionkeyi...@2a50e9a3 java.io.IOException: TIMED OUT at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906) --- 2) connection timeout is different from session timeout. connection timeout is the amount of time we allow for connection establishment (socket open) until the server accepts the connection, this value is the session timeout (as requested by the client) divided by the number of hosts in the host list. This could account for why the timeout (above snippet) occurred after 5 seconds. What timeout value is this client using? 15 seconds? We ask for a session timeout of 60 seconds -- the hbase default -- and our ticktime is 3 seconds. You are not troubled at all by the exceptions closing sessions above? Are these just noise? Thanks for the input, St.Ack
Re: Bit of help debugging a TIMED OUT session please
Thanks Patrick. See below. On Tue, Feb 23, 2010 at 1:19 PM, Patrick Hunt wrote: > Stack you might look at the following: > > 1) why does server 14 have such a low recv count? > > Received: 194 > > while the other servers are at 3.7k + received. Did server 14 fail at some > point? Or it's network? This may have caused the timeout seen by the client: > Ok. Will check into this the next time. I did take the dump after the observed TIMED_OUT, a good while after. Could this be why the numbers are low? > --snippet- > 2010-02-21 18:23:55,583 [main-SendThread] INFO > org.apache.zookeeper.ClientCnxn: Attempting connection to server > 14.u.XXX.com/X.X.X.141:2181 > 2010-02-21 18:24:00,423 > [regionserver/208.76.44.140:60020.compactor-SendThread] WARN > org.apache.zookeeper.ClientCnxn: Exception closing session > 0x226ed968a270003 to sun.nio.ch.selectionkeyi...@2a50e9a3 > java.io.IOException: TIMED OUT > at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906) > --- > > 2) connection timeout is different from session timeout. connection timeout > is the amount of time we allow for connection establishment (socket open) > until the server accepts the connection, this value is the session timeout > (as requested by the client) divided by the number of hosts in the host > list. This could account for why the timeout (above snippet) occurred after > 5 seconds. What timeout value is this client using? 15 seconds? > We ask for a session timeout of 60 seconds -- the hbase default -- and our ticktime is 3 seconds. You are not troubled at all by the exceptions closing sessions above? Are these just noise? Thanks for the input, St.Ack
Re: Bit of help debugging a TIMED OUT session please
Stack you might look at the following: 1) why does server 14 have such a low recv count? Received: 194 while the other servers are at 3.7k + received. Did server 14 fail at some point? Or it's network? This may have caused the timeout seen by the client: --snippet- 2010-02-21 18:23:55,583 [main-SendThread] INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server 14.u.XXX.com/X.X.X.141:2181 2010-02-21 18:24:00,423 [regionserver/208.76.44.140:60020.compactor-SendThread] WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x226ed968a270003 to sun.nio.ch.selectionkeyi...@2a50e9a3 java.io.IOException: TIMED OUT at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906) --- 2) connection timeout is different from session timeout. connection timeout is the amount of time we allow for connection establishment (socket open) until the server accepts the connection, this value is the session timeout (as requested by the client) divided by the number of hosts in the host list. This could account for why the timeout (above snippet) occurred after 5 seconds. What timeout value is this client using? 15 seconds? Patrick Stack wrote: Dang. Didn't save the log. Pardon me. I pasted exceptions only and thought it all about 0x26ed968d880001 session but now I see that what I posted above has TIMED_OUT on another session altogether. Above I skipped pasting exceptions thinking them on the same session but now it seems they probably were not. I'm trying to track a case where zk seems of a sudden, client-side, to give up the ghost w/ exceptions like those pasted above -- connectivity probs. There has been pollution in here where long gc pauses that are > session timeout would trigger TIMED_OUT but those have been tamed. I'll be back if I get another instance on my hook. Meantime, thanks for the comments. St.Ack On Mon, Feb 22, 2010 at 6:43 PM, Mahadev Konar wrote: HI stack, the other interesting part is with the session: 0x26ed968d880001 Looks like it gets disconnected from one of the servers (TIMEOUT). DO you see any of these messages: "Attempting connection to server" in the logs before you see all the consecutive org.apache.zookeeper.ClientCnxn: Exception closing session 0x26ed968d880001 to sun.nio.ch.selectionkeyi...@788ab708 java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:701) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945) and From the cient 0x26ed968d880001? Thanks mahadev On 2/22/10 11:42 AM, "Stack" wrote: The thing that seems odd to me is that the connectivity complaints are out of the zk client, right?, why is it failing getting to member 14 and why not move to another ensemble member if issue w/ 14?, and if there were a general connectivity issue, I'd think that the running hbase cluster would be complaining at about the same time (its talking to datanodes and masters at this time). (Thanks for the input lads) St.Ack On Mon, Feb 22, 2010 at 11:26 AM, Mahadev Konar wrote: I also looked at the logs. Ted might have a point. It does look like that zookeeper server's are doing fine (though as ted mentions the skew is a little concerning, though that might be due to very few packets served by the first server). Other than that the latencies of 300 ms at max should not cause any timeouts. Also, the number of packets received is pretty low - meaning that it wasn't serving huge traffic. Is there anyway we can check if the network connection from the client to the server is not flaky? Thanks mahadev On 2/22/10 10:40 AM, "Ted Dunning" wrote: Not sure this helps at all, but these times are remarkably asymmetrical. I would expect members of a ZK cluster to have very comparable times. Additionally, 345 ms is nowhere near large enough to cause a session to expire. My take is that ZK doesn't think it caused the timeout. On Mon, Feb 22, 2010 at 10:18 AM, Stack wrote: Latency min/avg/max: 2/125/345 ... Latency min/avg/max: 0/7/81 ... Latency min/avg/max: 1/1/1 Thanks for any pointers on how to debug.
Re: how to lock one-of-many ?
I think that the crux of Mahadev's suggestion is that you do as you say, but you should try the resources in randomized order. That will have very robust properties, especially with more than a handful of resources and is easy to code and to analyze. It won't work if you really mean "lock first available from this sequence". On Tue, Feb 23, 2010 at 4:05 AM, Martin Waite wrote: > For locking, I could loop through > the available ids, attempting to create a lock for that in the locked > directory. However this seems a bit clumsy and slow. Also, the locks are > held for a relatively short time (1 second on average), and by time I have > blundered through all the possible locks, ids that were locked at the start > might be available by time I finished. > -- Ted Dunning, CTO DeepDyve
Re: how to lock one-of-many ?
Hi Martin,
How about this-
you have resources in the a directory (say /locks)
each process which needs to lock, lists all the children of this directory
and then creates an ephemeral node called /locks/resource1/lock depending on
which resource it wants to lock.
This ephemeral node will be deleted by the process as soon as its done using
the resource. A process should only use to resource_{i} if its been able to
create /locks/resource_{i}/locks.
Would that work?
Thanks
mahadev
On 2/23/10 4:05 AM, "Martin Waite" wrote:
> Hi,
>
> I have a set of resources each of which has a unique identifier. Each
> resource element must be locked before it is used, and unlocked afterwards.
>
> The logic of the application is something like:
>
> lock any one element;
> if (none locked) then
>exit with error;
> else
>get resource-id from lock
>use resource
>unlock resource
> end
>
> Zookeeper looks like a good candidate for managing these locks, being fast
> and resilient, and it seems quite simple to recover from client failure.
>
> However, I cannot think of a good way to implement this sort of one-of-many
> locking.
>
> I could create a directory called "available" and another called "locked".
> Available would have one entry for each resource id ( or one entry
> containing a list of the resource-ids). For locking, I could loop through
> the available ids, attempting to create a lock for that in the locked
> directory. However this seems a bit clumsy and slow. Also, the locks are
> held for a relatively short time (1 second on average), and by time I have
> blundered through all the possible locks, ids that were locked at the start
> might be available by time I finished.
>
> Can anyone think of a more elegant and efficient way of doing this ?
>
> regards,
> Martin
how to lock one-of-many ?
Hi, I have a set of resources each of which has a unique identifier. Each resource element must be locked before it is used, and unlocked afterwards. The logic of the application is something like: lock any one element; if (none locked) then exit with error; else get resource-id from lock use resource unlock resource end Zookeeper looks like a good candidate for managing these locks, being fast and resilient, and it seems quite simple to recover from client failure. However, I cannot think of a good way to implement this sort of one-of-many locking. I could create a directory called "available" and another called "locked". Available would have one entry for each resource id ( or one entry containing a list of the resource-ids). For locking, I could loop through the available ids, attempting to create a lock for that in the locked directory. However this seems a bit clumsy and slow. Also, the locks are held for a relatively short time (1 second on average), and by time I have blundered through all the possible locks, ids that were locked at the start might be available by time I finished. Can anyone think of a more elegant and efficient way of doing this ? regards, Martin
