Re: txzookeeper - a twisted python client for zookeeper

Nice. Any chance of putting it back in zk? Would be useful. Thanks mahadev On 11/18/10 1:17 PM, "Kapil Thangavelu" wrote: At canonical we've been using zookeeper heavily in the development of a new project (ensemble) as noted by gustavo. I just wanted to give a quick overview of the client l

Re: Watcher examples

Yup. This would be really nice to have. More examples and documentaiton for them, would be really helpful for our users. Thanks mahadev On 11/15/10 12:54 PM, "Patrick Hunt" wrote: > It would be great to have more examples as part of the release > artifact. Would you mind creating a JIRA/patc

Re: JUnit tests do not produce logs if the JVM crashes

Hi Andras, Junit unit will always buffer the logs unless you print it out to console. To do that, try running this ant test -Dtest.output=yes This will print out the logs to console as they are logged. Thanks mahadev On 11/4/10 3:33 AM, "András Kövi" wrote: > Hi all, I'm new to Zookeeper

FW: [Hadoop Wiki] Update of "ZooKeeper/ZKClientBindings" by yfinkelstein

Nice to see this! Thanks mahadev -- Forwarded Message From: Apache Wiki Reply-To: Date: Tue, 2 Nov 2010 14:39:24 -0700 To: Apache Wiki Subject: [Hadoop Wiki] Update of "ZooKeeper/ZKClientBindings" by yfinkelstein Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hado

Re: Problem with Zookeeper cluster configuration

I think Jared pointed this out, given that your clientPort and quorum port are same: clientPort=5181 server.1=3.7.192.142:5181:5888 The above 2 ports should be different. Thanks mahadev On 10/27/10 10:19 AM, "Ted Dunning" wrote: > Sorry, didn't see this

Re: Unusual exception

Hi Avinash, Not sure if you got a response for your email. The exception that you mention mostly means that the client already closed the socket or shutdown. Looks like a client is trying to connect but disconnects before the server can respond. Do you have any such clients? Is this causing an

Re: Zookeeper on 60+Gb mem

Hi Maarteen, I definitely know of a group which uses around 3GB of memory heap for zookeeper but never heard of someone with such huge requirements. I would say it definitely would be a learning experience with such high memory which I definitely think would be very very useful for others in the

Re: Expiring session... timeout of 600000ms exceeded

Am not sure, if anyone responded to this or not. Are the clients getting session expired or getting Connectionloss? In any case, zookeeper client has its own thread to updated the server with active connection status. Did you take a look at the GC activity at your client? Thanks mahadev On 9/21/

Re: possible bug in zookeeper ?

Hi Yatir, Any update on this? Are you still struggling with this problem? Thanks mahadev On 9/15/10 12:56 AM, "Yatir Ben Shlomo" wrote: > Thanks to all who replied, I appreciate your efforts: > > 1. There is no connections problem from the client machine: > (ob1078)(tom...@cass3:~)$ echo ru

Re: SessionMovedException

Hi Jun, You can read more about the SessionMovedException at http://hadoop.apache.org/zookeeper/docs/r3.3.0/zookeeperProgrammers.html Thanks mahadev On 10/1/10 9:58 AM, "Jun Rao" wrote: Hi, Could someone explain what SessionMovedException means? Should it be treated as SessionExpiredExcept

Re: zkfuse

Hi Jun, I havent seen people using zkfuse recently. What kind of issues are you facing? Thanks mahadev On 9/19/10 6:46 PM, "俊贤" wrote: > Hi guys, > Has anyone succeeded in installing the zkfuse? > > > This email (including any attachments) is confidential an

Re: possible bug in zookeeper ?

Yatir, Can you try this out: From zook1, try running the zookeeper a simpole client library: http://hadoop.apache.org/zookeeper/docs/r3.2.1/zookeeperStarted.html bin/zkCli.sh -server zoo1:port,zoo2:port,zoo3:port And then try killing one of the servers and see if this client connects to th

Re: possible bug in zookeeper ?

Hi yatir, Can you confirm that zook1 , zook2 can be nslookedup from the client machine? We havent seen a bug like this. It would be great to nail this down. Thanks mahadev On 9/14/10 8:44 AM, "Yatir Ben Shlomo" wrote: > zook1:2181,zook2:2181,zook3:2181 > > > -Original Message- > F

Re: Understanding ZooKeeper data file management and LogFormatter

Hi Vishal, Usually the default retention policy is safe enough for operations. http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperAdmin.html Gives you an overview of how to use the purging library in zookeeper. Thanks mahadev On 9/8/10 12:01 PM, "Vishal K" wrote: > Hi All, > > Can you

Re: Lock example

Hi Tim, The lock recipe you mention is supposed to avoid her affect and prevent starvation (though it has bugs :)). Are you looking for something like that or just a simple lock and unlock that doesn't have to worry abt the above issues. If that's the case then just doing an ephemeral create an

Re: Receiving create events for self with synchronous create

ved. line 51. I'm trying to encapsulate the detection of additions and deletions of child nodes within this Watcher. All other events that occur due to a node being added or deleted should be handled externally by the clustermanager. Thanks, Todd On Thu, 2010-08-26 at 19:26 -0700, Mahade

Re: Logs and in memory operations

Hi Avinash, IN the source code the FinalRequestProcessor updates the in memory data structures and the SyncRequestProcessor logs to disk. For deciding when to delete take a look at PurgeTxnLog.java file. Thanks mahadev On 8/30/10 1:11 PM, "Avinash Lakshman" wrote: Hi All >From my understan

Re: getting created child on NodeChildrenChanged event

Hi Todd, We have always tried to lean on the side of keeping things lightweight and the api simple. The only way you would be able to do this is with sequential creates. 1. create nodes like /queueelement-$i where i is a monotonically increasing number. You could use the sequential flag of zook

Re: ZooKeeper C bindings and cygwin?

Hi Jan, It would be great to have some documentation on how to use the windows install. Would you mind submitting a patch with documentation with FAQ's and any other issues you might have faced? Thanks mahadev On 9/1/10 6:04 AM, "jdeinh...@ujam.com" wrote: Dear list readers, we've "solved"

Re: election recipe

Hi Eric, As Ted and you yourself mentioned its mostly to avoid herd affect. A herd affect would usually mean 1000¹s of client notified of some change and would try creating the same node on notification. With just 10¹s of clients you don¹t need to worry abt this herd effect at all. Thanks mahad

Re: Spew after call to close

Hi Stack, Looks like you are shutting down the server and shutting down the client at the same time? Is that the issue? Thanks mahadev On 9/3/10 4:47 PM, "Stack" wrote: > Have you fellas seen this before? I call close on zookeeper but it insists > on doing the below exceptions. Why is it doi

Re: client notification of partition in ensemble

Yes, the zookeeper node will stop accepting clients as soon as it disconnects. So the client will get a disconnect. Thanks mahadev On 8/26/10 6:36 PM, "Ryan King" wrote: > I'm new to zookeeper so forgive me if this question is naive, but I > can't seem to find an answer in the documentation or

Re: Receiving create events for self with synchronous create

Hi Todd, The code that you point to, I am not able to make out the sequence of steps. Can you be more clear on what you are trying to do in terms of zookeeper api? Thanks mahadev On 8/26/10 5:58 PM, "Todd Nine" wrote: Hi all, I'm running into a strange issue I could use a hand with. I'v

Re: Zookeeper stops

t; Patrick >>> >>> On 08/19/2010 07:43 AM, Ted Dunning wrote: >>> >>> Also, /tmp is not a great place to keep things that are intended for >>> persistence. >>> >>> On Thu, Aug 19, 2010 at 7:34 AM, Mahadev Konar>> >wrote: >>

Re: Searching more ZooKeeper content

I am definitely a +1 on this, given that its powered by Solr. Thanks mahadev On 8/25/10 9:22 AM, "Alex Baranau" wrote: > Hello guys, > > Over at http://search-hadoop.com we index ZooKeeper project's mailing lists, > wiki, web site, > source code, javadoc, jira... > > Would the community be i

Re: Size of a znode in memory

Hi Marten, The usual memory footprint of a znode is around 40-80 bytes. I think Ben is planning to document a way to calculate approximate memory footprint of your zk servers given a set of updates and there sizes. thanks mahadev On 8/25/10 11:49 AM, "Maarten Koopmans" wrote: > Hi, > > Is

Re: Non Hadoop scheduling frameworks

Hi Todd, Just to be clear, are you looking at solving UC1 and UC2 via zookeeper? Or is this a broader question for scheduling on cassandra nodes? For the latter this probably isnt the right mailing list. Thanks mahadev On 8/23/10 4:02 PM, "Todd Nine" wrote: Hi all, We're using Zookeeper

Re: Parent nodes & multi-step transactions

Hi Gustavo, Usually the paradigm I like to suggest is to have something like /A/init Every client watches for the existence of this node and this node is only created after /A has been initialized with the creation of /A/C or other stuff. Would that work for you? Thanks mahadev On 8/23/10 7:

Re: Zookeeper stops

Hi Wim, It mostly looks like that zookeeper is not able to create files on the /tmp filesystem. Is there is a space shortage or is it possible the file is being deleted as its being written to? Sometimes admins have a crontab on /tmp that cleans up the /tmp filesystem. Thanks mahadev On 8/1

Re: A question about Watcher

Hi Qian, The watcher information is saved at the client, and the client will reattach the watches to the new server it connects to. Hope that helps. Thanks mahadev On 8/16/10 9:28 AM, "Qian Ye" wrote: > thx for explaination. Since the watcher can be preserved when the client > switch the zo

Re: How to handle "Node does not exist" error?

HI Dr Hao, Can you please post the configuration of all the 3 zookeeper servers? I suspect it might be misconfigured clusters and they might not belong to the same ensemble. Just to be clear: /xpe/queues/3bd7851e79381ef4bfd1a5857b5e34c04e5159e5/msgs/msg002807 And other such nodes exist on o

Re: Sequence Number Generation With Zookeeper

Hi David, I think it would be really useful. It would be very helpful for someone looking for geenrating unique tokens/generations ids ( I can think of plenty of applications for this). Please do consider contributing it back to the community! Thanks mahadev On 8/6/10 7:10 AM, "David Rosenstra

Re: zkperl - skipped tests

Hi Martin, You might have to look into the tests. t/50_access.t is the file you might want to take a look at. I am not a perl guru so am not of much help but let me know if you cant work out the details on the skipped tests. I will try to dig into the perl code. Thanks mahadev On 8/4/10 6:16

Re: node symlinks

HI Maarteen, Can you elaborate on your use case of ZooKeeper? We currently don't have any symlinks feature in zookeeper. The only way to do it for you would be a client side hash/lookup table that buckets data to different zookeeper servers. Or you could also store this hash/lookup table in one

Re: ZK recovery questions

Hi Ashwin, We have seen people wanting to have something like ZooKeeper without the reliability of permanent storage and are willing to work with loosened guarantees of current Zookeeper. What you mention on log files is certainly a valid use case. It would be great to see how much throughput yo

Re: unit test failure

HI Martin, Can you check if you have a stale java process (ZooKeeperServer) running on your machine? That might cause some issues with the tests. Thanks mahadev On 7/14/10 8:03 AM, "Martin Waite" wrote: > Hi, > > I am attempting to build the C client on debian lenny. > > autoconf, config

Re: building client tools

Hi Martin, There is a list of tools, i.e cppunit. That is the only required tool to build the zookeeper c library. The readme says that it can be done without cppunit being installed but there has been a open bug regarding this. So cppunit is required as of now. Thanks mahadev On 7/13/10 10:09

Re: running the systest

Hi Stuart, The instructions are just out of date. If you could open a jira and post a patch to it that would be great! We should try getting this in 3.3.2! That would be useful! Thanks mahadev On 7/9/10 6:36 AM, "Stuart Halloway" wrote: > Hi all, > > I am trying to run the systest and have

Re: Frequent SessionTimeoutException[Client] - CancelledKeyException[Server]

HI Lakshman, The ping requests are also queued up. They are not answered immediately. They go through the request queue and make sure that the server is functioning properly. Is there any chance of under provisioning on the server side? The high latencies are usually a sign of Garbage collection

Re: Suggested way to simulate client session expiration in unit tests?

Hi Jeremy, zk.disconnect() is the right way to disconnect from the servers. For session expiration you just have to make sure that the client stays disconnected for more than the session expiration interval. Hope that helps. Thanks mahadev On 7/6/10 9:09 AM, "Jeremy Davis" wrote: > Is there

Re: Guaranteed message delivery until session timeout?

When a connectionloss happens all the watches are triggered saying that connectionloss occurred. But on a reconnect the watches are reset automagically on the new server and will be fired if the change has already happened or will be reset! I hope that answers your question. Thanks mahadev On

Re: Securing ZooKeeper connections

Hi Vishal, Ben (Benjamin Reed) has been working on a netty based client server protocol in ZooKeeper. I think there is an open jira for it. My network connection is pretty slow so am finding it hard to search for it. We have been thinking abt enabling secure connections via this netty based conn

Re: Zookeeper EventThread and SendThread

Hi Nick, These threads are spawned with each zookeeper client handle. As soon as you create a zookeeper client object these threads are spawned. Are yu creating too many zookeeper client objects in your application? Htanks mahadev On 5/20/10 11:30 AM, "Nick Bailey" wrote: > Hey guys, > > Qu

Re: Using ZooKeeper for managing solrCloud

Hi Rakhi, You can read more abt monitoring zookeeper servers at http://hadoop.apache.org/zookeeper/docs/r3.3.0/zookeeperAdmin.html#sc_monito ring Thanks mahadev On 5/14/10 4:09 AM, "Rakhi Khatwani" wrote: > Hi, >I just went through the zookeeper tutorial and successfully managed > t

Re: Can't ls with large node count and I don't understand the use of jute.maxbuffer

Hi Aaaron, Each of the requests and response between client and servers is sent an (buflen, buffer) packet. The content of the packets are then deserialized from this buffer. Looks like the size of the packet (buflen) is big in yoru case. We usually avoid sending/receiving large packets just to

Re: Problems with ZooKeeper apache wiki.

arbara, CA 93106, USA > http://www.cs.ucsb.edu/~sudipto > > > On Tue, May 11, 2010 at 4:52 PM, Mahadev Konar wrote: > >> Thanks Paul! >> >> Does this mean folks will not be able to upload and download presentations >> on wiki from now on? >> >

Re: Xid out of order. Got 8 expected 7

Hi Jordan, Can you create a jira for this? And attach all the server logs and client logs related to this timeline? How did you start up the servers? Is there some changes you might have made accidentatlly to the servers? Thanks mahadev On 5/12/10 10:49 AM, "Jordan Zimmerman" wrote: > We've

Re: Problems with ZooKeeper apache wiki.

been disabled due to a spammer earlier this week. > > I don't know how soon or if we will enable file attachments again. > > On Tue, May 11, 2010 at 4:36 PM, Mahadev Konar wrote: >> Hi, >>  We are having problems with apache wiki wherein we are getting an error of >

Re: New ZooKeeper client library "Cages"

Hi Dominic, Good to see this. I like the name "cages" :). You might want to post to the list what cages is useful for. I think quite a few folks would be interested in something like this. Are you guys currently using it with cassandra? Thanks mahadev On 5/11/10 4:02 PM, "Dominic Williams"

Re: ZookeeperPresentations Wiki

I just emailed in...@apache to ask for there help on this. I wasn't able to figure out what the problem is! Thanks for pointing it out. mahadev On 5/11/10 4:01 PM, "Sudipto Das" wrote: > Hi, > > I am trying to download some presentation slides from the > ZookeeperPresentations wiki ( > http:

Re: ZKClient

Hi Adam, I don't think zk is very very hard to get right. There are exmaples in src/recipes which implements locks/queues/others. There is ZOOKEEPER-22 to make it even more easier for application to use. Regarding re registration of watches, you can deifnitely write code and submit is as a part

Re: avoiding deadlocks on client handle close w/ python/c api

Sure, Ill take a look at it. Thanks mahadev On 5/4/10 2:32 PM, "Patrick Hunt" wrote: > Thanks Kapil, Mahadev perhaps you could take a look at this as well? > > Patrick > > On 05/04/2010 06:36 AM, Kapil Thangavelu wrote: >> I've constructed a simple example just using the zkpython library wi

Re: Dynamic adding/removing ZK servers on client

Yeah, that was one of the ideas, I think its been on the jira somewhere ( I forget)... But could be and would definitely be one soln for it. Thanks mahadev On 5/3/10 2:12 PM, "Ted Dunning" wrote: > Should this be a znode in the privileged namespace? > > On Mon, May 3, 2010 at 1:45 PM, Dave Wr

Re: Dynamic adding/removing ZK servers on client

Hi Dave, Just a question on how do you see it being used, meaning who would call addserver and removeserver? It does seem useful to be able to do this. This is definitely worth working on. You can link it as a subtask of ZOOKEEPER-107. Thanks mahadev On 5/3/10 7:03 AM, "Dave Wright" wrote: >

Re: Question on maintaining leader/membership status in zookeeper

case. We are talking about asymmetric network failure. >>> Yes. Leader could consider all the slaves being down if it tracks the status >>> of all slaves himself. But I guess if ZK is used for for membership >>> management, neither the leader nor the slaves will be consider

Re: Question on maintaining leader/membership status in zookeeper

where the master is cut off from ZK, but not from the slave. In both cases, the master will get a connection loss event and stop trying to act like a master and the slave will be notified that the master has dropped out of its role. -- On 4/30/10 4:14 PM, "Mah

Re: Question on maintaining leader/membership status in zookeeper

ership > management, neither the leader nor the slaves will be considered > disconnected because they can all connect to ZK. > > Thanks, > > Lei > > > On 4/30/10 3:47 PM, "Mahadev Konar" wrote: > >> Hi Lei, >> >> In this case, the Lea

Re: Question on maintaining leader/membership status in zookeeper

Hi Lei, In this case, the Leader will be disconnected from ZK cluster and will give up its leadership. Since its disconnected, ZK cluster will realize that the Leader is dead! When Zk cluster realizes that the Leader is dead (this is because the zk cluster hasn't heard from the Leader for a c

Re: Question on maintaining leader/membership status in zookeeper

Hi Lei, In this case, its up to application to decide what to do when this happens. The application will be notified that its disconnected from the ZooKeeper cluster. In such a case some of the applications might decide to not proceed at all, (since it might lead to some state corruption) and some

Re: Question on maintaining leader/membership status in zookeeper

Hi Lei, I think you might be confusing Leader Election within ZooKeeper itself with electing a leader for your application? Is that so? Leader Election within ZooKeeper is totally internal to zookeeper service and is not visible to applications. I am little confused, what your problem statement

Re: Partially partitioned connectivity

Hi Kevin, I had the response set up but didn't hit send. Ted already answered your question, but to give you a more technical background assuming that you know a little bit more about transaction ids in ZooKeeper and server ids: If B and C are partitioned from each other, and A is the leader, th

Re: Embedding ZK in another application

We do set that Chad but it doesn't seem to help on some systems (especially bsd)... Thanks mahadev On 4/29/10 11:22 AM, "Chad Harrington" wrote: > On Thu, Apr 29, 2010 at 8:49 AM, Patrick Hunt wrote: > >> This is not foolproof however. We found that in general this would work, >> however the

Re: Misbehaving zk servers

Hi Travis, How many clients did you have connected to this server? Usually the default is 8K file descriptors. Did you have clients more than that? Also, if clients fail to attach to a server, they will run off to another server. We do not do any blacklisting because we expect the server to heal

Re: Zookeeper client

mization occur even in this case? > > Thanks > Avinash > > On Tue, Apr 27, 2010 at 11:00 AM, Mahadev Konar wrote: > >> HI Avinash, >> The zk client does itself maintain liveness information and also >> randomizes the list of servers to balance the number of cl

Re: Zookeeper client

HI Avinash, The zk client does itself maintain liveness information and also randomizes the list of servers to balance the number of clients connected to a single ZooKeeper server. Hope that helps. Thanks mahadev On 4/27/10 10:56 AM, "Avinash Lakshman" wrote: > Let's assume I have 100 clien

Re: Embedding ZK in another application

That's true! Thanks mahadev On 4/23/10 11:41 AM, "Asankha C. Perera" wrote: > Hi Mahadev >> I think Ted and Pat had somewhat comentted on this before. >> >> Reiterating these comments below. If you are ok with these points I see no >> concern in ZooKeeper as an embedded application... > >

Re: Embedding ZK in another application

Hi Vishal and Ashanka, I think Ted and Pat had somewhat comentted on this before. Reiterating these comments below. If you are ok with these points I see no concern in ZooKeeper as an embedded application. Also, as Pat mentioned earlier there are some cases where the server code will "system.

Re: bug: wrong heading in recipes doc

I think we should be using zookeeper locks to create jiras :) . Looks like both of you created one!!! :) Thanks mahadev On 4/22/10 1:37 PM, "Patrick Hunt" wrote: > No problem. > https://issues.apache.org/jira/browse/ZOOKEEPER-752 > > I've seen alot of traffic on infrastruct...@apache, yo

Re: odd error message

Ok, I think this is possible. So here is what happens currently. This has been a long standing bug and should be fixed in 3.4 https://issues.apache.org/jira/browse/ZOOKEEPER-335 A newly elected leader currently doesn't log the new leader transaction to its database In your case, the follower

Re: Would this work?

Hi Avinash, This mostly looks like the zookeeper client is not able to find the zookeeper server running on the port that you have specified it on. Are you sure you are running zookeeper server on the port you are passing to the zookeeper client? You can check by running Echo stat| nc localhost

Re: Recovery issue - how to debug?

Hi Hao, As Vishal already asked, how are you determining if the writes are being received? Also, what was the status of C2 when you checked for these writes? Do you have the output of echo "stat" | nc localhost port? How long did you wait when you say that C2 did not received the writes? What

Re: rolling upgrade 3.2.1 -> 3.3.0

Hi Charity, Looks like you are hitting a bug recently found in 3.3.0. https://issues.apache.org/jira/browse/ZOOKEEPER-737 Is the bug, wherein the server does not show the right status. Looks like in your case the server is running fine but bin/zkserver.sh status is not returning the right res

Re: znode cversion decreasing?

). Thanks mahadev On 4/12/10 2:26 PM, "Kevin Webb" wrote: > On Mon, 12 Apr 2010 09:27:46 -0700 > Mahadev Konar wrote: > >> HI Kevin, >> >> The cversion should be monotonically increasing for the the znode. >> It would be a bug if its not. Can you please

Re: feed queue fetcher with hadoop/zookeeper/gearman?

Hi Thomas, There are a couple of projects inside Yahoo! that use ZooKeeper as an event manager for feed processing. I am little bit unclear on your example below. As I understand it- 1. There are 1 million feeds that will be stored in Hbase. 2. A map reduce job will be run on these feeds to f

Re: znode cversion decreasing?

HI Kevin, The cversion should be monotonically increasing for the the znode. It would be a bug if its not. Can you please elaborate in which cases you are seeing the cversion decreasing? If you can reproduce with an example that would be great. Thanks mahadev On 4/11/10 3:53 PM, "Kevin Webb"

Re: Errors while running sytest

Great. I was just responding with a different soln: '--- Looks like the fatjar does not include junit class. Also, the -jar option does not use the classpath environment variable. Here is an excerpt from the man page of java: -jar Execute a program encapsulated in

Re: deleting a node - command line tool

Hi Karthik, You can use bin/zkCli.sh which provides a nice command line shell interface for executing commands. Thanks mahadev On 3/26/10 9:42 AM, "Karthik K" wrote: > Hi - > I am looking to delete a node (say, /katta) from a running zk ensemble > altogether and curious if there is any com

Re: permanent ZSESSIONMOVED

Hi Lukasz, I looked at the logs and havent been able to determine how the session move can happen. You should go ahead and open a jira for this! Please attach the logs to that jira. Mark it for 3.3.0 until we can say that it isnt a problem as of now. Also, please go ahead and attach non fi

Re: java heap size

I am not sure I understand it. Lei, the default JVM size actually would not fit the JVM heap size of most of the applications that use ZooKeeper. Its always good to understand the requirements that are needed to run a system. I am not sure how its confusing. Is it the language or how to set it? T

Re: Zookeeper unit tester?

Hi David, We don't really have a mock test ZooKeeper client which does not do any I/O. We have been thinking about using mockito sometime soon to use for this kind of testing, but currently there is none. Thanks mahadev On 3/9/10 2:23 PM, "David Rosenstrauch" wrote: > Just wondering if there

Re: Managing multi-site clusters with Zookeeper

pend some more money >>> to >>> fix this. >>> >>> regards, >>> Martin >>> >>> On 8 March 2010 02:14, Ted Dunning wrote: >>> >>> If you can stand the latency for updates then zk should work well for >>>> yo

Re: Ok to share ZK nodes with Hadoop nodes?

Hi David, Sharing the cluster with HDFS and Map reduce might cause significant problems. Mapreduce is very IO intensive and this might cause lot of unnecessary hiccups in your cluster. I would suggest atleast providing something like this, if you really want to share the nodes. - atleast conside

Re: Managing multi-site clusters with Zookeeper

tact with each > other, this issue does not exist. > > regards, > Martin > > On 7 March 2010 21:43, Mahadev Konar wrote: > >> Hi Martin, >> As Ted rightly mentions that ZooKeeper usually is run within a colo >> because >> of the low latency requirements o

Re: Managing multi-site clusters with Zookeeper

Hi Martin, As Ted rightly mentions that ZooKeeper usually is run within a colo because of the low latency requirements of applications that it supports. Its definitely reasnoble to use it in a multi data center environments but you should realize the implications of it. The high latency/low throu

Re: zookeeper utils

Hi David, There is an implementation for locks and queues in src/recipes. The documentation residres in src/recipes/{lock/queue}/README.txt. Thanks mahadev On 3/2/10 1:04 PM, "David Rosenstrauch" wrote: > Was reading through the zookeeper docs on the web - specifically the > "recipes and sol

Re: how to lock one-of-many ?

Hi martin, Currently you cannot access the server that the client is connected to. This was fixed in this jira http://issues.apache.org/jira/browse/ZOOKEEPER-544 But again this does not tell you if you are connected to the primary or the other followers. So you will anyway have to do some manual

Re: is there a good pattern for leases ?

2/24/10 11:09 AM, "Martin Waite" wrote: > Hi Mahadev, > > That is interesting. All I need to do is hold the connection for the > required time of a session that created an ephemeral node. > > Zookeeper is an interesting tool. > > Thanks again, > Martin

Re: is there a good pattern for leases ?

Hi Martin, There isnt an inherent model for leases in the zookeeper library itself. To implement leases you will have to implement them at your application side with timeouts triggers (lease triggers) leading to session close at the client. Thanks mahadev On 2/24/10 3:40 AM, "Martin Waite" w

Re: how to lock one-of-many ?

Hi Martin, How about this- you have resources in the a directory (say /locks) each process which needs to lock, lists all the children of this directory and then creates an ephemeral node called /locks/resource1/lock depending on which resource it wants to lock. This ephemeral node will b

Re: Bit of help debugging a TIMED OUT session please

ut lads) > > St.Ack > > > On Mon, Feb 22, 2010 at 11:26 AM, Mahadev Konar wrote: >> I also looked at the logs. Ted might have a point. It does look like that >> zookeeper server's are doing fine (though as ted mentions the skew is a >> little concerning, th

Re: Bit of help debugging a TIMED OUT session please

I also looked at the logs. Ted might have a point. It does look like that zookeeper server's are doing fine (though as ted mentions the skew is a little concerning, though that might be due to very few packets served by the first server). Other than that the latencies of 300 ms at max should not ca

Re: Zab documentation ?

Hi Kay, There is a paper "Brief Announcement Zab A Practical Totally Ordered Broadcast Protocol." in DISC 2009 By ben and flavio which has the details of zab in it. Thanks mahadev On 2/18/10 6:07 PM, "Kay Kay" wrote: > I am trying to figure out more information about Zab and came across th

Re: ZooKeeper packages for Ubuntu

Great to hear this mahadev On 2/16/10 11:55 AM, "Gustavo Niemeyer" wrote: > Hello everyone, > > Thanks to Matthias Klose and Thierry Carrez, we've got ZooKeeper > packaged for Ubuntu: > > https://launchpad.net/~ttx/+archive/ppa > > This is a Personal Package Archive at the moment, b

Re: How do I get added child znode?

Hi Kim, The zookeeper api does not provide an api to get the znode that was added or deleted. You will have to compare the last set of children and new set of children to see which one was added or deleted. Thanks mahadev On 2/16/10 5:47 AM, "neptune" wrote: > Hi all, I'm kimhj. > > I have

Re: Ordering guarantees for async callbacks vs watchers

Hi martin, a call like getchildren(final String path, Watcher watcher, ChildrenCallback cb, Object ctx) Means that set a watch on this node for any further changes on the server. A client will see the response to getchildren data before the above watch is fired. Hope that helps. Thanks mahadev

Re: When session expired event fired?

Hi, a zookeeper client does not expire a session until and unless it is able to connect to one of the servers. In your case if you kill all the servers, the client is not able to connect to any of the servers and will keep trying to connect to the three servers. It cannot expire a session on its o

ZOOKEEPER-22 and release 3.3

Hi all, I had been working on zookeeper-22 and found out that it needs quite a few extensive changes. We will need to do some memory measurements to see if it has any memory impacts or not. Since we are targetting 3.3 release for early march, ZOOKEEPER-22 would be hard to get into 3.3. I am prop

Re: Q about ZK internal: how commit is being remembered

Qian, ZooKeeper gurantees that if a client sees some transaction response, then it will persist but the one's that a client does not see might be discarded or committed. So in case a quorum does not log the transaction, there might be a case wherein a zookeeper server which does not have the log

Re: Q about ZK internal: how commit is being remembered

Qing, This is how it happens - Leader will issue a COMMIT only when a quorum of servers have logged the transaction to disk. The issuing of COMMIT just means that all the servers can make the transaction visible to the client. The COMMIT message is never logged. On leader crash, another mach

Re: Using zookeeper to assign a bunch of long-running tasks to nodes (without unhandled tasks and double-handled tasks)

Qian, That's a good point, for 2) the application certainly needs to know that CONNECTIONLOSS has happened and it cannot connect to any of the servers. After spending some time on ZOOKEEPER-22, I am a little concerned about how much ZOOKEEPER-22 can do. I will post my thoughts/findings/concerns on

  1   2   3   >