FW: [ANN] katta-0.1.0 release - distribute lucene indexes in a grid

2008-09-17 Thread Mahadev Konar
-- Forwarded Message From: Stefan Groschupf [EMAIL PROTECTED] Reply-To: [EMAIL PROTECTED] Date: Wed, 17 Sep 2008 17:06:19 -0700 To: [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: [ANN] katta-0.1.0 release - distribute lucene indexes in a grid After 5

Re: ZooKeeper Roadmap - 3.1.0 and beyond.

2008-11-18 Thread Mahadev Konar
Hi Krishna, Sorry for the delayed response. The responses are in line. On 11/18/08 12:02 PM, Krishna Sankar (ksankar) [EMAIL PROTECTED] wrote: Have a couple of questions on the proposed multi-tenancy feature (pardon me if they are obvious, as I am slowly getting up to speed): a) First,

Re: ZooKeeper 3.0 Fix Release slated for end of this week.

2008-12-01 Thread Mahadev Konar
Hi Jake, We are in the process of releasing. The release is up for vote. As soon as the release passes, we will put up the release. Here is the email by Pat. On 11/24/08 5:50 PM, Patrick Hunt [EMAIL PROTECTED] wrote: From: Patrick Hunt [EMAIL PROTECTED] Date: November 24, 2008 5:50:24 PM

Re: InterruptedException

2009-01-06 Thread Mahadev Konar
Hi Kevin, The interrupt exception would be thrown in case any other thread tries to interrupt zookeeper threads during a client call (its not really interrupting the server but interrupting the client threads). Its like any synchronous operation that waits throwing an interrupted exception if

Re: event re-issue on reconnect?

2009-01-06 Thread Mahadev Konar
Does onData mean a datawatch? onConnect onData path: /foo, version: 4, data: '2333' onDisconnect onConnect onData path: /foo, version: 4, data: '2333' Are these the sequence of events that you get on the client? mahadev On 1/6/09 5:03 PM, Kevin Burton bur...@spinn3r.com wrote: I have an

Re: Simpler ZooKeeper event interface....

2009-01-07 Thread Mahadev Konar
Hi Vinod, I think what Ben meant was this-- The client will never know of a session expiration until and unless its connected to one of the servers. So the leader cannot demote itself since its connected to one of the servers. It might have lost its session (which all the others except itself

Re: ouch, zookeeper infinite loop

2009-01-07 Thread Mahadev Konar
The version of Jute we use is really an ancient version of recordio ser/deser library in hadoop. We do want to move to some better(versioned/fast/well accepted) ser/deser library. mahadev On 1/7/09 12:08 PM, Kevin Burton bur...@spinn3r.com wrote: Ah... you think it was because it was empty?

Re: Does session expiration only happen during total ensemble failure or network split?

2009-01-07 Thread Mahadev Konar
Kevin, the case you mention, the session is not really expired unless the quorum decides to expire it. So the client assuming that the session expired would be wrong to say. It is possible that as soon as you bring up the servers, the client reconnects with the same session and the session is

Re: Maximum number of children

2009-01-12 Thread Mahadev Konar
I was going to suggest bucketing with predifined hashes. /root/template/data/hashbucket/hash For the issue raised by Joshua regarding the length of the output from the server -- This is a bug. We seem to allow any number of children ( int) of a node and the getchildren call fails to return the

Delaying 3.2 release by 2 to 3 weeks?

2009-01-15 Thread Mahadev Konar
Hi all, I needed to get quotas in zookeeper 3.2.0 and wanted to see if delaying the release by 2-3 weeks is ok with everyone? Here is the jira for it - http://issues.apache.org/jira/browse/ZOOKEEPER-231 Please respond if you have any issues with the delay. thanks mahadev

Re: Delaying 3.1 release by 2 to 3 weeks?

2009-01-15 Thread Mahadev Konar
That was release 3.1 and not 3.2 :) mahadev On 1/15/09 4:26 PM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi all, I needed to get quotas in zookeeper 3.2.0 and wanted to see if delaying the release by 2-3 weeks is ok with everyone? Here is the jira for it - http://issues.apache.org

Re: Dealing with session expired

2009-02-12 Thread Mahadev Konar
Hi Tom, We prefer to discard the zookeeper instance if a session expires. Maintaining a one to one relationship between a client handle and a session makes it much simpler for users to understand the existence and disappearance of ephemeral nodes and watches created by a zookeeper client.

Re: Dealing with session expired

2009-02-12 Thread Mahadev Konar
On Thu, Feb 12, 2009 at 2:12 PM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi Tom, We prefer to discard the zookeeper instance if a session expires. Maintaining a one to one relationship between a client handle and a session makes it much simpler for users to understand the existence

Re: Watcher guarantees

2009-02-13 Thread Mahadev Konar
If client sets a watcher on a znode by doing a getData operation is it guaranteed to get the next change after the value it read, or can a change be missed? The watch is just a notification that the node changed. If you do a getData on the node, you their might have been more updates on the

Re: Recommended session timeout

2009-02-23 Thread Mahadev Konar
Hi Joey, here is a link to information on session timeouts. http://hadoop.apache.org/zookeeper/docs/r3.0.1/zookeeperProgrammers.html#ch_ zkSessions The session timeouts depends on how sensitive you want your application to be. A very low session timeout like (1-2 seconds) might lead to your

Re: Anyone using Zookeeper in AWS (Amazon Cloud)?

2009-02-26 Thread Mahadev Konar
Hi Greg, As for cross datacenters, we have tested zookeeper cross data centers and it works fine. The only thing is that you might have to tweak synclimit and tickTime to a little higher values for Zookeeper. http://hadoop.apache.org/zookeeper/docs/r3.1.0/zookeeperAdmin.html#sc_config uration

Re: How large an ensemble can one build with Zookeeper?

2009-03-03 Thread Mahadev Konar
HI Chad, The maximum number of zookeeper servers we have tested with is 13. Even with 13 the performance starts to degrade very quickly (compared to ensemble of 5 and 7). I am not sure we have the current numbers (we have made 3x or so performance improvements) but with the old number in

Re: Distributed Lock Libraries

2009-03-06 Thread Mahadev Konar
Hi Fernando, Our 3.2 release is focussed more on having such recipes in a clieaner and reliable way. One of the jiras that focusses on this is http://issues.apache.org/jira/browse/ZOOKEEPER-78. We hope to add more of such recipes in 3.2. Thanks mahadev On 3/6/09 2:30 PM, Fernando Padilla

Re: Dynamic addition of servers to Zookeeper cluster

2009-03-13 Thread Mahadev Konar
Hi Raghu, You are right that the cluster configuration is a static one. To deal with this problem, we usually suggest that you change your configs for every server and then re hup all of them at the same time (almost the same time I mean). The clients would lose connections to the servers but

Re: Dynamic addition of servers to Zookeeper cluster

2009-03-13 Thread Mahadev Konar
By re hup I mean restart. Thanks mahadev On 3/13/09 4:54 PM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi Raghu, You are right that the cluster configuration is a static one. To deal with this problem, we usually suggest that you change your configs for every server and then re hup all

Re: Semantics of ConnectionLoss exception

2009-03-25 Thread Mahadev Konar
Hi Nitay, - Does this event happening mean my ephemeral nodes will go away? No. the client will try connecting to other servers and if its not able to reconnect to the servers within the remaining session timeout. If the client is not able to connect within the remaining session timeout, the

Re: Semantics of ConnectionLoss exception

2009-03-26 Thread Mahadev Konar
Isn't it the case that the client won't get session expired until it's able to connect to a server, right? So what might happen is that the client loses connection to the server, the server eventually expires the client and deletes ephemerals (notifying all watchers) but the client won't

Re: Semantics of ConnectionLoss exception

2009-03-26 Thread Mahadev Konar
ephemeral nodes are gone? Why not deliver the Session Expired event on the client side after the right time has passed without communication to any server? On Thu, Mar 26, 2009 at 10:58 AM, Mahadev Konar maha...@yahoo-inc.comwrote: Isn't it the case that the client won't get session expired

Re: Some one send me some demo of programming with C client API for Zookeeper

2009-04-16 Thread Mahadev Konar
Please take a look at src/c/src/cli.c for some examples on zookeeper c client usage. Also you can see the test cases. Also http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperProgrammers.html Will give you some exmaple code for c clients. mahadev On 4/16/09 2:30 AM, Qian Ye

Re: Server-client connection timeout

2009-04-21 Thread Mahadev Konar
Hi raghu, http://wiki.apache.org/hadoop/ZooKeeper/FAQ Explains on what timeouts mean for a zookeeper client. A timeout does not mean a closed session. The client will reconnect to another server and then renew the session. A closed session will make the zoookeeper handle invalid. Hope this

Re: Does anyone have done some strict performance testing on current Zookeeper?

2009-04-23 Thread Mahadev Konar
Hi Qian, I can give you an example of one of our systems that uses zookeeper (crawling for our Yahoo! search engine). It has of the order of 4K clients - expecting to grow to 8K to 12K. Their write load is around 100 writes/sec (this is pretty low) and 4K reads/sec (max reads) with an ensemble

Re: Multiple ZooKeeper client instances

2009-04-24 Thread Mahadev Konar
HI Satish, A zookeeper client usually has a very small footprint for memory and cpu. The mutithreaded version of zookeeper client creates an internal thread to do the io and callbacks. I would suggest using the same zookeeper client across the objects to have less number of threads in your

Re: Dynamic server addition/deletion

2009-05-01 Thread Mahadev Konar
Hi Raghu and Ted, Theire is already an open jira on this -- http://issues.apache.org/jira/browse/ZOOKEEPER-107 You can go through the suggestions on it and can continue the discussion on the jira. Please feel free to add your ideas to the jira. Also, I don't think anyone is working on it (to

Re: Moving ZooKeeper Servers

2009-05-04 Thread Mahadev Konar
Hi Satish, Is the re generation of state in production something that is not acceptable? Copying over the whole datadir and datalogdir as it is maintaining the dirctory structure would be necessary. Also, in general this is a bad idea (just to warn you) since you would have to be careful with

Re: Moving ZooKeeper Servers

2009-05-06 Thread Mahadev Konar
from the pre-production cluster and write to the production one is much more sound. If you can't do that, you may need to rethink your processes since they are likely to be delicate for other reasons as well. On Mon, May 4, 2009 at 2:35 PM, Mahadev Konar maha...@yahoo-inc.com wrote: So

Re: Some thoughts on Zookeeper after using it for a while in the CXF/DOSGi subproject

2009-05-29 Thread Mahadev Konar
Hi David, I second this. If you folks want to host at http://scala-tools.org we'd be happy to host this non-Scala but super mega interesting and valuable project. We'll even do builds on our hudson server. We have had plans to publish our release on maven repos but havent had the time to

Re: ZooKeeper heavy CPU utilisation

2009-06-02 Thread Mahadev Konar
Hi Satish, Can you attach this trace to a jira? Please open one for this. Also, can you do the following - For all the threads for the zookeeper server you are seeing the problem on, Can you do an strace on all the threads and see which thread is spinning? Also, can you upload the configs of

Re: Errors during shutdown/startup of ZooKeeper

2009-06-02 Thread Mahadev Konar
Hi Nitay, This is not an error but should be a warning. I have opened up a jira for it. http://issues.apache.org/jira/browse/ZOOKEEPER-428 The message just says that a client is connecting to a server that is behind that a server is was connected to earlier. The log should be warn and not

Re: Win32 as a production platform

2009-06-04 Thread Mahadev Konar
Hi Marc, The only thing missing would be testing and support. We do most of our testing on linux boxes and for the same reason its easy for us to support the platforms that we use. We do not have access to windows boxes to test and (therefore) support windows as a suggested production platform.

Re: General Question about Zookeeper

2009-06-25 Thread Mahadev Konar
Hi Harold, As Henry mentioned, what acl's provide you is preventing access to znodes. If someone has access to zookeeper's data stored on zookeeper's server machines, they should be able to resconstruct the data and read it (using zookeeper deserialization code). I am not sure what kind of

Re: General Question about Zookeeper

2009-06-25 Thread Mahadev Konar
Hi Harold, Let me explain the whole concept of ZooKeeper Acls. 1) Zookeeper servers are run using some user id say X 2) zookeeper client use ZooKeeper client libaryr to create zookeeper nodes on zookeeper servers. They could be running as user id C. They can provide acl's to create such nodes

Re: Help to compile Zookeeper C API on a old system

2009-07-06 Thread Mahadev Konar
, 2009 at 2:22 AM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi Qian, What issues do you face? I have never tried compiling with the configuration below, but I could give it a try in my free time to see if I can get it to compile. mahadev On 7/6/09 7:37 AM, Qian Ye yeqian

Re: Instantiating HashSet for DataNode?

2009-07-14 Thread Mahadev Konar
Hi Erik, I am not sure if that would a considerable opitmization but even if you wanted to do it, it would be much more than just adding a check in the constructor (the serialization/deserialization would need to have specialized code). Right now all the datanodes are treated equally for

Re: Queue code

2009-07-17 Thread Mahadev Konar
Also are there any performance numbers of zookeeeper based queues. How does it compare with JMS. thanks Kishore G Hi Kishore, We do not have any performance number fr queues on zookeeper. I think you can get a rough idea of those numbers from your usage of zookeeper (number of reads/writes

Re: Leader Elections

2009-07-20 Thread Mahadev Konar
Both of the options that Scott mentioned are quite interesting. Quite a few of our users are interested in these two features. I think for 2, we should be able to use observers with a subscription to the master cluster with interested in a special subtree. That avoids too much of cross talk.

Bug in 3.2 release.

2009-07-23 Thread Mahadev Konar
Hi folks, We just discovered a bug in 3.2 release http://issues.apache.org/jira/browse/ZOOKEEPER-484. This bug will affect your clients whenever they switch zookeeper servers - from a zookeeper server that is a follower to a server that is leader. We should have a fix out by next week in

Re: bad svn url : test-patch

2009-07-30 Thread Mahadev Konar
Hi Todd, Yes this happens with the branch 3.2. The test-patch link is broken becasuse of the hadoop split. This file is used for hudson test environment. It isnt used anywhere else, so the svn co otherwise should be fine. We should fix it anyways. Thanks mahadev On 7/30/09 2:57 PM, Todd

Re: c client error message with chroot

2009-08-03 Thread Mahadev Konar
This looks like a bug. Does this happen without doing any reads/writes using the zookeeper handle? Please do open a jira for this. Thanks mahadev On 8/2/09 10:53 PM, Michi Mutsuzaki mi...@cs.stanford.edu wrote: Hello, I'm doing something like this (using zookeeper-3.2.0): zhandle_t*

Re: question about watcher

2009-08-05 Thread Mahadev Konar
Hi Qian, There isnt any such api. We have been thinking abt adding an api on cancelling a cleints watches. We have been thinking about adding a proc filesystem wherein a cleintt will have a list of all the watches. This data can be used to know which clients are watching what znode, but this has

Re: Runtime Interrogation of the Ensemble

2009-08-31 Thread Mahadev Konar
Hi Todd, You can use jmx to to find such information. Also you can just do this Echo stat | nc localhost clientport To get status from the zookeeper servers. This is all documented in the forrest docs at http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperAdmin.html Hope this helps.

Re: zookeeper on ec2

2009-09-01 Thread Mahadev Konar
Hi Satish, Connectionloss is a little trickier than just retrying blindly. Please read the following sections on this - http://wiki.apache.org/hadoop/ZooKeeper/ErrorHandling And the programmers guide: http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperProgrammers.html To learn more

Re: Infinite ping after calling setData()

2009-09-15 Thread Mahadev Konar
they to long to add to email at this time. Unfortunately I am having completely different issues now with the servers not shutting down. When I get past that and if I run into this issue again I will give more details. Thanks. -Original Message- From: Mahadev Konar

Re: ACL question w/ Zookeeper 3.1.1

2009-09-17 Thread Mahadev Konar
HI todd, From what I understand, you are sayin that a creator_all_acl does not work with auth? I tried the following with CREATOR_ALL_ACL and it seemed to work for me... import org.apache.zookeeper.CreateMode; import org.apache.zookeeper.WatchedEvent; import org.apache.zookeeper.Watcher;

Re: Start problem of Running Replicated ZooKeeper

2009-09-23 Thread Mahadev Konar
Hi Le, Is there some chance of the these servers not being able to talk to each other? IS the zookeeper prcoess running on debian-1? What error do you see on debian-1? The connection refused error suggests that debian-0 is not able to talk to debian-1 machine. Thanks mahadev On 9/23/09

Re: specifying the location of zookeeper.log

2009-10-16 Thread Mahadev Konar
Hi Leonard, You should be able to set the ZOO_LOG_DIR as an environment variable to get a different log directory. I think you are using bin/zkServer.sh to start the server? Also, please open a jira for this. It would be good to fix the documentation for this. Thanks mahadev On 10/16/09

Re: specifying the location of zookeeper.log

2009-10-16 Thread Mahadev Konar
} \ -cp $CLASSPATH $JVMFLAGS $ZOOMAIN $ZOOCFG I double checked by echo'ing the value of ZOO_LOG_DIR just before the java command. It's set correctly ... but it has no effect on the location of zookeeper.log :-( Leonard On 10/16/09 11:08 AM, Mahadev Konar maha...@yahoo-inc.com wrote

Re: specifying the location of zookeeper.log

2009-10-16 Thread Mahadev Konar
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout log4j.appender.FILE.layout.ConversionPattern=%d{ISO8601} - %-5p [%t:%c...@%l] - %m%n Will let you log to the output directory $dir. Hope that helps! mahadev On 10/16/09 11:35 AM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi Leonard, Looks

Re: specifying the location of zookeeper.log

2009-10-16 Thread Mahadev Konar
and forgotten that I'd done so. Thanks for your attention. Leonard On 10/16/09 11:35 AM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi Leonard, Looks like you are right. bin/zkServer.sh just logs the output to console, so you should be able to redirect to any file you want. No? Anyways

Re: Cluster Configuration Issues

2009-10-20 Thread Mahadev Konar
HI Mark, ZooKeeper does not create the myid file in the data directory. Looking at the config file it looks like it is missing the quorum configuration for other servers. Please take alook at http://hadoop.apache.org/zookeeper/docs/r3.2.1/zookeeperAdmin.html#sc_zkMuli tServerSetup You will

Re: Restarting a single zookeeper Server on the same port within the process

2009-10-22 Thread Mahadev Konar
Hi Siddharth, Usually the time of releasing the port is dependent on the OS. So you can try sleeping a few more seconds to see if the port has been released or it .. Or just poll on the port to see if its in use or not There isnt an easier way to restart on the same port. mahadev On

Re: zookeeper viewer

2009-10-24 Thread Mahadev Konar
Hi Hamoun, Can you please mention which link is broken? Are you a looking for a zookeeper tree browser? Pat created a dashboard for zookeeper at github. Below is the link: http://github.com/phunt/zookeeper_dashboard Also, there is an open jira for a zookeeper browser which you can try

Bugfix release 3.2.2

2009-10-30 Thread Mahadev Konar
Hi all, We are planning to make a bugfix release 3.2.2 which will include a critical bugfix in the c client code. The jira is ZOOKEEPER-562, http://issues.apache.org/jira/browse/ZOOKEEPER-562. If you would like some fix to be considered for this bugfix release please feel free to post on the

Re: zkfuse

2009-11-24 Thread Mahadev Konar
Hi Maarten, zkfuse does not have any support for acls. We havent had much time to focus on zkfuse. Create/read/write/delete/ls are all supported. It was built mostly for infrequent updates and more of a browsing interface on filesystem. I don't think zkfuse is being used in production anywhere.

Re: Fetching sequential children

2009-12-23 Thread Mahadev Konar
Hi ohad, there isnt a way to get a selected set of children from the servers. So you will have to get all of them and filter out the unwanted ones. Also, what Steve suggested in the other email might be useful for you. Thanks mahadev On 12/23/09 12:29 AM, Ohad Ben Porat o...@outbrain.com

Re: Killing a zookeeper server

2010-01-13 Thread Mahadev Konar
Hi Adam, That seems fair to file as an improvement. Running 'stat' did return the right stats right? Saying the servers werent able to elect a leader? mahadev On 1/13/10 11:52 AM, Adam Rosien a...@rosien.net wrote: On a related note, it was initially confusing to me that the server

Re: Question regarding Membership Election

2010-01-14 Thread Mahadev Konar
Hi Vijay, Unfortunately you wont be able to keep running the observer in the other DC if the quorum in the DC 1 is dead. Most of the folks we have talked to also want to avoid voiting across colos. They usually run two instances of Zookeeper in 2 DC's and copy state of zookeeper (using a bridge)

Re: Namespace partitioning ?

2010-01-14 Thread Mahadev Konar
Hi kay, the namespace partitioning in zookeeper has been on a back burner for a long time. There isnt any jira open on it. There had been some discussions on this but no real work. Flavio/Ben have had this on there minds for a while but no real work/proposal is out yet. May I know is this

Re: Server exception when closing session

2010-01-22 Thread Mahadev Konar
Hi Josh, This warning is not of any concern. Just a quick question, is there any reason for you to runn the server on a DEBUG level? Thanks mahadev On 1/22/10 5:19 PM, Josh Scheid jsch...@velocetechnologies.com wrote: Is it normal for client session close() to cause a server exception?

Re: Server exception when closing session

2010-01-22 Thread Mahadev Konar
at: http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview You can take a look at that as well and see what the expected performance should be for your workload. Thanks mahadev On 1/22/10 5:40 PM, Josh Scheid jsch...@velocetechnologies.com wrote: On Fri, Jan 22, 2010 at 17:22, Mahadev

Re: Q about ZK internal: how commit is being remembered

2010-01-28 Thread Mahadev Konar
Qian, ZooKeeper gurantees that if a client sees some transaction response, then it will persist but the one's that a client does not see might be discarded or committed. So in case a quorum does not log the transaction, there might be a case wherein a zookeeper server which does not have the

ZOOKEEPER-22 and release 3.3

2010-02-02 Thread Mahadev Konar
Hi all, I had been working on zookeeper-22 and found out that it needs quite a few extensive changes. We will need to do some memory measurements to see if it has any memory impacts or not. Since we are targetting 3.3 release for early march, ZOOKEEPER-22 would be hard to get into 3.3. I am

Re: Ordering guarantees for async callbacks vs watchers

2010-02-10 Thread Mahadev Konar
Hi martin, a call like getchildren(final String path, Watcher watcher, ChildrenCallback cb, Object ctx) Means that set a watch on this node for any further changes on the server. A client will see the response to getchildren data before the above watch is fired. Hope that helps. Thanks

Re: Bit of help debugging a TIMED OUT session please

2010-02-22 Thread Mahadev Konar
at 11:26 AM, Mahadev Konar maha...@yahoo-inc.com wrote: I also looked at the logs. Ted might have a point. It does look like that zookeeper server's are doing fine (though as ted mentions the skew is a little concerning, though that might be due to very few packets served by the first server). Other

Re: how to lock one-of-many ?

2010-02-23 Thread Mahadev Konar
Hi Martin, How about this- you have resources in the a directory (say /locks) each process which needs to lock, lists all the children of this directory and then creates an ephemeral node called /locks/resource1/lock depending on which resource it wants to lock. This ephemeral node will

Re: is there a good pattern for leases ?

2010-02-24 Thread Mahadev Konar
waite@googlemail.com wrote: Hi Mahadev, That is interesting. All I need to do is hold the connection for the required time of a session that created an ephemeral node. Zookeeper is an interesting tool. Thanks again, Martin On 24 February 2010 17:00, Mahadev Konar maha...@yahoo

Re: how to lock one-of-many ?

2010-02-24 Thread Mahadev Konar
Hi martin, Currently you cannot access the server that the client is connected to. This was fixed in this jira http://issues.apache.org/jira/browse/ZOOKEEPER-544 But again this does not tell you if you are connected to the primary or the other followers. So you will anyway have to do some

Re: zookeeper utils

2010-03-02 Thread Mahadev Konar
Hi David, There is an implementation for locks and queues in src/recipes. The documentation residres in src/recipes/{lock/queue}/README.txt. Thanks mahadev On 3/2/10 1:04 PM, David Rosenstrauch dar...@darose.net wrote: Was reading through the zookeeper docs on the web - specifically the

Re: Managing multi-site clusters with Zookeeper

2010-03-07 Thread Mahadev Konar
Hi Martin, As Ted rightly mentions that ZooKeeper usually is run within a colo because of the low latency requirements of applications that it supports. Its definitely reasnoble to use it in a multi data center environments but you should realize the implications of it. The high latency/low

Re: Managing multi-site clusters with Zookeeper

2010-03-07 Thread Mahadev Konar
21:43, Mahadev Konar maha...@yahoo-inc.com wrote: Hi Martin, As Ted rightly mentions that ZooKeeper usually is run within a colo because of the low latency requirements of applications that it supports. Its definitely reasnoble to use it in a multi data center environments but you should

Re: Managing multi-site clusters with Zookeeper

2010-03-08 Thread Mahadev Konar
to better than zk does and still maintain correctness. Do note that you can, probalbly bias client to use a local server. That should make things more efficient. Sent from my iPhone On Mar 7, 2010, at 3:00 PM, Mahadev Konar maha...@yahoo-inc.com wrote: The inter-site links

Re: Zookeeper unit tester?

2010-03-09 Thread Mahadev Konar
Hi David, We don't really have a mock test ZooKeeper client which does not do any I/O. We have been thinking about using mockito sometime soon to use for this kind of testing, but currently there is none. Thanks mahadev On 3/9/10 2:23 PM, David Rosenstrauch dar...@darose.net wrote: Just

Re: deleting a node - command line tool

2010-03-26 Thread Mahadev Konar
Hi Karthik, You can use bin/zkCli.sh which provides a nice command line shell interface for executing commands. Thanks mahadev On 3/26/10 9:42 AM, Karthik K oss@gmail.com wrote: Hi - I am looking to delete a node (say, /katta) from a running zk ensemble altogether and curious if

Re: Errors while running sytest

2010-04-07 Thread Mahadev Konar
Great. I was just responding with a different soln: '--- Looks like the fatjar does not include junit class. Also, the -jar option does not use the classpath environment variable. Here is an excerpt from the man page of java: -jar Execute a program encapsulated in

Re: feed queue fetcher with hadoop/zookeeper/gearman?

2010-04-12 Thread Mahadev Konar
Hi Thomas, There are a couple of projects inside Yahoo! that use ZooKeeper as an event manager for feed processing. I am little bit unclear on your example below. As I understand it- 1. There are 1 million feeds that will be stored in Hbase. 2. A map reduce job will be run on these feeds to

Re: rolling upgrade 3.2.1 - 3.3.0

2010-04-14 Thread Mahadev Konar
Hi Charity, Looks like you are hitting a bug recently found in 3.3.0. https://issues.apache.org/jira/browse/ZOOKEEPER-737 Is the bug, wherein the server does not show the right status. Looks like in your case the server is running fine but bin/zkserver.sh status is not returning the right

Re: Recovery issue - how to debug?

2010-04-19 Thread Mahadev Konar
Hi Hao, As Vishal already asked, how are you determining if the writes are being received? Also, what was the status of C2 when you checked for these writes? Do you have the output of echo stat | nc localhost port? How long did you wait when you say that C2 did not received the writes? What

Re: odd error message

2010-04-20 Thread Mahadev Konar
Ok, I think this is possible. So here is what happens currently. This has been a long standing bug and should be fixed in 3.4 https://issues.apache.org/jira/browse/ZOOKEEPER-335 A newly elected leader currently doesn't log the new leader transaction to its database In your case, the

Re: bug: wrong heading in recipes doc

2010-04-22 Thread Mahadev Konar
I think we should be using zookeeper locks to create jiras :) . Looks like both of you created one!!! :) Thanks mahadev On 4/22/10 1:37 PM, Patrick Hunt ph...@apache.org wrote: No problem. https://issues.apache.org/jira/browse/ZOOKEEPER-752 I've seen alot of traffic on

Re: Embedding ZK in another application

2010-04-23 Thread Mahadev Konar
Hi Vishal and Ashanka, I think Ted and Pat had somewhat comentted on this before. Reiterating these comments below. If you are ok with these points I see no concern in ZooKeeper as an embedded application. Also, as Pat mentioned earlier there are some cases where the server code will

Re: Embedding ZK in another application

2010-04-23 Thread Mahadev Konar
That's true! Thanks mahadev On 4/23/10 11:41 AM, Asankha C. Perera asan...@apache.org wrote: Hi Mahadev I think Ted and Pat had somewhat comentted on this before. Reiterating these comments below. If you are ok with these points I see no concern in ZooKeeper as an embedded

Re: Zookeeper client

2010-04-27 Thread Mahadev Konar
HI Avinash, The zk client does itself maintain liveness information and also randomizes the list of servers to balance the number of clients connected to a single ZooKeeper server. Hope that helps. Thanks mahadev On 4/27/10 10:56 AM, Avinash Lakshman avinash.laksh...@gmail.com wrote: Let's

Re: Embedding ZK in another application

2010-04-29 Thread Mahadev Konar
We do set that Chad but it doesn't seem to help on some systems (especially bsd)... Thanks mahadev On 4/29/10 11:22 AM, Chad Harrington chad.harring...@gmail.com wrote: On Thu, Apr 29, 2010 at 8:49 AM, Patrick Hunt ph...@apache.org wrote: This is not foolproof however. We found that in

Re: Question on maintaining leader/membership status in zookeeper

2010-04-30 Thread Mahadev Konar
Hi Lei, In this case, the Leader will be disconnected from ZK cluster and will give up its leadership. Since its disconnected, ZK cluster will realize that the Leader is dead! When Zk cluster realizes that the Leader is dead (this is because the zk cluster hasn't heard from the Leader for a

Re: Question on maintaining leader/membership status in zookeeper

2010-04-30 Thread Mahadev Konar
, neither the leader nor the slaves will be considered disconnected because they can all connect to ZK. Thanks, Lei On 4/30/10 3:47 PM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi Lei, In this case, the Leader will be disconnected from ZK cluster and will give up its leadership

Re: Question on maintaining leader/membership status in zookeeper

2010-04-30 Thread Mahadev Konar
connect to ZK. Thanks, Lei On 4/30/10 3:47 PM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi Lei, In this case, the Leader will be disconnected from ZK cluster and will give up its leadership. Since its disconnected, ZK cluster will realize that the Leader is dead! When Zk

Re: Dynamic adding/removing ZK servers on client

2010-05-03 Thread Mahadev Konar
Hi Dave, Just a question on how do you see it being used, meaning who would call addserver and removeserver? It does seem useful to be able to do this. This is definitely worth working on. You can link it as a subtask of ZOOKEEPER-107. Thanks mahadev On 5/3/10 7:03 AM, Dave Wright

Re: Dynamic adding/removing ZK servers on client

2010-05-03 Thread Mahadev Konar
Yeah, that was one of the ideas, I think its been on the jira somewhere ( I forget)... But could be and would definitely be one soln for it. Thanks mahadev On 5/3/10 2:12 PM, Ted Dunning ted.dunn...@gmail.com wrote: Should this be a znode in the privileged namespace? On Mon, May 3, 2010 at

Re: avoiding deadlocks on client handle close w/ python/c api

2010-05-04 Thread Mahadev Konar
Sure, Ill take a look at it. Thanks mahadev On 5/4/10 2:32 PM, Patrick Hunt ph...@apache.org wrote: Thanks Kapil, Mahadev perhaps you could take a look at this as well? Patrick On 05/04/2010 06:36 AM, Kapil Thangavelu wrote: I've constructed a simple example just using the zkpython

Re: ZKClient

2010-05-04 Thread Mahadev Konar
Hi Adam, I don't think zk is very very hard to get right. There are exmaples in src/recipes which implements locks/queues/others. There is ZOOKEEPER-22 to make it even more easier for application to use. Regarding re registration of watches, you can deifnitely write code and submit is as a part

Re: ZookeeperPresentations Wiki

2010-05-11 Thread Mahadev Konar
I just emailed in...@apache to ask for there help on this. I wasn't able to figure out what the problem is! Thanks for pointing it out. mahadev On 5/11/10 4:01 PM, Sudipto Das sudi...@cs.ucsb.edu wrote: Hi, I am trying to download some presentation slides from the ZookeeperPresentations

Re: New ZooKeeper client library Cages

2010-05-11 Thread Mahadev Konar
Hi Dominic, Good to see this. I like the name cages :). You might want to post to the list what cages is useful for. I think quite a few folks would be interested in something like this. Are you guys currently using it with cassandra? Thanks mahadev On 5/11/10 4:02 PM, Dominic Williams

Re: Xid out of order. Got 8 expected 7

2010-05-12 Thread Mahadev Konar
Hi Jordan, Can you create a jira for this? And attach all the server logs and client logs related to this timeline? How did you start up the servers? Is there some changes you might have made accidentatlly to the servers? Thanks mahadev On 5/12/10 10:49 AM, Jordan Zimmerman

Re: Can't ls with large node count and I don't understand the use of jute.maxbuffer

2010-05-13 Thread Mahadev Konar
Hi Aaaron, Each of the requests and response between client and servers is sent an (buflen, buffer) packet. The content of the packets are then deserialized from this buffer. Looks like the size of the packet (buflen) is big in yoru case. We usually avoid sending/receiving large packets just

Re: Using ZooKeeper for managing solrCloud

2010-05-14 Thread Mahadev Konar
Hi Rakhi, You can read more abt monitoring zookeeper servers at http://hadoop.apache.org/zookeeper/docs/r3.3.0/zookeeperAdmin.html#sc_monito ring Thanks mahadev On 5/14/10 4:09 AM, Rakhi Khatwani rkhatw...@gmail.com wrote: Hi, I just went through the zookeeper tutorial and

Re: Zookeeper EventThread and SendThread

2010-05-20 Thread Mahadev Konar
Hi Nick, These threads are spawned with each zookeeper client handle. As soon as you create a zookeeper client object these threads are spawned. Are yu creating too many zookeeper client objects in your application? Htanks mahadev On 5/20/10 11:30 AM, Nick Bailey nicholas.bai...@rackspace.com

  1   2   >