Re: Question about resetting offsets and the high-level consumer

2012-11-13 Thread Jun Rao
It's more than that. For example, what if 2 consumers want to set different offsets for the same partition? Jun On Tue, Nov 13, 2012 at 4:08 PM, Michal Haris wrote: > Of course assuming that none of the consumers in the group are interested > in the missed messages in such case. > On Nov 12, 201

Re: version of zookeeper

2012-11-13 Thread Neha Narkhede
Thanks for pulling up that list Jason ! I think that makes sense and we can experiment with ZK 3.3.5. Not sure how widely 3.3.6 is tested though. Regarding the 3.4.x stability, the fact that serious issues are still being reported makes me uncomfortable - https://issues.apache.org/jira/browse/ZOO

Re: version of zookeeper

2012-11-13 Thread Jason Rosenberg
According to the zookeeper home page, the current stable releases are "3.3.6" and "3.4.4". There seem to be a few interesting bug fixes in the release notes for 3.3.5 & 3.3.6: http://zookeeper.apache.org/doc/r3.3.5/releasenotes.html#changes http://zookeeper.apache.org/doc/r3.3.6/releasenotes.html#

Re: Question about resetting offsets and the high-level consumer

2012-11-13 Thread Michal Haris
Of course assuming that none of the consumers in the group are interested in the missed messages in such case. On Nov 12, 2012 6:03 AM, "Jun Rao" wrote: > It's easy if there is only a single consumer in the group. In general, > however, there could be multiple consumers in the same group. So, som

Re: Invalid KafkaStreams as a result of Decoder problems

2012-11-13 Thread Neha Narkhede
I meant the former. Thanks, Neha On Tue, Nov 13, 2012 at 10:03 AM, Jason Rosenberg wrote: > Neha, > > If I understand correctly, are you saying that it should be possible to > re-create KafkaStreams, without having to create a new ConsumerConnector? > > Or are you saying that we should never get

Re: version of zookeeper

2012-11-13 Thread Neha Narkhede
> So, there are known api changes that would preclude use of 3.4.x with > kafka, it sounds like? Not really. Zookeeper 3.4.x is backwards compatible. Kafka 0.8 should still work with any of the stable zookeeper versions out there. Once ZK 3.4.x is stable, Kafka 0.8 would certainly benefit from the

Re: version of zookeeper

2012-11-13 Thread Jason Rosenberg
If you do ./sbt make-pom, it generates a pom with the version set to 3.3.4. And I do think that is the current version (if you look at the release notes, I think for 0.7.1, it mentions that one of the changes is to upgrade to 3.3.4 zookeeper client). Jason On Tue, Nov 13, 2012 at 12:38 PM, Josep

Re: Invalid KafkaStreams as a result of Decoder problems

2012-11-13 Thread Jason Rosenberg
Neha, If I understand correctly, are you saying that it should be possible to re-create KafkaStreams, without having to create a new ConsumerConnector? Or are you saying that we should never get an exception when iterating over a KafkaStream (e.g. the underlying implementation should just either

Re: version of zookeeper

2012-11-13 Thread Jason Rosenberg
Ok, So, there are known api changes that would preclude use of 3.4.x with kafka, it sounds like? Is there a notion of a supported version of zookeeper for each kafka release? How about upgrading the recommended version for 0.8 at least to the latest 3.3.x (3.3.6)? Jason On Tue, Nov 13, 2012

Re: Invalid KafkaStreams as a result of Decoder problems

2012-11-13 Thread Neha Narkhede
Bob/Jason, This is a bug. Ideally, we should allow restarting the consumer streams when they encounter an error. Can you file a JIRA ? Thanks, Neha On Mon, Nov 12, 2012 at 6:31 AM, Bob Cotton wrote: > Handling the exceptions in the Decoder is not a problem. > > It was just a larger question abo

Re: Re: kafka broker re-register

2012-11-13 Thread Neha Narkhede
Zookeeper session can expire due to a variety of reasons ranging from network connectivity to JVM garbage collections. I would start by looking into the latter at the client as well as server side. To answer your 2nd question, once the broker establishes a new session, it re-registers topics from i

Re: version of zookeeper

2012-11-13 Thread Neha Narkhede
I've been observing the zookeeper mailing list for 3.4.x and haven't found it to be particularly stable. There are still serious issues being reported like leader election failures, ephemeral node deletion. I'm sure they are being fixed and there are many early adopters. But my guess is that we sho

Re: version of zookeeper

2012-11-13 Thread Jun Rao
Jason, We are just not sure about the stability of ZK 3.4.4. Does anybody know? In fact, some of the features in ZK 3.4.x (like the multi operation) can improve the availability of Kafka in 0.8. So, we'd like to move to ZK 3.4.4 if it's been widely used and is stable. Thanks, Jun On Mon, Nov 12

.NET Client

2012-11-13 Thread John Haigh
Hello, I am working to get the latest version of the .NET Client to compile under Visual Studio 2010. I have what I think is the latest version of the .NET client from https://svn.apache.org/repos/asf/incubator/kafka/branches/legacy_client_libraries/csharp/ The version I have has the Apache licens

Re: version of zookeeper

2012-11-13 Thread David Arthur
The ZooKeeper API has been stable for a while, I do not suspect there would be any problems between 3.3.4 and 3.4.x. Personally, I have run Kafka 0.7.x with ZooKeeper 3.4.3 with no problems -David On Nov 13, 2012, at 2:30 AM, Jason Rosenberg wrote: > Hi, > > I'm wondering if there's a reason