Re: mirror maker against 0.8.2 source cluster and 0.9.0 destination cluster

Hi Stephen, Newer brokers support older clients, but not the other way around. You could try 0.8.2 MirrorMaker against 0.8.2 source and 0.9.0 target clusters perhaps? Ismael On 6 Jan 2016 11:18, "Stephen Powis" wrote: > Hey! > > So I'm trying to get mirror maker going

Kafka Connect Embedded API

Hi, I wanted to use Kafka Connect in an application and wanted to if the Embedded API discussed in KIP-26 available in 0.9.0.0 or is there an alternative? Thanks, Shiti

mirror maker against 0.8.2 source cluster and 0.9.0 destination cluster

Hey! So I'm trying to get mirror maker going between two different clusters. My source cluster is version 0.8.2 and my destination cluster is 0.9.0 running the mirror maker code from the 0.9.0 release. Does anyone know if this is possible to do? I'm aware that the protocol changed slightly

Partition rebalancing after broker removal

Hi there, Kafka's `kafka-reassign-partitions.sh` tool currently has no mechanism for removing brokers. However, it does have the ability to generate partition plans across arbitrary sets of brokers, by using `--generate`, passing all the topics in the cluster into it, then passing the generated

Re: Best way to commit offset on demand

> in case we later changed the logic to only permit commits on assigned partitions I experienced this yesterday and was wondering why Kafka allows commits to partitions from other consumers than the assigned one. Does any one know of the reasoning behind this? Martin On 5 Jan 2016 18:29, "Jason

Leader Election

Hi Folks, I am trying to use the REST proxy, but I have some fundamental questions about how the leader election works. My understanding of how the way the leader elections work is that the proxy hides all of that complexity and processes my produce/consume request transparently. Is that the

Re: Programmable API for Kafka Connect ?

Hi Shiti, In the context of your question, there are three relevant components to Connect: connectors, tasks, and workers. Connectors do a number of things but at a high level they specify the tasks (Java classes) to perform the work, how many to start, and some coordination. The tasks do the

Re: Leader Election

Hi Heath, I assume you're referring to the partition leader. This is not something you need to worry about when using the REST API. Kafka handles leader election, failure recovery, etc. for you behind the scenes. Do know that if a leader fails, you'll experience a small latency hit because a new

Security in Kafka

In 0.9 release it's not clear if Security features of LDAP authentication and authorization are available? If authN and authZ are available can somebody point me to relevant documentation that shows how to configure Kafka to enable authN and authZ?

0.9 consumer reading a range of log messages

I want to use the new 0.9 consumer for a particular application. My use case is the following: i) The TopicPartition I need to poll has a short log say 10 mins odd (log.retention.minutes is set to 10). ii) I don't use a consumer group i.e. I manage the partition assignment myself. iii)

Re: Migrate a topic which has no brokers

Hi Gwen, Thanks for the suggestion, we are using Kafka 0.9's "auto broker ID" feature, is there a way to force a running kafka instance to take on a new broker ID? Thanks, Ben On Tue, Jan 5, 2016 at 6:19 PM, Gwen Shapira wrote: > Stevo pointed you at the correct document

Re: 0.9 consumer reading a range of log messages

Thanks or the replies Jason! A couple of follow up questions: i. If I call seekToEnd() on the consumer and then use position() is the position() call making a blocking IO call. It is not entirely clear from the documentation whether this will just block or not. ii. If the position() call does

Re: Migrate a topic which has no brokers

Hey, You can modify a broker ID by taking it down, changing the id configuration in the properties file and starting it. Note that the broker will lose its existing partitions and take the partitions of the broker it is replacing (i.e. a broker can't have two identities at the same time), so you

Re: 0.9 consumer reading a range of log messages

Hi Rajiv, Answers below: i) How do I get the last log offset from the Kafka consumer? To get the last offset, first call seekToEnd() and then use position(). ii) If I ask the consumer to seek to the beginning via the consumer > .seekToBeginning(newTopicPartition) call, will it handle the

Re: Security in Kafka

Mohit, In 0.9, Kafka supports Kerberos. So, you can authenticate with any directory service that supports Kerberos (e.g., Active Directory). Our default authorization is based on individual users, not on groups though. You can find more a bit more info on security in the following links.

Re: 0.9 consumer reading a range of log messages

On Wed, Jan 6, 2016 at 5:31 PM, Jason Gustafson wrote: > > > > i. If I call seekToEnd() on the consumer and then use position() is the > > position() call making a blocking IO call. It is not entirely clear from > > the documentation whether this will just block or not. > > >

Re: 0.9 consumer reading a range of log messages

> > i. If I call seekToEnd() on the consumer and then use position() is the > position() call making a blocking IO call. It is not entirely clear from > the documentation whether this will just block or not. > Yes, position() will block when the offset needs to be reset. You can depend on it

Re: Best way to commit offset on demand

Not sure there's a great reason. In the initial design, the server itself only permitted commits from consumers that were assigned the respective partitions, but we lost this when we generalized the group coordination protocol. It seems like it still makes sense to do it on the client though, so