Re: Consumer - Failed to find leader

Hi Harsha, This is my Kafka_server_jaas.config file. This is passed as JVM param to the Kafka broker while start up. = KafkaServer { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true storeKey=true serviceName="kafka"

Re: Migrate a topic which has no brokers

Hello Ben, Yes, you can use apply different replica assignment. See related docs: http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor Kind regards, Stevo Slavic. On Tue, Jan 5, 2016 at 5:32 PM, Ben Davison wrote: > Hi All, > > On our dev

Migrate a topic which has no brokers

Hi All, On our dev kafka environment our brokers went down with a topic on it, can we just reassign the partitions to another broker? Kafka 0.9 Thanks Ben -- This email, including attachments, is private and confidential. If you have received this email in error please notify the sender

Re: Best way to commit offset on demand

Yes, in this case you should use assign() instead of subscribe(). I'm not sure it's strictly necessary at the moment to use assign() in this case, but it would protect your code in case we later changed the logic to only permit commits on assigned partitions. It also doesn't really cost anything.

Re: Does MirrorMaker delete topics?

AFAIK "if the right options are set" actually means "if auto.create.topics.enable is left to default, set to true". As on any Kafka cluster with this configuration option, this will allow implicit topic creation, e.g. on first message being published to the topic, if topic does not exist it will

Does MirrorMaker delete topics?

Hi all, In the kafka docs, it mentions that MirrorMaker will automatically create topics on the mirror cluster if the right options are set. Does it automatically delete topics on the mirror that are deleted on the main cluster as well? Thanks, SimonC

RE: Does MirrorMaker delete topics?

Hmm, could this feature be added in the future? Although a better solution might be to turn mirrormaker into something like 'hidden replicas', where a certain set of brokers in a cross-site super-cluster replicate topics onto the secondary site but don't take part in topic leadership - is this

Re: Find current kafka memory usage

Hi Dillian, You can use a Linux script, general monitoring/alerting tool, or JMX (see here, for example: http://stackoverflow.com/questions/6487802/check-available-heapsize-programatically ). Let me know if you have a more specific question. Alex On Mon, Jan 4, 2016 at 4:42 PM, Dillian

Re: Migrate a topic which has no brokers

Stevo pointed you at the correct document for moving topics around. However, if you lost a broker, by far the easiest way to recover is to start a new broker and give it the same ID as the one that went down. On Tue, Jan 5, 2016 at 8:49 AM, Stevo Slavić wrote: > Hello Ben,

Re: Topic Deletion Issues

Hi Brenden, your previous email went through, and you got a response (that I believe answers your question). See here: http://mail-archives.apache.org/mod_mbox/kafka-users/201512.mbox/browser Alex On Mon, Jan 4, 2016 at 7:18 PM, Brenden Cobb wrote: > I might have sent

Re: Topic Deletion Issues

Thanks Alex (and sorry all for the spam) I'll give it another go as suggested, however in my tests I was just using the kafka-topics command and there was definitely a few seconds between creation/deletion. -BC On Tue, Jan 5, 2016 at 1:24 PM, Alex Loddengaard wrote: > Hi

Re: Topic Deletion Issues

Hi Brenden, I sent the wrong permalink. Try this: http://mail-archives.apache.org/mod_mbox/kafka-users/201512.mbox/%3CCAH7vnfhUQHQcCJ1R3pS_DYn0LvmA8fJtLtvCuvvDboWSqm-NBg%40mail.gmail.com%3E Alex On Tue, Jan 5, 2016 at 10:21 AM, Alex Loddengaard wrote: > Hi Brenden, your

Re: Topic Deletion Issues

Hi Gwen- No, unfortunately it doesn't appear to be consistent. One day I had no problem with rapid deletion/recreation, the next I did. Kafka is still obsessing about the last failed deletion attempt and won't move on to subsequent delete requests. I've planned downtime today so I can restart

Re: Topic Deletion Issues

If you have any way of capturing logs at trace level from the point of the failed attempt until 10 minutes after, it will be super useful. We know there are issues in deleting topics, but it doesn't reproduce well, we don't know if its one issue or several and we don't have a good starting point

Re: Does MirrorMaker delete topics?

Not sure, if this should be supported. For example. if we have 2 kafka clusters C1 and C2 mirroring data to C3-Aggregate using mirror-maker M1 and M2 respectively and if the topic exist in both C1 and C2 and if we delete topic from C1, M1 should not delete the topic from C3-Aggregate. Thanks,

Re: Does MirrorMaker delete topics?

Yes, there are quite a few related tickets in JIRA (see here ) On Tue, Jan 5, 2016 at

RE: kafka java producer security to access kerberos

How to pass Kerberos keytab file to kakfa producer properties The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the

Programmable API for Kafka Connect ?

Hi, Does Kafka Connect have an API which can be used by applications to start Kafka Connect, add/remove Connectors? I also do not want to use the REST API and do not want to start the REST server. Thanks, Shiti

Re: Unexpected 0.9 Consumer Behavior

Thanks for the clear and concise explanation Jason, makes sense now! On Tue, Jan 5, 2016 at 3:42 PM, Jason Gustafson wrote: > Hi Ben, > > The new consumer is single-threaded, so each instance should be given a > dedicated thread. Using multiple consumers in the same thread

End to end testing

Hi -- I got Kafka 0.8.2 and looking for ways to setup some end to end testing in Kafka to produce and consume messages to each partition for a given topic and replication is active. Is there a simple test suite that is available to perform these? Thanks!

Re: Topic Deletion Issues

By the way, https://issues.apache.org/jira/browse/KAFKA-2937 was filed recently and it includes a PR. Ismael On Tue, Jan 5, 2016 at 7:14 PM, Gwen Shapira wrote: > If you have any way of capturing logs at trace level from the point of the > failed attempt until 10 minutes

Unexpected 0.9 Consumer Behavior

Hi, Wondering if anyone can provide some insight into some unexpected behavior we're seeing with the 0.9 consumer: 1) We create two consumer instances, each with the same group.id and subscribe them to the same topic. 2) We would expect each consumer to be assigned 1/2 the topic partitions, but

Re: Unexpected 0.9 Consumer Behavior

Hi Ben, The new consumer is single-threaded, so each instance should be given a dedicated thread. Using multiple consumers in the same thread won't really work as expected because poll() blocks while the group is rebalancing. If both consumers aren't actively call poll(), then they won't be both