Re: kafka streaming rocks db lock bug?

This was in 10.1.0. What happened was that a kafka broker went down and then this happened on the kafka streaming instance which had connection to this broker. I can send you all logs I got. Ara. On Oct 24, 2016, at 10:41 PM, Guozhang Wang mailto:wangg...@gmail.com>> wrote: Hello Ara, Your e

Re: kafka streaming rocks db lock bug?

Hello Ara, Your encountered issue seems to be KAFKA-3812 , and KAFKA-3938 . Could you try to upgrade to the newly released 0.10.1.0 version and see if this issue goes away? If not I would love to in

Re: customised event time

Hi Hamza, You can create a windowed store in the processor API via the Stores factory class: org.apache.kafka.streams.state.Stores More specifically, you you do sth. like: Stores.create().withKeys().withValues().persistent().windowed(/* you can specify window size, retention period etc here */)

RE: A question about kafka

Hi, Anybody could help to answer below question? If compression type could be modified through command " bin/kafka-console-producer.sh --producer.config "? Thanks! Best Regards Johnny -Original Message- From: ZHU Hua B Sent: 2016年10月17日 14:52 To: users@kafka.apache.org; Radosl

RE: Mirror multi-embedded consumer's configuration

Hi, It is OK now, thanks for your help! Best Regards Johnny -Original Message- From: tao xiao [mailto:xiaotao...@gmail.com] Sent: 2016年10月24日 17:43 To: users@kafka.apache.org Subject: Re: Mirror multi-embedded consumer's configuration auto.offset.reset=smallest takes effect onl

programmatic way to check for topic existence?

Hiya! I've been trying to merge https://github.com/zendesk/maxwell/pull/457, which adds a much-requested feature of Maxwell, that of being able to have a topic-per-mysql-table. When we receive a row we programmatically generate the topic name, and the first thing we do is call `KafkaProducer#part

Re: Consumer error : This consumer has already been closed

Hello, Could someone reply me if I made a mistake here or not. Did I make a rookie mistake here ? Is this not the correct forum here to put my question ? It's a bit strange to have this error : Exception in thread "main" java.lang.IllegalStateException: This consumer has already been closed On the

Re: Kafka Streaming

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 It's a client issues... But CP 3.1 should be our in about 2 weeks... Of course, you can use Kafka 0.10.1.0 for now. It was released last week and does contain the fix. - -Matthias On 10/24/16 9:19 AM, Mohit Anchlia wrote: > Would this be an issue i

Re: Kafka Streaming

Would this be an issue if I connect to a remote Kafka instance running on the Linux box? Or is this a client issue. What's rockdb used for to keep state? On Mon, Oct 24, 2016 at 12:08 AM, Matthias J. Sax wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > Kafka 0.10.1.0 which was rele

Re: Kafka unable to find topic details after sudden EC2 server restart

You probably didn't setup permissions properly on the /data directory you created. You might want to do some free online training to learn some more Linux if you are planning to become a Kafka administrator. https://en.wikipedia.org/wiki/Chmod https://www.edx.org/course/introduction-linux-linuxfoun

Re: Removing kafka topic contents from Java

The Kafka Java Admin client is actively being worked on now as part of KIP-4 see https://issues.apache.org/jira/browse/KAFKA-1694 specifically sub-task #9 Create Java Admin Client The Server side and wire protocol for admin commands like create/delete/modify topic are already in Kafka 0.10.1 -h

Re: Removing kafka topic contents from Java

Are you looking something like this String command = String.format("--delete --zookeeper %s --topic %s", "localhost:2181", "test_topic"); String[] split = command.split(" "); TopicCommand.main(split); t SunilKalva On Mon, Oct 24, 2016 at 9:09 PM, Ali Akhtar wrote: > There isn't a java API for

Re: Removing kafka topic contents from Java

There isn't a java API for this, you'd have to mess around with bash scripts which I haven't found to be worth it. Just let the data expire and get deleted. Set a short expiry time for the topic if necessary. On Mon, Oct 24, 2016 at 6:30 PM, Demian Calcaprina wrote: > Hi Guys, > > Is there a w

RE: Removing kafka topic contents from Java

There is a script kafka-topics.sh which calls kafka.admin.TopicCommand. I think you could probably call that same class from your application if you figure out which jars are needed. When I do testing I'll clear out queues by setting the retention time to a second and then wait for kafka to r

Removing kafka topic contents from Java

Hi Guys, Is there a way to remove a kafka topic from the java api? I have the following scenario: We have a subscription service, which returns responses to subscribed clients through kafka topic. Per subscription, we create a new kafka topic and send the data by publishing in this new topic. And

RE: Mirror multi-embedded consumer's configuration

Yes. //h...@confluent.io Original message From: ZHU Hua B Date: 10/24/16 12:09 AM (GMT-08:00) To: users@kafka.apache.org Subject: RE: Mirror multi-embedded consumer's configuration Hi, Many thanks for your confirm! I have another question, if I deleted a mirrored topic on

Removing kafka topic contents from Java

Hi Guys, Is there a way to remove a kafka topic from the java api? I have the following scenario: We have a subscription service, which returns responses to subscribed clients through kafka topic. Per subscription, we create a new kafka topic and send the data by publishing in this new topic. And

Re: Mirror multi-embedded consumer's configuration

auto.offset.reset=smallest takes effect only when there is no offset committed prior or offset being requested is out of range. I believe your mirror maker has committed offset therefore auto.offset.reset=smallest wont take effect. You can create a new consumer group and try again On Mon, 24 Oct

Re: Kafka unable to find topic details after sudden EC2 server restart

No issues, I have solved the problem by changing the permission of /data Thanks, Gourab On Mon, Oct 24, 2016 at 2:20 PM, Gourab Chowdhury wrote: > Thanks for the reply, I tried changing the data directory as follows:- > dataDir=/data/zookeeper > > I also create a /data but still I am getting th

RE: Mirror multi-embedded consumer's configuration

Hi, Thanks for your info! I set auto.offset.reset=smallest in consumer.properties on target cluster and retest, it still failed. Seems the topic with original messages could not mirror to target cluster while running mirror maker first time, unless produce a new message to this topic to trigg

RE: customised event time

And the start time and end time of the window. In other words i need the notion of windows in the proecessor API. Is this possible ? De : Hamza HACHANI Envoyé : dimanche 23 octobre 2016 20:43:05 À : users@kafka.apache.org Objet : RE: customised event time To be

Re: Mirror multi-embedded consumer's configuration

You need to set auto.offset.reset=smallest to mirror data from beginning On Mon, 24 Oct 2016 at 17:07 ZHU Hua B wrote: > Hi, > > > Thanks for your info! > > > Before I launch mirror maker first time, there is a topic include some > messages, which have been produced and consumed on source Kafka

RE: Mirror multi-embedded consumer's configuration

Hi, Thanks for your info! Before I launch mirror maker first time, there is a topic include some messages, which have been produced and consumed on source Kafka cluster. Then I launch mirror maker first time on target cluster, but this topic did not mirror from source cluster. So I keep mir

Re: Mirror multi-embedded consumer's configuration

Yes, it should mirror again. To mirror the topic from beginning, you may have to start with different consumer group, On Mon, Oct 24, 2016 at 12:39 PM, ZHU Hua B wrote: > Hi, > > > Many thanks for your confirm! > > I have another question, if I deleted a mirrored topic on the target Kafka > clus

Re: Kafka unable to find topic details after sudden EC2 server restart

Thanks for the reply, I tried changing the data directory as follows:- dataDir=/data/zookeeper I also create a /data but still I am getting the following error. ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server. ZooKeeperServerMain) java.io.IOException: Unable to create

RE: customised event time

To be more specific. What id do really need is the property of the retention time dor the window in the processor API. Because for the window i think that i can manage to do this. Hamza De : Hamza HACHANI Envoyé : dimanche 23 octobre 2016 20:30:13 À : users@

RE: customised event time

Hi, I think that maybe i'm asking much. But Ineed the aspect of windowing in the processor API not in the Stram DSL. Is this possible? The second question is how can i get rid of the intermediate results because i'm only interested in the final result given by the window. Hamza _

RE: Mirror multi-embedded consumer's configuration

Hi, Many thanks for your confirm! I have another question, if I deleted a mirrored topic on the target Kafka cluster, if Kafka mirror maker could mirror the same topic again from source cluster when I launch mirror maker next time? Thanks! Best Regards Johnny -Original Message

Re: Kafka Streaming

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Kafka 0.10.1.0 which was release last week does contain the fix already. The fix will be in CP 3.1 coming up soon! (sorry that I did mix up versions in a previous email) - -Matthias On 10/23/16 12:10 PM, Mohit Anchlia wrote: > So if I get it right

Re: Mirror multi-embedded consumer's configuration

I think we need to update the docs. This behavior got changed longtime ago. we need to pass only one consumer configuration. This config will be applicable for all the consumer threads/instances. On Mon, Oct 24, 2016 at 11:43 AM, ZHU Hua B wrote: > Hi, > > > Because from some Kafka wiki I saw "A