Re: How to create a topic using the Java API / Client?

I'm guessing its not possible to delete topics? On Thu, Sep 15, 2016 at 5:43 AM, Ali Akhtar wrote: > Thank you Martin > > On 15 Sep 2016 3:05 am, "Mathieu Fenniak" > wrote: > >> Hey Ali, >> >> If you have auto create turned on, which it

Kafka Source Connector - JSON with Escape double quotes

Hi, I am using Kafka connect to feed data into kafka, please find sample code snippet from Source Task. My question is , why kafka connect dematerialize JSON message by escaping double quotes? Is there way to get the same without escaping? byte[] message = null; Charset charset =

Re: what's the relationship between Zookeeper and Kafka ?

In addition to what Michael noted, this question has been asked a few times before too and here's one such previous discussion https://www.quora.com/What-is-the-actual-role-of-ZooKeeper-in-Kafka -Jaikiran On Wednesday 14 September 2016 03:50 AM, Michael Noll wrote: Eric, the latest versions

Re: kafkaproducer send blocks until broker is available

This is a known issue and is being tracked in this JIRA https://issues.apache.org/jira/browse/KAFKA-3539 -Jaikiran On Saturday 10 September 2016 12:20 AM, Peter Sinoros Szabo wrote: Hi, I'd like to use the Java Kafka producer in a non-blocking async mode. My assuptions were that until the new

Re: Too many open files

What does the output of: lsof -p show on that specific node? -Jaikiran On Monday 12 September 2016 10:03 PM, Michael Sparr wrote: 5-node Kafka cluster, bare metal, Ubuntu 14.04.x LTS with 64GB RAM, 8-core, 960GB SSD boxes and a single node in cluster is filling logs with the following:

Re: Too many open files

What does the output of: lsof -p show? -Jaikiran On Monday 12 September 2016 10:03 PM, Michael Sparr wrote: 5-node Kafka cluster, bare metal, Ubuntu 14.04.x LTS with 64GB RAM, 8-core, 960GB SSD boxes and a single node in cluster is filling logs with the following: [2016-09-12

Re: Exception while deserializing in kafka streams

Guozhang, I am using 0.10.0.0. Could the below log be the cause? 16/09/14 17:24:35 WARN ConsumerConfig: The configuration schema.registry.url = http://192.168.50.6: 8081 was supplied but isn't a known config. 16/09/14 17:24:35 INFO AppInfoParser: Kafka version :

Anyone running Kafka on Kubernetes in production?

If so, can you please share if you're using a publicly available deployment, or if you created you own, how you did it? (I.e which services / replication controllers you have) Also, how has the performance been for you? I've read a report which said the performance suffered running kafka as a

Re: How to create a topic using the Java API / Client?

Thank you Martin On 15 Sep 2016 3:05 am, "Mathieu Fenniak" wrote: > Hey Ali, > > If you have auto create turned on, which it sounds like you do, and you're > happy with using the broker's configured partition count and replication > factor, then you can call

Consumer stops after reaching an offset of 1644

Hi All, I am trying to do a simple benchmark test  for Kafka using single broker, producer and consumer however my consumer doesn't seem to receive all the messages produced by the producer so not sure what is going on any help? Here is the full description of the problem.

Re: Exception while deserializing in kafka streams

Hello Walter, Which version of Kafka were you using? I ask this because there was a bug causing the serde passed through config to NOT being configured when constructed: https://issues.apache.org/jira/browse/KAFKA-3639 Which is fixed in the 0.10.0.0 release, which means you will only hit it if

Kafka consumer group problem

Hi, I created a few consumers that belong to the same group_id, but I noticed that each consumer get all messages instead of only some of the messages. As for the topic, I did create the topic with a few partitions. Anyone else had the same problem? Is there any configuration parameter I need

Re: How to create a topic using the Java API / Client?

Hey Ali, If you have auto create turned on, which it sounds like you do, and you're happy with using the broker's configured partition count and replication factor, then you can call "partitionsFor(String topic)" on your producer. This will create the topic without sending a message to it. I'm

Re: How to create a topic using the Java API / Client?

You can make following setting to set partition count, etc auto.create.topics.enable = true(enable auto creation of topic on the server) num.partitions = 1 (number of log partitions per topic) On 9/14/16, 2:41 PM, "Ali Akhtar" wrote:

Re: How to create a topic using the Java API / Client?

It looks like if I just send a message to a non-existent topic, it is created. But this has the downside that the first message of the topic is null or otherwise invalid. Also the partition count can't be specified. Is there a way to create a topic without needing to post a null message? Do I

Re: MockClientSupplier

Hello Andy, You can just call KafkaStreams.toString(). I'm copying its java doc string here: "Produces a string representation contain useful information about Kafka Streams * Such as thread IDs, task IDs and a representation of the topology. This is useful * in debugging scenarios."

Re: Unexpected KStream-KStream join behavior with asymmetric time window

Hello Elias, Thanks for reporting! I will follow-up with you on the created JIRA ticket. Guozhang On Mon, Sep 12, 2016 at 4:01 PM, Elias Levy wrote: > https://issues.apache.org/jira/browse/KAFKA-4153 > https://github.com/apache/kafka/pull/1846 > > On Mon, Sep 12,

Exception while deserializing in kafka streams

Hello, I get the below exception when deserilaizing avro records using KafkaAvroDeserializer. 16/09/14 17:24:39 INFO StreamThread: Stream thread shutdown complete [StreamThread-1] Exception in thread "StreamThread-1" org.apache.kafka.common.errors.SerializationException: Error deserializing Avro

Re: KAFKA-3933: Kafka OOM During Log Recovery Due to Leaked Native Memory

Here are the links: https://github.com/apache/kafka/commit/8a417c89d2f0b7861b2dec26f02e4e302b64b604 (trunk) https://github.com/apache/kafka/commit/c47c3b0b583a849fdf3ed0a06835427a2801950a (0.10.0) Ismael On Wed, Sep 14, 2016 at 4:38 PM, feifei hsu wrote: > Hi, Ismael >

Re: KAFKA-3933: Kafka OOM During Log Recovery Due to Leaked Native Memory

Hi, Ismael So many thanks for the quick reply I checked the trunk tree at github, I did not see the merge. did I make some mistake? sorry for that. for example, the pull 1598. file related to the LogSegment.scala. the one of the PR add try catch to close the leaking resource. but I did

Re: ZooKeeper client > 3.4.6

We upgraded to 3.4.8 for this exact reason and have not had any issues. On 09/14/2016 09:45 AM, Bill de hÓra wrote: Hi, I was wondering if anyone has experience (good or bad!) running Kafka with a ZooKeeper client > 3.4.6. The reason to do this is to avoid having to bounce brokers when

ZooKeeper client > 3.4.6

Hi, I was wondering if anyone has experience (good or bad!) running Kafka with a ZooKeeper client > 3.4.6. The reason to do this is to avoid having to bounce brokers when ZooKeeper node IPs change; it seems the 3.4.6 client caches IP addresses indefinitely but this may have be fixed in

RE: handling generics(solved) but new problem unearthed

the end goal was to power thru the kafka parsing/compilation to then see if all kafka java classes would actually compile I found this bug where kafka is attempting implement joptsimple kafka.tools.StreamsResetter.java fails the build

Re: KAFKA-3933: Kafka OOM During Log Recovery Due to Leaked Native Memory

Hi, We did merge the PR to trunk and 0.10.0. Ismael On Wed, Sep 14, 2016 at 9:21 AM, feifei hsu wrote: > Hi Tom and Ismael. >I am following the kafka-3933. the memory leak. but I did not see the > pr #1598 #1614 #1660 are merged into the trunk. > Do you know what the

KAFKA-3933: Kafka OOM During Log Recovery Due to Leaked Native Memory

Hi Tom and Ismael. I am following the kafka-3933. the memory leak. but I did not see the pr #1598 #1614 #1660 are merged into the trunk. Do you know what the current status? So many thanks. We are also thinking backport it to 0.9.0.1 --easy

Re: Question regarding functionality of MirrorMaker

Hello cs, Apologies for delayed response. I found one topic in my Kafka env which has no leaders and no replicas. That was pretty weird and I am not sure what caused this. Because of this topic MirrorMaker was hanging and printing messages like "No leader found for topic ..". Due to this hang MM