How start with Kafka?

I am Kafka beginner. I have compiled under Windows and Visual Studio 2015 librdkafka and cppkafka . I have two examples: kafka_consumer.cpp and kafka_producer.cpp. I try: >producer -b 127.0.0.1:9092 -t topic >consumer -b 127.0.0.1:9092 -t topic -g group but is bad connection

Newbie how to get key/value pojo out of a stream?

kafka. Or maybe I want to plot the data,… I looked at the java doc and did not find anything Any idea what I should “google” to to find a code example? Kind regards Andy

Re: Best Way to Check Producer's Connection to Kafka

On Apr 15, 2015, at 12:09 AM, Alex Melville wrote: > > Hi All, > > What is the lightest way to check if the connection to Kafka is up or not? > Currently I'm having the producer attempt to send a message to a topic, and > then catching a TimeoutException and handling logic inside of the catch >

Offset Storage

"can we achieve scalable offset management just by setting "offset.storage=kafka", and calling "consumer.commitOffsets" after processing each message?" Or would we need to adapt the code in [1] to our particular needs. Cheers, Andy [1]: https://cwiki.apache.org/con

Re: One more Kafka Meetup hosted by LinkedIn in 2016 (this time in San Francisco) - does anyone want to talk?

Wow you guys plan a long time ahead ;-) On Fri, Oct 30, 2015 at 12:49 PM, Ed Yakabosky wrote: > Hi all, > > LinkedIn is hoping to host one more Apache Kafka meetup this year on > November 18 in our San Francisco office. We're working on building the > agenda now. Does anyone want to talk? Ple

newbie: kafka 0.9.0.0 producer does not terminate after producer.close()

I wrote a little test client that reads from a file an publishes using the 0.9.0.0 API. I am contacting to an older 0.8.x sever. I am able to send messages how ever I noticed that once I am done reading the input file my test program hangs Any idea what I am doing wrong? Kind regards Andy

Re: newbie: kafka 0.9.0.0 producer does not terminate after producer.close()

sever. I am able to consume my test messages using kafka_2.10-0.8.2.1/bin/kafka-console-consumer.sh Kind regards Andy ERROR 17:12:14 kafka-producer-network-thread | producer-1 o.a.k.c.p.i.Sender run line:130 Uncaught error in kafka producer I/O thread

Re: newbie: kafka 0.9.0.0 producer does not terminate after producer.close()

Thanks Andy From: Kamal C Reply-To: Date: Sunday, May 22, 2016 at 8:36 AM To: Subject: Re: newbie: kafka 0.9.0.0 producer does not terminate after producer.close() > Andy, > > Kafka 0.9.0 server supports the previous versions of the clients (0.8.2, > 0.8.1..). > But, ne

FW: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Does anyone know if spark plans to upgrade? I think the current version is 0.8x? Kind regards Andy From: Gwen Shapira Reply-To: Date: Tuesday, May 24, 2016 at 9:24 AM To: , , Subject: [ANNOUCE] Apache Kafka 0.10.0.0 Released > The Apache Kafka community is pleased to announce

newbie: trouble configuring topic default retention (i.e. time to live)

? Thanks in advance Andy

performance problem when reading lots of small files created by spark streaming.

then concatenating the files into a few much large files, then using Œhadoop fs put¹ to move them to hdfs. Do you think this would improve the spark count() performance issue? Does anyone know of heuristics for determining the number or size of the concatenated files? Thanks in advance Andy

Re: performance problem when reading lots of small files created by spark streaming.

to a normal file system on one of > my workers and then concatenating the files into a few much large files, then > using Œhadoop fs put¹ to move them to hdfs. Do you think this would improve > the spark count() performance issue? > > Does anyone know of heuristics for determining the number or size of the > concatenated files? > > Thanks in advance > > Andy >

newbie does python del, gc.collect() release all resources?

call unpersist() or will del,gc.collect() do this for me? Thanks Andy

Partitioning at the edges

Hey Folks, We are having quite a bit trouble modelling the flow of data through a very kafka centric system As I understand it, every stream you might want to join with another must be partitioned the same way. But often streams at the edges of a system *cannot* be partitioned the same way becaus

Re: Partitioning at the edges

at re-partitions it's input as necessary for scale. Cheers, Andy On Sat, Sep 3, 2016 at 6:13 AM, Eno Thereska wrote: > Hi Andy, > > Could you share a bit more info or pseudocode so that we can understand > the scenario a bit better? Especially around the streams at the edges.

Re: Partitioning at the edges

an actually do this with one job): > > T1 -> Job_T1 -> T1’ > T2 -> Job_T2 -> T2’ > > T1’ and T2’ would be partitioned on your join key and would have the > same number of partitions so that you have the guarantees you need to do > the join. (i.e. join

MockClientSupplier

0.0.1"] [org.apache.kafka/kafka-clients "0.10.0.1"] [org.apache.kafka/kafka_2.11 "0.10.0.1"] but none seem to include the class org.apache.kafka.test.MockClientSupplier Cheers, Andy

Re: MockClientSupplier

check they are all glued together correctly. Cheers, Andy On Sun, Sep 11, 2016 at 8:56 PM, Guozhang Wang wrote: > Hello Andy, > > Unfortunately the included package `o.a.k.test` is not included in the > released maven. > > There are some discussions about moving these test

Re: programmatic way to check for topic existence?

You could just catch the exception but if this is per row, that is probably prohibitively expensive. Doesn't the binlog get "create table" events? Wouldn't that be a better time to create the topic? -- Andy On Mon, Oct 24, 2016 at 2:32 PM, Ben Osheroff wrote: > Hiya!

Re: KafkaAvroSerializer to produce to a single topic with different schemas used for records

ned in the union. This would allow you to retain the compatibility features of the schema registry. Cheers, Andy On Mon, Jan 30, 2017 at 1:48 PM, Mike Cargal wrote: > This helps some. W’re planning to write a non-homogeneous set of records > to a single topic (to preserve order). T

Missing Response Formats in Kafka Protocol Guide

itions Response* — Not listed at all. - *Fetch Response (Version: 17)* — Only versions 4–16 are documented, though version 17 of the request is included. Just wanted to flag in case this was unintentional. Thanks for all the great work on Kafka! Best, Andy Li