Kafka - Roadmap - 2017

Hello, I was wondering if anyone from kafka (users/dev or commits) can send or direct me to the kafka software roadmap for 2017 and onwards. Let me know if I need to subscribe or if this is available somwhere on your website. Greatly appreciated! Costa Tsirbas 514.443.1439

Re: Kafka - Roadmap - 2017

Our community does not have a roadmap as such. But there are few initiatives that are currently being worked on and are likely to be included in 2017 releases: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals If members of the community will have additional

Re: KafkaStream: puncutuate() never called even when data is received by process()

I know that the Kafka team is working on a new way to reason about time. My team's solution was to not use punctuate...but this only works if you have guarantees that all of the tasks will receive messages..if not all the partitions. Another solution is to periodically send canaries to all

Re: no luck with kafka-connect on secure cluster

Koert, I think what you're seeing is that there are actually 3 different ways Connect can interact with Kafka. For both standalone and distributed mode, you have producers and consumers that are part of the source and sink connector implementations, respectively. Security for these are configured

Re: Convert log file in Avro json encoding to Avro binary encoding and send into Kafka

Avro JSON encoding is a wire-level format. The AvroConverter accepts Java runtime data (e.g. primitive types like Strings & Integers, Maps, Arrays, and Connect Structs). The component that most closely matches your needs is Confluent's REST proxy, which supports the Avro JSON encoding when

Re: no luck with kafka-connect on secure cluster

ah okay that makes sense. also explains why for a distributed source i actually has to set it twice: security.protocol=SASL_PLAINTEXT producer.security.protocol=SASL_PLAINTEXT if anyone runs into this issue and just wants it to work... this is what is in my configs now:

Re: Interactive Queries

I would start here: http://docs.confluent.io/3.1.0/streams/index.html On 11/26/16, 8:27 PM, "Alan Kash" wrote: Hi, New to Kafka land. I am looking into Interactive queries feature, which transforms Topics into Tables with history, neat ! 1.

sharing storage topics between different distributed connect clusters

if i were to run multiple distributed connect clusters (so with different group.id) does each connect cluster need its own offset.storage.topic, config.storage.topic and status.storage.topic? or can they safely be shared between the clusters? thanks! koert

Re: sharing storage topics between different distributed connect clusters

If you want independent clusters based on the same Kafka cluster, they need independent values for config/offset/status topics. The Kafka Connect framework doesn't provide its own sort of namespacing or anything, so if you used the same topics in the same cluster, the values between different

Strange issue in Kafka Windows where Log directory getting increased day by day

Dear * I am facing some issue in Kafka Windows, the Log directories size are keep on increasing day by day.. Currently the traffic on my kafka set up is minimal , but if this log directory size does not get fixed then its going to be serious issue... My Kafka SetUp - 2 Windows Server -

Spring Kafka is not working with Spring 3.0.5.RELEASE

Hi users, My project is already developed with Spring 3.0.5.RELEASE, We are planning to use Kafka for new requirements. I am trying to use spring-kafka (from 1.0.x to 1.1.x) but it is not supporting with Spring 3.0.5.RELEASE. Application is throwing "class not found exception

Re: Error "Unknown magic byte" occurred while deserializing Avro message

Are you sure you have not produced any other data into that topic, e.g. perhaps you were testing the regular kafka-console-producer before? This would cause it to fail on the non-Avro messages (as Dayong says, because the initial magic byte mismatches). Can you try starting the consumer first

Re: Kafka Consumer not consuming new messages randomly

The REST proxy cannot guarantee that if there are messages in Kafka it will definitely return them. There will always be some latency between the request to the REST proxy and fetching data from Kafka, and because of the way the Kafka protocol works this could be delayed by the fetch timeout. The

Re: Custom Partitioner explanation?

Yes, that's correct. For reference you can just take a look at the DefaultPartitioner which does nearly same (with additional logic to do round robin when there isn't a key): https://github.com/ apache/kafka/blob/trunk/clients/src/main/java/org/

Re: Minor documentation error

I think you're seeing one of the confusing name changes between old and new consumers. A quick grep suggests that you are correct that the parameter for the old consumer is fetch.wait.max.ms, but the parameter for the new consumer is fetch.max.wait.ms. Since the link you gave is for the new

Re: Kafka Mirror viable hardware

Kevin, Generally you're right that mirroring, whether with MirrorMaker or Confluent's Replicator, shouldn't be too expensive wrt CPU. However, do be aware that in both cases, if you are using compression you'll need to decompress and recompress due to the way they copy data. This could possibly

Re: Balancing based on lag

Jens, Sorry, I'm very late to this thread but figure it might be worth following up since I think this is a cool feature of the new consumer but isn't well known. You actually have *quite* a bit of flexibility in controlling how partition assignment happens. The key hook is the

Re: Reg: DefaultParititioner in Kafka

When a key is available, you generally include it because you want all messages with the same key to always end up in the same partition. This allows all messages with the same key to be processed by the same consumer (e.g. allowing you to aggregate all data for a single user if you key on user

Interactive Queries

Hi, New to Kafka land. I am looking into Interactive queries feature, which transforms Topics into Tables with history, neat ! 1. What kind of queries we can run on the store ? Point or Range ? 2. Is Indexing supported ? primary or seconday ? 3. Query language - SQL ? Custom Java Native Query