Re: kafka compacted topic

I want to confirm if kafka has to re-compact all log segments, as log grows doesn't it become slower as well? On Tue, Nov 28, 2017 at 11:33 PM, Jakub Scholz wrote: > There is quite a nice section on this in the documentation - > http://kafka.apache.org/documentation/#compaction

Re: Lost messages and messed up offsets

Do you commit the received messages? Either by doing it manually or setting enable.auto.commit and auto.commit.interval.ms? On Wed, Nov 29, 2017 at 11:15 PM, Tom van den Berge < tom.vandenbe...@gmail.com> wrote: > I'm using Kafka 0.10.0. > > I'm reading messages from a single topic (20

RE: [EXTERNAL] - Lost messages and messed up offsets

Hi; With default configuration, your consumers are set with auto.offset.reset=latest So on restart, the consumers start to read the offset of 0 minutes ago, not the offset of 30 minutes ago (or whatever the lag was). https://kafka.apache.org/documentation/#configuration auto.offset.reset What

Lost messages and messed up offsets

I'm using Kafka 0.10.0. I'm reading messages from a single topic (20 partitions), using 4 consumers (one group), using a standard java consumer with default configuration, except for the key and value deserializer, and a group id; no other settings. We've been experiencing a serious problem a

Re: My kafka docker image

El mié., 29 nov. 2017 9:09 PM, Christian F. Gonzalez Di Antonio < christian...@gmail.com> escribió: > uhh, so sorry, I forgot it. > > Dockek Hub: https://hub.docker.com/r/christiangda/kafka/ > > Github: https://github.com/christiangda/docker-kafka > > Regards, > > Christian > > > > El mié., 29

Re: Plans to extend streams?

Hello Adrienne, I think your suggested feature to to use not only Kafka as inter-process communication but also configurable to use TCP directly, right? There are a few people asking about this before, especially for not using Kafka for repartitioning (think: shuffling in the batch world), but

Re: Plans to extend streams?

We had some discussion if we can/should replace re-partitioning topic via a direct network connection between instances. It's a tricky problem though with many string attached... Thus, it comes with pros and cons and it's still unclear what the exact trade-off is. Thus, it might happen, but it's

My kafka docker image

Hi everyone, I would like to share my recently kafka's docker image with you. I made this docker image because I try to get the major focus of kafka configuration in my development environment and also because any of the others images available was not easy to understand (how to use) outside

Re: Too many open files in kafka 0.9

There is KAFKA-3317 which is still open. Have you seen this ? http://search-hadoop.com/m/Kafka/uyzND1KvOlt1p5UcE?subj=Re+Brokers+is+down+by+java+io+IOException+Too+many+open+files+ On Wed, Nov 29, 2017 at 8:55 AM, REYMOND Jean-max (BPCE-IT - SYNCHRONE TECHNOLOGIES)

Kafka 1.0.0 - Creating a topic and waiting for it to be ready

Hello everyone. We are currently using AdminUtils.createTopic to create topics at runtime and we were wondering if it's possible to implement some check to validate that the topic is "ready" for producer/consumer wiring. Thank you. Cos.

RE: Kafka & Canary Release

Hi, It doesn’t so simple only micro service 1 knows about it - if micro service 2 send to micro service 1, it doesn’t have any idea that micro service 1 was updated. So we need to route it on message hub - Kafka. Thanks, Assaf From: Wim Van Leuven [mailto:wim.vanleu...@highestpoint.biz] Sent:

Too many open files in kafka 0.9

We have a cluster with 3 brokers and kafka 0.9.0.1. One week ago, we decide to adjust log.retention.hours from 10 days to 2 days. Stop and go the cluster and it is ok. But for one broker, we have every day more and more datas and two days later crash with message too many open files. lsof

Re: Kafka & Canary Release

Isn't that simply a matter of changing your key to route the element to the right topic for 1 or the other microservice? -w On Wed, 29 Nov 2017 at 17:09 Yuval Alon wrote: > Hello, > We are using Apache Kafka 0.10.2, which we understand is not supporting > “Canary

Kafka Topic Exists Exception - On upgrading to 1.0

Hi I am trying to upgrade a single node kafka broker to latest 1.0 from 0.10. The steps followed 1. Stopped Kafka Broker 2. Replaced bin/, libs & site-docs/ with the latest 3. Started Kafka But I am seeing the below exception in logs - Anyone any thoughts? How can I get around this?

Re: Plans to extend streams?

Hi, You misunderstood the focus of the post perhaps or I could not explain properly. I am not claiming the streams is limited to single node. Although the whole topology instance can be limited to a single node (each node run all topology), this is sth else. Also, I think that "moving 100s of GB

Kafka & Canary Release

Hello, We are using Apache Kafka 0.10.2, which we understand is not supporting “Canary Release” Is it in your pipeline, if so, when it’s going to be released? Explanation of the issue: We hold two different versions of micro services that

Re: Plans to extend streams?

Hey, you making some wrong assumptions here. Kafka Streams is in no way single threaded or limited to one physical instance. Having connectivity issues to your brokers is IMO a problem with the deployment and not at all with how kafka streams is designed and works. Kafka Streams moves hundreds

Re: Plans to extend streams?

Not necessarily. I would avoid the term "reimplemented". Btw, Apache Storm is not also the best (streaming) system that can utilize the network and it does not support runtime scale in/out (at least by design). So, can streams preserve its current selling points (ex:dynamicity) while introducing

Re: Plans to extend streams?

What you are actually asking is if Kafka Streams should be reimplemented as Apache Storm? -wim On Wed, 29 Nov 2017 at 15:10 Adrienne Kole wrote: > Hi, > > The purpose of this email is to get overall intuition for the future plans > of streams library. > > The main

Plans to extend streams?

Hi, The purpose of this email is to get overall intuition for the future plans of streams library. The main question is that, will it be a single threaded application in the long run and serve microservices use-cases, or are there any plans to extend it to multi-node execution framework with

Re: Kafka streams - regarding WordCountInteractiveQueriesExample

Giridhar, Thanks for reporting this, I'll take a look. On Wed, Nov 29, 2017 at 5:37 AM, Giridhar Addepalli wrote: > Hi, > > I am newbie to Kafka streams. > > Tried below example : > > https://github.com/confluentinc/kafka-streams- >

Kafka beginner: problem with interpreting logs, troubleshooting running in docker swarm

Hi, I am running one kafka broker, against a 3 node zookeeper cluster, all in AWS. Using Kafka Tool on it, which works flawlessly against my local Kafka (in docker) Kafka Tool can connect, and comes up with information, but I can't create topics, even though it reports success. On the other

Re: KAFKA-5413 and 0.10.x

Just a quick note. On Tue, Nov 28, 2017 at 8:09 PM, Philippe Laflamme wrote: > Upgrading our cluster to a 0.11 version is certainly an option, but is a > risky one given that it could introduce move bugs (especially since it has > a new storage format and a whole slew of

Kafka streams - regarding WordCountInteractiveQueriesExample

Hi, I am newbie to Kafka streams. Tried below example : https://github.com/confluentinc/kafka-streams-examples/blob/4.0.x/src/main/java/io/confluent/examples/streams/interactivequeries/WordCountInteractiveQueriesExample.java http://localhost:7070/state/instances [ { "host": "localhost",