Compression in Kafka

Hi Kafka users, I am trying to understand the behavior of compression in Kafka. Consider a scenario, where producer sets compression.codec "snappy" and broker's compression.code "lz4"? In this scenario, what is the behavior of the compression? As far as I have understood is the following, The

Re: error when attempting a unit test of spring kafka producer

LoginType was in 0.10.x release. This seems to indicate Kafka version mismatch. Can you check the dependencies of your test ? Thanks On Tue, Feb 13, 2018 at 8:03 PM, Ian Ewing wrote: > I have been trying to figure out how to unit test a kafka producer. Should > take in a

error when attempting a unit test of spring kafka producer

I have been trying to figure out how to unit test a kafka producer. Should take in a simple integer and perform some addition. Followed what I could find on spring kafka unit testing but keep running into this error: 19:53:12.788 [main] ERROR kafka.server.KafkaServer - [Kafka Server 0], Fatal

questions about replication factor of __consumer_offsets topic

We noticed that the replication factor is six for *__consumer_offsets* and as we have been moving consumers over from zk to kf offsets latency for FetchFollower on the group coordinator for a busy consumer_group increases significantly (in hundreds of ms), but offset commit latency and cpu

RE: Kafka Stream tuning.

I have also check the commit-latency-avg, it's around 23 millis per commit. That translate to about the same throughput that I'm getting now (0.04message/millis). Does anyone got any benchmark for kafka stream's commit-latency-avg? Is it possible to tune it to be faster? I just want to

RE: Kafka Stream tuning.

Hey Damian and folks, I've also tried 1000 and 500 and the performance state is exactly the same. Any other ideas? Regards, Brilly -Original Message- From: Damian Guy [mailto:damian@gmail.com] Sent: Tuesday, February 13, 2018 4:48 PM To: users@kafka.apache.org Subject: Re: Kafka

Re: Kafka consumer commit behaviour on rebalance

Yes. If a consumer when down, all the polled messages that were not committed will be redelivered to another consumer. regards On Tue, Feb 13, 2018 at 9:31 AM, pradeep s wrote: > Hi All, > I am running a Kafka consumer(Single threaded) on kubernetes . Application

Re: [VOTE] 1.0.1 RC1

+1 Ran tests, verified streams quickstart works On Tue, 13 Feb 2018 at 17:52 Damian Guy wrote: > Thanks Ewen - i had the staging repo set up as profile that i forgot to > add to my maven command. All good. > > On Tue, 13 Feb 2018 at 17:41 Ewen Cheslack-Postava

Re: [VOTE] 1.0.1 RC1

Thanks Ewen - i had the staging repo set up as profile that i forgot to add to my maven command. All good. On Tue, 13 Feb 2018 at 17:41 Ewen Cheslack-Postava wrote: > Damian, > > Which quickstart are you referring to? The streams quickstart only executes > pre-built stuff

Re: [VOTE] 1.0.1 RC1

Damian, Which quickstart are you referring to? The streams quickstart only executes pre-built stuff afaict. In any case, if you're building a maven streams project, did you modify it to point to the staging repository at https://repository.apache.org/content/groups/staging/ in addition to the

Kafka consumer commit behaviour on rebalance

Hi All, I am running a Kafka consumer(Single threaded) on kubernetes . Application is polling the records and accummulating in memory . There is a scheduled write of these records to S3 . Only after that i am committing the offsets back to Kafka. There are 4 partitions and 4 consumers(4 kubernetes

The Problem about Kafka 1.0

Hi Kafka guys, I faced the following Exception when I sent message to Kafka 1.0 - 2018-02-13 04:45:49,861 INFO [STDOUT] org.apache.kafka.common.KafkaException: Cannot perform send because at least one previous transactional or idempotent request has failed with errors. 2018-02-13 04:45:49,861

Re: one topic, one partition, many consumer instances

So...if you have a consumer running on A, and then you start another one on B, I would think it would trigger a rebalance. I am not 100% sure whether A would just keep being the active consumer or if it would switch to B. That being said, yes, I think the general idea is correct. In terms of

Re: one topic, one partition, many consumer instances

On Tue, Feb 13, 2018 at 2:59 PM, Subhash Sriram wrote: Hey Xavier, > > Within a consumer group, you can only have as many consumers as you have > partitions in a topic. If you start more consumers than partitions within > the same group, they will just be idle. >

Re: why kafka index file use memory mapped files ,however log file doesn't

1. Perhaps a human-readable log, being write-only, and which may buffer on the user side or in the kernel, may be more efficient because small writes are accumulated in a buffer (cheap) actually pushed to disk (less cheap)? If you mmap'd this instead, how do you feel it would behave? 2. Did you

Re: question on serialization ..

ah .. ok .. thanks for the clarification .. for reduce I guess the overload with Materialized does the same thing .. regards. On Tue, Feb 13, 2018 at 2:24 PM, Damian Guy wrote: > There is an overload `leftJoin(KTable, ValuJoiner, Joined)` > > Joined is where you specify

Re: one topic, one partition, many consumer instances

Hey Xavier, Within a consumer group, you can only have as many consumers as you have partitions in a topic. If you start more consumers than partitions within the same group, they will just be idle. Thanks, Subhash Sent from my iPhone > On Feb 13, 2018, at 8:30 AM, Xavier Noria

one topic, one partition, many consumer instances

Can a consumer group that listens to one topic, which has only one partition, have multiple consumer instances in different processes/machines? Or can you have at most as many consumer instances as partitions?

Re: why kafka index file use memory mapped files ,however log file doesn't

If that is like what you said , why index file use the memory mapped file? From: jan Sent: Monday, February 12, 2018 9:26 PM To: users@kafka.apache.org Subject: Re: why kafka index file use memory mapped files ,however log file doesn't

Re: Kafka Consumer Offsets unavailable during rebalancing

I logged https://issues.apache.org/jira/browse/KAFKA-6557 Best regards, Wouter On 13 February 2018 at 12:50, Jan Filipiak wrote: > I would encourage you todo so. > I also think its not reasonable behavior > > On 13.02.2018 11:28, Wouter Bancken wrote: > >> We have

Authentication between Producer and Broker

Hi, Im using Kafka to collect Device Status and Log from IoT gateway which are installed different location in users place. Im trying to setup Authentication option between client (producer) and the broker., In Password based authentication what if I want to change password frequently as a best

Re: Kafka Consumer Offsets unavailable during rebalancing

I would encourage you todo so. I also think its not reasonable behavior On 13.02.2018 11:28, Wouter Bancken wrote: We have upgraded our Kafka version as an attempt to solve this issue. However, the issue is still present in Kafka 1.0.0. Can I log a bug for this in JIRA? Wouter On 5 February

Re: [VOTE] 1.0.1 RC1

Hi Ewen, I'm trying to run the streams quickstart and I'm getting: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.0.1:generate (default-cli) on project standalone-pom: The desired archetype does not exist (org.apache.kafka:streams-quickstart-java:1.0.1)

Re: Kafka Consumer Offsets unavailable during rebalancing

We have upgraded our Kafka version as an attempt to solve this issue. However, the issue is still present in Kafka 1.0.0. Can I log a bug for this in JIRA? Wouter On 5 February 2018 at 09:22, Wouter Bancken wrote: > The consumers in consumer group 'X' do not have a

Re: [VOTE] 1.0.1 RC1

+1 (non-binding) ran quick-start, unit tests on the src. On Tue, Feb 13, 2018 at 5:31 AM, Ewen Cheslack-Postava wrote: > Thanks for the heads up, I forgot to drop the old ones, I've done that and > rc1 artifacts should be showing up now. > > -Ewen > > > On Mon, Feb 12,

Re: question on serialization ..

There is an overload `leftJoin(KTable, ValuJoiner, Joined)` Joined is where you specify the Serde for the KTable and for the resulting type. We don't need the Serde for the stream at this point as the value has already been deserialized. HTH, Damian On Tue, 13 Feb 2018 at 05:39 Debasish Ghosh

Re: Kafka Stream tuning.

Hi Brilly, My initial guess is that it is the overhead of committing. Commit is synchronous and you have the commit interval set to 50ms. Perhaps try increasing it. Thanks, Damian On Tue, 13 Feb 2018 at 07:49 TSANG, Brilly wrote: > Hi kafka users, > > I created a