Re: KTable Suppress not working

Thanks John, I'll try increasing the "CACHE_MAX_BYTES_BUFFERING_CONFIG" and "COMMIT_INTERVAL_MS_CONFIG" configurations. Thanks, Sushrut On Sat, Jan 18, 2020 at 11:31 AM John Roesler wrote: > Ah, I should add, if you actually want to use suppression, or > you need to resolve a similar error

Re: KTable Suppress not working

Ah, I should add, if you actually want to use suppression, or you need to resolve a similar error message in the future, you probably need to tweak the batch sizes and/or timeout configs of the various clients, and maybe the server as well. That error message kind of sounds like the server went

Re: KTable Suppress not working

Hi Sushrut, That's frustrating... I haven't seen that before, but looking at the error in combination with what you say happens without suppress makes me think there's a large volume of data involved here. Probably, the problem isn't specific to suppression, but it's just that the interactions on

Re: Clustered MirrorMaker 2 configuration update

That's right. The leader(s) will apply the config by writing it to the mm2-config topic(s), which the followers will pick up. If the entire cluster is bounced, the new config will have been applied. Ryanne On Fri, Jan 17, 2020, 4:28 AM Péter Sinóros-Szabó wrote: > Am I right that the

KTable Suppress not working

Hey, I'm building a streams application where I'm trying to aggregate a stream of events and getting a list of events per key. `eventStream .groupByKey(Grouped.with(Serdes.String(), eventSerde)) .windowedBy(TimeWindows.of(Duration.ofMillis(50)).grace(Duration.ofMillis(1))) .aggregate(

Minor version upgrade instructions from 0.11.0.0 to 0.11.0.3

Hi, We are using kafka 0.11.0.0 and I believe hitting this defect - [KAFKA-6003] Replication Fetcher thread for a partition with no data fails to start - ASF JIRA | | | | [KAFKA-6003] Replication Fetcher thread for a partition with no data fai... | | | Solution was patched to

Re: Mirrormaker 2.0

Peter, KIP-382 includes LegacyReplicationPolicy for this purpose, but no, it has not been implemented yet. If you are interested in writing the PR, it would not require a separate KIP before merging. Looks like you are already doing the work :) It is possible, as you point out, that returning

Re: Sort data across partitions and put it in another topic

I'm not familiar with Kafka Streams API, but I guess it's possible to use it since the data need to be consumed from some source, processed, and the results should be produced into some other destination. The main point here, that you would need to specify which source partitions should be used

Re: Mirrormaker 2.0

Hi Sebastian & Ryanne, do you have maybe an implementation of this is just some ideas about how to implement the policy that does not rename topics? I am checking the ReplicationPolicy interface and don't really know what the impact will be if I implement this: public String

Re: Kafka encoding UTF-8 problem

Hi Marco, On Fri, 17 Jan 2020 at 13:08, Marco Di Falco wrote: > Hi. > I'm sorry, I'm new to kafka and I'm afraid I haven't understood your > question. > How do I try a serialization / deserialization if I communicate in two > windows shell? I have no explicit serialization / deserialization, >

R: Kafka encoding UTF-8 problem

Hi. I'm sorry, I'm new to kafka and I'm afraid I haven't understood your question. How do I try a serialization / deserialization if I communicate in two windows shell? I have no explicit serialization / deserialization, communication is transparent to me. I think I have to set a kafka

Re: Kafka encoding UTF-8 problem

Hi, On Fri, 17 Jan 2020 at 11:18, Marco Di Falco wrote: > Hello guys! > I have a producer and consumer running in a windows shell. > I write this message ‘questo è un test’ and in consumer receive this: > “questo ´┐¢ un test” . > > What properties should I use to set up character coding in

Re: unpacking a column from a JSON structured msg, using stream processing

ignore comment just saw the kSQL button. G On Fri, Jan 17, 2020 at 1:43 PM George wrote: > Hi Bruno > > From the description this looks exactly whats needed, will have to see how > to do this as I haven't touched Java in 20 yrs, someone did mention to me > this is also possible via kSQL and a

Re: unpacking a column from a JSON structured msg, using stream processing

Hi Bruno >From the description this looks exactly whats needed, will have to see how to do this as I haven't touched Java in 20 yrs, someone did mention to me this is also possible via kSQL and a user defined method. Any chance anyone has a example? G On Fri, Jan 17, 2020 at 10:21 AM Bruno

Re: MM2 produce timeouts

I figured it out. The correct config is: my_cluster_name.offset.flush.timeout.ms = 15000 Peter On Thu, 16 Jan 2020 at 16:48, Péter Sinóros-Szabó < peter.sinoros-sz...@transferwise.com> wrote: > Hey, > > running MM2 that tried to process its backlog of 1 week just after it was > started. > > I

Kafka encoding UTF-8 problem

Hello guys! I have a producer and consumer running in a windows shell. I write this message ‘questo è un test’ and in consumer receive this: “questo ´┐¢ un test” . What properties should I use to set up character coding in utf-8? thank you Marco

Re: Clustered MirrorMaker 2 configuration update

Am I right that the configuration that is actually being used will come from the leader? Well, except the exception like the bootstrap list... Asking because I'd like to have a clear process to change the configuration in a way that at the end of the process I can be sure that all nodes use the

Is there a way to auto scale topic partitions in kafka?

Hello, I am working on developing a microservice based system which uses kafka as a messaging infrastructure. The microservices application are mainly kafka consumers and kafka streams applications and are deployed as docker containers on kubernetes. The system should be designed to be auto