Re: is ktable-ktable join stil having the risk of non determinism?

2020-07-22 Thread Matthias J. Sax
The answer is many-fold: (1) There is only a final result if the input topics don't get new data. For this case, the result is deterministic. You can consider it eventually consistent. The non-determinism really only applies to "intermediate results". Both input-topic partitions for a join are

Re: Confluent Platform- KTable clarification

2020-07-22 Thread Matthias J. Sax
The ides is "sum value by key" in this example, what is maybe not the perfect example. However, if you have a KTable, you can do a `groupBy(...).aggregate(...)` and the same update logic applies: k1 : a,1 k2 : a,2 k3 : b,3 k4 : b,4 If you groupBy the first attribute in the value and sum, you

Re: Confluent Platform- KTable clarification

2020-07-22 Thread John Roesler
Hello Nag, Yes, your conclusion sounds right. “Sum the values per key” is a statement that doesn’t really make sense in a KTable context, since there is always just one value per key (the latest update). I think the docs are just trying to drive the point home that in a KTable, there is just

kafka tuning(vm.max_map_count) and logs retention.

2020-07-22 Thread Vitalii Stoianov
Hi All, According to this: https://docs.confluent.io/current/kafka/deployment.html vm.max_map_count is depend on number of index file: *find /tmp/kafka_logs -name '*index' | wc -l* In our test lab we have next setup: *Topic:test PartitionCount:256 ReplicationFactor:2

Re: [VOTE] 2.6.0 RC1

2020-07-22 Thread Randall Hauch
Any thoughts, Rajini? On Mon, Jul 20, 2020 at 9:55 PM Randall Hauch wrote: > > When I was checking the documentation for RC1 after the tag was pushed, I > noticed that the fix Rajini mentioned in the RC0 vote thread ( > https://github.com/apache/kafka/pull/8979 >

Re: Enabling Unclean leader election

2020-07-22 Thread Ricardo Ferreira
You should be able to use the command `kafka-leader-election` to accomplish this. This command has an option called "--election-type" that you can use to specify whether the election is preferred or unclean. -- Ricardo On 7/22/20 11:31 AM, nitin agarwal wrote: Hi, Is there a way to enable

Confluent Docker Images

2020-07-22 Thread Nag Y
The Docker images are huge for each confluent component - ZK, registry etc .. Is there any other place I can download one image that contains all the components ? REPOSITORY TAG IMAGE ID CREATED SIZE

Enabling Unclean leader election

2020-07-22 Thread nitin agarwal
Hi, Is there a way to enable Unclean leader election in Kafka without restarting the broker? We have a use case where we want to enable the Unclean leader election conditionally. Thanks, Nitin

Confluent Platform- KTable clarification

2020-07-22 Thread Nag Y
I understood A KStream is an abstraction of a record stream and A KTable is an abstraction of a changelog stream ( updates or inserts) and the semantics around it. However, this is where some confusion arises .. From confluent documentation

Kafka - Controller Broker

2020-07-22 Thread Nag Y
come across this phrase from https://niqdev.github.io/devops/kafka/ and https://livebook.manning.com/book/kafka-streams-in-action/chapter-2/109 (Kafka Streams in Action ) The controller broker is responsible for setting up leader/follower relationships for all partitions of a topic. If a Kafka

Confluent Kafka - Schema Registry on windows

2020-07-22 Thread Nag Y
I happened to see an example how to run schema registry using "schema-registry-start.bat" from windows on 5.0.1 I didnt see the file in 5.5.0 . Is the schema registry not supported in windows now ? IT seems only the way to go about running schema registry in windows through dockers . Please