[jira] [Commented] (KAFKA-7595) Kafka Streams: KTrable to KTable join introduces duplicates in downstream KTable
[
https://issues.apache.org/jira/browse/KAFKA-7595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688927#comment-16688927
]
Vik Gamov commented on KAFKA-7595:
--
[~mjsax] [~vvcephei]
Thank you for input. I was able to rewrite my app according to Matthias
recommendations
[https://github.com/gAmUssA/streams-movie-demo/blob/refactoring/1/streams/src/main/java/io/confluent/demo/StreamsDemo.java#L165]
> Kafka Streams: KTrable to KTable join introduces duplicates in downstream
> KTable
>
>
> Key: KAFKA-7595
> URL: https://issues.apache.org/jira/browse/KAFKA-7595
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.0.0
>Reporter: Vik Gamov
>Priority: Major
>
> When perform KTable to KTable join after aggregation, there are duplicates in
> resulted KTable.
> 1. caching disabled, no materialized => duplicates
> {{streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG,
> 0);}}
> {{KTable ratingCounts = ratingsById.count();}}
> {{KTable ratingSums = ratingsById.reduce((v1, v2) -> v1 + v2);}}
> {{KTable ratingAverage = ratingSums.join(ratingCounts,}}
> {{ (sum, count) -> sum / count.doubleValue());}}
> 2. caching disabled, materialized => duplicate
> {{streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG,
> 0);}}{{KTable ratingCounts = ratingsById.count();}}
> {{KTable ratingSums = ratingsById.reduce((v1, v2) -> v1 + v2);}}
> {{KTable ratingAverage = ratingSums.join(ratingCounts,}}
> {{ (sum, count) -> sum / count.doubleValue(),}}
> {{ Materialized.as("average-ratings"));}}
> 3. caching enabled, materiazlized => all good
> {{// Enable record cache of size 10 MB.}}
> {{streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 10
> * 1024 * 1024L);}}
> {{// Set commit interval to 1 second.}}
> {{streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG,
> 1000);}}{{KTable ratingCounts = ratingsById.count();}}
> {{KTable ratingSums = ratingsById.reduce((v1, v2) -> v1 + v2);}}
> {{KTable ratingAverage = ratingSums.join(ratingCounts,}}
> {{ (sum, count) -> sum / count.doubleValue(),}}
> {{ Materialized.as("average-ratings"));}}
>
> Demo app
> [https://github.com/tlberglund/streams-movie-demo/blob/master/streams/src/main/java/io/confluent/demo/StreamsDemo.java#L107]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7595) Kafka Streams: KTrable to KTable join introduces duplicates in downstream KTable
Vik Gamov created KAFKA-7595:
Summary: Kafka Streams: KTrable to KTable join introduces
duplicates in downstream KTable
Key: KAFKA-7595
URL: https://issues.apache.org/jira/browse/KAFKA-7595
Project: Kafka
Issue Type: Bug
Components: streams
Affects Versions: 2.0.0
Reporter: Vik Gamov
When perform KTable to KTable join after aggregation, there are duplicates in
resulted KTable.
1. caching disabled, no materialized => duplicates
{{streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);}}
{{KTable ratingCounts = ratingsById.count();}}
{{KTable ratingSums = ratingsById.reduce((v1, v2) -> v1 + v2);}}
{{KTable ratingAverage = ratingSums.join(ratingCounts,}}
{{ (sum, count) -> sum / count.doubleValue());}}
2. caching disabled, materialized => duplicate
{{streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG,
0);}}{{KTable ratingCounts = ratingsById.count();}}
{{KTable ratingSums = ratingsById.reduce((v1, v2) -> v1 + v2);}}
{{KTable ratingAverage = ratingSums.join(ratingCounts,}}
{{ (sum, count) -> sum / count.doubleValue(),}}
{{ Materialized.as("average-ratings"));}}
3. caching enabled, materiazlized => all good
{{// Enable record cache of size 10 MB.}}
{{streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 10 *
1024 * 1024L);}}
{{// Set commit interval to 1 second.}}
{{streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG,
1000);}}{{KTable ratingCounts = ratingsById.count();}}
{{KTable ratingSums = ratingsById.reduce((v1, v2) -> v1 + v2);}}
{{KTable ratingAverage = ratingSums.join(ratingCounts,}}
{{ (sum, count) -> sum / count.doubleValue(),}}
{{ Materialized.as("average-ratings"));}}
Demo app
[https://github.com/tlberglund/streams-movie-demo/blob/master/streams/src/main/java/io/confluent/demo/StreamsDemo.java#L107]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-6574) Support Headers in console-consumer and console-producer
Vik Gamov created KAFKA-6574: Summary: Support Headers in console-consumer and console-producer Key: KAFKA-6574 URL: https://issues.apache.org/jira/browse/KAFKA-6574 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 1.0.0 Reporter: Vik Gamov Message headers were introduced in 1.0.0 but console producer and consumer don't support them, -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-2967) Move Kafka documentation to ReStructuredText
[ https://issues.apache.org/jira/browse/KAFKA-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357798#comment-16357798 ] Vik Gamov edited comment on KAFKA-2967 at 2/9/18 1:27 AM: -- Guys, Have we ever consider to use asciidoc [http://asciidoctor.org/docs/what-is-asciidoc/] for writing documentation rather rst or markdown? There are bunch of known projects who use asciidoc [http://asciidoctor.org/docs/what-is-asciidoc/#who-s-using-asciidoc] (like GIT documentation and Neo4j database docs, github supports it) It's very powerful and suitable for writing complex text (like books) Oreilly uses it for book writers, there are plugins for gradle to generate html5, pdf, and even mobile optimized formats like epub and mobi. It has wide range integrations (including gradle and maven) [http://asciidoctor.org/docs/#references-and-developer-resources] Here is how it different than markdown (for example) [http://asciidoctor.org/docs/user-manual/#compared-to-markdown]. Thank you was (Author: vikgamov): Guys, Have we ever consider to use asciidoc [http://asciidoctor.org/docs/what-is-asciidoc/] for writing documentation rather rst or markdown? It's very powerful and suitable for writing complex text (like books) Oreilly uses it for book writers, there are plugins for gradle to generate html5, pdf, and even mobile optimized formats like epub and mobi. It has wide range integrations (including gradle and maven) [http://asciidoctor.org/docs/#references-and-developer-resources] Here is how it different than markdown (for example) http://asciidoctor.org/docs/user-manual/#compared-to-markdown. Thank you > Move Kafka documentation to ReStructuredText > > > Key: KAFKA-2967 > URL: https://issues.apache.org/jira/browse/KAFKA-2967 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira >Assignee: Gwen Shapira >Priority: Major > > Storing documentation as HTML is kind of BS :) > * Formatting is a pain, and making it look good is even worse > * Its just HTML, can't generate PDFs > * Reading and editting is painful > * Validating changes is hard because our formatting relies on all kinds of > Apache Server features. > I suggest: > * Move to RST > * Generate HTML and PDF during build using Sphinx plugin for Gradle. > Lots of Apache projects are doing this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-2967) Move Kafka documentation to ReStructuredText
[ https://issues.apache.org/jira/browse/KAFKA-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357798#comment-16357798 ] Vik Gamov commented on KAFKA-2967: -- Guys, Have we ever consider to use asciidoc [http://asciidoctor.org/docs/what-is-asciidoc/] for writing documentation rather rst or markdown? It's very powerful and suitable for writing complex text (like books) Oreilly uses it for book writers, there are plugins for gradle to generate html5, pdf, and even mobile optimized formats like epub and mobi. It has wide range integrations (including gradle and maven) [http://asciidoctor.org/docs/#references-and-developer-resources] Here is how it different than markdown (for example) http://asciidoctor.org/docs/user-manual/#compared-to-markdown. Thank you > Move Kafka documentation to ReStructuredText > > > Key: KAFKA-2967 > URL: https://issues.apache.org/jira/browse/KAFKA-2967 > Project: Kafka > Issue Type: Bug >Reporter: Gwen Shapira >Assignee: Gwen Shapira >Priority: Major > > Storing documentation as HTML is kind of BS :) > * Formatting is a pain, and making it look good is even worse > * Its just HTML, can't generate PDFs > * Reading and editting is painful > * Validating changes is hard because our formatting relies on all kinds of > Apache Server features. > I suggest: > * Move to RST > * Generate HTML and PDF during build using Sphinx plugin for Gradle. > Lots of Apache projects are doing this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
