[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7895:
---
Fix Version/s: (was: 2.1.2)
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.2.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Blocker
> Fix For: 2.3.0, 2.2.1
>
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
> Could you please take a look?
> Thanks
>
>
> Added by John:
> Acceptance Criteria:
> * add suppress to system tests, such that it's exercised with crash/shutdown
> recovery, rebalance, etc.
> ** [https://github.com/apache/kafka/pull/6278]
> * make sure that there's some system test coverage with caching disabled.
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7943
> * test with tighter time bounds with windows of say 30 seconds and use
> system time without adding any extra time for verification
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7944
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7895:
---
Fix Version/s: 2.3.0
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.2.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Blocker
> Fix For: 2.3.0, 2.1.2, 2.2.1
>
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
> Could you please take a look?
> Thanks
>
>
> Added by John:
> Acceptance Criteria:
> * add suppress to system tests, such that it's exercised with crash/shutdown
> recovery, rebalance, etc.
> ** [https://github.com/apache/kafka/pull/6278]
> * make sure that there's some system test coverage with caching disabled.
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7943
> * test with tighter time bounds with windows of say 30 seconds and use
> system time without adding any extra time for verification
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7944
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler updated KAFKA-7895:
Fix Version/s: (was: 2.2.0)
2.2.1
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.2.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Blocker
> Fix For: 2.1.2, 2.2.1
>
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
> Could you please take a look?
> Thanks
>
>
> Added by John:
> Acceptance Criteria:
> * add suppress to system tests, such that it's exercised with crash/shutdown
> recovery, rebalance, etc.
> ** [https://github.com/apache/kafka/pull/6278]
> * make sure that there's some system test coverage with caching disabled.
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7943
> * test with tighter time bounds with windows of say 30 seconds and use
> system time without adding any extra time for verification
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7944
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler updated KAFKA-7895:
Affects Version/s: 2.2.0
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.2.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Blocker
> Fix For: 2.2.0, 2.1.2
>
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
> Could you please take a look?
> Thanks
>
>
> Added by John:
> Acceptance Criteria:
> * add suppress to system tests, such that it's exercised with crash/shutdown
> recovery, rebalance, etc.
> ** [https://github.com/apache/kafka/pull/6278]
> * make sure that there's some system test coverage with caching disabled.
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7943
> * test with tighter time bounds with windows of say 30 seconds and use
> system time without adding any extra time for verification
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7944
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler updated KAFKA-7895:
Priority: Blocker (was: Major)
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Blocker
> Fix For: 2.2.0, 2.1.2
>
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
> Could you please take a look?
> Thanks
>
>
> Added by John:
> Acceptance Criteria:
> * add suppress to system tests, such that it's exercised with crash/shutdown
> recovery, rebalance, etc.
> ** [https://github.com/apache/kafka/pull/6278]
> * make sure that there's some system test coverage with caching disabled.
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7943
> * test with tighter time bounds with windows of say 30 seconds and use
> system time without adding any extra time for verification
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7944
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bill Bejeck updated KAFKA-7895:
---
Fix Version/s: 2.1.2
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Major
> Fix For: 2.2.0, 2.1.2
>
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
> Could you please take a look?
> Thanks
>
>
> Added by John:
> Acceptance Criteria:
> * add suppress to system tests, such that it's exercised with crash/shutdown
> recovery, rebalance, etc.
> ** [https://github.com/apache/kafka/pull/6278]
> * make sure that there's some system test coverage with caching disabled.
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7943
> * test with tighter time bounds with windows of say 30 seconds and use
> system time without adding any extra time for verification
> ** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7944
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler updated KAFKA-7895:
Description:
Hi, We are using kstreams to get the aggregated counts per vendor(key) within a
specified window.
Here's how we configured the suppress operator to emit one final record per
key/window.
{code:java}
KTable, Long> windowedCount = groupedStream
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
.count(Materialized.with(Serdes.Integer(),Serdes.Long()))
.suppress(Suppressed.untilWindowCloses(unbounded()));
{code}
But we are getting more than one record for the same key/window as shown below.
{code:java}
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
[KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
[KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
[KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
[KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
Could you please take a look?
Thanks
Added by John:
Acceptance Criteria:
* add suppress to system tests, such that it's exercised with crash/shutdown
recovery, rebalance, etc.
** [https://github.com/apache/kafka/pull/6278]
* make sure that there's some system test coverage with caching disabled.
** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7943
* test with tighter time bounds with windows of say 30 seconds and use system
time without adding any extra time for verification
** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7944
was:
Hi, We are using kstreams to get the aggregated counts per vendor(key) within a
specified window.
Here's how we configured the suppress operator to emit one final record per
key/window.
{code:java}
KTable, Long> windowedCount = groupedStream
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
.count(Materialized.with(Serdes.Integer(),Serdes.Long()))
.suppress(Suppressed.untilWindowCloses(unbounded()));
{code}
But we are getting more than one record for the same key/window as shown below.
{code:java}
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
[KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
[KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
[KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
[KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
Could you please take a look?
Thanks
Added by John:
Acceptance Criteria:
* add suppress to system tests, such that it's exercised with crash/shutdown
recovery, rebalance, etc.
** [https://github.com/apache/kafka/pull/6278]
* make sure that there's some system test coverage with caching disabled.
** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7943
* test with tighter time bounds with windows of say 30 seconds and use system
time without adding any extra time for verification
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Major
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]:
[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler updated KAFKA-7895:
Description:
Hi, We are using kstreams to get the aggregated counts per vendor(key) within a
specified window.
Here's how we configured the suppress operator to emit one final record per
key/window.
{code:java}
KTable, Long> windowedCount = groupedStream
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
.count(Materialized.with(Serdes.Integer(),Serdes.Long()))
.suppress(Suppressed.untilWindowCloses(unbounded()));
{code}
But we are getting more than one record for the same key/window as shown below.
{code:java}
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
[KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
[KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
[KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
[KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
Could you please take a look?
Thanks
Added by John:
Acceptance Criteria:
* add suppress to system tests, such that it's exercised with crash/shutdown
recovery, rebalance, etc.
** [https://github.com/apache/kafka/pull/6278]
* make sure that there's some system test coverage with caching disabled.
** Follow-on ticket: https://issues.apache.org/jira/browse/KAFKA-7943
* test with tighter time bounds with windows of say 30 seconds and use system
time without adding any extra time for verification
was:
Hi, We are using kstreams to get the aggregated counts per vendor(key) within a
specified window.
Here's how we configured the suppress operator to emit one final record per
key/window.
{code:java}
KTable, Long> windowedCount = groupedStream
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
.count(Materialized.with(Serdes.Integer(),Serdes.Long()))
.suppress(Suppressed.untilWindowCloses(unbounded()));
{code}
But we are getting more than one record for the same key/window as shown below.
{code:java}
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
[KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
[KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
[KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
[KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
Could you please take a look?
Thanks
Added by John:
Acceptance Criteria:
* add suppress to system tests, such that it's exercised with crash/shutdown
recovery, rebalance, etc.
** https://github.com/apache/kafka/pull/6278
* make sure that there's some system test coverage with caching disabled.
* test with tighter time bounds with windows of say 30 seconds and use system
time without adding any extra time for verification
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Major
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
> [KTABLE-TOSTREAM-10]:
[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler updated KAFKA-7895:
Description:
Hi, We are using kstreams to get the aggregated counts per vendor(key) within a
specified window.
Here's how we configured the suppress operator to emit one final record per
key/window.
{code:java}
KTable, Long> windowedCount = groupedStream
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
.count(Materialized.with(Serdes.Integer(),Serdes.Long()))
.suppress(Suppressed.untilWindowCloses(unbounded()));
{code}
But we are getting more than one record for the same key/window as shown below.
{code:java}
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
[KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
[KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
[KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
[KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
Could you please take a look?
Thanks
Added by John:
Acceptance Criteria:
* add suppress to system tests, such that it's exercised with crash/shutdown
recovery, rebalance, etc.
** https://github.com/apache/kafka/pull/6278
* make sure that there's some system test coverage with caching disabled.
* test with tighter time bounds with windows of say 30 seconds and use system
time without adding any extra time for verification
was:
Hi, We are using kstreams to get the aggregated counts per vendor(key) within a
specified window.
Here's how we configured the suppress operator to emit one final record per
key/window.
{code:java}
KTable, Long> windowedCount = groupedStream
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
.count(Materialized.with(Serdes.Integer(),Serdes.Long()))
.suppress(Suppressed.untilWindowCloses(unbounded()));
{code}
But we are getting more than one record for the same key/window as shown below.
{code:java}
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
[KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
[KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
[KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
[KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
Could you please take a look?
Thanks
Added by John:
Acceptance Criteria:
* add suppress to system tests, such that it's exercised with crash/shutdown
recovery, rebalance, etc.
* make sure that there's some system test coverage with caching disabled.
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Major
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
> Could you please take a look?
> Thanks
>
>
> Added by John:
> Acceptance Criteria:
> * add suppress to system tests, such that it's exercised with crash/shutdown
> recovery, rebalance, etc.
> **
[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler updated KAFKA-7895:
Description:
Hi, We are using kstreams to get the aggregated counts per vendor(key) within a
specified window.
Here's how we configured the suppress operator to emit one final record per
key/window.
{code:java}
KTable, Long> windowedCount = groupedStream
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
.count(Materialized.with(Serdes.Integer(),Serdes.Long()))
.suppress(Suppressed.untilWindowCloses(unbounded()));
{code}
But we are getting more than one record for the same key/window as shown below.
{code:java}
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
[KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
[KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
[KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
[KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
Could you please take a look?
Thanks
Added by John:
Acceptance Criteria:
* add suppress to system tests, such that it's exercised with crash/shutdown
recovery, rebalance, etc.
* make sure that there's some system test coverage with caching disabled.
was:
Hi, We are using kstreams to get the aggregated counts per vendor(key) within a
specified window.
Here's how we configured the suppress operator to emit one final record per
key/window.
{code:java}
KTable, Long> windowedCount = groupedStream
.windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
.count(Materialized.with(Serdes.Integer(),Serdes.Long()))
.suppress(Suppressed.untilWindowCloses(unbounded()));
{code}
But we are getting more than one record for the same key/window as shown below.
{code:java}
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
[KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
[KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
[KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
[KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
[KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
[KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
Could you please take a look?
Thanks
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Major
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
> Could you please take a look?
> Thanks
>
>
> Added by John:
> Acceptance Criteria:
> * add suppress to system tests, such that it's exercised with crash/shutdown
> recovery, rebalance, etc.
> * make sure that there's some system test coverage with caching disabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7895) Ktable supress operator emitting more than one record for the same key per window
[
https://issues.apache.org/jira/browse/KAFKA-7895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7895:
---
Affects Version/s: 2.1.1
2.1.0
> Ktable supress operator emitting more than one record for the same key per
> window
> -
>
> Key: KAFKA-7895
> URL: https://issues.apache.org/jira/browse/KAFKA-7895
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0, 2.1.1
>Reporter: prasanthi
>Assignee: John Roesler
>Priority: Major
>
> Hi, We are using kstreams to get the aggregated counts per vendor(key) within
> a specified window.
> Here's how we configured the suppress operator to emit one final record per
> key/window.
> {code:java}
> KTable, Long> windowedCount = groupedStream
> .windowedBy(TimeWindows.of(Duration.ofMinutes(1)).grace(ofMillis(5L)))
> .count(Materialized.with(Serdes.Integer(),Serdes.Long()))
> .suppress(Suppressed.untilWindowCloses(unbounded()));
> {code}
> But we are getting more than one record for the same key/window as shown
> below.
> {code:java}
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1039
> [KTABLE-TOSTREAM-10]: [131@154906704/154906710], 1162
> [KTABLE-TOSTREAM-10]: [9@154906704/154906710], 6584
> [KTABLE-TOSTREAM-10]: [88@154906704/154906710], 107
> [KTABLE-TOSTREAM-10]: [108@154906704/154906710], 315
> [KTABLE-TOSTREAM-10]: [119@154906704/154906710], 119
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 746
> [KTABLE-TOSTREAM-10]: [154@154906704/154906710], 809{code}
> Could you please take a look?
> Thanks
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
