[jira] [Commented] (KAFKA-8611) Make topic optional when using through() operations in DSL
[ https://issues.apache.org/jira/browse/KAFKA-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875091#comment-16875091 ] Levani Kokhreidze commented on KAFKA-8611: -- Great! Thank you, will do, > Make topic optional when using through() operations in DSL > -- > > Key: KAFKA-8611 > URL: https://issues.apache.org/jira/browse/KAFKA-8611 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Assignee: Levani Kokhreidze >Priority: Minor > Labels: needs-kip > > When using DSL in Kafka Streams, data re-partition happens only when > key-changing operation is followed by stateful operation. On the other hand, > in DSL, stateful computation can happen using _transform()_ operation as > well. Problem with this approach is that, even if any upstream operation was > key-changing before calling _transform()_, no auto-repartition is triggered. > If repartitioning is required, a call to _through(String)_ should be > performed before _transform()_. With the current implementation, burden of > managing and creating the topic falls on user and introduces extra complexity > of managing Kafka Streams application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8611) Make topic optional when using through() operations in DSL
[ https://issues.apache.org/jira/browse/KAFKA-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875087#comment-16875087 ] Matthias J. Sax commented on KAFKA-8611: [~lkokhreidze], I added you to the list of contributors and assigned this ticket to you. You can now self-assign tickets. too. Feel free to reassign the other two tickets to yourself, and add this ticket to the KIP-221 wiki page. Thanks for contributing to Kafka! > Make topic optional when using through() operations in DSL > -- > > Key: KAFKA-8611 > URL: https://issues.apache.org/jira/browse/KAFKA-8611 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Assignee: Levani Kokhreidze >Priority: Minor > Labels: needs-kip > > When using DSL in Kafka Streams, data re-partition happens only when > key-changing operation is followed by stateful operation. On the other hand, > in DSL, stateful computation can happen using _transform()_ operation as > well. Problem with this approach is that, even if any upstream operation was > key-changing before calling _transform()_, no auto-repartition is triggered. > If repartitioning is required, a call to _through(String)_ should be > performed before _transform()_. With the current implementation, burden of > managing and creating the topic falls on user and introduces extra complexity > of managing Kafka Streams application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8611) Make topic optional when using through() operations in DSL
[ https://issues.apache.org/jira/browse/KAFKA-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875085#comment-16875085 ] Matthias J. Sax commented on KAFKA-8611: Might be easiest to do all at once with one umbrella KIP. KIP-221 has broader scope, so I might be easiest to discard KIP-485, and embed it in KIP-221 – if fact, the idea to allow removing the name requirement was already considered in the discussion of KIP-221. > Make topic optional when using through() operations in DSL > -- > > Key: KAFKA-8611 > URL: https://issues.apache.org/jira/browse/KAFKA-8611 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Priority: Minor > Labels: needs-kip > > When using DSL in Kafka Streams, data re-partition happens only when > key-changing operation is followed by stateful operation. On the other hand, > in DSL, stateful computation can happen using _transform()_ operation as > well. Problem with this approach is that, even if any upstream operation was > key-changing before calling _transform()_, no auto-repartition is triggered. > If repartitioning is required, a call to _through(String)_ should be > performed before _transform()_. With the current implementation, burden of > managing and creating the topic falls on user and introduces extra complexity > of managing Kafka Streams application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8611) Make topic optional when using through() operations in DSL
[ https://issues.apache.org/jira/browse/KAFKA-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875078#comment-16875078 ] Levani Kokhreidze commented on KAFKA-8611: -- Happy to work on that as well, can be really useful feature. In terms of organization, what would you suggest work on them separately or unite all three issues (KAFKA-8611, KAFKA-6037, KAFKA-4835) under one umbrella KIP? > Make topic optional when using through() operations in DSL > -- > > Key: KAFKA-8611 > URL: https://issues.apache.org/jira/browse/KAFKA-8611 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Priority: Minor > Labels: needs-kip > > When using DSL in Kafka Streams, data re-partition happens only when > key-changing operation is followed by stateful operation. On the other hand, > in DSL, stateful computation can happen using _transform()_ operation as > well. Problem with this approach is that, even if any upstream operation was > key-changing before calling _transform()_, no auto-repartition is triggered. > If repartitioning is required, a call to _through(String)_ should be > performed before _transform()_. With the current implementation, burden of > managing and creating the topic falls on user and introduces extra complexity > of managing Kafka Streams application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8611) Make topic optional when using through() operations in DSL
[ https://issues.apache.org/jira/browse/KAFKA-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875060#comment-16875060 ] Matthias J. Sax commented on KAFKA-8611: Thanks the ticket. Seems to overlap with https://issues.apache.org/jira/browse/KAFKA-6037 and https://issues.apache.org/jira/browse/KAFKA-4835 (ie, [https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint]) I think we should consolidate the work. Nodody is actively working on KIP-221, thus I think you could take over all if it. \cc [~bchen225242], would this be ok with you? > Make topic optional when using through() operations in DSL > -- > > Key: KAFKA-8611 > URL: https://issues.apache.org/jira/browse/KAFKA-8611 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Levani Kokhreidze >Priority: Minor > Labels: needs-kip > > When using DSL in Kafka Streams, data re-partition happens only when > key-changing operation is followed by stateful operation. On the other hand, > in DSL, stateful computation can happen using _transform()_ operation as > well. Problem with this approach is that, even if any upstream operation was > key-changing before calling _transform()_, no auto-repartition is triggered. > If repartitioning is required, a call to _through(String)_ should be > performed before _transform()_. With the current implementation, burden of > managing and creating the topic falls on user and introduces extra complexity > of managing Kafka Streams application. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
