Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thanks a lot Levani! On 11/16/19 4:00 AM, Levani Kokhreidze wrote: > Matthias, > > Yes, I agree. KIP is updated: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint > >

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Matthias, Yes, I agree. KIP is updated: https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+DSL+with+Connecting+Topic+Creation+and+Repartition+Hint and

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Levani, do you agree to the current proposal? It's basically a de-scoping of the already voted KIP. If you agree, could you update the KIP wiki page accordingly, including the "Rejected Alternative" section (and mabye a link to a follow up Jira ticket). Because it's a descope, and John and

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thanks Sophie, I think your concern is valid, and also that your idea to make a ticket is a good idea. Creating a ticket has some very positive effects: * It allows us to record the thinking at this point in time so we don't have to dig through the mail archives later * It demonstrates that we

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

While I'm concerned that "not augmenting groupBy as part of this KIP" really translates to "will not get around to augmenting groupBy for a long time if not as part of this KIP", like I said I don't want to hold up the new .repartition operator that it seems we do, at least, all agree on. It's a

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi all, I think that Sophie is asking a good question, and I do think that such "blanket configurations" are plausible. For example, we currently support (and I would encourage) "I don't know if this is going to create a repartition topic, but if it does, then use this name instead of generating

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hello, Just fyi, PR was updated and now it incorporates the latest suggestions about joins. `CopartitionedTopicsEnforcer` will throw an exception if number of partitions aren’t the same when using `repartition` operation along with `join`. For more details please take a look at the PR:

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thanks a lot for the input Sophie. Your example is quite useful, and I would use it to support my claim that a "partition hint" for `Grouped` seems "useless" and does not improve the user experience. 1) You argue that a new user would be worries about repartitions topics with too many paritions.

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

It seems like we all agree at this point (please correct me if wrong!) that we should NOT change the existing repartitioning behavior, ie we should allow Streams to continue to determine when and where to repartition -- *unless* explicitly informed to by the use of a .through or the new

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Side remark: If the user specifies `repartition()` on both side of the join, we can actually throw the execption earlier, ie, when we build the topology. Current, we can do this check only after Kafka Streams was started, within `StreamPartitionAssignor#assign()` -- we still need to keep this

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thanks, all, I can get behind just totally leaving out reparation-via-groupBy. If we only introduce `repartition()` for now, we're making the minimal change to gain the desired capability. Plus, since we agree that `repartition()` should never be optimizable, it's a future-compatible proposal.

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thanks a lot John. I think the way you decompose the operators is super helpful for this discussion. What you suggest with regard to using `Grouped` and enforcing repartitioning if the number of partitions is specified is certainly possible. However, I am not sure if we _should_ do this. My

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thank you all for an interesting discussion. This is very enlightening. Thank you Matthias for your explanation. Your arguments are very true. It makes sense that if user specifies number of partitions he/she really cares that those specifications are applied to internal topics. Unfortunately,

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Ah, thanks for the clarification. I missed your point. I like the framework you've presented. It does seem simpler to assume that they either care about the partition count and want to repartition to realize it, or they don't care about the number. Returning to this discussion, it does seem

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

> it seems like we do want to allow >> people to optionally specify a partition count as part of this >> operation, but we don't want that option to _force_ repartitioning Correct, ie, that is my suggestions. > "Use P partitions if repartitioning is necessary" I disagree here, because my

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thanks for those thoughts, Matthias, I find your reasoning about the optimization behavior compelling. The `through` operation is very simple and clear to reason about. It just passes the data exactly at the specified point in the topology exactly through the specified topic. Likewise, if a user

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Sorry for late reply. I guess, the question boils down to the intended semantics of `repartition()`. My understanding is as follows: - KS does auto-repartitioning for correctness reasons (using the upstream topic to determine the number of partitions) - KS does auto-repartitioning only for

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi Sophie, Thank you for your reply, very insightful. Looking forward hearing others opinion as well on this. Kind regards, Levani > On Nov 6, 2019, at 1:30 AM, Sophie Blee-Goldman wrote: > >> Personally, I think Matthias’s concern is valid, but on the other hand > Kafka Streams has already

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

> Personally, I think Matthias’s concern is valid, but on the other hand Kafka Streams has already > optimizer in place which alters topology independently from user I agree (with you) and think this is a good way to put it -- we currently auto-repartition for the user so that they don't have to

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hello all, While https://github.com/apache/kafka/pull/7170 is under review and it’s almost done, I want to resurrect discussion about this KIP to address couple of concerns raised by Matthias and John. As a reminder, idea of the KIP-221 was to allow

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hey Levani, I think people are busy with the upcoming 2.4 release, and don't have much spare time at the moment. It's kind of a difficult time to get attention on things, but feel free to pick up something else to work on in the meantime until things have calmed down a bit! Cheers, Sophie On

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hello all, Sorry for bringing this thread again, but I would like to get some attention on this PR: https://github.com/apache/kafka/pull/7170 It's been a while now and I would love to move on to other KIPs as well. Please let me know if you have any

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi all, Here’s voting thread for this KIP: https://www.mail-archive.com/dev@kafka.apache.org/msg99680.html Regards, Levani > On Jul 24, 2019, at 11:15 PM, Levani Kokhreidze > wrote: > > Hi Matthias, > > Thanks for the

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi Matthias, Thanks for the suggestion. I Don’t have strong opinion on that one. Agree that avoiding unnecessary method overloads is a good idea. Updated KIP Regards, Levani > On Jul 24, 2019, at 8:50 PM, Matthias J. Sax wrote: > > One question: > > Why do we add > >>

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

One question: Why do we add > Repartitioned#with(final String name, final int numberOfPartitions) It seems that `#with(String name)`, `#numberOfPartitions(int)` in combination with `withName()` and `withNumberOfPartitions()` should be sufficient. Users can chain the method calls. (I think it's

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hello, Thanks all for your feedback. I started voting procedure for this KIP. If there’re any other concerns about this KIP, please let me know. Regards, Levani > On Jul 20, 2019, at 8:39 PM, Levani Kokhreidze wrote: > > Hi Matthias, > > Thanks for the suggestion, makes sense. > I’ve

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi Matthias, Thanks for the suggestion, makes sense. I’ve updated KIP (https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thanks for driving the KIP. I agree that users need to be able to specify a partitioning strategy. Sophie raises a fair point about topic configs and producer configs. My take is, that consider `Repartitioned` as an "extension" to `Produced`, that adds topic configuration, is a good way to think

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Totally agree. I think in KStream interface it makes sense to have some duplicate configurations between operators in order to keep API simple and usable. Also, as more surface API has, harder it is to have proper backward compatibility. While initial idea of keeping topic level configs

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

I think that is a good point about trying to keep producer level configurations and (repartition) topic level considerations separate. Number of partitions is definitely purely a topic level configuration. But on some level, serdes and partitioners are just as much a topic configuration as a

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi Bill, Thanks a lot for the feedback. Yes, that makes sense. I’ve updated KIP with `Repartitioned#partitioner` configuration. In the beginning, I wanted to introduce a class for topic level configuration and keep topic level and producer level configurations (such as Produced) separately

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi Lavani, Thanks for resurrecting this KIP. I'm also a +1 for adding a partition option. In addition to the reason provided by John, my reasoning is: 1. Users may want to use something other than hash-based partitioning 2. Users may wish to partition on something different than the key

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi John, In my mind it makes sense. If we add partitioner configuration to Repartitioned class, with the combination of specifying number of partitions for internal topics, user will have opportunity to ensure co-partitioning before join operation. I think this can be quite powerful feature.

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Yes, I believe that's what I had in mind. Again, not totally sure it makes sense, but I believe something similar is the rationale for having the partitioner option in Produced. Thanks, -John On Wed, Jul 17, 2019 at 3:20 PM Levani Kokhreidze wrote: > > Hey John, > > Oh that’s interesting

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hey John, Oh that’s interesting use-case. Do I understand this correctly, in your example I would first issue repartition(Repartitioned) with proper partitioner that essentially would be the same as the topic I want to join with and then do the KStream#join with DSL? Regards, Levani > On Jul

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hey, all, just to chime in, I think it might be useful to have an option to specify the partitioner. The case I have in mind is that some data may get repartitioned and then joined with an input topic. If the right-side input topic uses a custom partitioning strategy, then the repartitioned

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Yes, I was thinking about it as well. To be honest I’m not sure about it yet. As Kafka Streams DSL user, I don’t really think I would need control over partitioner for internal topics. As a user, I would assume that Kafka Streams knows best how to partition data for internal topics. In this KIP

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thanks for clearing that up. I agree that Repartitioned would be a useful addition. I'm wondering if it might also need to have a withStreamPartitioner method/field, similar to Produced? I'm not sure how widely this feature is really used, but seems it should be available for repartition topics.

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hey Sophie, In both cases KStream#repartition and KStream#repartition(Repartitioned) topic will be created and managed by Kafka Streams. Idea of Repartitioned is to give user more control over the topic such as num of partitions. I feel like Repartitioned parameter is something that is

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hey Levani, Thanks for the KIP! Can you clarify one thing for me -- for the KStream#repartition signature taking a Repartitioned, will the topic be auto-created by Streams (which seems to be the case for the signature without a Repartitioned) or does it have to be pre-created? The wording in the

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hello, One more bump about KIP-221 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint )

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hello, Kind reminder about this KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hello, In order to move this KIP forward, I’ve updated confluence page with the new proposal https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hello Matthias, Thanks for the feedback and ideas. I like the idea of introducing dedicated `Topic` class for topic configuration for internal operators like `groupedBy`. Would be great to hear others opinion about this as well. Kind regards, Levani > On Jul 3, 2019, at 7:00 AM, Matthias J.

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Levani, Thanks for picking up this KIP! And thanks for summarizing everything. Even if some points may have been discussed already (can't really remember), it's helpful to get a good summary to refresh the discussion. I think your reasoning makes sense. With regard to the distinction between

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Actually, giving it more though - maybe enhancing Produced with num of partitions configuration is not the best approach. Let me explain why: 1) If we enhance Produced class with this configuration, this will also affect KStream#to operation. Since KStream#to is the final sink of the topology,

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hello, I would like to resurrect discussion around KIP-221. Going through the discussion thread, there’s seems to agreement around usefulness of this feature. Regarding the implementation, as far as I understood, the most optimal solution for me seems the following: 1) Add two method

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi Lei, Please feel free to take over the KIP. Cheers, Jeyhun On Fri, Sep 21, 2018, 22:27 Lei Chen wrote: > Hi, > > Just want to know is anyone actively working on this and also KAFKA-4835 > ? Seems like the JIRA > has been inactive for couple

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

It seems Jeyhun (cc'ed) is not working on the KIP any longer. If there is no response within a week from Jeyhun, feel free to take over the KIP. One more side comment: we recently accepted KIP-372, that overlaps with this KIP. Thus, if you resume KIP-221, please consider the changes of KIP-372.

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi, Just want to know is anyone actively working on this and also KAFKA-4835 ? Seems like the JIRA has been inactive for couple months. We want this feature and would like to move it forward if no one else is working on it. Lei On Wed, Jun 20,

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

No worries. It's just good to know. It seems that some other people are interested to drive this further. So we will just "reassign" it to them. Thanks for letting us know. -Matthias On 6/20/18 2:51 PM, Jeyhun Karimov wrote: > Hi Matthias, all, > > Currently, I am not able to complete this

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi Matthias, all, Currently, I am not able to complete this KIP. Please accept my apologies for that. Cheers, Jeyhun On Mon, Jun 11, 2018 at 2:25 AM Matthias J. Sax wrote: > What is the status of this KIP? > > -Matthias > > > On 2/13/18 1:43 PM, Matthias J. Sax wrote: > > Is there any update

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

What is the status of this KIP? -Matthias On 2/13/18 1:43 PM, Matthias J. Sax wrote: > Is there any update for this KIP? > > > -Matthias > > On 12/4/17 2:08 PM, Matthias J. Sax wrote: >> Jeyhun, >> >> thanks for updating the KIP. >> >> I am wondering if you intend to add a new class

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Is there any update for this KIP? -Matthias On 12/4/17 2:08 PM, Matthias J. Sax wrote: > Jeyhun, > > thanks for updating the KIP. > > I am wondering if you intend to add a new class `Produced`? There is > already `org.apache.kafka.streams.kstream.Produced`. So if we want to > add a new class,

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Jeyhun, thanks for updating the KIP. I am wondering if you intend to add a new class `Produced`? There is already `org.apache.kafka.streams.kstream.Produced`. So if we want to add a new class, it must have a different name -- or we might be able to merge both into one? Also, for the KStream

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi, Thanks for your comments. I agree with Matthias partially. I think we should relax some requirements related with to() and through() methods. IMHO, Produced class can cover (existing/to be created) topic information, and which will ease our effort: KStream.to(Produced topicInfo)

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

@Jan: The `Produced` class was introduced in 1.0 to specify key and valud Serdes (and partitioner) if data is written into a topic. Old API: KStream#to("topic", keySerde, valueSerde); New API: KStream#to("topic", Produced.with(keySerde, valueSerde)); This allows to reduce the number of

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi Gouzhang, this felt like these questions are supposed to be answered by me. I do not understand the first one. I don't understand why the user shouldn't be able to specify a suffix for the topic name. For the third question I am not 100% familiar if the Produced class came to existence at

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

A few clarification questions on the proposal details. 1. API: although the repartition only happens at the final stateful operations like agg / join, the repartition flag info was actually passed from an earlier operator like map / groupBy. So what should be the new API look like? For example,

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

bq. enlarge the score of through() I guess you meant scope. On Mon, Nov 6, 2017 at 1:15 PM, Jeyhun Karimov wrote: > Hi, > > Sorry for the late reply. I am convinced that we should enlarge the score > of through() (add more overloads) instead of introducing a separate set

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi, Sorry for the late reply. I am convinced that we should enlarge the score of through() (add more overloads) instead of introducing a separate set of overloads to other methods. I will update the KIP soon based on the discussion and inform. Cheers, Jeyhun On Mon, Nov 6, 2017 at 9:18 PM Jan

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Sorry for not beeing 100% up to date. Back then we had the discussion that when an operation puts a >Sink< into the topology, a >Produced< parameter is added. This produced parameter could have internal or external. If internal I think the name would still make a great suffix for the topic

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

I absolute agree with what you say. It's not a requirement to specify a topic name -- and this was the idea -- if user does specify a name, we treat as is -- if users does not specify a name, Streams create an internal topic. The goal of the Jira is to allow a simplified way to control

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Could you clarify exactly what you mean by keeping the current distinction? Actually, re-reading the KIP and JIRA, it's not clear that being able to specify a custom name is actually a requirement. If the goal is to control repartitioning and tune parallelism, maybe we can just sidestep this

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

That's a good point. In current design, we strictly distinguish both. For example, the reset tools deletes internal topics (starting with prefix `-` and ending with either `-repartition` or `-changelog`. Thus, from my point of view, it would make sense to keep the current distinction. -Matthias

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

I think this sounds good as well. It's worth clarifying whether topics that are named by the user but created by streams are considered "internal" topics also. On Sun, 2017-11-05 at 23:02 +0100, Matthias J. Sax wrote: My idea was, to relax the requirement for through() that a topic must be

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

That would be nice I think On 05.11.2017 23:02, Matthias J. Sax wrote: My idea was, to relax the requirement for through() that a topic must be created manually before startup. Thus, if no through() call is made, a (internal) topic is created the same way we do it currently. If one uses

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

My idea was, to relax the requirement for through() that a topic must be created manually before startup. Thus, if no through() call is made, a (internal) topic is created the same way we do it currently. If one uses `through(String topicName)` we keep the current behavior and require users to

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi. Im not 100 % up to date what version 1.0 DSL looks like ATM. I just would argue that repartitioning should be an own API call like through or something. One can use through or to already to get this. I would argue one should look there instead of overloads Best Jan On 04.11.2017 16:01,

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Hi, Thanks for your comments. @Ted API is given without much javadoc on the role / meaning of method > parameters. - I thought they are self-explanatory but I will add some more comments in the document. @Matthias - how does this relate to `KStream#through()` ? - The main difference

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

w.r.t. KIP-182, KAFKA-5651 has been resolved. But the KIP is still labeled Under Discussion. Should KIP-182 be moved to Adopted state ? On Sat, Nov 4, 2017 at 10:06 AM, Matthias J. Sax wrote: > Thanks for the KIP. > > Two comments/questions: > > - how does this relate

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

Thanks for the KIP. Two comments/questions: - how does this relate to `KStream#through()` ? - do we really need to add new overload or can we just extends exiting options (cf KIP-182); it would be create if we could just extend existing classes like `Produced` instead of adding new methods to

Re: [DISCUSS] KIP-221: Repartition Topic Hints in Streams

API is given without much javadoc on the role / meaning of method parameters. Can you enrich with descriptive javadoc ? On Sat, Nov 4, 2017 at 8:01 AM, Jeyhun Karimov wrote: > Dear community, > > I would like to initiate discussion on KIP-221 [1] based on issue [2]. >

[DISCUSS] KIP-221: Repartition Topic Hints in Streams

Dear community, I would like to initiate discussion on KIP-221 [1] based on issue [2]. Please feel free to comment. [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Repartition+Topic+Hints+in+Streams [2] https://issues.apache.org/jira/browse/KAFKA-6037 Cheers, Jeyhun