Hello Nick,
Thanks for the patient explanations. I think I'm convinced that we should
not exclude repartitioning inside the recursive operator indeed, and in
fact we cannot programmingly forbid it by the UnaryOperator itself anyways.
Just a few nits on the page itself:
1) could you still
The more I think about this, the more I think that automatic repartitioning
is not required in the "recursively" method itself. I've removed references
to this from the KIP, which further simplifies everything.
I don't see any need to restrict users from repartitioning, either before,
after or
Hi Guozhang,
I mentioned this in the "Rejected Alternatives" section. Repartitioning
gives us several significant advantages over using an explicit topic and
"to":
- Repartition topics are automatically created and managed by the
Streams runtime; explicit topics have to be created and
Hello Nick,
Thanks for the re-written KIP! I read through it again, and so far have
just one quick question on top of my head regarding repartitioning: it
seems to me that when there's an intermediate topic inside the recursion
step, then using this new API would basically give us the same
Hi everyone,
I've re-written the KIP, with a new design that I think resolves the issues
you highlighted, and also simplifies usage.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-857%3A+Streaming+recursion+in+Kafka+Streams
Note: I'm still working out the "automatic repartitioning" in my
Hi Sophie,
The reason I chose to add a new overload of "to", instead of creating a new
method, is simply because I felt that "to" was about sending records "to"
somewhere, and that "somewhere" just happens to currently be exclusively
topics. By re-using "to", we can send records *to other
Hey Nick,
Sounds like an interesting KIP, and I agree the current way of achieving
this in Streams
seems wildly overcomplicated. So I'm definitely +1 on adding a smooth API
that abstracts
away a lot of the complexity and unnecessary topic management.
That said, I've found much of the discussion
Hello Nick,
Thanks for the replies! They are very thoughtful. I think I agree with you
that requiring the output stream to a source stream is not sufficient for a
valid recursion, and even without the proposed API users today can still
create a broken recursive topology.
Just want to clarify
Hi everyone,
On Guozhang's point 1): I actually considered a "recursion" API, something
like you suggested, however it won't work, because to do the recursion you
need to know both the end of the KStream that you want to recurse, AND the
beginning of the stream you want to feed it back into. Your
Hello Nick,
Thanks for bringing this KIP! Just a few thoughts:
1) I agree with Sagar that, we'd probably think about two routes to
rephrase / restructure the proposal:
* we can propose a couple of new APIs, and just list "more
convenient recursion" as one of its benefits. Then we'd need to be
Hey Nick,
Since we are adding a new method to the public interface, we should
probably decide the necessity of doing so, more so when you say that it's
an alternative to something already existing. My suggestion would be to
still modify the KIP around the new API, highlight how it's an
Hi Sagar,
Thanks for reading through my proposal.
While the 2 new methods were originally intended for the recursive
use-case, they could also be used as an alternative means of wiring two
different KStreams together. The main reason I didn't document this in the
KIP is that using the API for
Hey Nick,
Thanks for the KIP. This seems like a great addition. However, just
wondering if the 2 new methods that you plan to add are meant only for
streaming recursion? I would imagine they could be repurposed for other use
cases as well? If yes, then probably the KIP should revolve around the
Hi everyone,
URL:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-857%3A+Streaming+recursion+in+Kafka+Streams
Here's a KIP for extending the Streams DSL API to support "streaming
recursion". See the Motivation section for details on what I mean by this,
along with an example of recursively
14 matches
Mail list logo
