[
https://issues.apache.org/jira/browse/KAFKA-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381457#comment-16381457
]
Matthias J. Sax commented on KAFKA-4936:
----------------------------------------
Thanks for you interest in picking this up.
If we integrate this in the library, I don't think that we need an additional
producer, but can extend the existing `SinkNode` implementation. This would
avoid the requirement of sync flushs and also work with EOS enabled.
The SinkNode could accept a "topic extractor" function that computes the topic
for each record and would not be configured with a single topic name. That's at
least my thinking. Can you share your ideas how to implement this?
I am still wondering about the existence of output topics (ie, should we
require this?) vs auto-topic creation? It should be clear to the user what the
impact requirements/impact is with this regard. Maybe I am overthinking this,
but if auto-topic create is enabled, and something goes wrong (ie, the "topic
name extractor" return the wrong field from an AVRO record), users might end up
with many topics that get created on error what could result in a quite messed
up cluster – I am a little concerned about this. Even with small input rates
(let's say 1000 records per second) on could create 1000 different topic per
second...
> Allow dynamic routing of output records
> ---------------------------------------
>
> Key: KAFKA-4936
> URL: https://issues.apache.org/jira/browse/KAFKA-4936
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Matthias J. Sax
> Priority: Major
> Labels: needs-kip
>
> Currently, all used output topics must be know beforehand, and thus, it's not
> possible to send output records to topic in a dynamic fashion.
> There have been couple of request for this feature and we should consider
> adding it. There are many open questions however, with regard to topic
> creation and configuration (replication factor, number of partitions) etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
