[
https://issues.apache.org/jira/browse/KAFKA-7397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16612785#comment-16612785
]
John Roesler commented on KAFKA-7397:
-------------------------------------
Hi [~frederic.tardif],
Thanks for the feature request, and also for the workaround code.
Just to put this out there, are you interested in writing a KIP and
implementing the feature? If so, I can help guide you through the process.
Thanks,
-John
> Ability to apply DSL stateless transformation on a global table
> ---------------------------------------------------------------
>
> Key: KAFKA-7397
> URL: https://issues.apache.org/jira/browse/KAFKA-7397
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Frederic Tardif
> Priority: Major
> Labels: needs-kip
> Attachments: kafka.zip
>
>
> When consuming a globalKTable (with the expectation of caching all the data
> of a topic in a consumer store), we can't apply any stateless transformation
> (filter, map), prior to materializing. To achieve this, while ensuring to
> consume the records of all the partitions, we must first run a stream app
> that does preprocessing on the ingress topic into an exact K1,V1 egress topic
> as we want to store in our GlobalKTable. This looks unnecessarily complex,
> and causes to double the storage of the topic, while the only goal is to
> adapt statelessly the data prior to storing (rockDB) at the receiving end.
> See discussion on
> :[https://stackoverflow.com/questions/50993292/kafka-streams-shared-changelog-topic]
> As a workaround, I have used `new Builder().addGlobalStore(....)` with a
> Custom Processor able to filter and map prior to store (see attached).
> Although this seem to work, I believe this functionality should be part of
> the basic dsl api when working with a globalTable (`new
> StreamsBuilder().globalTable().filter(...).map()... `).
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
