[jira] [Comment Edited] (KAFKA-5386) [Kafka Streams] - custom name for state-store change-log topic
[
https://issues.apache.org/jira/browse/KAFKA-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099058#comment-16099058
]
Bart Vercammen edited comment on KAFKA-5386 at 7/25/17 7:03 AM:
[~mjsax] {{--changelog}} that's exactly the problem
in our case ;-)
We have some strict rules about the naming of kafka topics in our project and
would like to have full control over them, including the ACLs on each
kafka-topic.
So basically we do not want applications to start creating whatever topics all
over the place.
This being said, {{--changelog}} would work if we
could add some wildcard ACLs on the creation of Kafka topics, but that would
imply writing an own authorizer in Kafka allowing only the creation of topics
that comply to a specific naming schema.
Changing the default {{--changelog}} schema into
something more configurable would be easier for KafkaStreams applications.
The (dirty) patch I made was simply to allow {{ProcessorStateManager}} to
accept a {{static}} _prefix_ and _suffix_ configuration and just replace the
{{storeChangelogTopic}} function to return {{..}}
when they are defined. This would of course not work for the dynamically
created repartitioning topics, but that could then be solved with your tip to
use {{through("...")}} ...
Let me see how far I get to make this cleaner and more configurable without
impacting the public API too much ... ;-)
was (Author: cloutrix):
[~mjsax] {{--changelog}} that's exactly the problem
in our case ;-)
We have some strict rules about the naming of kafka topics in our project and
would like to have full control over them, including the ACLs on each
kafka-topic.
So basically we do not want applications to start creating whatever topics all
over the place.
This being said, {{--changelog}} would work if we
could add some wildcard ACLs on the creation of Kafka topics, but that would
imply writing an own authorizer in Kafka allowing only the creation of topics
that comply to a specific naming schema.
Changing the default {{--changelog}} schema into
something more configurable would be easier for KafkaStreams applications.
The (dirty) patch I made was simply to allow {{ProcessorStateManager}} to
accept a {{static}} _prefix_ and _suffix_ configuration and just replace the
{{storeChangelogTopic}} function to return {{..}}
when they are defined. This would of course not work for the dynamically
created repartitioning topics, but that could then be solved with your tip to
use {{through("...")}} ...
Let me see how far I get to make this cleaner and more configurable without
impacting the public API to much ... ;-)
> [Kafka Streams] - custom name for state-store change-log topic
> --
>
> Key: KAFKA-5386
> URL: https://issues.apache.org/jira/browse/KAFKA-5386
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Bart Vercammen
> Labels: needs-kip
>
> Currently, when working with Kafka backed state stores in Kafka Streams,
> these log compacted topics are given a hardcoded name :
> _my.app.id-storename-changelog_
> {noformat}public static String storeChangelogTopic(String applicationId,
> String storeName) {
> return applicationId + "-" + storeName + STATE_CHANGELOG_TOPIC_SUFFIX;
> }{noformat}
> It would be nice if somehow I would be able to override this functionality
> and provide the topic-name myself when creating the state-store.
> Any comments?
> Would it be OK to submit a PR for this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5386) [Kafka Streams] - custom name for state-store change-log topic
[
https://issues.apache.org/jira/browse/KAFKA-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098591#comment-16098591
]
Matthias J. Sax edited comment on KAFKA-5386 at 7/24/17 3:39 PM:
-
Well. With regard to ACL, you can still know the names of the changelog topics:
They follow the pattern {{--changelog}} -- thus, as
long as you specify a store name for each `builder.table()` and
count/reduce/aggregate and joins, you would know the changelog topic names and
could adjust the ACL accordingly. (only if you omit a store name, Streams
generates one).
ATM, this feature request does not seem to be high priority. It always depends
how many people ask for it. Of course, we are more than happy if anybody picks
this up :) I guess, we would need a KIP though as this change impacts the
public API.
was (Author: mjsax):
Well. With regard to ACL, you can still know the names of the changelog topics:
They follow the pattern `--changelog` -- thus, as
long as you specify a store name for each `builder.table()` and
count/reduce/aggregate and joins, you would know the changelog topic names and
could adjust the ACL accordingly. (only if you omit a store name, Streams
generates one).
ATM, this feature request does not seem to be high priority. It always depends
how many people ask for it. Of course, we are more than happy if anybody picks
this up :) I guess, we would need a KIP though as this change impacts the
public API.
> [Kafka Streams] - custom name for state-store change-log topic
> --
>
> Key: KAFKA-5386
> URL: https://issues.apache.org/jira/browse/KAFKA-5386
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Bart Vercammen
> Labels: needs-kip
>
> Currently, when working with Kafka backed state stores in Kafka Streams,
> these log compacted topics are given a hardcoded name :
> _my.app.id-storename-changelog_
> {noformat}public static String storeChangelogTopic(String applicationId,
> String storeName) {
> return applicationId + "-" + storeName + STATE_CHANGELOG_TOPIC_SUFFIX;
> }{noformat}
> It would be nice if somehow I would be able to override this functionality
> and provide the topic-name myself when creating the state-store.
> Any comments?
> Would it be OK to submit a PR for this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5386) [Kafka Streams] - custom name for state-store change-log topic
[
https://issues.apache.org/jira/browse/KAFKA-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098554#comment-16098554
]
Bart Vercammen edited comment on KAFKA-5386 at 7/24/17 3:18 PM:
[~mjsax] Thanks for the reply. The {{through("...")}} would already help me a
little bit, thanks for the tip.
With respect to the changelog topics, will this be included in upcoming
versions of KafkaStreams?
It actually does make sense to be able to define the topic names myself,
especially with Kafka ACLs in play.
It would be nice to be able to configure the changelog topic _prefix_ and
_suffix_ somehow.
I have a local fork where I implemented this, it's a bit dirty and purely as a
proof of concept, but would it make sense to rework this into a valid PR?
was (Author: cloutrix):
[~mjsax] Thanks for the reply. The `through("...")` would already help me a
little bit, thanks for the tip.
With respect to the changelog topics, will this be included in upcoming
versions of KafkaStreams?
It actually does make sense to be able to define the topic names myself,
especially with Kafka ACLs in play.
It would be nice to be able to configure the changelog topic _prefix_ and
_suffix_ somehow.
I have a local fork where I implemented this, it's a bit dirty and purely as a
proof of concept, but would it make sense to rework this into a valid PR?
> [Kafka Streams] - custom name for state-store change-log topic
> --
>
> Key: KAFKA-5386
> URL: https://issues.apache.org/jira/browse/KAFKA-5386
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Bart Vercammen
>
> Currently, when working with Kafka backed state stores in Kafka Streams,
> these log compacted topics are given a hardcoded name :
> _my.app.id-storename-changelog_
> {noformat}public static String storeChangelogTopic(String applicationId,
> String storeName) {
> return applicationId + "-" + storeName + STATE_CHANGELOG_TOPIC_SUFFIX;
> }{noformat}
> It would be nice if somehow I would be able to override this functionality
> and provide the topic-name myself when creating the state-store.
> Any comments?
> Would it be OK to submit a PR for this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KAFKA-5386) [Kafka Streams] - custom name for state-store change-log topic
[
https://issues.apache.org/jira/browse/KAFKA-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091463#comment-16091463
]
Bart Vercammen edited comment on KAFKA-5386 at 7/18/17 12:01 PM:
-
Actually this is going to be a little harder than initially expected, as the
KStream API can create a bunch of _repartition_ topics under the hood, that I
also wish to comply to a certain known naming convention.
Basically what I'm trying to achieve is that every kafka-topic that is going to
be created is known up front, and that it complies to a certain naming
strategy. It would be nice if this naming strategy could be configured somehow.
Any suggestions on this?
was (Author: cloutrix):
Actually this is going to be a little harder than initially expected, as the
KStreams API can create a bunch of _repartition_ topics under the hood, that I
also wish to comply to a certain known naming convention.
Basically what I'm trying to achieve is that every kafka-topic that is going to
be created is known up front, and that it complies to a certain naming
strategy. It would be nice if this naming strategy could be configured somehow.
Any suggestions on this?
> [Kafka Streams] - custom name for state-store change-log topic
> --
>
> Key: KAFKA-5386
> URL: https://issues.apache.org/jira/browse/KAFKA-5386
> Project: Kafka
> Issue Type: Improvement
> Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Bart Vercammen
>
> Currently, when working with Kafka backed state stores in Kafka Streams,
> these log compacted topics are given a hardcoded name :
> _my.app.id-storename-changelog_
> {noformat}public static String storeChangelogTopic(String applicationId,
> String storeName) {
> return applicationId + "-" + storeName + STATE_CHANGELOG_TOPIC_SUFFIX;
> }{noformat}
> It would be nice if somehow I would be able to override this functionality
> and provide the topic-name myself when creating the state-store.
> Any comments?
> Would it be OK to submit a PR for this?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
