[
https://issues.apache.org/jira/browse/KAFKA-14069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566695#comment-17566695
]
Emmanuel Brard commented on KAFKA-14069:
----------------------------------------
One thing I did not mention yet is that we use version 2.8.0. I guess those API
calls are triggered by the streaming application itself, should we run it in
debug log mode to see the calls?
> Allow custom configuration of foreign key join internal topics
> --------------------------------------------------------------
>
> Key: KAFKA-14069
> URL: https://issues.apache.org/jira/browse/KAFKA-14069
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 2.8.0
> Reporter: Emmanuel Brard
> Priority: Minor
>
> Internal topic supporting foreign key joins (-subscription-registration-topic
> and -subscription-response-topic) are automatically created with_ infinite
> retention_ (retention.ms=-1, retention.bytes=-1).
> As far as I understand those topics are used for communication between tasks
> that are involved in the FK, the intermediate result though is persisted in a
> compacted topic (-subscription-store-changelog).
> This means, if I understood right, that during normal operation of the stream
> application, once a message is read from the registration/subscription topic,
> it will not be read again, even in case of recovery (the position in those
> topics is committed).
> Because we have very large tables being joined this way with very high
> changes frequency, we end up with FK internal topics in the order of 1 or 2
> TB. This is complicated to maintain especially in term of disk space.
> I was wondering if:
> - this infinite retention is really a required configuration and if not
> - this infinite retention could be replaced with a configurable one (for
> example of 1 week, meaning that I accept that in case of failure I must this
> my app within one week)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
