[
https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16555990#comment-16555990
]
Guozhang Wang commented on KAFKA-7190:
--------------------------------------
I've discussed about this with [~hachikuji] and here are our current proposal:
1. Currently the PID will be removed from the broker's cache once its last
produced message is truncated (details of the transactional messaging design
can be found in KIP-98). We will remove this logic, and only rely on the
broker-side config "transactional.id.expiration.ms".
2. Note that the config "transactional.id.expiration.ms" is used for multiple
purposes, and its default value is 7 days which would be too long if we are
going to do 1) above and hence are going to rely ONLY on it to cleanup PID
caches. So we'd probably want to add a separate config for PID expiration only
whose default value would be much smaller, say 1 hour.
[~lambdaliu] I understand that the above two requires some knowledge on the
broker-side transactional messaging feature implementations, and please let me
know if you feel comfortable working on it.
------------------------------------
Another caveat we need to fix which is related to this issue, is that today
DeleteRecord request would only try to avoid deleting beyond the
high-watermark, but with transactional messaging we should also make sure that
deleting would not go beyond LSO (latest stable offset) to make sure that no
uncommitted data would be deleted as they are not exposed to consumers yet.
[~lindong] since you added the delete record request, would like to know if you
are willing to make this modifications as well.
> Under low traffic conditions purging repartition topics cause WARN statements
> about UNKNOWN_PRODUCER_ID
> ---------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-7190
> URL: https://issues.apache.org/jira/browse/KAFKA-7190
> Project: Kafka
> Issue Type: Improvement
> Components: core, streams
> Affects Versions: 1.1.0, 1.1.1
> Reporter: Bill Bejeck
> Assignee: lambdaliu
> Priority: Major
>
> When a streams application has little traffic, then it is possible that
> consumer purging would delete
> even the last message sent by a producer (i.e., all the messages sent by
> this producer have been consumed and committed), and as a result, the broker
> would delete that producer's ID. The next time when this producer tries to
> send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case,
> this error is retriable: the producer would just get a new producer id and
> retries, and then this time it will succeed.
>
> Possible fixes could be on the broker side, i.e., delaying the deletion of
> the produderIDs for a more extended period or on the streams side developing
> a more conservative approach to deleting offsets from repartition topics
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
