[
https://issues.apache.org/jira/browse/KAFKA-7190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16556011#comment-16556011
]
Dong Lin commented on KAFKA-7190:
---------------------------------
[~guozhang] Certainly, I am happy to make the modification after we agree on
the solution.
Here is my thought regarding whether we should allow client to delete beyond
last stable offset. The main reason we need to prevent client from deleting
beyond HW is because otherwise follower may receive OutOfRangeException and the
broker logic may be messed up. Is there similar concern in the broker if we
delete beyond LSO?
If the goal of not deleting beyond LSO is to make sure that messages can be
exposed to consumers before being deleted, then I am not sure if this change is
justified. My understanding is that we currently rely on user to make sure that
messages have been consumed by all consumer groups which need the message,
before user calls deleteRecords() to delete the message. If user does follow
this rule, then user will not delete beyond LSO regardless of the constraint in
the broker. If user does not follow this rule, then even with the extra
protection in the broker, user can still delete the messages before the LSO
that has not been consumed yet. Does this make sense?
> Under low traffic conditions purging repartition topics cause WARN statements
> about UNKNOWN_PRODUCER_ID
> ---------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-7190
> URL: https://issues.apache.org/jira/browse/KAFKA-7190
> Project: Kafka
> Issue Type: Improvement
> Components: core, streams
> Affects Versions: 1.1.0, 1.1.1
> Reporter: Bill Bejeck
> Assignee: lambdaliu
> Priority: Major
>
> When a streams application has little traffic, then it is possible that
> consumer purging would delete
> even the last message sent by a producer (i.e., all the messages sent by
> this producer have been consumed and committed), and as a result, the broker
> would delete that producer's ID. The next time when this producer tries to
> send, it will get this UNKNOWN_PRODUCER_ID error code, but in this case,
> this error is retriable: the producer would just get a new producer id and
> retries, and then this time it will succeed.
>
> Possible fixes could be on the broker side, i.e., delaying the deletion of
> the produderIDs for a more extended period or on the streams side developing
> a more conservative approach to deleting offsets from repartition topics
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
