[
https://issues.apache.org/jira/browse/KAFKA-19781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianet Magrans updated KAFKA-19781:
-----------------------------------
Description:
Currently, the consumer attempts to update positions for all partitions it owns
that don't have a position, even when the partition may be already being
revoked. This could lead to NoOffsetForPartitionException for such partitions
(if there is no reset strategy defined). There are 2 main issues around this:
* is probably unneeded work to fetch offsets and update positions for
partitions being revoked. At that point we're actually not allowing fetching
from there anymore
* throwing NoOffsetForPartitionException may lead applications to take action
to set positions themselves, which will most probably fail (the partition will
most probably not owned by the consumer anymore since it was already being
revoked when the NoOffsetForPartitionException was generated). We've seen this
on Kafka Streams, that handled NoOffsetForPartitionException by calling seek to
set positions.
This task is to review if there are no other implications I may be missing?
Then fix to ensure we don't update positions for partitions being revoked
(aligned with how we don't allow fetching from them)
was:
Currently, the consumer attempts to update positions for all partitions it owns
that don't have a position, even when the partition may be already being
revoked. This could lead to NoOffsetForPartitionException for such partitions
(if there is no reset strategy defined). There are 2 main issues around this:
* is probably unneeded work to fetch offsets and update positions for
partitions being revoked, and at that point we're actually not allowing
fetching from there anymore
* throwing NoOffsetForPartitionException may lead applications to take action
to set positions themselves, which will most probably fail (the partition will
most probably not owned by the consumer anymore since it was already being
revoked when the NoOffsetForPartitionException was generated). We've seen this
on Kafka Streams, that handled NoOffsetForPartitionException by calling seek to
set positions.
This task is to review if there are no other implications I may be missing?
Then fix to ensure we don't update positions for partitions being revoked
(aligned with how we don't allow fetching from them)
> Consumer NoOffsetForPartitionException for partitions being revoked
> -------------------------------------------------------------------
>
> Key: KAFKA-19781
> URL: https://issues.apache.org/jira/browse/KAFKA-19781
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
> Reporter: Lianet Magrans
> Assignee: Lianet Magrans
> Priority: Major
> Fix For: 4.2.0
>
>
> Currently, the consumer attempts to update positions for all partitions it
> owns that don't have a position, even when the partition may be already being
> revoked. This could lead to NoOffsetForPartitionException for such partitions
> (if there is no reset strategy defined). There are 2 main issues around this:
> * is probably unneeded work to fetch offsets and update positions for
> partitions being revoked. At that point we're actually not allowing fetching
> from there anymore
> * throwing NoOffsetForPartitionException may lead applications to take
> action to set positions themselves, which will most probably fail (the
> partition will most probably not owned by the consumer anymore since it was
> already being revoked when the NoOffsetForPartitionException was generated).
> We've seen this on Kafka Streams, that handled NoOffsetForPartitionException
> by calling seek to set positions.
> This task is to review if there are no other implications I may be missing?
> Then fix to ensure we don't update positions for partitions being revoked
> (aligned with how we don't allow fetching from them)
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
