[
https://issues.apache.org/jira/browse/KAFKA-21104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianet Magrans updated KAFKA-21104:
-----------------------------------
Description:
The onPartitionsAssigned callback runs after the new assignment has been fully
applied on the client (both Classic and Consumer protocols), and with the
current behaviour it's just a notification triggered only once with the added
partitions, letting the app know that an assignment was applied.
If the onPartitionsAssigned callback fails, the error is propagated to the app
on the consumer.poll, but it's never retried with those partitions (retried
with empty assignment). Internally, the partitions are successfully assigned to
the consumer, but remain unfetchable due to the callback failure. Once the
callback is called again with empty assignment, the partitions are not blocked
anymore waiting for the callback (all, including the ones that intiailly
failed), but since the callback is never called again with those partitions
that failed, the logic inside it may not be retried for them (e.g, callback
setting positions for the added partitions...this wouldn't be retried)
This would be a behaviour change for both consumer, so this task is only to
think about potential improvements here. Note that the onPartitionsRevoked
behaves differently, it is retried on every poll if it fails, receiving the
same partitions as parameters (the consumer remains rebalancing/reconciling, so
it's all triggered again on a next poll)
was:
The onPartitionsAssigned callback runs after the new assignment has been fully
applied on the client (both Classic and Consumer protocols), and with the
current behaviour it's just a notification triggered only once with the added
partitions, letting the app know that an assignment was applied.
If the onPartitionsAssigned callback fails, the error is propagated to the app
on the consumer.poll, but it's never retried with those partitions (retried
with empty assignment). Internally, the partitions are successfully assigned to
the consumer, but remain unfetchable due to the callback failure.
This would be a behaviour change for both consumer, so this task is only to
think about potential improvements here. Note that the onPartitionsRevoked
behaves differently, it is retried on every poll if it fails, receiving the
same partitions as parameters (the consumer remains rebalancing/reconciling, so
it's all triggered again on a next poll)
> Consider supporting retries on onPartitionsAssigned callback on failure
> -----------------------------------------------------------------------
>
> Key: KAFKA-21104
> URL: https://issues.apache.org/jira/browse/KAFKA-21104
> Project: Kafka
> Issue Type: Improvement
> Components: clients, consumer
> Reporter: Lianet Magrans
> Assignee: Lianet Magrans
> Priority: Major
>
> The onPartitionsAssigned callback runs after the new assignment has been
> fully applied on the client (both Classic and Consumer protocols), and with
> the current behaviour it's just a notification triggered only once with the
> added partitions, letting the app know that an assignment was applied.
> If the onPartitionsAssigned callback fails, the error is propagated to the
> app on the consumer.poll, but it's never retried with those partitions
> (retried with empty assignment). Internally, the partitions are successfully
> assigned to the consumer, but remain unfetchable due to the callback failure.
> Once the callback is called again with empty assignment, the partitions are
> not blocked anymore waiting for the callback (all, including the ones that
> intiailly failed), but since the callback is never called again with those
> partitions that failed, the logic inside it may not be retried for them (e.g,
> callback setting positions for the added partitions...this wouldn't be
> retried)
> This would be a behaviour change for both consumer, so this task is only to
> think about potential improvements here. Note that the onPartitionsRevoked
> behaves differently, it is retried on every poll if it fails, receiving the
> same partitions as parameters (the consumer remains rebalancing/reconciling,
> so it's all triggered again on a next poll)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
