[ 
https://issues.apache.org/jira/browse/KAFKA-13500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17456955#comment-17456955
 ] 

Matthias J. Sax commented on KAFKA-13500:
-----------------------------------------

Not 100% sure: for individual configs, maybe. – But it won't solve the 
monitoring issue. Users might still see that the "restore consumer" has a 
non-zero lag if they monitor the consumer lag metric if there a lagging 
standbys and we are not actually restoring state for active tasks.

I marked this ticket as "minor", so if we think it's not worth it (for now?) 
for with me. But I don't think that our code would become (much) more 
complicated and thus, it might be an easy win... (the main impact might be that 
we would need to add a new `standby.producer` config prefix... so we might 
technically need a KIP)

Personally still in favor to add a new consumer, but I leave it up to you. 
Also, we can always make this change later on, too.

> Consider adding a dedicated standby consumer
> --------------------------------------------
>
>                 Key: KAFKA-13500
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13500
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Priority: Minor
>
> We currently use the restore consumer to recover state for active tasks and 
> to maintain standby tasks during regular processing. This setup has a few 
> disadvantages
>  # During state recovery, we might want to apply different consumer configs 
> compared to standby maintenance during regular processing.
>  # It make monitoring confusing: because we never commit offsets for 
> changelog topics, users can only monitor the client's "lag metric" to 
> observer restore progress (without the need to register a restore listener). 
> However, if they are interesting in a restore metric, during regular 
> processing it would report the standby lag, which can be rather confusing.
> Because the restore consumer does not use consumer group management, it seems 
> to be low overhead to actually use a third consumer, because there won't be 
> any heartbeat thread.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Reply via email to