[
https://issues.apache.org/jira/browse/SPARK-33833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258726#comment-17258726
]
L. C. Hsieh commented on SPARK-33833:
-------------------------------------
Yea, but this can be easily overcome here. We just need to have a user-provided
group id for committing offset purpose. As users need to specify it when they
want to commit offset and track the progress, this is used by users with
caution. Even for committing with currently static group ID given by users, I
do not think it is really a reason to reject the committing offset idea. Once
users decide to commit offset and track the progress, they should be cautious
with the risk.
Anyway, this seems not the reason causing the previous PR to be closed.
> Allow Spark Structured Streaming report Kafka Lag through Burrow
> ----------------------------------------------------------------
>
> Key: SPARK-33833
> URL: https://issues.apache.org/jira/browse/SPARK-33833
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.0.1
> Reporter: Sam Davarnia
> Priority: Major
>
> Because structured streaming tracks Kafka offset consumption by itself,
> It is not possible to track total Kafka lag using Burrow similar to DStreams
> We have used Stream hooks as mentioned
> [here|https://medium.com/@ronbarabash/how-to-measure-consumer-lag-in-spark-structured-streaming-6c3645e45a37]
>
> It would be great if Spark supports this feature out of the box.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
