L. C. Hsieh created SPARK-34297:
-----------------------------------
Summary: Add metrics for data loss and offset out range for
KafkaMicroBatchStream
Key: SPARK-34297
URL: https://issues.apache.org/jira/browse/SPARK-34297
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.2.0
Reporter: L. C. Hsieh
Assignee: L. C. Hsieh
When testing SS, I found it is hard to track data loss of SS reading from
Kafka. The micro scan node has only one metric, number of output rows. Users
have no idea how many times offsets to fetch are out of Kafak now, how many
times data loss happens.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
