[
https://issues.apache.org/jira/browse/SPARK-47733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhan updated SPARK-47733:
------------------------------
Description:
Add metrics to improve observability for the newly added operator
TransformWithState and some changes we've made into RocksDB.
Proposed metrics to add:
* on the RocksDB StateStore metrics side, we will add the following:
** num external col families
** num internal col families
* on the operator side, we will add the following:
** number of state vars
** count of state vars by type
** output mode
** timeout mode
** registered timers in batch
** expired timers in batch
** initial state enabled or not
** number of state vars removed in batch
> Add operational metrics for TWS operators
> -----------------------------------------
>
> Key: SPARK-47733
> URL: https://issues.apache.org/jira/browse/SPARK-47733
> Project: Spark
> Issue Type: Task
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Jing Zhan
> Priority: Major
> Labels: pull-request-available
>
> Add metrics to improve observability for the newly added operator
> TransformWithState and some changes we've made into RocksDB.
> Proposed metrics to add:
> * on the RocksDB StateStore metrics side, we will add the following:
> ** num external col families
> ** num internal col families
> * on the operator side, we will add the following:
> ** number of state vars
> ** count of state vars by type
> ** output mode
> ** timeout mode
> ** registered timers in batch
> ** expired timers in batch
> ** initial state enabled or not
> ** number of state vars removed in batch
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
