Chaoqin Li created SPARK-45794:
----------------------------------
Summary: Introduce state metadata source to query the streaming
state metadata information
Key: SPARK-45794
URL: https://issues.apache.org/jira/browse/SPARK-45794
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.5.0
Reporter: Chaoqin Li
Introduce a new data source so that user can query the metadata of each state
store of a streaming query, the schema of the result will be
operatorId INT | operatorName STRING | stateStoreName STRING | numPartitions
INT | numColsPrefixKey INT | minBatchId LONG | minBatchId LONG
To use this source, specify the source format and checkpoint path
_df = spark.read.format(“state-metadata”).load(“/checkpointPath”)_
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
