L. C. Hsieh created SPARK-39134:
-----------------------------------
Summary: Add custom metric of skipped null values for stream join
operator
Key: SPARK-39134
URL: https://issues.apache.org/jira/browse/SPARK-39134
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.4.0
Reporter: L. C. Hsieh
When STATE_STORE_SKIP_NULLS_FOR_STREAM_STREAM_JOINS is enabled, Spark will skip
null values in hash-based stream-stream joins. But currently we cannot track if
this is happened and how many it is. For our customers, they have an interest
to know the number of values skipped for better tracking purpose.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
