Jungtaek Lim created SPARK-45889:
------------------------------------
Summary: Implement push-down filter with partition ID and grouping
key (if possible)
Key: SPARK-45889
URL: https://issues.apache.org/jira/browse/SPARK-45889
Project: Spark
Issue Type: Task
Components: Structured Streaming
Affects Versions: 4.0.0
Reporter: Jungtaek Lim
If the query filters the state data via partition ID, it is a good chance for
state data source to avoid spinning all state store instances and wasting
resource. We can spin state store instances for only necessary partitions.
Same thing applies to grouping keys, although the criteria on distribution is
bound to the operator rather than the key in state store, hence it could be
very tricky unless we can follow the same criteria on distribution for the
operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
