[
https://issues.apache.org/jira/browse/SPARK-45889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-45889:
---------------------------------
Summary: Implement push-down filter with partition ID and grouping key (if
possible) for state data source reader (was: Implement push-down filter with
partition ID and grouping key (if possible))
> Implement push-down filter with partition ID and grouping key (if possible)
> for state data source reader
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-45889
> URL: https://issues.apache.org/jira/browse/SPARK-45889
> Project: Spark
> Issue Type: Task
> Components: Structured Streaming
> Affects Versions: 4.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> If the query filters the state data via partition ID, it is a good chance for
> state data source to avoid spinning all state store instances and wasting
> resource. We can spin state store instances for only necessary partitions.
> Same thing applies to grouping keys, although the criteria on distribution is
> bound to the operator rather than the key in state store, hence it could be
> very tricky unless we can follow the same criteria on distribution for the
> operator.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
