[
https://issues.apache.org/jira/browse/SPARK-60693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60693:
-----------------------------------
Labels: pull-request-available (was: )
> Add a task-scoped lookup cache to external lookup join
> ------------------------------------------------------
>
> Key: SPARK-60693
> URL: https://issues.apache.org/jira/browse/SPARK-60693
> Project: Spark
> Issue Type: Task
> Components: SQL, Structured Streaming
> Affects Versions: 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> Part of SPARK-60031.
> In scope:
> - A per-task LRU cache of complete results, including misses, with an
> optional TTL. Results larger than `maxRowsPerKey` are streamed but not cached.
> - Confs
> `spark.sql.join.externalLookup.cache.\{enabled,maxEntries,ttl,maxRowsPerKey}`.
> Defaults: enabled, 10000 entries, no TTL (0), 1000 rows per key.
> - Per-join hint options as quoted `'key=value'` strings: `cache_enabled`,
> `cache_max_entries`, `cache_ttl` and `cache_max_rows_per_key`. Invalid
> options fail with `INVALID_EXTERNAL_LOOKUP_OPTION`.
> - A `cacheHits` metric.
> For discussion: the cache is enabled by default, and the SPIP does not state
> a default.
> Out of scope:
> - Caches shared across tasks or queries.
> - Predicate pushdown through the join.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
