Jungtaek Lim created SPARK-60693:
------------------------------------
Summary: Add a task-scoped lookup cache to external lookup join
Key: SPARK-60693
URL: https://issues.apache.org/jira/browse/SPARK-60693
Project: Spark
Issue Type: Task
Components: SQL, Structured Streaming
Affects Versions: 4.4.0
Reporter: Jungtaek Lim
Part of SPARK-60031.
In scope:
- A per-task LRU cache of complete results, including misses, with an optional
TTL. Results larger than `maxRowsPerKey` are streamed but not cached.
- Confs
`spark.sql.join.externalLookup.cache.\{enabled,maxEntries,ttl,maxRowsPerKey}`.
Defaults: enabled, 10000 entries, no TTL (0), 1000 rows per key.
- Per-join hint options as quoted `'key=value'` strings: `cache_enabled`,
`cache_max_entries`, `cache_ttl` and `cache_max_rows_per_key`. Invalid options
fail with `INVALID_EXTERNAL_LOOKUP_OPTION`.
- A `cacheHits` metric.
For discussion: the cache is enabled by default, and the SPIP does not state a
default.
Out of scope:
- Caches shared across tasks or queries.
- Predicate pushdown through the join.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
