cgivre commented on PR #3086:
URL: https://github.com/apache/drill/pull/3086#issuecomment-6061492484
**Suggestion: make the plan cache size and expiration configurable**
`PlanCache` hard-codes its limits (`PlanCache.java:98-102`): a 32 MB maximum
weight and `expireAfterWrite(10, MINUTES)`. Operators can't tune either one. A
deployment with many distinct query templates may want a larger cache. One
where tables change often, or where memory on the planning Drillbit is tight,
may want a smaller cache or a shorter lifetime.
Could these be boot options in `drill-module.conf`, read in
`DrillbitContext` and passed to the `PlanCache` constructor? The generated-code
cache already does this with `drill.exec.compile.cache_max_size`. For example:
```
drill.exec.plan_cache: {
# Upper bound on the total size of cached plan JSON, per Drillbit
max_size_bytes: 33554432,
# How long an entry may live after it is written; 0 disables time-based
expiry
expire_after_write: 10m,
# Optional: evict entries that have not been used for this long; 0 disables
expire_after_access: 0
}
```
A few related points:
- The weigher uses `value.json.length()`, which counts characters, not
bytes. If the setting is named in bytes, the weigher should measure bytes (or
the setting should say characters).
- `expireAfterAccess` would let rarely used templates age out sooner, while
a hot query isn't replanned every 10 minutes just because it was written 10
minutes ago.
- It would help to log the settings in use when the Drillbit starts, and to
document them in the plan-cache design doc.
- Startup settings are a good fit because the cache is created once per
Drillbit, and Guava can't change these limits on a cache that's already built.
`planner.enable_plan_cache` stays the runtime on/off switch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
