baibaichen opened a new issue, #12671:
URL: https://github.com/apache/gluten/issues/12671
### Description
When `spark.gluten.sql.columnar.backend.velox.cacheEnabled` is on,
`VeloxBackend::initCache()`
builds an `MmapAllocator` sized from `memCacheSize` and hands it to the
`AsyncDataCache`:
```cpp
options.capacity = memCacheSize;
cacheAllocator_ = std::make_shared<velox::memory::MmapAllocator>(options);
if (ssdCacheSize == 0) {
// TODO: this is not tracked by Spark.
asyncDataCache_ =
velox::cache::AsyncDataCache::create(cacheAllocator_.get());
} else {
// TODO: this is not tracked by Spark.
auto ssd = initSsdCache(ssdCacheSize);
asyncDataCache_ =
velox::cache::AsyncDataCache::create(cacheAllocator_.get(), std::move(ssd));
}
```
Spark never sees that memory. The two `// TODO: this is not tracked by
Spark.` comments in
`cpp/velox/compute/VeloxBackend.cc` record exactly this.
#### Why it matters
An executor ends up occupying its Spark budget **plus** the whole cache.
Every decision Spark
makes from its own books is then made against a number that is short by up
to the cache size:
- when to spill,
- when to fail a task with an OOM,
- how much a task may acquire.
The cache is also fixed at its configured size for the lifetime of the
executor. It cannot give
anything back when queries need the memory, which is the wrong priority: a
query that has to
spill costs far more than the cache misses that shrinking would cause.
#### Proposal
Make the cache capacity something Spark can move at runtime, and register
what the cache occupies
with Spark's memory manager, so that:
1. the cache's footprint is visible to Spark on **all** supported Spark
versions;
2. the cache yields memory when execution needs it, and reclaims it when the
pressure lifts;
3. usage stays bounded by what Spark has agreed to.
An earlier attempt used Spark 4.1's `UnmanagedMemoryConsumer`, which only
works on 4.1+ and reacts
only at task boundaries.
This description was written with the assistance of generative AI tooling
(GitHub Copilot CLI).
### Gluten version
main branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
