WangGuangxin opened a new pull request, #13605:
URL: https://github.com/apache/gluten/pull/13605

   ## What changes are proposed in this pull request?
   
   With `spark.gluten.sql.columnar.backend.velox.cacheEnabled=true`, 
`VeloxBackend::initCache()` always builds the `AsyncDataCache` on Velox 
`MmapAllocator`. `MmapAllocator` tracks memory in 4 KB pages 
(`AllocationTraits::kPageSize`), and its constructor rejects any other system 
page size (`MmapAllocator::isPageSizeSupported()`). On aarch64 hosts running a 
64 KB-page kernel, which is the recommended setting for NVIDIA Grace / Vera, 
every executor therefore fails during native backend initialization:
   
   ```
   MmapAllocator.cpp:77 ... Expression: isPageSizeSupported() MmapAllocator 
requires the system page size to match AllocationTraits::kPageSize (4096 
bytes); system page size is 65536 bytes. Use MemoryAllocator::Kind::kMalloc on 
this system instead.
   ERROR NativeBackendInitializer: Failed to call native backend's initialize 
method
   ```
   
   This PR keeps `MmapAllocator` when the system page size is supported and 
otherwise falls back to `MallocAllocator`, as the Velox error message suggests. 
The cache capacity is still `memCacheSize`. `cacheAllocator_` is widened to 
`std::shared_ptr<MemoryAllocator>` so it can hold either allocator. Nothing 
changes on 4 KB-page hosts or when the cache is disabled.
   
   `MmapAllocator::isPageSizeSupported()` and `MmapAllocator::systemPageSize()` 
are available in the Velox version pinned on `main` (IBM `dft-2026_11_01`, 
which includes velox `0223a0ce7` "fix(memory): Reject unsupported page size in 
MmapAllocator").
   
   ## How was this patch tested?
   
   - Built and run as the same change on top of `3673833c2` (branch 
`velox-cache-malloc-64k-20261104`, Velox `be88416a3`): GCC 15.2, 
`-mcpu=olympus`, on a Vera host (aarch64, 64 KB pages, 42 executors × 8 cores, 
Spark 4.0.1). It was not rebuilt on `main` itself; the cherry-pick applied 
cleanly and the touched code is identical there.
   - TPCDS-H SF1K with `cacheEnabled=true`, `memCacheSize=8g`, 
`ssdCacheSize=0`, `loadQuantum=8MB`:
     - Without the patch, every executor fails at native init (above).
     - With the patch, all queries complete. Executor logs show the cache 
running on `MallocAllocator` (`MallocAllocator.cpp:67 ... the memory allocator 
capacity is 8.00GB`) and filling to capacity.
   - No unit test was added. The fallback only triggers on a non-4 KB page 
size, which the CI hosts do not have.
   - `clang-format-15` was not available on the build host. The changed lines 
follow the surrounding style and stay within the 120-column limit.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Opus 5.5 (claude-opus-5-5)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to