Yes, for the 10TB TPC-DS benchmark, LLAP IO was disabled for performance. For executing TPC-DS queries sequentially, enabling LLAP IO actually hurts the performance, mostly because the cache hit ratio is low. Moreover, in our test cluster, no extra memory is available to allocate to LLAP after assigning 6GB ~ 8GB to each executor.
Thank you. Sungwoo On Tue, Sep 15, 2026 at 8:56 PM László Bodor <[email protected]> wrote: > Thanks for sharing the configs. > Quick question about *hive.llap.io.enabled=false*: should I assume that, > for the Hive LLAP tests, the I/O cache was fully disabled? > > Regarding the LLAP cache: I agree with NVMe SSDs' wide availability. The > cluster we're running for performance tests utilizes NVMe SSDs for both > intermediate data spill and IO cache mmap. While mmap is very fast on a > warm hit, it has a penalty in every other case (e.g. mmap is not > cache-content aware). > The main difference between mmap and a "native file cache," as far as I > can understand, is who decides what stays in RAM: in the case of mmap, it's > the OS, whereas in the case of a "native file cache," it's the application > itself. For better caching decisions, a good app-level implementation > should win. So yeah, playing with cache implementations would be awesome! > Would you be interested in that *Lisoda*? > > > "Since mmap requires mapping file contents into a region of memory, the >> host's memory capacity directly limits how much data we can cache. " > > > Small clarification on this point, I think the framing slightly conflates > two things: mmap reserves virtual address space, but only touched pages are > actually resident in RAM; the rest live on the SSD and are paged in on > access. So the size of an mmap-backed cache isn't bounded by host memory, > it's bounded by SSD capacity. What host RAM does bound is the hot working > set (pages the kernel keeps resident) and the on-heap cache index (LLAP > tracks each cached buffer on the JVM heap, which is where the ~6% figure in > the HDInsight guide comes from). > > Regards, > Laszlo Bodor > > > On Fri, 11 Sept 2026 at 14:35, Denys Kuzmenko <[email protected]> > wrote: > >> Thank you! >> >
