Hi Denys, Thank you for sharing your results. I am glad to hear that the LLAP IO works very well in cloud environments with Parquet + S3.
In my opinion, the two results do not really contradict each other (i.e., our result saying that LLAP IO is not useful, while your result saying that LLAP IO is a clear win) because in our test cluster, we set hive.llap.io.allocator.mmap=false and set hive.llap.io.memory.size=72Gb. In our tests, we set hive.llap.io.allocator.mmap=false, so the LLAP IO cache consumes physical memory. Allocating memory for LLAP IO, in turn, implies smaller memory for executors and consequently, higher memory contention among executors. For the 10TB TPC-DS benchmark where we execute all queries sequentially, I remember that with hive.llap.io.memory.size=72Gb, the cache hit ratio was not high enough, implying that the overhead of managing LLAP IO could outweigh the performance boost attributed to LLAP IO. In your tests, you assign 400GB x 24 = 9.6TB LLAP IO cache, so the high cache hit ratio is not surprising to me. I don't know how Parquet data is stored in LLAP IO cache, but it might be that the entire dataset fits in the 9.6TB LLAP IO cache, even after taking into account the space overhead. To summarize, 1. In our tests, the LLAP IO cache was configured to consume memory and its size was relatively small. So, the cache hit ratio was not high and LLAP IO was not very useful (even detrimental) to the 10TB TPC-DS benchmark. (Our test cluster uses slow SSDs with limited capacity, so it does not make sense to set hive.llap.io.allocator.mmap=true.) 2. In your cluster, the LLAP IO cache was configured to exploit NVMe and its size was relatively large, So, there is a clear benefit from enabling LLAP IO, much more so because of the separation of compute and S3 storage. I think that running times obtained from different environments should be compared carefully. Our cluster (using Xeon CPUs and SSDs from the 2016 era) may be slow from today's standard, but your cluster uses Parquet stored on S3. So, I am not sure what we can learn from a direct comparison of 4226.3s vs 6654.5s. Personally I am interested with the cache hit ratio with respect to the total size of LLAP IO cache. I cannot run TPC-DS experiments with hive.llap.io.allocator.mmap=true in our local cluster, but can try LLAP IO with hive.llap.io.allocator.mmap=false and report cache hit ratios. Thank you. Sungwoo On Wed, Sep 16, 2026 at 11:43 PM Denys Kuzmenko <[email protected]> wrote: > Hi Sungwoo, > > I disagree - on our side LLAP IO is a clear win on sequential 10 TB > TPC-DS, and the cache hit ratio is not low. Some numbers, since we built > our cluster specifically to replicate your benchmark. > > Setup. 24 × r6id.4xlarge (16 vCPU / 8 physical cores / 128 GiB / local > NVMe) running LLAP plus 2 Tez AMs, with HiveServer2, HMS, ZooKeeper and > Postgres on a separate m5.2xlarge pool. That is 192 physical cores, 12 > executors per daemon, 288 executor slots, 48 GiB heap per daemon and a 400 > GB IO cache per daemon. TPC-DS 10 TB, 99 queries × 3 iterations with the > first discarded, run sequentially. > > On sizing, we matched your cluster's memory exactly, which leaves us at > 80% of your physical cores. Matching cores instead would have meant 30 > nodes — 240 cores, 14 executors per daemon, 420 slots, 97% of yours — but > that would also have handed us 25% more memory than you had, so we took the > configuration that does not favour us. > > One further difference worth stating up front: your published suite is ORC > on HDFS, ours is Parquet on S3. Not like-for-like, and both differences run > against us — ORC is the format Hive's LLAP IO was built around, and S3 adds > object-store latency that local HDFS does not. > > Cache on vs off — q13, warm pass of two iterations, same cluster, one > variable changed: > > | Configuration | Warm Cache Hit Ratio | q13 Warm Time | > | :-------------------------------- | :----------------------------- | > :-------------------- | > | native Parquet cache on | 99.3% | 67.2 s > | > | native Parquet cache off | — | 79.4 s > | > > 15.4% faster with the cache, at a 99.3% hit ratio, running sequentially. > Across the full suite the run finished 99.9% data-local — 601,759 of > 602,611 tasks. > > Full 99-query suite: 4226.3 s, against 6654.5 s for Hive 4.2 (LLAP) in > your published figures — 36.5% faster on the same 99 queries, with Parquet > on S3 against your ORC on HDFS. We are still behind Hive 4.2 on MR3 3.0 > (3085.2 s) and Trino 483 (3140.1 s), but those are ORC-on-HDFS numbers too, > so we are behind while carrying both handicaps. > > That is really the point I would make about the LLAP figure. Hive's LLAP > IO was built around ORC; Parquet never had an equivalent native > cached-decode path, going through the byte cache with parquet-mr > re-decoding on every read. Your 6654.5 s is LLAP on its best format and > local storage. We beat it by 36.5% on the format LLAP is least optimized > for, over S3 — which says the LLAP configuration in that measurement was > leaving a lot on the table, not that LLAP IO costs you performance. The > 15.4% above is a gain on top of the existing byte-cache path, not a > comparison against LLAP IO being off. > > On memory. The constraint is real, but the cache does not have to compete > with executor heap. We run hive.llap.io.memory.mode=cache with > hive.llap.io.allocator.mmap=true pointed at instance NVMe > (/apps/llap/work), giving 400 GB per daemon on nodes with only 128 GB of > RAM. Executors keep their 6–8 GB; the cache lives on the local SSD. On > instance types with NVMe there is no need to take memory from executors to > get a large cache. > > On the hit ratio generally. It tracks task locality almost exactly. If a > task is not scheduled onto the daemon holding the data, the cache cannot be > hit however large it is, and the hit rate ends up pinned to the data-local > rate rather than reflecting anything about the workload. DATA_LOCAL_TASKS > vs RACK_LOCAL_TASKS in the DAG counters (hive.tez.exec.print.summary=true) > is the quickest way to tell a caching problem from a placement one. > > Happy to share the full run records. > > > On Wed, Sep 16, 2026 at 4:43 PM Sungwoo Park <[email protected]> wrote: > >> Yes, for the 10TB TPC-DS benchmark, LLAP IO was disabled for performance. >> For executing TPC-DS queries sequentially, enabling LLAP IO actually hurts >> the performance, mostly because the cache hit ratio is low. Moreover, in >> our test cluster, no extra memory is available to allocate to LLAP after >> assigning 6GB ~ 8GB to each executor. >> >> Thank you. >> >> Sungwoo >> >> On Tue, Sep 15, 2026 at 8:56 PM László Bodor <[email protected]> >> wrote: >> >>> Thanks for sharing the configs. >>> Quick question about *hive.llap.io.enabled=false*: should I assume >>> that, for the Hive LLAP tests, the I/O cache was fully disabled? >>> >>> Regarding the LLAP cache: I agree with NVMe SSDs' wide availability. The >>> cluster we're running for performance tests utilizes NVMe SSDs for both >>> intermediate data spill and IO cache mmap. While mmap is very fast on a >>> warm hit, it has a penalty in every other case (e.g. mmap is not >>> cache-content aware). >>> The main difference between mmap and a "native file cache," as far as I >>> can understand, is who decides what stays in RAM: in the case of mmap, it's >>> the OS, whereas in the case of a "native file cache," it's the application >>> itself. For better caching decisions, a good app-level implementation >>> should win. So yeah, playing with cache implementations would be awesome! >>> Would you be interested in that *Lisoda*? >>> >>> >>> "Since mmap requires mapping file contents into a region of memory, the >>>> host's memory capacity directly limits how much data we can cache. " >>> >>> >>> Small clarification on this point, I think the framing slightly >>> conflates two things: mmap reserves virtual address space, but only touched >>> pages are actually resident in RAM; the rest live on the SSD and are paged >>> in on access. So the size of an mmap-backed cache isn't bounded by host >>> memory, it's bounded by SSD capacity. What host RAM does bound is the hot >>> working set (pages the kernel keeps resident) and the on-heap cache index >>> (LLAP tracks each cached buffer on the JVM heap, which is where the ~6% >>> figure in the HDInsight guide comes from). >>> >>> Regards, >>> Laszlo Bodor >>> >>> >>> On Fri, 11 Sept 2026 at 14:35, Denys Kuzmenko <[email protected]> >>> wrote: >>> >>>> Thank you! >>>> >>>
