Hi Sungwoo,
I disagree - on our side LLAP IO is a clear win on sequential 10 TB TPC-DS,
and the cache hit ratio is not low. Some numbers, since we built our
cluster specifically to replicate your benchmark.
Setup. 24 × r6id.4xlarge (16 vCPU / 8 physical cores / 128 GiB / local
NVMe) running LLAP plus 2 Tez AMs, with HiveServer2, HMS, ZooKeeper and
Postgres on a separate m5.2xlarge pool. That is 192 physical cores, 12
executors per daemon, 288 executor slots, 48 GiB heap per daemon and a 400
GB IO cache per daemon. TPC-DS 10 TB, 99 queries × 3 iterations with the
first discarded, run sequentially.
On sizing, we matched your cluster's memory exactly, which leaves us at 80%
of your physical cores. Matching cores instead would have meant 30 nodes —
240 cores, 14 executors per daemon, 420 slots, 97% of yours — but that
would also have handed us 25% more memory than you had, so we took the
configuration that does not favour us.
One further difference worth stating up front: your published suite is ORC
on HDFS, ours is Parquet on S3. Not like-for-like, and both differences run
against us — ORC is the format Hive's LLAP IO was built around, and S3 adds
object-store latency that local HDFS does not.
Cache on vs off — q13, warm pass of two iterations, same cluster, one
variable changed:
| Configuration | Warm Cache Hit Ratio | q13 Warm Time |
| :-------------------------------- | :----------------------------- |
:-------------------- |
| native Parquet cache on | 99.3% | 67.2 s
|
| native Parquet cache off | — | 79.4 s
|
15.4% faster with the cache, at a 99.3% hit ratio, running sequentially.
Across the full suite the run finished 99.9% data-local — 601,759 of
602,611 tasks.
Full 99-query suite: 4226.3 s, against 6654.5 s for Hive 4.2 (LLAP) in your
published figures — 36.5% faster on the same 99 queries, with Parquet on S3
against your ORC on HDFS. We are still behind Hive 4.2 on MR3 3.0 (3085.2
s) and Trino 483 (3140.1 s), but those are ORC-on-HDFS numbers too, so we
are behind while carrying both handicaps.
That is really the point I would make about the LLAP figure. Hive's LLAP IO
was built around ORC; Parquet never had an equivalent native cached-decode
path, going through the byte cache with parquet-mr re-decoding on every
read. Your 6654.5 s is LLAP on its best format and local storage. We beat
it by 36.5% on the format LLAP is least optimized for, over S3 — which says
the LLAP configuration in that measurement was leaving a lot on the table,
not that LLAP IO costs you performance. The 15.4% above is a gain on top of
the existing byte-cache path, not a comparison against LLAP IO being off.
On memory. The constraint is real, but the cache does not have to compete
with executor heap. We run hive.llap.io.memory.mode=cache with
hive.llap.io.allocator.mmap=true pointed at instance NVMe
(/apps/llap/work), giving 400 GB per daemon on nodes with only 128 GB of
RAM. Executors keep their 6–8 GB; the cache lives on the local SSD. On
instance types with NVMe there is no need to take memory from executors to
get a large cache.
On the hit ratio generally. It tracks task locality almost exactly. If a
task is not scheduled onto the daemon holding the data, the cache cannot be
hit however large it is, and the hit rate ends up pinned to the data-local
rate rather than reflecting anything about the workload. DATA_LOCAL_TASKS
vs RACK_LOCAL_TASKS in the DAG counters (hive.tez.exec.print.summary=true)
is the quickest way to tell a caching problem from a placement one.
Happy to share the full run records.
On Wed, Sep 16, 2026 at 4:43 PM Sungwoo Park <[email protected]> wrote:
> Yes, for the 10TB TPC-DS benchmark, LLAP IO was disabled for performance.
> For executing TPC-DS queries sequentially, enabling LLAP IO actually hurts
> the performance, mostly because the cache hit ratio is low. Moreover, in
> our test cluster, no extra memory is available to allocate to LLAP after
> assigning 6GB ~ 8GB to each executor.
>
> Thank you.
>
> Sungwoo
>
> On Tue, Sep 15, 2026 at 8:56 PM László Bodor <[email protected]>
> wrote:
>
>> Thanks for sharing the configs.
>> Quick question about *hive.llap.io.enabled=false*: should I assume that,
>> for the Hive LLAP tests, the I/O cache was fully disabled?
>>
>> Regarding the LLAP cache: I agree with NVMe SSDs' wide availability. The
>> cluster we're running for performance tests utilizes NVMe SSDs for both
>> intermediate data spill and IO cache mmap. While mmap is very fast on a
>> warm hit, it has a penalty in every other case (e.g. mmap is not
>> cache-content aware).
>> The main difference between mmap and a "native file cache," as far as I
>> can understand, is who decides what stays in RAM: in the case of mmap, it's
>> the OS, whereas in the case of a "native file cache," it's the application
>> itself. For better caching decisions, a good app-level implementation
>> should win. So yeah, playing with cache implementations would be awesome!
>> Would you be interested in that *Lisoda*?
>>
>>
>> "Since mmap requires mapping file contents into a region of memory, the
>>> host's memory capacity directly limits how much data we can cache. "
>>
>>
>> Small clarification on this point, I think the framing slightly conflates
>> two things: mmap reserves virtual address space, but only touched pages are
>> actually resident in RAM; the rest live on the SSD and are paged in on
>> access. So the size of an mmap-backed cache isn't bounded by host memory,
>> it's bounded by SSD capacity. What host RAM does bound is the hot working
>> set (pages the kernel keeps resident) and the on-heap cache index (LLAP
>> tracks each cached buffer on the JVM heap, which is where the ~6% figure in
>> the HDInsight guide comes from).
>>
>> Regards,
>> Laszlo Bodor
>>
>>
>> On Fri, 11 Sept 2026 at 14:35, Denys Kuzmenko <[email protected]>
>> wrote:
>>
>>> Thank you!
>>>
>>
