andygrove commented on PR #214: URL: https://github.com/apache/datafusion-site/pull/214#issuecomment-6531596687
I reran the suite on OSS Spark 4.1.3 with Iceberg to compare the Comet 1.1.0 release with the upcoming 1.2.0. The results line up with the post: Spark took 1,519 s in total (the post has 1,515 s on Spark 4.0), and Comet 1.2.0 took 790 s. Best of 3 iterations, in seconds: | Query | Spark 4.1.3 | Comet 1.1.0 | Comet 1.2.0 | Speedup, 1.1.0 | Speedup, 1.2.0 | |---|---:|---:|---:|---:|---:| | q01 | 84.9 | 17.0 | 14.8 | 5.01x | 5.73x | | q02 | 39.3 | 23.4 | 27.0 | 1.68x | 1.45x | | q03 | 39.7 | 21.6 | 20.1 | 1.83x | 1.97x | | q04 | 45.2 | 19.8 | 16.1 | 2.28x | 2.81x | | q05 | 82.6 | 48.5 | 43.9 | 1.70x | 1.88x | | q06 | 5.0 | 2.2 | 2.1 | 2.22x | 2.41x | | q07 | 58.6 | 35.3 | 33.5 | 1.66x | 1.75x | | q08 | 84.6 | 64.1 | 59.3 | 1.32x | 1.43x | | q09 | 97.1 | 73.2 | 64.9 | 1.33x | 1.50x | | q10 | 50.9 | 29.2 | 26.6 | 1.74x | 1.91x | | q11 | 59.9 | 41.8 | 40.0 | 1.43x | 1.50x | | q12 | 37.1 | 17.8 | 15.2 | 2.08x | 2.44x | | q13 | 25.6 | 9.3 | 8.7 | 2.75x | 2.95x | | q14 | 8.1 | 8.8 | 4.3 | 0.93x | 1.91x | | q15 | 34.2 | 21.1 | 23.0 | 1.62x | 1.49x | | q16 | 31.9 | 14.7 | 15.4 | 2.16x | 2.08x | | q17 | 133.6 | 48.1 | 42.8 | 2.78x | 3.13x | | q18 | 152.6 | 76.3 | 56.4 | 2.00x | 2.71x | | q19 | 32.1 | 17.2 | 15.0 | 1.87x | 2.14x | | q20 | 34.3 | 24.6 | 24.3 | 1.39x | 1.41x | | q21 | 332.0 | 218.9 | 202.1 | 1.52x | 1.64x | | q22 | 49.8 | 36.2 | 35.0 | 1.38x | 1.43x | | **Total** | **1,519** | **869** | **790** | **1.75x** | **1.92x** | **Setup** - **Data:** the nested SF1000 tables made by `nest_tpch.scala` from flat TPC-H, stored as Iceberg in a Hadoop catalog on S3. The files are zstd Parquet with 512 MiB row groups, partitioned as in the post. - **Queries:** `benchmarks/nested` from Comet `main`, run from `spark-shell`. Each query runs 3 times in one application with no warm-up, timed from `spark.sql()` to the end of `collect()`. Every query returned the expected TPC-H row count. - **Versions:** Spark 4.1.3 (Scala 2.13, JDK 17), Iceberg 1.11.0 (`iceberg-spark-runtime-4.1`), Comet 1.1.0, and Comet 1.2.0 from `branch-1.2` at `746246049` (the 1.2.0 docs commit). Both Comet versions use the native Iceberg scan and the default build, which uses glibc's allocator. - **Cluster:** Kubernetes, with 32 executors of 16 task slots each (8 CPUs per executor pod), shuffle and spill files on a block volume attached to each executor, and `spark.sql.shuffle.partitions=512`. Both engines share these settings: G1 GC, Kryo, `spark.io.compression.codec=zstd`, `spark.memory.fraction=0.6`, `spark.memory.storageFraction=0.2`, `spark.sql.sources.ignoreDataLocality=true`, and `spark.executor.memoryOverhead=10g`. Spark ran with `spark.executor.memory=64g` and no off-heap memory. Comet ran with these settings: ``` spark.executor.memory=32g spark.memory.offHeap.enabled=true spark.memory.offHeap.size=64g spark.executor.memoryOverhead=10g spark.plugins=org.apache.spark.CometPlugin spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager spark.comet.exec.memoryPool.fraction=0.8 spark.comet.parquet.prefetchNextFileMetadata.enabled=true spark.comet.scan.icebergNative.enabled=true spark.comet.scan.icebergNative.dataFileConcurrencyLimit=4 spark.executor.defaultJavaOptions=-XX:TrimNativeHeapInterval=5000 -Xlog:trimnative ``` So each Comet executor gets the same 106 GiB container as a Spark executor. Comet moves 32 GiB of the heap to off-heap, giving each task 4 GiB of native memory. **Two things we had to change to run the full suite** - **Off-heap memory.** With a 64 GiB heap and 32 GiB off-heap, each task gets 2 GiB of native memory. That is not enough for q21's sort-merge join stage on 1.2.0: it allocates about 2.2 GiB per task and reserves up to 3.1 GiB, so tasks failed and had to be retried. - **`-XX:TrimNativeHeapInterval`.** Without it, glibc keeps native memory that Comet has freed, and the executors' resident memory grows from one query to the next. Late in the suite, q21 got 11 executors OOMKilled while Comet's memory pools were well inside their limits. Trimming every 5 s kept the executors at 61 to 70 GiB of resident memory, and q21 ran steadily: 219-221 s on 1.1.0 and 202-205 s on 1.2.0. The trims cost about 4% on the other queries. There are more details in https://github.com/apache/datafusion-comet/pull/7304#issuecomment-6528220294, and the advice is going into the tuning guide in https://github.com/apache/datafusion-comet/pull/7342. Comet 1.2.0 is 9% faster than 1.1.0 overall. The largest gains are q14 (2.0x), q18 (1.35x) and q04 (1.23x). 1.1.0 is slightly faster on q02, q15 and q16, but this is one run of each version, so small differences may be noise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
