This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 1870de0b329a [SPARK-45110][BUILD] Upgrade rocksdbjni to 8.8.1 1870de0b329a is described below commit 1870de0b329ac5ef35a331a653b4debd85eaa684 Author: panbingkun <pbk1...@gmail.com> AuthorDate: Thu Feb 1 06:37:00 2024 -0600 [SPARK-45110][BUILD] Upgrade rocksdbjni to 8.8.1 ### What changes were proposed in this pull request? The pr aims to upgrade rocksdbjni from `8.3.2` to `8.8.1`. Why version `8.8.1`? Because so far, `32` tests have been conducted based on version `8.6.7` or `8.8.1`, and no previous core issues have been found. The later versions have not been rigorously validated. ### Why are the changes needed? 1.The full release notes: - https://github.com/facebook/rocksdb/releases/tag/v8.8.1 - https://github.com/facebook/rocksdb/releases/tag/v8.7.3 - https://github.com/facebook/rocksdb/releases/tag/v8.6.7 - https://github.com/facebook/rocksdb/releases/tag/v8.5.4 - https://github.com/facebook/rocksdb/releases/tag/v8.5.3 - https://github.com/facebook/rocksdb/releases/tag/v8.4.4 - https://github.com/facebook/rocksdb/releases/tag/v8.3.3 2.Bug Fixes, eg: - Fixed a bug where compaction read under non direct IO still falls back to RocksDB internal prefetching after file system's prefetching returns non-OK status other than Status::NotSupported() - Fix a bug with atomic_flush=true that can cause DB to stuck after a flush fails - Fix a bug where if there is an error reading from offset 0 of a file from L1+ and that the file is not the first file in the sorted run, data can be lost in compaction and read/scan can return incorrect results. - Fix a bug where iterator may return incorrect result for DeleteRange() users if there was an error reading from a file. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - Pass GA. - Manually test: ``` ./build/mvn clean install -pl core -am -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest -fn ``` ### Was this patch authored or co-authored using generative AI tooling? No. Closes #43924 from panbingkun/upgrade_rocksdbjni. Lead-authored-by: panbingkun <pbk1...@gmail.com> Co-authored-by: panbingkun <panbing...@baidu.com> Signed-off-by: Sean Owen <sro...@gmail.com> --- dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 +- pom.xml | 2 +- ...StoreBasicOperationsBenchmark-jdk21-results.txt | 70 ++++++++++----------- .../StateStoreBasicOperationsBenchmark-results.txt | 72 +++++++++++----------- 4 files changed, 73 insertions(+), 73 deletions(-) diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index fcb3350e5de2..e02733883642 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -239,7 +239,7 @@ parquet-jackson/1.13.1//parquet-jackson-1.13.1.jar pickle/1.3//pickle-1.3.jar py4j/0.10.9.7//py4j-0.10.9.7.jar remotetea-oncrpc/1.1.2//remotetea-oncrpc-1.1.2.jar -rocksdbjni/8.3.2//rocksdbjni-8.3.2.jar +rocksdbjni/8.8.1//rocksdbjni-8.8.1.jar scala-collection-compat_2.13/2.7.0//scala-collection-compat_2.13-2.7.0.jar scala-compiler/2.13.12//scala-compiler-2.13.12.jar scala-library/2.13.12//scala-library-2.13.12.jar diff --git a/pom.xml b/pom.xml index 6e118bb27f5a..2fc14a4cdede 100644 --- a/pom.xml +++ b/pom.xml @@ -677,7 +677,7 @@ <dependency> <groupId>org.rocksdb</groupId> <artifactId>rocksdbjni</artifactId> - <version>8.3.2</version> + <version>8.8.1</version> </dependency> <dependency> <groupId>${leveldbjni.group}</groupId> diff --git a/sql/core/benchmarks/StateStoreBasicOperationsBenchmark-jdk21-results.txt b/sql/core/benchmarks/StateStoreBasicOperationsBenchmark-jdk21-results.txt index f92ae8668e16..c0d710873aed 100644 --- a/sql/core/benchmarks/StateStoreBasicOperationsBenchmark-jdk21-results.txt +++ b/sql/core/benchmarks/StateStoreBasicOperationsBenchmark-jdk21-results.txt @@ -6,33 +6,33 @@ OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor putting 10000 rows (10000 rows to overwrite - rate 100): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative --------------------------------------------------------------------------------------------------------------------------------------- -In-memory 5 6 0 1.8 541.4 1.0X -RocksDB (trackTotalNumberOfRows: true) 40 41 2 0.2 4023.4 0.1X -RocksDB (trackTotalNumberOfRows: false) 15 15 1 0.7 1452.5 0.4X +In-memory 6 7 1 1.8 556.4 1.0X +RocksDB (trackTotalNumberOfRows: true) 43 44 2 0.2 4292.9 0.1X +RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1525.0 0.4X OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor putting 10000 rows (5000 rows to overwrite - rate 50): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------- -In-memory 5 6 0 1.9 536.6 1.0X -RocksDB (trackTotalNumberOfRows: true) 33 35 1 0.3 3349.3 0.2X -RocksDB (trackTotalNumberOfRows: false) 15 15 1 0.7 1452.4 0.4X +In-memory 6 7 1 1.8 550.7 1.0X +RocksDB (trackTotalNumberOfRows: true) 36 37 1 0.3 3607.8 0.2X +RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1535.3 0.4X OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor putting 10000 rows (1000 rows to overwrite - rate 10): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------- -In-memory 5 6 1 1.9 516.3 1.0X -RocksDB (trackTotalNumberOfRows: true) 28 29 1 0.4 2781.8 0.2X -RocksDB (trackTotalNumberOfRows: false) 14 15 1 0.7 1450.0 0.4X +In-memory 5 6 1 1.9 535.0 1.0X +RocksDB (trackTotalNumberOfRows: true) 29 31 1 0.3 2940.4 0.2X +RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1518.9 0.4X OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor putting 10000 rows (0 rows to overwrite - rate 0): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative --------------------------------------------------------------------------------------------------------------------------------- -In-memory 5 5 1 2.0 511.0 1.0X -RocksDB (trackTotalNumberOfRows: true) 26 27 1 0.4 2628.2 0.2X -RocksDB (trackTotalNumberOfRows: false) 15 15 0 0.7 1450.1 0.4X +In-memory 5 6 1 1.9 518.3 1.0X +RocksDB (trackTotalNumberOfRows: true) 28 29 1 0.4 2755.7 0.2X +RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.7 1512.0 0.3X ================================================================================================ @@ -43,33 +43,33 @@ OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor trying to delete 10000 rows from 10000 rows(10000 rows are non-existing - rate 100): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 0 0 0 27.9 35.8 1.0X -RocksDB (trackTotalNumberOfRows: true) 26 27 1 0.4 2630.5 0.0X -RocksDB (trackTotalNumberOfRows: false) 14 15 0 0.7 1430.3 0.0X +In-memory 0 1 0 27.2 36.8 1.0X +RocksDB (trackTotalNumberOfRows: true) 28 29 0 0.4 2784.0 0.0X +RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1538.2 0.0X OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor trying to delete 10000 rows from 10000 rows(5000 rows are non-existing - rate 50): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ----------------------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 4 4 0 2.5 395.7 1.0X -RocksDB (trackTotalNumberOfRows: true) 34 34 1 0.3 3372.0 0.1X -RocksDB (trackTotalNumberOfRows: false) 14 15 1 0.7 1441.5 0.3X +In-memory 4 5 1 2.4 413.9 1.0X +RocksDB (trackTotalNumberOfRows: true) 36 37 1 0.3 3583.1 0.1X +RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.6 1546.3 0.3X OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor trying to delete 10000 rows from 10000 rows(1000 rows are non-existing - rate 10): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ----------------------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 4 5 0 2.3 443.8 1.0X -RocksDB (trackTotalNumberOfRows: true) 39 40 1 0.3 3911.8 0.1X -RocksDB (trackTotalNumberOfRows: false) 14 15 0 0.7 1441.4 0.3X +In-memory 5 5 1 2.2 459.3 1.0X +RocksDB (trackTotalNumberOfRows: true) 42 43 1 0.2 4189.6 0.1X +RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.6 1539.5 0.3X OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor trying to delete 10000 rows from 10000 rows(0 rows are non-existing - rate 0): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 5 5 0 2.2 451.4 1.0X -RocksDB (trackTotalNumberOfRows: true) 40 41 1 0.2 4035.9 0.1X -RocksDB (trackTotalNumberOfRows: false) 14 15 1 0.7 1443.9 0.3X +In-memory 5 5 1 2.1 471.5 1.0X +RocksDB (trackTotalNumberOfRows: true) 43 44 1 0.2 4323.6 0.1X +RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.6 1546.7 0.3X ================================================================================================ @@ -80,32 +80,32 @@ OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor evicting 10000 rows (maxTimestampToEvictInMillis: 9999) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 4 5 0 2.3 439.6 1.0X -RocksDB (trackTotalNumberOfRows: true) 39 40 1 0.3 3938.0 0.1X -RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1534.3 0.3X +In-memory 4 5 0 2.2 449.1 1.0X +RocksDB (trackTotalNumberOfRows: true) 41 42 1 0.2 4146.8 0.1X +RocksDB (trackTotalNumberOfRows: false) 15 15 0 0.7 1534.8 0.3X OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor evicting 5000 rows (maxTimestampToEvictInMillis: 4999) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------------------ -In-memory 4 4 0 2.5 399.2 1.0X -RocksDB (trackTotalNumberOfRows: true) 21 22 0 0.5 2115.0 0.2X +In-memory 4 4 0 2.5 405.7 1.0X +RocksDB (trackTotalNumberOfRows: true) 22 23 1 0.4 2233.4 0.2X RocksDB (trackTotalNumberOfRows: false) 9 9 0 1.1 919.4 0.4X OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor evicting 1000 rows (maxTimestampToEvictInMillis: 999) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ----------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 3 4 0 2.9 341.4 1.0X -RocksDB (trackTotalNumberOfRows: true) 7 7 0 1.5 658.2 0.5X -RocksDB (trackTotalNumberOfRows: false) 4 4 0 2.4 419.7 0.8X +In-memory 3 4 0 2.9 349.5 1.0X +RocksDB (trackTotalNumberOfRows: true) 7 7 0 1.4 698.9 0.5X +RocksDB (trackTotalNumberOfRows: false) 4 4 0 2.3 438.7 0.8X OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor evicting 0 rows (maxTimestampToEvictInMillis: -1) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 0 0 0 24.2 41.3 1.0X -RocksDB (trackTotalNumberOfRows: true) 3 3 0 3.5 284.7 0.1X -RocksDB (trackTotalNumberOfRows: false) 3 3 0 3.5 284.9 0.1X +In-memory 0 0 0 25.7 38.8 1.0X +RocksDB (trackTotalNumberOfRows: true) 3 3 0 3.3 307.0 0.1X +RocksDB (trackTotalNumberOfRows: false) 3 3 0 3.3 305.9 0.1X diff --git a/sql/core/benchmarks/StateStoreBasicOperationsBenchmark-results.txt b/sql/core/benchmarks/StateStoreBasicOperationsBenchmark-results.txt index dc5bf70c9b48..bad581489294 100644 --- a/sql/core/benchmarks/StateStoreBasicOperationsBenchmark-results.txt +++ b/sql/core/benchmarks/StateStoreBasicOperationsBenchmark-results.txt @@ -6,33 +6,33 @@ OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor putting 10000 rows (10000 rows to overwrite - rate 100): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative --------------------------------------------------------------------------------------------------------------------------------------- -In-memory 6 7 1 1.8 558.3 1.0X -RocksDB (trackTotalNumberOfRows: true) 42 44 1 0.2 4221.6 0.1X -RocksDB (trackTotalNumberOfRows: false) 16 17 1 0.6 1601.4 0.3X +In-memory 6 7 1 1.6 606.4 1.0X +RocksDB (trackTotalNumberOfRows: true) 43 44 1 0.2 4257.1 0.1X +RocksDB (trackTotalNumberOfRows: false) 16 16 1 0.6 1553.7 0.4X OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor putting 10000 rows (5000 rows to overwrite - rate 50): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------- -In-memory 6 8 1 1.6 625.4 1.0X -RocksDB (trackTotalNumberOfRows: true) 36 37 1 0.3 3626.6 0.2X -RocksDB (trackTotalNumberOfRows: false) 15 17 1 0.7 1538.3 0.4X +In-memory 6 7 1 1.6 629.8 1.0X +RocksDB (trackTotalNumberOfRows: true) 36 37 1 0.3 3590.6 0.2X +RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1514.3 0.4X OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor putting 10000 rows (1000 rows to overwrite - rate 10): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------- -In-memory 6 6 1 1.8 550.3 1.0X -RocksDB (trackTotalNumberOfRows: true) 29 30 1 0.3 2917.1 0.2X -RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.6 1548.4 0.4X +In-memory 6 6 0 1.8 568.2 1.0X +RocksDB (trackTotalNumberOfRows: true) 29 30 1 0.3 2906.0 0.2X +RocksDB (trackTotalNumberOfRows: false) 15 15 1 0.7 1479.4 0.4X OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor putting 10000 rows (0 rows to overwrite - rate 0): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative --------------------------------------------------------------------------------------------------------------------------------- -In-memory 5 6 0 1.9 518.2 1.0X -RocksDB (trackTotalNumberOfRows: true) 27 28 0 0.4 2728.7 0.2X -RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1533.3 0.3X +In-memory 6 6 0 1.8 555.9 1.0X +RocksDB (trackTotalNumberOfRows: true) 27 28 1 0.4 2737.5 0.2X +RocksDB (trackTotalNumberOfRows: false) 15 15 1 0.7 1487.7 0.4X ================================================================================================ @@ -43,33 +43,33 @@ OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor trying to delete 10000 rows from 10000 rows(10000 rows are non-existing - rate 100): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 0 1 0 26.8 37.3 1.0X -RocksDB (trackTotalNumberOfRows: true) 28 29 1 0.4 2779.9 0.0X -RocksDB (trackTotalNumberOfRows: false) 16 16 1 0.6 1567.2 0.0X +In-memory 1 1 0 18.5 54.0 1.0X +RocksDB (trackTotalNumberOfRows: true) 27 28 1 0.4 2709.4 0.0X +RocksDB (trackTotalNumberOfRows: false) 14 15 0 0.7 1449.2 0.0X OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor trying to delete 10000 rows from 10000 rows(5000 rows are non-existing - rate 50): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ----------------------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 4 5 1 2.5 404.0 1.0X -RocksDB (trackTotalNumberOfRows: true) 35 36 1 0.3 3478.4 0.1X -RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1485.5 0.3X +In-memory 5 5 0 2.2 450.0 1.0X +RocksDB (trackTotalNumberOfRows: true) 36 36 1 0.3 3552.9 0.1X +RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.7 1524.4 0.3X OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor trying to delete 10000 rows from 10000 rows(1000 rows are non-existing - rate 10): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ----------------------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 5 5 1 2.2 452.8 1.0X -RocksDB (trackTotalNumberOfRows: true) 41 42 1 0.2 4063.0 0.1X -RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.7 1482.8 0.3X +In-memory 5 6 1 1.9 515.2 1.0X +RocksDB (trackTotalNumberOfRows: true) 41 43 1 0.2 4127.9 0.1X +RocksDB (trackTotalNumberOfRows: false) 15 16 1 0.7 1460.0 0.4X OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor trying to delete 10000 rows from 10000 rows(0 rows are non-existing - rate 0): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 5 6 1 2.2 459.4 1.0X -RocksDB (trackTotalNumberOfRows: true) 42 43 1 0.2 4192.6 0.1X -RocksDB (trackTotalNumberOfRows: false) 15 16 0 0.7 1534.4 0.3X +In-memory 5 6 1 2.0 509.8 1.0X +RocksDB (trackTotalNumberOfRows: true) 42 43 1 0.2 4186.4 0.1X +RocksDB (trackTotalNumberOfRows: false) 14 15 0 0.7 1436.9 0.4X ================================================================================================ @@ -80,32 +80,32 @@ OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor evicting 10000 rows (maxTimestampToEvictInMillis: 9999) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 5 5 0 2.2 460.3 1.0X -RocksDB (trackTotalNumberOfRows: true) 42 43 1 0.2 4188.4 0.1X -RocksDB (trackTotalNumberOfRows: false) 16 16 0 0.6 1576.7 0.3X +In-memory 5 5 0 2.2 451.2 1.0X +RocksDB (trackTotalNumberOfRows: true) 41 42 1 0.2 4103.1 0.1X +RocksDB (trackTotalNumberOfRows: false) 16 16 0 0.6 1555.1 0.3X OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor evicting 5000 rows (maxTimestampToEvictInMillis: 4999) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------------------ -In-memory 4 5 0 2.4 412.1 1.0X -RocksDB (trackTotalNumberOfRows: true) 23 23 0 0.4 2262.6 0.2X -RocksDB (trackTotalNumberOfRows: false) 10 10 0 1.0 974.5 0.4X +In-memory 4 5 0 2.3 431.0 1.0X +RocksDB (trackTotalNumberOfRows: true) 22 23 0 0.4 2225.6 0.2X +RocksDB (trackTotalNumberOfRows: false) 10 10 0 1.0 965.2 0.4X OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor evicting 1000 rows (maxTimestampToEvictInMillis: 999) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ----------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 4 4 0 2.8 353.0 1.0X -RocksDB (trackTotalNumberOfRows: true) 7 7 0 1.4 717.7 0.5X -RocksDB (trackTotalNumberOfRows: false) 5 5 0 2.1 465.8 0.8X +In-memory 4 4 0 2.8 359.7 1.0X +RocksDB (trackTotalNumberOfRows: true) 7 7 0 1.4 718.8 0.5X +RocksDB (trackTotalNumberOfRows: false) 5 5 0 2.1 468.4 0.8X OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1053-azure AMD EPYC 7763 64-Core Processor evicting 0 rows (maxTimestampToEvictInMillis: -1) from 10000 rows: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------------------------- -In-memory 0 0 0 21.5 46.6 1.0X -RocksDB (trackTotalNumberOfRows: true) 3 3 0 3.0 330.8 0.1X -RocksDB (trackTotalNumberOfRows: false) 3 3 0 3.0 330.6 0.1X +In-memory 0 1 0 21.2 47.1 1.0X +RocksDB (trackTotalNumberOfRows: true) 3 3 0 3.0 333.7 0.1X +RocksDB (trackTotalNumberOfRows: false) 3 3 0 3.0 332.7 0.1X --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org