This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new c71edaa6c897 [SPARK-57939][SQL][TEST] Add wide-schema benchmark cases
to `UnsafeProjectionBenchmark`
c71edaa6c897 is described below
commit c71edaa6c897a8aa6650d2e98fa8b26b985d3942
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sun Jul 5 16:06:50 2026 -0700
[SPARK-57939][SQL][TEST] Add wide-schema benchmark cases to
`UnsafeProjectionBenchmark`
### What changes were proposed in this pull request?
This PR adds wide-schema benchmark cases to `UnsafeProjectionBenchmark`:
for each of
`BooleanType`, `ByteType`, `ShortType`, `IntegerType`, and `FloatType`,
repeatedly
project a single row with 400 non-nullable fields of that type.
- One section per type, so a future improvement can be evaluated per type.
- A single row is projected repeatedly so the source stays in L1 cache and
the
field-write path is measured purely.
### Why are the changes needed?
The existing cases use at most 7 fields, so per-row fixed overhead
dominates and
improvements in the field-write path are not visible. With 400 fields,
field writing
dominates. A follow-up PR improving `UnsafeRowWriter`'s fixed-width field
writes will
use these cases to show before/after numbers.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Pass the CIs and check the generated result files.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #57014 from dongjoon-hyun/SPARK-57939.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit e6793ed7981760e585577576014c21fca5f6dc27)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../UnsafeProjectionBenchmark-jdk21-results.txt | 65 ++++++++++++++++++++--
.../UnsafeProjectionBenchmark-jdk25-results.txt | 65 ++++++++++++++++++++--
.../UnsafeProjectionBenchmark-results.txt | 65 ++++++++++++++++++++--
.../spark/sql/UnsafeProjectionBenchmark.scala | 27 +++++++++
4 files changed, 207 insertions(+), 15 deletions(-)
diff --git
a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk21-results.txt
b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk21-results.txt
index 4fac4cee9d3d..b82303892a77 100644
--- a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk21-results.txt
+++ b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk21-results.txt
@@ -2,13 +2,68 @@
unsafe projection
================================================================================================
-OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
unsafe projection: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-single long 1335 1335
0 201.0 5.0 1.0X
-single nullable long 2392 2397
7 112.2 8.9 0.6X
-7 primitive types 7134 7134
1 37.6 26.6 0.2X
-7 nullable primitive types 10679 10689
14 25.1 39.8 0.1X
+single long 1342 1342
0 200.0 5.0 1.0X
+single nullable long 2380 2393
19 112.8 8.9 0.6X
+7 primitive types 7161 7163
3 37.5 26.7 0.2X
+7 nullable primitive types 10670 10672
3 25.2 39.7 0.1X
+
+
+================================================================================================
+wide unsafe projection on boolean fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 boolean fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 boolean fields 6574 6577
4 0.6 1567.4 1.0X
+
+
+================================================================================================
+wide unsafe projection on byte fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 byte fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 byte fields 6745 6745
1 0.6 1608.1 1.0X
+
+
+================================================================================================
+wide unsafe projection on short fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 short fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 short fields 6817 6829
17 0.6 1625.2 1.0X
+
+
+================================================================================================
+wide unsafe projection on integer fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 integer fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 integer fields 6646 6652
8 0.6 1584.6 1.0X
+
+
+================================================================================================
+wide unsafe projection on float fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 float fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 float fields 6717 6730
19 0.6 1601.4 1.0X
diff --git
a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk25-results.txt
b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk25-results.txt
index 8ea2de901da3..14fe9dab53c3 100644
--- a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk25-results.txt
+++ b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-jdk25-results.txt
@@ -2,13 +2,68 @@
unsafe projection
================================================================================================
-OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.14.0-1017-azure
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
unsafe projection: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-single long 1325 1325
0 202.7 4.9 1.0X
-single nullable long 2427 2429
3 110.6 9.0 0.5X
-7 primitive types 7476 7484
11 35.9 27.9 0.2X
-7 nullable primitive types 10587 10608
29 25.4 39.4 0.1X
+single long 1335 1336
1 201.0 5.0 1.0X
+single nullable long 2399 2399
0 111.9 8.9 0.6X
+7 primitive types 7794 7803
13 34.4 29.0 0.2X
+7 nullable primitive types 10564 10575
15 25.4 39.4 0.1X
+
+
+================================================================================================
+wide unsafe projection on boolean fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 boolean fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 boolean fields 6583 6586
4 0.6 1569.5 1.0X
+
+
+================================================================================================
+wide unsafe projection on byte fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 byte fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 byte fields 6870 6872
3 0.6 1638.0 1.0X
+
+
+================================================================================================
+wide unsafe projection on short fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 short fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 short fields 6776 6786
14 0.6 1615.4 1.0X
+
+
+================================================================================================
+wide unsafe projection on integer fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 integer fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 integer fields 6676 6680
6 0.6 1591.7 1.0X
+
+
+================================================================================================
+wide unsafe projection on float fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 float fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 float fields 6669 6671
2 0.6 1590.1 1.0X
diff --git a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-results.txt
b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-results.txt
index cd1004d57431..7582ce227eec 100644
--- a/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-results.txt
+++ b/sql/catalyst/benchmarks/UnsafeProjectionBenchmark-results.txt
@@ -2,13 +2,68 @@
unsafe projection
================================================================================================
-OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
AMD EPYC 7763 64-Core Processor
unsafe projection: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-single long 1376 1377
1 195.0 5.1 1.0X
-single nullable long 2454 2454
0 109.4 9.1 0.6X
-7 primitive types 7049 7049
0 38.1 26.3 0.2X
-7 nullable primitive types 10384 10384
0 25.8 38.7 0.1X
+single long 1296 1296
0 207.2 4.8 1.0X
+single nullable long 2454 2454
0 109.4 9.1 0.5X
+7 primitive types 7017 7017
0 38.3 26.1 0.2X
+7 nullable primitive types 10331 10337
8 26.0 38.5 0.1X
+
+
+================================================================================================
+wide unsafe projection on boolean fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 boolean fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 boolean fields 6529 6536
9 0.6 1556.7 1.0X
+
+
+================================================================================================
+wide unsafe projection on byte fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 byte fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 byte fields 6689 6692
5 0.6 1594.8 1.0X
+
+
+================================================================================================
+wide unsafe projection on short fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 short fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 short fields 6701 6701
1 0.6 1597.6 1.0X
+
+
+================================================================================================
+wide unsafe projection on integer fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 integer fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 integer fields 6594 6596
3 0.6 1572.1 1.0X
+
+
+================================================================================================
+wide unsafe projection on float fields
+================================================================================================
+
+OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux 6.17.0-1018-azure
+AMD EPYC 7763 64-Core Processor
+single row with 400 float fields: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+single row with 400 float fields 6569 6572
4 0.6 1566.2 1.0X
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/UnsafeProjectionBenchmark.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/UnsafeProjectionBenchmark.scala
index 186ae44a1084..937a530a5f38 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/UnsafeProjectionBenchmark.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/UnsafeProjectionBenchmark.scala
@@ -130,5 +130,32 @@ object UnsafeProjectionBenchmark extends BenchmarkBase {
benchmark.run()
}
+
+ Seq(BooleanType, ByteType, ShortType, IntegerType, FloatType).foreach {
dataType =>
+ val iters = 1024 * 4096
+ val numFields = 400
+ val caseName = s"single row with $numFields ${dataType.typeName} fields"
+
+ runBenchmark(s"wide unsafe projection on ${dataType.typeName} fields") {
+ val benchmark = new Benchmark(caseName, iters.toLong, output = output)
+
+ val schema = (1 to numFields).foldLeft(new StructType())(
+ (schema, i) => schema.add(s"c$i", dataType, false))
+ val attrs = DataTypeUtils.toAttributes(schema)
+ val row = generateRows(schema, 1).head
+ val projection = UnsafeProjection.create(attrs, attrs)
+
+ benchmark.addCase(caseName) { _ =>
+ var sum = 0L
+ var i = 0
+ while (i < iters) {
+ sum += projection(row).getLong(0)
+ i += 1
+ }
+ }
+
+ benchmark.run()
+ }
+ }
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
