This is an automated email from the ASF dual-hosted git repository.
sunchao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new c0a12cfbb82 [SPARK-39231][SQL] Use `ConstantColumnVector` instead of
`On/OffHeapColumnVector` to store partition columns in
`VectorizedParquetRecordReader`
c0a12cfbb82 is described below
commit c0a12cfbb82a710d786158bcd70a7f7bd531751b
Author: yangjie01 <[email protected]>
AuthorDate: Wed Jun 29 09:22:14 2022 -0700
[SPARK-39231][SQL] Use `ConstantColumnVector` instead of
`On/OffHeapColumnVector` to store partition columns in
`VectorizedParquetRecordReader`
### What changes were proposed in this pull request?
This pr change to use `ConstantColumnVector` to store partition columns in
`VectorizedParquetRecordReader` because partition column vector always constant
vector.
### Why are the changes needed?
1. Partition columns vector alway constant vector.
2. **Performance improvement**: `ConstantColumnVector` has better reading
and writing performance than `OnHeapColumnVector` and `OffHeapColumnVector`.
From the microbench results, the performance improvement is obvious for
`StringType` : the read throughput is increased by about 2 times, and the
write throughput is increased by more than 100 times.
3. **Memory saving**: `ConstantColumnVector` saves more memory than
`OnHeapColumnVector` and `OffHeapColumnVector`, for `UTF8String` type Vector
with length of 4096(default `batchSize`), 'ConstantColumnVector' can save more
than 90% of memory compared with `OnHeapColumnVector`：
- - `ConstantColumnVector` only stores an `UTF8String`
- - `OnHeapColumnVector` needs `arrayOffsets(int[4096])` +
`arrayLengths(int[4096])` + `(UTF8String * 4096)`
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Pass Github Action
- Add new UTs to test the new method introduced by this pr:
`ColumnVectorUtils.fill(ConstantColumnVector col, InternalRow row, int
fieldIdx)`
- Add new micro benchmark to compare the read and write performance of
constant vector(simulate partition column scene) between `OnHeapColumnVector`,
`OffHeapColumnVector` and `ConstantColumnVector`
Closes #36616 from LuciferYang/SPARK-39231.
Authored-by: yangjie01 <[email protected]>
Signed-off-by: Chao Sun <[email protected]>
---
...ConstantColumnVectorBenchmark-jdk11-results.txt | 280 +++++++++++++++++++
...ConstantColumnVectorBenchmark-jdk17-results.txt | 280 +++++++++++++++++++
.../ConstantColumnVectorBenchmark-results.txt | 280 +++++++++++++++++++
.../parquet/VectorizedParquetRecordReader.java | 21 +-
.../execution/vectorized/ColumnVectorUtils.java | 89 +++++++
.../execution/vectorized/ConstantColumnVector.java | 13 +
.../datasources/parquet/ParquetFileFormat.scala | 6 +-
.../benchmark/ConstantColumnVectorBenchmark.scala | 296 +++++++++++++++++++++
.../vectorized/ColumnVectorUtilsSuite.scala | 163 ++++++++++++
9 files changed, 1414 insertions(+), 14 deletions(-)
diff --git
a/sql/core/benchmarks/ConstantColumnVectorBenchmark-jdk11-results.txt
b/sql/core/benchmarks/ConstantColumnVectorBenchmark-jdk11-results.txt
new file mode 100644
index 00000000000..775f5499bf8
--- /dev/null
+++ b/sql/core/benchmarks/ConstantColumnVectorBenchmark-jdk11-results.txt
@@ -0,0 +1,280 @@
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 583637.9 0.0 1.0X
+OnHeapColumnVector 3638 3638
0 112.6 8.9 0.0X
+OffHeapColumnVector 4601 4602
0 89.0 11.2 0.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2 2
0 266508.4 0.0 1.0X
+OnHeapColumnVector 4721 4721
0 86.8 11.5 0.0X
+OffHeapColumnVector 6553 6553
0 62.5 16.0 0.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2 2
0 266508.9 0.0 1.0X
+OnHeapColumnVector 5220 5224
6 78.5 12.7 0.0X
+OffHeapColumnVector 6510 6516
8 62.9 15.9 0.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 15: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 583804.3 0.0 1.0X
+OnHeapColumnVector 4747 4747
0 86.3 11.6 0.0X
+OffHeapColumnVector 7055 7057
3 58.1 17.2 0.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2 2
0 266508.6 0.0 1.0X
+OnHeapColumnVector 4929 4930
0 83.1 12.0 0.0X
+OffHeapColumnVector 6588 6589
1 62.2 16.1 0.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 30: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2 2
0 266508.4 0.0 1.0X
+OnHeapColumnVector 5300 5301
2 77.3 12.9 0.0X
+OffHeapColumnVector 6788 6790
2 60.3 16.6 0.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 943120.4 0.0 1.0X
+OnHeapColumnVector 10 10
0 39537.6 0.0 0.0X
+OffHeapColumnVector 139 139
0 2947.3 0.3 0.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 766174.6 0.0 1.0X
+OnHeapColumnVector 43 45
1 9504.2 0.1 0.0X
+OffHeapColumnVector 156 158
1 2622.8 0.4 0.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 817233.7 0.0 1.0X
+OnHeapColumnVector 10 10
0 40738.5 0.0 0.0X
+OffHeapColumnVector 139 139
0 2944.1 0.3 0.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 681073.0 0.0 1.0X
+OnHeapColumnVector 39 42
1 10369.7 0.1 0.0X
+OffHeapColumnVector 156 158
1 2633.1 0.4 0.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1190 1192
2 344.1 2.9 1.0X
+OnHeapColumnVector 2265 2268
4 180.8 5.5 0.5X
+OffHeapColumnVector 4599 4605
8 89.1 11.2 0.3X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1192 1192
0 343.5 2.9 1.0X
+OnHeapColumnVector 5648 5650
2 72.5 13.8 0.2X
+OffHeapColumnVector 4608 4609
1 88.9 11.3 0.3X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1192 1194
3 343.7 2.9 1.0X
+OnHeapColumnVector 5840 5844
6 70.1 14.3 0.2X
+OffHeapColumnVector 4602 4610
11 89.0 11.2 0.3X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 15: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1199 1202
5 341.6 2.9 1.0X
+OnHeapColumnVector 5634 5636
4 72.7 13.8 0.2X
+OffHeapColumnVector 4600 4603
4 89.0 11.2 0.3X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1193 1194
1 343.2 2.9 1.0X
+OnHeapColumnVector 5624 5629
6 72.8 13.7 0.2X
+OffHeapColumnVector 4599 4601
3 89.1 11.2 0.3X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 30: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1192 1193
1 343.7 2.9 1.0X
+OnHeapColumnVector 5623 5625
3 72.8 13.7 0.2X
+OffHeapColumnVector 4611 4617
8 88.8 11.3 0.3X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1161 1161
0 352.8 2.8 1.0X
+OnHeapColumnVector 1722 1728
8 237.9 4.2 0.7X
+OffHeapColumnVector 2839 2842
4 144.3 6.9 0.4X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1351 1353
2 303.1 3.3 1.0X
+OnHeapColumnVector 1598 1599
0 256.2 3.9 0.8X
+OffHeapColumnVector 2842 2845
4 144.1 6.9 0.5X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1163 1164
2 352.2 2.8 1.0X
+OnHeapColumnVector 1361 1363
2 300.9 3.3 0.9X
+OffHeapColumnVector 2522 2525
4 162.4 6.2 0.5X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1214 1214
0 337.5 3.0 1.0X
+OnHeapColumnVector 1478 1482
6 277.1 3.6 0.8X
+OffHeapColumnVector 2021 2021
0 202.7 4.9 0.6X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 1: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-----------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2418
2420 3 169.4 5.9 1.0X
+OnHeapColumnVector 5865
5867 2 69.8 14.3 0.4X
+OffHeapColumnVector 4640
4645 7 88.3 11.3 0.5X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 5: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-----------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2365
2366 1 173.2 5.8 1.0X
+OnHeapColumnVector 5852
5852 1 70.0 14.3 0.4X
+OffHeapColumnVector 4639
4641 2 88.3 11.3 0.5X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 10: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2365
2370 7 173.2 5.8 1.0X
+OnHeapColumnVector 5863
5864 0 69.9 14.3 0.4X
+OffHeapColumnVector 4639
4642 4 88.3 11.3 0.5X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 15: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2364
2365 1 173.3 5.8 1.0X
+OnHeapColumnVector 5842
5844 2 70.1 14.3 0.4X
+OffHeapColumnVector 4636
4642 9 88.4 11.3 0.5X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 20: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2363
2363 1 173.4 5.8 1.0X
+OnHeapColumnVector 5860
5864 6 69.9 14.3 0.4X
+OffHeapColumnVector 4646
4646 0 88.2 11.3 0.5X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 30: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2365
2367 3 173.2 5.8 1.0X
+OnHeapColumnVector 5850
5851 1 70.0 14.3 0.4X
+OffHeapColumnVector 4640
4643 4 88.3 11.3 0.5X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1338 1338
0 306.2 3.3 1.0X
+OnHeapColumnVector 2916 2917
0 140.4 7.1 0.5X
+OffHeapColumnVector 2917 2917
0 140.4 7.1 0.5X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2807 2809
2 145.9 6.9 1.0X
+OnHeapColumnVector 3230 3232
2 126.8 7.9 0.9X
+OffHeapColumnVector 2715 2715
0 150.9 6.6 1.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1752 1753
1 233.7 4.3 1.0X
+OnHeapColumnVector 2272 2284
18 180.3 5.5 0.8X
+OffHeapColumnVector 2012 2012
0 203.6 4.9 0.9X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2015 2024
12 203.2 4.9 1.0X
+OnHeapColumnVector 2414 2415
1 169.7 5.9 0.8X
+OffHeapColumnVector 2023 2024
1 202.4 4.9 1.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test isNull with StringType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1188 1188
0 344.7 2.9 1.0X
+OnHeapColumnVector 1456 1456
0 281.3 3.6 0.8X
+OffHeapColumnVector 1243 1244
1 329.5 3.0 1.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test isNull with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1188 1195
9 344.7 2.9 1.0X
+OnHeapColumnVector 1451 1453
3 282.4 3.5 0.8X
+OffHeapColumnVector 1243 1244
1 329.4 3.0 1.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test isNull with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1195 1197
4 342.8 2.9 1.0X
+OnHeapColumnVector 1456 1457
1 281.3 3.6 0.8X
+OffHeapColumnVector 1239 1244
7 330.6 3.0 1.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test isNull with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1192 1192
1 343.7 2.9 1.0X
+OnHeapColumnVector 1458 1461
4 280.9 3.6 0.8X
+OffHeapColumnVector 1242 1245
5 329.9 3.0 1.0X
+
+OpenJDK 64-Bit Server VM 11.0.15+10-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test isNull with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1190 1191
2 344.2 2.9 1.0X
+OnHeapColumnVector 1454 1458
5 281.7 3.5 0.8X
+OffHeapColumnVector 1237 1238
1 331.0 3.0 1.0X
+
diff --git
a/sql/core/benchmarks/ConstantColumnVectorBenchmark-jdk17-results.txt
b/sql/core/benchmarks/ConstantColumnVectorBenchmark-jdk17-results.txt
new file mode 100644
index 00000000000..2c74954fbe1
--- /dev/null
+++ b/sql/core/benchmarks/ConstantColumnVectorBenchmark-jdk17-results.txt
@@ -0,0 +1,280 @@
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write with StringType, row length = 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 567396.3 0.0 1.0X
+OnHeapColumnVector 2388 2389
2 171.6 5.8 0.0X
+OffHeapColumnVector 3665 3682
24 111.7 8.9 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write with StringType, row length = 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 638508.5 0.0 1.0X
+OnHeapColumnVector 3545 3552
10 115.6 8.7 0.0X
+OffHeapColumnVector 5482 5483
1 74.7 13.4 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write with StringType, row length = 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 638508.5 0.0 1.0X
+OnHeapColumnVector 3586 3586
0 114.2 8.8 0.0X
+OffHeapColumnVector 6296 6312
22 65.1 15.4 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write with StringType, row length = 15: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 638508.5 0.0 1.0X
+OnHeapColumnVector 4251 4252
2 96.4 10.4 0.0X
+OffHeapColumnVector 6586 6589
4 62.2 16.1 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write with StringType, row length = 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 638508.5 0.0 1.0X
+OnHeapColumnVector 4270 4274
4 95.9 10.4 0.0X
+OffHeapColumnVector 6698 6698
0 61.2 16.4 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write with StringType, row length = 30: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 638508.5 0.0 1.0X
+OnHeapColumnVector 4327 4328
2 94.7 10.6 0.0X
+OffHeapColumnVector 6746 6753
10 60.7 16.5 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 1135264.4 0.0 1.0X
+OnHeapColumnVector 12 12
0 34578.8 0.0 0.0X
+OffHeapColumnVector 85 85
0 4798.8 0.2 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 851389.3 0.0 1.0X
+OnHeapColumnVector 22 22
0 18511.9 0.1 0.0X
+OffHeapColumnVector 85 85
0 4804.0 0.2 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 1
0 928806.6 0.0 1.0X
+OnHeapColumnVector 12 12
0 34709.8 0.0 0.0X
+OffHeapColumnVector 85 85
0 4814.5 0.2 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 1
0 851389.3 0.0 1.0X
+OnHeapColumnVector 22 23
0 18516.6 0.1 0.0X
+OffHeapColumnVector 85 85
0 4804.2 0.2 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test read with StringType, row length = 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 3028 3035
10 135.3 7.4 0.0X
+OffHeapColumnVector 2359 2365
8 173.6 5.8 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test read with StringType, row length = 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 3038 3042
5 134.8 7.4 0.0X
+OffHeapColumnVector 2368 2536
238 172.9 5.8 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test read with StringType, row length = 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 3028 3030
3 135.3 7.4 0.0X
+OffHeapColumnVector 2358 2365
9 173.7 5.8 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test read with StringType, row length = 15: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 3023 3149
177 135.5 7.4 0.0X
+OffHeapColumnVector 2349 2352
3 174.3 5.7 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test read with StringType, row length = 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 3058 3169
157 133.9 7.5 0.0X
+OffHeapColumnVector 2358 2358
0 173.7 5.8 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test read with StringType, row length = 30: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 3064 3172
151 133.7 7.5 0.0X
+OffHeapColumnVector 2358 2361
5 173.7 5.8 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test read with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 2553647.8 0.0 1.0X
+OnHeapColumnVector 1 1
0 510791.3 0.0 0.2X
+OffHeapColumnVector 1277 1277
0 320.9 3.1 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test read with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 515808.6 0.0 1.0X
+OnHeapColumnVector 1 1
0 785886.3 0.0 1.5X
+OffHeapColumnVector 1194 1195
1 343.0 2.9 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test read with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 1135264.4 0.0 1.0X
+OnHeapColumnVector 1 1
0 408622.6 0.0 0.4X
+OffHeapColumnVector 1186 1186
0 345.3 2.9 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test read with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 464350.1 0.0 1.0X
+OnHeapColumnVector 3007 3009
4 136.2 7.3 0.0X
+OffHeapColumnVector 1273 1274
1 321.7 3.1 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write and read with StringType, row length = 1: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-----------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1
1 0 510790.7 0.0 1.0X
+OnHeapColumnVector 3510
3512 4 116.7 8.6 0.0X
+OffHeapColumnVector 2528
2528 0 162.0 6.2 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write and read with StringType, row length = 5: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-----------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1248
1249 2 328.2 3.0 1.0X
+OnHeapColumnVector 3515
3518 4 116.5 8.6 0.4X
+OffHeapColumnVector 2528
2531 3 162.0 6.2 0.5X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write and read with StringType, row length = 10: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1244
1250 8 329.3 3.0 1.0X
+OnHeapColumnVector 3515
3515 0 116.5 8.6 0.4X
+OffHeapColumnVector 2525
2528 3 162.2 6.2 0.5X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write and read with StringType, row length = 15: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1244
1245 1 329.2 3.0 1.0X
+OnHeapColumnVector 3507
3510 5 116.8 8.6 0.4X
+OffHeapColumnVector 2528
2528 1 162.0 6.2 0.5X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write and read with StringType, row length = 20: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1240
1240 1 330.4 3.0 1.0X
+OnHeapColumnVector 3506
3513 11 116.8 8.6 0.4X
+OffHeapColumnVector 2529
2530 2 162.0 6.2 0.5X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write and read with StringType, row length = 30: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1243
1243 0 329.4 3.0 1.0X
+OnHeapColumnVector 3504
3505 2 116.9 8.6 0.4X
+OffHeapColumnVector 2524
2530 8 162.3 6.2 0.5X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write and read with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 1459739.6 0.0 1.0X
+OnHeapColumnVector 1180 1181
0 347.0 2.9 0.0X
+OffHeapColumnVector 1277 1278
1 320.6 3.1 0.0X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write and read with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 965 971
7 424.4 2.4 1.0X
+OnHeapColumnVector 1254 1255
1 326.6 3.1 0.8X
+OffHeapColumnVector 1185 1186
0 345.6 2.9 0.8X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write and read with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 954 958
4 429.3 2.3 1.0X
+OnHeapColumnVector 1264 1266
2 323.9 3.1 0.8X
+OffHeapColumnVector 1183 1186
3 346.1 2.9 0.8X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test write and read with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 3282 3283
2 124.8 8.0 1.0X
+OnHeapColumnVector 1189 1191
2 344.4 2.9 2.8X
+OffHeapColumnVector 1274 1274
0 321.5 3.1 2.6X
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test isNull with StringType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 1 1
0 552174.9 0.0 0.0X
+OffHeapColumnVector 0 0
0 Infinity 0.0 NaNX
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test isNull with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 0 0
0 2078154.0 0.0 0.0X
+OffHeapColumnVector 0 0
0 Infinity 0.0 NaNX
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test isNull with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 0 0
0 2079198.4 0.0 0.0X
+OffHeapColumnVector 0 0
0 Infinity 0.0 NaNX
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test isNull with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 0 0
0 2079208.9 0.0 0.0X
+OffHeapColumnVector 0 0
0 Infinity 0.0 NaNX
+
+OpenJDK 64-Bit Server VM 17.0.3+7-LTS on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz
+Test isNull with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 Infinity 0.0 NaNX
+OnHeapColumnVector 0 0
0 2079208.9 0.0 0.0X
+OffHeapColumnVector 0 0
0 Infinity 0.0 NaNX
+
diff --git a/sql/core/benchmarks/ConstantColumnVectorBenchmark-results.txt
b/sql/core/benchmarks/ConstantColumnVectorBenchmark-results.txt
new file mode 100644
index 00000000000..5101ed7aa1b
--- /dev/null
+++ b/sql/core/benchmarks/ConstantColumnVectorBenchmark-results.txt
@@ -0,0 +1,280 @@
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 681069.6 0.0 1.0X
+OnHeapColumnVector 3073 3073
0 133.3 7.5 0.0X
+OffHeapColumnVector 3502 3503
2 117.0 8.6 0.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 680956.3 0.0 1.0X
+OnHeapColumnVector 3773 3773
0 108.6 9.2 0.0X
+OffHeapColumnVector 4834 4839
7 84.7 11.8 0.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 681069.6 0.0 1.0X
+OnHeapColumnVector 3980 3980
0 102.9 9.7 0.0X
+OffHeapColumnVector 4706 4706
1 87.0 11.5 0.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 15: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 681069.6 0.0 1.0X
+OnHeapColumnVector 4000 4001
1 102.4 9.8 0.0X
+OffHeapColumnVector 4834 4835
1 84.7 11.8 0.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 681069.6 0.0 1.0X
+OnHeapColumnVector 4400 4400
0 93.1 10.7 0.0X
+OffHeapColumnVector 4593 4594
2 89.2 11.2 0.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with StringType, row length = 30: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+---------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 681073.0 0.0 1.0X
+OnHeapColumnVector 4781 4781
0 85.7 11.7 0.0X
+OffHeapColumnVector 5185 5187
3 79.0 12.7 0.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 1021433.6 0.0 1.0X
+OnHeapColumnVector 70 70
0 5845.6 0.2 0.0X
+OffHeapColumnVector 139 139
0 2937.7 0.3 0.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 0 0
0 942911.9 0.0 1.0X
+OnHeapColumnVector 61 63
1 6759.3 0.1 0.0X
+OffHeapColumnVector 157 158
1 2612.7 0.4 0.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 817232.1 0.0 1.0X
+OnHeapColumnVector 70 70
0 5819.5 0.2 0.0X
+OffHeapColumnVector 139 139
0 2944.1 0.3 0.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1 1
0 766171.7 0.0 1.0X
+OnHeapColumnVector 55 59
1 7504.8 0.1 0.0X
+OffHeapColumnVector 154 157
1 2662.4 0.4 0.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 1: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1188 1190
3 344.7 2.9 1.0X
+OnHeapColumnVector 2907 2907
1 140.9 7.1 0.4X
+OffHeapColumnVector 4290 4295
7 95.5 10.5 0.3X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 5: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1168 1172
6 350.6 2.9 1.0X
+OnHeapColumnVector 5766 5770
6 71.0 14.1 0.2X
+OffHeapColumnVector 4278 4281
5 95.7 10.4 0.3X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 10: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1163 1165
4 352.3 2.8 1.0X
+OnHeapColumnVector 5761 5762
2 71.1 14.1 0.2X
+OffHeapColumnVector 4233 4249
23 96.8 10.3 0.3X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 15: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1158 1163
6 353.7 2.8 1.0X
+OnHeapColumnVector 5750 5754
5 71.2 14.0 0.2X
+OffHeapColumnVector 4231 4232
2 96.8 10.3 0.3X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 20: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1162 1164
2 352.4 2.8 1.0X
+OnHeapColumnVector 6277 6278
2 65.3 15.3 0.2X
+OffHeapColumnVector 4220 4236
22 97.1 10.3 0.3X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with StringType, row length = 30: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+--------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1159 1160
1 353.4 2.8 1.0X
+OnHeapColumnVector 6281 6284
4 65.2 15.3 0.2X
+OffHeapColumnVector 4233 4245
17 96.8 10.3 0.3X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1361 1362
1 300.9 3.3 1.0X
+OnHeapColumnVector 1807 1808
3 226.7 4.4 0.8X
+OffHeapColumnVector 2808 2811
4 145.8 6.9 0.5X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1315 1317
3 311.5 3.2 1.0X
+OnHeapColumnVector 2318 2318
1 176.7 5.7 0.6X
+OffHeapColumnVector 2512 2513
1 163.0 6.1 0.5X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1177 1179
2 347.9 2.9 1.0X
+OnHeapColumnVector 1334 1335
1 307.0 3.3 0.9X
+OffHeapColumnVector 1837 1838
1 222.9 4.5 0.6X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test read with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1178 1178
0 347.8 2.9 1.0X
+OnHeapColumnVector 1330 1331
1 307.9 3.2 0.9X
+OffHeapColumnVector 1835 1837
2 223.2 4.5 0.6X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 1: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-----------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1164
1166 2 351.8 2.8 1.0X
+OnHeapColumnVector 5884
5884 0 69.6 14.4 0.2X
+OffHeapColumnVector 4744
4746 4 86.3 11.6 0.2X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 5: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+-----------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1162
1164 4 352.6 2.8 1.0X
+OnHeapColumnVector 5545
5546 2 73.9 13.5 0.2X
+OffHeapColumnVector 4754
4756 4 86.2 11.6 0.2X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 10: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1164
1165 2 352.0 2.8 1.0X
+OnHeapColumnVector 5539
5541 2 73.9 13.5 0.2X
+OffHeapColumnVector 4745
4749 6 86.3 11.6 0.2X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 15: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1168
1168 1 350.8 2.9 1.0X
+OnHeapColumnVector 5541
5543 2 73.9 13.5 0.2X
+OffHeapColumnVector 4755
4756 2 86.1 11.6 0.2X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 20: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1163
1163 1 352.3 2.8 1.0X
+OnHeapColumnVector 5541
5542 2 73.9 13.5 0.2X
+OffHeapColumnVector 4755
4758 4 86.1 11.6 0.2X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with StringType, row length = 30: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1168
1168 0 350.8 2.9 1.0X
+OnHeapColumnVector 5538
5539 3 74.0 13.5 0.2X
+OffHeapColumnVector 4757
4758 0 86.1 11.6 0.2X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1099 1102
4 372.6 2.7 1.0X
+OnHeapColumnVector 3025 3026
1 135.4 7.4 0.4X
+OffHeapColumnVector 2800 2804
5 146.3 6.8 0.4X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 2466 2469
3 166.1 6.0 1.0X
+OnHeapColumnVector 2778 2779
1 147.4 6.8 0.9X
+OffHeapColumnVector 2511 2514
5 163.1 6.1 1.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1841 1843
3 222.5 4.5 1.0X
+OnHeapColumnVector 2237 2238
2 183.1 5.5 0.8X
+OffHeapColumnVector 1791 1793
4 228.8 4.4 1.0X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test write and read with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1645 1647
3 249.0 4.0 1.0X
+OnHeapColumnVector 2223 2225
3 184.2 5.4 0.7X
+OffHeapColumnVector 1767 1769
3 231.8 4.3 0.9X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test isNull with StringType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1098 1100
2 373.0 2.7 1.0X
+OnHeapColumnVector 1437 1438
2 285.0 3.5 0.8X
+OffHeapColumnVector 1166 1168
3 351.2 2.8 0.9X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test isNull with IntegerType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1097 1101
5 373.2 2.7 1.0X
+OnHeapColumnVector 1432 1433
1 286.1 3.5 0.8X
+OffHeapColumnVector 1171 1173
2 349.7 2.9 0.9X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test isNull with LongType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1103 1109
8 371.2 2.7 1.0X
+OnHeapColumnVector 1440 1443
4 284.5 3.5 0.8X
+OffHeapColumnVector 1177 1178
3 348.1 2.9 0.9X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test isNull with FloatType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1112 1113
1 368.3 2.7 1.0X
+OnHeapColumnVector 1449 1451
2 282.6 3.5 0.8X
+OffHeapColumnVector 1179 1180
1 347.3 2.9 0.9X
+
+OpenJDK 64-Bit Server VM 1.8.0_332-b09 on Linux 5.13.0-1023-azure
+Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
+Test isNull with DoubleType: Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
+------------------------------------------------------------------------------------------------------------------------
+ConstantColumnVector 1109 1110
1 369.4 2.7 1.0X
+OnHeapColumnVector 1451 1451
1 282.4 3.5 0.8X
+OffHeapColumnVector 1175 1179
4 348.5 2.9 0.9X
+
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
index 6a30876a3bc..4d71f3f6689 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
@@ -37,9 +37,9 @@ import org.apache.parquet.schema.Type;
import org.apache.spark.memory.MemoryMode;
import org.apache.spark.sql.catalyst.InternalRow;
import org.apache.spark.sql.execution.vectorized.ColumnVectorUtils;
+import org.apache.spark.sql.execution.vectorized.ConstantColumnVector;
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
-import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
-import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.vectorized.ColumnVector;
import org.apache.spark.sql.vectorized.ColumnarBatch;
import org.apache.spark.sql.types.StructField;
import org.apache.spark.sql.types.StructType;
@@ -243,18 +243,17 @@ public class VectorizedParquetRecordReader extends
SpecificParquetRecordReaderBa
for (StructField f: sparkSchema.fields()) {
batchSchema = batchSchema.add(f);
}
+ int constantColumnLength = 0;
if (partitionColumns != null) {
for (StructField f : partitionColumns.fields()) {
batchSchema = batchSchema.add(f);
}
+ constantColumnLength = partitionColumns.fields().length;
}
- WritableColumnVector[] vectors;
- if (memMode == MemoryMode.OFF_HEAP) {
- vectors = OffHeapColumnVector.allocateColumns(capacity, batchSchema);
- } else {
- vectors = OnHeapColumnVector.allocateColumns(capacity, batchSchema);
- }
+ ColumnVector[] vectors = ColumnVectorUtils.allocateColumns(
+ capacity, batchSchema, memMode == MemoryMode.OFF_HEAP,
constantColumnLength);
+
columnarBatch = new ColumnarBatch(vectors);
columnVectors = new ParquetColumnVector[sparkSchema.fields().length];
@@ -264,14 +263,14 @@ public class VectorizedParquetRecordReader extends
SpecificParquetRecordReaderBa
defaultValue = sparkRequestedSchema.existenceDefaultValues()[i];
}
columnVectors[i] = new
ParquetColumnVector(parquetColumn.children().apply(i),
- vectors[i], capacity, memMode, missingColumns, true, defaultValue);
+ (WritableColumnVector) vectors[i], capacity, memMode, missingColumns,
true, defaultValue);
}
if (partitionColumns != null) {
int partitionIdx = sparkSchema.fields().length;
for (int i = 0; i < partitionColumns.fields().length; i++) {
- ColumnVectorUtils.populate(vectors[i + partitionIdx], partitionValues,
i);
- vectors[i + partitionIdx].setIsConstant();
+ ColumnVectorUtils.populate(
+ (ConstantColumnVector) vectors[i + partitionIdx], partitionValues,
i);
}
}
}
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java
index 353a1282544..c55c59032e6 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java
@@ -30,6 +30,7 @@ import org.apache.spark.sql.Row;
import org.apache.spark.sql.catalyst.InternalRow;
import org.apache.spark.sql.catalyst.util.DateTimeUtils;
import org.apache.spark.sql.types.*;
+import org.apache.spark.sql.vectorized.ColumnVector;
import org.apache.spark.sql.vectorized.ColumnarArray;
import org.apache.spark.sql.vectorized.ColumnarBatch;
import org.apache.spark.sql.vectorized.ColumnarMap;
@@ -105,6 +106,61 @@ public class ColumnVectorUtils {
}
}
+ /**
+ * Populates the value of `row[fieldIdx]` into `ConstantColumnVector`.
+ */
+ public static void populate(ConstantColumnVector col, InternalRow row, int
fieldIdx) {
+ DataType t = col.dataType();
+
+ if (row.isNullAt(fieldIdx)) {
+ col.setNull();
+ } else {
+ if (t == DataTypes.BooleanType) {
+ col.setBoolean(row.getBoolean(fieldIdx));
+ } else if (t == DataTypes.BinaryType) {
+ col.setBinary(row.getBinary(fieldIdx));
+ } else if (t == DataTypes.ByteType) {
+ col.setByte(row.getByte(fieldIdx));
+ } else if (t == DataTypes.ShortType) {
+ col.setShort(row.getShort(fieldIdx));
+ } else if (t == DataTypes.IntegerType) {
+ col.setInt(row.getInt(fieldIdx));
+ } else if (t == DataTypes.LongType) {
+ col.setLong(row.getLong(fieldIdx));
+ } else if (t == DataTypes.FloatType) {
+ col.setFloat(row.getFloat(fieldIdx));
+ } else if (t == DataTypes.DoubleType) {
+ col.setDouble(row.getDouble(fieldIdx));
+ } else if (t == DataTypes.StringType) {
+ UTF8String v = row.getUTF8String(fieldIdx);
+ col.setUtf8String(v);
+ } else if (t instanceof DecimalType) {
+ DecimalType dt = (DecimalType) t;
+ Decimal d = row.getDecimal(fieldIdx, dt.precision(), dt.scale());
+ if (dt.precision() <= Decimal.MAX_INT_DIGITS()) {
+ col.setInt((int)d.toUnscaledLong());
+ } else if (dt.precision() <= Decimal.MAX_LONG_DIGITS()) {
+ col.setLong(d.toUnscaledLong());
+ } else {
+ final BigInteger integer = d.toJavaBigDecimal().unscaledValue();
+ byte[] bytes = integer.toByteArray();
+ col.setBinary(bytes);
+ }
+ } else if (t instanceof CalendarIntervalType) {
+ // The value of `numRows` is irrelevant.
+ col.setCalendarInterval((CalendarInterval) row.get(fieldIdx, t));
+ } else if (t instanceof DateType || t instanceof YearMonthIntervalType) {
+ col.setInt(row.getInt(fieldIdx));
+ } else if (t instanceof TimestampType || t instanceof TimestampNTZType ||
+ t instanceof DayTimeIntervalType) {
+ col.setLong(row.getLong(fieldIdx));
+ } else {
+ throw new RuntimeException(String.format("DataType %s is not
supported" +
+ " in column vectorized reader.", t.sql()));
+ }
+ }
+ }
+
/**
* Returns the array data as the java primitive array.
* For example, an array of IntegerType will return an int[].
@@ -235,4 +291,37 @@ public class ColumnVectorUtils {
batch.setNumRows(n);
return batch;
}
+
+ /**
+ * <b>This method assumes that all constant column are at the end of schema
+ * and `constantColumnLength` represents the number of constant column.<b/>
+ *
+ * This method allocates columns to store elements of each field of the
schema,
+ * the data columns use `OffHeapColumnVector` when `useOffHeap` is true and
+ * use `OnHeapColumnVector` when `useOffHeap` is false, the constant columns
+ * always use `ConstantColumnVector`.
+ *
+ * Capacity is the initial capacity of the vector, and it will grow as
necessary.
+ * Capacity is in number of elements, not number of bytes.
+ */
+ public static ColumnVector[] allocateColumns(
+ int capacity, StructType schema, boolean useOffHeap, int
constantColumnLength) {
+ StructField[] fields = schema.fields();
+ int fieldsLength = fields.length;
+ ColumnVector[] vectors = new ColumnVector[fieldsLength];
+ if (useOffHeap) {
+ for (int i = 0; i < fieldsLength - constantColumnLength; i++) {
+ vectors[i] = new OffHeapColumnVector(capacity, fields[i].dataType());
+ }
+ } else {
+ for (int i = 0; i < fieldsLength - constantColumnLength; i++) {
+ vectors[i] = new OnHeapColumnVector(capacity, fields[i].dataType());
+ }
+ }
+ for (int i = fieldsLength - constantColumnLength; i < fieldsLength; i++) {
+ vectors[i] = new ConstantColumnVector(capacity, fields[i].dataType());
+ }
+ return vectors;
+ }
+
}
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java
index 3a5dea479ca..5095e6b0c9c 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java
@@ -23,6 +23,7 @@ import org.apache.spark.sql.types.*;
import org.apache.spark.sql.vectorized.ColumnVector;
import org.apache.spark.sql.vectorized.ColumnarArray;
import org.apache.spark.sql.vectorized.ColumnarMap;
+import org.apache.spark.unsafe.types.CalendarInterval;
import org.apache.spark.unsafe.types.UTF8String;
/**
@@ -63,6 +64,9 @@ public class ConstantColumnVector extends ColumnVector {
} else if (type instanceof CalendarIntervalType) {
// Three columns. Months as int. Days as Int. Microseconds as Long.
this.childData = new ConstantColumnVector[3];
+ this.childData[0] = new ConstantColumnVector(1, DataTypes.IntegerType);
+ this.childData[1] = new ConstantColumnVector(1, DataTypes.IntegerType);
+ this.childData[2] = new ConstantColumnVector(1, DataTypes.LongType);
} else {
this.childData = null;
}
@@ -294,4 +298,13 @@ public class ConstantColumnVector extends ColumnVector {
public void setChild(int ordinal, ConstantColumnVector value) {
childData[ordinal] = value;
}
+
+ /**
+ * Sets the CalendarInterval `value` for all rows
+ */
+ public void setCalendarInterval(CalendarInterval value) {
+ this.childData[0].setInt(value.months);
+ this.childData[1].setInt(value.days);
+ this.childData[2].setLong(value.microseconds);
+ }
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
index 9534be81928..3349f335841 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
@@ -47,7 +47,7 @@ import org.apache.spark.sql.catalyst.util.DateTimeUtils
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.execution.WholeStageCodegenExec
import org.apache.spark.sql.execution.datasources._
-import org.apache.spark.sql.execution.vectorized.{OffHeapColumnVector,
OnHeapColumnVector}
+import org.apache.spark.sql.execution.vectorized.{ConstantColumnVector,
OffHeapColumnVector, OnHeapColumnVector}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.sources._
import org.apache.spark.sql.types._
@@ -176,13 +176,13 @@ class ParquetFileFormat
requiredSchema: StructType,
partitionSchema: StructType,
sqlConf: SQLConf): Option[Seq[String]] = {
- Option(Seq.fill(requiredSchema.fields.length +
partitionSchema.fields.length)(
+ Option(Seq.fill(requiredSchema.fields.length)(
if (!sqlConf.offHeapColumnVectorEnabled) {
classOf[OnHeapColumnVector].getName
} else {
classOf[OffHeapColumnVector].getName
}
- ))
+ ) ++
Seq.fill(partitionSchema.fields.length)(classOf[ConstantColumnVector].getName))
}
override def isSplitable(
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ConstantColumnVectorBenchmark.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ConstantColumnVectorBenchmark.scala
new file mode 100644
index 00000000000..9e4902f2fb5
--- /dev/null
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/ConstantColumnVectorBenchmark.scala
@@ -0,0 +1,296 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.benchmark
+
+import org.apache.commons.lang3.RandomStringUtils
+
+import org.apache.spark.benchmark.Benchmark
+import org.apache.spark.benchmark.BenchmarkBase
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.vectorized.{ColumnVectorUtils,
ConstantColumnVector, OffHeapColumnVector, OnHeapColumnVector}
+import org.apache.spark.sql.types._
+import org.apache.spark.sql.vectorized.ColumnVector
+import org.apache.spark.unsafe.UTF8StringBuilder
+
+/**
+ * Benchmark for constant ColumnVector read and write,
+ * include `ConstantColumnVector`, `OnHeapColumnVector` and
`OffHeapColumnVector`
+ * To run this benchmark:
+ * {{{
+ * 1. without sbt: bin/spark-submit --class <this class>
+ * --jars <spark core test jar>,<spark catalyst test jar> <spark sql
test jar>
+ * 2. build/sbt "sql/test:runMain <this class>"
+ * 3. generate result: SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt
"sql/test:runMain <this class>"
+ * Results will be written to
"benchmarks/ConstantColumnVectorBenchmark-results.txt".
+ * }}}
+ */
+object ConstantColumnVectorBenchmark extends BenchmarkBase {
+
+ private def readValues(dataType: DataType, batchSize: Int, vector:
ColumnVector): Unit = {
+ dataType match {
+ case IntegerType =>
+ (0 until batchSize).foreach(i => vector.getInt(i))
+ case LongType =>
+ (0 until batchSize).foreach(i => vector.getLong(i))
+ case FloatType =>
+ (0 until batchSize).foreach(i => vector.getFloat(i))
+ case DoubleType =>
+ (0 until batchSize).foreach(i => vector.getDouble(i))
+ case StringType =>
+ (0 until batchSize).foreach(i => vector.getUTF8String(i))
+ }
+ }
+
+ def testWrite(
+ valuesPerIteration: Int,
+ batchSize: Int,
+ dataType: DataType,
+ row: InternalRow): Unit = {
+
+ val onHeapColumnVector = new OnHeapColumnVector(batchSize, dataType)
+ val offHeapColumnVector = new OffHeapColumnVector(batchSize, dataType)
+ val constantColumnVector = new ConstantColumnVector(batchSize, dataType)
+
+ val other = if (dataType == StringType) {
+ s", row length = ${row.getUTF8String(0).toString.length}"
+ } else {
+ ""
+ }
+
+ val benchmark = new Benchmark(
+ s"Test write with $dataType$other",
+ valuesPerIteration * batchSize,
+ output = output)
+
+ benchmark.addCase("ConstantColumnVector") { _: Int =>
+ for (_ <- 0 until valuesPerIteration) {
+ ColumnVectorUtils.populate(constantColumnVector, row, 0)
+ }
+ }
+
+ benchmark.addCase("OnHeapColumnVector") { _: Int =>
+ for (_ <- 0 until valuesPerIteration) {
+ onHeapColumnVector.reset()
+ ColumnVectorUtils.populate(onHeapColumnVector, row, 0)
+ }
+ }
+
+ benchmark.addCase("OffHeapColumnVector") { _: Int =>
+ for (_ <- 0 until valuesPerIteration) {
+ offHeapColumnVector.reset()
+ ColumnVectorUtils.populate(offHeapColumnVector, row, 0)
+ }
+ }
+
+ benchmark.run()
+ onHeapColumnVector.close()
+ offHeapColumnVector.close()
+ constantColumnVector.close()
+ }
+
+ def testRead(
+ valuesPerIteration: Int,
+ batchSize: Int,
+ dataType: DataType,
+ row: InternalRow): Unit = {
+
+ val onHeapColumnVector = new OnHeapColumnVector(batchSize, dataType)
+ val offHeapColumnVector = new OffHeapColumnVector(batchSize, dataType)
+ val constantColumnVector = new ConstantColumnVector(batchSize, dataType)
+
+ onHeapColumnVector.reset()
+ ColumnVectorUtils.populate(onHeapColumnVector, row, 0)
+ offHeapColumnVector.reset()
+ ColumnVectorUtils.populate(offHeapColumnVector, row, 0)
+ ColumnVectorUtils.populate(constantColumnVector, row, 0)
+
+ val other = if (dataType == StringType) {
+ s", row length = ${row.getUTF8String(0).toString.length}"
+ } else {
+ ""
+ }
+
+ val benchmark = new Benchmark(
+ s"Test read with $dataType$other",
+ valuesPerIteration * batchSize,
+ output = output)
+
+ benchmark.addCase("ConstantColumnVector") { _: Int =>
+ for (_ <- 0 until valuesPerIteration) {
+ readValues(dataType, batchSize, constantColumnVector)
+ }
+ }
+
+ benchmark.addCase("OnHeapColumnVector") { _: Int =>
+ for (_ <- 0 until valuesPerIteration) {
+ readValues(dataType, batchSize, onHeapColumnVector)
+ }
+ }
+
+ benchmark.addCase("OffHeapColumnVector") { _: Int =>
+ for (_ <- 0 until valuesPerIteration) {
+ readValues(dataType, batchSize, offHeapColumnVector)
+ }
+ }
+
+ benchmark.run()
+ onHeapColumnVector.close()
+ offHeapColumnVector.close()
+ constantColumnVector.close()
+ }
+
+ def testWriteAndRead(
+ valuesPerIteration: Int,
+ batchSize: Int,
+ dataType: DataType,
+ row: InternalRow): Unit = {
+
+ val onHeapColumnVector = new OnHeapColumnVector(batchSize, dataType)
+ val offHeapColumnVector = new OffHeapColumnVector(batchSize, dataType)
+ val constantColumnVector = new ConstantColumnVector(batchSize, dataType)
+
+ val other = if (dataType == StringType) {
+ s", row length = ${row.getUTF8String(0).toString.length}"
+ } else {
+ ""
+ }
+
+ val benchmark = new Benchmark(
+ s"Test write and read with $dataType$other",
+ valuesPerIteration * batchSize,
+ output = output)
+
+ benchmark.addCase("ConstantColumnVector") { _: Int =>
+ ColumnVectorUtils.populate(constantColumnVector, row, 0)
+ for (_ <- 0 until valuesPerIteration) {
+ readValues(dataType, batchSize, constantColumnVector)
+ }
+ }
+
+ benchmark.addCase("OnHeapColumnVector") { _: Int =>
+ onHeapColumnVector.reset()
+ ColumnVectorUtils.populate(onHeapColumnVector, row, 0)
+ for (_ <- 0 until valuesPerIteration) {
+ readValues(dataType, batchSize, onHeapColumnVector)
+ }
+ }
+
+ benchmark.addCase("OffHeapColumnVector") { _: Int =>
+ offHeapColumnVector.reset()
+ ColumnVectorUtils.populate(offHeapColumnVector, row, 0)
+ for (_ <- 0 until valuesPerIteration) {
+ readValues(dataType, batchSize, offHeapColumnVector)
+ }
+ }
+
+ benchmark.run()
+ onHeapColumnVector.close()
+ offHeapColumnVector.close()
+ constantColumnVector.close()
+ }
+
+ def testIsNull(
+ valuesPerIteration: Int,
+ batchSize: Int,
+ dataType: DataType): Unit = {
+
+ val onHeapColumnVector = new OnHeapColumnVector(batchSize, dataType)
+ val offHeapColumnVector = new OffHeapColumnVector(batchSize, dataType)
+ val constantColumnVector = new ConstantColumnVector(batchSize, dataType)
+
+ onHeapColumnVector.putNulls(0, batchSize)
+ offHeapColumnVector.putNulls(0, batchSize)
+ constantColumnVector.setNull()
+
+ val benchmark = new Benchmark(
+ s"Test isNull with $dataType",
+ valuesPerIteration * batchSize,
+ output = output)
+
+ benchmark.addCase("ConstantColumnVector") { _: Int =>
+ for (_ <- 0 until valuesPerIteration) {
+ (0 until batchSize).foreach(constantColumnVector.isNullAt)
+ }
+ }
+
+ benchmark.addCase("OnHeapColumnVector") { _: Int =>
+ for (i <- 0 until valuesPerIteration) {
+ (0 until batchSize).foreach(onHeapColumnVector.isNullAt)
+ }
+ }
+
+ benchmark.addCase("OffHeapColumnVector") { _: Int =>
+ for (i <- 0 until valuesPerIteration) {
+ (0 until batchSize).foreach(offHeapColumnVector.isNullAt)
+ }
+ }
+
+ benchmark.run()
+ onHeapColumnVector.close()
+ offHeapColumnVector.close()
+ constantColumnVector.close()
+ }
+
+ override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
+ val valuesPerIteration = 100000
+ val batchSize = 4096
+
+ Seq(1, 5, 10, 15, 20, 30).foreach { length =>
+ val builder = new UTF8StringBuilder()
+ builder.append(RandomStringUtils.random(length))
+ val row = InternalRow(builder.build())
+ testWrite(valuesPerIteration, batchSize, StringType, row)
+ }
+
+ testWrite(valuesPerIteration, batchSize, IntegerType, InternalRow(100))
+ testWrite(valuesPerIteration, batchSize, LongType, InternalRow(100L))
+ testWrite(valuesPerIteration, batchSize, FloatType, InternalRow(100F))
+ testWrite(valuesPerIteration, batchSize, DoubleType, InternalRow(100D))
+
+
+ Seq(1, 5, 10, 15, 20, 30).foreach { length =>
+ val builder = new UTF8StringBuilder()
+ builder.append(RandomStringUtils.random(length))
+ val row = InternalRow(builder.build())
+ testRead(valuesPerIteration, batchSize, StringType, row)
+ }
+
+ testRead(valuesPerIteration, batchSize, IntegerType, InternalRow(100))
+ testRead(valuesPerIteration, batchSize, LongType, InternalRow(100L))
+ testRead(valuesPerIteration, batchSize, FloatType, InternalRow(100F))
+ testRead(valuesPerIteration, batchSize, DoubleType, InternalRow(100D))
+
+ Seq(1, 5, 10, 15, 20, 30).foreach { length =>
+ val builder = new UTF8StringBuilder()
+ builder.append(RandomStringUtils.random(length))
+ val row = InternalRow(builder.build())
+ testWriteAndRead(valuesPerIteration, batchSize, StringType, row)
+ }
+
+ testWriteAndRead(valuesPerIteration, batchSize, IntegerType,
InternalRow(100))
+ testWriteAndRead(valuesPerIteration, batchSize, LongType,
InternalRow(100L))
+ testWriteAndRead(valuesPerIteration, batchSize, FloatType,
InternalRow(100F))
+ testWriteAndRead(valuesPerIteration, batchSize, DoubleType,
InternalRow(100D))
+
+ testIsNull(valuesPerIteration, batchSize, StringType)
+ testIsNull(valuesPerIteration, batchSize, IntegerType)
+ testIsNull(valuesPerIteration, batchSize, LongType)
+ testIsNull(valuesPerIteration, batchSize, FloatType)
+ testIsNull(valuesPerIteration, batchSize, DoubleType)
+ }
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnVectorUtilsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnVectorUtilsSuite.scala
new file mode 100644
index 00000000000..6205484d6be
--- /dev/null
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnVectorUtilsSuite.scala
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.vectorized
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.CalendarInterval
+import org.apache.spark.unsafe.types.UTF8String
+
+class ColumnVectorUtilsSuite extends SparkFunSuite {
+
+ private def testConstantColumnVector(name: String, size: Int, dt: DataType)
+ (f: ConstantColumnVector => Unit): Unit = {
+ test(name) {
+ val vector = new ConstantColumnVector(size, dt)
+ f(vector)
+ vector.close()
+ }
+ }
+
+ testConstantColumnVector("fill null", 10, IntegerType) { vector =>
+
+ ColumnVectorUtils.populate(vector, InternalRow(null), 0)
+
+ assert(vector.hasNull)
+ assert(vector.numNulls() == 10)
+ (0 until 10).foreach { i =>
+ assert(vector.isNullAt(i))
+ }
+
+ vector.setNotNull()
+ assert(!vector.hasNull)
+ assert(vector.numNulls() == 0)
+ (0 until 10).foreach { i =>
+ assert(!vector.isNullAt(i))
+ }
+ }
+
+ testConstantColumnVector("fill boolean", 10, BooleanType) { vector =>
+ ColumnVectorUtils.populate(vector, InternalRow(true), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getBoolean(i))
+ }
+ }
+
+ testConstantColumnVector("fill byte", 10, ByteType) { vector =>
+ ColumnVectorUtils.populate(vector, InternalRow(3.toByte), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getByte(i) == 3.toByte)
+ }
+ }
+
+ testConstantColumnVector("fill short", 10, ShortType) { vector =>
+ ColumnVectorUtils.populate(vector, InternalRow(3.toShort), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getShort(i) == 3.toShort)
+ }
+ }
+
+ testConstantColumnVector("fill int", 10, IntegerType) { vector =>
+ ColumnVectorUtils.populate(vector, InternalRow(3), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getInt(i) == 3)
+ }
+ }
+
+ testConstantColumnVector("fill long", 10, LongType) { vector =>
+ ColumnVectorUtils.populate(vector, InternalRow(3L), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getLong(i) == 3L)
+ }
+ }
+
+ testConstantColumnVector("fill float", 10, FloatType) { vector =>
+ ColumnVectorUtils.populate(vector, InternalRow(3.toFloat), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getFloat(i) == 3.toFloat)
+ }
+ }
+
+ testConstantColumnVector("fill double", 10, DoubleType) { vector =>
+ ColumnVectorUtils.populate(vector, InternalRow(3.toDouble), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getDouble(i) == 3.toDouble)
+ }
+ }
+
+ testConstantColumnVector("fill decimal", 10, DecimalType(10, 0)) { vector =>
+ val decimal = Decimal(100L)
+ ColumnVectorUtils.populate(vector, InternalRow(decimal), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getDecimal(i, 10, 0) == decimal)
+ }
+ }
+
+ testConstantColumnVector("fill utf8string", 10, StringType) { vector =>
+ val string = UTF8String.fromString("hello")
+ ColumnVectorUtils.populate(vector, InternalRow(string), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getUTF8String(i) == string)
+ }
+ }
+
+ testConstantColumnVector("fill binary", 10, BinaryType) { vector =>
+ val binary = "hello".getBytes("utf8")
+ ColumnVectorUtils.populate(vector, InternalRow(binary), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getBinary(i) === binary)
+ }
+ }
+
+ testConstantColumnVector("fill calendar interval", 10,
+ CalendarIntervalType) { vector =>
+ val interval = new CalendarInterval(3, 5, 1000000)
+ ColumnVectorUtils.populate(vector, InternalRow(interval), 0)
+ (0 until 10).foreach { i =>
+ assert(vector.getInterval(i) === interval)
+ }
+ }
+
+ testConstantColumnVector("not supported: fill map", 10,
+ MapType(IntegerType, BooleanType)) { vector =>
+ val message = intercept[RuntimeException] {
+ ColumnVectorUtils.populate(vector, InternalRow("fakeMap"), 0)
+ }.getMessage
+ assert(message == "DataType MAP<INT, BOOLEAN> is not supported in column
vectorized reader.")
+ }
+
+ testConstantColumnVector("not supported: fill struct", 10,
+ new StructType()
+ .add(StructField("name", StringType))
+ .add(StructField("age", IntegerType))) { vector =>
+ val message = intercept[RuntimeException] {
+ ColumnVectorUtils.populate(vector, InternalRow("fakeStruct"), 0)
+ }.getMessage
+ assert(message ==
+ "DataType STRUCT<name: STRING, age: INT> is not supported in column
vectorized reader.")
+ }
+
+ testConstantColumnVector("not supported: fill array", 10,
+ ArrayType(IntegerType)) { vector =>
+ val message = intercept[RuntimeException] {
+ ColumnVectorUtils.populate(vector, InternalRow("fakeArray"), 0)
+ }.getMessage
+ assert(message == "DataType ARRAY<INT> is not supported in column
vectorized reader.")
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
