This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 021b347d3f6c [SPARK-56893][SQL] Optimize Parquet dictionary decoding
with hasNull fast path and per-class updater overrides
021b347d3f6c is described below
commit 021b347d3f6ca6049061ce520d7190db79cefb99
Author: Ismaël Mejía <[email protected]>
AuthorDate: Tue Jun 23 09:43:17 2026 +0200
[SPARK-56893][SQL] Optimize Parquet dictionary decoding with hasNull fast
path and per-class updater overrides
### What changes were proposed in this pull request?
This PR adds two optimizations to the Parquet vectorized dictionary decode
path (`ParquetVectorUpdater.decodeDictionaryIds`):
1. **`hasNull()` fast path**: A new `static decodeBatch` helper on
`ParquetVectorUpdater` splits decoding into two loops — when `values.hasNull()`
is false, the per-element `isNullAt(i)` check is skipped entirely.
2. **Per-class `decodeDictionaryIds` overrides** in six hot-path updaters
(`IntegerUpdater`, `IntegerToLongUpdater`, `LongUpdater`, `FloatUpdater`,
`FloatToDoubleUpdater`, `DoubleUpdater`): each override is a one-line
delegation to `decodeBatch(... this)`. Although the logic is identical to the
default method, each per-class override gives the C2 JIT compiler its own
bytecode — and therefore a monomorphic call site for `decodeSingleDictionaryId`
— enabling full inlining of the type-sp [...]
Class hierarchy of the change:
```
ParquetVectorUpdater (interface)
├── default decodeDictionaryIds(...) → delegates to static
decodeBatch(... this)
├── static decodeBatch(...) → hasNull() branch + two loops
calling updater.decodeSingleDictionaryId()
│
└── Concrete updaters in ParquetVectorUpdaterFactory:
├── IntegerUpdater Override decodeDictionaryIds →
decodeBatch(... this)
├── IntegerToLongUpdater Override decodeDictionaryIds →
decodeBatch(... this)
├── LongUpdater Override decodeDictionaryIds →
decodeBatch(... this)
├── FloatUpdater Override decodeDictionaryIds →
decodeBatch(... this)
├── FloatToDoubleUpdater Override decodeDictionaryIds →
decodeBatch(... this)
└── DoubleUpdater Override decodeDictionaryIds →
decodeBatch(... this)
```
### Why are the changes needed?
The default `decodeDictionaryIds` method has two performance issues:
- **Unconditional `isNullAt` check**: Even when the column has no nulls
(common case), every element pays for an `isNullAt(i)` call.
`WritableColumnVector.hasNull()` is an O(1) flag check that allows skipping the
per-element null check entirely.
- **Megamorphic dispatch**: Java interface default methods compile to a
single bytecode shared by all implementors. C2 profiles one call site for
`decodeSingleDictionaryId` across ~30 updater types → megamorphic → no
inlining. Per-class overrides create per-class bytecode → per-class C2 profiles
→ monomorphic devirtualization → full inlining of the decode expression.
### Benchmark results
Baseline vs optimized on **AMD EPYC 9V74** (1M rows, dict size 100, Rate
M/s — higher is better):
**No nulls (fast-path target):**
| Updater | JDK 17 Baseline | JDK 17 Optimized | Speedup | JDK 21 Baseline
| JDK 21 Optimized | Speedup | JDK 25 Baseline | JDK 25 Optimized | Speedup |
|---------|----:|----:|:---:|----:|----:|:---:|----:|----:|:---:|
| IntegerUpdater | 186.7 | 227.6 | **1.22x** | 177.6 | 288.3 | **1.62x** |
179.1 | 222.6 | **1.24x** |
| LongUpdater | 185.1 | 224.1 | **1.21x** | 176.4 | 285.4 | **1.62x** |
178.6 | 221.1 | **1.24x** |
| FloatUpdater | 188.6 | 229.7 | **1.22x** | 178.3 | 287.4 | **1.61x** |
178.1 | 222.9 | **1.25x** |
| DoubleUpdater | 187.6 | 228.2 | **1.22x** | 177.2 | 286.5 | **1.62x** |
177.5 | 221.7 | **1.25x** |
| IntegerToLongUpdater | 184.4 | 224.1 | **1.22x** | 175.8 | 281.6 |
**1.60x** | 175.1 | 218.9 | **1.25x** |
| FloatToDoubleUpdater | 187.3 | 227.8 | **1.22x** | 172.2 | 268.8 |
**1.56x** | 178.8 | 222.2 | **1.24x** |
**10% nulls:**
| Updater | JDK 17 Speedup | JDK 21 Speedup | JDK 25 Speedup |
|---------|:---:|:---:|:---:|
| IntegerUpdater | 1.00x | **1.26x** | 0.99x |
| LongUpdater | 0.96x | **1.28x** | 0.99x |
| FloatUpdater | 1.00x | **1.28x** | 0.97x |
| DoubleUpdater | 1.00x | **1.24x** | 1.00x |
| IntegerToLongUpdater | 0.98x | **1.26x** | 0.99x |
| FloatToDoubleUpdater | 0.99x | **1.29x** | 0.99x |
**50% nulls:**
| Updater | JDK 17 Speedup | JDK 21 Speedup | JDK 25 Speedup |
|---------|:---:|:---:|:---:|
| IntegerUpdater | 0.99x | **1.25x** | 0.99x |
| LongUpdater | 1.01x | **1.25x** | 1.03x |
| FloatUpdater | 0.99x | **1.26x** | 0.98x |
| DoubleUpdater | 0.99x | **1.26x** | 0.98x |
| IntegerToLongUpdater | 0.99x | **1.26x** | 1.02x |
| FloatToDoubleUpdater | 1.00x | **1.25x** | 0.99x |
**Key observations:**
- JDK 17/25 no-nulls: consistent **1.21-1.25x** improvement.
- JDK 21 no-nulls: **1.56-1.62x** — JDK 21 C2 benefits dramatically from
monomorphic devirtualization.
- JDK 21 with-nulls: **1.24-1.29x** — the monomorphic call site alone
(independent of hasNull branch) improves throughput.
- JDK 17/25 with-nulls: neutral (~1.0x) — as expected, `isNullAt` dominates.
Baseline workflow runs (before optimization): [JDK
17](https://github.com/iemejia/spark/actions/runs/27978726935), [JDK
21](https://github.com/iemejia/spark/actions/runs/27967169001), [JDK
25](https://github.com/iemejia/spark/actions/runs/27977735937)
Optimized workflow runs: [JDK
17](https://github.com/iemejia/spark/actions/runs/27981015508), [JDK
21](https://github.com/iemejia/spark/actions/runs/27989226064), [JDK
25](https://github.com/iemejia/spark/actions/runs/27986201001)
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- Existing tests: `ParquetVectorizedSuite`, `ParquetIOSuite` (123 tests
total) — all pass.
- Benchmark: `ParquetDictionaryDecodeBenchmark` (landed separately in
#56670) with global pre-warm that interleaves both `hasNull()` branches
(no-null and 50%-null) across all 6 updater types before measurement, avoiding
C2 uncommon-trap bias. Three benchmark groups: no nulls, 10% nulls, 50% nulls.
Baseline and optimized results committed on the same CPU (AMD EPYC 9V74).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenCode (Claude claude-opus-4.6)
Closes #55920 from iemejia/SPARK-56893-parquet-dict-decode-fast-path.
Authored-by: Ismaël Mejía <[email protected]>
Signed-off-by: Max Gekk <[email protected]>
---
...quetDictionaryDecodeBenchmark-jdk21-results.txt | 36 ++++++-------
...quetDictionaryDecodeBenchmark-jdk25-results.txt | 36 ++++++-------
.../ParquetDictionaryDecodeBenchmark-results.txt | 36 ++++++-------
.../datasources/parquet/ParquetVectorUpdater.java | 43 ++++++++++++++--
.../parquet/ParquetVectorUpdaterFactory.java | 60 ++++++++++++++++++++++
5 files changed, 152 insertions(+), 59 deletions(-)
diff --git
a/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-jdk21-results.txt
b/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-jdk21-results.txt
index 041372e7f0af..bca8e062ee23 100644
--- a/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-jdk21-results.txt
+++ b/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-jdk21-results.txt
@@ -6,12 +6,12 @@ OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux
6.17.0-1018-azure
AMD EPYC 9V74 80-Core Processor
Dictionary Decode (no nulls): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-IntegerUpdater 6 6
0 177.6 5.6 1.0X
-LongUpdater 6 6
0 176.4 5.7 1.0X
-FloatUpdater 6 6
0 178.3 5.6 1.0X
-DoubleUpdater 6 6
0 177.2 5.6 1.0X
-IntegerToLongUpdater (INT32 -> Long) 6 6
0 175.8 5.7 1.0X
-FloatToDoubleUpdater (FLOAT -> Double) 6 6
0 172.2 5.8 1.0X
+IntegerUpdater 4 4
0 288.3 3.5 1.0X
+LongUpdater 4 4
0 285.4 3.5 1.0X
+FloatUpdater 4 4
0 287.4 3.5 1.0X
+DoubleUpdater 4 4
0 286.5 3.5 1.0X
+IntegerToLongUpdater (INT32 -> Long) 4 4
0 281.6 3.6 1.0X
+FloatToDoubleUpdater (FLOAT -> Double) 4 4
0 268.8 3.7 0.9X
================================================================================================
@@ -22,12 +22,12 @@ OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux
6.17.0-1018-azure
AMD EPYC 9V74 80-Core Processor
Dictionary Decode (10% nulls): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-IntegerUpdater 7 7
0 159.9 6.3 1.0X
-LongUpdater 7 7
0 158.4 6.3 1.0X
-FloatUpdater 7 7
0 159.9 6.3 1.0X
-DoubleUpdater 7 7
0 159.8 6.3 1.0X
-IntegerToLongUpdater (INT32 -> Long) 7 7
0 160.4 6.2 1.0X
-FloatToDoubleUpdater (FLOAT -> Double) 7 7
0 157.0 6.4 1.0X
+IntegerUpdater 5 5
0 201.7 5.0 1.0X
+LongUpdater 5 5
0 202.2 4.9 1.0X
+FloatUpdater 5 5
0 203.9 4.9 1.0X
+DoubleUpdater 5 5
0 198.9 5.0 1.0X
+IntegerToLongUpdater (INT32 -> Long) 5 5
0 201.6 5.0 1.0X
+FloatToDoubleUpdater (FLOAT -> Double) 5 5
0 203.3 4.9 1.0X
================================================================================================
@@ -38,11 +38,11 @@ OpenJDK 64-Bit Server VM 21.0.11+10-LTS on Linux
6.17.0-1018-azure
AMD EPYC 9V74 80-Core Processor
Dictionary Decode (50% nulls): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-IntegerUpdater 9 9
0 113.2 8.8 1.0X
-LongUpdater 9 9
0 117.3 8.5 1.0X
-FloatUpdater 9 9
0 117.7 8.5 1.0X
-DoubleUpdater 9 9
0 117.4 8.5 1.0X
-IntegerToLongUpdater (INT32 -> Long) 9 9
0 113.5 8.8 1.0X
-FloatToDoubleUpdater (FLOAT -> Double) 9 9
0 117.6 8.5 1.0X
+IntegerUpdater 7 7
0 141.9 7.0 1.0X
+LongUpdater 7 7
0 147.2 6.8 1.0X
+FloatUpdater 7 7
0 147.8 6.8 1.0X
+DoubleUpdater 7 7
0 147.6 6.8 1.0X
+IntegerToLongUpdater (INT32 -> Long) 7 7
0 142.8 7.0 1.0X
+FloatToDoubleUpdater (FLOAT -> Double) 7 7
0 147.5 6.8 1.0X
diff --git
a/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-jdk25-results.txt
b/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-jdk25-results.txt
index 538cc8c3445b..7f342b50ac4d 100644
--- a/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-jdk25-results.txt
+++ b/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-jdk25-results.txt
@@ -6,12 +6,12 @@ OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux
6.17.0-1018-azure
AMD EPYC 9V74 80-Core Processor
Dictionary Decode (no nulls): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-IntegerUpdater 6 6
0 179.1 5.6 1.0X
-LongUpdater 6 6
0 178.6 5.6 1.0X
-FloatUpdater 6 6
0 178.1 5.6 1.0X
-DoubleUpdater 6 6
0 177.5 5.6 1.0X
-IntegerToLongUpdater (INT32 -> Long) 6 6
0 175.1 5.7 1.0X
-FloatToDoubleUpdater (FLOAT -> Double) 6 6
0 178.8 5.6 1.0X
+IntegerUpdater 5 5
0 222.6 4.5 1.0X
+LongUpdater 5 5
0 221.1 4.5 1.0X
+FloatUpdater 5 5
0 222.9 4.5 1.0X
+DoubleUpdater 5 5
0 221.7 4.5 1.0X
+IntegerToLongUpdater (INT32 -> Long) 5 5
0 218.9 4.6 1.0X
+FloatToDoubleUpdater (FLOAT -> Double) 5 5
0 222.2 4.5 1.0X
================================================================================================
@@ -22,12 +22,12 @@ OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux
6.17.0-1018-azure
AMD EPYC 9V74 80-Core Processor
Dictionary Decode (10% nulls): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-IntegerUpdater 7 7
0 159.0 6.3 1.0X
-LongUpdater 7 7
0 159.3 6.3 1.0X
-FloatUpdater 7 7
0 158.6 6.3 1.0X
-DoubleUpdater 7 7
0 158.2 6.3 1.0X
-IntegerToLongUpdater (INT32 -> Long) 7 7
0 158.6 6.3 1.0X
-FloatToDoubleUpdater (FLOAT -> Double) 7 7
0 159.2 6.3 1.0X
+IntegerUpdater 7 7
0 156.7 6.4 1.0X
+LongUpdater 7 7
0 157.1 6.4 1.0X
+FloatUpdater 7 7
0 154.1 6.5 1.0X
+DoubleUpdater 7 7
0 157.9 6.3 1.0X
+IntegerToLongUpdater (INT32 -> Long) 7 7
0 157.2 6.4 1.0X
+FloatToDoubleUpdater (FLOAT -> Double) 7 7
0 157.9 6.3 1.0X
================================================================================================
@@ -38,11 +38,11 @@ OpenJDK 64-Bit Server VM 25.0.3+9-LTS on Linux
6.17.0-1018-azure
AMD EPYC 9V74 80-Core Processor
Dictionary Decode (50% nulls): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-IntegerUpdater 9 9
0 116.4 8.6 1.0X
-LongUpdater 9 9
0 111.4 9.0 1.0X
-FloatUpdater 9 9
0 116.4 8.6 1.0X
-DoubleUpdater 9 9
0 116.6 8.6 1.0X
-IntegerToLongUpdater (INT32 -> Long) 9 9
0 114.0 8.8 1.0X
-FloatToDoubleUpdater (FLOAT -> Double) 9 9
0 116.2 8.6 1.0X
+IntegerUpdater 9 9
0 114.9 8.7 1.0X
+LongUpdater 9 9
0 114.8 8.7 1.0X
+FloatUpdater 9 9
0 114.6 8.7 1.0X
+DoubleUpdater 9 9
0 114.5 8.7 1.0X
+IntegerToLongUpdater (INT32 -> Long) 9 9
0 115.8 8.6 1.0X
+FloatToDoubleUpdater (FLOAT -> Double) 9 9
0 114.9 8.7 1.0X
diff --git a/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-results.txt
b/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-results.txt
index c43127858a13..53dcf356ec72 100644
--- a/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-results.txt
+++ b/sql/core/benchmarks/ParquetDictionaryDecodeBenchmark-results.txt
@@ -6,12 +6,12 @@ OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux
6.17.0-1018-azure
AMD EPYC 9V74 80-Core Processor
Dictionary Decode (no nulls): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-IntegerUpdater 6 6
0 186.7 5.4 1.0X
-LongUpdater 6 6
0 185.1 5.4 1.0X
-FloatUpdater 6 6
0 188.6 5.3 1.0X
-DoubleUpdater 6 6
0 187.6 5.3 1.0X
-IntegerToLongUpdater (INT32 -> Long) 6 6
0 184.4 5.4 1.0X
-FloatToDoubleUpdater (FLOAT -> Double) 6 6
0 187.3 5.3 1.0X
+IntegerUpdater 5 5
0 227.6 4.4 1.0X
+LongUpdater 5 5
0 224.1 4.5 1.0X
+FloatUpdater 5 5
0 229.7 4.4 1.0X
+DoubleUpdater 5 5
0 228.2 4.4 1.0X
+IntegerToLongUpdater (INT32 -> Long) 5 5
0 224.1 4.5 1.0X
+FloatToDoubleUpdater (FLOAT -> Double) 5 5
0 227.8 4.4 1.0X
================================================================================================
@@ -22,12 +22,12 @@ OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux
6.17.0-1018-azure
AMD EPYC 9V74 80-Core Processor
Dictionary Decode (10% nulls): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-IntegerUpdater 6 6
0 164.8 6.1 1.0X
-LongUpdater 6 6
0 165.7 6.0 1.0X
-FloatUpdater 6 6
0 166.1 6.0 1.0X
-DoubleUpdater 6 6
0 165.7 6.0 1.0X
-IntegerToLongUpdater (INT32 -> Long) 6 6
0 165.7 6.0 1.0X
-FloatToDoubleUpdater (FLOAT -> Double) 6 6
0 165.7 6.0 1.0X
+IntegerUpdater 6 6
0 164.2 6.1 1.0X
+LongUpdater 7 7
0 159.4 6.3 1.0X
+FloatUpdater 6 6
0 165.9 6.0 1.0X
+DoubleUpdater 6 6
0 165.2 6.1 1.0X
+IntegerToLongUpdater (INT32 -> Long) 6 6
0 163.1 6.1 1.0X
+FloatToDoubleUpdater (FLOAT -> Double) 6 6
0 163.2 6.1 1.0X
================================================================================================
@@ -38,11 +38,11 @@ OpenJDK 64-Bit Server VM 17.0.19+10-LTS on Linux
6.17.0-1018-azure
AMD EPYC 9V74 80-Core Processor
Dictionary Decode (50% nulls): Best Time(ms) Avg Time(ms)
Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
-IntegerUpdater 9 9
0 117.7 8.5 1.0X
-LongUpdater 9 9
0 116.2 8.6 1.0X
-FloatUpdater 9 9
0 117.7 8.5 1.0X
-DoubleUpdater 9 9
0 117.6 8.5 1.0X
-IntegerToLongUpdater (INT32 -> Long) 9 9
0 118.6 8.4 1.0X
-FloatToDoubleUpdater (FLOAT -> Double) 9 9
0 117.4 8.5 1.0X
+IntegerUpdater 9 9
0 117.0 8.5 1.0X
+LongUpdater 9 9
0 116.8 8.6 1.0X
+FloatUpdater 9 9
0 117.1 8.5 1.0X
+DoubleUpdater 9 9
0 116.9 8.6 1.0X
+IntegerToLongUpdater (INT32 -> Long) 9 9
0 117.1 8.5 1.0X
+FloatToDoubleUpdater (FLOAT -> Double) 9 9
0 117.0 8.5 1.0X
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdater.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdater.java
index 9bb852987e65..0774b4ad3475 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdater.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdater.java
@@ -58,6 +58,14 @@ public interface ParquetVectorUpdater {
* should have already been filled, and fills the non-null slots using
dictionary IDs from
* `dictionaryIds`, together with Parquet `dictionary`.
*
+ * <p>The default implementation delegates to {@link #decodeBatch}, which
calls
+ * {@link #decodeSingleDictionaryId} per non-null element. Because this
default method's
+ * bytecode is shared by every implementor, C2 sees a megamorphic call site
for
+ * {@code decodeSingleDictionaryId} and cannot inline the per-element
decode. Hot-path
+ * updaters (e.g. {@code IntegerUpdater}) override this method with an
identical one-line
+ * delegation to {@code decodeBatch}; the per-class bytecode gives C2 a
monomorphic call
+ * site, enabling full inlining of the decode expression.
+ *
* @param total total number slots to process in `values`
* @param offset starting offset in `values`
* @param values destination value vector
@@ -70,11 +78,7 @@ public interface ParquetVectorUpdater {
WritableColumnVector values,
WritableColumnVector dictionaryIds,
Dictionary dictionary) {
- for (int i = offset; i < offset + total; i++) {
- if (!values.isNullAt(i)) {
- decodeSingleDictionaryId(i, values, dictionaryIds, dictionary);
- }
- }
+ decodeBatch(total, offset, values, dictionaryIds, dictionary, this);
}
/**
@@ -91,4 +95,33 @@ public interface ParquetVectorUpdater {
WritableColumnVector values,
WritableColumnVector dictionaryIds,
Dictionary dictionary);
+
+ /**
+ * Batch-decodes dictionary IDs with a no-null fast path.
+ *
+ * <p>When {@code values.hasNull()} is false the per-element {@code
isNullAt} check is
+ * skipped entirely. The loop body calls {@code
updater.decodeSingleDictionaryId}, which
+ * C2 can devirtualize and inline when the {@code updater} reference has a
known concrete
+ * type -- i.e. when this helper is called from a per-class override of
+ * {@link #decodeDictionaryIds} rather than from the shared default method.
+ */
+ static void decodeBatch(
+ int total,
+ int offset,
+ WritableColumnVector values,
+ WritableColumnVector dictionaryIds,
+ Dictionary dictionary,
+ ParquetVectorUpdater updater) {
+ if (!values.hasNull()) {
+ for (int i = offset; i < offset + total; i++) {
+ updater.decodeSingleDictionaryId(i, values, dictionaryIds, dictionary);
+ }
+ } else {
+ for (int i = offset; i < offset + total; i++) {
+ if (!values.isNullAt(i)) {
+ updater.decodeSingleDictionaryId(i, values, dictionaryIds,
dictionary);
+ }
+ }
+ }
+ }
}
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java
index 35936dc80e88..4fa4183e411c 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java
@@ -355,6 +355,16 @@ public class ParquetVectorUpdaterFactory {
values.putInt(offset, valuesReader.readInteger());
}
+ @Override
+ public void decodeDictionaryIds(
+ int total,
+ int offset,
+ WritableColumnVector values,
+ WritableColumnVector dictionaryIds,
+ Dictionary dictionary) {
+ ParquetVectorUpdater.decodeBatch(total, offset, values, dictionaryIds,
dictionary, this);
+ }
+
@Override
public void decodeSingleDictionaryId(
int offset,
@@ -388,6 +398,16 @@ public class ParquetVectorUpdaterFactory {
values.putLong(offset, valuesReader.readInteger());
}
+ @Override
+ public void decodeDictionaryIds(
+ int total,
+ int offset,
+ WritableColumnVector values,
+ WritableColumnVector dictionaryIds,
+ Dictionary dictionary) {
+ ParquetVectorUpdater.decodeBatch(total, offset, values, dictionaryIds,
dictionary, this);
+ }
+
@Override
public void decodeSingleDictionaryId(
int offset,
@@ -680,6 +700,16 @@ public class ParquetVectorUpdaterFactory {
values.putLong(offset, valuesReader.readLong());
}
+ @Override
+ public void decodeDictionaryIds(
+ int total,
+ int offset,
+ WritableColumnVector values,
+ WritableColumnVector dictionaryIds,
+ Dictionary dictionary) {
+ ParquetVectorUpdater.decodeBatch(total, offset, values, dictionaryIds,
dictionary, this);
+ }
+
@Override
public void decodeSingleDictionaryId(
int offset,
@@ -985,6 +1015,16 @@ public class ParquetVectorUpdaterFactory {
values.putFloat(offset, valuesReader.readFloat());
}
+ @Override
+ public void decodeDictionaryIds(
+ int total,
+ int offset,
+ WritableColumnVector values,
+ WritableColumnVector dictionaryIds,
+ Dictionary dictionary) {
+ ParquetVectorUpdater.decodeBatch(total, offset, values, dictionaryIds,
dictionary, this);
+ }
+
@Override
public void decodeSingleDictionaryId(
int offset,
@@ -1018,6 +1058,16 @@ public class ParquetVectorUpdaterFactory {
values.putDouble(offset, valuesReader.readFloat());
}
+ @Override
+ public void decodeDictionaryIds(
+ int total,
+ int offset,
+ WritableColumnVector values,
+ WritableColumnVector dictionaryIds,
+ Dictionary dictionary) {
+ ParquetVectorUpdater.decodeBatch(total, offset, values, dictionaryIds,
dictionary, this);
+ }
+
@Override
public void decodeSingleDictionaryId(
int offset,
@@ -1051,6 +1101,16 @@ public class ParquetVectorUpdaterFactory {
values.putDouble(offset, valuesReader.readDouble());
}
+ @Override
+ public void decodeDictionaryIds(
+ int total,
+ int offset,
+ WritableColumnVector values,
+ WritableColumnVector dictionaryIds,
+ Dictionary dictionary) {
+ ParquetVectorUpdater.decodeBatch(total, offset, values, dictionaryIds,
dictionary, this);
+ }
+
@Override
public void decodeSingleDictionaryId(
int offset,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
