This is an automated email from the ASF dual-hosted git repository.
sunchao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new fd9b8a094b0 [SPARK-43410][SQL] Improve vectorized loop for Packed
skipValues
fd9b8a094b0 is described below
commit fd9b8a094b02b958817a6e2a924ebd5df24f7fb4
Author: ClownXC <[email protected]>
AuthorDate: Mon May 8 10:06:59 2023 -0700
[SPARK-43410][SQL] Improve vectorized loop for Packed skipValues
### What changes were proposed in this pull request?
Improve vectorized loop for Packed skipValues
### Why are the changes needed?
Performance improvement
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
Existing tests
Closes #41092 from clownxc/clownxc-spark-43410.
Authored-by: ClownXC <[email protected]>
Signed-off-by: Chao Sun <[email protected]>
---
.../sql/execution/datasources/parquet/VectorizedRleValuesReader.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
index 12fe5697954..41ce02e3c02 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
@@ -673,12 +673,14 @@ public final class VectorizedRleValuesReader extends
ValuesReader
}
break;
case PACKED:
+ int totalSkipNum = 0;
for (int i = 0; i < num; ++i) {
// Same as above, only skip non-null values from `valuesReader`
if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel)
{
- updater.skipValues(1, valuesReader);
+ ++totalSkipNum;
}
}
+ updater.skipValues(totalSkipNum, valuesReader);
break;
}
currentCount -= num;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
