This is an automated email from the ASF dual-hosted git repository.
weiting-chen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new caa71bce15 [VL] Add shiftrightunsigned native execution coverage
(#13214)
caa71bce15 is described below
commit caa71bce1567e9421104f9f74f2ab69937a90c9f
Author: Minni Mittal <[email protected]>
AuthorDate: Fri Nov 6 20:15:23 2026 +0530
[VL] Add shiftrightunsigned native execution coverage (#13214)
* [VL] Add shiftrightunsigned native execution coverage
Add column-backed parity coverage for Spark SQL shiftrightunsigned and the
unsigned right-shift operator across integer, bigint, boundary, and null
inputs. Mark both forms supported by the Velox backend.
Co-authored-by: Copilot <[email protected]>
* [VL] Fix shiftrightunsigned Spark version coverage
Keep the shiftrightunsigned function test available on all supported Spark
versions, and run the unsigned shift operator coverage only on Spark 4.0 and
later. Update the scalar-function support summary for the newly supported
expression.
Co-authored-by: Copilot <[email protected]>
* [VL] Reuse unsigned-shift test fixture
## Summary
Run function and operator comparisons against one Parquet input.
## Prompting Intent
Address fixture-reuse feedback while preserving Spark compatibility and
native-plan assertions.
## Linked Sources
- Pull request: https://github.com/apache/gluten/pull/13214
- Review comment:
https://github.com/apache/gluten/pull/13214#discussion_r4266933469
## Rationale
Keep both comparisons within the same scoped fixture to avoid a second
Parquet write. Retain view and path cleanup, and gate the operator check to
Spark 4.0 and later.
Generated-by: GitHub Copilot CLI 1.0.103 (GPT-6.1 Sol)
Co-authored-by: Copilot <[email protected]>
* [VL] Require fully native unsigned-shift test projections
Reject partial JVM projection in both unsigned-shift parity queries so
INTEGER and BIGINT coverage cannot pass with mixed execution.
Co-authored-by: Copilot <[email protected]>
* [VL] Clarify unsigned-shift test assertions
## Summary
Share the native-plan assertion between the function and operator queries,
include offending partial-project nodes in failures, and name both SQL
forms.
## Prompting Intent
Address review feedback on the unsigned-shift regression test without
changing
its input cases, SQL queries, or Spark-version compatibility.
## Linked Sources
- https://github.com/apache/gluten/pull/13214#discussion_r4479970958
- https://github.com/apache/gluten/pull/13214#discussion_r4478140222
- https://github.com/apache/gluten/pull/13214#discussion_r4478140329
## Rationale
Keep the helper local to the test so both queries retain the same
native-project
requirement and partial-projection rejection. Preserve the shared Parquet
fixture,
all 13 input rows, and the Spark 4.0 operator guard.
Co-authored-by: Copilot <[email protected]>
---------
Co-authored-by: minni31 <[email protected]>
Co-authored-by: Copilot <[email protected]>
---
.../functions/ScalarFunctionsValidateSuite.scala | 56 +++++++++++++++++++++-
.../velox/velox-backend-scalar-function-support.md | 6 +--
2 files changed, 58 insertions(+), 4 deletions(-)
diff --git
a/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
b/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
index 774a80529a..2a2ade72f0 100644
---
a/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
+++
b/backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
@@ -17,7 +17,7 @@
package org.apache.gluten.functions
import org.apache.gluten.config.GlutenConfig
-import org.apache.gluten.execution.{BatchScanExecTransformer,
FilterExecTransformer, ProjectExecTransformer}
+import org.apache.gluten.execution.{BatchScanExecTransformer,
ColumnarPartialProjectExec, FilterExecTransformer, ProjectExecTransformer}
import org.apache.gluten.expression.FormatNumberRestrictions
import org.apache.gluten.extension.columnar.FallbackTags
@@ -197,6 +197,60 @@ class ScalarFunctionsValidateSuite extends
FunctionsValidateSuite {
}
}
+ test("shiftrightunsigned and >>> (Spark 4.0+)") {
+ def assertNativeUnsignedShift(df: DataFrame): Unit = {
+ checkGlutenPlan[ProjectExecTransformer](df)
+ val partialProjects = getExecutedPlan(df).collect {
+ case p: ColumnarPartialProjectExec => p
+ }
+ assert(
+ partialProjects.isEmpty,
+ s"Unsigned shifts must not use partial JVM
projection:\n${partialProjects.mkString("\n")}")
+ }
+
+ withTempPath {
+ path =>
+ withTempView("shift_input") {
+ Seq[(Integer, java.lang.Long, Integer)](
+ (8, 8L, 1),
+ (-1, -1L, 1),
+ (Int.MinValue, Long.MinValue, 1),
+ (Int.MinValue, Long.MinValue, 31),
+ (-1, -1L, 32),
+ (-1, -1L, 33),
+ (-1, -1L, 63),
+ (-1, -1L, 64),
+ (-1, -1L, 65),
+ (-1, -1L, -1),
+ (Int.MinValue, Long.MinValue, Int.MaxValue),
+ (null, null, 1),
+ (1, 1L, null)
+ )
+ .toDF("int_value", "long_value", "shift")
+ .write
+ .parquet(path.getCanonicalPath)
+
+
spark.read.parquet(path.getCanonicalPath).createOrReplaceTempView("shift_input")
+ runQueryAndCompare(
+ """
+ |SELECT
+ | shiftrightunsigned(int_value, shift),
+ | shiftrightunsigned(long_value, shift)
+ |FROM shift_input
+ |""".stripMargin)(assertNativeUnsignedShift)
+ if (isSparkVersionGE("4.0")) {
+ runQueryAndCompare(
+ """
+ |SELECT
+ | int_value >>> shift,
+ | long_value >>> shift
+ |FROM shift_input
+ |""".stripMargin)(assertNativeUnsignedShift)
+ }
+ }
+ }
+ }
+
test("array_aggregate") {
withTempPath {
path =>
diff --git a/docs/backends/velox/velox-backend-scalar-function-support.md
b/docs/backends/velox/velox-backend-scalar-function-support.md
index 404b63442c..0ce993e2c5 100644
--- a/docs/backends/velox/velox-backend-scalar-function-support.md
+++ b/docs/backends/velox/velox-backend-scalar-function-support.md
@@ -1,6 +1,6 @@
# Scalar Functions Support Status
-**Out of 433 scalar functions in Spark 4.1, Gluten currently fully supports
261 functions and partially supports 32 functions.**
+**Out of 433 scalar functions in Spark 4.1, Gluten currently fully supports
262 functions and partially supports 32 functions.**
The status applies to `spark.sql.ansi.enabled=false`. When ANSI mode is
enabled, Gluten falls back to vanilla Spark
(see `spark.gluten.sql.ansiFallback.enabled`).
@@ -42,14 +42,14 @@ The status applies to `spark.sql.ansi.enabled=false`. When
ANSI mode is enabled,
| & | BitwiseAnd | S | |
| << | ShiftLeft | S | |
| >> | ShiftRight | S | |
-| >>> | ShiftRightUnsigned | | |
+| >>> | ShiftRightUnsigned | S | |
| ^ | BitwiseXor | S | |
| bit_count | BitwiseCount | S | |
| bit_get | BitwiseGet | S | |
| getbit | BitwiseGet | S | |
| shiftleft | ShiftLeft | S | |
| shiftright | ShiftRight | S | |
-| shiftrightunsigned | ShiftRightUnsigned | | |
+| shiftrightunsigned | ShiftRightUnsigned | S | |
| | | BitwiseOr | S | |
| ~ | BitwiseNot | S | |
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
