luis4a0 opened a new pull request, #12963:
URL: https://github.com/apache/gluten/pull/12963
## What changes are proposed in this pull request?
Spark 4.1 uses `OneRowRelationExec` as the physical source for zero-column,
one-row inputs in literal-only queries and scalar subqueries. The Velox
backend
does not currently transform this leaf, so an otherwise native plan must
retain
a Spark operator and cross a row-to-columnar boundary.
This PR adds native Spark 4.1 `OneRowRelationExec` support by:
- recognizing the operator in the Spark 4.1 shim without introducing a Spark
4.1 dependency into common code;
- replacing it only for the Velox backend while preserving the original Spark
plan for unsupported backends and Spark versions;
- representing the logical row as a zero-column, one-row Substrait virtual
table and converting it to a Velox `ValuesNode`;
- introducing an explicit, conservative no-input execution capability so a
supported dependency-free native stage receives exactly one Spark
partition;
- propagating that capability only through safe unary, union, and
streamed-side
broadcast-join paths;
- validating virtual-table schemas, row cardinality, null handling, malformed
values, and split-free native conversion; and
- documenting the supported virtual-table and `OneRowRelationExec` behavior.
This is needed for Spark 4.1 native-plan completeness. The existing Spark
fallback produces correct results, but it fragments literal-only and
scalar-subquery pipelines and leaves an avoidable execution boundary. The new
no-input behavior is deliberately opt-in: unrelated empty or broadcast-only
stages retain their previous failure behavior, and unsupported backends
retain
their original Spark plans.
This PR is not presented as a performance optimization. Its purpose is to
close
the Spark 4.1 capability gap with explicit execution semantics and regression
coverage.
## How was this patch tested?
Added and ran JVM unit, Spark integration, and native C++ tests covering:
- typed Spark 4.1 recognition and false-positive rejection;
- unsupported-backend identity preservation;
- Substrait virtual-table protobuf generation, batching, schema and null
validation, zero-column cardinality, and legal zero-row execution;
- dependency-free and broadcast-only partition discovery, including
concurrent
access and unsupported-stage rejection;
- AQE enabled and disabled, scalar-subquery reuse, native union, broadcast
hash
join, broadcast nested-loop join, and multi-partition file-scan
coexistence;
- Spark plan-tag preservation, exact output rows, native-stage presence, and
removal of the OneRow-specific fallback boundary; and
- native `ValuesNode` execution, malformed and non-literal rejection,
trivial-leaf registration, and file-split isolation.
Results on the current `main` base:
- 14 focused `gluten-substrait` tests passed.
- 3 Spark 4.1 shim tests passed.
- All 106 executed `MiscOperatorSuite` tests passed; one existing test was
ignored.
- The Spark 4.1 explain regression passed.
- 52 native plan-conversion tests passed; 7 existing round-trip tests were
skipped.
- Spotless, Scalastyle, Checkstyle, `clang-format-15`, license-header, and
source-hygiene checks passed.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: GitHub Copilot CLI 1.0.83-0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
