minni31 commented on code in PR #12631:
URL: https://github.com/apache/gluten/pull/12631#discussion_r3656032943
##########
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala:
##########
@@ -1423,6 +1423,31 @@ class VeloxSparkPlanExecApi extends SparkPlanExecApi
with Logging {
VeloxColumnarToCarrierRowExec.enforce(plan)
}
+ override def isSupportLocalTableScanExec(plan: LocalTableScanExec): Boolean
= {
+ // `rows` is @transient, so it becomes null after Java serialization (e.g.
an AQE sub-plan
+ // shipped across an RPC boundary). A null rows payload signals a
deserialized plan that can
+ // no longer be executed natively, so offload must be skipped to avoid a
later NPE.
+ if (plan.rows == null) {
+ logDebug("LocalTableScan offload skipped: deserialized plan with null
transient rows")
+ return false
+ }
+ // A streaming source (Spark 4.0+ only) must keep vanilla execution.
+ if (SparkShimLoader.getSparkShims.getLocalTableScanStream(plan).isDefined)
{
+ logDebug("LocalTableScan offload skipped: streaming source detected")
+ return false
+ }
Review Comment:
Addressed in b2323f7d0 by adding explicit test coverage for the streaming
path:
- **Spark 4.x streaming-detected path** — new
`Spark40LocalTableScanStreamSuite` and `Spark41LocalTableScanStreamSuite`
construct a `LocalTableScanExec` carrying a stream and assert
`getLocalTableScanStream(plan).isDefined` (plus `None` for a batch plan). These
live in the `shims/spark40` and `shims/spark41` modules because the `stream`
constructor parameter is Spark 4.x-only and cannot compile in the shared
cross-version `VeloxLocalTableScanSuite`.
- **Version-agnostic guard** — added a case to `VeloxLocalTableScanSuite`
asserting a batch `LocalTableScanExec` is classified as non-streaming (`None`)
and is therefore not falsely skipped by the streaming guard, on every supported
Spark version.
Both shim suites pass locally (2/2 each) under `-Pspark-4.0` and
`-Pspark-4.1`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
