sunchao commented on PR #59146: URL: https://github.com/apache/spark/pull/59146#issuecomment-5962984773
Updated in [e3b6ec55](https://github.com/apache/spark/commit/e3b6ec55d570bf2c7d2d79061c414796d4f2ae7a). The cross-shuffle regression now uses vectorized Parquet and checks both preconditions: the query uses the scan's columnar wrapper RDD, and the downstream task serialization visits the upstream scan plan. Applied the three cleanups and replied to all five inline comments. The PR description now states the wrapper-only scope and has a concise testing section, addressing the description feedback as well. Validation: recompiled the two changed files and ran the complete SparkPlanSuite: **14 passed, 0 failed**. Removing only the `executeColumnarRDD` annotation made both regressions fail for the intended reasons: the direct round trip reached the nonserializable sentinel, and the shuffle test found the upstream wrapper RDD. Scoped Scalastyle passed with 0 errors and 0 warnings; `git diff --check` passed. The local runs used Scala 2.13.18 / Java 17 and the exact exported test classpath from [CI run 37074734073](https://github.com/sunchao/spark/actions/runs/37074734073), whose compile artifact was built from CI merge 46131903a74c52e9c457400f445efe7fcb38f9a7 (Apache master 099c3ee2e2e71ef8c826ee06d35081cb6d82b29c plus this PR). The two source files were verified against that base before recompiling. The source build passed; the broader CI test jobs are still running. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
