wecharyu opened a new issue, #13633: URL: https://github.com/apache/gluten/issues/13633
### Backend VL (Velox) ### Bug description The inherited Spark `SPARK-25158` regression test intermittently fails in Gluten's Hive SQL suites because the Python script path points inside a Spark test JAR rather than to a filesystem file. The resulting subprocess startup failure races with the UDF exception that the test is intended to check. **Introduced by #12005** (`[GLUTEN-10072][UT] Enable hive execution tests`, merge commit [356fda012d](https://github.com/apache/gluten/commit/356fda012d859f693fc613e6ac3585b7661bdbc5)). That PR added the Hive `GlutenSQLQuerySuite` / `GlutenSQLQuerySuiteAE` wrappers and enabled them in `VeloxTestSettings`, bringing the inherited `SPARK-25158` test into Gluten's test coverage. It also introduced `GlutenHiveResourcePathSupport` and rewrote several tests to use workspace resources, but left this inherited test using the upstream resource lookup. This attribution concerns its introduction into Gluten CI; the original regression test already existed in Spark. **Expected behavior** The Python transformation script should resolve to a real file and start successfully. When the deliberately failing UDF throws, its exception should reach the task and satisfy the expected-message assertion, while the uncaught-exception handler remains empty. **Actual behavior** Python cannot open the JAR-based script path and exits with code `2`. Depending on timing, Spark reports the subprocess error instead of the expected UDF error, so the assertion fails and the CI job becomes red. **Observed CI failures** | PR | Spark | Failing suite | Job | | --- | --- | --- | --- | | #13628 | 3.4.4 | `org.apache.spark.sql.hive.execution.GlutenSQLQuerySuite` | [spark-test-spark34-slow (slow-hive)](https://github.com/apache/gluten/actions/runs/43206708219/job/131146427143) | | #13613 | 4.0.2 | `org.apache.spark.sql.hive.execution.GlutenSQLQuerySuiteAE` | [spark-test-spark40-slow (slow-hive)](https://github.com/apache/gluten/actions/runs/42909456682/job/130171962940) | | #13456 | 4.0.2 | `org.apache.spark.sql.hive.execution.GlutenSQLQuerySuiteAE` | [spark-test-spark40-slow (slow-hive)](https://github.com/apache/gluten/actions/runs/43230119992/job/131226342548) | **Root cause and why the failure is flaky** 1. The [upstream test](https://github.com/apache/spark/blob/v3.4.4/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala#L2335) calls `getTestResourcePath("data")` and appends `/scripts/test_transform.py` to construct the Python command. Its UDF deliberately throws `Failed to produce data.`. 2. [SparkFunSuite's resource helper](https://github.com/apache/spark/blob/v3.4.4/core/src/test/scala/org/apache/spark/SparkFunSuite.scala#L110) converts the ClassLoader resource URL through `new File(...getFile).getCanonicalPath`. In these Gluten CI runs, the resource comes from `spark-hive_*-tests.jar`. This conversion does not extract the resource; it produces a path containing `file:/...tests.jar!/data`, prefixed by the current working directory. Python cannot use that as a script file. 3. [HiveScriptTransformationExec](https://github.com/apache/spark/blob/v3.4.4/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationExec.scala#L160) starts a writer thread to consume upstream rows, including evaluating the UDF, while the task thread reads the subprocess output. 4. [checkFailureAndPropagate](https://github.com/apache/spark/blob/v3.4.4/sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala#L167) checks whether the writer has recorded an exception before checking the subprocess exit status. If the UDF error is already recorded, the expected assertion can pass. If Python exits before that error is recorded, the task can propagate the exit-code error and fail the assertion. A passing execution can therefore mask the invalid script path. The Spark 3.4 failure log is consistent with this ordering: the subprocess error is logged at `11:31:16.166`, followed by the writer's UDF error and the missing-script diagnostic at `11:31:16.171`. The timestamps support the analysis but are not a measurement of the precise exception-assignment time. The path failure is confirmed by the logs; the timing explanation follows from the execution code. No controlled repeated local reproduction or failure-rate measurement has been performed. **Suggested fix** Rewrite the inherited test in Gluten's Hive suite adapter to use a real workspace script path, for example `hiveResourcePath("data/scripts/test_transform.py").toFile.getCanonicalPath`, following the existing resource adaptations. Preserve both the UDF-message assertion and the uncaught-exception assertion. Cover both regular and AQE suites and audit the other Spark versions that inherit this test. Related tracking issue: #10072. This issue was prepared with assistance from OpenAI Codex (GPT-6). ### Gluten version Main branch. Example failing PR head: `0bd72a021508a708338bae5cedc501a244e2dafd` (#13628). ### Spark version Confirmed in CI with Spark 3.4.4 and Spark 4.0.2. ### Spark configurations The `slow-hive` jobs in the Velox Backend (x86) workflow, including the regular and AQE Hive SQL suites listed above. ### System information GitHub Actions CI. The linked jobs contain the runner, container, and build details. ### Relevant logs The Spark 3.4 run contains the following filesystem diagnostic: ```text python3: can't open file '/__w/gluten/gluten/gluten-ut/spark34/file:/root/.m2/repository/org/apache/spark/spark-hive_2.12/3.4.4/spark-hive_2.12-3.4.4-tests.jar!/data/scripts/test_transform.py' ``` The task reports `Subprocess exited with status 2`, and the assertion fails because the message does not contain the expected UDF text. The Spark 4.0 runs show the same malformed path pattern under `spark40`, using `spark-hive_2.13-4.0.2-tests.jar`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
