rahil-c commented on PR #19048:
URL: https://github.com/apache/hudi/pull/19048#issuecomment-4782171784
## CI investigation + fix (Spark 3.3 assertion)
After the test-hardening commit, the GitHub Actions matrix flagged
`test-spark-scala-other-tests (scala-2.12, spark3.3)`. Root-caused and fixed in
the current head (`aad92c8d`). Details for reviewers:
### What actually failed (mine)
Only **`test-spark-scala-other-tests` on Spark 3.3**. The production fix
worked there — it was my *test assertion* that was wrong for 3.3.
The helper `assertNativeUnresolvedColumn` branches on
`HoodieSparkUtils.gteqSpark3_4`:
- **Spark 3.4+** (error classes exist) → require the structured token
`UNRESOLVED_COLUMN`. Passed on 3.4 / 3.5 / 4.0 / 4.1 / 4.2.
- **Spark 3.3** (pre-error-class) → originally only accepted `"cannot
resolve"` / `"cannot be resolved"`.
Spark 3.3 actually phrases an unresolved column as:
```
Column 'nonexistent_column' does not exist. Did you mean one of the
following? []; line 4 pos 64;
```
i.e. **`does not exist`**, not "cannot be resolved" — so the 3.3 fallback
didn't match. One-line fix:
```scala
} else {
assert(msg.contains("cannot resolve") || msg.contains("cannot be
resolved") ||
msg.contains("does not exist"), // Spark 3.3 phrasing
s"Expected Spark's native unresolved-column error; got: $msg")
}
```
### The other red jobs on the prior run were not from this PR
On the previous run three other jobs were red:
- `test-spark-scala-other-tests (spark3.4)` — failed on `Test Validation -
Overlapping Transactions` / `Mixed Issues`; **this PR's test passed on 3.4**.
- `test-spark-java17-java-tests-part2 (spark4.1 / 4.2)` —
`TestHoodieClientMultiWriter.testHoodieClientBasicMultiWriterWithEarlyConflictDetectionDirect`
failing with `OutOfMemoryError` / `CLEANING FAILURE` / socket-bind errors.
That Java test job does not contain this PR's Scala test.
On the current head (`aad92c8d`) all four of those jobs report `pass`, which
confirms the three non-3.3 failures were pre-existing flakes and the 3.3
failure is resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
