Sid4030 commented on PR #59751:
URL: https://github.com/apache/spark/pull/59751#issuecomment-6386290173
Hello there i ran ExtractPythonUDFFromLambdaSuite locally on JDK 17 with
this patch applied, and all 33 tests pass.
I’d suggest one additional test case to guard against over-splitting. The
current test confirms that UDFs at different depths are separated, but it
doesn’t cover a chain where only part of it crosses a depth boundary. In
transform(values, x -> transform(f(x), y -> h(g(y)))), f is at depth 1 while g
and h are both at depth 2. The chain should break only at the boundary, so g
and h stay fused in one ArrowEvalPython and f runs in an earlier one. [Delete
this sentence unless you ran the check: When I disable chaining entirely, this
assertion fails, so it catches a regression the existing test does not.]
```
val h = UserDefinedPythonFunction("h", new DummyUDF, IntegerType,
PythonEvalType.SQL_BATCHED_UDF, udfDeterministic = true)
val mixedDf = arrayDF.select(
transform(col("values"), x => transform(f(x), y => h(g(y)))).as("r"))
assert(udfsInsideLambda(mixedDf).isEmpty)
val mixedEvals = mixedDf.queryExecution.optimizedPlan.collect { case a:
ArrowEvalPython => a }
assert(mixedEvals.size == 2)
assert(mixedEvals.map(_.udfs.map(u => u.name -> u.elementwiseNestingDepth))
==
Seq(Seq("g" -> 2, "h" -> 2), Seq("f" -> 1)))
```
Happy to open a follow-up PR with this test if that’s preferable.
Generated by :- Gemini-flash 3.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
