deepujain opened a new pull request, #59035:
URL: https://github.com/apache/spark/pull/59035
### What changes were proposed in this pull request?
Java and Python users inspecting the code Spark generates do not have the
named `debugCodegen()` method available through Scala's implicit helper. This
adds `df.debugCodegen()` directly to their APIs, giving them the same debugging
output through a discoverable method.
The aliases call the existing `explain("codegen")` implementation in the
shared APIs, so classic and Connect callers follow the same behavior. The Scala
implicit helper stays in place to preserve existing callers.
This continues Juanvi Soler's earlier work in
[#40608](https://github.com/apache/spark/pull/40608), adapted to the current
shared API layout. The commit credits the original author. The API reference
and version annotations identify the next compatible release, 4.4.0.
### Why are the changes needed?
[SPARK-35198](https://issues.apache.org/jira/browse/SPARK-35198) requests
this direct Java and Python access. A Java example using `df.debugCodegen()`
currently fails to compile because implicit Scala conversions are not ordinary
Java methods. With the alias, that same caller prints the generated Java code.
Python receives the matching method name; `explain(mode="codegen")` remains
supported.
### Does this PR introduce _any_ user-facing change?
Yes. Java, Scala, and Python callers gain a direct `debugCodegen()` alias.
Query execution and generated code are unchanged. Existing `explain` calls and
the Scala implicit helper remain available.
### How was this patch tested?
- A Java caller fails to compile against the original API and compiles
against the candidate API.
- A real local Java SparkSession produced 5,905 characters of
code-generation output through the new method, byte-identical to
`explain("codegen")`.
- The actual PySpark classic regression passed against the candidate
JVM/classes, checking generated-code output equality and the `None` return
value.
- The inherited PySpark Connect regression passed against a real local
Connect server: `python -m unittest
pyspark.sql.tests.connect.test_parity_dataframe.DataFrameParityTests.test_debug_codegen`
(1 passed). Classic used
`pyspark.sql.tests.test_dataframe.DataFrameTests.test_debug_codegen` (1
passed). Both check output equality and the `None` return value.
- `connect/Compile/compile`, `sql-api/scalastyle`, and `sql/test:scalastyle`
passed; both style checks reported zero errors and warnings.
- Added Java, Scala classic, and JVM Connect regression coverage; these
broader test suites were not completed locally. `git diff --check` passed.
The full SQL test-module compilation was killed with exit 137 in the local
container, including after increasing the compiler heap. No SQL or JVM Connect
suite success is inferred from that build. Direct runtime checks are reported
separately from the uncompleted broad test compilation.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
**JIRA assignee for credit:** deepujain
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
