spark-test commented on code in PR #59735:
URL: https://github.com/apache/spark/pull/59735#discussion_r4355439633
##########
sql/connect/server/src/test/scala/org/apache/spark/sql/connect/pipelines/PythonPipelineSuite.scala:
##########
@@ -404,6 +404,65 @@ class PythonPipelineSuite
Set(graphIdentifier("b"), graphIdentifier("d")))
}
+ test("SPARK-60494: referencing internal datasets through DataFrame arguments
of spark.sql") {
Review Comment:
**Nit (P3):** These tests return the `spark.sql(..., df=...)` result
directly, with one direct read per argument and datasets that resolve. That
leaves two behaviors this PR introduces without a failing signal:
- An analysis error in a DataFrame argument is now reported as a flow
resolution failure. `referencing internal datasets failed` covers `read.table`,
`table`, `readStream.table`, `spark.sql`, and `STREAM`, but not the
DataFrame-argument form. If that failure were lost, or reported against the
generated `_pyspark_connect_temp_view_<uuid>` alias instead of `src`, no
assertion would change.
- Deferral also applies to SQL-with-references below the flow's root
relation, because `deferSqlWithRefs` stays set for the whole
`transformRelation` call. Two flows depend on that: one that returns
`spark.sql("SELECT * FROM {df}", df=spark.read.table("src")).where("id > 0")`,
and one that passes an untransformed `spark.sql(..., df=...)` result as a
DataFrame argument of another `spark.sql` call. If deferral regressed to the
root relation only, the first run would fail at `DefineFlow` with
`TABLE_OR_VIEW_NOT_FOUND`, and a re-run would silently drop the dependency on
`src`. Both new tests would still pass.
To close both gaps, add a DataFrame-argument flow to `referencing internal
datasets failed` and add those two shapes to the new positive test. This leaves
out a flow that passes a further-transformed result into another `spark.sql`
call, because that shape still fails at the flow-function SQL-command guard.
See the decision challenge below.
**Verification:**
- **Behavior:** On a fresh pipeline, a flow that reads an undefined dataset
through a spark.sql DataFrame argument is reported as a flow resolution failure
whose message is TABLE_OR_VIEW_NOT_FOUND for the user's dataset name, not for
the generated DataFrame-argument alias.
- **Regression:** On a fresh pipeline, a flow that applies a further
DataFrame operation to a spark.sql-with-DataFrame-arguments result before
returning it, and a flow that passes an untransformed
spark.sql-with-DataFrame-arguments result as a DataFrame argument of another
spark.sql call, register successfully, list the upstream pipeline dataset among
their inputs, and produce the expected rows. These assertions must fail if
deferral applied only to a top-level SQL-with-references relation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
