zhengruifeng commented on PR #57122:
URL: https://github.com/apache/spark/pull/57122#issuecomment-5275919991
Thanks. To clarify, the existing pipeline already covers more than
unresolved-plan stability: `JVM DataFrame API → protobuf golden → analyzed
Catalyst plan golden`
`ProtoToParsedPlanTestSuite` transforms and analyzes each protobuf plan.
It seems straightforward to extend executable cases to also check the optimized
plan, output schema, and result through QueryExecution. Execution should be
opt-in because some existing plans are streaming, schema-only, or
otherwise non-executable.
The new Python framework does not assert protobuf stability and duplicates
much of the server-side golden machinery. I would prefer extending the existing
Scala framework for language-neutral DataFrame semantics. Any genuinely
Python-specific behavior can remain in focused, conventional PySpark tests.
Running a large golden corpus directly in Scala should also use fewer OSS
CI resources by avoiding Python, Connect round trips, repeated analysis, and
Python-side result conversion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
