Yicong-Huang opened a new pull request, #58322: URL: https://github.com/apache/spark/pull/58322
### What changes were proposed in this pull request? This adds an `optional` (best-effort) flag to the Spark Connect Python client's command-execution entry points, `SparkConnectClient.execute_command` and `SparkConnectClient.execute_command_as_iterator` (default `False`, so existing behavior is unchanged). Spark Connect is only guaranteed to be backward compatible: an older client can talk to a newer server. The reverse direction (a newer client talking to an older server) is not supported, so a request for an API the older server does not implement fails hard. Some commands are advisory and safe to skip when the server does not understand them; for those, failing the whole call is undesirable. When a command is sent with `optional=True` and the server rejects it with an error condition that unambiguously means "this server does not implement the requested API", the client now emits a `RuntimeWarning` and returns a no-op result instead of raising. Any other failure is always re-raised, and for the iterator path the failure is only downgraded when the server rejected the request before streaming any response (so partial results are never silently dropped). The recognized condition is `CONNECT_INVALID_PLAN.NO_HANDLER_FOR_EXTENSION`, which is what the server (`InvalidInputErrors.noHandlerFoundForExtension`) raises when a client sends an extension command/relation/expression the server has no handler for -- the canonical way a newer client's new API reaches an older server. The recognized set is a single module-level constant that is easy to extend, deliberately restricted to conditions that cannot also signal a genuine user error. No protocol/proto change is involved; this is purely additive client-side behavior. ### Why are the changes needed? Rollbacks and staged rollouts mean a newer client can transiently reach an older server. For advisory, best-effort commands, a graceful "warn and no-op" is much better than a hard failure, and gives callers a safe, explicit way to opt into that behavior per request. ### Does this PR introduce _any_ user-facing change? No. The new `optional` parameter defaults to `False`, preserving existing behavior. It only changes behavior when a caller explicitly passes `optional=True`. ### How was this patch tested? Added unit tests in `python/pyspark/sql/tests/connect/client/test_client.py`: - `optional=True` downgrades the unsupported-API rejection to a `RuntimeWarning` and returns a no-op result; the default (`optional=False`) still raises. - `optional=True` does not swallow unrelated failures (a different error condition still raises). - the iterator variant (`execute_command_as_iterator`) warns and ends without yielding on the unsupported-API rejection, and still raises by default. All tests in `SparkConnectClientTestCase` pass locally. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
