Haoyan Geng created SPARK-59135:
-----------------------------------
Summary: Spark Connect command streams fail on valid
non-dictionary ExecutePlan responses
Key: SPARK-59135
URL: https://issues.apache.org/jira/browse/SPARK-59135
Project: Spark
Issue Type: Bug
Components: Connect, PySpark
Affects Versions: 4.2.0
Reporter: Haoyan Geng
### Problem
`SparkConnectClient.execute_command_as_iterator` assumes every decoded
`ExecutePlanResponse` item is a dictionary. However, the response decoder
legitimately produces other values, including `PlanMetrics`,
`PlanObservedMetrics`, schemas, Arrow batches, and protobuf extensions.
Python Data Source workers currently register profiler accumulators even when
`spark.sql.pyspark.dataSource.profiler` is unset. Their empty accumulator
updates can therefore be returned as observed metrics alongside command
results. When this happens, `execute_command_as_iterator` raises
`UNKNOWN_RESPONSE` and terminates the command stream.
### Impact
This affects PySpark Spark Connect clients. Known affected consumers are the
streaming-query listener bus and Declarative Pipelines event stream, so the
problem is not limited to streaming queries. Accumulators are currently
registered unconditionally.
### Expected behavior
The low-level command iterator should preserve every valid decoded response.
Command-specific consumers should ignore recognized execution metadata while
continuing to reject malformed or unrelated responses. Python Data Source
workers should only create profiler accumulators when performance or memory
profiling is enabled. Ordinary accumulator updates must remain unaffected.
### Proposed change
- Make `execute_command_as_iterator` yield every decoded response item.
- Let streaming-listener and pipeline consumers ignore `PlanMetrics` and
`ObservedMetrics`.
- Continue raising errors for unexpected command-specific responses.
- Register Python Data Source profiler accumulators only for `perf` and
`memory` profiling modes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
