This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
from aac494e74c60 [SPARK-50134][SPARK-50130][SQL][CONNECT] Support
DataFrame API for SCALAR and EXISTS subqueries in Spark Connect
add c92021091502 [SPARK-50578][PYTHON][SS] Add support for new version of
state metadata for TransformWithStateInPandas
No new revisions were added by this update.
Summary of changes:
.../spark/api/python/StreamingPythonRunner.scala | 16 +-
python/pyspark/sql/pandas/group_ops.py | 24 ++
.../sql/streaming/proto/StateMessage_pb2.py | 4 +-
.../sql/streaming/proto/StateMessage_pb2.pyi | 22 +-
.../sql/streaming/stateful_processor_api_client.py | 14 +-
.../sql/streaming/stateful_processor_util.py | 1 +
.../transform_with_state_driver_worker.py | 102 +++++
.../pandas/test_pandas_transform_with_state.py | 423 ++++++++++++++++++++-
.../sql/execution/streaming/StateMessage.proto | 11 +-
.../datasources/v2/state/StateDataSource.scala | 11 +-
.../python/TransformWithStateInPandasExec.scala | 65 +++-
.../TransformWithStateInPandasPythonRunner.scala | 134 +++++--
.../TransformWithStateInPandasStateServer.scala | 11 +-
.../execution/streaming/IncrementalExecution.scala | 2 +-
.../streaming/TransformWithStateExec.scala | 103 ++---
.../TransformWithStateVariableUtils.scala | 110 +++++-
16 files changed, 887 insertions(+), 166 deletions(-)
create mode 100644
python/pyspark/sql/streaming/transform_with_state_driver_worker.py
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
