This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
from 9fbefb3e575c Revert "Revert "[SPARK-50227][CONNECT][PYTHON] Upgrade
buf plugins to `v28.3`""
add 36410f073cb9 [SPARK-49601][SS][PYTHON] Support Initial State Handling
for TransformWithStateInPandas
No new revisions were added by this update.
Summary of changes:
.../org/apache/spark/api/python/PythonRunner.scala | 3 +
.../reference/pyspark.sql/stateful_processor.rst | 3 +-
python/pyspark/sql/pandas/_typing/__init__.pyi | 1 +
python/pyspark/sql/pandas/functions.py | 2 +
python/pyspark/sql/pandas/group_ops.py | 108 +++++++++--
python/pyspark/sql/pandas/serializers.py | 76 ++++++++
python/pyspark/sql/streaming/stateful_processor.py | 7 +
.../pandas/test_pandas_transform_with_state.py | 197 +++++++++++++++++++
python/pyspark/util.py | 5 +-
python/pyspark/worker.py | 70 ++++++-
.../plans/logical/pythonLogicalOperators.scala | 45 ++++-
.../spark/sql/RelationalGroupedDataset.scala | 65 +++++--
.../spark/sql/execution/SparkStrategies.scala | 13 +-
.../python/TransformWithStateInPandasExec.scala | 215 +++++++++++++++------
.../TransformWithStateInPandasPythonRunner.scala | 180 ++++++++++++-----
.../execution/streaming/IncrementalExecution.scala | 4 +-
16 files changed, 846 insertions(+), 148 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
