This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
from 9c40c126b799 [SPARK-53946][BUILD] Upgrade SBT to 1.11.7
add 57b4cd2ccc64 [SPARK-53696][PYTHON][CONNECT][SQL] Default to bytes for
BinaryType in PySpark
No new revisions were added by this update.
Summary of changes:
docs/sql-ref-datatypes.md | 2 +-
.../source/migration_guide/pyspark_upgrade.rst | 11 +++
.../docs/source/tutorial/sql/type_conversions.rst | 5 +-
python/pyspark/sql/avro/functions.py | 6 +-
python/pyspark/sql/connect/dataframe.py | 21 ++++-
python/pyspark/sql/conversion.py | 41 ++++----
python/pyspark/sql/functions/builtin.py | 8 +-
python/pyspark/sql/pandas/serializers.py | 10 +-
.../sql/tests/arrow/test_arrow_python_udf.py | 85 ++++++++++++++++-
.../connect/arrow/test_parity_arrow_python_udf.py | 18 ++++
.../sql/tests/connect/client/test_client.py | 14 +++
.../sql/tests/connect/test_connect_collection.py | 79 ++++++++++++++++
python/pyspark/sql/tests/test_conversion.py | 37 ++++++++
python/pyspark/sql/tests/test_readwriter.py | 50 +++++++++-
python/pyspark/sql/tests/test_udf.py | 84 +++++++++++++++++
python/pyspark/sql/tests/test_udtf.py | 31 ++++++
.../golden_pandas_udf_input_types.txt | 4 +-
.../golden_pandas_udf_return_type_coercion.txt | 2 +-
.../golden_udf_input_types_arrow_disabled.txt | 4 +-
.../golden_udf_input_types_arrow_enabled.txt | 4 +-
.../golden_udf_input_types_arrow_legacy_pandas.txt | 4 +-
...den_udf_return_type_coercion_arrow_disabled.txt | 2 +-
...lden_udf_return_type_coercion_arrow_enabled.txt | 2 +-
...df_return_type_coercion_arrow_legacy_pandas.txt | 2 +-
.../sql/worker/data_source_pushdown_filters.py | 4 +-
python/pyspark/sql/worker/plan_data_source_read.py | 9 +-
.../pyspark/sql/worker/write_into_data_source.py | 9 +-
python/pyspark/testing/connectutils.py | 32 +++++++
python/pyspark/worker.py | 13 ++-
.../org/apache/spark/sql/internal/SQLConf.scala | 15 +++
.../spark/sql/api/python/PythonSQLUtils.scala | 2 +-
.../org/apache/spark/sql/classic/Dataset.scala | 14 ++-
.../v2/python/UserDefinedPythonDataSource.scala | 3 +
.../sql/execution/python/ArrowPythonRunner.scala | 5 +-
.../sql/execution/python/BatchEvalPythonExec.scala | 16 ++--
.../execution/python/BatchEvalPythonUDTFExec.scala | 3 +-
.../sql/execution/python/EvaluatePython.scala | 105 +++++++++++++++------
.../python/UserDefinedPythonFunction.scala | 3 +-
.../python/streaming/PythonForeachWriter.scala | 4 +-
39 files changed, 666 insertions(+), 97 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
