This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new e9f109abba8 [SPARK-42456][DOCS][PYTHON] Consolidating the PySpark
version upgrade note pages into a single page to make it easier to read
e9f109abba8 is described below
commit e9f109abba8ffe3a238fed23cecc715c30312607
Author: Allan Folting <[email protected]>
AuthorDate: Thu Feb 16 10:35:05 2023 +0900
[SPARK-42456][DOCS][PYTHON] Consolidating the PySpark version upgrade note
pages into a single page to make it easier to read
### What changes were proposed in this pull request?
Creating a new PySpark migration guide sub page and consolidating the
existing 9 separate pages into this one new page. This makes it easier to take
a look across multiple version upgrades by simply scrolling on the page instead
of having to navigate back and forth. Note that this is similar to the Spark
Core Migration Guide page here:
https://spark.apache.org/docs/latest/core-migration-guide.html
Also updating the existing main Migration Guide page to point to this new
sub page and making some minor language updates to help readers.
### Why are the changes needed?
To improve usability of the PySpark doc site.
### Does this PR introduce _any_ user-facing change?
Yes, the user facing PySpark documentation is updated.
### How was this patch tested?
Built and tested the PySpark documentation web site locally.
Closes #40044 from allanf-db/pyspark_doc_updates.
Authored-by: Allan Folting <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 10c0528cf71a092bd78bfec2c7cfe69a3ebd99ec)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
python/docs/source/migration_guide/index.rst | 33 +++---
.../migration_guide/pyspark_1.0_1.2_to_1.3.rst | 23 -----
.../source/migration_guide/pyspark_1.4_to_1.5.rst | 25 -----
.../source/migration_guide/pyspark_2.2_to_2.3.rst | 30 ------
.../pyspark_2.3.0_to_2.3.1_above.rst | 23 -----
.../source/migration_guide/pyspark_2.3_to_2.4.rst | 23 -----
.../source/migration_guide/pyspark_2.4_to_3.0.rst | 44 --------
.../source/migration_guide/pyspark_3.1_to_3.2.rst | 31 ------
.../source/migration_guide/pyspark_3.2_to_3.3.rst | 26 -----
.../source/migration_guide/pyspark_3.3_to_3.4.rst | 43 --------
.../source/migration_guide/pyspark_upgrade.rst | 111 +++++++++++++++++++++
11 files changed, 127 insertions(+), 285 deletions(-)
diff --git a/python/docs/source/migration_guide/index.rst
b/python/docs/source/migration_guide/index.rst
index 43c836fecba..b5ccaee0f8c 100644
--- a/python/docs/source/migration_guide/index.rst
+++ b/python/docs/source/migration_guide/index.rst
@@ -16,37 +16,36 @@
under the License.
-===============
-Migration Guide
-===============
+================
+Migration Guides
+================
-This page describes the migration guide specific to PySpark.
+This page includes links to guides that will help you migrate to PySpark.
+
+If you are upgrading from an older to a newer version of PySpark, refer to
+the following page for differences between versions:
+
+- :ref:`Upgrading PySpark</migration_guide/pyspark_upgrade.rst>`
.. toctree::
:maxdepth: 2
+ :hidden:
- pyspark_3.3_to_3.4
- pyspark_3.2_to_3.3
- pyspark_3.1_to_3.2
- pyspark_2.4_to_3.0
- pyspark_2.3_to_2.4
- pyspark_2.3.0_to_2.3.1_above
- pyspark_2.2_to_2.3
- pyspark_1.4_to_1.5
- pyspark_1.0_1.2_to_1.3
+ pyspark_upgrade
-The guide below is for those who are from `Koalas
<https://koalas.readthedocs.io/en/latest>`_.
+If you are migrating code from `Koalas
<https://koalas.readthedocs.io/en/latest>`_ to PySpark,
+this guide is for you:
.. toctree::
:maxdepth: 2
koalas_to_pyspark
-Many items of other migration guides can also be applied when migrating
PySpark to higher versions because PySpark internally shares other components.
-Please also refer other migration guides:
+A lot of content in other migration guides can also be helpful when migrating
to newer PySpark versions because PySpark internally shares the same components.
+Please also refer to the following migration guides:
-- `Migration Guide: Spark Core
<https://spark.apache.org/docs/latest/core-migration-guide.html>`_
- `Migration Guide: SQL, Datasets and DataFrame
<https://spark.apache.org/docs/latest/sql-migration-guide.html>`_
- `Migration Guide: Structured Streaming
<https://spark.apache.org/docs/latest/ss-migration-guide.html>`_
- `Migration Guide: MLlib (Machine Learning)
<https://spark.apache.org/docs/latest/ml-migration-guide.html>`_
+- `Migration Guide: Spark Core
<https://spark.apache.org/docs/latest/core-migration-guide.html>`_
diff --git a/python/docs/source/migration_guide/pyspark_1.0_1.2_to_1.3.rst
b/python/docs/source/migration_guide/pyspark_1.0_1.2_to_1.3.rst
deleted file mode 100644
index ef2b74b566a..00000000000
--- a/python/docs/source/migration_guide/pyspark_1.0_1.2_to_1.3.rst
+++ /dev/null
@@ -1,23 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-=====================================
-Upgrading from PySpark 1.0-1.2 to 1.3
-=====================================
-
-* When using DataTypes in Python you will need to construct them (i.e.
``StringType()``) instead of referencing a singleton.
\ No newline at end of file
diff --git a/python/docs/source/migration_guide/pyspark_1.4_to_1.5.rst
b/python/docs/source/migration_guide/pyspark_1.4_to_1.5.rst
deleted file mode 100644
index 2b42b17defc..00000000000
--- a/python/docs/source/migration_guide/pyspark_1.4_to_1.5.rst
+++ /dev/null
@@ -1,25 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-=================================
-Upgrading from PySpark 1.4 to 1.5
-=================================
-
-* Resolution of strings to columns in Python now supports using dots (.) to
qualify the column or access nested values. For example
``df['table.column.nestedField']``. However, this means that if your column
name contains any dots you must now escape them using backticks (e.g.,
``table.`column.with.dots`.nested``).
-
-* DataFrame.withColumn method in PySpark supports adding a new column or
replacing existing columns of the same name.
diff --git a/python/docs/source/migration_guide/pyspark_2.2_to_2.3.rst
b/python/docs/source/migration_guide/pyspark_2.2_to_2.3.rst
deleted file mode 100644
index e1b9bffe5a6..00000000000
--- a/python/docs/source/migration_guide/pyspark_2.2_to_2.3.rst
+++ /dev/null
@@ -1,30 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-=================================
-Upgrading from PySpark 2.2 to 2.3
-=================================
-
-* In PySpark, now we need Pandas 0.19.2 or upper if you want to use Pandas
related functionalities, such as ``toPandas``, ``createDataFrame`` from Pandas
DataFrame, etc.
-
-* In PySpark, the behavior of timestamp values for Pandas related
functionalities was changed to respect session timezone. If you want to use the
old behavior, you need to set a configuration
``spark.sql.execution.pandas.respectSessionTimeZone`` to False. See
`SPARK-22395 <https://issues.apache.org/jira/browse/SPARK-22395>`_ for details.
-
-* In PySpark, ``na.fill()`` or ``fillna`` also accepts boolean and replaces
nulls with booleans. In prior Spark versions, PySpark just ignores it and
returns the original Dataset/DataFrame.
-
-* In PySpark, ``df.replace`` does not allow to omit value when ``to_replace``
is not a dictionary. Previously, value could be omitted in the other cases and
had None by default, which is counterintuitive and error-prone.
-
diff --git
a/python/docs/source/migration_guide/pyspark_2.3.0_to_2.3.1_above.rst
b/python/docs/source/migration_guide/pyspark_2.3.0_to_2.3.1_above.rst
deleted file mode 100644
index 4de43b340eb..00000000000
--- a/python/docs/source/migration_guide/pyspark_2.3.0_to_2.3.1_above.rst
+++ /dev/null
@@ -1,23 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-===============================================
-Upgrading from PySpark 2.3.0 to 2.3.1 and above
-===============================================
-
-* As of version 2.3.1 Arrow functionality, including ``pandas_udf`` and
``toPandas()``/``createDataFrame()`` with ``spark.sql.execution.arrow.enabled``
set to ``True``, has been marked as experimental. These are still evolving and
not currently recommended for use in production.
\ No newline at end of file
diff --git a/python/docs/source/migration_guide/pyspark_2.3_to_2.4.rst
b/python/docs/source/migration_guide/pyspark_2.3_to_2.4.rst
deleted file mode 100644
index 394d2bd3bbe..00000000000
--- a/python/docs/source/migration_guide/pyspark_2.3_to_2.4.rst
+++ /dev/null
@@ -1,23 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-=================================
-Upgrading from PySpark 2.3 to 2.4
-=================================
-
-* In PySpark, when Arrow optimization is enabled, previously ``toPandas`` just
failed when Arrow optimization is unable to be used whereas ``createDataFrame``
from Pandas DataFrame allowed the fallback to non-optimization. Now, both
``toPandas`` and ``createDataFrame`` from Pandas DataFrame allow the fallback
by default, which can be switched off by
``spark.sql.execution.arrow.fallback.enabled``.
diff --git a/python/docs/source/migration_guide/pyspark_2.4_to_3.0.rst
b/python/docs/source/migration_guide/pyspark_2.4_to_3.0.rst
deleted file mode 100644
index ad800ddfc34..00000000000
--- a/python/docs/source/migration_guide/pyspark_2.4_to_3.0.rst
+++ /dev/null
@@ -1,44 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-=================================
-Upgrading from PySpark 2.4 to 3.0
-=================================
-
-* In Spark 3.0, PySpark requires a pandas version of 0.23.2 or higher to use
pandas related functionality, such as ``toPandas``, ``createDataFrame`` from
pandas DataFrame, and so on.
-
-* In Spark 3.0, PySpark requires a PyArrow version of 0.12.1 or higher to use
PyArrow related functionality, such as ``pandas_udf``, ``toPandas`` and
``createDataFrame`` with "spark.sql.execution.arrow.enabled=true", etc.
-
-* In PySpark, when creating a ``SparkSession`` with
``SparkSession.builder.getOrCreate()``, if there is an existing
``SparkContext``, the builder was trying to update the ``SparkConf`` of the
existing ``SparkContext`` with configurations specified to the builder, but the
``SparkContext`` is shared by all ``SparkSession`` s, so we should not update
them. In 3.0, the builder comes to not update the configurations. This is the
same behavior as Java/Scala API in 2.3 and above. If you want to [...]
-
-* In PySpark, when Arrow optimization is enabled, if Arrow version is higher
than 0.11.0, Arrow can perform safe type conversion when converting
pandas.Series to an Arrow array during serialization. Arrow raises errors when
detecting unsafe type conversions like overflow. You enable it by setting
``spark.sql.execution.pandas.convertToArrowArraySafely`` to true. The default
setting is false. PySpark behavior for Arrow versions is illustrated in the
following table:
-
- ======================================= ================
=========================
- PyArrow version Integer overflow Floating point
truncation
- ======================================= ================
=========================
- 0.11.0 and below Raise error Silently allows
- > 0.11.0, arrowSafeTypeConversion=false Silent overflow Silently allows
- > 0.11.0, arrowSafeTypeConversion=true Raise error Raise error
- ======================================= ================
=========================
-
-* In Spark 3.0, ``createDataFrame(..., verifySchema=True)`` validates LongType
as well in PySpark. Previously, LongType was not verified and resulted in None
in case the value overflows. To restore this behavior, verifySchema can be set
to False to disable the validation.
-
-* As of Spark 3.0, ``Row`` field names are no longer sorted alphabetically
when constructing with named arguments for Python versions 3.6 and above, and
the order of fields will match that as entered. To enable sorted fields by
default, as in Spark 2.4, set the environment variable
``PYSPARK_ROW_FIELD_SORTING_ENABLED`` to true for both executors and driver -
this environment variable must be consistent on all executors and driver;
otherwise, it may cause failures or incorrect answers. Fo [...]
-
-* In Spark 3.0, ``pyspark.ml.param.shared.Has*`` mixins do not provide any
``set*(self, value)`` setter methods anymore, use the respective
``self.set(self.*, value)`` instead. See `SPARK-29093
<https://issues.apache.org/jira/browse/SPARK-29093>`_ for details.
-
diff --git a/python/docs/source/migration_guide/pyspark_3.1_to_3.2.rst
b/python/docs/source/migration_guide/pyspark_3.1_to_3.2.rst
deleted file mode 100644
index 908d4d34d6e..00000000000
--- a/python/docs/source/migration_guide/pyspark_3.1_to_3.2.rst
+++ /dev/null
@@ -1,31 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-=================================
-Upgrading from PySpark 3.1 to 3.2
-=================================
-
-* In Spark 3.2, the PySpark methods from sql, ml, spark_on_pandas modules
raise the ``TypeError`` instead of ``ValueError`` when are applied to an param
of inappropriate type.
-
-* In Spark 3.2, the traceback from Python UDFs, pandas UDFs and pandas
function APIs are simplified by default without the traceback from the internal
Python workers. In Spark 3.1 or earlier, the traceback from Python workers was
printed out. To restore the behavior before Spark 3.2, you can set
``spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled`` to ``false``.
-
-* In Spark 3.2, pinned thread mode is enabled by default to map each Python
thread to the corresponding JVM thread. Previously,
- one JVM thread could be reused for multiple Python threads, which resulted
in one JVM thread local being shared to multiple Python threads.
- Also, note that now ``pyspark.InheritableThread`` or
``pyspark.inheritable_thread_target`` is recommended to use together for a
Python thread
- to properly inherit the inheritable attributes such as local properties in a
JVM thread, and to avoid a potential resource leak issue.
- To restore the behavior before Spark 3.2, you can set ``PYSPARK_PIN_THREAD``
environment variable to ``false``.
diff --git a/python/docs/source/migration_guide/pyspark_3.2_to_3.3.rst
b/python/docs/source/migration_guide/pyspark_3.2_to_3.3.rst
deleted file mode 100644
index d81008d63cb..00000000000
--- a/python/docs/source/migration_guide/pyspark_3.2_to_3.3.rst
+++ /dev/null
@@ -1,26 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-=================================
-Upgrading from PySpark 3.2 to 3.3
-=================================
-
-* In Spark 3.3, the ``pyspark.pandas.sql`` method follows [the standard Python
string
formatter](https://docs.python.org/3/library/string.html#format-string-syntax).
To restore the previous behavior, set ``PYSPARK_PANDAS_SQL_LEGACY`` environment
variable to ``1``.
-* In Spark 3.3, the ``drop`` method of pandas API on Spark DataFrame supports
dropping rows by ``index``, and sets dropping by index instead of column by
default.
-* In Spark 3.3, PySpark upgrades Pandas version, the new minimum required
version changes from 0.23.2 to 1.0.5.
-* In Spark 3.3, the ``repr`` return values of SQL DataTypes have been changed
to yield an object with the same value when passed to ``eval``.
diff --git a/python/docs/source/migration_guide/pyspark_3.3_to_3.4.rst
b/python/docs/source/migration_guide/pyspark_3.3_to_3.4.rst
deleted file mode 100644
index ca942c54979..00000000000
--- a/python/docs/source/migration_guide/pyspark_3.3_to_3.4.rst
+++ /dev/null
@@ -1,43 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements. See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership. The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied. See the License for the
- specific language governing permissions and limitations
- under the License.
-
-
-=================================
-Upgrading from PySpark 3.3 to 3.4
-=================================
-
-* In Spark 3.4, the schema of an array column is inferred by merging the
schemas of all elements in the array. To restore the previous behavior where
the schema is only inferred from the first element, you can set
``spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled`` to ``true``.
-
-* In Spark 3.4, if Pandas on Spark API ``Groupby.apply``'s ``func`` parameter
return type is not specified and ``compute.shortcut_limit`` is set to 0, the
sampling rows will be set to 2 (ensure sampling rows always >= 2) to make sure
infer schema is accurate.
-
-* In Spark 3.4, if Pandas on Spark API ``Index.insert`` is out of bounds, will
raise IndexError with ``index {} is out of bounds for axis 0 with size {}`` to
follow pandas 1.4 behavior.
-
-* In Spark 3.4, the series name will be preserved in Pandas on Spark API
``Series.mode`` to follow pandas 1.4 behavior.
-
-* In Spark 3.4, the Pandas on Spark API ``Index.__setitem__`` will first to
check ``value`` type is ``Column`` type to avoid raising unexpected
``ValueError`` in ``is_list_like`` like `Cannot convert column into bool:
please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame
boolean expressions.`.
-
-* In Spark 3.4, the Pandas on Spark API ``astype('category')`` will also
refresh ``categories.dtype`` according to original data ``dtype`` to follow
pandas 1.4 behavior.
-
-* In Spark 3.4, the Pandas on Spark API supports groupby positional indexing
in ``GroupBy.head`` and ``GroupBy.tail`` to follow pandas 1.4. Negative
arguments now work correctly and result in ranges relative to the end and start
of each group, Previously, negative arguments returned empty frames.
-
-* In Spark 3.4, the infer schema process of ``groupby.apply`` in Pandas on
Spark, will first infer the pandas type to ensure the accuracy of the pandas
``dtype`` as much as possible.
-
-* In Spark 3.4, the ``Series.concat`` sort parameter will be respected to
follow pandas 1.4 behaviors.
-
-* In Spark 3.4, the ``DataFrame.__setitem__`` will make a copy and replace
pre-existing arrays, which will NOT be over-written to follow pandas 1.4
behaviors.
-
-* In Spark 3.4, the ``SparkSession.sql`` and the Pandas on Spark API ``sql``
have got new parameter ``args`` which provides binding of named parameters to
their SQL literals.
diff --git a/python/docs/source/migration_guide/pyspark_upgrade.rst
b/python/docs/source/migration_guide/pyspark_upgrade.rst
new file mode 100644
index 00000000000..0a590548684
--- /dev/null
+++ b/python/docs/source/migration_guide/pyspark_upgrade.rst
@@ -0,0 +1,111 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+==================
+Upgrading PySpark
+==================
+
+Upgrading from PySpark 3.3 to 3.4
+---------------------------------
+
+* In Spark 3.4, the schema of an array column is inferred by merging the
schemas of all elements in the array. To restore the previous behavior where
the schema is only inferred from the first element, you can set
``spark.sql.pyspark.legacy.inferArrayTypeFromFirstElement.enabled`` to ``true``.
+* In Spark 3.4, if Pandas on Spark API ``Groupby.apply``'s ``func`` parameter
return type is not specified and ``compute.shortcut_limit`` is set to 0, the
sampling rows will be set to 2 (ensure sampling rows always >= 2) to make sure
infer schema is accurate.
+* In Spark 3.4, if Pandas on Spark API ``Index.insert`` is out of bounds, will
raise IndexError with ``index {} is out of bounds for axis 0 with size {}`` to
follow pandas 1.4 behavior.
+* In Spark 3.4, the series name will be preserved in Pandas on Spark API
``Series.mode`` to follow pandas 1.4 behavior.
+* In Spark 3.4, the Pandas on Spark API ``Index.__setitem__`` will first to
check ``value`` type is ``Column`` type to avoid raising unexpected
``ValueError`` in ``is_list_like`` like `Cannot convert column into bool:
please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame
boolean expressions.`.
+* In Spark 3.4, the Pandas on Spark API ``astype('category')`` will also
refresh ``categories.dtype`` according to original data ``dtype`` to follow
pandas 1.4 behavior.
+* In Spark 3.4, the Pandas on Spark API supports groupby positional indexing
in ``GroupBy.head`` and ``GroupBy.tail`` to follow pandas 1.4. Negative
arguments now work correctly and result in ranges relative to the end and start
of each group, Previously, negative arguments returned empty frames.
+* In Spark 3.4, the infer schema process of ``groupby.apply`` in Pandas on
Spark, will first infer the pandas type to ensure the accuracy of the pandas
``dtype`` as much as possible.
+* In Spark 3.4, the ``Series.concat`` sort parameter will be respected to
follow pandas 1.4 behaviors.
+* In Spark 3.4, the ``DataFrame.__setitem__`` will make a copy and replace
pre-existing arrays, which will NOT be over-written to follow pandas 1.4
behaviors.
+* In Spark 3.4, the ``SparkSession.sql`` and the Pandas on Spark API ``sql``
have got new parameter ``args`` which provides binding of named parameters to
their SQL literals.
+
+
+Upgrading from PySpark 3.2 to 3.3
+---------------------------------
+
+* In Spark 3.3, the ``pyspark.pandas.sql`` method follows [the standard Python
string
formatter](https://docs.python.org/3/library/string.html#format-string-syntax).
To restore the previous behavior, set ``PYSPARK_PANDAS_SQL_LEGACY`` environment
variable to ``1``.
+* In Spark 3.3, the ``drop`` method of pandas API on Spark DataFrame supports
dropping rows by ``index``, and sets dropping by index instead of column by
default.
+* In Spark 3.3, PySpark upgrades Pandas version, the new minimum required
version changes from 0.23.2 to 1.0.5.
+* In Spark 3.3, the ``repr`` return values of SQL DataTypes have been changed
to yield an object with the same value when passed to ``eval``.
+
+
+Upgrading from PySpark 3.1 to 3.2
+---------------------------------
+
+* In Spark 3.2, the PySpark methods from sql, ml, spark_on_pandas modules
raise the ``TypeError`` instead of ``ValueError`` when are applied to an param
of inappropriate type.
+* In Spark 3.2, the traceback from Python UDFs, pandas UDFs and pandas
function APIs are simplified by default without the traceback from the internal
Python workers. In Spark 3.1 or earlier, the traceback from Python workers was
printed out. To restore the behavior before Spark 3.2, you can set
``spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled`` to ``false``.
+* In Spark 3.2, pinned thread mode is enabled by default to map each Python
thread to the corresponding JVM thread. Previously,
+ one JVM thread could be reused for multiple Python threads, which resulted
in one JVM thread local being shared to multiple Python threads.
+ Also, note that now ``pyspark.InheritableThread`` or
``pyspark.inheritable_thread_target`` is recommended to use together for a
Python thread
+ to properly inherit the inheritable attributes such as local properties in a
JVM thread, and to avoid a potential resource leak issue.
+ To restore the behavior before Spark 3.2, you can set ``PYSPARK_PIN_THREAD``
environment variable to ``false``.
+
+
+Upgrading from PySpark 2.4 to 3.0
+---------------------------------
+
+* In Spark 3.0, PySpark requires a pandas version of 0.23.2 or higher to use
pandas related functionality, such as ``toPandas``, ``createDataFrame`` from
pandas DataFrame, and so on.
+* In Spark 3.0, PySpark requires a PyArrow version of 0.12.1 or higher to use
PyArrow related functionality, such as ``pandas_udf``, ``toPandas`` and
``createDataFrame`` with "spark.sql.execution.arrow.enabled=true", etc.
+* In PySpark, when creating a ``SparkSession`` with
``SparkSession.builder.getOrCreate()``, if there is an existing
``SparkContext``, the builder was trying to update the ``SparkConf`` of the
existing ``SparkContext`` with configurations specified to the builder, but the
``SparkContext`` is shared by all ``SparkSession`` s, so we should not update
them. In 3.0, the builder comes to not update the configurations. This is the
same behavior as Java/Scala API in 2.3 and above. If you want to [...]
+* In PySpark, when Arrow optimization is enabled, if Arrow version is higher
than 0.11.0, Arrow can perform safe type conversion when converting
pandas.Series to an Arrow array during serialization. Arrow raises errors when
detecting unsafe type conversions like overflow. You enable it by setting
``spark.sql.execution.pandas.convertToArrowArraySafely`` to true. The default
setting is false. PySpark behavior for Arrow versions is illustrated in the
following table:
+
+ ======================================= ================
=========================
+ PyArrow version Integer overflow Floating point
truncation
+ ======================================= ================
=========================
+ 0.11.0 and below Raise error Silently allows
+ > 0.11.0, arrowSafeTypeConversion=false Silent overflow Silently allows
+ > 0.11.0, arrowSafeTypeConversion=true Raise error Raise error
+ ======================================= ================
=========================
+
+* In Spark 3.0, ``createDataFrame(..., verifySchema=True)`` validates LongType
as well in PySpark. Previously, LongType was not verified and resulted in None
in case the value overflows. To restore this behavior, verifySchema can be set
to False to disable the validation.
+* As of Spark 3.0, ``Row`` field names are no longer sorted alphabetically
when constructing with named arguments for Python versions 3.6 and above, and
the order of fields will match that as entered. To enable sorted fields by
default, as in Spark 2.4, set the environment variable
``PYSPARK_ROW_FIELD_SORTING_ENABLED`` to true for both executors and driver -
this environment variable must be consistent on all executors and driver;
otherwise, it may cause failures or incorrect answers. Fo [...]
+* In Spark 3.0, ``pyspark.ml.param.shared.Has*`` mixins do not provide any
``set*(self, value)`` setter methods anymore, use the respective
``self.set(self.*, value)`` instead. See `SPARK-29093
<https://issues.apache.org/jira/browse/SPARK-29093>`_ for details.
+
+
+Upgrading from PySpark 2.3 to 2.4
+---------------------------------
+
+* In PySpark, when Arrow optimization is enabled, previously ``toPandas`` just
failed when Arrow optimization is unable to be used whereas ``createDataFrame``
from Pandas DataFrame allowed the fallback to non-optimization. Now, both
``toPandas`` and ``createDataFrame`` from Pandas DataFrame allow the fallback
by default, which can be switched off by
``spark.sql.execution.arrow.fallback.enabled``.
+
+
+Upgrading from PySpark 2.3.0 to 2.3.1 and above
+-----------------------------------------------
+
+* As of version 2.3.1 Arrow functionality, including ``pandas_udf`` and
``toPandas()``/``createDataFrame()`` with ``spark.sql.execution.arrow.enabled``
set to ``True``, has been marked as experimental. These are still evolving and
not currently recommended for use in production.
+
+
+Upgrading from PySpark 2.2 to 2.3
+---------------------------------
+
+* In PySpark, now we need Pandas 0.19.2 or upper if you want to use Pandas
related functionalities, such as ``toPandas``, ``createDataFrame`` from Pandas
DataFrame, etc.
+* In PySpark, the behavior of timestamp values for Pandas related
functionalities was changed to respect session timezone. If you want to use the
old behavior, you need to set a configuration
``spark.sql.execution.pandas.respectSessionTimeZone`` to False. See
`SPARK-22395 <https://issues.apache.org/jira/browse/SPARK-22395>`_ for details.
+* In PySpark, ``na.fill()`` or ``fillna`` also accepts boolean and replaces
nulls with booleans. In prior Spark versions, PySpark just ignores it and
returns the original Dataset/DataFrame.
+* In PySpark, ``df.replace`` does not allow to omit value when ``to_replace``
is not a dictionary. Previously, value could be omitted in the other cases and
had None by default, which is counterintuitive and error-prone.
+
+
+Upgrading from PySpark 1.4 to 1.5
+---------------------------------
+
+* Resolution of strings to columns in Python now supports using dots (.) to
qualify the column or access nested values. For example
``df['table.column.nestedField']``. However, this means that if your column
name contains any dots you must now escape them using backticks (e.g.,
``table.`column.with.dots`.nested``).
+* DataFrame.withColumn method in PySpark supports adding a new column or
replacing existing columns of the same name.
+
+
+Upgrading from PySpark 1.0-1.2 to 1.3
+-------------------------------------
+
+* When using DataTypes in Python you will need to construct them (i.e.
``StringType()``) instead of referencing a singleton.
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
