[
https://issues.apache.org/jira/browse/SPARK-38912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-38912:
------------------------------------
Assignee: Furcy Pin
> Clean "classproperty" workaround in pyspark.sql.session once support for
> Python 3.8 is dropped
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-38912
> URL: https://issues.apache.org/jira/browse/SPARK-38912
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 3.2.1
> Reporter: Furcy Pin
> Assignee: Furcy Pin
> Priority: Minor
> Labels: pull-request-available
>
> *Context*
> The proper bugfix of SPARK-38870 would be to use the @classmethod + @property
> decorators on the SparkSession.builder attribute to make it return a new
> builder every time like in Scala.
> However, @classmethod + @property decorators have only been made compatible
> since Python 3.9:
> [https://docs.python.org/3/howto/descriptor.html#class-methods]
> > The code path for {{hasattr(type(self.f), '_{_}get{_}_')}} was added in
> > Python 3.9 and makes it possible for
> > [{{classmethod()}}|https://docs.python.org/3/library/functions.html#classmethod]
> > to support chained decorators.
> Moreover, @classmethod + @property waq also affected by a bug which has been
> been fixed in Python 3.9.6 (see also SPARK-38964) :
> [https://github.com/python/cpython/pull/28838]
> So we had to use a temporary workaround by defining a custom decorator named
> @classproperty to fix SPARK-38870.
> *What should be done*
> Once support for Python 3.8 and "<=3.9.5" is officially dropped, we should
> remove this @classproperty workaround and replace it with @classmethod +
> @property (cf TODOs in the [pyspark.sql.session
> module|https://github.com/apache/spark/pull/36161/files]).
>
> *Further details*
> _Python 3.8 will end-of-life is 2024/10._
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
