[
https://issues.apache.org/jira/browse/SPARK-38912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Furcy Pin updated SPARK-38912:
------------------------------
Description:
*Context*
The proper bugfix of SPARK-38870 would be to use the @classmethod + @property
decorators on the SparkSession.builder attribute to make it return a new
builder every time like in Scala.
However, @classmethod + @property decorators have only been made compatible
since Python 3.9:
[https://docs.python.org/3/howto/descriptor.html#class-methods]
> The code path for {{hasattr(type(self.f), '__get__')}} was added in Python
> 3.9 and makes it possible for
> [{{classmethod()}}|https://docs.python.org/3/library/functions.html#classmethod]
> to support chained decorators.
Moreover, @classmethod + @property waq also affected by a bug which has been
been fixed in Python 3.9.6:
https://github.com/python/cpython/pull/28838
So we had to use a temporary workaround by defining a custom decorator named
@classproperty to fix SPARK-38870.
*What should be done*
Once support for Python 3.8 and "<=3.9.5" is officially dropped, we should
remove this @classproperty workaround and replace it with @classmethod +
@property (cf TODOs in the [pyspark.sql.session
module|https://github.com/apache/spark/pull/36161/files]).
*Further details*
_Python 3.8 will end-of-life is 2024/10._
was:
The proper bugfix of SPARK-38870 uses a feature of Python 3.9
and had to use a workaround for earlier version (cf TODOs in the
[pyspark.sql.session module|https://github.com/apache/spark/pull/36161/files])
Once support for Python 3.8 is officially dropped, we should remove this
workaround.
_Python 3.8 will end-of-life is 2024/10._
> Clean "classproperty" workaround in pyspark.sql.session once support for
> Python 3.8 is dropped
> ----------------------------------------------------------------------------------------------
>
> Key: SPARK-38912
> URL: https://issues.apache.org/jira/browse/SPARK-38912
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 3.2.1
> Reporter: Furcy Pin
> Priority: Minor
>
> *Context*
> The proper bugfix of SPARK-38870 would be to use the @classmethod + @property
> decorators on the SparkSession.builder attribute to make it return a new
> builder every time like in Scala.
> However, @classmethod + @property decorators have only been made compatible
> since Python 3.9:
> [https://docs.python.org/3/howto/descriptor.html#class-methods]
> > The code path for {{hasattr(type(self.f), '__get__')}} was added in Python
> > 3.9 and makes it possible for
> > [{{classmethod()}}|https://docs.python.org/3/library/functions.html#classmethod]
> > to support chained decorators.
> Moreover, @classmethod + @property waq also affected by a bug which has been
> been fixed in Python 3.9.6:
> https://github.com/python/cpython/pull/28838
> So we had to use a temporary workaround by defining a custom decorator named
> @classproperty to fix SPARK-38870.
> *What should be done*
> Once support for Python 3.8 and "<=3.9.5" is officially dropped, we should
> remove this @classproperty workaround and replace it with @classmethod +
> @property (cf TODOs in the [pyspark.sql.session
> module|https://github.com/apache/spark/pull/36161/files]).
>
> *Further details*
> _Python 3.8 will end-of-life is 2024/10._
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
