[
https://issues.apache.org/jira/browse/SPARK-52856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-52856.
-----------------------------------
Fix Version/s: 4.1.0
Resolution: Fixed
> Prevent session state mutation from within Pipelines Python definition files
> ----------------------------------------------------------------------------
>
> Key: SPARK-52856
> URL: https://issues.apache.org/jira/browse/SPARK-52856
> Project: Spark
> Issue Type: Umbrella
> Components: Declarative Pipelines
> Affects Versions: 4.1.0
> Reporter: Sandy Ryza
> Assignee: Jacky Wang
> Priority: Major
> Fix For: 4.1.0
>
>
> Mutating Spark session state is an imperative construct that can cause
> friction and unexpected behavior from within a pipeline declaration. E.g. it
> makes pipeline behavior sensitive to the order that Python files are imported
> in, which can be unpredictable.
> For most elements of session state, there are already existing declarative
> ways to specify these within pipelines.
> Raising an error when someone tries to directly mutate session state would
> help avoid this unpredictable behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
