Sandy Ryza created SPARK-52856:
----------------------------------
Summary: Prevent session state mutation from within Pipelines
Python definition files
Key: SPARK-52856
URL: https://issues.apache.org/jira/browse/SPARK-52856
Project: Spark
Issue Type: Improvement
Components: Declarative Pipelines
Affects Versions: 4.1.0
Reporter: Sandy Ryza
Mutating Spark session state is an imperative construct that can cause friction
and unexpected behavior from within a pipeline declaration. E.g. it makes
pipeline behavior sensitive to the order that Python files are imported in,
which can be unpredictable.
For most elements of session state, there are already existing declarative ways
to specify these within pipelines.
Raising an error when someone tries to directly mutate session state would help
avoid this unpredictable behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
