Sandy Ryza created SPARK-52853:
----------------------------------
Summary: Prevent setting Spark confs on the session within
Pipelines Python definition files
Key: SPARK-52853
URL: https://issues.apache.org/jira/browse/SPARK-52853
Project: Spark
Issue Type: Improvement
Components: Declarative Pipelines
Affects Versions: 4.1.0
Reporter: Sandy Ryza
Setting spark session confs is an imperative construct that can cause friction
and unexpected behavior from within a pipeline declaration. E.g. it makes
pipeline behavior sensitive to the order that Python files are imported in,
which can be unpredictable. There are already existing mechanisms for setting
Spark confs for pipelines:
* The configuration setting in the pipeline spec
* The `spark_conf` arguments on various Python decorators
Raising an error when someone tries to invoke to set a conf in this situation
would avoid this unpredictable behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
