Sudhanva Huruli created SPARK-61825:
---------------------------------------
Summary: Static or core Spark settings in a pipeline's
configuration fail every flow with CANNOT_MODIFY_STATIC_CONFIG
Key: SPARK-61825
URL: https://issues.apache.org/jira/browse/SPARK-61825
Project: Spark
Issue Type: Bug
Components: Declarative Pipelines
Affects Versions: 4.3.0
Reporter: Sudhanva Huruli
The spark-pipelines CLI sets the spec's configuration on the session builder
and also sends the same map to the pipeline graph as the default SQL
configuration, which is applied to every flow. Setting a static SQL
configuration (for example spark.sql.catalogImplementation) or a core setting
(for example spark.driver.memory) on a running session throws, so every flow
fails with CANNOT_MODIFY_STATIC_CONFIG or CANNOT_MODIFY_CONFIG even when the
value matches the session. The server should skip settings that cannot change
at run time when it applies per-flow configuration, and log a warning when the
requested value differs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]