[
https://issues.apache.org/jira/browse/SPARK-61825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146479#comment-18146479
]
Andreas Neumann commented on SPARK-61825:
-----------------------------------------
This is inaccurate. You will hit this problem equally with plain Spark. The
catalog implementation can only be configured once, and that is at startup.
Later attempts to set this conf will log the observed error and be ignored. The
pipeline spec should never contain that conf. But if it does, then the observed
behavior is exactly as expected.
> Static or core Spark settings in a pipeline's configuration fail every flow
> with CANNOT_MODIFY_STATIC_CONFIG
> ------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61825
> URL: https://issues.apache.org/jira/browse/SPARK-61825
> Project: Spark
> Issue Type: Bug
> Components: Declarative Pipelines
> Affects Versions: 4.3.0
> Reporter: Sudhanva Huruli
> Priority: Major
> Labels: pull-request-available
>
> The spark-pipelines CLI sets the spec's configuration on the session builder
> and also sends the same map to the pipeline graph as the default SQL
> configuration, which is applied to every flow. Setting a static SQL
> configuration (for example spark.sql.catalogImplementation) or a core setting
> (for example spark.driver.memory) on a running session throws, so every flow
> fails with CANNOT_MODIFY_STATIC_CONFIG or CANNOT_MODIFY_CONFIG even when the
> value matches the session. The server should skip settings that cannot change
> at run time when it applies per-flow configuration, and log a warning when
> the requested value differs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]