LuciferYang opened a new pull request, #56861: URL: https://github.com/apache/spark/pull/56861
### What changes were proposed in this pull request? Declarative Pipelines resolves flows in parallel on a shared `SparkSession` (`DataflowGraphTransformer`, parallelism 10). `FlowAnalysis.createFlowFunctionFromLogicalPlan` applied each flow's per-flow SQL confs by mutating that shared session's conf and restoring it afterwards. Because the session is shared, concurrent flows interleave those set/restore operations, so a flow can be analyzed under another flow's confs or have its own conf restored out from under it. This gives each flow a private `SQLConf` instead: clone the session's conf, apply the flow's overrides to the clone, and install it for the analyzing thread with `SQLConf.withExistingConf` while that flow is analyzed (the analyzer reads conf through `SQLConf.get`). Analysis still runs on the shared session, so the catalog, current catalog/database, temp views, and the resolved DataFrames are all left on that session; only the confs the analyzer reads are isolated per flow. ### Why are the changes needed? When more than one flow sets per-flow confs, parallel resolution can analyze a flow under another flow's confs, producing non-deterministic and occasionally incorrect analysis and schema inference. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added three tests in `ConnectValidPipelineSuite`: - one checks that a flow's per-flow conf is what the analyzer reads during analysis, but does not leak onto the session the pipeline is run from; - one resolves several flows in parallel, each setting a different value for the same conf, and asserts every flow's analysis observes its own value; - one resolves a graph through `resolveToDataflowGraph()` with a per-flow `spark.sql.caseSensitive` override and checks that analysis actually honors it: a column reference resolves under the default but fails for the flow that turns on case sensitivity. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
