uros-b commented on code in PR #56662:
URL: https://github.com/apache/spark/pull/56662#discussion_r3452345470
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowAnalysis.scala:
##########
@@ -44,17 +44,19 @@ object FlowAnalysis {
confs: Map[String, String],
queryContext: QueryContext,
queryOrigin: QueryOrigin) => {
+ // Flows are resolved in parallel on a shared session, so applying
per-flow confs by mutating
+ // that session would race across flows. Resolve each flow against its
own cloned session
+ // instead, which keeps its confs isolated from other flows and from the
session the pipeline
+ // is run from.
val ctx = FlowAnalysisContext(
allInputs = allInputs,
availableInputs = availableInputs,
queryContext = queryContext,
- spark = SparkSession.active
+ spark = SparkSession.active.cloneSession()
Review Comment:
With the per-flow `cloneSession()`, each flow resolution now clones a
session (which clones SessionState, including the catalog); whereas the
previous code reused one session. Do we have a rough idea about the perf
impact? Is it negligible or substantial?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
