rekbun commented on code in PR #59522:
URL: https://github.com/apache/spark/pull/59522#discussion_r4498497880
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala:
##########
@@ -425,16 +426,36 @@ class QueryExecution(
!hasShuffle(executedPlan)) {
this
} else {
- val session = SparkSession.getOrCloneSessionWithConfigsOff(
- sparkSession, Seq(SQLConf.LOCAL_PIPELINED_SHUFFLE_ENABLED))
- new QueryExecution(session, commandExecuted, mode = mode,
- shuffleCleanupModeOpt = shuffleCleanupModeOpt,
- refreshPhaseEnabled = refreshPhaseEnabled, analyzerOpt =
Some(analyzer))
+ withConfigsOff(Seq(SQLConf.LOCAL_PIPELINED_SHUFFLE_ENABLED))
}
}
private[sql] def withRegularShuffle: QueryExecution = lazyRegularShuffle.get
+ /**
+ * A checkpoint reports its plan's output partitioning to the queries that
read it later. Its
+ * root has no consumer when it is planned, so scan packing would withdraw a
keyed partitioning
+ * those queries could use. Plan it with packing off, as well as with
regular shuffles.
+ */
+ private val lazyForCheckpoint = LazyTry {
+ if (!sparkSession.sessionState.conf.v2ScanPartitionPackingEnabled) {
Review Comment:
Done in dc626a7c49a: a checkpoint changes nothing unless putting back the
packed scans that reach a non-adaptive root without crossing an exchange
changes the partitioning or ordering that root reports; then it reuses the
Dataset's prepared plan in the same session, copying only the root path needed
to put those scans back. Exchanges, query stages, adaptive roots and subquery
plans are left untouched; a put-back scan with runtime filters applies them
once more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
