cloud-fan commented on code in PR #57842:
URL: https://github.com/apache/spark/pull/57842#discussion_r3746492238
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamExecution.scala:
##########
@@ -458,6 +463,87 @@ abstract class StreamExecution(
}
}
+ /**
+ * Applies the configuration a Real-Time Mode query needs but that is not
the engine-wide default,
+ * because it is only the right choice for a low-latency, long-running
batch. Runs once at query
+ * start, before the logical plan is forced, so a config read during
planning sees the final
+ * value.
+ *
+ * There are two kinds of setting here, and the difference is deliberate:
+ *
+ * - SOFT DEFAULTS, applied only when the user has not set the key, so an
explicit choice always
+ * wins. These are performance choices, not correctness requirements, so
a user who sets one is
+ * assumed to mean it. The state store settings and
`changelogCheckpointing` are these.
+ * - HARD OVERRIDES, applied unconditionally because honouring the user's
value would break the
Review Comment:
Please update this section to describe `sortBeforeRepartition` as a soft
default whose incompatible explicit value is rejected by the earlier preflight.
The current text says it is overridden unconditionally and warning-logged, but
this method only sets it when absent.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/CommitLog.scala:
##########
@@ -76,7 +70,12 @@ class CommitLog(
/**
* Factory for creating a [[CommitMetadataBase]] for the requested wire
format version.
- * Defaults to the version configured via
[[SQLConf.STATE_STORE_CHECKPOINT_FORMAT_VERSION]].
+ *
+ * The version is a parameter rather than a field read from the session
config, so that a caller
+ * always supplies the version it resolved for this query run. Reading the
config here would
+ * capture whatever value happened to be set when this log was constructed
-- which is during
+ * [[StreamExecution]]'s own initialization, before per-query configuration
is applied -- and
Review Comment:
Please qualify this as
`[[org.apache.spark.sql.execution.streaming.runtime.StreamExecution]]`. The
bare link is unresolved from the `checkpointing` package because that runtime
class is neither in this package nor imported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
