This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new dacb7acadb20 [SPARK-47013][SS] Promote
`spark.sql.streaming.minBatchesToRetain` to a public config
dacb7acadb20 is described below
commit dacb7acadb20345474ea3a21c3a66ff037c86232
Author: BRIJ RAJ KISHORE <[email protected]>
AuthorDate: Tue Jul 21 23:13:52 2026 -0700
[SPARK-47013][SS] Promote `spark.sql.streaming.minBatchesToRetain` to a
public config
### What changes were proposed in this pull request?
Remove the `.internal()` marker from
`spark.sql.streaming.minBatchesToRetain` (`MIN_BATCHES_TO_RETAIN` in `SQLConf`)
and expand its doc string to also mention its role in checkpoint file cleanup.
### Why are the changes needed?
The config controls the minimum number of batches retained for recovery and
determines when old state/offset log checkpoint files become eligible for
deletion. It is useful for users tuning checkpoint storage, but was marked
`.internal()` so it never appeared in the public configuration reference.
### Does this PR introduce _any_ user-facing change?
Yes — `spark.sql.streaming.minBatchesToRetain` will now appear in the
generated configuration documentation. No behavior change; default value (100)
and semantics are unchanged.
### How was this patch tested?
Documentation-only change to config metadata. No test changes needed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6
Closes #56675 from brijrajk/SPARK-47013-document-minBatchesToRetain.
Authored-by: BRIJ RAJ KISHORE <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 6 ++++--
.../spark/sql/execution/streaming/runtime/StreamExecution.scala | 1 -
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 4758d063ca03..f71f27de5c6a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -3302,10 +3302,12 @@ object SQLConf {
.createWithDefault(false)
val MIN_BATCHES_TO_RETAIN =
buildConf("spark.sql.streaming.minBatchesToRetain")
- .internal()
- .doc("The minimum number of batches that must be retained and made
recoverable.")
+ .doc("The minimum number of batches that must be retained and made
recoverable. " +
+ "This also controls the lifecycle of checkpoint files: state, offset and
commit log " +
+ "files older than this many batches are eligible for cleanup. Must be
positive.")
.version("2.1.1")
.intConf
+ .checkValue(_ > 0, "minBatchesToRetain must be positive")
.createWithDefault(100)
val RATIO_EXTRA_SPACE_ALLOWED_IN_CHECKPOINT =
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamExecution.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamExecution.scala
index 65c1226a85df..38b4a58385eb 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamExecution.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamExecution.scala
@@ -88,7 +88,6 @@ abstract class StreamExecution(
protected val pollingDelayMs: Long =
sparkSession.sessionState.conf.streamingPollingDelay
protected val minLogEntriesToMaintain: Int =
sparkSession.sessionState.conf.minBatchesToRetain
- require(minLogEntriesToMaintain > 0, "minBatchesToRetain has to be positive")
/**
* A lock used to wait/notify when batches complete. Use a fair lock to
avoid thread starvation.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
