brijrajk opened a new pull request, #56675: URL: https://github.com/apache/spark/pull/56675
### What changes were proposed in this pull request? Remove the `.internal()` marker from `spark.sql.streaming.minBatchesToRetain` (`MIN_BATCHES_TO_RETAIN` in `SQLConf`) and expand its doc string to also mention its role in checkpoint file cleanup. ### Why are the changes needed? The config controls the minimum number of batches retained for recovery and determines when old state/offset log checkpoint files become eligible for deletion. It is useful for users tuning checkpoint storage, but was marked `.internal()` so it never appeared in the public configuration reference. ### Does this PR introduce _any_ user-facing change? Yes — `spark.sql.streaming.minBatchesToRetain` will now appear in the generated configuration documentation. No behavior change; default value (100) and semantics are unchanged. ### How was this patch tested? Documentation-only change to config metadata. No test changes needed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Sonnet 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
