This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 93b3995ec84b [SPARK-57377][INFRA] Add CI check to prevent new entries
in the config binding policy exceptions file
93b3995ec84b is described below
commit 93b3995ec84bc4cc0a8b921a3e894ddf3ebceff6
Author: Wenchen Fan <[email protected]>
AuthorDate: Thu Jun 11 07:06:01 2026 -0700
[SPARK-57377][INFRA] Add CI check to prevent new entries in the config
binding policy exceptions file
### What changes were proposed in this pull request?
This PR adds a new lightweight CI job `binding-policy` ("Config binding
policy exceptions check") to `build_and_test.yml`. The job diffs
`sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions`
against the latest apache/spark base (using the same fork-sync mechanism as
the existing `buf` job) and fails if the PR adds any entries to the file, with
an error message pointing the author to `.withBindingPolicy()` and the
`ConfigBindingPolicy` s [...]
The job runs on a bare runner in well under a minute, is gated on the
existing `lint` flag from the `precondition` job, only runs on forks (where PR
builds run, so the diff against apache/spark master is exactly the PR's change;
on apache/spark itself the job is skipped at the scheduler level and consumes
no runner), and skips cleanly on branches where the exceptions file does not
exist.
On top of the CI check, this PR also:
- Documents how to choose a binding policy: the `ConfigBindingPolicy`
scaladoc now has a 3-step decision flow (does the config change the result of
resolving a view/UDF/procedure body? should persisted objects freeze the
create-time value, like ANSI mode or session timezone? otherwise SESSION).
`SparkConfigBindingPolicySuite` and the CI job's error message point to that
scaladoc as the single source of truth, and the suite's failure message now
explicitly says not to grow the exceptio [...]
- Fixes the 12 recently-added configs that were put in the exceptions file
instead of declaring a policy, all with `NOT_APPLICABLE` (none of them can
change the result of resolving a view/UDF/procedure body):
`spark.sql.execution.useHashAggregateExec`,
`spark.sql.hive.thriftServer.http.sniHostCheckEnabled`,
`spark.sql.streaming.queryEvolution.enableSinkEvolution`,
`spark.sql.insertNestedTypeCoercion.enabled`,
`spark.testing.injectShuffleFetchFailures`, `spark.ui.jetty.sniHostCheckEnab
[...]
- Relabels the four `spark.sql.window.segmentTree.*` configs from `SESSION`
to `NOT_APPLICABLE` per the review discussion (runtime-identical, but
`NOT_APPLICABLE` is the accurate label for physical-execution toggles).
- Fixes the `version()` of
`spark.sql.streaming.queryEvolution.enableSinkEvolution` from 4.1.0 to 4.3.0
(it was added after the branch-4.2 cut).
### Why are the changes needed?
The exceptions file is a frozen list of pre-existing configs that were
created before the binding policy was introduced, and it should only ever
shrink. `SparkConfigBindingPolicySuite` forces every config to either declare a
binding policy or be listed in this file, so PR authors who add a new config
without a binding policy see a test failure and "fix" it by adding the config
to the exceptions file, which defeats the purpose of the enforcement (see
https://github.com/apache/spark/pul [...]
### Does this PR introduce _any_ user-facing change?
No. `SESSION` and `NOT_APPLICABLE` behave identically at runtime, and
declaring either for a config that previously had no policy only affects
view/UDF resolution for configs that are actually consulted there, which none
of the touched configs are.
### How was this patch tested?
The CI job was verified end to end on this PR itself: a temporary commit
adding an entry to the exceptions file made the job fail with the guidance
message in ~10 seconds
(https://github.com/cloud-fan/spark/actions/runs/27302316185), and the commits
that shrink the file pass the check. The config changes are covered by the
existing `SparkConfigBindingPolicySuite`, which verifies that every config
either declares a policy or is listed in the exceptions file, and that no
config does both.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
This pull request and its description were written by Isaac.
Closes #56437 from cloud-fan/binding-policy-ci-check.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 6693d43d11d7d6fb3fc3aa90bc0601c569e9c6ab)
Signed-off-by: Wenchen Fan <[email protected]>
---
.github/workflows/build_and_test.yml | 49 ++++++++++++++++++++++
.../internal/config/ConfigBindingPolicy.scala | 16 +++++++
.../org/apache/spark/internal/config/History.scala | 1 +
.../org/apache/spark/internal/config/Tests.scala | 1 +
.../org/apache/spark/internal/config/UI.scala | 1 +
.../org/apache/spark/internal/config/package.scala | 5 +++
.../org/apache/spark/sql/internal/SQLConf.scala | 13 +++---
.../apache/spark/sql/internal/StaticSQLConf.scala | 2 +
.../configs-without-binding-policy-exceptions | 12 ------
.../SparkConfigBindingPolicySuite.scala | 15 ++++++-
10 files changed, 96 insertions(+), 19 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 4c39826964ae..a07ec104dc64 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -1035,6 +1035,55 @@ jobs:
if: inputs.branch != 'branch-3.5'
run: ./dev/check-protos.py
+ # The config binding policy exceptions file lists pre-existing configs that
were created before
+ # the binding policy was introduced. It must never grow: new configs must
declare a binding
+ # policy via .withBindingPolicy() instead of being added to the exceptions
file. This job fails
+ # if a PR adds entries to the file. It only runs on forks because PR builds
run there, and the
+ # diff against apache/spark master is exactly the PR's change.
+ binding-policy:
+ needs: [precondition]
+ if: >-
+ (!cancelled()) &&
+ github.repository != 'apache/spark' &&
+ fromJson(needs.precondition.outputs.required).lint == 'true'
+ name: Config binding policy exceptions check
+ runs-on: ubuntu-latest
+ timeout-minutes: 10
+ steps:
+ - name: Checkout Spark repository
+ uses: actions/checkout@v6
+ with:
+ fetch-depth: 0
+ repository: apache/spark
+ ref: ${{ needs.precondition.outputs.head_sha }}
+ - name: Sync the current branch with the latest in Apache Spark
+ run: |
+ echo "APACHE_SPARK_REF=$(git rev-parse HEAD)" >> $GITHUB_ENV
+ git fetch https://github.com/$GITHUB_REPOSITORY.git
${GITHUB_REF#refs/heads/}
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' merge --no-commit --progress --squash
FETCH_HEAD
+ git -c user.name='Apache Spark Test Account' -c
user.email='[email protected]' commit -m "Merged commit" --allow-empty
+ - name: Check no new entries in the binding policy exceptions file
+ run: |
+
EXCEPTIONS_FILE="sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions"
+ if [ ! -f "$EXCEPTIONS_FILE" ]; then
+ echo "$EXCEPTIONS_FILE does not exist on this branch, skipping."
+ exit 0
+ fi
+ ADDED_ENTRIES=$(git diff "$APACHE_SPARK_REF" HEAD --
"$EXCEPTIONS_FILE" | grep '^+' | grep -v '^+++' || true)
+ if [ -n "$ADDED_ENTRIES" ]; then
+ echo "ERROR: This PR adds the following entries to $EXCEPTIONS_FILE:"
+ echo "$ADDED_ENTRIES"
+ echo ""
+ echo "This file is a frozen list of configs that predate the binding
policy, and new"
+ echo "entries must not be added to it. Instead, declare a binding
policy when building"
+ echo "the config entry, e.g."
+ echo "
.withBindingPolicy(ConfigBindingPolicy.SESSION/PERSISTED/NOT_APPLICABLE)"
+ echo "See the scaladoc of
org.apache.spark.internal.config.ConfigBindingPolicy for the"
+ echo "semantics of each policy and how to choose one."
+ exit 1
+ fi
+ echo "No new entries added to the binding policy exceptions file."
+
# Static analysis
lint:
needs: [precondition, infra-image]
diff --git
a/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBindingPolicy.scala
b/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBindingPolicy.scala
index 65009159f609..4369aa71b03a 100644
---
a/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBindingPolicy.scala
+++
b/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBindingPolicy.scala
@@ -29,6 +29,22 @@ package org.apache.spark.internal.config
* is PERSISTED, session-level config changes will not apply to
views/UDFs/procedures but only
* to outer queries. In order for session-level changes to propagate
correctly, this value must
* be explicitly set to SESSION.
+ *
+ * How to choose a policy for a new config:
+ *
+ * 1. Can the config change the result of resolving the body of a
view/UDF/procedure, i.e. the
+ * resolved plan? If not, use NOT_APPLICABLE. This covers most configs.
Note that the test
+ * is whether the config changes the resolution result, not whether it is
read during
+ * resolution: view analysis can itself trigger query execution (e.g. a
schema inference
+ * job), so even physical planning or runtime configs may be read while
resolving a view,
+ * but they do not change what the body resolves to.
+ * 2. If it can, should the persisted object use the create-time value of the
config,
+ * so that it keeps computing the same result as it did at creation no
matter who calls it
+ * later (e.g. ANSI mode, session timezone)? If so, use PERSISTED. Note
that changing query
+ * behavior alone does not justify PERSISTED: a bug-fix flag also changes
query behavior,
+ * but views should not freeze the buggy behavior.
+ * 3. Otherwise, use SESSION: the caller's session value applies, keeping
behavior uniform
+ * across the entire query.
*/
object ConfigBindingPolicy extends Enumeration {
type ConfigBindingPolicy = Value
diff --git a/core/src/main/scala/org/apache/spark/internal/config/History.scala
b/core/src/main/scala/org/apache/spark/internal/config/History.scala
index 90abc9d038db..1936ecf68103 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/History.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/History.scala
@@ -76,6 +76,7 @@ private[spark] object History {
"(e.g., S3 Lifecycle Policies) for those. " +
"Example: \"s3a://.*,gs://.*\" disables scanning for all S3 and GCS
directories.")
.version("4.2.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.stringConf
.toSequence
.createWithDefault(Nil)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/Tests.scala
b/core/src/main/scala/org/apache/spark/internal/config/Tests.scala
index 8ecb14be1dfb..ed88fbe71ffc 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/Tests.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/Tests.scala
@@ -43,6 +43,7 @@ private[spark] object Tests {
ConfigBuilder("spark.testing.injectShuffleFetchFailures")
.doc("Injecting fetch failures for shuffle stages by providing an
invalid BlockManager " +
"location for the first stage attempt. Testing only flag!")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
.createWithDefault(false)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/UI.scala
b/core/src/main/scala/org/apache/spark/internal/config/UI.scala
index 20c99794cd21..3fae4c6d2f78 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/UI.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/UI.scala
@@ -149,6 +149,7 @@ private[spark] object UI {
"backward compatibility with standalone deployments. Set to true to
enforce " +
"SNI host checking for stricter security.")
.version("4.2.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
.createWithDefault(false)
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala
b/core/src/main/scala/org/apache/spark/internal/config/package.scala
index 86e5422a8551..141e033ea992 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/package.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -2393,6 +2393,7 @@ package object config {
.doc("When true, scheduler log messages for streaming tasks include " +
"the structured streaming query ID and batch ID.")
.version("4.2.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
.createWithDefault(true)
@@ -2401,6 +2402,7 @@ package object config {
.doc("Maximum number of characters of the streaming query ID to include
" +
"in scheduler log messages. Set to -1 to include the full query ID.")
.version("4.2.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.intConf
.createWithDefault(5)
@@ -2971,6 +2973,7 @@ package object config {
"Application Master JVM options in client mode. In cluster mode, use "
+
"`spark.driver.limitActiveProcessorCount.enabled` instead.")
.version("4.2.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
.createWithDefault(false)
@@ -2979,6 +2982,7 @@ package object config {
.doc("Whether to add -XX:ActiveProcessorCount=<spark.driver.cores> to
the driver JVM " +
"options. Currently, this only takes effect in YARN cluster mode.")
.version("4.2.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
.createWithDefault(false)
@@ -2987,6 +2991,7 @@ package object config {
.doc("Whether to add -XX:ActiveProcessorCount=<spark.executor.cores> to
executor JVM " +
"options. Currently, this only takes effect in YARN mode.")
.version("4.2.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
.createWithDefault(false)
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 99b21f3b79b4..80a55cf5802d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -3260,7 +3260,8 @@ object SQLConf {
.doc("When true, streaming sinks can be named using the name() API on
DataStreamWriter. " +
"This enables sink evolution capability where sinks can be changed
while maintaining " +
"a historical record of all sinks used in the checkpoint.")
- .version("4.1.0")
+ .version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
.createWithDefault(false)
@@ -3823,6 +3824,7 @@ object SQLConf {
.doc("Decides if we use HashAggregateExec when possible, the rule takes
precedence " +
s"over ${USE_OBJECT_HASH_AGG.key}.")
.version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
.createWithDefault(true)
@@ -4239,7 +4241,7 @@ object SQLConf {
val WINDOW_SEGMENT_TREE_ENABLED =
buildConf("spark.sql.window.segmentTree.enabled")
- .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.doc("Use block-chunked segment tree for moving aggregate window frames
" +
"whose functions are all DeclarativeAggregate without
FILTER/DISTINCT.")
.version("4.2.0")
@@ -4248,7 +4250,7 @@ object SQLConf {
val WINDOW_SEGMENT_TREE_MIN_PARTITION_ROWS =
buildConf("spark.sql.window.segmentTree.minPartitionRows")
- .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.doc("Minimum partition row count to activate the segment-tree moving
frame. " +
"Partitions smaller than this fall back to the default sliding
implementation.")
.version("4.2.0")
@@ -4259,7 +4261,7 @@ object SQLConf {
val WINDOW_SEGMENT_TREE_BLOCK_SIZE =
buildConf("spark.sql.window.segmentTree.blockSize")
- .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.doc("Block size, in rows, for the block-chunked segment tree used by
moving " +
"window frames. Each leaf of the tree aggregates this many consecutive
rows. " +
"Smaller values reduce per-partition memory and speed up tree build
for small " +
@@ -4274,7 +4276,7 @@ object SQLConf {
val WINDOW_SEGMENT_TREE_FANOUT =
buildConf("spark.sql.window.segmentTree.fanout")
- .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.doc("Fanout of internal nodes for the block-chunked segment tree.")
.version("4.2.0")
.internal()
@@ -7446,6 +7448,7 @@ object SQLConf {
"top-level columns are filled with their default value (or null). This
is " +
"experimental and the semantics may change.")
.version("4.2.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
.createWithDefault(false)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
index 25114a93f04c..ad9106ee3e06 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
@@ -20,6 +20,7 @@ package org.apache.spark.sql.internal
import java.util.Locale
import java.util.concurrent.TimeUnit
+import org.apache.spark.internal.config.ConfigBindingPolicy
import
org.apache.spark.sql.connector.catalog.CatalogManager.SESSION_CATALOG_NAME
import org.apache.spark.util.Utils
@@ -128,6 +129,7 @@ object StaticSQLConf {
.doc("Whether to enable Jetty's SNI host check on the
ThriftHttpCLIService HTTPS " +
"connector. Set to false to restore the behavior prior to SPARK-45522
(Jetty 10+).")
.version("4.3.0")
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
.booleanConf
.createWithDefault(true)
diff --git
a/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
b/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
index 26f29ba24d10..a9d53bdb1036 100644
---
a/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
+++
b/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
@@ -43,7 +43,6 @@ spark.driver.extraClassPath
spark.driver.extraJavaOptions
spark.driver.extraLibraryPath
spark.driver.host
-spark.driver.limitActiveProcessorCount.enabled
spark.driver.log.allowErasureCoding
spark.driver.log.dfsDir
spark.driver.log.layout
@@ -120,7 +119,6 @@ spark.executor.id
spark.executor.instances
spark.executor.isolatedSessionCache.size
spark.executor.killOnFatalError.depth
-spark.executor.limitActiveProcessorCount.enabled
spark.executor.logs.redirectConsoleOutputs
spark.executor.logs.rolling.enableCompression
spark.executor.logs.rolling.maxRetainedFiles
@@ -169,7 +167,6 @@ spark.history.fs.numReplayThreads
spark.history.fs.safemodeCheck.interval
spark.history.fs.update.batchSize
spark.history.fs.update.interval
-spark.history.fs.update.scanDisabledPathPatterns
spark.history.kerberos.enabled
spark.history.kerberos.keytab
spark.history.kerberos.principal
@@ -319,8 +316,6 @@ spark.scheduler.numCancelledJobGroupsToTrack
spark.scheduler.resource.profileMergeConflicts
spark.scheduler.revive.interval
spark.scheduler.stage.legacyAbortAfterKillTasks
-spark.scheduler.streaming.idAwareLogging.enabled
-spark.scheduler.streaming.idAwareLogging.queryIdLength
spark.security.credentials.renewalRatio
spark.security.credentials.retryWait
spark.serializer
@@ -603,7 +598,6 @@ spark.sql.execution.replaceHashWithSortAgg
spark.sql.execution.reuseSubquery
spark.sql.execution.sortBeforeRepartition
spark.sql.execution.topKSortFallbackThreshold
-spark.sql.execution.useHashAggregateExec
spark.sql.execution.useObjectHashAggregateExec
spark.sql.execution.usePartitionEvaluator
spark.sql.extendedExplainProviders
@@ -649,7 +643,6 @@ spark.sql.hive.metastorePartitionPruningFastFallback
spark.sql.hive.metastorePartitionPruningInSetThreshold
spark.sql.hive.tablePropertyLengthThreshold
spark.sql.hive.thriftServer.async
-spark.sql.hive.thriftServer.http.sniHostCheckEnabled
spark.sql.hive.thriftServer.singleSession
spark.sql.hive.useDelegateForSymlinkTextInputFormat
spark.sql.hive.version
@@ -661,7 +654,6 @@ spark.sql.inMemoryColumnarStorage.hugeVectorReserveRatio
spark.sql.inMemoryColumnarStorage.hugeVectorThreshold
spark.sql.inMemoryColumnarStorage.partitionPruning
spark.sql.inMemoryTableScanStatistics.enable
-spark.sql.insertNestedTypeCoercion.enabled
spark.sql.join.preferSortMergeJoin
spark.sql.json.enableExactStringParsing
spark.sql.json.enablePartialResults
@@ -1038,7 +1030,6 @@ spark.sql.streaming.numRecentProgressUpdates
spark.sql.streaming.offsetLog.formatVersion
spark.sql.streaming.optimizeOneRowPlan.enabled
spark.sql.streaming.pollingDelay
-spark.sql.streaming.queryEvolution.enableSinkEvolution
spark.sql.streaming.queryEvolution.enableSourceEvolution
spark.sql.streaming.ratioExtraSpaceAllowedInCheckpoint
spark.sql.streaming.realTimeMode.allowlistCheck
@@ -1177,7 +1168,6 @@ spark.taskMetrics.trackUpdatedBlockStatuses
spark.test.noStageRetry
spark.testing
spark.testing.dynamicAllocation.schedule.enabled
-spark.testing.injectShuffleFetchFailures
spark.testing.memory
spark.testing.nCoresPerExecutor
spark.testing.nExecutorsPerHost
@@ -1194,7 +1184,6 @@ spark.ui.enabled
spark.ui.filters
spark.ui.groupSQLSubExecutionEnabled
spark.ui.heapHistogramEnabled
-spark.ui.jetty.sniHostCheckEnabled
spark.ui.jetty.stopTimeout
spark.ui.killEnabled
spark.ui.liveUpdate.minFlushPeriod
@@ -1229,7 +1218,6 @@ spark.unsafe.sorter.spill.read.ahead.enabled
spark.unsafe.sorter.spill.reader.buffer.size
spark.user.groups.mapping
spark.worker.executorStateSync.maxAttempts
-spark.yarn.am.limitActiveProcessorCount.enabled
spark.yarn.dist.forceDownloadSchemes
spark.yarn.dist.pyFiles
spark.yarn.isPython
diff --git
a/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala
b/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala
index e7b4669e2b96..78c3184d351e 100644
---
a/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala
+++
b/sql/hive/src/test/scala/org/apache/spark/sql/configaudit/SparkConfigBindingPolicySuite.scala
@@ -25,6 +25,14 @@ import
org.apache.spark.internal.config.{ConfigBindingPolicy, ConfigEntry}
import org.apache.spark.sql.hive.HiveUtils
import org.apache.spark.sql.internal.SQLConf
+/**
+ * Enforces that every Spark config declares a `ConfigBindingPolicy`. The
exceptions file
+ * `conf/binding-policy-exceptions/configs-without-binding-policy-exceptions`
is a frozen list
+ * of configs that predate the binding policy and must only ever shrink: new
configs must
+ * declare a policy via `.withBindingPolicy()` when building the config entry
(see the
+ * [[ConfigBindingPolicy]] scaladoc for how to choose a policy). The
`binding-policy` CI job
+ * rejects any PR that adds entries to the exceptions file.
+ */
class SparkConfigBindingPolicySuite extends SparkFunSuite {
override def beforeAll(): Unit = {
@@ -57,8 +65,11 @@ class SparkConfigBindingPolicySuite extends SparkFunSuite {
if (missingBindingPolicyConfigs.nonEmpty) {
fail(
s"The following configs do not have bindingPolicy field set. You need
to define it " +
- "by using .withBindingPolicy(ConfigBindingPolicy.SESSION/PERSISTED)
when you build " +
- "the config entry.\n" +
+ "by using
.withBindingPolicy(ConfigBindingPolicy.SESSION/PERSISTED/NOT_APPLICABLE) " +
+ "when you build the config entry. See the ConfigBindingPolicy scaladoc
for how to " +
+ "choose a policy. DO NOT add new entries to the exceptions file: it is
a frozen " +
+ "list of configs that predate the binding policy and must only ever
shrink (the " +
+ "binding-policy CI job rejects any addition).\n" +
missingBindingPolicyConfigs.mkString("\n")
)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
