This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new abc39b74f753 [SPARK-56460][CORE] Define configs in text files
abc39b74f753 is described below
commit abc39b74f753fdc395f3979716d926ede7569697
Author: Wenchen Fan <[email protected]>
AuthorDate: Mon Jul 27 23:45:45 2026 +0800
[SPARK-56460][CORE] Define configs in text files
### What changes were proposed in this pull request?
This PR adds a new module `common/config` which introduces a framework to
define configs in prototext files. When the Spark application starts, we load
config definitions from all prototext files and put them in a map with config
name as the key, and the proto binary of the config definition as the value.
These proto-backed configs are also registered in `ConfigEntry.knownConfigs`
via a wrapper `ProtoBackedConfigEntry`, together with existing Spark configs.
The proto schema defines each config entry with: key, value type, default
value, scope (cluster vs session), mutability (static vs dynamic, i.e. whether
the config can be changed after system initialization), visibility (public vs
internal), binding policy (session/persisted/not_applicable for SQL views,
UDFs, and procedures), documentation, and version. Enum values follow proto3
naming conventions with type-name prefixes (e.g. `SCOPE_CLUSTER`,
`VALUE_TYPE_BOOL`, `VISIBILITY_PUBLIC`, [...]
The new framework will co-exist with the existing config framework, until
we migrate all existing configs to this new framework.
For simple configs that are only accessed in one place, we can hardcode the
config name in the place that accesses it, with a new API
`SQLConf#getConfByKeyStrict` to avoid typo. Example in this PR:
`spark.sql.optimizer.datasourceV2ExprFolding`
For configs that are accessed in multiple places and we want to avoid
hardcoding config name, or configs that need custom validation, we can still
have an entry in `object SQLConf` to reference the config definition. Examples
in this PR: `spark.sql.optimizer.maxIterations` and
`spark.sql.shuffledHashJoinFactor`.
As part of the migration, the 4 example configs are removed from the
binding-policy exceptions allowlist
(`configs-without-binding-policy-exceptions`) and assigned `NOT_APPLICABLE`,
aligning them with standard binding-policy handling. Since `NOT_APPLICABLE`
configs do not affect view/UDF/procedure resolution results, no resolved plans
change.
### Why are the changes needed?
Defining configs in various Scala objects is a bad practice:
- Configs are registered when the Scala objects are loaded. To list all
configs we must know all these Scala objects and load them.
- It's hard to audit configs as they spread all over the codebase.
- We will hit JVM limitation one day sooner or later, as defining configs
in a Scala object is basically doing heavy work in the constructor, which has
limitation of 64 kb bytecode size.
### Does this PR introduce _any_ user-facing change?
No, it's developer facing
### How was this patch tested?
new tests
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #53488 from cloud-fan/conf.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 2d5e1b65350d0ae3d61c94c034a5207221a88950)
Signed-off-by: Wenchen Fan <[email protected]>
---
common/config/README.md | 122 +++++++
common/config/pom.xml | 187 ++++++++++
.../org/apache/spark/config/ConfigRegistry.java | 210 +++++++++++
common/config/src/main/protobuf/buf.yaml | 24 ++
.../org/apache/spark/config/config_schema.proto | 108 ++++++
.../spark/config/cluster_configs/sql.textproto | 28 ++
.../spark/config/session_configs/sql.textproto | 54 +++
.../apache/spark/config/ConfigRegistrySuite.java | 382 +++++++++++++++++++++
.../spark/config/duplicate_key_file2.textproto | 28 ++
.../invalid_missing_binding_policy.textproto | 27 ++
.../spark/config/invalid_missing_doc.textproto | 27 ++
.../spark/config/invalid_missing_key.textproto | 27 ++
.../config/invalid_missing_mutability.textproto | 27 ++
.../spark/config/invalid_missing_scope.textproto | 27 ++
.../config/invalid_missing_value_type.textproto | 27 ++
.../config/invalid_missing_visibility.textproto | 27 ++
.../invalid_testdefault_without_default.textproto | 28 ++
.../spark/config/invalid_unordered.textproto | 40 +++
.../org/apache/spark/config/test_configs.textproto | 91 +++++
common/utils/pom.xml | 5 +
.../apache/spark/internal/config/ConfigEntry.scala | 189 +++++++++-
dev/mima | 10 +-
pom.xml | 1 +
project/SparkBuild.scala | 36 +-
.../sql/catalyst/util/V2ExpressionBuilder.scala | 2 +-
.../org/apache/spark/sql/internal/SQLConf.scala | 84 +++--
.../apache/spark/sql/internal/StaticSQLConf.scala | 8 +-
.../org/apache/spark/sql/RuntimeConfigSuite.scala | 2 +
.../datasources/v2/DataSourceV2StrategySuite.scala | 2 +-
.../apache/spark/sql/internal/SQLConfSuite.scala | 83 +++++
.../configs-without-binding-policy-exceptions | 4 -
31 files changed, 1873 insertions(+), 44 deletions(-)
diff --git a/common/config/README.md b/common/config/README.md
new file mode 100644
index 000000000000..67c74ac48be2
--- /dev/null
+++ b/common/config/README.md
@@ -0,0 +1,122 @@
+# Spark Configuration Module
+
+This module provides a proto-based configuration system for defining Spark
configurations. Configurations are defined using Protocol Buffers text format
(`.textproto` files) and loaded at runtime by `ConfigRegistry`.
+
+## Directory Structure
+
+```
+common/config/src/main/
+├── java/org/apache/spark/config/
+│ └── ConfigRegistry.java # Loads and queries configs
+├── protobuf/org/apache/spark/config/
+│ └── config_schema.proto # Defines the config schema
+└── resources/org/apache/spark/config/
+ ├── cluster_configs/ # CLUSTER scope configs
+ │ └── sql.textproto
+ └── session_configs/ # SESSION scope configs
+ └── sql.textproto
+```
+
+## How to Define a New Config
+
+### Step 1: Choose the Right File
+
+- **CLUSTER scope** (`MUTABILITY_STATIC`): If the config applies to the entire
Spark cluster and cannot be changed at runtime, add to a file in
`cluster_configs/` directory.
+- **SESSION scope** (`MUTABILITY_DYNAMIC`): If the config applies to a Spark
session and can be changed at runtime (e.g., via `SET`), add to a file in
`session_configs/` directory.
+
+Currently, SQL configs go in `sql.textproto`. New categories can have their
own files.
+
+### Step 2: Add the Config Entry
+
+Add a new `configs` block to the appropriate `.textproto` file:
+
+```protobuf
+configs {
+ key: "spark.sql.myFeature.enabled"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_SESSION
+ doc: "When true, enables my new feature."
+ version: "4.0.0"
+}
+```
+
+**Notes**:
+- See `config_schema.proto` for field details and requirements
+- Configs within each file must be ordered alphabetically by key
+- A config that sets `test_default` must also set `default_value` (a test-only
override needs a
+ production default to override); this is validated at load time
+- These constraints are validated by tests
+
+### Step 3: Register the Config File (if new)
+
+If you created a new `.textproto` file, add it to `ConfigRegistry.java`:
+
+```java
+private static final String[] DEFAULT_CONFIG_FILES = {
+ "org/apache/spark/config/cluster_configs/sql.textproto",
+ "org/apache/spark/config/session_configs/sql.textproto",
+ "org/apache/spark/config/session_configs/your_new_file.textproto" // Add
here
+};
+```
+
+## Multi-line Documentation
+
+For long documentation strings, use protobuf string concatenation:
+
+```protobuf
+configs {
+ key: "spark.sql.myFeature.threshold"
+ value_type: VALUE_TYPE_INT
+ default_value: "100"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "This is a long documentation string that spans multiple lines. "
+ "Simply place multiple quoted strings adjacent to each other "
+ "and protobuf will concatenate them automatically."
+ version: "4.0.0"
+}
+```
+
+## Using Configs in Spark
+
+### Reading Config Values
+
+For configs accessed in only one or a few places, use `getConfByKeyStrict`
with the config key directly. This will fail at runtime if the key is not
defined in a `.textproto` file:
+
+```scala
+def myFunc(conf: SQLConf): Unit = {
+ if (conf.getConfByKeyStrict[Boolean]("spark.sql.myFeature.enabled")) {
+ doSomething()
+ }
+}
+```
+
+For configs accessed in many places, create a `ConfigEntry` in `object
SQLConf` (or its friends) to avoid hardcoding keys:
+
+```scala
+object SQLConf {
+ val MY_FEATURE_ENABLED =
buildConfFromConfigFile[Boolean]("spark.sql.myFeature.enabled")
+}
+
+def myFunc(conf: SQLConf): Unit = {
+ if (conf.getConf(SQLConf.MY_FEATURE_ENABLED)) {
+ doSomething()
+ }
+}
+```
+
+### Adding Config Value Validation
+
+For configs that need to validate their values, create a `ConfigEntry` and use
`checkValue`:
+
+```scala
+val MY_THRESHOLD =
buildConfFromConfigFile[Int]("spark.sql.myFeature.threshold")
+ .checkValue(_ > 0, "Threshold must be positive")
+```
+
diff --git a/common/config/pom.xml b/common/config/pom.xml
new file mode 100644
index 000000000000..b5dba2fcf9d9
--- /dev/null
+++ b/common/config/pom.xml
@@ -0,0 +1,187 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-parent_2.13</artifactId>
+ <version>5.0.0-SNAPSHOT</version>
+ <relativePath>../../pom.xml</relativePath>
+ </parent>
+
+ <artifactId>spark-config_2.13</artifactId>
+ <packaging>jar</packaging>
+ <name>Spark Project Config</name>
+ <url>https://spark.apache.org/</url>
+ <properties>
+ <sbt.project.name>config</sbt.project.name>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>com.google.protobuf</groupId>
+ <artifactId>protobuf-java</artifactId>
+ <scope>compile</scope>
+ </dependency>
+
+ <!--
+ This spark-tags test-dep is needed even though it isn't used in this
module, otherwise testing-cmds that exclude
+ them will yield errors.
+ -->
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-tags_${scala.binary.version}</artifactId>
+ <type>test-jar</type>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter</artifactId>
+ <scope>test</scope>
+ </dependency>
+ </dependencies>
+
+ <build>
+
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
+
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <!--
+ Override the parent pom's shade configuration completely (rather than
+ merging) so this module bundles protobuf-java relocated into a shared
+ org.sparkproject.protobuf package. This keeps the unshaded
protobuf-java
+ off the classpath of every consumer of this low-level module (notably
+ spark-common-utils, and transitively spark-core and the assembly),
and
+ removes protobuf-java from this module's published
(dependency-reduced)
+ pom. The relocation package is intentionally general (not
config-specific)
+ so other low-level modules can share it in the future.
+ -->
+ <configuration combine.self="override">
+ <shadedArtifactAttached>false</shadedArtifactAttached>
+ <shadeTestJar>false</shadeTestJar>
+ <artifactSet>
+ <includes>
+ <include>com.google.protobuf:*</include>
+ </includes>
+ </artifactSet>
+ <relocations>
+ <relocation>
+ <pattern>com.google.protobuf</pattern>
+
<shadedPattern>${spark.shade.packageName}.protobuf</shadedPattern>
+ <includes>
+ <include>com.google.protobuf.**</include>
+ </includes>
+ </relocation>
+ </relocations>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>google/protobuf/**</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ </configuration>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+
+ <profiles>
+ <profile>
+ <id>default-protoc</id>
+ <activation>
+ <property>
+ <name>!skipDefaultProtoc</name>
+ </property>
+ </activation>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.github.os72</groupId>
+ <artifactId>protoc-jar-maven-plugin</artifactId>
+ <version>${protoc-jar-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ <configuration>
+
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}</protocArtifact>
+ <protocVersion>${protobuf.version}</protocVersion>
+ <inputDirectories>
+ <include>src/main/protobuf</include>
+ </inputDirectories>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ <profile>
+ <id>user-defined-protoc</id>
+ <properties>
+
<spark.protoc.executable.path>${env.SPARK_PROTOC_EXEC_PATH}</spark.protoc.executable.path>
+ </properties>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>com.github.os72</groupId>
+ <artifactId>protoc-jar-maven-plugin</artifactId>
+ <version>${protoc-jar-maven-plugin.version}</version>
+ <executions>
+ <execution>
+ <phase>generate-sources</phase>
+ <goals>
+ <goal>run</goal>
+ </goals>
+ <configuration>
+
<protocArtifact>com.google.protobuf:protoc:${protobuf.version}</protocArtifact>
+ <protocVersion>${protobuf.version}</protocVersion>
+
<protocCommand>${spark.protoc.executable.path}</protocCommand>
+ <inputDirectories>
+ <include>src/main/protobuf</include>
+ </inputDirectories>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ </profile>
+ </profiles>
+</project>
diff --git
a/common/config/src/main/java/org/apache/spark/config/ConfigRegistry.java
b/common/config/src/main/java/org/apache/spark/config/ConfigRegistry.java
new file mode 100644
index 000000000000..08080081762a
--- /dev/null
+++ b/common/config/src/main/java/org/apache/spark/config/ConfigRegistry.java
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.config;
+
+import java.io.BufferedReader;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+
+import com.google.protobuf.TextFormat;
+
+import org.apache.spark.config.protobuf.ConfigEntries;
+import org.apache.spark.config.protobuf.ConfigEntry;
+
+/**
+ * A registry for Spark configuration entries.
+ *
+ * <p>This class loads configuration entries from .textproto files on the
classpath
+ * and provides methods to query them by key.
+ *
+ * <p>For production use, use the static methods which access the default
singleton instance.
+ * For testing, create a new instance with custom config file paths using the
constructor.
+ *
+ * <p>Config files are organized into two directories:
+ * <ul>
+ * <li>cluster_configs/: Contains configs with CLUSTER scope</li>
+ * <li>session_configs/: Contains configs with SESSION scope</li>
+ * </ul>
+ */
+public class ConfigRegistry {
+
+ // Explicit list of config files to load for production (package-private for
testing)
+ static final String[] DEFAULT_CONFIG_FILES = {
+ "org/apache/spark/config/cluster_configs/sql.textproto",
+ "org/apache/spark/config/session_configs/sql.textproto"
+ };
+
+ // Lazy initialization holder class idiom for thread-safe lazy loading of
default instance
+ private static class DefaultInstanceHolder {
+ static final ConfigRegistry INSTANCE = new
ConfigRegistry(DEFAULT_CONFIG_FILES);
+ }
+
+ // Instance field: map of config key -> ConfigEntry
+ private final Map<String, ConfigEntry> configMap;
+
+ /**
+ * Create a ConfigRegistry with custom config file paths.
+ *
+ * @param configFiles the resource paths of .textproto files to load
+ */
+ public ConfigRegistry(String... configFiles) {
+ this.configMap = loadConfigs(configFiles);
+ }
+
+ /**
+ * Get a config entry by its key.
+ *
+ * @param key the config key (e.g., "spark.sql.optimizer.maxIterations")
+ * @return the ConfigEntry if found, null otherwise
+ */
+ public ConfigEntry get(String key) {
+ return configMap.get(key);
+ }
+
+ /**
+ * Check if a config key exists in the registry.
+ *
+ * @param key the config key
+ * @return true if the key exists, false otherwise
+ */
+ public boolean contains(String key) {
+ return configMap.containsKey(key);
+ }
+
+ /**
+ * Get all registered config keys.
+ *
+ * @return an unmodifiable set of all config keys
+ */
+ public Set<String> keys() {
+ return configMap.keySet();
+ }
+
+ /**
+ * Get all registered config entries.
+ *
+ * @return an unmodifiable collection of all config entries
+ */
+ public Collection<ConfigEntry> all() {
+ return configMap.values();
+ }
+
+ // ==========================================================================
+ // Static methods for production use (delegate to default singleton instance)
+ // ==========================================================================
+
+ /**
+ * Get the default singleton instance.
+ *
+ * @return the default ConfigRegistry instance
+ */
+ public static ConfigRegistry getInstance() {
+ return DefaultInstanceHolder.INSTANCE;
+ }
+
+ /**
+ * Get a config entry by its key from the default registry.
+ *
+ * @param key the config key (e.g., "spark.sql.optimizer.maxIterations")
+ * @return the ConfigEntry if found, null otherwise
+ */
+ public static ConfigEntry getConfig(String key) {
+ return getInstance().get(key);
+ }
+
+ /**
+ * Check if a config key exists in the default registry.
+ *
+ * @param key the config key
+ * @return true if the key exists, false otherwise
+ */
+ public static boolean containsConfig(String key) {
+ return getInstance().contains(key);
+ }
+
+ /**
+ * Get all registered config keys from the default registry.
+ *
+ * @return an unmodifiable set of all config keys
+ */
+ public static Set<String> allKeys() {
+ return getInstance().keys();
+ }
+
+ /**
+ * Get all registered config entries from the default registry.
+ *
+ * @return an unmodifiable collection of all config entries
+ */
+ public static Collection<ConfigEntry> allConfigs() {
+ return getInstance().all();
+ }
+
+ // ==========================================================================
+ // Private helper methods
+ // ==========================================================================
+
+ /**
+ * Load configs from .textproto files.
+ */
+ private static Map<String, ConfigEntry> loadConfigs(String[] configFiles) {
+ Map<String, ConfigEntry> result = new HashMap<>();
+ ClassLoader classLoader = ConfigRegistry.class.getClassLoader();
+
+ for (String file : configFiles) {
+ List<ConfigEntry> configs = loadConfigFile(classLoader, file);
+ for (ConfigEntry config : configs) {
+ result.put(config.getKey(), config);
+ }
+ }
+
+ return Collections.unmodifiableMap(result);
+ }
+
+ /**
+ * Load configs from a single .textproto file.
+ * Package-private for testing.
+ */
+ static List<ConfigEntry> loadConfigFile(
+ ClassLoader classLoader,
+ String resourcePath) {
+ InputStream inputStream = classLoader.getResourceAsStream(resourcePath);
+ if (inputStream == null) {
+ throw internalError("Config file not found on classpath: " +
resourcePath);
+ }
+
+ try (BufferedReader reader = new BufferedReader(
+ new InputStreamReader(inputStream, StandardCharsets.UTF_8))) {
+ ConfigEntries.Builder builder = ConfigEntries.newBuilder();
+ TextFormat.getParser().merge(reader, builder);
+ return builder.build().getConfigsList();
+ } catch (Exception e) {
+ throw internalError("Failed to load config file: " + resourcePath, e);
+ }
+ }
+
+ private static AssertionError internalError(String message) {
+ return new AssertionError(message);
+ }
+
+ private static AssertionError internalError(String message, Throwable cause)
{
+ return new AssertionError(message, cause);
+ }
+}
diff --git a/common/config/src/main/protobuf/buf.yaml
b/common/config/src/main/protobuf/buf.yaml
new file mode 100644
index 000000000000..3b3903fc97c9
--- /dev/null
+++ b/common/config/src/main/protobuf/buf.yaml
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+version: v1
+breaking:
+ use:
+ - FILE
+lint:
+ use:
+ - BASIC
+
diff --git
a/common/config/src/main/protobuf/org/apache/spark/config/config_schema.proto
b/common/config/src/main/protobuf/org/apache/spark/config/config_schema.proto
new file mode 100644
index 000000000000..c46dbdbb97d4
--- /dev/null
+++
b/common/config/src/main/protobuf/org/apache/spark/config/config_schema.proto
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = "proto3";
+package org.apache.spark.config;
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.config.protobuf";
+
+// The scope of the config
+enum Scope {
+ SCOPE_UNSPECIFIED = 0;
+ // The config is effective for the entire Spark cluster.
+ SCOPE_CLUSTER = 1;
+ // The config is effective for the current session.
+ SCOPE_SESSION = 2;
+}
+
+// The type of the config value
+enum ValueType {
+ VALUE_TYPE_UNSPECIFIED = 0;
+ VALUE_TYPE_BOOL = 1;
+ VALUE_TYPE_INT = 2;
+ VALUE_TYPE_LONG = 3;
+ VALUE_TYPE_DOUBLE = 4;
+ VALUE_TYPE_STRING = 5;
+}
+
+// The visibility of the config
+enum Visibility {
+ VISIBILITY_UNSPECIFIED = 0;
+ VISIBILITY_PUBLIC = 1;
+ VISIBILITY_INTERNAL = 2;
+}
+
+// The binding policy of the config for SQL views, UDFs, and procedures.
+enum BindingPolicy {
+ BINDING_POLICY_UNSPECIFIED = 0;
+ // The config value propagates from the active session to
views/UDFs/procedures.
+ BINDING_POLICY_SESSION = 1;
+ // The config uses the value saved at view/UDF/procedure creation time,
+ // or the Spark default if none was saved.
+ BINDING_POLICY_PERSISTED = 2;
+ // The config does not apply to views/UDFs/procedures. At runtime, behaves
like SESSION.
+ BINDING_POLICY_NOT_APPLICABLE = 3;
+}
+
+// Whether the config can be changed after system initialization.
+enum Mutability {
+ MUTABILITY_UNSPECIFIED = 0;
+ // The config is fixed once the Spark application starts.
+ MUTABILITY_STATIC = 1;
+ // The config can be changed at runtime (e.g., via SET).
+ MUTABILITY_DYNAMIC = 2;
+}
+
+// A single config entry
+message ConfigEntry {
+ // [Required] The config key.
+ optional string key = 1;
+
+ // [Required] The type of the config value
+ optional ValueType value_type = 2;
+
+ // The default value as a string representation
+ optional string default_value = 3;
+
+ // The default value to use during testing, if different from default_value.
+ // Requires default_value to be set, since it only overrides the production
default in tests.
+ optional string test_default = 4;
+
+ // [Required] The scope of the config (CLUSTER or SESSION).
+ optional Scope scope = 5;
+
+ // [Required] Whether the config can be changed after system initialization.
+ optional Mutability mutability = 6;
+
+ // [Required] The visibility of the config (PUBLIC or INTERNAL).
+ optional Visibility visibility = 7;
+
+ // [Required] The binding policy for SQL views, UDFs, and procedures.
+ optional BindingPolicy binding_policy = 8;
+
+ // [Required] Documentation for the config.
+ optional string doc = 9;
+
+ // The Spark version when this config was introduced.
+ optional string version = 10;
+}
+
+// A collection of config entries
+message ConfigEntries {
+ repeated ConfigEntry configs = 1;
+}
diff --git
a/common/config/src/main/resources/org/apache/spark/config/cluster_configs/sql.textproto
b/common/config/src/main/resources/org/apache/spark/config/cluster_configs/sql.textproto
new file mode 100644
index 000000000000..f599ed52d794
--- /dev/null
+++
b/common/config/src/main/resources/org/apache/spark/config/cluster_configs/sql.textproto
@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# SQL configuration entries (CLUSTER scope)
+
+configs {
+ key: "spark.sql.ui.retainedExecutions"
+ value_type: VALUE_TYPE_INT
+ default_value: "1000"
+ scope: SCOPE_CLUSTER
+ mutability: MUTABILITY_STATIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "Number of executions to retain in the Spark UI."
+ version: "1.5.0"
+}
diff --git
a/common/config/src/main/resources/org/apache/spark/config/session_configs/sql.textproto
b/common/config/src/main/resources/org/apache/spark/config/session_configs/sql.textproto
new file mode 100644
index 000000000000..06eaf7ee0277
--- /dev/null
+++
b/common/config/src/main/resources/org/apache/spark/config/session_configs/sql.textproto
@@ -0,0 +1,54 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# SQL configuration entries (SESSION scope)
+
+configs {
+ key: "spark.sql.optimizer.datasourceV2ExprFolding"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_INTERNAL
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "When this config is set to true, do safe constant folding for the "
+ "expressions before translation and pushdown."
+ version: "4.1.0"
+}
+
+configs {
+ key: "spark.sql.optimizer.maxIterations"
+ value_type: VALUE_TYPE_INT
+ default_value: "100"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_INTERNAL
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "The max number of iterations the optimizer runs."
+ version: "2.0.0"
+}
+
+configs {
+ key: "spark.sql.shuffledHashJoinFactor"
+ value_type: VALUE_TYPE_INT
+ default_value: "3"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "The shuffle hash join can be selected if the data size of small "
+ "side multiplied by this factor is still smaller than the large side."
+ version: "3.3.0"
+}
diff --git
a/common/config/src/test/java/org/apache/spark/config/ConfigRegistrySuite.java
b/common/config/src/test/java/org/apache/spark/config/ConfigRegistrySuite.java
new file mode 100644
index 000000000000..9ec05c1a8328
--- /dev/null
+++
b/common/config/src/test/java/org/apache/spark/config/ConfigRegistrySuite.java
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.config;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.junit.jupiter.api.Test;
+
+import org.apache.spark.config.protobuf.BindingPolicy;
+import org.apache.spark.config.protobuf.ConfigEntry;
+import org.apache.spark.config.protobuf.Mutability;
+import org.apache.spark.config.protobuf.Scope;
+import org.apache.spark.config.protobuf.ValueType;
+import org.apache.spark.config.protobuf.Visibility;
+
+import static org.junit.jupiter.api.Assertions.*;
+
+/**
+ * Tests for {@link ConfigRegistry}.
+ */
+public class ConfigRegistrySuite {
+
+ private static final String TEST_CONFIG_FILE =
+ "org/apache/spark/config/test_configs.textproto";
+
+ private static final ConfigRegistry registry = new
ConfigRegistry(TEST_CONFIG_FILE);
+
+ private static final ClassLoader CLASS_LOADER =
ConfigRegistrySuite.class.getClassLoader();
+
+ // ==========================================================================
+ // Validation helpers
+ // ==========================================================================
+
+ /**
+ * Validate configs in one file: required fields and alphabetical ordering.
+ * Convenience method that loads the file first.
+ *
+ * @param file the resource path of the .textproto file
+ */
+ private static void validateConfigsInOneFile(String file) {
+ List<ConfigEntry> configs = ConfigRegistry.loadConfigFile(CLASS_LOADER,
file);
+ validateConfigsInOneFile(configs, file);
+ }
+
+ /**
+ * Validate configs: required fields and alphabetical ordering.
+ *
+ * @param configs the list of config entries
+ * @param file the file path (for error messages)
+ */
+ private static void validateConfigsInOneFile(List<ConfigEntry> configs,
String file) {
+ String previousKey = null;
+ for (ConfigEntry config : configs) {
+ // Validate required fields
+ List<String> missingFields = new ArrayList<>();
+ if (config.getKey().isEmpty()) {
+ missingFields.add("key");
+ }
+ if (config.getValueType() == ValueType.VALUE_TYPE_UNSPECIFIED) {
+ missingFields.add("value_type");
+ }
+ if (config.getScope() == Scope.SCOPE_UNSPECIFIED) {
+ missingFields.add("scope");
+ }
+ if (config.getVisibility() == Visibility.VISIBILITY_UNSPECIFIED) {
+ missingFields.add("visibility");
+ }
+ if (config.getBindingPolicy() ==
BindingPolicy.BINDING_POLICY_UNSPECIFIED) {
+ missingFields.add("binding_policy");
+ }
+ if (config.getMutability() == Mutability.MUTABILITY_UNSPECIFIED) {
+ missingFields.add("mutability");
+ }
+ if (config.getDoc().isEmpty()) {
+ missingFields.add("doc");
+ }
+ if (!missingFields.isEmpty()) {
+ String keyInfo = config.getKey().isEmpty() ? "" : " for key '" +
config.getKey() + "'";
+ fail("Missing required fields " + missingFields + keyInfo + " in " +
file);
+ }
+
+ // TODO: Relax this constraint once the config framework fully supports
+ // cluster-scoped dynamic configs and session-scoped static configs.
+ if (config.getScope() == Scope.SCOPE_CLUSTER
+ && config.getMutability() != Mutability.MUTABILITY_STATIC) {
+ fail("CLUSTER scope config must be STATIC" +
+ " for key '" + config.getKey() + "' in " + file);
+ }
+ if (config.getScope() == Scope.SCOPE_SESSION
+ && config.getMutability() != Mutability.MUTABILITY_DYNAMIC) {
+ fail("SESSION scope config must be DYNAMIC" +
+ " for key '" + config.getKey() + "' in " + file);
+ }
+
+ // A test_default only overrides default_value in test environments, so
it requires a
+ // default_value to override. Otherwise the entry would have a default
under test but none
+ // in production, diverging its shape between the two.
+ if (config.hasTestDefault() && !config.hasDefaultValue()) {
+ fail("test_default requires default_value" +
+ " for key '" + config.getKey() + "' in " + file);
+ }
+
+ // Validate alphabetical ordering
+ String key = config.getKey();
+ if (previousKey != null && key.compareTo(previousKey) < 0) {
+ fail("Config keys must be ordered alphabetically in " + file +
+ ": '" + key + "' should come before '" + previousKey + "'");
+ }
+ previousKey = key;
+ }
+ }
+
+ /**
+ * Validate all config files: required fields, ordering, and no duplicate
keys.
+ * Each file is loaded only once.
+ *
+ * @param configFiles the list of config file paths to check
+ */
+ private static void validateAllConfigFiles(String[] configFiles) {
+ Map<String, String> keyToFile = new HashMap<>();
+ for (String file : configFiles) {
+ List<ConfigEntry> configs = ConfigRegistry.loadConfigFile(CLASS_LOADER,
file);
+ // Validate required fields and ordering
+ validateConfigsInOneFile(configs, file);
+ // Validate no duplicate keys across files
+ for (ConfigEntry config : configs) {
+ String key = config.getKey();
+ if (keyToFile.containsKey(key)) {
+ fail("Duplicate config key '" + key + "': " +
+ "first defined in " + keyToFile.get(key) + ", duplicated in " +
file);
+ }
+ keyToFile.put(key, file);
+ }
+ }
+ }
+
+ // ==========================================================================
+ // Basic functionality tests
+ // ==========================================================================
+
+ @Test
+ public void testLoadConfigs() {
+ assertEquals(6, registry.keys().size());
+ assertTrue(registry.contains("spark.test.bool.config"));
+ assertTrue(registry.contains("spark.test.int.config"));
+ assertTrue(registry.contains("spark.test.string.config"));
+ assertTrue(registry.contains("spark.test.long.doc.config"));
+ assertTrue(registry.contains("spark.test.optional.config"));
+ assertTrue(registry.contains("spark.test.testdefault.config"));
+ }
+
+ @Test
+ public void testBoolConfig() {
+ ConfigEntry config = registry.get("spark.test.bool.config");
+ assertNotNull(config);
+ assertEquals("spark.test.bool.config", config.getKey());
+ assertEquals(ValueType.VALUE_TYPE_BOOL, config.getValueType());
+ assertEquals("true", config.getDefaultValue());
+ assertEquals(Scope.SCOPE_SESSION, config.getScope());
+ assertEquals(Visibility.VISIBILITY_PUBLIC, config.getVisibility());
+ assertEquals("A test boolean config", config.getDoc());
+ assertEquals("4.0.0", config.getVersion());
+ assertEquals(BindingPolicy.BINDING_POLICY_SESSION,
config.getBindingPolicy());
+ }
+
+ @Test
+ public void testIntConfig() {
+ ConfigEntry config = registry.get("spark.test.int.config");
+ assertNotNull(config);
+ assertEquals("spark.test.int.config", config.getKey());
+ assertEquals(ValueType.VALUE_TYPE_INT, config.getValueType());
+ assertEquals("42", config.getDefaultValue());
+ assertEquals(Scope.SCOPE_CLUSTER, config.getScope());
+ assertEquals(Visibility.VISIBILITY_INTERNAL, config.getVisibility());
+ assertEquals("A test integer config", config.getDoc());
+ assertEquals("4.0.0", config.getVersion());
+ assertEquals(BindingPolicy.BINDING_POLICY_NOT_APPLICABLE,
config.getBindingPolicy());
+ }
+
+ @Test
+ public void testStringConfig() {
+ ConfigEntry config = registry.get("spark.test.string.config");
+ assertNotNull(config);
+ assertEquals("spark.test.string.config", config.getKey());
+ assertEquals(ValueType.VALUE_TYPE_STRING, config.getValueType());
+ assertEquals("default_value", config.getDefaultValue());
+ assertEquals(Scope.SCOPE_SESSION, config.getScope());
+ assertEquals(Visibility.VISIBILITY_PUBLIC, config.getVisibility());
+ assertEquals("A test string config", config.getDoc());
+ assertEquals("4.0.0", config.getVersion());
+ assertEquals(BindingPolicy.BINDING_POLICY_NOT_APPLICABLE,
config.getBindingPolicy());
+ }
+
+ @Test
+ public void testGetNonExistent() {
+ assertNull(registry.get("spark.nonexistent.config"));
+ assertFalse(registry.contains("spark.nonexistent.config"));
+ }
+
+ @Test
+ public void testAll() {
+ assertEquals(6, registry.all().size());
+ }
+
+ @Test
+ public void testOptionalConfig() {
+ // A config without a default_value: hasDefaultValue is false and the
field is empty.
+ ConfigEntry config = registry.get("spark.test.optional.config");
+ assertNotNull(config);
+ assertEquals("spark.test.optional.config", config.getKey());
+ assertEquals(ValueType.VALUE_TYPE_STRING, config.getValueType());
+ assertFalse(config.hasDefaultValue());
+ assertEquals("", config.getDefaultValue());
+ }
+
+ @Test
+ public void testTestDefaultConfig() {
+ // A config with a test_default distinct from default_value.
+ ConfigEntry config = registry.get("spark.test.testdefault.config");
+ assertNotNull(config);
+ assertEquals("spark.test.testdefault.config", config.getKey());
+ assertEquals("10", config.getDefaultValue());
+ assertTrue(config.hasTestDefault());
+ assertEquals("20", config.getTestDefault());
+ }
+
+ @Test
+ public void testMultiLineDoc() {
+ ConfigEntry config = registry.get("spark.test.long.doc.config");
+ assertNotNull(config);
+ assertEquals("spark.test.long.doc.config", config.getKey());
+ assertEquals(ValueType.VALUE_TYPE_STRING, config.getValueType());
+ assertEquals("test", config.getDefaultValue());
+ assertEquals(Scope.SCOPE_SESSION, config.getScope());
+ assertEquals(Visibility.VISIBILITY_PUBLIC, config.getVisibility());
+ // Verify multi-line string concatenation works
+ String expectedDoc = "This is a very long documentation string that spans
multiple lines. " +
+ "It demonstrates the prototext multi-line string concatenation
feature. " +
+ "Each quoted segment will be concatenated together into a single
string, " +
+ "which is useful for configs with lengthy descriptions.";
+ assertEquals(expectedDoc, config.getDoc());
+ assertEquals("4.0.0", config.getVersion());
+ }
+
+ // ==========================================================================
+ // Validation of production config files
+ // ==========================================================================
+
+ @Test
+ public void testProductionConfigsAreValid() {
+ validateAllConfigFiles(ConfigRegistry.DEFAULT_CONFIG_FILES);
+ }
+
+ // ==========================================================================
+ // Tests for validation logic using invalid config files
+ // ==========================================================================
+
+ @Test
+ public void testValidationMissingKey() {
+ String file = "org/apache/spark/config/invalid_missing_key.textproto";
+ Throwable error = assertThrows(Throwable.class,
+ () -> validateConfigsInOneFile(file));
+ assertTrue(error.getMessage().contains("Missing required fields"));
+ assertTrue(error.getMessage().contains("key"));
+ assertTrue(error.getMessage().contains(file));
+ }
+
+ @Test
+ public void testValidationMissingValueType() {
+ String file =
"org/apache/spark/config/invalid_missing_value_type.textproto";
+ Throwable error = assertThrows(Throwable.class,
+ () -> validateConfigsInOneFile(file));
+ assertTrue(error.getMessage().contains("Missing required fields"));
+ assertTrue(error.getMessage().contains("value_type"));
+ assertTrue(error.getMessage().contains("spark.test.missing.value.type"));
+ assertTrue(error.getMessage().contains(file));
+ }
+
+ @Test
+ public void testValidationMissingScope() {
+ String file = "org/apache/spark/config/invalid_missing_scope.textproto";
+ Throwable error = assertThrows(Throwable.class,
+ () -> validateConfigsInOneFile(file));
+ assertTrue(error.getMessage().contains("Missing required fields"));
+ assertTrue(error.getMessage().contains("scope"));
+ assertTrue(error.getMessage().contains("spark.test.missing.scope"));
+ assertTrue(error.getMessage().contains(file));
+ }
+
+ @Test
+ public void testValidationMissingVisibility() {
+ String file =
"org/apache/spark/config/invalid_missing_visibility.textproto";
+ Throwable error = assertThrows(Throwable.class,
+ () -> validateConfigsInOneFile(file));
+ assertTrue(error.getMessage().contains("Missing required fields"));
+ assertTrue(error.getMessage().contains("visibility"));
+ assertTrue(error.getMessage().contains("spark.test.missing.visibility"));
+ assertTrue(error.getMessage().contains(file));
+ }
+
+ @Test
+ public void testValidationMissingDoc() {
+ String file = "org/apache/spark/config/invalid_missing_doc.textproto";
+ Throwable error = assertThrows(Throwable.class,
+ () -> validateConfigsInOneFile(file));
+ assertTrue(error.getMessage().contains("Missing required fields"));
+ assertTrue(error.getMessage().contains("doc"));
+ assertTrue(error.getMessage().contains("spark.test.missing.doc"));
+ assertTrue(error.getMessage().contains(file));
+ }
+
+ @Test
+ public void testValidationMissingBindingPolicy() {
+ String file =
"org/apache/spark/config/invalid_missing_binding_policy.textproto";
+ Throwable error = assertThrows(Throwable.class,
+ () -> validateConfigsInOneFile(file));
+ assertTrue(error.getMessage().contains("Missing required fields"));
+ assertTrue(error.getMessage().contains("binding_policy"));
+
assertTrue(error.getMessage().contains("spark.test.missing.binding.policy"));
+ assertTrue(error.getMessage().contains(file));
+ }
+
+ @Test
+ public void testValidationMissingMutability() {
+ String file =
"org/apache/spark/config/invalid_missing_mutability.textproto";
+ Throwable error = assertThrows(Throwable.class,
+ () -> validateConfigsInOneFile(file));
+ assertTrue(error.getMessage().contains("Missing required fields"));
+ assertTrue(error.getMessage().contains("mutability"));
+ assertTrue(error.getMessage().contains("spark.test.missing.mutability"));
+ assertTrue(error.getMessage().contains(file));
+ }
+
+ @Test
+ public void testValidationTestDefaultWithoutDefault() {
+ String file =
"org/apache/spark/config/invalid_testdefault_without_default.textproto";
+ Throwable error = assertThrows(Throwable.class,
+ () -> validateConfigsInOneFile(file));
+ assertTrue(error.getMessage().contains("test_default requires
default_value"));
+
assertTrue(error.getMessage().contains("spark.test.testdefault.without.default"));
+ assertTrue(error.getMessage().contains(file));
+ }
+
+ @Test
+ public void testValidationUnorderedConfigs() {
+ String file = "org/apache/spark/config/invalid_unordered.textproto";
+ Throwable error = assertThrows(Throwable.class,
+ () -> validateConfigsInOneFile(file));
+ assertTrue(error.getMessage().contains("must be ordered alphabetically"));
+ assertTrue(error.getMessage().contains(file));
+ }
+
+ @Test
+ public void testValidationDuplicateKeys() {
+ String file1 = TEST_CONFIG_FILE;
+ String file2 = "org/apache/spark/config/duplicate_key_file2.textproto";
+ Throwable error = assertThrows(Throwable.class,
+ () -> validateAllConfigFiles(new String[]{file1, file2}));
+ assertTrue(error.getMessage().contains("Duplicate config key"));
+ assertTrue(error.getMessage().contains("spark.test.bool.config"));
+ assertTrue(error.getMessage().contains(file1));
+ assertTrue(error.getMessage().contains(file2));
+ }
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/duplicate_key_file2.textproto
b/common/config/src/test/resources/org/apache/spark/config/duplicate_key_file2.textproto
new file mode 100644
index 000000000000..5221f87af883
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/duplicate_key_file2.textproto
@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Second test file with a key that duplicates one in test_configs.textproto
+
+configs {
+ key: "spark.test.bool.config"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "false"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_SESSION
+ doc: "Duplicate of spark.test.bool.config"
+ version: "4.0.0"
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/invalid_missing_binding_policy.textproto
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_binding_policy.textproto
new file mode 100644
index 000000000000..3f99ec3facb6
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_binding_policy.textproto
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test file with missing binding_policy field
+
+configs {
+ key: "spark.test.missing.binding.policy"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ doc: "A config with missing binding_policy"
+ version: "4.0.0"
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/invalid_missing_doc.textproto
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_doc.textproto
new file mode 100644
index 000000000000..a0dcbfd6afb6
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_doc.textproto
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test file with missing doc field
+
+configs {
+ key: "spark.test.missing.doc"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ version: "4.0.0"
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/invalid_missing_key.textproto
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_key.textproto
new file mode 100644
index 000000000000..14f561da7c50
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_key.textproto
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test file with missing key field
+
+configs {
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A config with missing key"
+ version: "4.0.0"
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/invalid_missing_mutability.textproto
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_mutability.textproto
new file mode 100644
index 000000000000..e8b759941515
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_mutability.textproto
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test file with missing mutability field
+
+configs {
+ key: "spark.test.missing.mutability"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ scope: SCOPE_SESSION
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A config with missing mutability"
+ version: "4.0.0"
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/invalid_missing_scope.textproto
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_scope.textproto
new file mode 100644
index 000000000000..56f3661f4d03
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_scope.textproto
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test file with missing scope field
+
+configs {
+ key: "spark.test.missing.scope"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A config with missing scope"
+ version: "4.0.0"
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/invalid_missing_value_type.textproto
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_value_type.textproto
new file mode 100644
index 000000000000..7652ac680f66
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_value_type.textproto
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test file with missing value_type field
+
+configs {
+ key: "spark.test.missing.value.type"
+ default_value: "true"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A config with missing value_type"
+ version: "4.0.0"
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/invalid_missing_visibility.textproto
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_visibility.textproto
new file mode 100644
index 000000000000..e1a5a731e616
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/invalid_missing_visibility.textproto
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test file with missing visibility field
+
+configs {
+ key: "spark.test.missing.visibility"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A config with missing visibility"
+ version: "4.0.0"
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/invalid_testdefault_without_default.textproto
b/common/config/src/test/resources/org/apache/spark/config/invalid_testdefault_without_default.textproto
new file mode 100644
index 000000000000..a9abe3b60a13
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/invalid_testdefault_without_default.textproto
@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test file with test_default but no default_value
+
+configs {
+ key: "spark.test.testdefault.without.default"
+ value_type: VALUE_TYPE_INT
+ test_default: "20"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A test config with a test default but no default value"
+ version: "4.0.0"
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/invalid_unordered.textproto
b/common/config/src/test/resources/org/apache/spark/config/invalid_unordered.textproto
new file mode 100644
index 000000000000..99a6c983717d
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/invalid_unordered.textproto
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test file with unordered config keys (z comes before a)
+
+configs {
+ key: "spark.test.z.config"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "Z config"
+ version: "4.0.0"
+}
+
+configs {
+ key: "spark.test.a.config"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A config - should come before Z"
+ version: "4.0.0"
+}
diff --git
a/common/config/src/test/resources/org/apache/spark/config/test_configs.textproto
b/common/config/src/test/resources/org/apache/spark/config/test_configs.textproto
new file mode 100644
index 000000000000..3ce51f2d0b4c
--- /dev/null
+++
b/common/config/src/test/resources/org/apache/spark/config/test_configs.textproto
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Test configuration entries
+
+configs {
+ key: "spark.test.bool.config"
+ value_type: VALUE_TYPE_BOOL
+ default_value: "true"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_SESSION
+ doc: "A test boolean config"
+ version: "4.0.0"
+}
+
+configs {
+ key: "spark.test.int.config"
+ value_type: VALUE_TYPE_INT
+ default_value: "42"
+ scope: SCOPE_CLUSTER
+ mutability: MUTABILITY_STATIC
+ visibility: VISIBILITY_INTERNAL
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A test integer config"
+ version: "4.0.0"
+}
+
+configs {
+ key: "spark.test.long.doc.config"
+ value_type: VALUE_TYPE_STRING
+ default_value: "test"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_PERSISTED
+ doc: "This is a very long documentation string that spans multiple lines. "
+ "It demonstrates the prototext multi-line string concatenation feature.
"
+ "Each quoted segment will be concatenated together into a single
string, "
+ "which is useful for configs with lengthy descriptions."
+ version: "4.0.0"
+}
+
+configs {
+ key: "spark.test.optional.config"
+ value_type: VALUE_TYPE_STRING
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A test config without a default value"
+ version: "4.0.0"
+}
+
+configs {
+ key: "spark.test.string.config"
+ value_type: VALUE_TYPE_STRING
+ default_value: "default_value"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A test string config"
+ version: "4.0.0"
+}
+
+configs {
+ key: "spark.test.testdefault.config"
+ value_type: VALUE_TYPE_INT
+ default_value: "10"
+ test_default: "20"
+ scope: SCOPE_SESSION
+ mutability: MUTABILITY_DYNAMIC
+ visibility: VISIBILITY_PUBLIC
+ binding_policy: BINDING_POLICY_NOT_APPLICABLE
+ doc: "A test config with a distinct test default"
+ version: "4.0.0"
+}
diff --git a/common/utils/pom.xml b/common/utils/pom.xml
index 2bbfc37b7c66..09ea31e5ad56 100644
--- a/common/utils/pom.xml
+++ b/common/utils/pom.xml
@@ -39,6 +39,11 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-tags_${scala.binary.version}</artifactId>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-config_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-common-utils-java_${scala.binary.version}</artifactId>
diff --git
a/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala
b/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala
index 97fddf632ed2..abede2d57d51 100644
---
a/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala
+++
b/common/utils/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala
@@ -17,6 +17,11 @@
package org.apache.spark.internal.config
+import org.apache.spark.SparkException
+import org.apache.spark.config.ConfigRegistry
+import org.apache.spark.config.protobuf.{BindingPolicy, ConfigEntry =>
ProtoConfigEntry, ValueType, Visibility}
+import org.apache.spark.util.SparkEnvUtils
+
//
====================================================================================
// The guideline for naming configurations
//
====================================================================================
@@ -283,6 +288,107 @@ private[spark] class FallbackConfigEntry[T] (
}
}
+/**
+ * Marker trait for config entries backed by proto definitions.
+ */
+private[spark] sealed trait ProtoBackedBase
+
+/**
+ * A proto-backed config entry with a default value.
+ */
+private[spark] class ProtoBackedConfigEntry[T](
+ protoEntry: ProtoConfigEntry,
+ valueConverter: String => T,
+ stringConverter: T => String)
+ extends ConfigEntry[T](
+ key = protoEntry.getKey,
+ prependedKey = None,
+ prependSeparator = "",
+ alternatives = Nil,
+ valueConverter = valueConverter,
+ stringConverter = stringConverter,
+ doc = protoEntry.getDoc,
+ isPublic = protoEntry.getVisibility != Visibility.VISIBILITY_INTERNAL,
+ version = protoEntry.getVersion,
+ bindingPolicy = ProtoBackedConfigEntry.toBindingPolicy(protoEntry)
+ ) with ProtoBackedBase {
+
+ override def defaultValueString: String =
+ defaultValue.map(stringConverter).getOrElse(ConfigEntry.UNDEFINED)
+
+ // The default is derived from immutable proto fields (and the fixed testing
flag), so convert it
+ // once and cache rather than re-parsing the string on every
default-fallback read.
+ private lazy val cachedDefaultValue: Option[T] = {
+ val defaultStrOpt = if (SparkEnvUtils.isTesting &&
protoEntry.hasTestDefault) {
+ Some(protoEntry.getTestDefault)
+ } else if (protoEntry.hasDefaultValue) {
+ Some(protoEntry.getDefaultValue)
+ } else {
+ None
+ }
+ defaultStrOpt.map(valueConverter)
+ }
+
+ override def defaultValue: Option[T] = cachedDefaultValue
+
+ override def readFrom(reader: ConfigReader): T = {
+ readString(reader).map(valueConverter).getOrElse(defaultValue.get)
+ }
+
+ def checkValue(validator: T => Boolean, errorMsg: String):
ProtoBackedConfigEntry[T] = {
+ new ProtoBackedConfigEntry[T](
+ protoEntry,
+ str => {
+ val v = valueConverter(str)
+ if (!validator(v)) {
+ throw ConfigHelpers.configRequirementError(key, str, errorMsg)
+ }
+ v
+ },
+ stringConverter
+ )
+ }
+}
+
+/**
+ * A proto-backed config entry that does not have a default value.
+ */
+private[spark] class ProtoBackedOptionalConfigEntry[T](
+ protoEntry: ProtoConfigEntry,
+ rawValueConverter: String => T,
+ rawStringConverter: T => String)
+ extends ConfigEntry[Option[T]](
+ key = protoEntry.getKey,
+ prependedKey = None,
+ prependSeparator = "",
+ alternatives = Nil,
+ valueConverter = s => Some(rawValueConverter(s)),
+ stringConverter = v => v.map(rawStringConverter).orNull,
+ doc = protoEntry.getDoc,
+ isPublic = protoEntry.getVisibility != Visibility.VISIBILITY_INTERNAL,
+ version = protoEntry.getVersion,
+ bindingPolicy = ProtoBackedConfigEntry.toBindingPolicy(protoEntry)
+ ) with ProtoBackedBase {
+
+ override def defaultValueString: String = ConfigEntry.UNDEFINED
+
+ override def readFrom(reader: ConfigReader): Option[T] = {
+ readString(reader).map(rawValueConverter)
+ }
+}
+
+private[spark] object ProtoBackedConfigEntry {
+ def toBindingPolicy(
+ protoEntry: ProtoConfigEntry): Option[ConfigBindingPolicy.Value] = {
+ protoEntry.getBindingPolicy match {
+ case BindingPolicy.BINDING_POLICY_SESSION =>
Some(ConfigBindingPolicy.SESSION)
+ case BindingPolicy.BINDING_POLICY_PERSISTED =>
Some(ConfigBindingPolicy.PERSISTED)
+ case BindingPolicy.BINDING_POLICY_NOT_APPLICABLE =>
Some(ConfigBindingPolicy.NOT_APPLICABLE)
+ case _ => None
+ }
+ }
+}
+
private[spark] object ConfigEntry {
val UNDEFINED = "<undefined>"
@@ -290,13 +396,94 @@ private[spark] object ConfigEntry {
private[spark] val knownConfigs =
new java.util.concurrent.ConcurrentHashMap[String, ConfigEntry[_]]()
+ // Register all proto-backed configs at object initialization.
+ // These can be overwritten later by modules that need to add validation.
+ ConfigRegistry.allConfigs().forEach { protoEntry =>
+ createProtoBackedConfigEntry(protoEntry)
+ }
+
def registerEntry(entry: ConfigEntry[_]): Unit = {
val existing = knownConfigs.putIfAbsent(entry.key, entry)
- require(existing == null, s"Config entry ${entry.key} already registered!")
+ if (existing != null) {
+ // A key registered twice is normally a bug (typo or accidental
duplicate), so we fail loudly.
+ // The one exception is the enhancement pattern: a proto-backed config
is registered eagerly
+ // at object init, then a Scala entry built via
`buildConfFromConfigFile` (e.g. to add
+ // `checkValue`) may intentionally replace it. We allow the overwrite
only when the existing
+ // entry is proto-backed; this deliberately trades the
duplicate-detection net for that key,
+ // so the Scala side must still reference a key that is genuinely
defined in a .textproto
+ // file.
+ require(existing.isInstanceOf[ProtoBackedBase] &&
entry.isInstanceOf[ProtoBackedBase],
+ s"Config entry ${entry.key} already registered!")
+ knownConfigs.put(entry.key, entry)
+ }
}
def findEntry(key: String): ConfigEntry[_] = knownConfigs.get(key)
def listAllEntries(): java.util.Collection[ConfigEntry[_]] =
knownConfigs.values()
+ def findProtoBackedEntry(key: String): ProtoBackedConfigEntry[_] = {
+ knownConfigs.get(key) match {
+ case entry: ProtoBackedConfigEntry[_] => entry
+ case _ => null
+ }
+ }
+
+ def findProtoDefinedEntry(key: String): ConfigEntry[_] = {
+ if (ConfigRegistry.containsConfig(key)) knownConfigs.get(key) else null
+ }
+
+ def listAllProtoDefinedConfigs(): java.util.Collection[ConfigEntry[_]] = {
+ new java.util.AbstractCollection[ConfigEntry[_]]() {
+ override def iterator(): java.util.Iterator[ConfigEntry[_]] = {
+ val keys = ConfigRegistry.allKeys().iterator()
+ new java.util.Iterator[ConfigEntry[_]]() {
+ override def hasNext: Boolean = keys.hasNext
+ override def next(): ConfigEntry[_] = knownConfigs.get(keys.next())
+ }
+ }
+
+ override def size(): Int = ConfigRegistry.allKeys().size()
+ }
+ }
+
+ private def createProtoBackedConfigEntry(
+ protoEntry: ProtoConfigEntry): ConfigEntry[_] = {
+ // `test_default` only overrides `default_value` in test environments; a
config that sets
+ // `test_default` without `default_value` would be a has-default entry
under test but an
+ // optional entry in production, diverging the entry's shape (and
`buildConfFromConfigFile`'s
+ // success/failure) between the two. Reject it here so the
misconfiguration fails consistently.
+ require(!protoEntry.hasTestDefault || protoEntry.hasDefaultValue,
+ s"Config entry ${protoEntry.getKey} sets test_default without
default_value; " +
+ "a config with a test default must also declare a default value.")
+ // With the invariant above, `test_default` implies `default_value`, so
having a default is
+ // equivalent to having a `default_value` in every environment.
+ val hasDefault = protoEntry.hasDefaultValue
+ protoEntry.getValueType match {
+ case ValueType.VALUE_TYPE_BOOL =>
+ createTypedEntry[Boolean](protoEntry, hasDefault, _.toBoolean,
_.toString)
+ case ValueType.VALUE_TYPE_INT =>
+ createTypedEntry[Int](protoEntry, hasDefault, _.toInt, _.toString)
+ case ValueType.VALUE_TYPE_LONG =>
+ createTypedEntry[Long](protoEntry, hasDefault, _.toLong, _.toString)
+ case ValueType.VALUE_TYPE_DOUBLE =>
+ createTypedEntry[Double](protoEntry, hasDefault, _.toDouble,
_.toString)
+ case ValueType.VALUE_TYPE_STRING =>
+ createTypedEntry[String](protoEntry, hasDefault, identity[String],
identity[String])
+ case other =>
+ throw SparkException.internalError(s"Unsupported value type: $other")
+ }
+ }
+
+ private def createTypedEntry[T](
+ protoEntry: ProtoConfigEntry,
+ hasDefault: Boolean,
+ valueConverter: String => T,
+ stringConverter: T => String): ConfigEntry[_] = {
+ if (hasDefault) {
+ new ProtoBackedConfigEntry[T](protoEntry, valueConverter,
stringConverter)
+ } else {
+ new ProtoBackedOptionalConfigEntry[T](protoEntry, valueConverter,
stringConverter)
+ }
+ }
}
diff --git a/dev/mima b/dev/mima
index 17558be37fc8..39be02a1dd55 100755
--- a/dev/mima
+++ b/dev/mima
@@ -25,8 +25,16 @@ FWDIR="$(cd "`dirname "$0"`"/..; pwd)"
cd "$FWDIR"
SPARK_PROFILES=${1:-"-Pkubernetes -Pyarn -Pspark-ganglia-lgpl -Pkinesis-asl
-Phive-thriftserver -Phive"}
+
+# Capture sbt output to show errors if the command fails
+OLD_DEPS_OUTPUT="$(build/sbt -DcopyDependencies=false $SPARK_PROFILES "export
oldDeps/fullClasspath" 2>&1)" || {
+ echo "ERROR: Failed to export oldDeps classpath:"
+ echo "$OLD_DEPS_OUTPUT" | grep -E "^\[error\]"
+ exit 1
+}
+OLD_DEPS_CLASSPATH="$(echo "$OLD_DEPS_OUTPUT" | grep jar | tail -n1)"
+
TOOLS_CLASSPATH="$(build/sbt -DcopyDependencies=false "export
tools/fullClasspath" | grep jar | tail -n1)"
-OLD_DEPS_CLASSPATH="$(build/sbt -DcopyDependencies=false $SPARK_PROFILES
"export oldDeps/fullClasspath" | grep jar | tail -n1)"
rm -f .generated-mima*
diff --git a/pom.xml b/pom.xml
index 9f1665d342b1..50f8579ecc2c 100644
--- a/pom.xml
+++ b/pom.xml
@@ -85,6 +85,7 @@
<module>common/utils-java</module>
<module>common/variant</module>
<module>common/tags</module>
+ <module>common/config</module>
<module>sql/connect/shims</module>
<module>core</module>
<module>graphx</module>
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 64fee813c9c2..9e301a79f547 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -64,10 +64,10 @@ object BuildCommons {
val allProjects@Seq(
core, graphx, mllib, mllibLocal, repl, networkCommon, networkShuffle,
launcher, unsafe, tags, sketch, kvstore,
- commonUtils, commonUtilsJava, variant, pipelines, _*
+ commonUtils, commonUtilsJava, variant, pipelines, sparkConfig, _*
) = Seq(
"core", "graphx", "mllib", "mllib-local", "repl", "network-common",
"network-shuffle", "launcher", "unsafe",
- "tags", "sketch", "kvstore", "common-utils", "common-utils-java",
"variant", "pipelines"
+ "tags", "sketch", "kvstore", "common-utils", "common-utils-java",
"variant", "pipelines", "config"
).map(ProjectRef(buildLocation, _)) ++ sqlProjects ++ streamingProjects ++
connectProjects ++
udfWorkerProjects
@@ -373,6 +373,7 @@ object SparkBuild extends PomBuild {
"-groups",
"-skip-packages", Seq(
"org.apache.spark.api.python",
+ "org.apache.spark.config",
"org.apache.spark.deploy",
"org.apache.spark.kafka010",
"org.apache.spark.network",
@@ -415,7 +416,7 @@ object SparkBuild extends PomBuild {
Seq(
spark, hive, hiveThriftServer, repl, networkCommon, networkShuffle,
networkYarn,
unsafe, tags, tokenProviderKafka010, sqlKafka010, pipelines,
connectCommon, connect,
- connectJdbc, connectClient, variant, connectShims, profiler,
commonUtilsJava,
+ connectJdbc, connectClient, variant, connectShims, profiler,
commonUtilsJava, sparkConfig,
udfWorkerProto, udfWorkerCore, udfWorkerGrpc
).contains(x)
}
@@ -475,6 +476,9 @@ object SparkBuild extends PomBuild {
/* UDF Worker gRPC settings */
enable(UDFWorkerGrpc.settings)(udfWorkerGrpc)
+ /* Config module protobuf settings */
+ enable(SparkConfig.settings)(sparkConfig)
+
enable(DockerIntegrationTests.settings)(dockerIntegrationTests)
enable(KubernetesIntegrationTests.settings)(kubernetesIntegrationTests)
@@ -682,6 +686,31 @@ object Core {
}
}
+object SparkConfig {
+ import BuildCommons.protoVersion
+ lazy val settings = Seq(
+ // Setting version for the protobuf compiler.
+ PB.protocVersion := BuildCommons.protoVersion,
+ libraryDependencies ++= {
+ Seq(
+ "com.google.protobuf" % "protobuf-java" % protoVersion % "protobuf"
+ )
+ },
+ (Compile / PB.targets) := Seq(
+ PB.gens.java -> (Compile / sourceManaged).value
+ )
+ ) ++ {
+ val sparkProtocExecPath = sys.props.get("spark.protoc.executable.path")
+ if (sparkProtocExecPath.isDefined) {
+ Seq(
+ PB.protocExecutable := file(sparkProtocExecPath.get)
+ )
+ } else {
+ Seq.empty
+ }
+ }
+}
+
object SparkConnectCommon {
import BuildCommons.protoVersion
@@ -1638,6 +1667,7 @@ object Unidoc {
protected def ignoreUndocumentedPackages(packages: Seq[Seq[File]]):
Seq[Seq[File]] = {
packages
.map(_.filterNot(_.getName.contains("$")))
+ .map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/config")))
.map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/deploy")))
.map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/examples")))
.map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/internal")))
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala
index e40a6e6df88b..3fb20dcd6420 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala
@@ -95,7 +95,7 @@ class V2ExpressionBuilder(e: Expression, isPredicate: Boolean
= false) extends L
expr: Expression, isPredicate: Boolean = false): Option[V2Expression] =
expr match {
case literal: Literal => Some(translateLiteral(literal))
case _ if expr.contextIndependentFoldable
- && SQLConf.get.getConf(SQLConf.DATA_SOURCE_V2_EXPR_FOLDING) =>
+ &&
SQLConf.get.getConfByKeyStrict[Boolean]("spark.sql.optimizer.datasourceV2ExprFolding")
=>
// If the expression is context independent foldable, we can convert it
to a literal.
// This is useful for increasing the coverage of V2 expressions.
val constantExpr = ConstantFolding.constantFolding(expr)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 77c69494a15a..c6e0c9bc60bf 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -35,6 +35,8 @@ import org.apache.hadoop.mapreduce.OutputCommitter
import org.slf4j.event.Level
import org.apache.spark.{ErrorMessageFormat, SparkConf, SparkContext,
SparkException, TaskContext}
+import org.apache.spark.config.ConfigRegistry
+import org.apache.spark.config.protobuf.Mutability
import org.apache.spark.internal.Logging
import org.apache.spark.internal.config.{ConfigBindingPolicy, _}
import org.apache.spark.internal.io.FileCommitProtocol
@@ -75,6 +77,9 @@ object SQLConf {
private def register(entry: ConfigEntry[_]): Unit =
sqlConfEntriesUpdateLock.synchronized {
require(!sqlConfEntries.containsKey(entry.key),
s"Duplicate SQLConfigEntry. ${entry.key} has been registered")
+ require(!ConfigRegistry.containsConfig(entry.key),
+ s"${entry.key} is defined in a .textproto config file. " +
+ s"Use buildConfFromConfigFile instead of buildConf.")
val updatedMap = new java.util.HashMap[String,
ConfigEntry[_]](sqlConfEntries)
updatedMap.put(entry.key, entry)
sqlConfEntries = updatedMap
@@ -88,11 +93,28 @@ object SQLConf {
}
private[internal] def getConfigEntry(key: String): ConfigEntry[_] = {
- sqlConfEntries.get(key)
+
Option(sqlConfEntries.get(key)).getOrElse(ConfigEntry.findProtoDefinedEntry(key))
}
+ // TODO: once all configs are migrated to textproto, this can be replaced by
+ // ConfigEntry.listAllEntries() and callers can filter by config properties.
private[sql] def getConfigEntries(): util.Collection[ConfigEntry[_]] = {
- sqlConfEntries.values()
+ // Lazy concatenating view - no intermediate allocation
+ new util.AbstractCollection[ConfigEntry[_]]() {
+ private lazy val protoConfigs = ConfigEntry.listAllProtoDefinedConfigs()
+
+ override def iterator(): util.Iterator[ConfigEntry[_]] = {
+ val first = sqlConfEntries.values().iterator()
+ val second = protoConfigs.iterator()
+ new util.Iterator[ConfigEntry[_]]() {
+ override def hasNext: Boolean = first.hasNext || second.hasNext
+ override def next(): ConfigEntry[_] =
+ if (first.hasNext) first.next() else second.next()
+ }
+ }
+
+ override def size(): Int = sqlConfEntries.size() + protoConfigs.size()
+ }
}
private[internal] def containsConfigEntry(entry: ConfigEntry[_]): Boolean = {
@@ -100,7 +122,7 @@ object SQLConf {
}
private[sql] def containsConfigKey(key: String): Boolean = {
- sqlConfEntries.containsKey(key)
+ sqlConfEntries.containsKey(key) || ConfigRegistry.containsConfig(key)
}
def registerStaticConfigKey(key: String): Unit =
staticConfKeysUpdateLock.synchronized {
@@ -109,10 +131,27 @@ object SQLConf {
staticConfKeys = updated
}
- def isStaticConfigKey(key: String): Boolean = staticConfKeys.contains(key)
+ def isStaticConfigKey(key: String): Boolean = {
+ staticConfKeys.contains(key) ||
+ Option(ConfigRegistry.getConfig(key)).exists(_.getMutability ==
Mutability.MUTABILITY_STATIC)
+ }
def buildConf(key: String): ConfigBuilder =
ConfigBuilder(key).onCreate(register)
+ def buildConfFromConfigFile[T](key: String): ProtoBackedConfigEntry[T] = {
+
Option(ConfigEntry.findProtoBackedEntry(key)).map(_.asInstanceOf[ProtoBackedConfigEntry[T]])
+ .getOrElse {
+ if (ConfigRegistry.containsConfig(key)) {
+ // The key is defined in a .textproto file but has no default value,
so it is backed by
+ // a ProtoBackedOptionalConfigEntry. Such configs must be accessed
via getConfByKeyStrict.
+ throw SparkException.internalError(
+ s"Config entry $key has no default value; use getConfByKeyStrict
to read it.")
+ } else {
+ throw SparkException.internalError(s"Config entry $key not found in
ConfigRegistry")
+ }
+ }
+ }
+
def buildStaticConf(key: String): ConfigBuilder = {
ConfigBuilder(key).onCreate { entry =>
SQLConf.registerStaticConfigKey(entry.key)
@@ -129,7 +168,7 @@ object SQLConf {
private[sql] def mergeNonStaticSQLConfigs(
sqlConf: SQLConf,
configs: Map[String, String]): Unit = {
- for ((k, v) <- configs if !staticConfKeys.contains(k)) {
+ for ((k, v) <- configs if !isStaticConfigKey(k)) {
sqlConf.setConfString(k, v)
}
}
@@ -570,12 +609,8 @@ object SQLConf {
.stringConf
.createOptional
- val OPTIMIZER_MAX_ITERATIONS = buildConf("spark.sql.optimizer.maxIterations")
- .internal()
- .doc("The max number of iterations the optimizer runs.")
- .version("2.0.0")
- .intConf
- .createWithDefault(100)
+ val OPTIMIZER_MAX_ITERATIONS =
+ buildConfFromConfigFile[Int]("spark.sql.optimizer.maxIterations")
val OPTIMIZER_INSET_CONVERSION_THRESHOLD =
buildConf("spark.sql.optimizer.inSetConversionThreshold")
@@ -1001,13 +1036,9 @@ object SQLConf {
.bytesConf(ByteUnit.BYTE)
.createWithDefaultString("10MB")
- val SHUFFLE_HASH_JOIN_FACTOR = buildConf("spark.sql.shuffledHashJoinFactor")
- .doc("The shuffle hash join can be selected if the data size of small" +
- " side multiplied by this factor is still smaller than the large side.")
- .version("3.3.0")
- .intConf
- .checkValue(_ >= 1, "The shuffle hash join factor cannot be negative.")
- .createWithDefault(3)
+ val SHUFFLE_HASH_JOIN_FACTOR =
+ buildConfFromConfigFile[Int]("spark.sql.shuffledHashJoinFactor")
+ .checkValue(_ >= 1, "The shuffle hash join factor must be at least 1.")
val LIMIT_INITIAL_NUM_PARTITIONS =
buildConf("spark.sql.limit.initialNumPartitions")
.internal()
@@ -2217,15 +2248,6 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
- val DATA_SOURCE_V2_EXPR_FOLDING =
- buildConf("spark.sql.optimizer.datasourceV2ExprFolding")
- .internal()
- .version("4.1.0")
- .doc("When this config is set to true, do safe constant folding for the
" +
- "expressions before translation and pushdown.")
- .booleanConf
- .createWithDefault(true)
-
// This is used to set the default data source
val DEFAULT_DATA_SOURCE_NAME = buildConf("spark.sql.sources.default")
.doc("The default data source to use in input/output.")
@@ -9475,6 +9497,14 @@ class SQLConf extends Serializable with Logging with
SqlApiConf {
}
}
+ def getConfByKeyStrict[T](key: String): T = {
+ Option(ConfigEntry.findProtoDefinedEntry(key))
+ .map(_.asInstanceOf[ConfigEntry[T]].readFrom(reader))
+ .getOrElse {
+ throw SparkException.internalError(s"Config entry $key not found in
ConfigRegistry")
+ }
+ }
+
private var definedConfsLoaded = false
/**
* Init [[StaticSQLConf]] and [[org.apache.spark.sql.hive.HiveUtils]] so
that all the defined
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
index 658ab5f5312d..72e36250b068 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
@@ -33,7 +33,7 @@ import org.apache.spark.util.Utils
*/
object StaticSQLConf {
- import SQLConf.buildStaticConf
+ import SQLConf.{buildConfFromConfigFile, buildStaticConf}
val WAREHOUSE_PATH = buildStaticConf("spark.sql.warehouse.dir")
.doc("The default location for managed databases and tables.")
@@ -198,11 +198,7 @@ object StaticSQLConf {
.createOptional
val UI_RETAINED_EXECUTIONS =
- buildStaticConf("spark.sql.ui.retainedExecutions")
- .doc("Number of executions to retain in the Spark UI.")
- .version("1.5.0")
- .intConf
- .createWithDefault(1000)
+ buildConfFromConfigFile[Int]("spark.sql.ui.retainedExecutions")
val SHUFFLE_EXCHANGE_MAX_THREAD_THRESHOLD =
buildStaticConf("spark.sql.shuffleExchange.maxThreadThreshold")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/RuntimeConfigSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/RuntimeConfigSuite.scala
index 8c9e3eae4981..1ca88ce408c8 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/RuntimeConfigSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/RuntimeConfigSuite.scala
@@ -67,6 +67,8 @@ class RuntimeConfigSuite extends SparkFunSuite {
// SQL configs
assert(!conf.isModifiable(GLOBAL_TEMP_DATABASE.key))
assert(conf.isModifiable(CHECKPOINT_LOCATION.key))
+ // Proto-backed cluster config is not modifiable
+ assert(!conf.isModifiable("spark.sql.ui.retainedExecutions"))
// Core configs
assert(!conf.isModifiable(config.CPUS_PER_TASK.key))
assert(!conf.isModifiable("spark.executor.cores"))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StrategySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StrategySuite.scala
index 0301c1d0f5ba..4af5a3251534 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StrategySuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2StrategySuite.scala
@@ -1026,7 +1026,7 @@ class DataSourceV2StrategySuite extends
SharedSparkSession {
val expr = Abs(Literal(-5), failOnError = true)
checkV2Conversion(expr, LiteralValue(5, IntegerType))
- withSQLConf(SQLConf.DATA_SOURCE_V2_EXPR_FOLDING.key -> "false") {
+ withSQLConf("spark.sql.optimizer.datasourceV2ExprFolding" -> "false") {
// when spark.sql.optimizer.datasourceV2ExprFolding = false
// expression will be converted to V2 expressions, but not folded
checkV2Conversion(expr,
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala
index 94d860174cec..f29d91ba7bdf 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/internal/SQLConfSuite.scala
@@ -23,6 +23,7 @@ import org.apache.hadoop.fs.Path
import org.apache.logging.log4j.Level
import org.apache.spark.{SPARK_DOC_ROOT, SparkIllegalArgumentException,
SparkNoSuchElementException}
+import org.apache.spark.internal.config.{ConfigBuilder, ConfigEntry}
import org.apache.spark.network.util.ByteUnit
import org.apache.spark.sql.{AnalysisException, Row}
import org.apache.spark.sql.catalyst.parser.ParseException
@@ -183,6 +184,88 @@ class SQLConfSuite extends SharedSparkSession {
}
}
+ test("proto-backed config - OPTIMIZER_MAX_ITERATIONS") {
+ // Verify that the config is loaded from the prototext file with correct
properties
+ val config = SQLConf.OPTIMIZER_MAX_ITERATIONS
+ assert(config.key === "spark.sql.optimizer.maxIterations")
+ assert(config.defaultValueString === "100")
+ assert(config.doc === "The max number of iterations the optimizer runs.")
+ assert(config.version === "2.0.0")
+ // This config is internal (not public)
+ assert(!config.isPublic)
+
+ // Verify the config works correctly at runtime
+ sqlConf.clear()
+ assert(sqlConf.getConf(config) === 100)
+ sql(s"set ${config.key}=50")
+ assert(sqlConf.getConf(config) === 50)
+ sqlConf.clear()
+ }
+
+ test("proto-backed config cannot be replaced by an ordinary config entry") {
+ val key = SQLConf.OPTIMIZER_MAX_ITERATIONS.key
+ val original = ConfigEntry.findEntry(key)
+
+ intercept[IllegalArgumentException] {
+ ConfigBuilder(key).intConf.createWithDefault(1)
+ }
+ assert(ConfigEntry.findEntry(key) eq original)
+ }
+
+ test("proto-backed config with checkValue - SHUFFLE_HASH_JOIN_FACTOR") {
+ val config = SQLConf.SHUFFLE_HASH_JOIN_FACTOR
+ assert(config.key === "spark.sql.shuffledHashJoinFactor")
+ assert(config.defaultValueString === "3")
+ assert(config.version === "3.3.0")
+ assert(config.isPublic)
+
+ // Verify the config works correctly at runtime
+ sqlConf.clear()
+ assert(sqlConf.getConf(config) === 3)
+ sql(s"set ${config.key}=5")
+ assert(sqlConf.getConf(config) === 5)
+
+ // Verify the checkValue validation works - value must be >= 1
+ checkError(
+ exception = intercept[SparkIllegalArgumentException] {
+ spark.conf.set(config.key, 0)
+ },
+ condition = "INVALID_CONF_VALUE.REQUIREMENT",
+ parameters = Map(
+ "confName" -> config.key,
+ "confValue" -> "0",
+ "confRequirement" -> "The shuffle hash join factor must be at least
1."))
+
+ sqlConf.clear()
+ }
+
+ test("getConfByKeyStrict - read proto-backed config by key") {
+ sqlConf.clear()
+
+ // Test reading default value
+
assert(sqlConf.getConfByKeyStrict[Int]("spark.sql.optimizer.maxIterations") ===
100)
+ assert(sqlConf.getConfByKeyStrict[Boolean](
+ "spark.sql.optimizer.datasourceV2ExprFolding") === true)
+
+ // Test reading configured value
+ sql("set spark.sql.optimizer.maxIterations=50")
+
assert(sqlConf.getConfByKeyStrict[Int]("spark.sql.optimizer.maxIterations") ===
50)
+
+ // Test that non-existent key throws error
+ val e = intercept[Exception] {
+ sqlConf.getConfByKeyStrict[Int]("spark.nonexistent.config")
+ }
+ assert(e.getMessage.contains("not found in ConfigRegistry"))
+
+ // Test that type mismatch throws ClassCastException
+ // spark.sql.optimizer.maxIterations is an INT config, reading as Boolean
should fail
+ intercept[ClassCastException] {
+ sqlConf.getConfByKeyStrict[Boolean]("spark.sql.optimizer.maxIterations")
+ }
+
+ sqlConf.clear()
+ }
+
test("reset - user-defined conf") {
sqlConf.clear()
val userDefinedConf = "x.y.z.reset"
diff --git
a/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
b/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
index 75a905223e67..37598f184c02 100644
---
a/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
+++
b/sql/hive/src/test/resources/conf/binding-policy-exceptions/configs-without-binding-policy-exceptions
@@ -786,7 +786,6 @@ spark.sql.optimizer.avoidCollapseUDFWithExpensiveExpr
spark.sql.optimizer.avoidDoubleFilterEval
spark.sql.optimizer.canChangeCachedPlanOutputPartitioning
spark.sql.optimizer.collapseProjectAlwaysInline
-spark.sql.optimizer.datasourceV2ExprFolding
spark.sql.optimizer.datasourceV2JoinPushdown
spark.sql.optimizer.decorrelateExistsSubqueryLegacyIncorrectCountHandling.enabled
spark.sql.optimizer.decorrelateInnerQuery.enabled
@@ -811,7 +810,6 @@ spark.sql.optimizer.expression.nestedPruning.enabled
spark.sql.optimizer.expressionProjectionCandidateLimit
spark.sql.optimizer.inSetConversionThreshold
spark.sql.optimizer.inSetSwitchThreshold
-spark.sql.optimizer.maxIterations
spark.sql.optimizer.metadataOnly
spark.sql.optimizer.nestedPredicatePushdown.supportedFileSources
spark.sql.optimizer.nestedSchemaPruning.enabled
@@ -947,7 +945,6 @@ spark.sql.shuffle.partitions
spark.sql.shuffleDependency.fileCleanup.enabled
spark.sql.shuffleDependency.skipMigration.enabled
spark.sql.shuffleExchange.maxThreadThreshold
-spark.sql.shuffledHashJoinFactor
spark.sql.sort.enableRadixSort
spark.sql.sortMergeJoinExec.buffer.in.memory.threshold
spark.sql.sortMergeJoinExec.buffer.spill.size.threshold
@@ -1103,7 +1100,6 @@ spark.sql.transposeMaxValues
spark.sql.truncateTable.ignorePermissionAcl.enabled
spark.sql.tvf.allowMultipleTableArguments.enabled
spark.sql.ui.explainMode
-spark.sql.ui.retainedExecutions
spark.sql.unionOutputPartitioning
spark.sql.useCommonExprIdForAlias
spark.sql.variable.substitute
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]