jiangzho opened a new pull request, #713:
URL: https://github.com/apache/spark-kubernetes-operator/pull/713
### What changes were proposed in this pull request?
This PR makes runtime dynamic config override opt-in and enforces it
centrally when dynamic config is refreshed:
1. `ConfigOption`
- `enableDynamicOverride` now defaults to `false` — an option must
explicitly opt in via `.enableDynamicOverride(true)` to be overridable at
runtime.
- Each option registers itself (keyed by its config key) into a static
registry on construction, exposing `dynamicOverrideEnabledKeys()` as the
allow-list of overridable keys. `enableDynamicOverride` on each option remains
the single source of truth.
2. `SparkOperatorConfManager.refresh(Map)` now filters the incoming
ConfigMap data against `ConfigOption.dynamicOverrideEnabledKeys()`, applying
only overridable keys and logging any dropped (unknown or non-overridable) keys
at WARN. `getValue(String)` and `getAll()` therefore no longer surface override
values for keys that are never honored.
3. `SparkOperatorConf` options now set `enableDynamicOverride`explicitly;
this also fixes docs for `LOG_CONF` and `PERIODIC_GC_INTERVAL_SECONDS`, which
are only read at startup and are non-overridable.
The existing per-option check in `ConfigOption.resolveValue()` is kept as
defense-in-depth.
### Why are the changes needed?
Previously `refresh()` stored the entire ConfigMap into the override map,
and `enableDynamicOverride` defaulted to `true`. As a result, an override value
could be stored (and surfaced via `getValue`/`getAll`/the `LOG_CONF` dump) for
keys that are not meant to be hot-reloaded, and any new option was overridable
by default. Defaulting to opt-in and dropping non-allow-listed keys at refresh
time makes the override surface explicit, safer, and self-documenting, and
avoids silently ignored config updates.
### Does this PR introduce _any_ user-facing change?
Yes. Dynamic config updates for keys that are not explicitly marked
`enableDynamicOverride(true)` are now ignored (and logged at WARN) instead of
being stored. In particular `spark.logConf` and
`spark.kubernetes.operator.periodicGC.intervalSeconds` are no longer
dynamically overridable (both are only read at startup). The generated config
documentation's "Allow Hot Reloading" column reflects each option's flag.
### How was this patch tested?
Updated `ConfigOptionTest` and `SparkOperatorConfManagerTest`, and added
`SparkOperatorConfManagerTest#testRefreshDropsUnknownAndNonOverridableKeys`
covering the allow-list filtering. Existing
`SparkOperatorConfigMapReconcilerTest` continues to exercise the dynamic-config
refresh path with an overridable key.
### Was this patch authored or co-authored using generative AI tooling?
Co-Authored-By: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
