jackylee-ch opened a new pull request, #12549:
URL: https://github.com/apache/gluten/pull/12549
## What changes were proposed in this pull request?
**[DNM] Do Not Merge — opened for design discussion and early feedback.**
Currently, conf keys passed to the native side are maintained in hard-coded
string lists that are decoupled from where the configs are defined:
- `nativeKeys` in `GlutenConfig` (a plain string set mixing common keys,
Spark keys, and Velox-specific keys),
- `extraNativeSessionConfKeys()` / `extraNativeBackendConfKeys()` in backend
settings,
- plus various prefix-matching rules inside `getNativeSessionConf` /
`getNativeBackendConf`.
Adding a new native conf requires defining the `ConfigEntry` in one place
and remembering to append its key string to one of these lists in another
place. Default values also end up duplicated between Scala and C++.
This PR introduces an **additive** registration mechanism (it does not
replace or modify any existing key list or prefix rule):
- `ConfigBuilder.passToNative(scope)`: marks a Gluten config entry to be
passed to the native side. Registration happens automatically when the entry is
created, so declaring a config and declaring its native passing are done at the
same place:
```scala
val MY_CONF = buildConf("spark.gluten.sql.columnar.foo")
.passToNative(NativeConfScope.SESSION)
.booleanConf
.createWithDefault(true)
```
- `NativeConfScope` (`SESSION` / `BACKEND` / `BOTH`): declares whether the
conf is passed on each native runtime creation (`getNativeSessionConf`), once
at native backend initialization (`getNativeBackendConf`), or both.
- `NativeConfRegistry` in `gluten-core`: collects the registrations. Raw
Spark/Hadoop keys that have no Gluten `ConfigEntry` (e.g. `spark.sql.*`,
`spark.hadoop.fs.s3a.*`) can be registered via `registerRaw`, optionally with a
`defaultToPass` value that is always sent to native even when not set by the
user. This allows each module (backend, connector, etc.) to register its own
native confs on demand, without touching common conf-passing code in
`gluten-substrait`.
- `getNativeSessionConf` / `getNativeBackendConf` additionally include the
registered confs. Existing hard-coded key lists, default-value lists and prefix
rules are untouched, so current behavior is fully preserved.
### Follow-ups (out of scope for this PR)
- Incrementally migrate keys in `nativeKeys` and `extraNative*ConfKeys()` to
`passToNative` at their `ConfigEntry` definitions.
- Migrate the always-send default-value lists in `getNative*Conf` to
`registerRaw(..., defaultToPass = ...)`.
- Explore generating C++ conf key constants from the registry to remove
JVM/native duplication.
## How was this patch tested?
New unit test `NativeConfRegistrySuite` in gluten-core covering:
- automatic registration via `ConfigBuilder.passToNative` for all creation
paths, with scope checks;
- duplicate-registration rejection;
- session/backend scoped selection from a conf map;
- `defaultToPass` semantics (always sent; user-set value wins).
Also ran existing `GlutenConfigUtilSuite` (gluten-core) and
`SparkConfigUtilSuite` (gluten-substrait) to verify no regression. When nothing
is registered, `getNativeSessionConf` / `getNativeBackendConf` behavior is
unchanged.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude (Cursor Agent)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
