jackylee-ch opened a new issue, #12694:
URL: https://github.com/apache/gluten/issues/12694
### Description
Which configurations get delivered from JVM to native side is currently
decided by hard-coded
string lists inside `GlutenConfig.getNativeSessionConf` /
`getNativeBackendConf`:
- a `nativeKeys` set of 40+ raw key strings;
- two "configs having default values" `Seq`s that restate each conf's key
**and** its default;
- per-key special cases inlined into the selection methods (byte-unit
conversion for
`spark.shuffle.file.buffer`, upper-casing for
`spark.sql.legacy.timeParserPolicy`, …);
- `BackendSettingsApi.extraNativeSessionConfKeys` /
`extraNativeBackendConfKeys` as the escape
hatch for backends.
#### Why it matters
**The declaration lives far from the definition.** Adding a native conf
means editing a central
list in `gluten-substrait`, not the conf's own definition. Nothing links the
two, so the usual
failure mode is a conf that looks wired up but never reaches native — or the
reverse, a key that
keeps being delivered long after its native reader was removed.
`spark.gluten.velox.fs.s3a.retry.mode`
is an instance of the latter: it has had no native reader since #8123 moved
the S3 config path to
Velox's `S3Config`, yet it is still on the list.
**Defaults are duplicated and can drift.** The default-value `Seq`s repeat
what the `ConfigEntry`
already declares. Two copies of the same default, updated by hand.
**Backend-specific keys live in common code.** `gluten-substrait` enumerates
Velox S3 keys, so a
ClickHouse-only deployment carries Velox key names, and vice versa. There is
no module boundary.
**Delivery scope is invisible at the definition.** A conf is delivered on
the backend channel, the
runtime channel, or both purely by virtue of which list it appears in.
Reading the conf's
definition tells you nothing about it, and a mismatch between the declared
mutability of a conf and
where native actually consumes it is impossible to notice. Surveying the
current lists turned up
several: `velox.cudf.enableTableScan` is declared static yet read per query;
`memoryOverhead.size.in.bytes` is declared modifiable yet consumed only at
backend init.
**Third parties cannot participate.** An out-of-tree backend or component
has no way to declare its
own native confs. `extraNativeBackendConfKeys` is the only hook, it takes a
bare `Set[String]`
(no per-channel scope, no default, no normalization), and no backend in the
tree has ever
overridden it.
#### Proposal
Move the declaration to each conf's definition, and let the conf's own
properties decide how it is
delivered.
1. **Declare at the definition site.** A marker on the config builder, e.g.
`passToNative()`,
states that the conf reaches native. Related markers cover the two
remaining behaviors the old
per-key special cases provided: delivering the conf's default when the
user did not set it (for
keys native relies on being present), and normalizing a value before
delivery.
2. **Derive the channel from mutability rather than stating it.** A conf
that is modifiable at any
time should be delivered wherever native might read it — both at backend
initialization and on
each native runtime creation. A conf that is set at backend init and
immutable afterwards only
needs delivering once, at backend init. This makes the channel a
consequence of a property the
declaration already states, so there is nothing extra to get wrong, and
it forces the
mutability/consumption mismatches above to be resolved rather than
papered over.
3. **Support Spark- and Hadoop-owned keys.** Keys such as
`spark.sql.orc.compression.codec` or
`spark.hadoop.input.read.timeout` must **not** get a Gluten `ConfigEntry`
— their owner already
registered them, and registering again conflicts with `SQLConf`. They
need a declaration form
that states only the native delivery.
4. **Express "overrides a Spark conf" declaratively.** Several Gluten confs
exist only to depart
from a Spark default (`spark.gluten.sql.columnar.shuffle.codec` over
`spark.io.compression.codec`). Today the relationship is hand-written at
each read site. A
fallback declaration should carry it, including whether a value came from
the Gluten key or was
inherited — callers that validate the two differently need that
distinction.
5. **Give components an entry point.** A conf object is a Scala `object`, so
declaring one is not
enough: its declarations only take effect once something touches it.
Components need a hook that
Gluten invokes early enough for the backend channel — i.e. before any
backend's driver/executor
startup, where native backend init happens. This is what makes the
mechanism usable from
out-of-tree code, and it also makes declarations naturally modular: a
backend's declarations only
exist when that backend is loaded, so Velox keys never leak into a
ClickHouse deployment.
Prefix-based rules (`spark.gluten.sql.columnar.backend.<backend>`,
`spark.hadoop.fs.s3a.`, …) are
out of scope — they cover open-ended key families and are pattern rules
rather than enumerable
declarations.
#### Acceptance criteria
- No hard-coded native conf key list, default-value list, or per-key special
case remains in the
selection methods.
- The delivered key/value set is unchanged for every existing conf, except
where a mismatch is
deliberately corrected — each such difference stated explicitly.
- A conf's native delivery is readable from its definition alone.
- An out-of-tree backend or component can declare native confs without
modifying Gluten common
code, verified from outside the `org.apache.gluten` package.
- The declared mutability of every native-passed conf matches where native
consumes it.
This description was written with the assistance of generative AI tooling
(Claude Code).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
