zhangshenghang opened a new issue, #13226:
URL: https://github.com/apache/seatunnel/issues/13226
## Motivation
Today the `env` block of a SeaTunnel job config is consumed only at
job/engine level (`job.mode`, `parallelism`, checkpoint settings, etc.). Plugin
factories created through `TableFactoryContext` cannot see it, and SQL UDFs
(`ZetaUDF` / `CalciteUdf`) have no way to read user-defined values from `env`
at all.
Users need to define job-scoped custom parameters once, for example:
```hocon
env {
job.mode = "BATCH"
custom_parameters {
udf_salt = "abc123"
region = "cn-north"
}
}
```
and read them from:
- custom Source / Sink / Transform factories,
- `ZetaUDF` implementations used by the `Sql` transform,
- `CalciteUdf` implementations used by the `Calcite` transform.
`EnvCommonOptions.CUSTOM_PARAMETERS` already exists and is already allowed
by `EnvOptionRule`, but it is not exposed anywhere and is undocumented.
## Proposal
### 1. Expose env on the factory context (seatunnel-api)
- Add `ReadonlyConfig getEnvOptions()` to `TableFactoryContext`.
- Add env-aware constructor overloads to:
- `TableSourceFactoryContext(options, classLoader, metadataConfig,
envOptions)`
- `TableSinkFactoryContext(catalogTable, options, classLoader, envOptions)`
- `MultiTableFactoryContext(options, classLoader, sinks, envOptions)`
- `TableTransformFactoryContext(catalogTables, options, classLoader,
envOptions)`
- Add `TableSinkFactoryContext.replacePlaceholderAndCreate(...,
envOptions)`; env must stay outside placeholder rewriting.
- Add trailing `envOptions` overloads to the five `FactoryUtil` entry points:
`createAndPrepareSource`, `restoreAndPrepareSource`,
`createAndPrepareSink`, `createMultiTableSink`,
`createAndPrepareMultiTableTransform`.
- All existing constructors/methods stay and delegate with an empty env, so
existing connectors, tests and Flink/Spark starters are unaffected.
### 2. Wire the Zeta execution path
`MultipleTableJobConfigParser` already has the parsed `envOptions` field;
pass it to the source / restore-source / transform / multi-sink / sink factory
calls. CLI and REST submission both go through this parser.
Also pass env through `--dry-run=connect` (`DryRunConnectValidator`) for
source, transform **and sink** validation so that dry-run semantics match
runtime.
### 3. Expose env to SQL UDFs (seatunnel-transforms-v2)
- `Sql` transform:
- `SQLTransformFactory` reads `context.getEnvOptions()` and passes it to
`SQLMultiCatalogFlatMapTransform` -> `SQLTransform`.
- `SQLTransform` keeps env as a serializable field (transient SQL engine
is rebuilt on the worker with the same env).
- `SQLEngine.init(...)` gets a default 5-arg overload; `ZetaSQLEngine`
stores env and calls `udf.open(envOptions)`.
- `ZetaUDF` gets `default void open(ReadonlyConfig envOptions) throws
Exception { open(); }`.
- `Calcite` transform:
- Same propagation through `CalciteTransformFactory` ->
`CalciteMultiCatalogTransform` -> `CalciteTransform` -> `CalciteSQLEngine`.
- `BuiltinFunctions.discoverAndRegister` calls `udf.open(envOptions)`.
- `CalciteUdf` gets `default void open(ReadonlyConfig envOptions) {
open(); }`.
- `CalciteUdfContext.enter(tableId, rowKind, envOptions)` and
`getEnvOptions()` expose env to static `eval` methods, because Calcite UDFs are
invoked statically.
Deep-copied env must be injected into the multi-catalog wrapper **before**
inner per-table transforms are built: `AbstractMultiCatalogTransform` builds
them in its own constructor, so env must live on the base class, not in
subclass fields (subclass fields are still `null` at that point).
### 4. Documentation
Document `env.custom_parameters` as the recommended namespace for plugin/UDF
parameters, with examples for:
- `TableFactoryContext#getEnvOptions()` in a connector factory,
- `ZetaUDF#open(ReadonlyConfig)` / `evaluate`,
- `CalciteUdf` via `CalciteUdfContext.current().getEnvOptions()`.
Also document the limitations below.
## Design constraints
- **Do not merge env into plugin options.** Env must remain a separate
`getEnvOptions()` view. Merging would break unknown-key validation
(`ConfigValidator.validateUnknownKeys`), sink table-placeholder rewriting, and
connector `OptionRule` validation.
- **Backward compatibility is a hard requirement.** No existing constructor,
method signature, default value or config option may change.
- Env is carried as `ReadonlyConfig` (already `Serializable`) so it follows
transform serialization to workers.
- `ZetaSQLEngine`'s existing 4-arg `init` must keep working and default to
an empty env, because existing tests and callers invoke it directly.
## Non-goals (possible follow-ups)
- Wiring Flink/Spark starters (their contexts keep receiving an empty env;
behavior unchanged).
- Extending `ConfigShadeUtils` decryption to the `env` block (env values are
**not** decrypted today, so secrets should not be placed there).
- Runtime row-level contexts (`SourceReader.Context`, writer context, etc.)
exposing env.
## Known limitations to document
- `ZetaUDF.resultType()` is invoked during schema inference before
`open(env)`, so it cannot read env.
- A Calcite UDF's static `eval` may also run during planning/constant
folding, where `CalciteUdfContext.current()` is `null`; UDF code must
null-check.
- Calcite UDFs should not cache job-specific env in static fields (multiple
jobs can share a JVM); use `CalciteUdfContext.current()` inside `eval`.
- Adding a non-transient field to `SQLTransform` / `CalciteTransform`
changes their implicit `serialVersionUID`, which may affect restore from
checkpoints/savepoints written by older builds; this should be handled
explicitly (declare a stable UID or document the incompatibility).
- `env` is not decrypted by `ConfigShadeUtils` (it only processes
`source`/`sink`/`transform`), so do not put encrypted values in `env`.
## Acceptance criteria
1. Given a job with `env.custom_parameters.my_key = "my_value"`, a custom
`ZetaUDF` receives that value through `open(ReadonlyConfig)` when the job runs
on Zeta (local and cluster), including REST submission.
2. A `CalciteUdf` can read the same value via
`CalciteUdfContext.current().getEnvOptions()` inside `eval`.
3. Source / Sink / Transform factories can read `context.getEnvOptions()` on
the Zeta path.
4. `--check-config` / `--dry-run=connect` do not report unknown option keys
because of env, and dry-run passes env consistently to source, transform and
sink validation.
5. Flink/Spark execution and all existing connectors behave exactly as
before (empty env).
6. New unit tests cover: context getters + legacy empty default;
`FactoryUtil` overloads; `Sql` UDF env end-to-end; `Calcite` UDF env
end-to-end; parser-level env propagation.
7. Documentation updated in `docs/en` and `docs/zh`.
## Additional context
This capability depends on `EnvCommonOptions.CUSTOM_PARAMETERS` (already
registered in `EnvOptionRule`), so no new config option is required. The change
is additive API only; no existing signature is removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]