zhangshenghang commented on issue #13226:
URL: https://github.com/apache/seatunnel/issues/13226#issuecomment-6797291462
Thanks for the review. Agreed on the staged delivery and on keeping `env`
out of the credential path. I have documented the compatibility contract below
and will keep stage 1 and stage 2 reviewable separately.
## Stage 1 — factory env view + Zeta wiring (PR #13233)
Implemented in the first PR:
- Immutable `ReadonlyConfig getEnvOptions()` on `TableFactoryContext`; empty
env is the default and is never merged into plugin options.
- Env-aware constructors for `TableSourceFactoryContext`,
`TableSinkFactoryContext`, `MultiTableFactoryContext` and
`TableTransformFactoryContext`.
- Env-aware `FactoryUtil` overloads for `createAndPrepareSource`,
`restoreAndPrepareSource`, `createAndPrepareSink`, `createMultiTableSink` and
`createAndPrepareMultiTableTransform`.
- `MultipleTableJobConfigParser` passes env through the Zeta source,
restore-source, transform, multi-sink and sink paths (CLI and REST share this
parser).
- `DryRunConnectValidator` passes env to source, transform and sink dry-run
contexts.
- Flink/Spark behavior is unchanged (empty env).
- Focused API tests, a Zeta-path parser test and EN/ZH documentation.
`env.custom_parameters` is explicitly **not** a secret store:
`ConfigShadeUtils` only decrypts `source`/`sink`/`transform`, and the docs
state this.
## Constructor / overload compatibility table
All rows are additive: no existing signature is removed or changed.
| API | Existing signature (preserved) | New overload |
| --- | --- | --- |
| `TableFactoryContext` | `(ReadonlyConfig, ClassLoader)` |
`(ReadonlyConfig, ClassLoader, ReadonlyConfig envOptions)` |
| `TableSourceFactoryContext` | `(options, classLoader)`, `(options,
classLoader, metadataConfig)` | `(options, classLoader, metadataConfig,
envOptions)` |
| `TableSinkFactoryContext` | `(catalogTable, options, classLoader)` |
`(catalogTable, options, classLoader, envOptions)` |
| `TableSinkFactoryContext.replacePlaceholderAndCreate` | `(catalogTable,
options, classLoader, excludeKeys)` | `(catalogTable, options, classLoader,
excludeKeys, envOptions)` |
| `MultiTableFactoryContext` | `(options, classLoader, sinks)` | `(options,
classLoader, sinks, envOptions)` |
| `TableTransformFactoryContext` | `(catalogTables, options, classLoader)` |
`(catalogTables, options, classLoader, envOptions)` |
| `FactoryUtil.createAndPrepareSource` | 6-arg form | 7-arg form with
trailing `envOptions` |
| `FactoryUtil.restoreAndPrepareSource` | 7-arg form | 8-arg form with
trailing `envOptions` |
| `FactoryUtil.createAndPrepareSink` | 6-arg form | 7-arg form with trailing
`envOptions` |
| `FactoryUtil.createMultiTableSink` | 3-arg form | 4-arg form with trailing
`envOptions` |
| `FactoryUtil.createAndPrepareMultiTableTransform` | 4-arg form | 5-arg
form with trailing `envOptions` |
| `SQLEngine.init` | 4-arg abstract method | 5-arg default method delegating
to the 4-arg form |
| `ZetaUDF.open` | `default void open() throws Exception` | `default void
open(ReadonlyConfig) throws Exception` delegating to `open()` |
| `CalciteUdf.open` | `default void open()` | `default void
open(ReadonlyConfig)` delegating to `open()` |
| `CalciteUdfContext.enter` | `(tableId, rowKind)` | `(tableId, rowKind,
envOptions)` |
Old constructors and methods delegate with an empty environment. Adding
constructors, overloads and default interface methods is binary compatible;
existing third-party factories and UDF implementations keep their current
behavior.
## Serialization compatibility treatment (stage 2)
- `ReadonlyConfig` is already `Serializable` (`serialVersionUID = 1L`) and
env values from HOCON rendering are Jackson map/list/scalar types.
- `SQLTransform` and `CalciteTransform` carry env as a non-transient field;
their SQL engines are `transient` and are rebuilt with the same env on the
worker.
- A legacy stream does not contain the new `envOptions` field.
Deserialization leaves it `null`; `open()` treats `null` as an empty env, and
`AbstractMultiCatalogTransform#getEnvOptions()` is null-safe. Legacy transforms
therefore keep the previous (empty-env) behavior.
- To keep old streams readable, the pre-change implicit UIDs were computed
from `dev` with `serialver` and pinned explicitly:
- `SQLTransform`: `5502594799389514109L`
- `CalciteTransform`: `426108317364037266L`
- `AbstractMultiCatalogTransform` already declares an explicit UID and is
unaffected.
- New streams remain readable by older builds (unknown fields are ignored).
## Stage 2 — SQL / Calcite UDF integration (follow-up PR)
Planned/covered tests:
- Local submission (Zeta container E2E) for both `Sql` and `Calcite`.
- REST submission through the in-process Zeta cluster, asserting the job
reaches `FINISHED` with an Assert sink (the job fails if env does not reach the
UDF).
- Job isolation when UDF classes share a JVM: two transforms/engines with
different env values, executed interleaved; Zeta keeps env per UDF instance,
Calcite reads it from the per-row context.
- `CalciteUdfContext` cleanup: the thread-local scope is removed after
`execute`, so `current()` returns `null` and no job-specific env remains.
- Restore/serialization: round-trip of the parsed transform plus the
UID/legacy-stream treatment above.
I will narrow PR #13233 to stage 1 and keep the UDF work on a stacked branch
for the follow-up PR once the factory API contract is settled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]