zhangshenghang opened a new pull request, #13233:
URL: https://github.com/apache/seatunnel/pull/13233

   ## Contribution acknowledgement
   
   Yes — I am glad to contribute this feature to **Apache SeaTunnel**.
   
   ## Summary
   
   This PR exposes the job-level `env` block to plugin factories and SQL UDFs, 
so users can define job-scoped values once (for example under 
`env.custom_parameters`) and read them from:
   
   - Source / Sink / Transform factories through 
`TableFactoryContext#getEnvOptions()`.
   - `ZetaUDF#open(ReadonlyConfig)` in the `Sql` transform.
   - `CalciteUdfContext#current().getEnvOptions()` inside a static `eval` 
method in the `Calcite` transform.
   
   The `env` options are intentionally **not merged into plugin options**, so 
existing option-rule validation, unknown-key checks and sink table-placeholder 
rewriting are unaffected.
   
   Closes #13226.
   
   ## Motivation
   
   Today `env` is only consumed at job/engine level. Plugin factories cannot 
see it and SQL UDFs have no way to read user-defined parameters from it. 
`EnvCommonOptions.CUSTOM_PARAMETERS` already exists and is allowed by 
`EnvOptionRule`, but it was neither exposed nor documented.
   
   Example:
   
   ```hocon
   env {
     job.mode = "BATCH"
     custom_parameters {
       my_key = "my_value"
     }
   }
   ```
   
   ## Changes
   
   ### seatunnel-api
   - Add `TableFactoryContext#getEnvOptions()` and an empty-env default.
   - Add env-aware constructors to `TableSourceFactoryContext`, 
`TableSinkFactoryContext`, `MultiTableFactoryContext` and 
`TableTransformFactoryContext`.
   - Add env-aware overloads to all five `FactoryUtil` entry points 
(`createAndPrepareSource`, `restoreAndPrepareSource`, `createAndPrepareSink`, 
`createMultiTableSink`, `createAndPrepareMultiTableTransform`).
   - Preserve all existing constructors and methods; they delegate with an 
empty env.
   
   ### Zeta execution path
   - `MultipleTableJobConfigParser` passes its parsed `envOptions` to source, 
restore-source, transform, multi-table-sink and sink creation.
   - `DryRunConnectValidator` receives env and passes it to source, transform 
and sink dry-run contexts.
   - CLI and REST submissions share the same parser, so both get the capability.
   
   ### SQL transforms and UDFs
   - `Sql`: the factory passes env to `SQLMultiCatalogFlatMapTransform` -> 
`SQLTransform`; the transform keeps env as a serializable field and injects it 
into the transient SQL engine on the worker.
   - `SQLEngine#init(..., envOptions)` gets a default overload; `ZetaSQLEngine` 
stores env and calls `ZetaUDF#open(ReadonlyConfig)`.
   - The multi-table wrapper stores env on the base class **before** building 
per-table transforms, because `buildTransform` is invoked from the base 
constructor (subclass fields are still `null` at that point).
   - `Calcite`: env flows through `CalciteTransformFactory` -> 
`CalciteMultiCatalogTransform` -> `CalciteTransform` -> `CalciteSQLEngine`; 
`BuiltinFunctions` opens UDFs with env; `CalciteUdfContext` exposes env to 
static `eval` calls.
   
   ### Documentation
   - Documented `env.custom_parameters` and the plugin/UDF access patterns in 
`docs/en` and `docs/zh` (`concepts/config.md` and 
`configuration/JobEnvConfig.md`), including limitations.
   
   ## Compatibility
   
   - All existing constructors and method signatures are preserved as overloads 
delegating with an empty env.
   - Adding default methods to `ZetaUDF`, `CalciteUdf` and `SQLEngine` is 
binary compatible for existing implementations.
   - Flink/Spark execution is unchanged: their factory contexts and UDFs 
receive an empty env.
   - `env` is not decrypted by `ConfigShadeUtils` (which only processes 
`source`/`sink`/`transform`), so encrypted/sensitive values should not be 
placed in `env`; this is documented.
   - Adding a field to `SQLTransform` / `CalciteTransform` changes their 
implicit `serialVersionUID`; checkpoint/savepoint restore from older builds is 
not guaranteed across this change.
   
   ## Tests
   
   ### Unit / integration tests (all passing)
   
   ```bash
   ./mvnw -pl seatunnel-api test \
     -Dtest='TableFactoryContextEnvTest,FactoryUtilEnvOptionsTest'
   
   ./mvnw -pl seatunnel-transforms-v2 test \
     
-Dtest='SQLTransformFactoryTest,CalciteTransformFactoryTest,SQLMultiCatalogFlatMapTransformTest,AbstractMultiCatalogTransformTest,ZetaSQLEngineTest,CalciteSQLEngineTest,ZetaUdfEnvOptionsTest,CalciteUdfEnvOptionsTest'
   
   ./mvnw -pl seatunnel-engine/seatunnel-engine-client test \
     -Dtest=MultipleTableJobConfigParserTest
   
   ./mvnw -pl seatunnel-core/seatunnel-starter test \
     -Dtest=SeaTunnelConfValidateCommandTest
   ```
   
   Results: 8 API tests, 387 transform tests, 17 parser tests and 30 starter 
tests passed. The parser tests also serialize/deserialize the parsed transform 
to simulate the client -> worker hop.
   
   ### E2E tests (passing on Zeta, Testcontainers)
   
   ```bash
   ./mvnw -pl 
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-udf 
verify \
     -DskipUT=true -DskipIT=false \
     -Dit.test=ExampleUdfIT#testCustomUdfReadsEnvOptions
   
   ./mvnw -pl 
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-part-2 
verify \
     -DskipUT=true -DskipIT=false \
     -Dit.test=TestCalciteIT#testCalciteUdfReadsEnvOptions
   ```
   
   - `custom_udf_env.conf`: `env.custom_parameters.probe` reaches 
`ZetaUDF#open(ReadonlyConfig)` and the Assert sink verifies `env-value:Hello 
World`.
   - `calcite_env_udf.conf`: the same value is read through 
`CalciteUdfContext#current().getEnvOptions()` in a static `eval`; the test is 
disabled on Flink/Spark because env propagation is Zeta-only.
   
   ### Formatting / packaging
   
   ```bash
   ./mvnw spotless:apply
   ./mvnw -pl <changed modules> -DskipTests verify
   ```
   
   ## Known limitations (documented)
   
   - Flink/Spark provide an empty env to factories and UDFs in this change.
   - `ZetaUDF.resultType(...)` runs before `open(...)`, so it cannot read env 
yet.
   - A Calcite UDF static `eval` may run during planning constant folding where 
`CalciteUdfContext.current()` is `null`; UDFs must null-check.
   - Calcite UDFs should not store job-scoped env in static fields; read it 
from the per-row context instead.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to