JAEKWANG97 opened a new issue, #13099:
URL: https://github.com/apache/seatunnel/issues/13099

   ## Background
   
   FieldEncrypt reads `key` from configuration and keeps it in a String field. 
The first call to `transformRow()` finds an `Encryptor` and calls `init(String 
key)`. The initialized encryptor is transient. The existing interface also 
defines `support(String)`, `encrypt(String)`, and `decrypt(String)`.
   
   At commit `67941ddefacebbcadab6ba72f7e3877d10cbaee9`, two other parts of the 
code affect startup. `AbstractMultiCatalogTransform` does not forward `open()` 
or `close()` to its children. Zeta's `TransformFlowLifeCycle.open()` catches 
initialization exceptions and logs them without rethrowing them. An `open()` 
override in `FieldEncryptTransform` alone cannot ensure that key preparation 
succeeds before row processing starts.
   
   ## Motivation
   
   Users need to get data keys from an external key system. They should not 
need to put a raw data key in the job configuration. FieldEncrypt should 
support this choice while existing static-key jobs continue to work.
   
   ## Goals
   
   - Keep existing options, defaults, ciphertext, and `Encryptor` 
implementations compatible.
   - Separate key preparation from local encryption and decryption.
   - Prepare external keys before processing rows. Fail startup if preparation 
fails.
   - Recover the same keys after restart without remote calls during row 
processing.
   - Define a public interface for official and custom providers before 
implementing it.
   - Agree on the new ciphertext format, credential handling, and module layout 
before adding a reference provider.
   
   ## Non-Goals
   
   - Generate a new data key each time a worker starts.
   - Add a SeaTunnel key database or require a particular sink.
   - Refresh keys during row processing or automatically encrypt stored data 
again.
   - Fetch an unconfigured historical key when a row requests it.
   - Export KMS master keys or call remote encryption APIs for each field.
   - Add several vendor integrations in the first change.
   
   ## Design
   
   ### User configuration
   
   I propose two paths. Without `key_provider`, FieldEncrypt uses the existing 
`key` option and ciphertext format. A missing or blank `key` fails validation. 
It keeps the current algorithm defaults and first-row initialization. We can 
extract that initialization into a helper without changing when it runs.
   
   With `key_provider`, FieldEncrypt prepares keys in `open()` and uses the new 
format. This example proposes the configuration shape. These options and the 
`example` provider do not exist yet.
   
   ```hocon
   transform {
     FieldEncrypt {
       fields = ["phone"]
       mode = "encrypt"
       algorithm = "AES_GCM"
   
       key_provider = "example"
       key_provider_config {
         endpoint = "https://keys.example.invalid";
         credential_source = "worker"
         write_key = { id = "K2", reference = "phone-key/version-2" }
         read_keys = [
           { id = "K1", reference = "phone-key/version-1" },
           { id = "K2", reference = "phone-key/version-2" }
         ]
       }
   
       ciphertext_format = "v1"
       ciphertext_prefix = "ENC:"
     }
   }
   ```
   
   The transform owns `key_provider`, `key_provider_config`, 
`ciphertext_format`, and `ciphertext_prefix`. Each new option needs a SeaTunnel 
`Option` definition. The proposed defaults are no provider, an empty provider 
block, `legacy` format, and `ENC:` prefix. The prefix applies only to the new 
format. A provider requires explicit `ciphertext_format = "v1"`; other 
combinations fail validation.
   
   The selected provider defines and validates every field inside 
`key_provider_config`. The example uses versioned references and credentials 
from the worker environment. It contains no raw key or credential. A real 
provider must document its authentication method and recovery inputs.
   
   When both `key_provider` and `key` exist, the provider takes priority. A 
provider failure fails startup; it never selects the static key as a fallback. 
The ignored `key` can still enter existing job state. Review must settle this 
mixed-configuration risk before the provider release.
   
   ### Proposed interfaces
   
   The public SPI lets users add a provider without changing FieldEncrypt. SPI 
means an interface that external implementations can use. The following 
signatures are a proposal for review. They are not existing classes or an 
implementation patch.
   
   ```java
   public interface KeyProviderFactory {
       String identifier();
       void validate(ReadonlyConfig config) throws KeyProviderException;
       KeyProvider create();
   }
   
   public interface KeyProvider extends AutoCloseable {
       PreparedKeys prepare(ReadonlyConfig config, KeyRequest request)
               throws KeyProviderException;
       void close() throws KeyProviderException;
   }
   
   public interface PreparedKeys extends AutoCloseable {
       Optional<PreparedKey> writeKey();
       Map<String, PreparedKey> readKeys();
       void close() throws KeyProviderException;
   }
   
   public interface DataCipher extends AutoCloseable {
       String identifier();
       void init(PreparedKeys keys, CipherOptions options) throws 
CipherException;
       String encrypt(String value) throws CipherException;
       String decrypt(String value) throws CipherException;
       void close() throws CipherException;
   }
   ```
   
   `KeyProviderFactory.identifier()` supplies the name used by `key_provider`. 
Service discovery selects one factory by that name. Duplicate names fail 
validation. `validate()` checks configuration without remote calls. `create()` 
creates a worker-local provider without contacting the service.
   
   `KeyRequest` contains the mode and algorithm. `prepare()` opens the provider 
session and gets the required keys once per transform instance. If it fails, 
the provider releases resources that it created before throwing a sanitized 
exception.
   
   `PreparedKey` contains a logical ID and a `java.security.Key` usable by the 
selected cipher. It may contain a local cryptographic handle. It must not 
contain an exported KMS master key. `PreparedKeys` returns an immutable map 
indexed by logical ID. Reject conflicting recovery references for the same ID. 
Encrypt mode requires a write key and includes it in the read map. Decrypt mode 
requires read keys but no write key.
   
   `CipherOptions` contains the algorithm, format, prefix, and size limits. It 
contains no provider credentials or SDK clients. `DataCipher` checks format 
headers and performs local cryptographic operations. Unknown keys fail through 
the local map. It cannot fetch more keys.
   
   The transform owns the provider, prepared keys, and cipher. It closes the 
cipher first, then the prepared keys, then the provider. The cipher borrows 
keys and must not destroy them. `PreparedKeys.close()` releases key resources. 
`KeyProvider.close()` releases sessions and clients. Each close operation must 
tolerate partial initialization and repeated calls. Exceptions must identify 
the operation without exposing secret values or unsafe nested exceptions.
   
   I propose a separate `DataCipher` SPI for the new path. The existing 
`Encryptor` interface and service registration stay unchanged. Separate service 
discovery selects a new cipher by its `identifier()` and the existing 
`algorithm` option. Duplicate cipher identifiers fail validation. Existing 
custom `Encryptor` implementations continue to work on the legacy path. They 
need a `DataCipher` implementation to support provider keys and the new format.
   
   ### Startup, rows, and cleanup
   
   `FactoryUtil` validates the option rules from 
`FieldEncryptTransformFactory`. The factory returns a creator for the 
multi-table wrapper. Neither contacts the key service or calls `open()`. The 
proposed worker flow is:
   
   ```text
   Zeta calls open() on the runtime transform
     -> outer wrappers forward open(), if present
     -> FieldEncryptMultiCatalogTransform calls each child's open()
        -> the child creates a KeyProvider
        -> prepare() gets the configured keys
        -> DataCipher.init() checks and accepts the keys
        -> the child becomes ready
   
   For each row
     -> transformRow() uses the ready DataCipher
     -> the cipher selects a prepared key and processes the value locally
   
   On shutdown
     -> wrappers forward close() to their children
        -> the child closes the cipher, prepared keys, and provider
   ```
   
   I recommend separate prerequisite changes for common lifecycle forwarding 
and Zeta startup failure propagation. Forwarding belongs in the common wrappers 
and chains because they own their children. A FieldEncrypt-only override would 
leave the same problem in composed transform chains. These changes need 
regression tests for other transforms, Map/FlatMap chains, and error-handling 
wrappers.
   
   If one child fails, it cleans up its partial initialization. The wrapper 
closes earlier children and rethrows the startup failure. Cleanup continues if 
another close operation fails. The engine must stop startup before processing 
rows. A child rejects rows before readiness, after failure, and after close.
   
   Each child owns its resources. There is no shared JVM key cache. Mutable 
cipher state must be isolated between concurrent calls. Providers need finite 
timeouts, bounded retries, and cancellation support. Their limits need 
agreement before implementation. Engine adapters and schema-change paths need 
explicit verification before they are listed as supported.
   
   ### Restart and key rotation
   
   A data encryption key, or DEK, encrypts field values. A KMS master key 
protects that DEK and stays inside the key system. Both the old and new paths 
use worker memory. Restart creates new runtime objects from configuration and 
the key system.
   
   For v1, users supply references to existing keys. The provider must recover 
the same DEK from each reference after restart. Some providers also need an 
immutable version or a protected data-key blob. Their configuration must supply 
those recovery inputs. SeaTunnel adds no database for them.
   
   In the example, `K1` and `K2` are IDs that the provider returns with the 
keys. The versioned references tell the provider how to recover them. New 
writes use K2. Reads can use K1 or K2 because startup prepares both keys. To 
rotate again, users change the write reference and restart the job. They keep 
older read references only while older data needs decryption.
   
   The write key stays fixed during that execution. An unknown ciphertext key 
ID fails locally. The provider does not generate a new key on each restart. 
Review must set limits on the read-key count, startup duration, and memory use.
   
   Revoking remote access does not remove a key already held in worker memory. 
Provider documentation must explain the required stop/restart procedure. 
Cleanup attempts to release key resources but cannot promise complete JVM 
memory erasure.
   
   ### Ciphertext format and prefix
   
   Legacy output stays unchanged. For the provider path, I propose a prefix 
followed by a Base64-encoded envelope. The envelope contains a version, logical 
key ID, algorithm ID, nonce, encrypted payload, and authentication tag. The 
prefix must contain a character outside the legacy Base64 alphabet. The default 
`ENC:` meets this rule.
   
   The cipher must authenticate the prefix, header, and payload. AES-GCM is the 
built-in choice for the new path. Custom ciphers must provide equivalent 
authentication and local processing. Legacy AES-CBC stays available through the 
existing path. A header does not add authentication to CBC.
   
   I propose these default behaviors:
   
   - Encrypt unmarked input as plaintext.
   - If input has the configured prefix, parse and authenticate it with the 
prepared read keys before preserving it unchanged.
   - Reject malformed marked input, unknown keys or versions, and 
authentication failures.
   - In decrypt mode, require the configured prefix and format. Reject a 
mismatch.
   - Select legacy decoding through the legacy configuration. Do not guess 
between formats or silently return plaintext.
   
   Plaintext that starts with the reserved prefix therefore fails unless it is 
a valid authenticated envelope. This restriction needs documentation. A mixed 
legacy/new dataset needs separate processing or an agreed migration tool. The 
binary field layout, encoding rules, and parser limits still need format review 
before implementation.
   
   ### Security and modules
   
   Only recovery references and approved non-secret options belong in 
serialized provider configuration. Workers resolve credentials and create 
runtime key objects. SDK clients, prepared keys, and live handles must not 
enter job or checkpoint state. Each provider must define an allowed 
configuration schema and reject inline secrets for this path.
   
   Existing literal-key state and Zeta's row DEBUG/dry-run output need separate 
review. The current code does not satisfy the issue's full secrecy requirement. 
Check configuration output, exception causes, suppressed exceptions, metrics, 
and error-row storage as well. The provider release must address these paths 
before claiming that they expose no secrets.
   
   I recommend a small encryption SPI jar separate from `seatunnel-api` and 
vendor SDKs. A proposed name is `seatunnel-transform-encrypt-api`. The 
transform and optional provider jars depend on it. Vendor SDKs belong only in 
optional provider jars, such as `seatunnel-transform-encrypt-provider-<id>`. 
These are proposed artifacts; the first code change creates none of them.
   
   Packaged-jar tests must check discovery, classloader isolation, dependency 
relocation, and licenses. The first reference provider remains a review choice. 
A provider-free test double must first prove the lifecycle and zero remote 
calls during row processing.
   
   ### Delivery phases
   
   1. Agree on this design before implementation.
   2. Extract static-key preparation without changing existing public 
interfaces or first-row behavior. Add no cloud SDK, provider module, or public 
SPI.
   3. Review lifecycle forwarding and startup failure propagation in separate 
prerequisite changes. Prove the flow with an internal provider-free test double.
   4. After design acceptance, add the agreed public interfaces and artifacts, 
then one reference provider. Add other providers separately.
   
   ### Decisions requested before acceptance
   
   1. Accept the proposed configuration and interfaces, including separate 
legacy and provider paths?
   2. Accept common lifecycle fixes as separate prerequisites, and define the 
engines and schema changes that v1 must support?
   3. Accept configured key recovery and rotation by restart, and select one 
reference provider?
   4. Agree on the binary format, prefix behavior, and resource limits?
   5. Resolve mixed-configuration secrets, credential transport, and row 
diagnostics before releasing a provider?
   
   ## Compatibility, Deprecation, and Migration Plan
   
   Keep every existing option, default, `Encryptor` method, package, and 
registration path. Existing AES-CBC/AES-GCM data must remain decryptable. 
Repeated encryption can produce different bytes because it uses random IVs. 
Compatibility requires the same encoding and successful cross-decryption.
   
   The new format requires explicit selection. Older releases may not read it. 
Deploy compatible readers before changing writers and document rollback 
restrictions. Keep the current `max_field_length` behavior. Define separate 
limits for new encoded input and output.
   
   Schema changes must preserve field selection and resource ownership. 
FieldEncrypt computes field indexes in its constructor. Test index updates or 
safely reject unsupported changes before processing affected rows. 
Restart/restore must reconstruct transient readiness, keys, and cipher state. 
Tests must cover same-version recovery; cross-version saved-job compatibility 
needs separate evidence.
   
   ## Test Plan
   
   Extend existing FieldEncrypt/Encryptor tests and `TestFieldEncryptIT`. Cover 
legacy fixtures, cross-decryption, plugin loading, and null, empty, and long 
values. Exercise the factory, wrappers, and worker together. Include multiple 
tables, schema changes, initialization failures, cleanup failures, 
cancellation, and restart/restore.
   
   For 0, 1, and 10,000 rows, count calls during startup and row processing 
separately. Assert zero remote calls during row processing, including failures 
for unknown keys. After restart, decrypt K1 data with K1/K2 prepared and K2 
selected for writes.
   
   Test header and tag tampering, prefix collisions, authenticated 
double-encryption checks, and resource limits. Use synthetic secret values to 
detect exposure in logs, exceptions, metrics, and state. Test the reference 
provider's recovery and installation with packaged jars and E2E tests. Update 
English and Chinese documentation with the implementation.
   
   These are planned tests. This draft reports no implementation, Maven build, 
provider call, E2E result, or performance measurement. The implementation must 
measure startup cost and throughput.
   
   ## Alternatives Considered
   
   - Per-row KMS calls add remote access to row processing. Startup preparation 
avoids this dependency.
   - New keys on each restart require durable storage and recovery. Existing 
versioned references avoid that addition in v1.
   - A SeaTunnel key store adds another service to operate. Existing key 
systems and deployment configuration can supply recovery inputs.
   - Mandatory legacy initialization in `open()` changes direct calls and 
failure timing for empty input. Keep first-row behavior initially.
   - A JVM-wide cache can reduce startup calls but needs shared ownership, 
isolation, and cleanup rules.
   
   ## References
   
   - Feature and discussion: https://github.com/apache/seatunnel/issues/12407
   - Implementation sequencing guidance: 
https://github.com/apache/seatunnel/issues/12407#issuecomment-6246034419
   - dev@ discussion request: 
https://github.com/apache/seatunnel/issues/12407#issuecomment-5771146218
   - Contribution process: 
https://seatunnel.apache.org/community/contribution_guide/STIP/
   - Current transform: 
https://github.com/apache/seatunnel/blob/67941ddefacebbcadab6ba72f7e3877d10cbaee9/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/encrypt/FieldEncryptTransform.java
   - Existing Encryptor contract: 
https://github.com/apache/seatunnel/blob/67941ddefacebbcadab6ba72f7e3877d10cbaee9/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/encrypt/encryptor/Encryptor.java
   - Multi-table lifecycle: 
https://github.com/apache/seatunnel/blob/67941ddefacebbcadab6ba72f7e3877d10cbaee9/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/common/AbstractMultiCatalogTransform.java
   - Zeta lifecycle/diagnostics: 
https://github.com/apache/seatunnel/blob/67941ddefacebbcadab6ba72f7e3877d10cbaee9/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/flow/TransformFlowLifeCycle.java
   - Configuration baseline: 
https://github.com/apache/seatunnel/blob/67941ddefacebbcadab6ba72f7e3877d10cbaee9/seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/encrypt/FieldEncryptTransformConfig.java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to