nzw921rx opened a new issue, #12886:
URL: https://github.com/apache/seatunnel/issues/12886
### Search before asking
- [x] I searched existing SeaTunnel issues and STIPs for Kubernetes
Application Mode high availability and found no equivalent follow-up task.
### Background
The umbrella issue #11857 tracks Native Zeta Application Mode and external
resource management. STIP-41 (#12457) and its initial implementation (#12460)
established the single-master Kubernetes MVP.
That MVP intentionally uses:
- one Zeta master;
- `backup-count = 0`;
- no live master failover or automatic worker replacement.
If the master or a worker is lost, the current application fails and
recovery requires submitting a new application that restores from the latest
durable checkpoint.
This child issue tracks Kubernetes Application Mode high availability as a
separate follow-up. The HA implementation should extend the already merged
application lifecycle rather than create another deployment or recovery model.
### Goal
Allow a Kubernetes Application Mode cluster to survive loss of an
application Pod while the SeaTunnel job remains active, without requiring the
submitter to recreate the application.
The implementation should reuse Zeta's existing HA mechanisms:
- Hazelcast member discovery and active-master election;
- partition backups and IMap recovery;
- checkpoint restore;
- existing member-loss pipeline restoration.
### Required boundaries
1. **Reuse the Zeta HA model**
Do not introduce a second leader-election or job-recovery state machine
in the Kubernetes provider. Kubernetes owns Pod lifecycle and stable service
discovery; Zeta remains responsible for master election, cluster state,
pipeline recovery and checkpoint semantics.
2. **Keep application Pods online while the application is active**
The Kubernetes workload must maintain the desired number of control-plane
Pods until the SeaTunnel application reaches a terminal state.
A standard Kubernetes `Job` may mark the whole application complete when
one master Pod exits. The implementation must define a controller model that
keeps the desired replicas alive during failover. Possible designs include a
StatefulSet plus an application coordinator, or a custom Job lifecycle
contract, but the ownership and termination semantics must be explicit.
3. **Provide a stable ClusterIP service**
Application clients should use a stable `ClusterIP` service instead of
depending on a single master Pod address.
The Service must continue to work after the active master changes. If
Hazelcast member discovery still requires an internal headless service or
Kubernetes API discovery, retain that as an implementation detail rather than
using it as the external application endpoint.
4. **Require backups for HA**
An HA application must have at least two master replicas and
`seatunnel.engine.backup-count > 0`. The initial recommendation is
`backup-count: 1` with at least two masters, so the cluster can retain
partition state after one master is lost.
Invalid combinations, such as HA enabled with one master or HA enabled
with `backup-count: 0`, must fail fast with a clear configuration error.
5. **Preserve worker availability**
Worker Pods that participate in a running application must remain online
while the application is active. If a worker Pod is lost, Kubernetes should
replace or rejoin it when the selected recovery policy requires replacement,
and Zeta should continue through its existing member-loss recovery path.
The implementation must define the worker-failure policy and avoid
allowing a single Pod exit to make the owner Kubernetes workload terminal while
recovery is still possible.
6. **Keep application management APIs valid across failover**
`submit`, `status`, `cancel`, cleanup and checkpoint restore must
continue to work with the native Kubernetes application ID after the active
master changes. The application must not be reported as successful or failed
solely because a master Pod was replaced.
7. **Preserve terminal cleanup**
Once the job reaches a terminal state or is canceled, the application
must release its workers, masters, Service, owned Secret/ConfigMap resources
and other application-owned objects. Durable user-owned checkpoint storage must
not be deleted.
### Acceptance criteria
- A Kubernetes Application Mode job can start with at least two master
replicas and `backup-count > 0`.
- Invalid HA configurations fail fast before submitting the application.
- The application workload keeps the required master Pods online for the
lifetime of the application.
- Killing the active master Pod triggers Kubernetes replacement and Zeta
active-master re-election without client resubmission.
- The ClusterIP endpoint remains usable after the active master changes.
- The job restores affected pipelines through the existing Zeta member-loss
recovery and checkpoint path.
- Killing a standby master does not interrupt the job.
- The defined worker-loss scenario uses the existing Zeta recovery path
without silently changing the application to a terminal state.
- `status` and `cancel` continue to operate by Kubernetes application ID
across failover.
- Cleanup removes all application-owned Pods and resources after normal
completion, failure or cancellation.
- English and Chinese documentation describe the HA configuration, minimum
replicas, backup count and failover guarantees.
### Suggested test coverage
- Configuration tests for valid and invalid HA replica/backup-count
combinations.
- Unit tests for lifecycle ownership and terminal-state handling when a
master Pod exits.
- K3s or Kubernetes E2E coverage with at least two or three master Pods.
- Active-master Pod deletion during a streaming job, including recovery from
checkpoint.
- Standby-master deletion.
- Worker Pod deletion according to the selected policy.
- ClusterIP connectivity after active-master replacement.
- Status, cancel and cleanup behavior across failover.
### Implementation areas to inspect
- `ApplicationJobRunner`
- `KubernetesResourceManagerDriver`
- Kubernetes master and worker CLIs
- Kubernetes Job, Service, Secret and owner-reference construction
- Application lifecycle and status mapping
- Application Mode configuration and generated Hazelcast configuration
The implementation may require a small update to STIP-41 if the Kubernetes
workload changes from one Job-controlled master Pod to a multi-replica
controller model.
### Out of scope
- YARN Application Mode HA.
- Autoscaling or dynamic worker count.
- Multi-job applications.
- Cross-cluster or cross-region recovery.
- Kerberos and delegation-token support.
- Changes to standalone cluster HA behavior.
### Related work
- Umbrella: #11857
- STIP-41: #12457
- Initial Kubernetes Application Mode implementation: #12460
### Are you willing to submit a PR?
- [x] Yes, I am willing to submit a PR.
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
