SEZ9 commented on issue #11857: URL: https://github.com/apache/seatunnel/issues/11857#issuecomment-5788019087
@det101 thanks for the +1 and the offer to help — the motivation you describe (wanting native Zeta job management while your capacity lives on YARN, with SeaTunnel-on-Flink giving a weaker management experience) is exactly the gap this issue is meant to close, and having someone with a real YARN deployment in the loop is valuable. I agree with @nzw921rx that YARN should take priority for the first pass, and that the right next step is a STIP rather than jumping straight into code. Today the source only has placeholders in this direction (`DeployType` enumerating `STANDALONE` / `YARN` / `KUBERNETES`, `ResourceManagerFactory` routing to `YarnResourceManager` / `KubernetesResourceManager`, and `ThirdPartyResourceManager` exposing only `createNewWorker(...)` / `releaseWorker(...)`, with those lifecycle methods still returning `null`), so the design doc needs to define the actual application-mode contract before implementation can be split out. @det101, if you have the bandwidth to drive the STIP, I'd suggest it cover, for YARN v1 specifically: 1. Scope: one job == one application / one Zeta cluster; autoscaling, HA, Kerberos and multi-job tenancy explicitly out of v1. 2. Application lifecycle separated from worker lifecycle — who creates the application/master, how startup is coordinated, how workers join, and how cleanup behaves when bootstrap or worker provisioning partially fails. 3. Lifecycle states and their transition owners, ideally shaped so Kubernetes can share them later with platform-specific details underneath. 4. The minimum config/package changes needed for v1 — the project-structure proposal @nzw921rx posted (a root-level `resource-managers/` module alongside `seatunnel-engine`, with `deployment/` types in `seatunnel-engine-common` / `seatunnel-engine-client` and a `ResourceManagerDriver` layer in `seatunnel-engine-server`) is a good starting point to react to in the doc rather than restart from scratch. Once the STIP is up, let's do the in-depth discussion there and then open separate implementation issues for YARN and Kubernetes. Please link it back here when it's ready. <!-- streview-comment:1249 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
