SEZ9 commented on issue #12457: URL: https://github.com/apache/seatunnel/issues/12457#issuecomment-5964917249
@nzw921rx thanks for updating the design. The comment only says "Update design", so it is hard to tell from the thread what actually changed. Could you add a short changelog to the STIP (or here) that maps the revisions to the earlier review points: whether v1 is now YARN-first with Kubernetes kept as a design comparison, how the existing `YarnResourceManager`/`KubernetesResourceManager` placeholders relate to the proposed `ResourceManagerDriver` (single owner for allocation, release, retries and terminal cleanup), and whether the YARN contract items (submission/rollback ownership, worker-registration deadline, detached status/cancel semantics, option names/defaults, checkpoint restore identity, AM/worker exit behavior) are now frozen in the text. A pointer to the updated sections is enough; I don't want to re-review the whole document to find the diff. @LeonYoah the upload cost of a large `yarn.distribution` archive is a fair operational concern, and the `spark.yarn.archive` / `spark.yarn.jars` model is a reasonable reference. As discussed above, though, a pre-staged HDFS archive or per-jar cache brings in versioning/integrity, HDFS permission and Kerberos handling, cache lifecycle, and connector/classloader upgrade semantics, so let's keep it out of the first implementation. The current design should stay with one self-contained archive per submission, and a follow-up STIP can evaluate a versioned shared-artifact option. @nzw921rx, please note this explicitly in the design as a non-goal for v1 so the scope is clear. Remaining asks: 1. @nzw921rx: post the changelog of the design update against the points above and confirm the YARN-first split and the single resource-management owner. 2. @nzw921rx: record the shared-HDFS-artifact idea as an out-of-scope follow-up in the STIP. 3. Keep the linked PR description in sync with the updated design once those are in. <!-- streview-comment:1484 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
