csurong opened a new pull request, #4289: URL: https://github.com/apache/amoro/pull/4289
## Why are the changes needed? Spark defaults `spark.kubernetes.submission.waitAppCompletion` to `true`. For long-running Spark optimizers on Kubernetes, this keeps each local `spark-submit` launcher process alive in the AMS pod for the lifetime of the optimizer, even though AMS does not use the launcher to track or manage the application. These retained processes add unnecessary memory pressure to AMS. AMS manages Kubernetes optimizers using a deterministic submission ID, so the launcher can exit after submitting the driver pod. Closes #4126. ## Brief change log - Default `spark.kubernetes.submission.waitAppCompletion` to `false` for Spark optimizers deployed on Kubernetes. - Preserve an explicitly configured value from the optimizing container or group. - Keep the YARN submission path unchanged. - Add regression tests for the Kubernetes default, explicit override, and YARN isolation. ## How was this patch tested? - [x] Add test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] Run test locally before making a pull request `mvn -q -pl amoro-ams -am -Dtest=TestSparkOptimizerContainer -Dsurefire.failIfNoSpecifiedTests=false -Pskip-dashboard-build test` Result: 4 tests run, 0 failures, 0 errors, 0 skipped. ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
