csurong opened a new issue, #4301: URL: https://github.com/apache/amoro/issues/4301
### What happened? When AMS builds the Spark submit command for a Kubernetes-native optimizer, `KUBERNETES_EXECUTOR_LABEL_PREFIX` is set to `spark.kubernetes.driver.label.`, the same value as the driver prefix. `addKubernetesProperties()` writes three labels for the driver and then three intended labels for executors. Because Spark options are stored in a map, the second set overwrites the same driver keys. The generated command contains only `spark.kubernetes.driver.label.*` options and no `spark.kubernetes.executor.label.*` options. As a result, executor pods do not receive Amoro's structured optimizer labels: `optimizer-group`, `optimizer-implementation`, and `optimizer-id`. This does not prevent optimizer startup, execution, or release, but it affects observability and label-based operations. Expected behavior: emit both driver and executor label options using Spark's separate documented prefixes. ### Affects Versions master (ec28554a2) ### What table formats are you seeing the problem on? _No response_ ### What engines are you seeing the problem on? Optimizer, Spark ### How to reproduce 1. Create a `SparkOptimizerContainer` whose `optimizingContainerProperties.master` uses a `k8s://` URL. 2. Call `generateStartupArgs(...)` for an optimizer resource. 3. Inspect the generated `--conf` arguments. 4. Observe that the three `spark.kubernetes.driver.label.*` options are present, while the corresponding `spark.kubernetes.executor.label.*` options are absent. ### Relevant log output ```shell ``` ### Anything else Spark documents separate driver and executor label prefixes: https://spark.apache.org/docs/3.5.8/running-on-kubernetes.html#spark-properties The incorrect prefix appears to have been introduced with the Kubernetes-native Spark optimizer support. A one-line constant correction plus regression coverage for both driver and executor startup arguments should address the issue. ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's Code of Conduct -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
