databius commented on issue #35911: URL: https://github.com/apache/airflow/issues/35911#issuecomment-2002920286
I think the update caused the new issue since SparkSubmitOperator accidentally overwrote the Base Operator's parameter. https://github.com/apache/airflow/blob/main/airflow/providers/apache/spark/operators/spark_submit.py#L76 > :param queue: The name of the YARN queue to which the application is submitted. (will overwrite any yarn queue defined in the connection's extra JSON) https://github.com/apache/airflow/blob/main/airflow/models/baseoperator.py#L587 > :param queue: which queue to target when running this job. Not all executors implement queue management, the CeleryExecutor does support targeting specific queues. So if you are using CeleryKubernetesExecutor, you cannot decide the Operator run on KubernetesExecutor by setting the `queue` to [kubernetes_queue](https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/configurations-ref.html#kubernetes-queue) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
