HadarSha opened a new issue, #41797:
URL: https://github.com/apache/airflow/issues/41797
### Apache Airflow version
2.10.0
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When trying to use hybrid executors and aliasing one of them (in my case AWS
ECS Executor), I get an error when Airflow is being initialized (my deployment
uses the Airflow Official Helm Chart on AWS EKS, this exception happens in the
`airflow-run-migrations` pod).
My offending configuration bit is the following environment variable trying
to set the `AwsEcsExecutor` to have an alias of `FargateExecutor` :
```
- name: "AIRFLOW__CORE__EXECUTOR"
value:
"CeleryExecutor,KubernetesExecutor,airflow.providers.amazon.aws.executors.ecs.AwsEcsExecutor:FargateExecutor"
```
The traceback I am getting is as follows:
```
....................
Last check result:
$ airflow db check
/home/airflow/.local/lib/python3.12/site-packages/airflow/plugins_manager.py:30
DeprecationWarning: 'cgitb' is deprecated and slated for removal in Python 3.13
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/__main__.py", line
38, in <module>
from airflow.cli import cli_parser
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/cli_parser.py",
line 62, in <module>
for executor_name in ExecutorLoader.get_executor_names():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/executor_loader.py",
line 150, in get_executor_names
return cls._get_executor_names()
^^^^^^^^^^^^^^^^^^^^^^^^^
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/executor_loader.py",
line 111, in _get_executor_names
raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: Incorrectly formatted executor
configuration. Second portion of an executor configuration must be a module
path or plugin but received: FargateExecutor
2024-08-27T14:53:00.179007881Z
```
### What you think should happen instead?
Based on the [official Airflow documentation for using Hybrid executors with
aliasing](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html#configuration),
my configuration should be valid and I expect the ECSExecutor to be available
for use as `FargateExecutor` as per my alias given in the configuration.
### How to reproduce
1. Make sure you are using Airflow Helm Chart 1.15.0 and Airflow Version
2.10.0
2. Make sure Celery and Kubernetes providers are also installed, as I was
using the `CeleryKubernetesExecutor` as my default executor (though this might
not be critical for testing the alias issue per se) - e.g. `pip install
apache-airflow-providers-celery==3.8.0 && pip install
apache-airflow-providers-cncf-kubernetes==8.4.0)
3. Install the Amazon Provider package version 8.28.0 (e.g. `pip install
apache-airflow-providers-amazon==8.28.0`)
4. Set the `executor` key in the Helm chart to be `CeleryKubernetesExecutor`
5. Set the following environment variable:
`AIRFLOW__CORE__EXECUTOR=CeleryExecutor,KubernetesExecutor,airflow.providers.amazon.aws.executors.ecs.AwsEcsExecutor:FargateExecutor`
6. Make sure the `migrateDatabaseJob` is enabled in the Helm chart
7. Deploy Airflow and watch the migrations job when it runs as it's where I
had encountered my issue (but it might pop up in other places as well)
### Operating System
Debian GNU/Linux 12 (bookworm)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.28.0
apache-airflow-providers-atlassian-jira==2.7.0
apache-airflow-providers-celery==3.8.0
apache-airflow-providers-cncf-kubernetes==8.4.0
apache-airflow-providers-common-compat==1.1.0
apache-airflow-providers-common-io==1.4.0
apache-airflow-providers-common-sql==1.15.0
apache-airflow-providers-fab==1.2.2
apache-airflow-providers-ftp==3.10.1
apache-airflow-providers-google==10.19.0
apache-airflow-providers-http==4.12.0
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-jenkins==3.7.0
apache-airflow-providers-openlineage==1.10.0
apache-airflow-providers-postgres==5.12.0
apache-airflow-providers-salesforce==5.8.0
apache-airflow-providers-slack==8.9.0
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-sqlite==3.8.2
apache-airflow-providers-tableau==4.6.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
K8s deployment on AWS EKS, offending pod is `airflow-run-migrations` (AKA
`migrateDatabaseJob` in the Helm chart), the executor being aliased in my case
is the AwsEcsExecutor from the apache-airflow-providers-amazon package.
Airflow Helm Chart Version - 1.15.0
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
