o-nikolas commented on code in PR #1052: URL: https://github.com/apache/airflow-site/pull/1052#discussion_r1710119155
########## landing-pages/site/content/en/blog/airflow-2.10.0/index.md: ########## @@ -0,0 +1,174 @@ +--- +title: "Apache Airflow 2.10.0 is here" +linkTitle: "Apache Airflow 2.10.0 is here" +author: "Utkarsh Sharma" +github: "utkarsharma2" +linkedin: "utkarsh-sharma-5791ab8a" +description: "Apache Airflow 2.10.0 is a game-changer, with powerful Dataset improvements and the groundbreaking Hybrid Executor, set to redefine your workflow capabilities!" +tags: [Release] +date: "2024-08-08" +--- + +I'm happy to announce that Apache Airflow 2.10.0 is now available, bringing an array of noteworthy enhancements and new features that will greatly serve our community. + +Apache Airflow 2.10.0 contains over 135 commits, which include 43 new features, 85 improvements, 43 bug fixes, and 26 documentation changes. + +**Details**: + +📦 PyPI: https://pypi.org/project/apache-airflow/2.10.0/ \ +📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.10.0/ \ +🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html \ +🐳 Docker Image: "docker pull apache/airflow:2.10.0" \ +🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.10.0 + + +## Hybrid Execution Review Comment: ```suggestion ## Multiple Executor Configuration (formerly "Hybrid Execution") ``` ########## landing-pages/site/content/en/blog/airflow-2.10.0/index.md: ########## @@ -0,0 +1,174 @@ +--- +title: "Apache Airflow 2.10.0 is here" +linkTitle: "Apache Airflow 2.10.0 is here" +author: "Utkarsh Sharma" +github: "utkarsharma2" +linkedin: "utkarsh-sharma-5791ab8a" +description: "Apache Airflow 2.10.0 is a game-changer, with powerful Dataset improvements and the groundbreaking Hybrid Executor, set to redefine your workflow capabilities!" +tags: [Release] +date: "2024-08-08" +--- + +I'm happy to announce that Apache Airflow 2.10.0 is now available, bringing an array of noteworthy enhancements and new features that will greatly serve our community. + +Apache Airflow 2.10.0 contains over 135 commits, which include 43 new features, 85 improvements, 43 bug fixes, and 26 documentation changes. + +**Details**: + +📦 PyPI: https://pypi.org/project/apache-airflow/2.10.0/ \ +📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.10.0/ \ +🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html \ +🐳 Docker Image: "docker pull apache/airflow:2.10.0" \ +🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.10.0 + + +## Hybrid Execution Review Comment: I'd also like to write a blog post entirely for multi exec config, since there is some nuance and details there that's worth spelling out and having it on the website blog like [this example](https://airflow.apache.org/blog/introducing_setup_teardown/). I have not created one before, should I just follow this PR as an example of how to do that? I have the content written already but would need to translate it into this formatting. ########## landing-pages/site/content/en/blog/airflow-2.10.0/index.md: ########## @@ -0,0 +1,174 @@ +--- +title: "Apache Airflow 2.10.0 is here" +linkTitle: "Apache Airflow 2.10.0 is here" +author: "Utkarsh Sharma" +github: "utkarsharma2" +linkedin: "utkarsh-sharma-5791ab8a" +description: "Apache Airflow 2.10.0 is a game-changer, with powerful Dataset improvements and the groundbreaking Hybrid Executor, set to redefine your workflow capabilities!" +tags: [Release] +date: "2024-08-08" +--- + +I'm happy to announce that Apache Airflow 2.10.0 is now available, bringing an array of noteworthy enhancements and new features that will greatly serve our community. + +Apache Airflow 2.10.0 contains over 135 commits, which include 43 new features, 85 improvements, 43 bug fixes, and 26 documentation changes. + +**Details**: + +📦 PyPI: https://pypi.org/project/apache-airflow/2.10.0/ \ +📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.10.0/ \ +🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html \ +🐳 Docker Image: "docker pull apache/airflow:2.10.0" \ +🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.10.0 + + +## Hybrid Execution Review Comment: CC @jedcunningham ########## landing-pages/site/content/en/blog/airflow-2.10.0/index.md: ########## @@ -0,0 +1,129 @@ +--- +title: "Apache Airflow 2.10.0 is here" +linkTitle: "Apache Airflow 2.10.0 is here" +author: "Utkarsh Sharma" +github: "utkarsharma2" +linkedin: "utkarsh-sharma-5791ab8a" +description: "Apache Airflow 2.10.0 is a game-changer, with powerful Dataset improvements and the groundbreaking Hybrid Executor, set to redefine your workflow capabilities!" +tags: [Release] +date: "2024-08-08" +--- + +I'm happy to announce that Apache Airflow 2.10.0 is now available, bringing an array of noteworthy enhancements and new features that will greatly serve our community. + +Apache Airflow 2.10.0 contains over 135 commits, which include 43 new features, 85 improvements, 43 bug fixes, and 26 documentation changes. + +**Details**: + +📦 PyPI: https://pypi.org/project/apache-airflow/2.10.0/ \ +📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.10.0/ \ +🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html \ +🐳 Docker Image: "docker pull apache/airflow:2.10.0" \ +🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.10.0 + + +## Hybrid Execution + +Each executor comes with its unique set of strengths and weaknesses, typically balancing latency, isolation, and compute efficiency. Traditionally, an Airflow environment is limited to a single executor, requiring users to make trade-offs, as no single executor is perfectly suited for all types of tasks. + +We are introducing a new feature that allows for the concurrent use of multiple executors within a single Airflow environment. This flexibility enables users to take advantage of the specific strengths of different executors for various tasks, improving overall efficiency and mitigating weaknesses. Users can set a default executor for the entire environment and, if necessary, assign particular executors to individual DAGs or tasks. + +To configure multiple executors we can pass comma separated list in airflow configuration. The first executor in the list will be the default executor for the environment. + +``` +[core] +executor = 'LocalExecutor,CeleryExecutor' +``` +To make it easier for dag authors, we can also specify aliases for executors that can be specified in the executor configuration +```commandline +[core] +executor = 'LocalExecutor,my.custom.module.ExecutorClass:ShortName' +``` + +DAG authors can specify executors to use at the task +```python +BashOperator( + task_id="hello_world", + executor="LocalExecutor", Review Comment: I would suggest changing the config to something like: `'LocalExecutor,KubernetesExecutor,my.custom.module.ExecutorClass:ShortName'` And then in some examples use `ShortName` and in others use `KubernetesExecutor` so that folks can see the flexibility. ########## landing-pages/site/content/en/blog/airflow-2.10.0/index.md: ########## @@ -0,0 +1,129 @@ +--- +title: "Apache Airflow 2.10.0 is here" +linkTitle: "Apache Airflow 2.10.0 is here" +author: "Utkarsh Sharma" +github: "utkarsharma2" +linkedin: "utkarsh-sharma-5791ab8a" +description: "Apache Airflow 2.10.0 is a game-changer, with powerful Dataset improvements and the groundbreaking Hybrid Executor, set to redefine your workflow capabilities!" +tags: [Release] +date: "2024-08-08" +--- + +I'm happy to announce that Apache Airflow 2.10.0 is now available, bringing an array of noteworthy enhancements and new features that will greatly serve our community. + +Apache Airflow 2.10.0 contains over 135 commits, which include 43 new features, 85 improvements, 43 bug fixes, and 26 documentation changes. + +**Details**: + +📦 PyPI: https://pypi.org/project/apache-airflow/2.10.0/ \ +📚 Docs: https://airflow.apache.org/docs/apache-airflow/2.10.0/ \ +🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html \ +🐳 Docker Image: "docker pull apache/airflow:2.10.0" \ +🚏 Constraints: https://github.com/apache/airflow/tree/constraints-2.10.0 + + +## Hybrid Execution + +Each executor comes with its unique set of strengths and weaknesses, typically balancing latency, isolation, and compute efficiency. Traditionally, an Airflow environment is limited to a single executor, requiring users to make trade-offs, as no single executor is perfectly suited for all types of tasks. + +We are introducing a new feature that allows for the concurrent use of multiple executors within a single Airflow environment. This flexibility enables users to take advantage of the specific strengths of different executors for various tasks, improving overall efficiency and mitigating weaknesses. Users can set a default executor for the entire environment and, if necessary, assign particular executors to individual DAGs or tasks. + +To configure multiple executors we can pass comma separated list in airflow configuration. The first executor in the list will be the default executor for the environment. + +``` +[core] +executor = 'LocalExecutor,CeleryExecutor' +``` +To make it easier for dag authors, we can also specify aliases for executors that can be specified in the executor configuration +```commandline +[core] +executor = 'LocalExecutor,my.custom.module.ExecutorClass:ShortName' +``` + +DAG authors can specify executors to use at the task +```python +BashOperator( + task_id="hello_world", + executor="LocalExecutor", Review Comment: +1 Also LocalExecutor, in this example, is the default executor (first in the list), so it will run any task without a specific executor configured on the task, so specifying it is kind of redundant. If this is copied from our docs we should fix it there as well! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
