This is an automated email from the ASF dual-hosted git repository.
ephraimanierobi pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/main by this push:
new d9889f78dd Add blog post for 2.10.0 (#1052)
d9889f78dd is described below
commit d9889f78dd00018a7e1ca1ed5860ac9b9663ea86
Author: Utkarsh Sharma <[email protected]>
AuthorDate: Fri Aug 16 12:22:19 2024 +0530
Add blog post for 2.10.0 (#1052)
* [WIP] Add blog post for 2.10.0
* Update landing-pages/site/content/en/blog/airflow-2.10.0/index.md
* Add Task Instance History
* Add UI changes
* Add dataset details changes
* Update landing-pages/site/content/en/blog/airflow-2.10.0/index.md
Co-authored-by: Niko Oliveira <[email protected]>
* Add code suggestions
* Update post
* Update images
* Update blogpost
* Minor changes
* style: replace tab with spaces
---------
Co-authored-by: Niko Oliveira <[email protected]>
Co-authored-by: Wei Lee <[email protected]>
---
.../airflow-2.10.0/DAG_reparse_button_detail.png | Bin 0 -> 16792 bytes
.../airflow-2.10.0/DAG_reparsing_button_list.png | Bin 0 -> 22343 bytes
.../en/blog/airflow-2.10.0/airflow_dark_mode.png | Bin 0 -> 1477678 bytes
.../en/blog/airflow-2.10.0/airflow_light_mode.png | Bin 0 -> 1392945 bytes
.../airflow-2.10.0/dag_dependencies_legend.png | Bin 0 -> 100011 bytes
.../en/blog/airflow-2.10.0/dataset_details.png | Bin 0 -> 234016 bytes
.../en/blog/airflow-2.10.0/dataset_info.png | Bin 0 -> 281014 bytes
.../en/blog/airflow-2.10.0/dataset_list.png | Bin 0 -> 347612 bytes
.../en/blog/airflow-2.10.0/dataset_toggle_off.png | Bin 0 -> 118831 bytes
.../en/blog/airflow-2.10.0/dataset_toggle_on.png | Bin 0 -> 150428 bytes
.../en/blog/airflow-2.10.0/dependency_graph.png | Bin 0 -> 844792 bytes
.../site/content/en/blog/airflow-2.10.0/index.md | 190 +++++++++++++++++++++
.../airflow-2.10.0/render_dataset_conditions.png | Bin 0 -> 119992 bytes
.../blog/airflow-2.10.0/task_instance_history.png | Bin 0 -> 118727 bytes
.../airflow-2.10.0/task_instance_history_log.png | Bin 0 -> 156398 bytes
15 files changed, 190 insertions(+)
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/DAG_reparse_button_detail.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/DAG_reparse_button_detail.png
new file mode 100644
index 0000000000..2241271ed3
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/DAG_reparse_button_detail.png
differ
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/DAG_reparsing_button_list.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/DAG_reparsing_button_list.png
new file mode 100644
index 0000000000..06c1dc98cf
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/DAG_reparsing_button_list.png
differ
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/airflow_dark_mode.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/airflow_dark_mode.png
new file mode 100644
index 0000000000..84affe1dcd
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/airflow_dark_mode.png differ
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/airflow_light_mode.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/airflow_light_mode.png
new file mode 100644
index 0000000000..b3dd4dbcb3
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/airflow_light_mode.png
differ
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/dag_dependencies_legend.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/dag_dependencies_legend.png
new file mode 100644
index 0000000000..256ab7df82
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/dag_dependencies_legend.png
differ
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_details.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_details.png
new file mode 100644
index 0000000000..274159cedf
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_details.png differ
diff --git a/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_info.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_info.png
new file mode 100644
index 0000000000..d7cb19e426
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_info.png differ
diff --git a/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_list.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_list.png
new file mode 100644
index 0000000000..20f3675773
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_list.png differ
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_toggle_off.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_toggle_off.png
new file mode 100644
index 0000000000..3830df66b8
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_toggle_off.png
differ
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_toggle_on.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_toggle_on.png
new file mode 100644
index 0000000000..550602f512
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/dataset_toggle_on.png differ
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/dependency_graph.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/dependency_graph.png
new file mode 100644
index 0000000000..c8048c1339
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/dependency_graph.png differ
diff --git a/landing-pages/site/content/en/blog/airflow-2.10.0/index.md
b/landing-pages/site/content/en/blog/airflow-2.10.0/index.md
new file mode 100644
index 0000000000..befc121f96
--- /dev/null
+++ b/landing-pages/site/content/en/blog/airflow-2.10.0/index.md
@@ -0,0 +1,190 @@
+---
+title: "Apache Airflow 2.10.0 is here"
+linkTitle: "Apache Airflow 2.10.0 is here"
+author: "Utkarsh Sharma"
+github: "utkarsharma2"
+linkedin: "utkarsh-sharma-5791ab8a"
+description: "Apache Airflow 2.10.0 is a game-changer, with powerful Dataset
improvements and the groundbreaking Hybrid Executor, set to redefine your
workflow capabilities!"
+tags: [Release]
+date: "2024-08-08"
+---
+
+I'm happy to announce that Apache Airflow 2.10.0 is now available, bringing an
array of noteworthy enhancements and new features that will greatly serve our
community.
+
+**Details**:
+
+📦 PyPI: <https://pypi.org/project/apache-airflow/2.10.0/> \
+📚 Docs: <https://airflow.apache.org/docs/apache-airflow/2.10.0/> \
+🛠 Release Notes:
<https://airflow.apache.org/docs/apache-airflow/2.10.0/release_notes.html> \
+🐳 Docker Image: "docker pull apache/airflow:2.10.0" \
+🚏 Constraints: <https://github.com/apache/airflow/tree/constraints-2.10.0>
+
+## Multiple Executor Configuration (formerly "Hybrid Execution")
+
+Each executor comes with its unique set of strengths and weaknesses, typically
balancing latency, isolation, and compute efficiency. Traditionally, an Airflow
environment is limited to a single executor, requiring users to make
trade-offs, as no single executor is perfectly suited for all types of tasks.
+
+We are introducing a new feature that allows for the concurrent use of
multiple executors within a single Airflow environment. This flexibility
enables users to take advantage of the specific strengths of different
executors for various tasks, improving overall efficiency and mitigating
weaknesses. Users can set a default executor for the entire environment and, if
necessary, assign particular executors to individual DAGs or tasks.
+
+To configure multiple executors we can pass comma separated list in airflow
configuration. The first executor in the list will be the default executor for
the environment.
+
+```
+[core]
+executor = 'LocalExecutor,CeleryExecutor'
+```
+
+To make it easier for dag authors, we can also specify aliases for executors
that can be specified in the executor configuration
+
+```commandline
+[core]
+executor =
'LocalExecutor,KubernetesExecutor,my.custom.module.ExecutorClass:ShortName'
+```
+
+DAG authors can specify executors to use at the task
+
+```python
+BashOperator(
+ task_id="hello_world",
+ executor="ShortName",
+ bash_command="echo 'hello world!'",
+)
+
+@task(executor="KubernetesExecutor")
+def hello_world():
+ print("hello world!")
+```
+
+We can also specify executors on the DAG level
+
+```python
+def hello_world():
+ print("hello world!")
+
+def hello_world_again():
+ print("hello world again!")
+
+with DAG(
+ dag_id="hello_worlds",
+ default_args={"executor": "ShortName"}, # Applies to all tasks in the DAG
+) as dag:
+ # All tasks will use the executor from default args automatically
+ hw = hello_world()
+ hw_again = hello_world_again()
+```
+
+## Dynamic Dataset scheduling through DatasetAlias
+
+Airflow 2.10 comes with `DatasetAlias` class which can be passed as a value in
the `outlets`, `inlets` on a task, and `schedule` on a DAG. An instance of
`DatasetAlias` is resolved dynamically to a real dataset. Downstream can depend
on either the resolved dataset or on an alias itself.
+
+`DatasetAlias` has one argument `name` that uniquely identifies the dataset.
The task must first declare the alias as an outlet, and use `outlet_events` or
`yield Metadata` to add events to it.
+
+### Emit a dataset event during task execution through outlet_events
+
+```python
+from airflow.datasets import DatasetAlias
+
+@task(outlets=[DatasetAlias("my-task-outputs")])
+def my_task_with_outlet_events(*, outlet_events):
+ outlet_events["my-task-outputs"].add(Dataset("s3://bucket/my-task"))
+```
+
+### Emit a dataset event during task execution by yielding Metadata
+
+```python
+from airflow.datasets.metadata import Metadata
+
+@task(outlets=[DatasetAlias("my-task-outputs")])
+def my_task_with_metadata():
+ s3_dataset = Dataset("s3://bucket/my-task}")
+ yield Metadata(s3_dataset, alias="my-task-outputs")
+```
+
+There are two options for scheduling based on dataset aliases. Schedule based
on `DatasetAlias` or real datasets.
+
+```python
+with DAG(dag_id="dataset-alias-producer"):
+ @task(outlets=[DatasetAlias("example-alias")])
+ def produce_dataset_events(*, outlet_events):
+ outlet_events["example-alias"].add(Dataset("s3://bucket/my-task"))
+
+with DAG(dag_id="dataset-consumer", schedule=Dataset("s3://bucket/my-task")):
+ ...
+
+with DAG(dag_id="dataset-alias-consumer",
schedule=DatasetAlias("example-alias")):
+ ...
+```
+
+### Dataset Aliases UI Enhancements
+
+Now users can see Dataset Aliases in legend of each cross-dag dependency graph
with a corresponded icon/color.
+
+
+
+## Dark Mode for Airflow UI
+
+Airflow 2.10 comes with new Dark Mode feature which is designed to enhance
user experience by offering an alternative visual theme that is easier on the
eyes, especially in low-light conditions. You can toggle the crescent icon on
the right side of the navigation bar to switch between light and dark mode.
+
+
+
+
+
+## Task Instance History
+
+In Apache Airflow 2.10.0, when a task instance is retried or cleared, its
execution history is maintained. You can view this history by clicking on the
task instance in the Grid view, allowing you to access information about each
attempt, such as logs, execution durations, and any failures. This feature
improves transparency into the task's execution process, making it easier to
troubleshoot and analyze your DAGs.
+
+
+
+The history displays the final values of the task instance attributes for each
specific run. On the log page, you can also access the logs for each attempt of
the task instance. This information is valuable for debugging purposes.
+
+
+
+## Dataset UI Enhancements
+
+The dataset page has been revamped to include a focused dataset events section
with additional details such as extras, consuming DAGs, and producing tasks.
+
+
+We now have separate dependency graph and dataset list pages in new tabs,
enhancing the user experience.
+
+
+
+Dataset events are now displayed in both the Details tab of each DAG run and
within the DAG graph.
+
+
+
+### Toggle datasets in Graph
+
+We can now toggle the datasets in the DAG graph
+
+
+
+
+### Dataset Conditions in DAG Graph view
+
+We now display the graph view with logical gates. Datasets with actual events
are highlighted with a different border, making it easier to see what triggered
the selected run.
+
+
+
+### Dataset event info in DAG Graph
+
+For a DAG run, users can now view the dataset events connected to it directly
in the graph view.
+
+
+
+## On-demand DAG Re-parsing
+
+In 2.10 users can now reparse the DAGs on demand using below button on DAG
list and DAG detail pages
+
+
+
+
+## Additional new features
+
+Here are just a few interesting new features since there are too many to list
in full:
+
+* Deferrable operators can now execute directly from the triggerer without
needing to go through the worker. This is especially efficient for certain
operators, like sensors, and can help teams save both time and money.
+* Crucial executor logs are now integrated into the task logs. If the executor
fails to start a task, the relevant error messages will be available in the
task logs, simplifying the debugging process.
+
+## Contributors
+
+Thanks to everyone who contributed to this release, including Andrey Anshin,
Brent Bovenzi, Daniel Standish, Ephraim Anierobi, Hussein Awala, Jarek Potiuk,
Jed Cunningham, Jens Scheffler, Tzu-ping Chung, Vincent, and over 63 others!
+
+I hope you enjoy using Apache Airflow 2.10.0!
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/render_dataset_conditions.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/render_dataset_conditions.png
new file mode 100644
index 0000000000..f5abe8466d
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/render_dataset_conditions.png
differ
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/task_instance_history.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/task_instance_history.png
new file mode 100644
index 0000000000..d1dcf27a5e
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/task_instance_history.png
differ
diff --git
a/landing-pages/site/content/en/blog/airflow-2.10.0/task_instance_history_log.png
b/landing-pages/site/content/en/blog/airflow-2.10.0/task_instance_history_log.png
new file mode 100644
index 0000000000..c1d83f3ed3
Binary files /dev/null and
b/landing-pages/site/content/en/blog/airflow-2.10.0/task_instance_history_log.png
differ
