jedcunningham commented on code in PR #41184: URL: https://github.com/apache/airflow/pull/41184#discussion_r1700537032
########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Review Comment: ```suggestion ``` This was never released to even be experimental. Its confusing with the following section stating traces have been added. ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) Review Comment: Not a new feature, right? Maybe collapse some inlet stuff into a single entry? ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) +- Send important executor logs to task logs (#40468) +- Enhance User Experience by Opening external UI Links in new Tab on browser. (#40635) +- Attempt to add ReactJSON view to rendered templates (#40639) +- Speeding up regex match time for custom warnings (#40513) +- Refactor DAG.dataset_triggers into the timetable class (#39321) +- Ensure try_number incremented for empty operator (#40426) +- Show TI history for log links, more ti details and mapped tasks (#40375) +- add next_kwargs to StartTriggerArgs (#40376) +- Improve Task TryHistory UI (#40357) +- Improve UI error handling (#40350) +- Remove double warning in CLI when config value is deprecated (#40319) +- AIP 64: Add TI try history to Task Instance Details, Logs, and Gantt chart (#40304) +- Bump ``ws`` from 7.5.5 to 7.5.10 in /airflow/www (#40288) +- Implement XComArg concat() (#40172) +- Added ``get_extra_dejson`` method with nested parameter which allows you to specify if you want the nested json as string to be also deserialized (#39811) +- Add executor field to the task instance API (#40034) +- Support checking for db path absoluteness on Windows (#40069) +- Remove unnecessary nginx redirect rule from reverse proxy documentation (#38953) +- Introduce StartTriggerArgs and prevent start trigger initialization in scheduler (#39585) +- Add task documentation to details tab in grid view. (#39899) +- Allow executors to be specified with only the class name of the Executor (#40131) +- Remove obsolete conditional logic related to try_number (#40104) +- Add listeners for Dag import errors (#39739) Review Comment: New feature ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) Review Comment: ```suggestion - Rotate fernet key in batches to limit memory usage (#40786) ``` or something ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) +- Send important executor logs to task logs (#40468) +- Enhance User Experience by Opening external UI Links in new Tab on browser. (#40635) Review Comment: ```suggestion - Open external links in new tabs (#40635) ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) Review Comment: ```suggestion ``` AIP-61. I'm going to stop suggesting these, but we should do a pass to make sure things for AIPs aren't here and the PRs are labeled appropriately. ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for Review Comment: We should link to the docs. ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) Review Comment: ```suggestion - Use params instead of dag_run.conf in example DAG (#40759) ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) Review Comment: ```suggestion ``` This should be covered by the AIP-44 entry above. ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) Review Comment: ```suggestion - Stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) Review Comment: It might be worth having significant entry for scarf. ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) +- Send important executor logs to task logs (#40468) +- Enhance User Experience by Opening external UI Links in new Tab on browser. (#40635) +- Attempt to add ReactJSON view to rendered templates (#40639) +- Speeding up regex match time for custom warnings (#40513) +- Refactor DAG.dataset_triggers into the timetable class (#39321) +- Ensure try_number incremented for empty operator (#40426) Review Comment: This is a bugfix. ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) +- Send important executor logs to task logs (#40468) +- Enhance User Experience by Opening external UI Links in new Tab on browser. (#40635) +- Attempt to add ReactJSON view to rendered templates (#40639) +- Speeding up regex match time for custom warnings (#40513) +- Refactor DAG.dataset_triggers into the timetable class (#39321) +- Ensure try_number incremented for empty operator (#40426) +- Show TI history for log links, more ti details and mapped tasks (#40375) +- add next_kwargs to StartTriggerArgs (#40376) +- Improve Task TryHistory UI (#40357) +- Improve UI error handling (#40350) +- Remove double warning in CLI when config value is deprecated (#40319) +- AIP 64: Add TI try history to Task Instance Details, Logs, and Gantt chart (#40304) +- Bump ``ws`` from 7.5.5 to 7.5.10 in /airflow/www (#40288) +- Implement XComArg concat() (#40172) +- Added ``get_extra_dejson`` method with nested parameter which allows you to specify if you want the nested json as string to be also deserialized (#39811) +- Add executor field to the task instance API (#40034) +- Support checking for db path absoluteness on Windows (#40069) +- Remove unnecessary nginx redirect rule from reverse proxy documentation (#38953) Review Comment: docs only ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) +- Send important executor logs to task logs (#40468) +- Enhance User Experience by Opening external UI Links in new Tab on browser. (#40635) +- Attempt to add ReactJSON view to rendered templates (#40639) +- Speeding up regex match time for custom warnings (#40513) +- Refactor DAG.dataset_triggers into the timetable class (#39321) +- Ensure try_number incremented for empty operator (#40426) +- Show TI history for log links, more ti details and mapped tasks (#40375) +- add next_kwargs to StartTriggerArgs (#40376) +- Improve Task TryHistory UI (#40357) +- Improve UI error handling (#40350) +- Remove double warning in CLI when config value is deprecated (#40319) +- AIP 64: Add TI try history to Task Instance Details, Logs, and Gantt chart (#40304) +- Bump ``ws`` from 7.5.5 to 7.5.10 in /airflow/www (#40288) +- Implement XComArg concat() (#40172) +- Added ``get_extra_dejson`` method with nested parameter which allows you to specify if you want the nested json as string to be also deserialized (#39811) +- Add executor field to the task instance API (#40034) +- Support checking for db path absoluteness on Windows (#40069) +- Remove unnecessary nginx redirect rule from reverse proxy documentation (#38953) +- Introduce StartTriggerArgs and prevent start trigger initialization in scheduler (#39585) +- Add task documentation to details tab in grid view. (#39899) +- Allow executors to be specified with only the class name of the Executor (#40131) +- Remove obsolete conditional logic related to try_number (#40104) +- Add listeners for Dag import errors (#39739) +- Enable templating in ``extraContainers`` and ``extraInitContainers`` (#38507) +- Allow Task Group Ids to be passed as branches in BranchMixIn (#38883) +- Fix Todo remove hybrid property hack (#39765) +- Javascript connection form will apply CodeMirror to all textarea's dynamically (#39812) +- Improve typing for allowed/failed_states in TriggerDagRunOperator (#39855) +- Determine needs_expansion at time of serialization (#39604) +- Add indexes on dag_id column in referencing tables to speed up deletion of dag records (#39638) +- Add task failed dependencies to details page. (#38449) +- Support failing tasks stuck in queued for hybrid executors (#39624) +- Remove webserver try_number adjustment (#39623) +- Implement slicing in lazy sequence (#39483) +- Unify lazy db sequence implementations (#39426) +- Add ``__getattr__`` to task decorator stub (#39425) +- Allow passing labels to FAB Views registered via Plugins (#39444) +- Simpler error message when trying to offline migrate with sqlite (#39441) +- feat: soft_fail TriggerDagRunOperator (#39173) +- Rename "dataset event" in context to use "outlet" (#39397) +- Resolve ``RemovedIn20Warning`` in ``airflow task`` command (#39244) +- Determine fail_stop on client side when db isolated (#39258) +- Refactor cloudpickle support in Python operators/decorators (#39270) +- Update trigger kwargs migration to specify existing_nullable (#39361) +- Allowing tasks to start execution directly from triggerer without going to worker (#38674) +- Better ``db migrate`` error messages (#39268) +- Add stacklevel into the ``suppress_and_warn`` warning (#39263) +- Add support to search by dag_display_name. (#39008) Review Comment: ```suggestion - Support searching by dag_display_name (#39008) ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) +- Send important executor logs to task logs (#40468) +- Enhance User Experience by Opening external UI Links in new Tab on browser. (#40635) +- Attempt to add ReactJSON view to rendered templates (#40639) +- Speeding up regex match time for custom warnings (#40513) +- Refactor DAG.dataset_triggers into the timetable class (#39321) +- Ensure try_number incremented for empty operator (#40426) +- Show TI history for log links, more ti details and mapped tasks (#40375) +- add next_kwargs to StartTriggerArgs (#40376) +- Improve Task TryHistory UI (#40357) +- Improve UI error handling (#40350) +- Remove double warning in CLI when config value is deprecated (#40319) +- AIP 64: Add TI try history to Task Instance Details, Logs, and Gantt chart (#40304) +- Bump ``ws`` from 7.5.5 to 7.5.10 in /airflow/www (#40288) +- Implement XComArg concat() (#40172) +- Added ``get_extra_dejson`` method with nested parameter which allows you to specify if you want the nested json as string to be also deserialized (#39811) +- Add executor field to the task instance API (#40034) +- Support checking for db path absoluteness on Windows (#40069) +- Remove unnecessary nginx redirect rule from reverse proxy documentation (#38953) +- Introduce StartTriggerArgs and prevent start trigger initialization in scheduler (#39585) +- Add task documentation to details tab in grid view. (#39899) +- Allow executors to be specified with only the class name of the Executor (#40131) +- Remove obsolete conditional logic related to try_number (#40104) +- Add listeners for Dag import errors (#39739) +- Enable templating in ``extraContainers`` and ``extraInitContainers`` (#38507) +- Allow Task Group Ids to be passed as branches in BranchMixIn (#38883) +- Fix Todo remove hybrid property hack (#39765) +- Javascript connection form will apply CodeMirror to all textarea's dynamically (#39812) +- Improve typing for allowed/failed_states in TriggerDagRunOperator (#39855) +- Determine needs_expansion at time of serialization (#39604) +- Add indexes on dag_id column in referencing tables to speed up deletion of dag records (#39638) +- Add task failed dependencies to details page. (#38449) Review Comment: ```suggestion - Add task failed dependencies to details page (#38449) ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) +- Send important executor logs to task logs (#40468) +- Enhance User Experience by Opening external UI Links in new Tab on browser. (#40635) +- Attempt to add ReactJSON view to rendered templates (#40639) +- Speeding up regex match time for custom warnings (#40513) +- Refactor DAG.dataset_triggers into the timetable class (#39321) +- Ensure try_number incremented for empty operator (#40426) +- Show TI history for log links, more ti details and mapped tasks (#40375) +- add next_kwargs to StartTriggerArgs (#40376) +- Improve Task TryHistory UI (#40357) +- Improve UI error handling (#40350) +- Remove double warning in CLI when config value is deprecated (#40319) +- AIP 64: Add TI try history to Task Instance Details, Logs, and Gantt chart (#40304) +- Bump ``ws`` from 7.5.5 to 7.5.10 in /airflow/www (#40288) +- Implement XComArg concat() (#40172) +- Added ``get_extra_dejson`` method with nested parameter which allows you to specify if you want the nested json as string to be also deserialized (#39811) +- Add executor field to the task instance API (#40034) +- Support checking for db path absoluteness on Windows (#40069) +- Remove unnecessary nginx redirect rule from reverse proxy documentation (#38953) +- Introduce StartTriggerArgs and prevent start trigger initialization in scheduler (#39585) +- Add task documentation to details tab in grid view. (#39899) +- Allow executors to be specified with only the class name of the Executor (#40131) +- Remove obsolete conditional logic related to try_number (#40104) +- Add listeners for Dag import errors (#39739) +- Enable templating in ``extraContainers`` and ``extraInitContainers`` (#38507) +- Allow Task Group Ids to be passed as branches in BranchMixIn (#38883) +- Fix Todo remove hybrid property hack (#39765) +- Javascript connection form will apply CodeMirror to all textarea's dynamically (#39812) +- Improve typing for allowed/failed_states in TriggerDagRunOperator (#39855) Review Comment: probably better as misc ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) +- Send important executor logs to task logs (#40468) +- Enhance User Experience by Opening external UI Links in new Tab on browser. (#40635) +- Attempt to add ReactJSON view to rendered templates (#40639) +- Speeding up regex match time for custom warnings (#40513) +- Refactor DAG.dataset_triggers into the timetable class (#39321) +- Ensure try_number incremented for empty operator (#40426) +- Show TI history for log links, more ti details and mapped tasks (#40375) +- add next_kwargs to StartTriggerArgs (#40376) +- Improve Task TryHistory UI (#40357) +- Improve UI error handling (#40350) +- Remove double warning in CLI when config value is deprecated (#40319) +- AIP 64: Add TI try history to Task Instance Details, Logs, and Gantt chart (#40304) +- Bump ``ws`` from 7.5.5 to 7.5.10 in /airflow/www (#40288) Review Comment: Probably should be misc? ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) +- Send important executor logs to task logs (#40468) +- Enhance User Experience by Opening external UI Links in new Tab on browser. (#40635) +- Attempt to add ReactJSON view to rendered templates (#40639) +- Speeding up regex match time for custom warnings (#40513) +- Refactor DAG.dataset_triggers into the timetable class (#39321) +- Ensure try_number incremented for empty operator (#40426) +- Show TI history for log links, more ti details and mapped tasks (#40375) +- add next_kwargs to StartTriggerArgs (#40376) +- Improve Task TryHistory UI (#40357) +- Improve UI error handling (#40350) +- Remove double warning in CLI when config value is deprecated (#40319) +- AIP 64: Add TI try history to Task Instance Details, Logs, and Gantt chart (#40304) +- Bump ``ws`` from 7.5.5 to 7.5.10 in /airflow/www (#40288) +- Implement XComArg concat() (#40172) +- Added ``get_extra_dejson`` method with nested parameter which allows you to specify if you want the nested json as string to be also deserialized (#39811) +- Add executor field to the task instance API (#40034) +- Support checking for db path absoluteness on Windows (#40069) +- Remove unnecessary nginx redirect rule from reverse proxy documentation (#38953) +- Introduce StartTriggerArgs and prevent start trigger initialization in scheduler (#39585) +- Add task documentation to details tab in grid view. (#39899) +- Allow executors to be specified with only the class name of the Executor (#40131) +- Remove obsolete conditional logic related to try_number (#40104) +- Add listeners for Dag import errors (#39739) +- Enable templating in ``extraContainers`` and ``extraInitContainers`` (#38507) Review Comment: ```suggestion ``` This isn't a core PR, it's for the helm chart. ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,310 @@ .. towncrier release notes start +Airflow 2.10.0 (2024-08-12) +--------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Datasets no longer trigger inactive DAGs (#38891) +""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, when a DAG is paused or removed, incoming dataset events would still +trigger it, and the DAG would run when it is unpaused or added back in a DAG +file. This has been changed; a DAG's dataset schedule can now only be satisfied +by events that occur when the DAG is active. While this is a breaking change, +the previous behavior is considered a bug. + +The behavior of time-based scheduling is unchanged, including the timetable part +of ``DatasetOrTimeSchedule``. + +``try_number`` is no longer incremented during task execution (#39336) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Previously, the try number (``try_number``) was incremented at the beginning of task execution on the worker. This was problematic for many reasons. +For one it meant that the try number was incremented when it was not supposed to, namely when resuming from reschedule or deferral. And it also resulted in +the try number being "wrong" when the task had not yet started. The workarounds for these two issues caused a lot of confusion. + +Now, instead, the try number for a task run is determined at the time the task is scheduled, and does not change in flight, and it is never decremented. +So after the task runs, the observed try number remains the same as it was when the task was running; only when there is a "new try" will the try number be incremented again. + +One consequence of this change is, if users were "manually" running tasks (e.g. by calling ``ti.run()`` directly, or command line ``airflow tasks run``), +try number will no longer be incremented. Airflow assumes that tasks are always run after being scheduled by the scheduler, so we do not regard this as a breaking change. + +``/logout`` endpoint in FAB Auth Manager is now CSRF protected (#40145) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +The ``/logout`` endpoint's method in FAB Auth Manager has been changed from ``GET`` to ``POST`` in all existing +AuthViews (``AuthDBView``, ``AuthLDAPView``, ``AuthOAuthView``, ``AuthOIDView``, ``AuthRemoteUserView``), and +now includes CSRF protection to enhance security and prevent unauthorized logouts. + +Support for OpenTelemetry Traces is no longer "Experimental" (#40874) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +OpenTelemetry Traces for Apache Airflow (#37948). +""""""""""""""""""""""""""""""""""""""""""""""""" +This new feature adds capability for Apache Airflow to emit 1) airflow system traces of scheduler, +triggerer, executor, processor 2) DAG run traces for deployed DAG runs in OpenTelemetry format. Previously, only metrics were supported which emitted metrics in OpenTelemetry. +This new feature will add richer data for users to use OpenTelemetry standard to emit and send their trace data to OTLP compatible endpoints. + +Using Multiple Executors Concurrently (#40701) +"""""""""""""""""""""""""""""""""""""""""""""" +Previously known as hybrid executors, this new feature allows Airflow to use multiple executors concurrently. DAGs, or even individual tasks, can be configured +to use a specific executor that suits its needs best. A single DAG can contain tasks all using different executors. Please see the Airflow documentation for +more details. Note: This feature is still experimental. + +New Features +"""""""""""" +- AIP-61 Hybrid Execution (`AIP-61 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3Aarea%3Ahybrid-executors+is%3Aclosed+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-62 Getting Lineage from Hook Instrumentation (`AIP-62 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-62+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-64 TaskInstance Try History (`AIP-64 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-64+milestone%3A%22Airflow+2.10.0%22>`_) +- AIP-44 Internal API (`AIP-44 <https://github.com/apache/airflow/pulls?q=is%3Apr+label%3AAIP-44+milestone%3A%22Airflow+2.10.0%22+is%3Aclosed>`_) +- Retrieve inlet dataset events through dataset aliases (#40809) +- Extend get_datasets endpoint to include dataset aliases (#40830) +- Feature/add token authentication to internal api (#40899) +- Scheduling based on dataset aliases (#40693) +- Add start execution from triggerer support to dynamic task mapping (#39912) +- Add try_number to log table (#40739) +- Added ds_format_locale method in macros which allows localizing datetime formatting using Babel (#40746) +- Link dataset event to dataset alias (#40723) +- Add DatasetAlias to support dynamic Dataset Event Emission and Dataset Creation (#40478) +- Use sentinel to mark dag as removed on re-serialization (#39825) +- Support hybrid executors in backfill jobs (#40472) +- Scheduler job: main loop and event processing for multi executors (#40017) +- Add parameter for the last number of queries to the DB in DAG file processing stats (#40323) +- ADD prototype version dark mode for Airflow UI (#39355) +- Move import to make PythonOperator working on Windows (#40424) +- Dag test without sensor (#40010) +- Handle db isolation for mapped operators and task groups (#39259) +- feature: callable for template_fields (#37028) +- Home: filter running/failed and active/paused dags (#39701) +- Fix typo in example inlet events DAG (#39909) +- Add example DAGs for inlet_events (#39893) +- Add metrics about task CPU and memory usage (#39650) +- UI changes for DAG Re-parsing feature (#39636) +- Add Scarf based telemetry (#39510) +- Add dag re-parsing request endpoint (#39138) +- Task adoption for hybrid executors (#39531) +- Redirect to new DAGRun after trigger from Grid view (#39569) +- Display ``endDate`` in task instance tooltip. (#39547) +- Implement ``accessors`` to read dataset events defined as inlet (#39367) +- Add color to log lines in UI for error and warnings based on keywords (#39006) +- Add Rendered k8s pod spec tab to ti details view (#39141) +- Make audit log before/after filterable (#39120) +- Consolidate grid collapse actions to a single full screen toggle (#39070) +- Implement Metadata to emit runtime extra (#38650) +- Add ``AirflowInternalRuntimeError`` for raise ``non catchable`` errors (#38778) +- Add executor field to the DB and parameter to the operators (#38474) +- Implement context accessor for DatasetEvent extra (#38481) +- Add health endpoint to the RPC server (#38551) +- Add dataset event info to dag graph (#41012) + +Improvements +"""""""""""" +- Update metrics names to allow multiple executors to report metrics (#40778) +- Format DAG run count (#39684) +- Update styles for ``renderedjson`` component (#40964) +- Improve ATTRIBUTE_REMOVED sentinel to use class and more context (#40920) +- Make XCom display as react json (#40640) +- Replace usages of task context logger with the log table (#40867) +- Rollback for all retry exceptions (#40882) (#40883) +- feat: support rendering ObjectStoragePath value (#40638) +- Add try_number and map_index as params for log event endpoint (#40845) +- Fernet-key-rotation-optimization (#40786) +- Add gauge metric for 'last_num_of_db_queries' parameter (#40833) +- Set parallelism log messages to warning level for better visibility (#39298) +- Add error handling for encoding the dag runs (#40222) +- Use params instead of dag_run.conf in example (#40759) +- ``Hybrid-ize`` tasks run_task executor entrypoint (#40762) +- Load Example Plugins with Example DAGs (#39999) +- chore(core): stop deferring TimeDeltaSensorAsync task when the target_dttm is in the past (#40719) +- Send important executor logs to task logs (#40468) +- Enhance User Experience by Opening external UI Links in new Tab on browser. (#40635) +- Attempt to add ReactJSON view to rendered templates (#40639) +- Speeding up regex match time for custom warnings (#40513) +- Refactor DAG.dataset_triggers into the timetable class (#39321) +- Ensure try_number incremented for empty operator (#40426) +- Show TI history for log links, more ti details and mapped tasks (#40375) +- add next_kwargs to StartTriggerArgs (#40376) +- Improve Task TryHistory UI (#40357) +- Improve UI error handling (#40350) +- Remove double warning in CLI when config value is deprecated (#40319) +- AIP 64: Add TI try history to Task Instance Details, Logs, and Gantt chart (#40304) +- Bump ``ws`` from 7.5.5 to 7.5.10 in /airflow/www (#40288) +- Implement XComArg concat() (#40172) +- Added ``get_extra_dejson`` method with nested parameter which allows you to specify if you want the nested json as string to be also deserialized (#39811) +- Add executor field to the task instance API (#40034) +- Support checking for db path absoluteness on Windows (#40069) +- Remove unnecessary nginx redirect rule from reverse proxy documentation (#38953) +- Introduce StartTriggerArgs and prevent start trigger initialization in scheduler (#39585) +- Add task documentation to details tab in grid view. (#39899) Review Comment: ```suggestion - Add task documentation to details tab in grid view (#39899) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
