This is an automated email from the ASF dual-hosted git repository. kaxilnaik pushed a commit to branch v3-1-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 8636e33c69917e670b053e5fa06b8a77c3b15e93 Author: Kaxil Naik <[email protected]> AuthorDate: Tue Sep 16 20:08:09 2025 +0100 Update 3.1.0 release notes based on latest cherry-picks --- RELEASE_NOTES.rst | 141 +++++++++++++++++++++-- airflow-core/newsfragments/46929.bugfix.rst | 1 - airflow-core/newsfragments/49779.significant.rst | 20 ---- airflow-core/newsfragments/50374.feature.rst | 1 - airflow-core/newsfragments/50693.significant.rst | 41 ------- airflow-core/newsfragments/51424.significant.rst | 17 --- airflow-core/newsfragments/51639.significant.rst | 17 --- airflow-core/newsfragments/52651.significant.rst | 53 --------- airflow-core/newsfragments/52860.significant.rst | 17 --- airflow-core/newsfragments/53631.misc.rst | 1 - airflow-core/newsfragments/53727.feature.rst | 1 - airflow-core/newsfragments/53796.misc.rst | 1 - airflow-core/newsfragments/54145.bugfix.rst | 1 - airflow-core/newsfragments/54857.significant.rst | 14 --- airflow-core/newsfragments/55508.feature.rst | 1 - airflow-core/newsfragments/55707.significant.rst | 23 ---- reproducible_build.yaml | 4 +- 17 files changed, 135 insertions(+), 219 deletions(-) diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst index c2c375242e6..ba6da524157 100644 --- a/RELEASE_NOTES.rst +++ b/RELEASE_NOTES.rst @@ -85,9 +85,14 @@ Example use cases: - Notify stakeholders 30 minutes before a critical deadline - Escalate when resource-constrained DAGs remain queued too long +**Current Limitations**: Deadline Alerts currently support only asynchronous callbacks (``AsyncCallback``). +Support for synchronous callbacks (``SyncCallback``) is planned for a future release. + For configuration details and examples, see :doc:`/howto/deadline-alerts`. -**Warning**: Deadline Alerts are experimental in 3.1 and may change in future versions based on user feedback. +.. warning:: + + Deadline Alerts are experimental in 3.1 and may change in future versions based on user feedback. UI Internationalization """"""""""""""""""""""" @@ -145,9 +150,20 @@ For more details and examples, see :doc:`/howto/custom-view-plugin`. Enhanced UI Views and Filtering """""""""""""""""""""""""""""""" -Airflow 3.1 brings Calendar and Gantt chart views to the modern React UI, along with comprehensive filtering -capabilities. The Calendar and Gantt views from Airflow 2.x have been rebuilt for the modern React UI, -along with enhanced filtering capabilities across all views. +Airflow 3.1 brings significant UI improvements including rebuilt Calendar and Gantt chart views for the modern React UI, +comprehensive filtering capabilities, and a refreshed visual design system. + +**Visual Design Improvements** + +The UI now features an updated color palette leveraging Chakra UI semantic tokens, providing better consistency, +accessibility, and theme support across the interface. This modernization improves readability and creates +a more cohesive visual experience throughout Airflow. + +**Rebuilt Views and Enhanced Filtering** + +The Calendar and Gantt views from Airflow 2.x have been rebuilt for the modern React UI, along with enhanced +filtering capabilities across all views. These improvements provide better performance and a more consistent +user experience with the rest of the modern Airflow interface. Inference Execution (Synchronous DAGs) """""""""""""""""""""""""""""""""""""" @@ -190,6 +206,112 @@ Python 3.13 support added & 3.9 support removed Support for Python 3.9 has been removed, as it has reached end-of-life. Airflow 3.1.0 requires Python 3.10, 3.11, 3.12 or 3.13. +Configuration Changes and Cleanup +"""""""""""""""""""""""""""""""""" + +**Webserver Configuration Reorganization** + +Several webserver configuration options have been moved to the ``api`` section for better organization: + +- ``[webserver] log_fetch_timeout_sec`` → ``[api] log_fetch_timeout_sec`` +- ``[webserver] hide_paused_dags_by_default`` → ``[api] hide_paused_dags_by_default`` +- ``[webserver] page_size`` → ``[api] page_size`` +- ``[webserver] default_wrap`` → ``[api] default_wrap`` +- ``[webserver] require_confirmation_dag_change`` → ``[api] require_confirmation_dag_change`` +- ``[webserver] auto_refresh_interval`` → ``[api] auto_refresh_interval`` + +Unused configuration options have been removed: + +- ``[webserver] instance_name_has_markup`` +- ``[webserver] warn_deployment_exposure`` + +**API Server Logging Configuration** + +The API server configuration option ``[api] access_logfile`` has been replaced with ``[api] log_config`` to align with uvicorn's logging configuration. The new option accepts a path to a logging configuration file compatible with ``logging.config.fileConfig``, providing more flexible logging configuration. + +**Security Improvement: XCom Deserialization** + +The ``enable_xcom_deserialize_support`` configuration option has been removed as a security improvement. This option previously allowed deserializing unknown objects in the API, which posed a security risk due to potential remote code execution vulnerabilities when deserializing arbitrary Python objects. + +The XCom display improvements now handle showing non-native XComs (like custom objects, Assets, datetime objects) in a human-readable way through safer methods that don't require deserializing unknown objects in the API server. This provides better user experience when viewing XCom data in the Airflow UI while eliminating the security risk. + +API Changes +""""""""""" + +**Asset API Key Rename** + +The ``consuming_dags`` key in asset API responses has been renamed to ``scheduled_dags`` to better reflect its purpose. This key contains only DAGs that use the asset in their ``schedule`` argument, not all DAGs that technically use the asset. + +Task SDK Interface Changes +"""""""""""""""""""""""""" + +**Removed Functions** + +The following functions have been removed from the task-sdk (``airflow.sdk.definitions.taskgroup``) and moved to server-side API services: + +- ``get_task_group_children_getter`` +- ``task_group_to_dict`` + +These functions are now internal to Airflow's API layer and should not be imported directly by users. + +Reduce default API server workers to 1 +"""""""""""""""""""""""""""""""""""""" + +The default number of API server workers (``[api] workers``) has been reduced from ``4`` to ``1``. + +With FastAPI, sync code runs in external thread pools, making multiple workers within a single +process less necessary. Additionally, with uvicorn's spawn behavior instead of fork, there is +no shared copy-on-write memory between workers, so horizontal scaling with multiple API server +instances is now the recommended approach for better resource utilization and fault isolation. + +A good starting point for the number of workers is to set it to the number of CPU cores available. +If you do have multiple CPU cores available for the API server, consider deploying multiple API +server instances instead of increasing the number of workers. + +Airflow now uses `structlog <https://www.structlog.org/en/stable/>`_ everywhere +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + +Most users should not notice the difference, but it is now possible to emit structured log key/value pairs from tasks. + +If your class subclasses ``LoggingMixin`` (which all ``BaseHook`` and ``BaseOperator`` do -- i.e. all hooks and +operators) then ``self.log`` is now a `<structlog logger>`_. + +The advantage of using structured logging is that it is much easier to find specific information about log +message, especially when using a central store such as ``OpenSearch``/``Elastic``/``Splunk`` etc. +You don't have to make any changes, but you can now take advantage of this. + +.. code-block:: python + + # Inside a Task/Hook etc. + + # Before: + # self.log.info("Registering adapter %r", item.name) + # Now: + self.log.info("Registering adapter", name=item.name) + +This will produce a log that (in the UI) will look something like this:: + + [2025-09-16 10:36:13] INFO - Registering adapter name="adapter1" + +or in JSON (i.e. the log files on disk):: + + {"timestamp": "2025-09-16T10:36:13Z", "log_level": "info", "event": "Registering adapter", "name": "adapter1"} + +You can also use ``structlog`` loggers at the top level of modules etc, and ``stdlib`` both continue to work: + +.. code-block:: python + + import logging + import structlog + + log1 = logging.getLogger(__name__) + log2 = strcutlog.get_logger(__name__) + + log1.info("Loading something from %s", __name__) + log2.info("Loading something", source=__name__) + +(You can't add arbitrary key/value pairs to ``stdlib``, but the normal ``percent-formatter`` approaches still work fine.) + New Features ^^^^^^^^^^^^ @@ -226,7 +348,7 @@ New Features - Implement deadline alert system for proactive DAG monitoring (AIP-86) (#53951, #53903, #53201, #55086) - Add configurable reference points and notification callbacks (#50677, #50093) - Add deadline calculation and tracking in DAG execution lifecycle (#51638, #50925) -- Add comprehensive UI translation support for 14 languages (#51266, #51038, #51219, #50929, #50981, #51793) +- Add comprehensive UI translation support for 16 languages (#51266, #51038, #51219, #50929, #50981, #51793 and more) - Add right-to-left (RTL) layout support for Arabic and Hebrew (#51376) - Add language selection interface and browser preference detection (#51369) - Add translation completeness validation and automated checks (#51166, #51131) @@ -287,6 +409,7 @@ New Features - Switch all airflow logging to structlog (#52651, #55434, #55431, #55638) - Add Filter Bar to Audit Log (#55487) - Add Filters UI for Asset View (#54640) +- Update color palette and leverage Chakra semantic tokens (#53981, #55739) Bug Fixes ^^^^^^^^^ @@ -342,7 +465,9 @@ Bug Fixes - Fix incorrect log timestamps in UI when ``default_timezone`` is not UTC (#54431) - Fix handling of priority_weight for DAG processor callbacks (#55436) - Fix pointless requests from Gantt view when there is no Run ID (#55668) -- Ensure filename and lineno of logger calls are present in Task Logs (#55581) +- Ensure filename and ``lineno`` of logger calls are present in Task Logs (#55581) +- Fix DAG disappearing after callback execution in stale detection (#55698) +- Fix DB downgrade to Airflow 2 when fab tables exists (#55738) Miscellaneous ^^^^^^^^^^^^^ @@ -366,13 +491,13 @@ Miscellaneous - Add guards for registering middlewares from plugins (#55399) - Optimize Gantt group expansion with de-bouncing and deferred rendering (#55334) - Differentiate between triggers and watchers currently running for better visibility (#55376) -- Update color palette and leverage Chakra semantic tokens (#53981) - Removed unused config: ``dag_stale_not_seen_duration`` (#55601, #55684) - Update UI's query client strategy for improved performance (#55528) - Unify datetime format across the UI for consistency (#55572) - Mark React Apps as Experimental for Airflow 3.1 release (#55478) - Improve OOM error messaging for clearer task failure diagnosis (#55602) - Display responder username for better audit trail in HITL workflows (#55509) +- The constraint file do not contain developer dependencies anymore (#53631) Doc Only Changes ^^^^^^^^^^^^^^^^ @@ -382,7 +507,7 @@ Doc Only Changes - Make term Dag consistent in docs task-sdk (#55100) - Add DAG bundles triggerer limitation documentation (#55232) - Add deadline alerts usage guides and best practices (#53727) -- Remove ``--preview `` flag from ``ruff check`` instructions for Airflow 3 upgrade path (#55516) +- Remove ``--preview`` flag from ``ruff check`` instructions for Airflow 3 upgrade path (#55516) - Add documentation for context parameter (#55377) Airflow 3.0.6 (2025-08-29) diff --git a/airflow-core/newsfragments/46929.bugfix.rst b/airflow-core/newsfragments/46929.bugfix.rst deleted file mode 100644 index bff83a3807d..00000000000 --- a/airflow-core/newsfragments/46929.bugfix.rst +++ /dev/null @@ -1 +0,0 @@ -Added validation in XCom.set() to disallow None or empty string keys; XCom.get() still allows None as key but disallows empty strings. diff --git a/airflow-core/newsfragments/49779.significant.rst b/airflow-core/newsfragments/49779.significant.rst deleted file mode 100644 index 187f6ac1991..00000000000 --- a/airflow-core/newsfragments/49779.significant.rst +++ /dev/null @@ -1,20 +0,0 @@ -SecretCache class has been moved to ``airflow.sdk.execution_time.cache`` from ``airflow.secrets.cache`` - -* Types of change - - * [ ] Dag changes - * [ ] Config changes - * [ ] API changes - * [ ] CLI changes - * [ ] Behaviour changes - * [ ] Plugin changes - * [ ] Dependency changes - * [x] Code interface changes - -* Migration rules needed - - * ruff - - * AIR301 - - * [ ] ``airflow.secrets.cache.SecretCache`` → ``airflow.sdk.execution_time.cache.SecretCache`` diff --git a/airflow-core/newsfragments/50374.feature.rst b/airflow-core/newsfragments/50374.feature.rst deleted file mode 100644 index 85e4dc165e9..00000000000 --- a/airflow-core/newsfragments/50374.feature.rst +++ /dev/null @@ -1 +0,0 @@ -When a Dag specifies ``schedule="@once"`` without an explicit ``start_date``, run it as soon as convenient. (Previously, the Dag would never run.) diff --git a/airflow-core/newsfragments/50693.significant.rst b/airflow-core/newsfragments/50693.significant.rst deleted file mode 100644 index 6368ddf0ee5..00000000000 --- a/airflow-core/newsfragments/50693.significant.rst +++ /dev/null @@ -1,41 +0,0 @@ -Unused webserver configuration options have been removed - -The following webserver options were moved into the ``api`` section: - -* ``[webserver] log_fetch_timeout_sec`` → ``[api] log_fetch_timeout_sec`` -* ``[webserver] hide_paused_dags_by_default`` → ``[api] hide_paused_dags_by_default`` -* ``[webserver] page_size`` → ``[api] page_size`` -* ``[webserver] default_wrap`` → ``[api] default_wrap`` -* ``[webserver] require_confirmation_dag_change`` → ``[api] require_confirmation_dag_change`` -* ``[webserver] auto_refresh_interval`` → ``[api] auto_refresh_interval`` - -The following configuration options are now unused and have been removed: - -- ``[webserver] instance_name_has_markup`` -- ``[webserver] warn_deployment_exposure`` - -* Types of change - - * [ ] Dag changes - * [x] Config changes - * [ ] API changes - * [ ] CLI changes - * [ ] Behaviour changes - * [ ] Plugin changes - * [ ] Dependency changes - * [ ] Code interface changes - -.. List the migration rules needed for this change (see https://github.com/apache/airflow/issues/41641) - -* Migration rules needed - - * ``airflow config lint`` - - * [ ] Remove configuration option ``[webserver] instance_name_has_markup`` - * [ ] Remove configuration option ``[webserver] warn_deployment_exposure`` - * [ ] [webserver] log_fetch_timeout_sec`` → ``[api] log_fetch_timeout_sec`` - * [ ] [webserver] hide_paused_dags_by_default`` → ``[api] hide_paused_dags_by_default`` - * [ ] [webserver] page_size`` → ``[api] page_size`` - * [ ] [webserver] default_wrap`` → ``[api] default_wrap`` - * [ ] [webserver] require_confirmation_dag_change`` → ``[api] require_confirmation_dag_change`` - * [ ] [webserver] auto_refresh_interval`` → ``[api] auto_refresh_interval`` diff --git a/airflow-core/newsfragments/51424.significant.rst b/airflow-core/newsfragments/51424.significant.rst deleted file mode 100644 index a455a28eb1c..00000000000 --- a/airflow-core/newsfragments/51424.significant.rst +++ /dev/null @@ -1,17 +0,0 @@ -The ``consuming_dags`` key in asset API has been renamed to ``scheduled_dags``. - -The previous name caused confusion to users since the list does not contain all -Dags that technically *use* the asset, but only those that use it in their -``schedule`` argument. As a bug fix, the key has been renamed to clarify its -intention. - -* Types of change - - * [ ] Dag changes - * [ ] Config changes - * [x] API changes - * [ ] CLI changes - * [ ] Behaviour changes - * [ ] Plugin changes - * [ ] Dependency changes - * [ ] Code interface changes diff --git a/airflow-core/newsfragments/51639.significant.rst b/airflow-core/newsfragments/51639.significant.rst deleted file mode 100644 index 78708b4eb19..00000000000 --- a/airflow-core/newsfragments/51639.significant.rst +++ /dev/null @@ -1,17 +0,0 @@ -``enable_xcom_deserialize_support`` configuration option has been removed. - -This configuration was previously marked as a security risk due to potential remote code execution vulnerabilities -when deserializing arbitrary Python objects that came in from XComs. The removal is a security improvement since -all custom XCom serialization/deserialization is now handled safely at the worker level, making this configuration -unnecessary in core. Users should migrate to not setting this configuration. - -* Types of change - - * [ ] Dag changes - * [x] Config changes - * [ ] API changes - * [ ] CLI changes - * [ ] Behaviour changes - * [ ] Plugin changes - * [ ] Dependency changes - * [ ] Code interface changes diff --git a/airflow-core/newsfragments/52651.significant.rst b/airflow-core/newsfragments/52651.significant.rst deleted file mode 100644 index d71834af938..00000000000 --- a/airflow-core/newsfragments/52651.significant.rst +++ /dev/null @@ -1,53 +0,0 @@ -Airflow now uses `https://www.structlog.org/en/stable/ <structlog>`_ everywhere. - -Most users should not notice the difference, but it is now possible to emit structured log key/value pairs from tasks. - -If your class subclasses LoggingMixin (which all BaseHook and BaseOperator do -- i.e. all hooks and operators) then ``self.log`` is now a ` <structlog logger>`_. - -The advantage of using structured logging is that it is much easier to find specific information about log message, especially when using a central store such as OpenSearch/Elastic/Splunk etc. You don't have to make any changes, but you can now take advantage of this. - -.. code-block:: python - - # Inside a Task/Hook etc. - - # Before: - # self.log.info("Registering adapter %r", item.name) - # Now: - self.log.info("Registering adapter", name=item.name) - -This will produce a log that (in the UI) will look something like this:: - - [2025-09-16 10:36:13] INFO - Registering adapter name="adapter1" - - -or in JSON (i.e. the log files on disk):: - - {"timestamp": "2025-09-16T10:36:13Z", "log_level": "info", "event": "Registering adapter", "name": "adapter1"} - - -You can also use structlog loggers at the top level of modules etc, and stdlib both continue to work: - -.. code-block:: python - - import logging - import structlog - - log1 = logging.getLogger(__name__) - log2 = strcutlog.get_logger(__name__) - - log1.info("Loading something from %s", __name__) - log2.info("Loading something", source=__name__) - -(You can't add arbitrary key/value pairs to stdlib, but the normal percent-formatter approaches still work fine.) - - -* Types of change - - * [ ] Dag changes - * [ ] Config changes - * [x] API changes - * [ ] CLI changes - * [ ] Behaviour changes - * [ ] Plugin changes - * [ ] Dependency changes - * [ ] Code interface changes diff --git a/airflow-core/newsfragments/52860.significant.rst b/airflow-core/newsfragments/52860.significant.rst deleted file mode 100644 index 5962897ec20..00000000000 --- a/airflow-core/newsfragments/52860.significant.rst +++ /dev/null @@ -1,17 +0,0 @@ -Replace API server ``access_logfile`` configuration with ``log_config`` - -The API server configuration option ``[api] access_logfile`` has been replaced with ``[api] log_config`` to align with uvicorn's logging configuration instead of the legacy gunicorn approach. -The new ``log_config`` option accepts a path to a logging configuration file compatible with ``logging.config.fileConfig``, providing more flexible logging configuration for the API server. - -This change also removes the dependency on gunicorn for daemonization, making the API server ``--daemon`` option consistent with other Airflow components like scheduler and triggerer. - -* Types of change - - * [ ] Dag changes - * [x] Config changes - * [ ] API changes - * [ ] CLI changes - * [ ] Behaviour changes - * [ ] Plugin changes - * [ ] Dependency changes - * [ ] Code interface changes diff --git a/airflow-core/newsfragments/53631.misc.rst b/airflow-core/newsfragments/53631.misc.rst deleted file mode 100644 index a6cef457b58..00000000000 --- a/airflow-core/newsfragments/53631.misc.rst +++ /dev/null @@ -1 +0,0 @@ -The constraint do not contain developer dependencies as of Airflow 3.1.0 diff --git a/airflow-core/newsfragments/53727.feature.rst b/airflow-core/newsfragments/53727.feature.rst deleted file mode 100644 index 90272000d1d..00000000000 --- a/airflow-core/newsfragments/53727.feature.rst +++ /dev/null @@ -1 +0,0 @@ -Add Airflow Deadlines feature in Airflow 3.1. See https://airflow.apache.org/docs/apache-airflow/stable/howto/deadline-alerts.html. diff --git a/airflow-core/newsfragments/53796.misc.rst b/airflow-core/newsfragments/53796.misc.rst deleted file mode 100644 index a582154dc3a..00000000000 --- a/airflow-core/newsfragments/53796.misc.rst +++ /dev/null @@ -1 +0,0 @@ -Add ``RemovedInAirflow4Warning`` warnings for ``airflow.security.permissions`` imports and ``access_control`` Dag attribute usage. diff --git a/airflow-core/newsfragments/54145.bugfix.rst b/airflow-core/newsfragments/54145.bugfix.rst deleted file mode 100644 index 2c30aae8972..00000000000 --- a/airflow-core/newsfragments/54145.bugfix.rst +++ /dev/null @@ -1 +0,0 @@ -Filename and line number of logging calls is once again visible in Task logs (#54145) diff --git a/airflow-core/newsfragments/54857.significant.rst b/airflow-core/newsfragments/54857.significant.rst deleted file mode 100644 index 84ba878db59..00000000000 --- a/airflow-core/newsfragments/54857.significant.rst +++ /dev/null @@ -1,14 +0,0 @@ -Remove ``get_task_group_children_getter`` and ``task_group_to_dict`` from task-sdk - -The ``get_task_group_children_getter`` and ``task_group_to_dict`` functions have been removed from the task-sdk (``airflow.sdk.definitions.taskgroup``) and moved to server-side API services. These functions are now internal to Airflow's API layer and should not be imported directly by users. - -* Types of change - - * [ ] Dag changes - * [ ] Config changes - * [ ] API changes - * [ ] CLI changes - * [ ] Behaviour changes - * [ ] Plugin changes - * [ ] Dependency changes - * [x] Code interface changes diff --git a/airflow-core/newsfragments/55508.feature.rst b/airflow-core/newsfragments/55508.feature.rst deleted file mode 100644 index 47285553ba2..00000000000 --- a/airflow-core/newsfragments/55508.feature.rst +++ /dev/null @@ -1 +0,0 @@ -Output on stdout/stderr from within tasks is now filterable in the Sources list in the UI log view diff --git a/airflow-core/newsfragments/55707.significant.rst b/airflow-core/newsfragments/55707.significant.rst deleted file mode 100644 index 07a42ba3b93..00000000000 --- a/airflow-core/newsfragments/55707.significant.rst +++ /dev/null @@ -1,23 +0,0 @@ -Reduce default API server workers to 1 - -The default number of API server workers (``[api] workers``) has been reduced from 4 to 1. - -With FastAPI, sync code runs in external thread pools, making multiple workers within a single -process less necessary. Additionally, with uvicorn's spawn behavior instead of fork, there is -no shared copy-on-write memory between workers, so horizontal scaling with multiple API server -instances is now the recommended approach for better resource utilization and fault isolation. - -A good starting point for the number of workers is to set it to the number of CPU cores available. -If you do have multiple CPU cores available for the API server, consider deploying multiple API -server instances instead of increasing the number of workers. - -* Types of change - - * [ ] Dag changes - * [x] Config changes - * [ ] API changes - * [ ] CLI changes - * [ ] Behaviour changes - * [ ] Plugin changes - * [ ] Dependency changes - * [ ] Code interface changes diff --git a/reproducible_build.yaml b/reproducible_build.yaml index 7226a3bc1ae..d03870c8423 100644 --- a/reproducible_build.yaml +++ b/reproducible_build.yaml @@ -1,2 +1,2 @@ -release-notes-hash: 231c1dea9192040721d8048592300cac -source-date-epoch: 1757972431 +release-notes-hash: 82291e0ddd593e41fd9349cd2b12f227 +source-date-epoch: 1758062538
