jedcunningham commented on code in PR #30643: URL: https://github.com/apache/airflow/pull/30643#discussion_r1167203222
########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,271 @@ .. towncrier release notes start +Airflow 2.6.0 (2023-04-20) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) + have the same group set as main group. Previously the default was also other-writeable and the user might choose + to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` + and ``file_task_handler_new_file_permissions`` in ``logging`` section. Review Comment: ```suggestion Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) have the same group set as main group. Previously the default was also other-writeable and the user might choose to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` and ``file_task_handler_new_file_permissions`` in ``logging`` section. ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,271 @@ .. towncrier release notes start +Airflow 2.6.0 (2023-04-20) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) + have the same group set as main group. Previously the default was also other-writeable and the user might choose + to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` + and ``file_task_handler_new_file_permissions`` in ``logging`` section. + +SLA callbacks no longer add files to the dag processor manager's queue (#30076) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, + and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state. Review Comment: ```suggestion This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state. ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,271 @@ .. towncrier release notes start +Airflow 2.6.0 (2023-04-20) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) + have the same group set as main group. Previously the default was also other-writeable and the user might choose + to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` + and ``file_task_handler_new_file_permissions`` in ``logging`` section. + +SLA callbacks no longer add files to the dag processor manager's queue (#30076) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, + and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state. + +The ``cleanup()`` method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This is potentially a breaking change for any custom trigger implementations that override the ``cleanup()`` + method and uses synchronous code, however using synchronous operations in cleanup was technically wrong, + because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays + impacting other triggers. The change is unlikely to affect any existing trigger implementations. Review Comment: ```suggestion This is potentially a breaking change for any custom trigger implementations that override the ``cleanup()`` method and uses synchronous code, however using synchronous operations in cleanup was technically wrong, because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays impacting other triggers. The change is unlikely to affect any existing trigger implementations. ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,271 @@ .. towncrier release notes start +Airflow 2.6.0 (2023-04-20) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) + have the same group set as main group. Previously the default was also other-writeable and the user might choose + to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` + and ``file_task_handler_new_file_permissions`` in ``logging`` section. + +SLA callbacks no longer add files to the dag processor manager's queue (#30076) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, + and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state. + +The ``cleanup()`` method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This is potentially a breaking change for any custom trigger implementations that override the ``cleanup()`` + method and uses synchronous code, however using synchronous operations in cleanup was technically wrong, + because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays + impacting other triggers. The change is unlikely to affect any existing trigger implementations. + +The gauge ``scheduler.tasks.running`` no longer exist (#30374) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The gauge has never been working and its value has always been 0. Having an accurate + value for this metric is complex so it has been decided that removing this gauge makes + more sense than fixing it with no certainty of the correctness of its value. + +Consolidate handling of tasks stuck in queued under new ``task_queued_timeout`` config (#30375) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations + responsible for timing out stuck queued tasks have been deprecated and merged into + ``[scheduler] task_queued_timeout``. The configurations that have been deprecated are + ``[kubernetes] worker_pods_pending_timeout``, ``[celery] stalled_task_timeout``, and + ``[celery] task_adoption_timeout``. If any of these configurations are set, the longest timeout will be + respected. For example, if ``[celery] stalled_task_timeout`` is 1200, and ``[scheduler] task_queued_timeout`` + is 600, Airflow will set ``[scheduler] task_queued_timeout`` to 1200. + +Improvement Changes +^^^^^^^^^^^^^^^^^^^ + +Display only the running configuration in configurations view (#28892) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The configurations view now only displays the running configuration. Previously, the default configuration + was displayed at the top but it was not obvious whether this default configuration was overridden or not. + Subsequently, the non-documented endpoint ``/configuration?raw=true`` is deprecated and will be removed in + Airflow 3.0. The HTTP response now returns an additional ``Deprecation`` header. The ``/config`` endpoint on + the REST API is the standard way to fetch Airflow configuration programmatically. + +Explicit skipped states list for ExternalTaskSensor (#29933) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + ExternalTaskSensor now has an explicit ``skipped_states`` list + +Miscellaneous Changes +^^^^^^^^^^^^^^^^^^^^^ + +Handle OverflowError on exponential backoff in next_run_calculation (#28172) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Maximum retry task delay is set to be 24h (86400s) by default. You can change it globally via ``core.max_task_retry_delay`` + parameter. Review Comment: ```suggestion Maximum retry task delay is set to be 24h (86400s) by default. You can change it globally via ``core.max_task_retry_delay`` parameter. ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,271 @@ .. towncrier release notes start +Airflow 2.6.0 (2023-04-20) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) + have the same group set as main group. Previously the default was also other-writeable and the user might choose + to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` + and ``file_task_handler_new_file_permissions`` in ``logging`` section. + +SLA callbacks no longer add files to the dag processor manager's queue (#30076) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, + and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state. + +The ``cleanup()`` method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This is potentially a breaking change for any custom trigger implementations that override the ``cleanup()`` + method and uses synchronous code, however using synchronous operations in cleanup was technically wrong, + because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays + impacting other triggers. The change is unlikely to affect any existing trigger implementations. + +The gauge ``scheduler.tasks.running`` no longer exist (#30374) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The gauge has never been working and its value has always been 0. Having an accurate + value for this metric is complex so it has been decided that removing this gauge makes + more sense than fixing it with no certainty of the correctness of its value. + +Consolidate handling of tasks stuck in queued under new ``task_queued_timeout`` config (#30375) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations + responsible for timing out stuck queued tasks have been deprecated and merged into + ``[scheduler] task_queued_timeout``. The configurations that have been deprecated are + ``[kubernetes] worker_pods_pending_timeout``, ``[celery] stalled_task_timeout``, and + ``[celery] task_adoption_timeout``. If any of these configurations are set, the longest timeout will be + respected. For example, if ``[celery] stalled_task_timeout`` is 1200, and ``[scheduler] task_queued_timeout`` + is 600, Airflow will set ``[scheduler] task_queued_timeout`` to 1200. Review Comment: ```suggestion Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations responsible for timing out stuck queued tasks have been deprecated and merged into ``[scheduler] task_queued_timeout``. The configurations that have been deprecated are ``[kubernetes] worker_pods_pending_timeout``, ``[celery] stalled_task_timeout``, and ``[celery] task_adoption_timeout``. If any of these configurations are set, the longest timeout will be respected. For example, if ``[celery] stalled_task_timeout`` is 1200, and ``[scheduler] task_queued_timeout`` is 600, Airflow will set ``[scheduler] task_queued_timeout`` to 1200. ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,271 @@ .. towncrier release notes start +Airflow 2.6.0 (2023-04-20) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) + have the same group set as main group. Previously the default was also other-writeable and the user might choose + to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` + and ``file_task_handler_new_file_permissions`` in ``logging`` section. + +SLA callbacks no longer add files to the dag processor manager's queue (#30076) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, + and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state. + +The ``cleanup()`` method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This is potentially a breaking change for any custom trigger implementations that override the ``cleanup()`` + method and uses synchronous code, however using synchronous operations in cleanup was technically wrong, + because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays + impacting other triggers. The change is unlikely to affect any existing trigger implementations. + +The gauge ``scheduler.tasks.running`` no longer exist (#30374) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The gauge has never been working and its value has always been 0. Having an accurate + value for this metric is complex so it has been decided that removing this gauge makes + more sense than fixing it with no certainty of the correctness of its value. + +Consolidate handling of tasks stuck in queued under new ``task_queued_timeout`` config (#30375) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations + responsible for timing out stuck queued tasks have been deprecated and merged into + ``[scheduler] task_queued_timeout``. The configurations that have been deprecated are + ``[kubernetes] worker_pods_pending_timeout``, ``[celery] stalled_task_timeout``, and + ``[celery] task_adoption_timeout``. If any of these configurations are set, the longest timeout will be + respected. For example, if ``[celery] stalled_task_timeout`` is 1200, and ``[scheduler] task_queued_timeout`` + is 600, Airflow will set ``[scheduler] task_queued_timeout`` to 1200. + +Improvement Changes +^^^^^^^^^^^^^^^^^^^ + +Display only the running configuration in configurations view (#28892) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The configurations view now only displays the running configuration. Previously, the default configuration + was displayed at the top but it was not obvious whether this default configuration was overridden or not. + Subsequently, the non-documented endpoint ``/configuration?raw=true`` is deprecated and will be removed in + Airflow 3.0. The HTTP response now returns an additional ``Deprecation`` header. The ``/config`` endpoint on + the REST API is the standard way to fetch Airflow configuration programmatically. + +Explicit skipped states list for ExternalTaskSensor (#29933) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + ExternalTaskSensor now has an explicit ``skipped_states`` list Review Comment: I feel like this can just be folded into the main list of improvements. ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,271 @@ .. towncrier release notes start +Airflow 2.6.0 (2023-04-20) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) + have the same group set as main group. Previously the default was also other-writeable and the user might choose + to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` + and ``file_task_handler_new_file_permissions`` in ``logging`` section. + +SLA callbacks no longer add files to the dag processor manager's queue (#30076) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, + and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state. + +The ``cleanup()`` method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This is potentially a breaking change for any custom trigger implementations that override the ``cleanup()`` + method and uses synchronous code, however using synchronous operations in cleanup was technically wrong, + because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays + impacting other triggers. The change is unlikely to affect any existing trigger implementations. + +The gauge ``scheduler.tasks.running`` no longer exist (#30374) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The gauge has never been working and its value has always been 0. Having an accurate + value for this metric is complex so it has been decided that removing this gauge makes + more sense than fixing it with no certainty of the correctness of its value. Review Comment: ```suggestion The gauge has never been working and its value has always been 0. Having an accurate value for this metric is complex so it has been decided that removing this gauge makes more sense than fixing it with no certainty of the correctness of its value. ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,271 @@ .. towncrier release notes start +Airflow 2.6.0 (2023-04-20) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) + have the same group set as main group. Previously the default was also other-writeable and the user might choose + to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` + and ``file_task_handler_new_file_permissions`` in ``logging`` section. + +SLA callbacks no longer add files to the dag processor manager's queue (#30076) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, + and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state. + +The ``cleanup()`` method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This is potentially a breaking change for any custom trigger implementations that override the ``cleanup()`` + method and uses synchronous code, however using synchronous operations in cleanup was technically wrong, + because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays + impacting other triggers. The change is unlikely to affect any existing trigger implementations. + +The gauge ``scheduler.tasks.running`` no longer exist (#30374) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The gauge has never been working and its value has always been 0. Having an accurate + value for this metric is complex so it has been decided that removing this gauge makes + more sense than fixing it with no certainty of the correctness of its value. + +Consolidate handling of tasks stuck in queued under new ``task_queued_timeout`` config (#30375) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations + responsible for timing out stuck queued tasks have been deprecated and merged into + ``[scheduler] task_queued_timeout``. The configurations that have been deprecated are + ``[kubernetes] worker_pods_pending_timeout``, ``[celery] stalled_task_timeout``, and + ``[celery] task_adoption_timeout``. If any of these configurations are set, the longest timeout will be + respected. For example, if ``[celery] stalled_task_timeout`` is 1200, and ``[scheduler] task_queued_timeout`` + is 600, Airflow will set ``[scheduler] task_queued_timeout`` to 1200. + +Improvement Changes +^^^^^^^^^^^^^^^^^^^ + +Display only the running configuration in configurations view (#28892) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The configurations view now only displays the running configuration. Previously, the default configuration + was displayed at the top but it was not obvious whether this default configuration was overridden or not. + Subsequently, the non-documented endpoint ``/configuration?raw=true`` is deprecated and will be removed in + Airflow 3.0. The HTTP response now returns an additional ``Deprecation`` header. The ``/config`` endpoint on + the REST API is the standard way to fetch Airflow configuration programmatically. + +Explicit skipped states list for ExternalTaskSensor (#29933) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + ExternalTaskSensor now has an explicit ``skipped_states`` list + +Miscellaneous Changes +^^^^^^^^^^^^^^^^^^^^^ + +Handle OverflowError on exponential backoff in next_run_calculation (#28172) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Maximum retry task delay is set to be 24h (86400s) by default. You can change it globally via ``core.max_task_retry_delay`` + parameter. + +Move Hive macros to the provider (#28538) +""""""""""""""""""""""""""""""""""""""""" + The Hive Macros (``hive.max_partition``, ``hive.closest_ds_partition``) are available only when Hive Provider is + installed. Please install Hive Provider > 5.1.0 when using those macros. Review Comment: ```suggestion The Hive Macros (``hive.max_partition``, ``hive.closest_ds_partition``) are available only when Hive Provider is installed. Please install Hive Provider > 5.1.0 when using those macros. ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,271 @@ .. towncrier release notes start +Airflow 2.6.0 (2023-04-20) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) + have the same group set as main group. Previously the default was also other-writeable and the user might choose + to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` + and ``file_task_handler_new_file_permissions`` in ``logging`` section. + +SLA callbacks no longer add files to the dag processor manager's queue (#30076) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, + and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state. + +The ``cleanup()`` method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This is potentially a breaking change for any custom trigger implementations that override the ``cleanup()`` + method and uses synchronous code, however using synchronous operations in cleanup was technically wrong, + because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays + impacting other triggers. The change is unlikely to affect any existing trigger implementations. + +The gauge ``scheduler.tasks.running`` no longer exist (#30374) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The gauge has never been working and its value has always been 0. Having an accurate + value for this metric is complex so it has been decided that removing this gauge makes + more sense than fixing it with no certainty of the correctness of its value. + +Consolidate handling of tasks stuck in queued under new ``task_queued_timeout`` config (#30375) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations + responsible for timing out stuck queued tasks have been deprecated and merged into + ``[scheduler] task_queued_timeout``. The configurations that have been deprecated are + ``[kubernetes] worker_pods_pending_timeout``, ``[celery] stalled_task_timeout``, and + ``[celery] task_adoption_timeout``. If any of these configurations are set, the longest timeout will be + respected. For example, if ``[celery] stalled_task_timeout`` is 1200, and ``[scheduler] task_queued_timeout`` + is 600, Airflow will set ``[scheduler] task_queued_timeout`` to 1200. + +Improvement Changes +^^^^^^^^^^^^^^^^^^^ + +Display only the running configuration in configurations view (#28892) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The configurations view now only displays the running configuration. Previously, the default configuration + was displayed at the top but it was not obvious whether this default configuration was overridden or not. + Subsequently, the non-documented endpoint ``/configuration?raw=true`` is deprecated and will be removed in + Airflow 3.0. The HTTP response now returns an additional ``Deprecation`` header. The ``/config`` endpoint on + the REST API is the standard way to fetch Airflow configuration programmatically. Review Comment: ```suggestion The configurations view now only displays the running configuration. Previously, the default configuration was displayed at the top but it was not obvious whether this default configuration was overridden or not. Subsequently, the non-documented endpoint ``/configuration?raw=true`` is deprecated and will be removed in Airflow 3.0. The HTTP response now returns an additional ``Deprecation`` header. The ``/config`` endpoint on the REST API is the standard way to fetch Airflow configuration programmatically. ``` ########## RELEASE_NOTES.rst: ########## @@ -21,6 +21,271 @@ .. towncrier release notes start +Airflow 2.6.0 (2023-04-20) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Default permissions of file task handler log directories and files has been changed to "owner + group" writeable (#29506). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Default setting handles case where impersonation is needed and both users (airflow and the impersonated user) + have the same group set as main group. Previously the default was also other-writeable and the user might choose + to use the other-writeable setting if they wish by configuring ``file_task_handler_new_folder_permissions`` + and ``file_task_handler_new_file_permissions`` in ``logging`` section. + +SLA callbacks no longer add files to the dag processor manager's queue (#30076) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This stops SLA callbacks from keeping the dag processor manager permanently busy. It means reduced CPU, + and fixes issues where SLAs stop the system from seeing changes to existing dag files. Additional metrics added to help track queue state. + +The ``cleanup()`` method in BaseTrigger is now defined as asynchronous (following async/await) pattern (#30152). +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + This is potentially a breaking change for any custom trigger implementations that override the ``cleanup()`` + method and uses synchronous code, however using synchronous operations in cleanup was technically wrong, + because the method was executed in the main loop of the Triggerer and it was introducing unnecessary delays + impacting other triggers. The change is unlikely to affect any existing trigger implementations. + +The gauge ``scheduler.tasks.running`` no longer exist (#30374) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The gauge has never been working and its value has always been 0. Having an accurate + value for this metric is complex so it has been decided that removing this gauge makes + more sense than fixing it with no certainty of the correctness of its value. + +Consolidate handling of tasks stuck in queued under new ``task_queued_timeout`` config (#30375) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + Logic for handling tasks stuck in the queued state has been consolidated, and the all configurations + responsible for timing out stuck queued tasks have been deprecated and merged into + ``[scheduler] task_queued_timeout``. The configurations that have been deprecated are + ``[kubernetes] worker_pods_pending_timeout``, ``[celery] stalled_task_timeout``, and + ``[celery] task_adoption_timeout``. If any of these configurations are set, the longest timeout will be + respected. For example, if ``[celery] stalled_task_timeout`` is 1200, and ``[scheduler] task_queued_timeout`` + is 600, Airflow will set ``[scheduler] task_queued_timeout`` to 1200. + +Improvement Changes +^^^^^^^^^^^^^^^^^^^ + +Display only the running configuration in configurations view (#28892) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + The configurations view now only displays the running configuration. Previously, the default configuration + was displayed at the top but it was not obvious whether this default configuration was overridden or not. + Subsequently, the non-documented endpoint ``/configuration?raw=true`` is deprecated and will be removed in + Airflow 3.0. The HTTP response now returns an additional ``Deprecation`` header. The ``/config`` endpoint on + the REST API is the standard way to fetch Airflow configuration programmatically. + +Explicit skipped states list for ExternalTaskSensor (#29933) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" + ExternalTaskSensor now has an explicit ``skipped_states`` list Review Comment: ```suggestion ExternalTaskSensor now has an explicit ``skipped_states`` list ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
