jedcunningham commented on code in PR #27895:
URL: https://github.com/apache/airflow/pull/27895#discussion_r1031692178
##########
RELEASE_NOTES.rst:
##########
@@ -21,6 +21,279 @@
.. towncrier release notes start
+Airflow 2.5.0 (2022-11-28)
Review Comment:
```suggestion
Airflow 2.5.0 (2022-11-30)
```
##########
RELEASE_NOTES.rst:
##########
@@ -21,6 +21,279 @@
.. towncrier release notes start
+Airflow 2.5.0 (2022-11-28)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+- ``airflow dags test`` no longer performs a backfill job.
+
+ In order to make ``airflow dags test`` more useful as a testing and
debugging tool, we no
+ longer run a backfill job and instead run a "local task runner". Users can
still backfill
+ their DAGs using the ``airflow dags backfill`` command. (#26400)
+- Airflow config section ``kubernetes`` renamed to ``kubernetes_executor``
Review Comment:
```suggestion
Airflow config section ``kubernetes`` renamed to ``kubernetes_executor``
(#26873)
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
```
##########
RELEASE_NOTES.rst:
##########
@@ -21,6 +21,279 @@
.. towncrier release notes start
+Airflow 2.5.0 (2022-11-28)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+- ``airflow dags test`` no longer performs a backfill job.
+
+ In order to make ``airflow dags test`` more useful as a testing and
debugging tool, we no
+ longer run a backfill job and instead run a "local task runner". Users can
still backfill
+ their DAGs using the ``airflow dags backfill`` command. (#26400)
Review Comment:
```suggestion
their DAGs using the ``airflow dags backfill`` command.
```
##########
RELEASE_NOTES.rst:
##########
@@ -21,6 +21,279 @@
.. towncrier release notes start
+Airflow 2.5.0 (2022-11-28)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+- ``airflow dags test`` no longer performs a backfill job.
+
+ In order to make ``airflow dags test`` more useful as a testing and
debugging tool, we no
+ longer run a backfill job and instead run a "local task runner". Users can
still backfill
+ their DAGs using the ``airflow dags backfill`` command. (#26400)
+- Airflow config section ``kubernetes`` renamed to ``kubernetes_executor``
+
+ KubernetesPodOperator no longer considers any core kubernetes config params,
so this section now only applies to kubernetes executor. Renaming it reduces
potential for confusion. (#26873)
+- ``ExternalTaskSensor`` no longer hangs indefinitely when ``failed_states``
is set, an ``execute_date_fn`` is used, and some but not all of the dependent
tasks fail. Instead, an ``AirflowException`` is thrown as soon as any of the
dependent tasks fail.
+
+ Any code handling this failure in addition to timeouts should move to
cathing the ``AirflowException`` baseclass and not only the
``AirflowSensorTimeout`` subclass. (#27190)
Review Comment:
This will need a similar change as above, but we probably need a shorter
title for this?
##########
RELEASE_NOTES.rst:
##########
@@ -21,6 +21,279 @@
.. towncrier release notes start
+Airflow 2.5.0 (2022-11-28)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+- ``airflow dags test`` no longer performs a backfill job.
Review Comment:
```suggestion
``airflow dags test`` no longer performs a backfill job (#26400)
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""
```
##########
RELEASE_NOTES.rst:
##########
@@ -21,6 +21,279 @@
.. towncrier release notes start
+Airflow 2.5.0 (2022-11-28)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+- ``airflow dags test`` no longer performs a backfill job.
+
+ In order to make ``airflow dags test`` more useful as a testing and
debugging tool, we no
+ longer run a backfill job and instead run a "local task runner". Users can
still backfill
+ their DAGs using the ``airflow dags backfill`` command. (#26400)
+- Airflow config section ``kubernetes`` renamed to ``kubernetes_executor``
+
+ KubernetesPodOperator no longer considers any core kubernetes config params,
so this section now only applies to kubernetes executor. Renaming it reduces
potential for confusion. (#26873)
+- ``ExternalTaskSensor`` no longer hangs indefinitely when ``failed_states``
is set, an ``execute_date_fn`` is used, and some but not all of the dependent
tasks fail. Instead, an ``AirflowException`` is thrown as soon as any of the
dependent tasks fail.
+
+ Any code handling this failure in addition to timeouts should move to
cathing the ``AirflowException`` baseclass and not only the
``AirflowSensorTimeout`` subclass. (#27190)
+
+New Features
+^^^^^^^^^^^^
+- ``TaskRunner``: notify of component start and finish (#27855)
+- Add DagRun state change to the Listener plugin system(#27113)
+- Metric for raw task return codes (#27155)
+- Add logic for XComArg to pull specific map indexes (#27771)
+- Clear TaskGroup (#26658)
+- Add critical section query duration metric (#27700)
+- Add: #23880 :: Audit log for ``AirflowModelViews(Variables/Connection)``
(#24079)
+- Add postgres 15 support (#27444)
+- Expand tasks in mapped group at run time (#27491)
+- reset commits, clean submodules (#27560)
+- scheduler_job, add metric for scheduler loop timer (#27605)
+- Allow datasets to be used in taskflow (#27540)
+- Add expanded_ti_count to ti context (#27680)
+- Add user comment to task instance and dag run (#26457, #27849, #27867)
+- Enable copying DagRun JSON to clipboard (#27639)
+- Implement extra controls for SLAs (#27557)
+- add dag parsed time in DAG view (#27573)
+- Add max_wait for exponential_backoff in BaseSensor (#27597)
+- Expand tasks in mapped group at parse time (#27158)
+- Add disable retry flag on backfill (#23829)
+- Adding sensor decorator (#22562)
+- Api endpoint update ti (#26165)
+- Filtering datasets by recent update events (#26942)
+- Support Is /not Null filter for value is None on webui (#26584)
+- Add search to datasets list (#26893)
+- Split out and handle 'params' in mapped operator (#26100)
+- Add authoring API for TaskGroup mapping (#26844)
+- Add ``one_done`` trigger rule (#26146)
+- Create a more efficient airflow dag test command that also has better local
logging (#26400)
+- Support add/remove permissions to roles commands (#26338)
+- Auto tail file logs in Web UI (#26169)
+- Add triggerer info to task instance in API (#26249)
+- Flag to deserialize value on custom XCom backend (#26343)
+
+Bug Fixes
+^^^^^^^^^
+- Redirect to home view when there are no valid tags in the URL (#25715)
+- Make MappedTaskGroup depend on its expand inputs (#27876)
+- Make DagRun state updates for paused DAGs faster (#27725)
+- Don't explicitly set include_examples to False on task run command (#27813)
+- Fix menu border color (#27789)
+- Fix backfill queued task getting reset to scheduled state. (#23720)
+- Fix clearing child dag mapped tasks from parent dag (#27501)
+- Handle json encoding of ``V1Pod`` in task callback (#27609)
+- Fix ExternalTaskSensor can't check zipped dag (#27056)
+- Avoid re-fetching DAG run in TriggerDagRunOperator (#27635)
+- Continue on exception when retrieving metadata (#27665)
+- Fix double logging with some task logging handler (#27591)
+- External task sensor fail fix (#27190)
+- Replace FAB url filtering function with Airflows (#27576)
+- Fix mini scheduler expansion of mapped task (#27506)
+- Add the default None when pop actions (#27537)
+- Display parameter values from serialized dag in trigger dag view. (#27482)
+- Fix getting the dag/task ids from base executor (#27550)
+- Fix sqlalchemy primary key black-out error on DDRQ (#27538)
+- Move TriggerDagRun conf check to execute (#27035)
+- SLAMiss is nullable and not always given back when pulling task instances
(#27423)
+- Fix behavior of ``_`` when searching for DAGs (#27448)
+- Add case insensitive constraint to username (#27266)
+- Fix python external template keys (#27256)
+- Resolve trigger assignment race condition (#27072)
+- Update google_analytics.html (#27226)
+- Fix IntegrityError during webserver startup (#27297)
+- reduce extraneous task log requests (#27233)
+- Make RotatingFilehandler used in DagProcessor non-caching (#27223)
+- set executor.job_id to BackfillJob.id for backfills (#27020)
+- Fix some bug in web ui dags list page (auto-refresh & jump search null
state) (#27141)
+- Listener: Set task on sqlalchemy taskinstance object (#27167)
+- fix yarn warning from d3-color (#27139)
+- Don't overwrite connection extra with invalid json (#27142)
+- Simplify origin string cleaning (#27143)
+- Don't re-patch pods that are already controlled by current worker (#26778)
+- Add separate error handler for 405(Method not allowed) errors (#26880)
+- Avoid 500 on dag redirect (#27064)
+- Make tracebacks opt-in (#27059)
Review Comment:
```suggestion
```
Hmm, these were already in 2.4.2? Might be others as well.
##########
RELEASE_NOTES.rst:
##########
@@ -21,6 +21,279 @@
.. towncrier release notes start
+Airflow 2.5.0 (2022-11-28)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+- ``airflow dags test`` no longer performs a backfill job.
+
+ In order to make ``airflow dags test`` more useful as a testing and
debugging tool, we no
+ longer run a backfill job and instead run a "local task runner". Users can
still backfill
+ their DAGs using the ``airflow dags backfill`` command. (#26400)
+- Airflow config section ``kubernetes`` renamed to ``kubernetes_executor``
+
+ KubernetesPodOperator no longer considers any core kubernetes config params,
so this section now only applies to kubernetes executor. Renaming it reduces
potential for confusion. (#26873)
+- ``ExternalTaskSensor`` no longer hangs indefinitely when ``failed_states``
is set, an ``execute_date_fn`` is used, and some but not all of the dependent
tasks fail. Instead, an ``AirflowException`` is thrown as soon as any of the
dependent tasks fail.
+
+ Any code handling this failure in addition to timeouts should move to
cathing the ``AirflowException`` baseclass and not only the
``AirflowSensorTimeout`` subclass. (#27190)
+
+New Features
+^^^^^^^^^^^^
+- ``TaskRunner``: notify of component start and finish (#27855)
+- Add DagRun state change to the Listener plugin system(#27113)
+- Metric for raw task return codes (#27155)
+- Add logic for XComArg to pull specific map indexes (#27771)
+- Clear TaskGroup (#26658)
+- Add critical section query duration metric (#27700)
+- Add: #23880 :: Audit log for ``AirflowModelViews(Variables/Connection)``
(#24079)
+- Add postgres 15 support (#27444)
+- Expand tasks in mapped group at run time (#27491)
+- reset commits, clean submodules (#27560)
+- scheduler_job, add metric for scheduler loop timer (#27605)
+- Allow datasets to be used in taskflow (#27540)
+- Add expanded_ti_count to ti context (#27680)
+- Add user comment to task instance and dag run (#26457, #27849, #27867)
+- Enable copying DagRun JSON to clipboard (#27639)
+- Implement extra controls for SLAs (#27557)
+- add dag parsed time in DAG view (#27573)
+- Add max_wait for exponential_backoff in BaseSensor (#27597)
+- Expand tasks in mapped group at parse time (#27158)
+- Add disable retry flag on backfill (#23829)
+- Adding sensor decorator (#22562)
+- Api endpoint update ti (#26165)
+- Filtering datasets by recent update events (#26942)
+- Support Is /not Null filter for value is None on webui (#26584)
+- Add search to datasets list (#26893)
+- Split out and handle 'params' in mapped operator (#26100)
+- Add authoring API for TaskGroup mapping (#26844)
+- Add ``one_done`` trigger rule (#26146)
+- Create a more efficient airflow dag test command that also has better local
logging (#26400)
+- Support add/remove permissions to roles commands (#26338)
+- Auto tail file logs in Web UI (#26169)
+- Add triggerer info to task instance in API (#26249)
+- Flag to deserialize value on custom XCom backend (#26343)
+
+Bug Fixes
+^^^^^^^^^
+- Redirect to home view when there are no valid tags in the URL (#25715)
+- Make MappedTaskGroup depend on its expand inputs (#27876)
+- Make DagRun state updates for paused DAGs faster (#27725)
+- Don't explicitly set include_examples to False on task run command (#27813)
+- Fix menu border color (#27789)
+- Fix backfill queued task getting reset to scheduled state. (#23720)
+- Fix clearing child dag mapped tasks from parent dag (#27501)
+- Handle json encoding of ``V1Pod`` in task callback (#27609)
+- Fix ExternalTaskSensor can't check zipped dag (#27056)
+- Avoid re-fetching DAG run in TriggerDagRunOperator (#27635)
+- Continue on exception when retrieving metadata (#27665)
+- Fix double logging with some task logging handler (#27591)
+- External task sensor fail fix (#27190)
+- Replace FAB url filtering function with Airflows (#27576)
+- Fix mini scheduler expansion of mapped task (#27506)
+- Add the default None when pop actions (#27537)
+- Display parameter values from serialized dag in trigger dag view. (#27482)
+- Fix getting the dag/task ids from base executor (#27550)
+- Fix sqlalchemy primary key black-out error on DDRQ (#27538)
+- Move TriggerDagRun conf check to execute (#27035)
+- SLAMiss is nullable and not always given back when pulling task instances
(#27423)
+- Fix behavior of ``_`` when searching for DAGs (#27448)
+- Add case insensitive constraint to username (#27266)
+- Fix python external template keys (#27256)
+- Resolve trigger assignment race condition (#27072)
+- Update google_analytics.html (#27226)
+- Fix IntegrityError during webserver startup (#27297)
+- reduce extraneous task log requests (#27233)
+- Make RotatingFilehandler used in DagProcessor non-caching (#27223)
+- set executor.job_id to BackfillJob.id for backfills (#27020)
+- Fix some bug in web ui dags list page (auto-refresh & jump search null
state) (#27141)
+- Listener: Set task on sqlalchemy taskinstance object (#27167)
+- fix yarn warning from d3-color (#27139)
+- Don't overwrite connection extra with invalid json (#27142)
+- Simplify origin string cleaning (#27143)
+- Don't re-patch pods that are already controlled by current worker (#26778)
+- Add separate error handler for 405(Method not allowed) errors (#26880)
+- Avoid 500 on dag redirect (#27064)
+- Make tracebacks opt-in (#27059)
+- Filter dataset dependency data on webserver (#27046)
+- Remove double collection of dags in ``airflow dags reserialize`` (#27030)
+- Fix auto refresh for graph view (#26926)
+- Add missing AUTOINC/SERIAL for FAB tables (#26885)
+- fix next run dataset modal links (#26897)
+- Change dag audit log sort by date from asc to desc (#26895)
+- Bump min version of jinja2 (#26866)
+- Add missing colors to state_color_mapping jinja global (#26822)
+- Retry on Airflow Schedule DAG Run DB Deadlock (#26347)
+- Fix running debuggers inside ``airflow tasks test`` (#26806)
+- Fix warning when using xcomarg dependencies (#26801)
+- demote Removed state in priority for displaying task summaries (#26789)
+- Ensure the log messages from operators during parsing go somewhere (#26779)
+- Add restarting state to TaskState Enum in REST API (#26776)
+- Remove DAG parsing from StandardTaskRunner (#26750)
+- Allow retrieving error message from data.detail (#26762)
+- Fix airflow tasks run --local when dags_folder differs from that of
processor (#26509)
+- Handle mapped tasks in task duration chart (#26722)
+- Remove TaskFail duplicates check (#26714)
+- Fix task duration cumulative chart (#26717)
+- Fix non-hidden cumulative chart on duration view (#26716)
+- Fixed broken URL for docker-compose.yaml (#26721)
+- When rendering template, unmap task in context (#26702)
+- Fix scroll overflow for ConfirmDialog (#26681)
+- Use COALESCE when ordering runs to handle NULL (#26626)
+- Fix xcom arg.py .zip bug (#26636)
+- Allow MapXComArg to resolve after serialization (#26591)
+- Resolve deprecation warning re Table.exists() (#26616)
+- Check user is active (#26635)
+- Resolve warning about DISTINCT ON query on dags view (#26608)
+- Suppress SQLALCHEMY_TRACK_MODIFICATIONS warning in db init (#26617)
+- Fix Deferrable stuck as "scheduled" during backfill (#26205)
+- No missing user warning for public admin (#26611)
+- Fix grid view log try numbers (#26556)
+- Fix 404 taskInstance errors and split into two tables (#26575)
+- Log warning when secret backend kwargs is invalid (#26580)
+- fix WSGI root app (#26549)
+- Fix browser warning of improper thead usage (#26551)
+- Correctly set json_provider_class on Flask app so it uses our encoder
(#26554)
+- Revert "No grid auto-refresh for backfill dag runs (#25042)" (#26463)
+- Fix deadlock when mapped task with removed upstream is rerun (#26518)
+- ExecutorConfigType should be cacheable (#26498)
+- template rendering issue fix (#26390)
+- Fix proper joining of the path for logs retrieved from celery workers
(#26493)
+- Don't update backfill run from the scheduler (#26342)
+- Fix version for a couple configurations (#26491)
+- DAG Deps extends ``base_template`` (#26439)
+- Fix UI redirect (#26409)
+- Clear autoregistered DAGs if there are any import errors (#26398)
+- Require dag_id arg for dags list-runs (#26357)
+- Properly build URL to retrieve logs independently from system (#26337)
+- Add the dag_id to AirflowDagCycleException message (#26204)
+- Fix 'from airflow import version' lazy import (#26239)
+
+Improvements
+^^^^^^^^^^^^
+- Allow depth-first execution (#27827)
+- UI: Update offset height if data changes (#27865)
+- Improve TriggerRuleDep typing and readability (#27810)
+- Make views requiring session, keyword only args (#27790)
+- Optimize ``TI.xcom_pull()`` with explicit task_ids and map_indexes (#27699)
+- Allow hyphens in pod id used by k8s executor (#27737)
+- optimise task instances filtering (#27102)
+- Use context managers to simplify log serve management (#27756)
+- Fix formatting leftovers (#27750)
+- Improve task deadlock messaging (#27734)
+- Improve "sensor timeout" messaging (#27733)
+- Replace urlparse with urlsplit (#27389)
+- Add new files to parsing queue on every loop of dag processing (#27060)
+- Make Kubernetes Executor & Scheduler resilient to error during PMH execution
(#27611)
+- Separate dataset deps into individual graphs (#27356)
+- Use log.exception where more economical than log.error (#27517)
+- Move validation ``branch_task_ids`` into ``SkipMixin`` (#27434)
+- Coerce LazyXComAccess to list when pushed to XCom (#27251)
+- Update cluster-policies.rst docs (#27362)
+- Add warning if connection type already registered within the provider
(#27520)
+- Activate debug logging in commands with --verbose option (#27447)
+- Add classic examples for Python Operators (#27403)
+- change ``.first()`` to ``.scalar()`` (#27323)
+- Improve reset_dag_run description (#26755)
+- Add examples and howtos about sensors (#27333)
+- Make grid view widths adjustable (#27273)
+- Sorting plugins custom menu links by category before name (#27152)
+- Simplify DagRun.verify_integrity (#26894)
+- Add mapped task group info to serialization (#27027)
+- Correct the JSON style used for Run config in Grid View (#27119)
+- No ``extra__conn_type__`` prefix required for UI behaviors (#26995)
+- Betterize dataset update blurb (#26878)
+- Rename kubernetes config section to kubernetes_executor (#26873)
+- decode params for dataset searches (#26941)
+- Get rid of the DAGRun details page & rely completely on Grid (#26837)
+- Fix scheduler crashloopbackoff when using ``hostname_callable`` (#24999)
+- Reduce log verbosity in KubernetesExecutor. (#26582)
+- Don't iterate tis list twice for no reason (#26740)
+- Clearer code for PodGenerator.deserialize_model_file (#26641)
+- Don't import kubernetes unless you have a V1Pod (#26496)
+- Add updated_at column to DagRun and Ti tables (#26252)
+- Move the deserialization of custom XCom Backend to 2.4.0 (#26392)
+- Avoid calculating all elements when one item is needed (#26377)
+- Add ``__future__``.annotations automatically by isort (#26383)
+- Handle list when serializing expand_kwargs (#26369)
+- Apply PEP-563 (Postponed Evaluation of Annotations) to core airflow (#26290)
+- Add more weekday operator and sensor examples #26071 (#26098)
+- Align TaskGroup semantics to AbstractOperator (#27723)
+
+Misc/Internal
+^^^^^^^^^^^^^
+- Fix old-style typing in Base Sensor (#27871)
+- Switch (back) to late imports (#27730)
+- Completed D400 for multiple folders (#27748)
+- simplify notes accordion test (#27757)
+- completed D400 for ``airflow/callbacks/* airflow/cli/*`` (#27721)
+- Completed D400 for ``airflow/api_connexion/* directory`` (#27718)
+- Completed D400 for ``airflow/listener/* directory`` (#27731)
+- Completed D400 for ``airflow/lineage/* directory`` (#27732)
+- Update API & Python Client versions (#27642)
+- Completed D400 & D401 for ``airflow/api/*`` directory (#27716)
+- Completed D400 for multiple folders (#27722)
+- Bump minimatch from ``3.0.4 to 3.0.8`` in ``/airflow/www`` (#27688)
+- Bump loader-utils from ``1.4.1 to 1.4.2 ``in ``/airflow/www`` (#27697)
+- Disable nested task mapping for now (#27681)
+- bump alembic minimum version (#27629)
+- remove unused code.html (#27585)
+- Enable python string normalization everywhere (#27588)
+- Bump loader-utils from ``1.4.0 to 1.4.1`` in /airflow/www (#27552)
+- Upgrade dependencies in order to avoid backtracking (#27531)
+- Strengthen a bit and clarify importance of triaging issues (#27262)
+- Deduplicate type hints (#27508)
+- Add stub 'yield' to ``BaseTrigger.run`` (#27416)
+- Remove upper-bound limit to dask (#27415)
+- Limit Dask to under ``2022.10.1 ``(#27383)
+- Update old style typing (#26872)
+- Enable string normalization for docs (#27269)
+- Slightly faster up/downgrade tests (#26939)
+- Deprecate use of core get_kube_client in PodManager (#26848)
+- Reduce log level for k8s ``TCP_KEEPALIVE`` etc warnings (#26981)
+- Add memray files to gitignore / dockerignore (#27001)
+- Clean-ups around task-mapping code (#26879)
+- Move user-facing string to template (#26815)
+- add icon legend to datasets graph (#26781)
+- Bump sphinx and sphinx-autoapi (#26743)
+- Simplify ``RTIF.delete_old_records()`` (#26667)
+- migrate last react files to typescript (#26112)
+- Bump FAB to ``4.1.4`` (#26393)
+- Add min attrs version (#26408)
+- Work around pyupgrade edge cases (#26384)
+
+Doc only changes
+^^^^^^^^^^^^^^^^
+- Document dag_file_processor_timeouts metric as deprecated (#27067)
+- Drop support for PostgreSQL 10 (#27594)
+- Correct timer units to seconds from milliseconds. (#27360)
+- Update index.rst (#27529)
+- Fix wording in DAG Runs description(#27470)
+- Add note about pushing the lazy XCom proxy to XCom (#27250)
+- Document that KubernetesExecutor overwrites container args (#27450)
+- Fix BaseOperator link (#27441)
+- [docs] best-practices add use variable with template example. (#27316)
+- docs for custom view using plugin (#27244)
+- Update SLA wording to reflect it is relative to Dag Run start. (#27111)
+- Add missed import in the Trigger Rules example (#27309)
+- Update dags.rst (#27110)
+- Fix typo in ``set-up-database.rst`` (#27137)
Review Comment:
```suggestion
```
##########
RELEASE_NOTES.rst:
##########
@@ -21,6 +21,279 @@
.. towncrier release notes start
+Airflow 2.5.0 (2022-11-28)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+- ``airflow dags test`` no longer performs a backfill job.
+
+ In order to make ``airflow dags test`` more useful as a testing and
debugging tool, we no
+ longer run a backfill job and instead run a "local task runner". Users can
still backfill
+ their DAGs using the ``airflow dags backfill`` command. (#26400)
+- Airflow config section ``kubernetes`` renamed to ``kubernetes_executor``
+
+ KubernetesPodOperator no longer considers any core kubernetes config params,
so this section now only applies to kubernetes executor. Renaming it reduces
potential for confusion. (#26873)
Review Comment:
```suggestion
KubernetesPodOperator no longer considers any core kubernetes config
params, so this section now only applies to kubernetes executor. Renaming it
reduces potential for confusion.
```
##########
RELEASE_NOTES.rst:
##########
@@ -21,6 +21,279 @@
.. towncrier release notes start
+Airflow 2.5.0 (2022-11-28)
+--------------------------
+
+Significant Changes
+^^^^^^^^^^^^^^^^^^^
+
+- ``airflow dags test`` no longer performs a backfill job.
+
+ In order to make ``airflow dags test`` more useful as a testing and
debugging tool, we no
+ longer run a backfill job and instead run a "local task runner". Users can
still backfill
+ their DAGs using the ``airflow dags backfill`` command. (#26400)
+- Airflow config section ``kubernetes`` renamed to ``kubernetes_executor``
+
+ KubernetesPodOperator no longer considers any core kubernetes config params,
so this section now only applies to kubernetes executor. Renaming it reduces
potential for confusion. (#26873)
+- ``ExternalTaskSensor`` no longer hangs indefinitely when ``failed_states``
is set, an ``execute_date_fn`` is used, and some but not all of the dependent
tasks fail. Instead, an ``AirflowException`` is thrown as soon as any of the
dependent tasks fail.
+
+ Any code handling this failure in addition to timeouts should move to
cathing the ``AirflowException`` baseclass and not only the
``AirflowSensorTimeout`` subclass. (#27190)
+
+New Features
+^^^^^^^^^^^^
+- ``TaskRunner``: notify of component start and finish (#27855)
+- Add DagRun state change to the Listener plugin system(#27113)
+- Metric for raw task return codes (#27155)
+- Add logic for XComArg to pull specific map indexes (#27771)
+- Clear TaskGroup (#26658)
+- Add critical section query duration metric (#27700)
+- Add: #23880 :: Audit log for ``AirflowModelViews(Variables/Connection)``
(#24079)
+- Add postgres 15 support (#27444)
+- Expand tasks in mapped group at run time (#27491)
+- reset commits, clean submodules (#27560)
+- scheduler_job, add metric for scheduler loop timer (#27605)
+- Allow datasets to be used in taskflow (#27540)
+- Add expanded_ti_count to ti context (#27680)
+- Add user comment to task instance and dag run (#26457, #27849, #27867)
+- Enable copying DagRun JSON to clipboard (#27639)
+- Implement extra controls for SLAs (#27557)
+- add dag parsed time in DAG view (#27573)
+- Add max_wait for exponential_backoff in BaseSensor (#27597)
+- Expand tasks in mapped group at parse time (#27158)
+- Add disable retry flag on backfill (#23829)
+- Adding sensor decorator (#22562)
+- Api endpoint update ti (#26165)
+- Filtering datasets by recent update events (#26942)
+- Support Is /not Null filter for value is None on webui (#26584)
+- Add search to datasets list (#26893)
+- Split out and handle 'params' in mapped operator (#26100)
+- Add authoring API for TaskGroup mapping (#26844)
+- Add ``one_done`` trigger rule (#26146)
+- Create a more efficient airflow dag test command that also has better local
logging (#26400)
+- Support add/remove permissions to roles commands (#26338)
+- Auto tail file logs in Web UI (#26169)
+- Add triggerer info to task instance in API (#26249)
+- Flag to deserialize value on custom XCom backend (#26343)
+
+Bug Fixes
+^^^^^^^^^
+- Redirect to home view when there are no valid tags in the URL (#25715)
+- Make MappedTaskGroup depend on its expand inputs (#27876)
+- Make DagRun state updates for paused DAGs faster (#27725)
+- Don't explicitly set include_examples to False on task run command (#27813)
+- Fix menu border color (#27789)
+- Fix backfill queued task getting reset to scheduled state. (#23720)
+- Fix clearing child dag mapped tasks from parent dag (#27501)
+- Handle json encoding of ``V1Pod`` in task callback (#27609)
+- Fix ExternalTaskSensor can't check zipped dag (#27056)
+- Avoid re-fetching DAG run in TriggerDagRunOperator (#27635)
+- Continue on exception when retrieving metadata (#27665)
+- Fix double logging with some task logging handler (#27591)
+- External task sensor fail fix (#27190)
+- Replace FAB url filtering function with Airflows (#27576)
+- Fix mini scheduler expansion of mapped task (#27506)
+- Add the default None when pop actions (#27537)
+- Display parameter values from serialized dag in trigger dag view. (#27482)
+- Fix getting the dag/task ids from base executor (#27550)
+- Fix sqlalchemy primary key black-out error on DDRQ (#27538)
+- Move TriggerDagRun conf check to execute (#27035)
+- SLAMiss is nullable and not always given back when pulling task instances
(#27423)
+- Fix behavior of ``_`` when searching for DAGs (#27448)
+- Add case insensitive constraint to username (#27266)
+- Fix python external template keys (#27256)
+- Resolve trigger assignment race condition (#27072)
+- Update google_analytics.html (#27226)
+- Fix IntegrityError during webserver startup (#27297)
+- reduce extraneous task log requests (#27233)
+- Make RotatingFilehandler used in DagProcessor non-caching (#27223)
+- set executor.job_id to BackfillJob.id for backfills (#27020)
+- Fix some bug in web ui dags list page (auto-refresh & jump search null
state) (#27141)
+- Listener: Set task on sqlalchemy taskinstance object (#27167)
+- fix yarn warning from d3-color (#27139)
+- Don't overwrite connection extra with invalid json (#27142)
+- Simplify origin string cleaning (#27143)
+- Don't re-patch pods that are already controlled by current worker (#26778)
+- Add separate error handler for 405(Method not allowed) errors (#26880)
+- Avoid 500 on dag redirect (#27064)
+- Make tracebacks opt-in (#27059)
+- Filter dataset dependency data on webserver (#27046)
+- Remove double collection of dags in ``airflow dags reserialize`` (#27030)
+- Fix auto refresh for graph view (#26926)
+- Add missing AUTOINC/SERIAL for FAB tables (#26885)
+- fix next run dataset modal links (#26897)
+- Change dag audit log sort by date from asc to desc (#26895)
+- Bump min version of jinja2 (#26866)
+- Add missing colors to state_color_mapping jinja global (#26822)
+- Retry on Airflow Schedule DAG Run DB Deadlock (#26347)
+- Fix running debuggers inside ``airflow tasks test`` (#26806)
+- Fix warning when using xcomarg dependencies (#26801)
+- demote Removed state in priority for displaying task summaries (#26789)
+- Ensure the log messages from operators during parsing go somewhere (#26779)
+- Add restarting state to TaskState Enum in REST API (#26776)
+- Remove DAG parsing from StandardTaskRunner (#26750)
+- Allow retrieving error message from data.detail (#26762)
+- Fix airflow tasks run --local when dags_folder differs from that of
processor (#26509)
+- Handle mapped tasks in task duration chart (#26722)
+- Remove TaskFail duplicates check (#26714)
+- Fix task duration cumulative chart (#26717)
+- Fix non-hidden cumulative chart on duration view (#26716)
+- Fixed broken URL for docker-compose.yaml (#26721)
+- When rendering template, unmap task in context (#26702)
+- Fix scroll overflow for ConfirmDialog (#26681)
+- Use COALESCE when ordering runs to handle NULL (#26626)
+- Fix xcom arg.py .zip bug (#26636)
+- Allow MapXComArg to resolve after serialization (#26591)
+- Resolve deprecation warning re Table.exists() (#26616)
+- Check user is active (#26635)
+- Resolve warning about DISTINCT ON query on dags view (#26608)
+- Suppress SQLALCHEMY_TRACK_MODIFICATIONS warning in db init (#26617)
+- Fix Deferrable stuck as "scheduled" during backfill (#26205)
+- No missing user warning for public admin (#26611)
+- Fix grid view log try numbers (#26556)
+- Fix 404 taskInstance errors and split into two tables (#26575)
+- Log warning when secret backend kwargs is invalid (#26580)
+- fix WSGI root app (#26549)
+- Fix browser warning of improper thead usage (#26551)
+- Correctly set json_provider_class on Flask app so it uses our encoder
(#26554)
+- Revert "No grid auto-refresh for backfill dag runs (#25042)" (#26463)
+- Fix deadlock when mapped task with removed upstream is rerun (#26518)
+- ExecutorConfigType should be cacheable (#26498)
+- template rendering issue fix (#26390)
+- Fix proper joining of the path for logs retrieved from celery workers
(#26493)
+- Don't update backfill run from the scheduler (#26342)
+- Fix version for a couple configurations (#26491)
+- DAG Deps extends ``base_template`` (#26439)
+- Fix UI redirect (#26409)
+- Clear autoregistered DAGs if there are any import errors (#26398)
+- Require dag_id arg for dags list-runs (#26357)
+- Properly build URL to retrieve logs independently from system (#26337)
+- Add the dag_id to AirflowDagCycleException message (#26204)
+- Fix 'from airflow import version' lazy import (#26239)
+
+Improvements
+^^^^^^^^^^^^
+- Allow depth-first execution (#27827)
+- UI: Update offset height if data changes (#27865)
+- Improve TriggerRuleDep typing and readability (#27810)
+- Make views requiring session, keyword only args (#27790)
+- Optimize ``TI.xcom_pull()`` with explicit task_ids and map_indexes (#27699)
+- Allow hyphens in pod id used by k8s executor (#27737)
+- optimise task instances filtering (#27102)
+- Use context managers to simplify log serve management (#27756)
+- Fix formatting leftovers (#27750)
+- Improve task deadlock messaging (#27734)
+- Improve "sensor timeout" messaging (#27733)
+- Replace urlparse with urlsplit (#27389)
+- Add new files to parsing queue on every loop of dag processing (#27060)
+- Make Kubernetes Executor & Scheduler resilient to error during PMH execution
(#27611)
+- Separate dataset deps into individual graphs (#27356)
+- Use log.exception where more economical than log.error (#27517)
+- Move validation ``branch_task_ids`` into ``SkipMixin`` (#27434)
+- Coerce LazyXComAccess to list when pushed to XCom (#27251)
+- Update cluster-policies.rst docs (#27362)
+- Add warning if connection type already registered within the provider
(#27520)
+- Activate debug logging in commands with --verbose option (#27447)
+- Add classic examples for Python Operators (#27403)
+- change ``.first()`` to ``.scalar()`` (#27323)
+- Improve reset_dag_run description (#26755)
+- Add examples and howtos about sensors (#27333)
+- Make grid view widths adjustable (#27273)
+- Sorting plugins custom menu links by category before name (#27152)
+- Simplify DagRun.verify_integrity (#26894)
+- Add mapped task group info to serialization (#27027)
+- Correct the JSON style used for Run config in Grid View (#27119)
+- No ``extra__conn_type__`` prefix required for UI behaviors (#26995)
+- Betterize dataset update blurb (#26878)
+- Rename kubernetes config section to kubernetes_executor (#26873)
+- decode params for dataset searches (#26941)
+- Get rid of the DAGRun details page & rely completely on Grid (#26837)
+- Fix scheduler crashloopbackoff when using ``hostname_callable`` (#24999)
+- Reduce log verbosity in KubernetesExecutor. (#26582)
+- Don't iterate tis list twice for no reason (#26740)
+- Clearer code for PodGenerator.deserialize_model_file (#26641)
+- Don't import kubernetes unless you have a V1Pod (#26496)
+- Add updated_at column to DagRun and Ti tables (#26252)
+- Move the deserialization of custom XCom Backend to 2.4.0 (#26392)
+- Avoid calculating all elements when one item is needed (#26377)
+- Add ``__future__``.annotations automatically by isort (#26383)
+- Handle list when serializing expand_kwargs (#26369)
+- Apply PEP-563 (Postponed Evaluation of Annotations) to core airflow (#26290)
+- Add more weekday operator and sensor examples #26071 (#26098)
+- Align TaskGroup semantics to AbstractOperator (#27723)
+
+Misc/Internal
+^^^^^^^^^^^^^
+- Fix old-style typing in Base Sensor (#27871)
+- Switch (back) to late imports (#27730)
+- Completed D400 for multiple folders (#27748)
+- simplify notes accordion test (#27757)
+- completed D400 for ``airflow/callbacks/* airflow/cli/*`` (#27721)
+- Completed D400 for ``airflow/api_connexion/* directory`` (#27718)
+- Completed D400 for ``airflow/listener/* directory`` (#27731)
+- Completed D400 for ``airflow/lineage/* directory`` (#27732)
+- Update API & Python Client versions (#27642)
+- Completed D400 & D401 for ``airflow/api/*`` directory (#27716)
+- Completed D400 for multiple folders (#27722)
+- Bump minimatch from ``3.0.4 to 3.0.8`` in ``/airflow/www`` (#27688)
+- Bump loader-utils from ``1.4.1 to 1.4.2 ``in ``/airflow/www`` (#27697)
+- Disable nested task mapping for now (#27681)
+- bump alembic minimum version (#27629)
+- remove unused code.html (#27585)
+- Enable python string normalization everywhere (#27588)
+- Bump loader-utils from ``1.4.0 to 1.4.1`` in /airflow/www (#27552)
+- Upgrade dependencies in order to avoid backtracking (#27531)
+- Strengthen a bit and clarify importance of triaging issues (#27262)
+- Deduplicate type hints (#27508)
+- Add stub 'yield' to ``BaseTrigger.run`` (#27416)
+- Remove upper-bound limit to dask (#27415)
+- Limit Dask to under ``2022.10.1 ``(#27383)
+- Update old style typing (#26872)
+- Enable string normalization for docs (#27269)
+- Slightly faster up/downgrade tests (#26939)
+- Deprecate use of core get_kube_client in PodManager (#26848)
+- Reduce log level for k8s ``TCP_KEEPALIVE`` etc warnings (#26981)
+- Add memray files to gitignore / dockerignore (#27001)
+- Clean-ups around task-mapping code (#26879)
+- Move user-facing string to template (#26815)
+- add icon legend to datasets graph (#26781)
+- Bump sphinx and sphinx-autoapi (#26743)
+- Simplify ``RTIF.delete_old_records()`` (#26667)
+- migrate last react files to typescript (#26112)
+- Bump FAB to ``4.1.4`` (#26393)
+- Add min attrs version (#26408)
+- Work around pyupgrade edge cases (#26384)
+
+Doc only changes
+^^^^^^^^^^^^^^^^
+- Document dag_file_processor_timeouts metric as deprecated (#27067)
+- Drop support for PostgreSQL 10 (#27594)
+- Correct timer units to seconds from milliseconds. (#27360)
+- Update index.rst (#27529)
+- Fix wording in DAG Runs description(#27470)
+- Add note about pushing the lazy XCom proxy to XCom (#27250)
+- Document that KubernetesExecutor overwrites container args (#27450)
+- Fix BaseOperator link (#27441)
+- [docs] best-practices add use variable with template example. (#27316)
+- docs for custom view using plugin (#27244)
+- Update SLA wording to reflect it is relative to Dag Run start. (#27111)
+- Add missed import in the Trigger Rules example (#27309)
+- Update dags.rst (#27110)
Review Comment:
Typo, can skip this one.
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
