This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch v2-5-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit 0c2ee0ad958e5424be084793efbacf023dccd333 Author: Ephraim Anierobi <[email protected]> AuthorDate: Thu Nov 24 16:14:49 2022 +0100 Add release notes --- RELEASE_NOTES.rst | 273 ++++++++++++++++++++++++++++++++++++ newsfragments/26400.significant.rst | 5 - newsfragments/26873.significant.rst | 3 - newsfragments/27190.significant.rst | 3 - 4 files changed, 273 insertions(+), 11 deletions(-) diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst index fe2babef13..b94810118e 100644 --- a/RELEASE_NOTES.rst +++ b/RELEASE_NOTES.rst @@ -21,6 +21,279 @@ .. towncrier release notes start +Airflow 2.5.0 (2022-11-28) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +- ``airflow dags test`` no longer performs a backfill job. + + In order to make ``airflow dags test`` more useful as a testing and debugging tool, we no + longer run a backfill job and instead run a "local task runner". Users can still backfill + their DAGs using the ``airflow dags backfill`` command. (#26400) +- Airflow config section ``kubernetes`` renamed to ``kubernetes_executor`` + + KubernetesPodOperator no longer considers any core kubernetes config params, so this section now only applies to kubernetes executor. Renaming it reduces potential for confusion. (#26873) +- ``ExternalTaskSensor`` no longer hangs indefinitely when ``failed_states`` is set, an ``execute_date_fn`` is used, and some but not all of the dependent tasks fail. Instead, an ``AirflowException`` is thrown as soon as any of the dependent tasks fail. + + Any code handling this failure in addition to timeouts should move to cathing the ``AirflowException`` baseclass and not only the ``AirflowSensorTimeout`` subclass. (#27190) + +New Features +^^^^^^^^^^^^ +- ``TaskRunner``: notify of component start and finish (#27855) +- Add DagRun state change to the Listener plugin system(#27113) +- Metric for raw task return codes (#27155) +- Add logic for XComArg to pull specific map indexes (#27771) +- Clear TaskGroup (#26658) +- Add critical section query duration metric (#27700) +- Add: #23880 :: Audit log for ``AirflowModelViews(Variables/Connection)`` (#24079) +- Add postgres 15 support (#27444) +- Expand tasks in mapped group at run time (#27491) +- reset commits, clean submodules (#27560) +- scheduler_job, add metric for scheduler loop timer (#27605) +- Allow datasets to be used in taskflow (#27540) +- Add expanded_ti_count to ti context (#27680) +- Add user comment to task instance and dag run (#26457, #27849, #27867) +- Enable copying DagRun JSON to clipboard (#27639) +- Implement extra controls for SLAs (#27557) +- add dag parsed time in DAG view (#27573) +- Add max_wait for exponential_backoff in BaseSensor (#27597) +- Expand tasks in mapped group at parse time (#27158) +- Add disable retry flag on backfill (#23829) +- Adding sensor decorator (#22562) +- Api endpoint update ti (#26165) +- Filtering datasets by recent update events (#26942) +- Support Is /not Null filter for value is None on webui (#26584) +- Add search to datasets list (#26893) +- Split out and handle 'params' in mapped operator (#26100) +- Add authoring API for TaskGroup mapping (#26844) +- Add ``one_done`` trigger rule (#26146) +- Create a more efficient airflow dag test command that also has better local logging (#26400) +- Support add/remove permissions to roles commands (#26338) +- Auto tail file logs in Web UI (#26169) +- Add triggerer info to task instance in API (#26249) +- Flag to deserialize value on custom XCom backend (#26343) + +Bug Fixes +^^^^^^^^^ +- Redirect to home view when there are no valid tags in the URL (#25715) +- Make MappedTaskGroup depend on its expand inputs (#27876) +- Make DagRun state updates for paused DAGs faster (#27725) +- Don't explicitly set include_examples to False on task run command (#27813) +- Fix menu border color (#27789) +- Fix backfill queued task getting reset to scheduled state. (#23720) +- Fix clearing child dag mapped tasks from parent dag (#27501) +- Handle json encoding of ``V1Pod`` in task callback (#27609) +- Fix ExternalTaskSensor can't check zipped dag (#27056) +- Avoid re-fetching DAG run in TriggerDagRunOperator (#27635) +- Continue on exception when retrieving metadata (#27665) +- Fix double logging with some task logging handler (#27591) +- External task sensor fail fix (#27190) +- Replace FAB url filtering function with Airflows (#27576) +- Fix mini scheduler expansion of mapped task (#27506) +- Add the default None when pop actions (#27537) +- Display parameter values from serialized dag in trigger dag view. (#27482) +- Fix getting the dag/task ids from base executor (#27550) +- Fix sqlalchemy primary key black-out error on DDRQ (#27538) +- Move TriggerDagRun conf check to execute (#27035) +- SLAMiss is nullable and not always given back when pulling task instances (#27423) +- Fix behavior of ``_`` when searching for DAGs (#27448) +- Add case insensitive constraint to username (#27266) +- Fix python external template keys (#27256) +- Resolve trigger assignment race condition (#27072) +- Update google_analytics.html (#27226) +- Fix IntegrityError during webserver startup (#27297) +- reduce extraneous task log requests (#27233) +- Make RotatingFilehandler used in DagProcessor non-caching (#27223) +- set executor.job_id to BackfillJob.id for backfills (#27020) +- Fix some bug in web ui dags list page (auto-refresh & jump search null state) (#27141) +- Listener: Set task on sqlalchemy taskinstance object (#27167) +- fix yarn warning from d3-color (#27139) +- Don't overwrite connection extra with invalid json (#27142) +- Simplify origin string cleaning (#27143) +- Don't re-patch pods that are already controlled by current worker (#26778) +- Add separate error handler for 405(Method not allowed) errors (#26880) +- Avoid 500 on dag redirect (#27064) +- Make tracebacks opt-in (#27059) +- Filter dataset dependency data on webserver (#27046) +- Remove double collection of dags in ``airflow dags reserialize`` (#27030) +- Fix auto refresh for graph view (#26926) +- Add missing AUTOINC/SERIAL for FAB tables (#26885) +- fix next run dataset modal links (#26897) +- Change dag audit log sort by date from asc to desc (#26895) +- Bump min version of jinja2 (#26866) +- Add missing colors to state_color_mapping jinja global (#26822) +- Retry on Airflow Schedule DAG Run DB Deadlock (#26347) +- Fix running debuggers inside ``airflow tasks test`` (#26806) +- Fix warning when using xcomarg dependencies (#26801) +- demote Removed state in priority for displaying task summaries (#26789) +- Ensure the log messages from operators during parsing go somewhere (#26779) +- Add restarting state to TaskState Enum in REST API (#26776) +- Remove DAG parsing from StandardTaskRunner (#26750) +- Allow retrieving error message from data.detail (#26762) +- Fix airflow tasks run --local when dags_folder differs from that of processor (#26509) +- Handle mapped tasks in task duration chart (#26722) +- Remove TaskFail duplicates check (#26714) +- Fix task duration cumulative chart (#26717) +- Fix non-hidden cumulative chart on duration view (#26716) +- Fixed broken URL for docker-compose.yaml (#26721) +- When rendering template, unmap task in context (#26702) +- Fix scroll overflow for ConfirmDialog (#26681) +- Use COALESCE when ordering runs to handle NULL (#26626) +- Fix xcom arg.py .zip bug (#26636) +- Allow MapXComArg to resolve after serialization (#26591) +- Resolve deprecation warning re Table.exists() (#26616) +- Check user is active (#26635) +- Resolve warning about DISTINCT ON query on dags view (#26608) +- Suppress SQLALCHEMY_TRACK_MODIFICATIONS warning in db init (#26617) +- Fix Deferrable stuck as "scheduled" during backfill (#26205) +- No missing user warning for public admin (#26611) +- Fix grid view log try numbers (#26556) +- Fix 404 taskInstance errors and split into two tables (#26575) +- Log warning when secret backend kwargs is invalid (#26580) +- fix WSGI root app (#26549) +- Fix browser warning of improper thead usage (#26551) +- Correctly set json_provider_class on Flask app so it uses our encoder (#26554) +- Revert "No grid auto-refresh for backfill dag runs (#25042)" (#26463) +- Fix deadlock when mapped task with removed upstream is rerun (#26518) +- ExecutorConfigType should be cacheable (#26498) +- template rendering issue fix (#26390) +- Fix proper joining of the path for logs retrieved from celery workers (#26493) +- Don't update backfill run from the scheduler (#26342) +- Fix version for a couple configurations (#26491) +- DAG Deps extends ``base_template`` (#26439) +- Fix UI redirect (#26409) +- Clear autoregistered DAGs if there are any import errors (#26398) +- Require dag_id arg for dags list-runs (#26357) +- Properly build URL to retrieve logs independently from system (#26337) +- Add the dag_id to AirflowDagCycleException message (#26204) +- Fix 'from airflow import version' lazy import (#26239) + +Improvements +^^^^^^^^^^^^ +- Allow depth-first execution (#27827) +- UI: Update offset height if data changes (#27865) +- Improve TriggerRuleDep typing and readability (#27810) +- Make views requiring session, keyword only args (#27790) +- Optimize ``TI.xcom_pull()`` with explicit task_ids and map_indexes (#27699) +- Allow hyphens in pod id used by k8s executor (#27737) +- optimise task instances filtering (#27102) +- Use context managers to simplify log serve management (#27756) +- Fix formatting leftovers (#27750) +- Improve task deadlock messaging (#27734) +- Improve "sensor timeout" messaging (#27733) +- Replace urlparse with urlsplit (#27389) +- Add new files to parsing queue on every loop of dag processing (#27060) +- Make Kubernetes Executor & Scheduler resilient to error during PMH execution (#27611) +- Separate dataset deps into individual graphs (#27356) +- Use log.exception where more economical than log.error (#27517) +- Move validation ``branch_task_ids`` into ``SkipMixin`` (#27434) +- Coerce LazyXComAccess to list when pushed to XCom (#27251) +- Update cluster-policies.rst docs (#27362) +- Add warning if connection type already registered within the provider (#27520) +- Activate debug logging in commands with --verbose option (#27447) +- Add classic examples for Python Operators (#27403) +- change ``.first()`` to ``.scalar()`` (#27323) +- Improve reset_dag_run description (#26755) +- Add examples and howtos about sensors (#27333) +- Make grid view widths adjustable (#27273) +- Sorting plugins custom menu links by category before name (#27152) +- Simplify DagRun.verify_integrity (#26894) +- Add mapped task group info to serialization (#27027) +- Correct the JSON style used for Run config in Grid View (#27119) +- No ``extra__conn_type__`` prefix required for UI behaviors (#26995) +- Betterize dataset update blurb (#26878) +- Rename kubernetes config section to kubernetes_executor (#26873) +- decode params for dataset searches (#26941) +- Get rid of the DAGRun details page & rely completely on Grid (#26837) +- Fix scheduler crashloopbackoff when using ``hostname_callable`` (#24999) +- Reduce log verbosity in KubernetesExecutor. (#26582) +- Don't iterate tis list twice for no reason (#26740) +- Clearer code for PodGenerator.deserialize_model_file (#26641) +- Don't import kubernetes unless you have a V1Pod (#26496) +- Add updated_at column to DagRun and Ti tables (#26252) +- Move the deserialization of custom XCom Backend to 2.4.0 (#26392) +- Avoid calculating all elements when one item is needed (#26377) +- Add ``__future__``.annotations automatically by isort (#26383) +- Handle list when serializing expand_kwargs (#26369) +- Apply PEP-563 (Postponed Evaluation of Annotations) to core airflow (#26290) +- Add more weekday operator and sensor examples #26071 (#26098) +- Align TaskGroup semantics to AbstractOperator (#27723) + +Misc/Internal +^^^^^^^^^^^^^ +- Fix old-style typing in Base Sensor (#27871) +- Switch (back) to late imports (#27730) +- Completed D400 for multiple folders (#27748) +- simplify notes accordion test (#27757) +- completed D400 for ``airflow/callbacks/* airflow/cli/*`` (#27721) +- Completed D400 for ``airflow/api_connexion/* directory`` (#27718) +- Completed D400 for ``airflow/listener/* directory`` (#27731) +- Completed D400 for ``airflow/lineage/* directory`` (#27732) +- Update API & Python Client versions (#27642) +- Completed D400 & D401 for ``airflow/api/*`` directory (#27716) +- Completed D400 for multiple folders (#27722) +- Bump minimatch from ``3.0.4 to 3.0.8`` in ``/airflow/www`` (#27688) +- Bump loader-utils from ``1.4.1 to 1.4.2 ``in ``/airflow/www`` (#27697) +- Disable nested task mapping for now (#27681) +- bump alembic minimum version (#27629) +- remove unused code.html (#27585) +- Enable python string normalization everywhere (#27588) +- Bump loader-utils from ``1.4.0 to 1.4.1`` in /airflow/www (#27552) +- Upgrade dependencies in order to avoid backtracking (#27531) +- Strengthen a bit and clarify importance of triaging issues (#27262) +- Deduplicate type hints (#27508) +- Add stub 'yield' to ``BaseTrigger.run`` (#27416) +- Remove upper-bound limit to dask (#27415) +- Limit Dask to under ``2022.10.1 ``(#27383) +- Update old style typing (#26872) +- Enable string normalization for docs (#27269) +- Slightly faster up/downgrade tests (#26939) +- Deprecate use of core get_kube_client in PodManager (#26848) +- Reduce log level for k8s ``TCP_KEEPALIVE`` etc warnings (#26981) +- Add memray files to gitignore / dockerignore (#27001) +- Clean-ups around task-mapping code (#26879) +- Move user-facing string to template (#26815) +- add icon legend to datasets graph (#26781) +- Bump sphinx and sphinx-autoapi (#26743) +- Simplify ``RTIF.delete_old_records()`` (#26667) +- migrate last react files to typescript (#26112) +- Bump FAB to ``4.1.4`` (#26393) +- Add min attrs version (#26408) +- Work around pyupgrade edge cases (#26384) + +Doc only changes +^^^^^^^^^^^^^^^^ +- Document dag_file_processor_timeouts metric as deprecated (#27067) +- Drop support for PostgreSQL 10 (#27594) +- Correct timer units to seconds from milliseconds. (#27360) +- Update index.rst (#27529) +- Fix wording in DAG Runs description(#27470) +- Add note about pushing the lazy XCom proxy to XCom (#27250) +- Document that KubernetesExecutor overwrites container args (#27450) +- Fix BaseOperator link (#27441) +- [docs] best-practices add use variable with template example. (#27316) +- docs for custom view using plugin (#27244) +- Update SLA wording to reflect it is relative to Dag Run start. (#27111) +- Add missed import in the Trigger Rules example (#27309) +- Update dags.rst (#27110) +- Fix typo in ``set-up-database.rst`` (#27137) +- Add kerberos environment variables to the docs (#27028) +- Update graph view and grid view on overview page (#26909) +- Fixed triple quotes in task group example (#26829) +- Documentation fixes (#26819) +- make consistency on markup title string level (#26696) +- Add documentation to dag test function (#26713) +- Fix broken URL for ``docker-compose.yaml`` (#26726) +- Add a note against use of top level code in timetable (#26649) +- Document the non-sensitive-only option for expose_config (#26507) +- Clarify owner links document (#26515) +- Fix example_datasets dag names (#26495) +- Update docs: zip-like effect is now possible in task mapping (#26435) +- Fix invalid RST in dataset concepts doc (#26434) +- changing to task decorator in docs from classic operator use (#25711) + Airflow 2.4.3 (2022-11-14) -------------------------- diff --git a/newsfragments/26400.significant.rst b/newsfragments/26400.significant.rst deleted file mode 100644 index 75285d31cd..0000000000 --- a/newsfragments/26400.significant.rst +++ /dev/null @@ -1,5 +0,0 @@ -``airflow dags test`` no longer performs a backfill job. - -In order to make ``airflow dags test`` more useful as a testing and debugging tool, we no -longer run a backfill job and instead run a "local task runner". Users can still backfill -their DAGs using the ``airflow dags backfill`` command. diff --git a/newsfragments/26873.significant.rst b/newsfragments/26873.significant.rst deleted file mode 100644 index 15d920a2f5..0000000000 --- a/newsfragments/26873.significant.rst +++ /dev/null @@ -1,3 +0,0 @@ -Airflow config section ``kubernetes`` renamed to ``kubernetes_executor`` - -KubernetesPodOperator no longer considers any core kubernetes config params, so this section now only applies to kubernetes executor. Renaming it reduces potential for confusion. diff --git a/newsfragments/27190.significant.rst b/newsfragments/27190.significant.rst deleted file mode 100644 index 2fb2dc040d..0000000000 --- a/newsfragments/27190.significant.rst +++ /dev/null @@ -1,3 +0,0 @@ -``ExternalTaskSensor`` no longer hangs indefinitely when ``failed_states`` is set, an ``execute_date_fn`` is used, and some but not all of the dependent tasks fail. Instead, an ``AirflowException`` is thrown as soon as any of the dependent tasks fail. - -Any code handling this failure in addition to timeouts should move to cathing the ``AirflowException`` baseclass and not only the ``AirflowSensorTimeout`` subclass.
