This is an automated email from the ASF dual-hosted git repository. ephraimanierobi pushed a commit to branch v2-7-test in repository https://gitbox.apache.org/repos/asf/airflow.git
commit a9b5e1cf42ca2eb45f2e5e83876eb8cc88abd9c2 Author: Ephraim Anierobi <[email protected]> AuthorDate: Wed Aug 2 15:01:41 2023 +0100 Update RELEASE_NOTES.rst for 2.7.0 --- RELEASE_NOTES.rst | 268 ++++++++++++++++++++++++++++++++++++ docs/docker-stack/changelog.rst | 8 ++ newsfragments/30762.improvement.rst | 1 - newsfragments/32052.significant.rst | 17 --- newsfragments/32176.significant.rst | 10 -- newsfragments/32526.significant.rst | 5 - newsfragments/32572.significant.rst | 10 -- newsfragments/32767.significant.rst | 7 - 8 files changed, 276 insertions(+), 50 deletions(-) diff --git a/RELEASE_NOTES.rst b/RELEASE_NOTES.rst index d231cc864f..8a9276fafc 100644 --- a/RELEASE_NOTES.rst +++ b/RELEASE_NOTES.rst @@ -21,6 +21,274 @@ .. towncrier release notes start +Airflow 2.7.0 (2023-08-10) +-------------------------- + +Significant Changes +^^^^^^^^^^^^^^^^^^^ + +Remove Python 3.7 support (#30963) +"""""""""""""""""""""""""""""""""" +As of now, Python 3.7 is no longer supported by the Python community. +Therefore, to use Airflow 2.7.0, you must ensure your Python version is +either 3.8, 3.9, 3.10, or 3.11. + +Old Graph View is removed (#32958) +"""""""""""""""""""""""""""""""""" +The old Graph View is removed. The new Graph View is the default view now. + +Disable default allowing the testing of connections in UI, API and CLI(#32052) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" +For security reasons, the test connection functionality is disabled by default across Airflow UI, +API and CLI. The availability of the functionality can be controlled by the +``test_connection`` flag in the ``core`` section of the Airflow +configuration (``airflow.cfg``). It can also be controlled by the +environment variable ``AIRFLOW__CORE__TEST_CONNECTION``. +The following values are accepted for this config param: +1. ``Disabled``: Disables the test connection functionality and +disables the Test Connection button in the UI. +This is also the default value set in the Airflow configuration. +2. ``Enabled``: Enables the test connection functionality and +activates the Test Connection button in the UI. +3. ``Hidden``: Disables the test connection functionality and +hides the Test Connection button in UI. +For more information on capabilities of users, see the documentation: +https://airflow.apache.org/docs/apache-airflow/stable/security/security_model.html#capabilities-of-authenticated-ui-users +It is strongly advised to **not** enable the feature until you make sure that only +highly trusted UI/API users have "edit connection" permissions. +The ``xcomEntries`` API disables support for the ``deserialize`` flag by default (#32176) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" +For security reasons, the ``/dags/*/dagRuns/*/taskInstances/*/xcomEntries/*`` +API endpoint now disables the ``deserialize`` option to deserialize arbitrary +XCom values in the webserver. For backward compatibility, server admins may set +the ``[api] enable_xcom_deserialize_support`` config to *True* to enable the +flag and restore backward compatibility. + +However, it is strongly advised to **not** enable the feature, and perform +deserialization at the client side instead. + +Default name of the Celery application changed from ``airflow.executors.celery_executor`` to ``airflow.providers.celery.executors.celery_executor`` (#32526) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" +You should change both your configuration and Health check command to use the new name: + * in configuration (``celery_app_name`` configuration in ``celery`` section) use ``airflow.providers.celery.executors.celery_executor`` + * in your Health check command use ``airflow.providers.celery.executors.celery_executor.app`` + + +The default value for ``scheduler.max_tis_per_query`` is changed from 512 to 16 (#32572) +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" +This change is expected to make the Scheduler more responsive. + +``scheduler.max_tis_per_query`` needs to be lower than ``core.parallelism``. +If both were left to their default value previously, the effective default value of ``scheduler.max_tis_per_query`` was 32 +(because it was capped at ``core.parallelism``). + +To keep the behavior as close as possible to the old config, one can set ``scheduler.max_tis_per_query = 0``, +in which case it'll always use the value of ``core.parallelism``. + +The Kubernetes, Celery, CeleryKubernetes, LocalKubernetes, and Dask executors are moved to corresponding providers (#32767) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" +In order to use the executors, you need to install the providers: + +* for Celery executors you need to install ``apache-airflow-providers-celery`` package >= 3.3.0 +* for Kubernetes executors you need to install ``apache-airflow-providers-cncf-kubernetes`` package >= 7.4.0 +* For Dask executors you need to install ``apache-airflow-providers-daskexecutor`` package in any version +You can achieve it also by installing airflow with ``[celery]``, ``[cncf.kubernetes]``, ``[daskexecutor]`` extras respectively. + +Users who base their images on the ``apache/airflow`` reference image (not slim) should be unaffected - the base +reference image comes with all the three providers installed. + +Improvement Changes +^^^^^^^^^^^^^^^^^^^ + +PostgreSQL only improvement: Added index on taskinstance table (#30762) +""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" +This index seems to have great positive effect in a setup with tens of millions such rows. + +New Features +"""""""""""" +- Add OpenTelemetry to Airflow (`AIP-49 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-49+milestone%3A%22Airflow+2.7.0%22>`_) +- Trigger Button - Implement Part 2 of AIP-50 (#31583) +- Automatic setup and teardown tasks (`AIP-52 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-52+milestone%3A%22Airflow+2.7.0%22>`_) +- OpenLineage in Airflow (`AIP-53 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-53+milestone%3A%22Airflow+2.7.0%22>`_) +- Extensible user management (`AIP-56 <https://github.com/apache/airflow/pulls?q=is%3Apr+is%3Amerged+label%3AAIP-56+milestone%3A%22Airflow+2.7.0%22>`_) +- New gantt tab (#31806) +- Load plugins from providers (#32692) +- Add ``ExternalBranchPythonOperator`` (#32787) +- Add option for storing configuration description in providers (#32629) +- Add an option to use a direct DB connection in KEDA when ``pgbouncer`` is enabled (#32608) +- Introduce Heartbeat Parameter to Allow ``Per-LocalTaskJob`` Configuration (#32313) +- Add Executors discovery and documentation (#32532) +- Add JobState for job state constants (#32549) +- Add config to disable the 'deserialize' XCom API flag (#32176) +- Show task instance in web UI by custom operator name (#31852) +- Add default_deferrable config (#31712) +- Introducing ``AirflowClusterPolicySkipDag`` exception (#32013) +- Use ``reactflow`` for datasets graph (#31775) +- Add an option to load the dags from db for command tasks run (#32038) +- Add version of ``chain`` which doesn't require matched lists (#31927) +- Use operator_name instead of task_type in UI (#31662) +- Add ``--retry`` and ``--retry-delay`` to ``airflow db check`` (#31836) +- Allow skipped task state task_instance_schema.py (#31421) +- Add a new config for celery result_backend engine options (#30426) +- UI Add Cluster Activity Page (#31123, #32446) +- Adding keyboard shortcuts to common actions (#30950) +- Adding more information to kubernetes executor logs (#29929) +- Add support for configuring custom alembic file (#31415) +- Add running and failed status tab for DAGs on the UI (#30429) +- Add multi-select, proposals and labels for trigger form (#31441) +- Making webserver config customizable (#29926) +- Render DAGCode in the Grid View as a tab (#31113) +- Add rest endpoint to get option of configuration (#31056) +- Add ``section`` query param in get config rest API (#30936) +- Create metrics to track ``Scheduled->Queued->Running`` task state transition times (#30612) +- Mark Task Groups as Success/Failure (#30478) +- Add CLI command to list the provider trigger info (#30822) +- Add Fail Fast feature for DAGs (#29406) + +Improvements +"""""""""""" +- Always show gantt and code tabs (#33029) +- Move listener success hook to after SQLAlchemy commit (#32988) +- Rename ``db upgrade`` to ``db migrate`` and add ``connections create-default-connections`` (#32810) +- Remove old gantt chart and redirect to grid views gantt tab (#32908) +- Adjust graph zoom based on selected task (#32792) +- Call listener on_task_instance_running after rendering templates (#32716) +- Display execution_date in graph view task instance tooltip. (#32527) +- Allow configuration to be contributed by providers (#32604, #32755, #32812) +- Reduce default for max TIs per query, enforce ``<=`` parallelism (#32572) +- Store config description in Airflow configuration object (#32669) +- Use ``isdisjoint`` instead of ``not intersection`` (#32616) +- Speed up calculation of leaves and roots for task groups (#32592) +- Kubernetes Executor Load Time Optimizations (#30727) +- Save DAG parsing time if dag is not schedulable (#30911) +- Updates health check endpoint to include ``dag_processor`` status. (#32382) +- Disable default allowing the testing of connections in UI, API and CLI (#32052) +- Fix config var types under the scheduler section (#32132) +- Allow to sort Grid View alphabetically (#32179) +- Add hostname to triggerer metric ``[triggers.running]`` (#32050) +- Improve DAG ORM cleanup code (#30614) +- ``TriggerDagRunOperator``: Add ``wait_for_completion`` to ``template_fields`` (#31122) +- Open links in new tab that take us away from Airflow UI (#32088) +- Only show code tab when a task is not selected (#31744) +- Add descriptions for celery and dask cert configs (#31822) +- ``PythonVirtualenvOperator`` termination log in alert (#31747) +- Migration of all DAG details to existing grid view dag details panel (#31690) +- Add a diagram to help visualize timer metrics (#30650) +- Celery Executor load time optimizations (#31001) +- Update code style for ``airflow db`` commands to SQLAlchemy 2.0 style (#31486) +- Mark uses of md5 as "not-used-for-security" in FIPS environments (#31171) +- Add pydantic support to serde (#31565) +- Enable search in note column in DagRun and TaskInstance (#31455) +- Save scheduler execution time by adding new Index idea for dag_run (#30827) +- Save scheduler execution time by caching dags (#30704) +- Support for sorting DAGs by Last Run Date in the web UI (#31234) +- Better typing for Job and JobRunners (#31240) +- Add sorting logic by created_date for fetching triggers (#31151) +- Remove DAGs.can_create on access control doc, adjust test fixture (#30862) +- Split Celery logs into stdout/stderr (#30485) +- Decouple metrics clients and ``validators`` into their own modules (#30802) +- Description added for pagination in ``get_log`` api (#30729) +- Optimize performance of scheduling mapped tasks (#30372) +- Add sentry transport configuration option (#30419) +- Better message on deserialization error (#30588) + +Bug Fixes +""""""""" +- OpenAPI Spec fix nullable alongside ``$ref`` (#32887) +- Make the decorators of ``PythonOperator`` sub-classes extend its decorator (#32845) +- Fix check if ``virtualenv`` is installed in ``PythonVirtualenvOperator`` (#32939) +- Unwrap Proxy before checking ``__iter__`` in is_container() (#32850) +- Override base log folder by using task handler's base_log_folder (#32781) +- Catch arbitrary exception from run_job to prevent zombie scheduler (#32707) +- Fix depends_on_past work for dynamic tasks (#32397) +- Sort extra_links for predictable order in UI. (#32762) +- Fix prefix group false graph (#32764) +- Fix bad delete logic for dagruns (#32684) +- Fix bug in prune_dict where empty dict and list would be removed even in strict mode (#32573) +- Add explicit browsers list and correct rel for blank target links (#32633) +- Handle returned None when multiple_outputs is True (#32625) +- Fix returned value when ShortCircuitOperator condition is falsy and there is not downstream tasks (#32623) +- Fix returned value when ShortCircuitOperator condition is falsy (#32569) +- Fix rendering of ``dagRunTimeout`` (#32565) +- Fix permissions on ``/blocked`` endpoint (#32571) +- Bugfix, prevent force of unpause on trigger DAG (#32456) +- Fix data interval in ``cli.dags.trigger`` command output (#32548) +- Strip ``whitespaces`` from airflow connections form (#32292) +- Add timedelta support for applicable arguments of sensors (#32515) +- Fix incorrect default on ``readonly`` property in our API (#32510) +- Add xcom map_index as a filter to xcom endpoint (#32453) +- Fix CLI commands when custom timetable is used (#32118) +- Use WebEncoder to encode DagRun.conf in DagRun's list view (#32385) +- Fix logic of the skip_all_except method (#31153) +- Ensure dynamic tasks inside dynamic task group only marks the (#32354) +- Handle the cases that webserver.expose_config is set to non-sensitive-only instead of boolean value (#32261) +- Add retry functionality for handling process termination caused by database network issues (#31998) +- Adapt Notifier for sla_miss_callback (#31887) +- Fix XCOM view (#31807) +- Fix for "Filter dags by tag" flickering on initial load of dags.html (#31578) +- Fix where expanding ``resizer`` wouldn't expanse grid view (#31581) +- Fix MappedOperator-BaseOperator attr sync check (#31520) +- Always pass named ``type_`` arg to drop_constraint (#31306) +- Fix bad ``drop_constraint`` call in migrations (#31302) +- Resolving problems with redesigned grid view (#31232) +- Support ``requirepass`` redis sentinel (#30352) +- Fix webserver crash when calling get ``/config`` (#31057) + +Misc/Internal +""""""""""""" +- Remove old graph view (#32958) +- Move CeleryExecutor to the celery provider (#32526, #32628) +- Move all k8S classes to ``cncf.kubernetes`` provider (#32767, #32891) +- Refactor existence-checking SQL to helper (#32790) +- Extract Dask executor to new daskexecutor provider (#32772) +- Remove atlas configuration definition (#32776) +- Add Redis task handler (#31855) +- Move writing configuration for webserver to main (webserver limited) (#32766) +- Improve getting the query count in Airflow API endpoints (#32630) +- Remove click upper bound (#32634) +- Add D400 ``pydocstyle`` check - core Airflow only (#31297) +- D205 Support (#31742, #32575, #32213, #32212, #32591, #32449, #32450) +- Bump word-wrap from ``1.2.3 to 1.2.4`` in ``/airflow/www`` (#32680) +- Strong-type all single-state enum values (#32537) +- More strong typed state conversion (#32521) +- SQL query improvements in utils/db.py (#32518) +- Bump semver from ``6.3.0 to 6.3.1`` in ``/airflow/www`` (#32506) +- Bump jsonschema version to ``4.18.0`` (#32445) +- Bump ``stylelint`` from ``13.13.1 to 15.10.1`` in ``/airflow/www`` (#32435) +- Bump tough-cookie from ``4.0.0 to 4.1.3`` in ``/airflow/www`` (#32443) +- upgrade flask-appbuilder (#32054) +- Support ``Pydantic`` 2 (#32366) +- Limit click until we fix mypy issues (#32413) +- A couple of minor cleanups (#31890) +- Replace State usages with strong-typed ``enums`` (#31735) +- Upgrade ruff to ``0.272`` (#31966) +- Better error message when serializing callable without name (#31778) +- Improve the views module a bit (#31661) +- Remove ``asynctest`` (#31664) +- Refactor sqlalchemy queries to ``2.0`` style (#31569, #31772, #32350, #32339, #32474, #32645) +- Remove Python ``3.7`` support (#30963) +- Bring back min-airflow-version for preinstalled providers (#31469) +- Docstring improvements (#31375) +- Improve typing in SchedulerJobRunner (#31285) +- Upgrade ruff to ``0.0.262`` (#30809) +- Upgrade to MyPy ``1.2.0`` (#30687) + +Docs only changes +""""""""""""""""" +- Clean-up of our new security page (#32951) +- Cleans up Extras reference page (#32954) +- Update Dag trigger API and command docs (#32696) +- Add deprecation info to the Airflow modules and classes docstring (#32635) +- Formatting installation doc to improve readability (#32502) +- Fix triggerer HA doc (#32454) +- Add type annotation to code examples (#32422) +- Document cron and delta timetables (#32392) +- Update index.rst doc to correct grammar (#32315) +- Fixing small typo in python.py (#31474) +- Separate out and clarify policies for providers (#30657) +- Fix docs: add an "apache" prefix to pip install (#30681) + + Airflow 2.6.3 (2023-07-10) -------------------------- diff --git a/docs/docker-stack/changelog.rst b/docs/docker-stack/changelog.rst index 619dfd9e74..a7f9653c29 100644 --- a/docs/docker-stack/changelog.rst +++ b/docs/docker-stack/changelog.rst @@ -55,6 +55,14 @@ here so that users affected can find the reason for the changes. | | | | on 17 Jan 2022 | | +--------------+---------------------+-----------------------------------------+------------------------+----------------------------------------------+ +Airflow 2.7 +~~~~~~~~~~~ + +* 2.7.0 + + * As of now, Python 3.7 is no longer supported by the Python community. Therefore, to use Airflow 2.7.0, you must ensure your Python version is + either 3.8, 3.9, 3.10, or 3.11. + Airflow 2.6 ~~~~~~~~~~~~~ diff --git a/newsfragments/30762.improvement.rst b/newsfragments/30762.improvement.rst deleted file mode 100644 index 36aa3ca35b..0000000000 --- a/newsfragments/30762.improvement.rst +++ /dev/null @@ -1 +0,0 @@ -PostgreSQL only improvement: Added index on taskinstance table. This index seems to have great positive effect in a setup with tens of millions such rows. diff --git a/newsfragments/32052.significant.rst b/newsfragments/32052.significant.rst deleted file mode 100644 index d4ec9a934f..0000000000 --- a/newsfragments/32052.significant.rst +++ /dev/null @@ -1,17 +0,0 @@ -Disable default allowing the testing of connections in UI, API and CLI - -The test connection functionality is disabled by default across Airflow UI, -API and CLI. The availability of the functionality can be controlled by the -``test_connection`` flag in the ``core`` section of the Airflow -configuration (``airflow.cfg``). It can also be controlled by the -environment variable ``AIRFLOW__CORE__TEST_CONNECTION``. -The following values are accepted for this config param: -1. ``Disabled``: Disables the test connection functionality and -disables(greys out) the Test Connection button in the UI. -This is also the default value set in the Airflow configuration. -2. ``Enabled``: Enables the test connection functionality and -activates the Test Connection button in the UI. -3. ``Hidden``: Disables the test connection functionality and -hides the Test Connection button in UI. -For more information on capabilities of users, see the documentation: -https://airflow.apache.org/docs/apache-airflow/stable/security/index.html#capabilities-of-authenticated-ui-users diff --git a/newsfragments/32176.significant.rst b/newsfragments/32176.significant.rst deleted file mode 100644 index 492d5f955d..0000000000 --- a/newsfragments/32176.significant.rst +++ /dev/null @@ -1,10 +0,0 @@ -The ``xcomEntries`` API disables support for the ``deserialize`` flag by default - -For security reasons, the ``/dags/*/dagRuns/*/taskInstances/*/xcomEntries/*`` -API endpoint now disables the ``deserialize`` option to deserialize arbitrary -XCom values in the webserver. For backward compatibility, server admins may set -the ``[api] enable_xcom_deserialize_support`` config to *True* to enable the -flag and restore backward compatibility. - -However, it is strongly advised to **not** enable the feature, and perform -deserialization at the client side instead. diff --git a/newsfragments/32526.significant.rst b/newsfragments/32526.significant.rst deleted file mode 100644 index 8b3a71518e..0000000000 --- a/newsfragments/32526.significant.rst +++ /dev/null @@ -1,5 +0,0 @@ -Default name of the Celery application changed from ``airflow.executors.celery_executor`` to ``airflow.providers.celery.executors.celery_executor`` - -You should change both your configuration and Health check command to use the new name: - * in configuration (``celery_app_name`` configuration in ``celery`` section) use ``airflow.providers.celery.executors.celery_executor`` - * in your Health check command use ``airflow.providers.celery.executors.celery_executor.app`` diff --git a/newsfragments/32572.significant.rst b/newsfragments/32572.significant.rst deleted file mode 100644 index 3105e98315..0000000000 --- a/newsfragments/32572.significant.rst +++ /dev/null @@ -1,10 +0,0 @@ -The default value for ``scheduler.max_tis_per_query`` is changed from 512 to 16. - -This change is expected to make the Scheduler more responsive. - -``scheduler.max_tis_per_query`` needs to be lower than ``core.parallelism``. -If both were left to their default value previously, the effective default value of ``scheduler.max_tis_per_query`` was 32 -(because it was capped at ``core.parallelism``). - -To keep the behavior as close as possible to the old config, one can set ``scheduler.max_tis_per_query = 0``, -in which case it'll always use the value of ``core.parallelism``. diff --git a/newsfragments/32767.significant.rst b/newsfragments/32767.significant.rst deleted file mode 100644 index ec4420488a..0000000000 --- a/newsfragments/32767.significant.rst +++ /dev/null @@ -1,7 +0,0 @@ -The Kubernetes, Celery, CeleryKubernetes, LocalKubernetes, and Dask executors are moved to corresponding providers. - -In order to use the executors, you need to install the providers: - -* for Celery executors you need to install ``apache-airflow-providers-celery`` package >= 3.3.0 -* for Kubernetes executors you need to install ``apache-airflow-providers-cncf-kubernetes`` package >= 7.4.0 -* For Dask executors you need to install ``apache-airflow-providers-daskexecutor`` package in any version
