sjyangkevin commented on issue #54714: URL: https://github.com/apache/airflow/issues/54714#issuecomment-3708548445
Hi @Lee-W and @amoghrajesh , I've created a PR in the ruff repository: https://github.com/astral-sh/ruff/pull/22376. Please let me know if you have any feedback when you have a chance to review. A new code `AIR321` is created to catch and/or fix deprecated imports in Airflow 3.1. Below is the list of imports included in the PR at the moment. ### airflow.utils - [x] **(AIR321)** airflow.utils.setup_teardown.BaseSetupTeardownContext → airflow.sdk.definitions._internal.setup_teardown.BaseSetupTeardownContext - [x] **(AIR321)** airflow.utils.setup_teardown.SetupTeardownContext → airflow.sdk.definitions._internal.setup_teardown.SetupTeardownContext - [x] **(AIR321)** airflow.utils.xcom.XCOM_RETURN_KEY → airflow.models.xcom.XCOM_RETURN_KEY - [x] **(AIR321)** airflow.utils.task_group.TaskGroup → airflow.sdk.TaskGroup - [x] **(AIR321)** airflow.utils.timezone.* → airflow.sdk.timezone > - airflow.utils.timezone.coerce_datetime → airflow.sdk.timezone.coerce_datetime > - airflow.utils.timezone.convert_to_utc → airflow.sdk.timezone.convert_to_utc > - airflow.utils.timezone.datetime → airflow.sdk.timezone.datetime > - airflow.utils.timezone.make_naive → airflow.sdk.timezone.make_naive > - airflow.utils.timezone.parse → airflow.sdk.timezone.parse > - airflow.utils.timezone.utc → airflow.sdk.timezone.utc > - airflow.utils.timezone.utcnow → airflow.sdk.timezone.utcnow - [x] **(AIR321)** airflow.utils.decorators.remove_task_decorator → airflow.sdk.definitions._internal.decorators.remove_task_decorator - [x] **(AIR321)** airflow.utils.decorators.fixup_decorator_warning_stack → airflow.sdk.definitions._internal.decorators.fixup_decorator_warning_stack ### airflow.models - [x] **(AIR321)** airflow.models.abstractoperator.AbstractOperator → airflow.sdk.definitions._internal.abstractoperator.AbstractOperator - [x] **(AIR321)** airflow.models.abstractoperator.NotMapped → airflow.sdk.definitions._internal.abstractoperator.NotMapped - [x] **(AIR321)** airflow.models.abstractoperator.TaskStateChangeCallback → airflow.sdk.definitions._internal.abstractoperator.TaskStateChangeCallback - [x] **(AIR321)** airflow.models.baseoperator.BaseOperator → airflow.sdk.bases.operator.BaseOperator ### airflow.macros - [x] **(AIR321)** airflow.macros.* → airflow.sdk.execution_time.macros > - airflow.macros.ds_add → airflow.sdk.execution_time.macros.ds_add > - airflow.macros.ds_format → airflow.sdk.execution_time.macros.ds_format > - airflow.macros.datetime_diff_for_humans → airflow.sdk.execution_time.macros.datetime_diff_for_humans > - airflow.macros.ds_format_locale → airflow.sdk.execution_time.macros.ds_format_locale ### airflow.io - [x] **(AIR321)** airflow.io.get_fs → airflow.sdk.io.get_fs - [x] **(AIR321)** airflow.io.has_fs → airflow.sdk.io.has_fs - [x] **(AIR321)** airflow.io.Properties → airflow.sdk.io.Properties ## The imports need further clarification or potentially need to be fixed ### airflow.utils The following imports raise **ImportError** in Airflow 3.1.5. Looks like `get_task_group_children_getter` and `task_group_to_dict` are moved under `airflow-core/src/airflow/api_fastapi/core_api/services/ui/task_group.py`. - airflow.utils.task_group.get_task_group_children_getter → airflow.sdk.definitions.taskgroup.get_task_group_children_getter - airflow.utils.task_group.task_group_to_dict → airflow.sdk.definitions.taskgroup.task_group_to_dict ### airflow.sensors `airflow.sensors.base.poke_mode_only` is moved to `airflow.sdk.poke_mode_only`, which is implemented in [AIR311](https://github.com/astral-sh/ruff/blob/77ad1076173144e9877494eec49fc5c77d2b8c4f/crates/ruff_linter/src/rules/airflow/rules/suggested_to_update_3_0.rs#L303). However, `from airflow.sdk import poke_mode_only` raise **ImportError** in both Airflow 3.0.6 and 3.1.5. ### airflow.secrets airflow.secrets.cache.SecretCache → airflow.sdk.execution_time.cache.SecretCache The followings raise **ImportError** in Airflow 3.0.6. On the ruff side, there is a [rule](https://github.com/astral-sh/ruff/blob/77ad1076173144e9877494eec49fc5c77d2b8c4f/crates/ruff_linter/src/rules/airflow/rules/removal_in_3.rs#L727) implemented for this import. ```python from airflow.sdk import SecretCache from airflow.sdk.execution_time.cache import SecretCache ``` The above imports work in Airflow 3.1.5. I feel this would need a fix on the ruff side, to remove this rule from AIR301 and include it into AIR321. ### airflow.models I also noticed `DeprecationWarning` in Airflow 3.0.6 for the following imports. Wondering if we should make it in `AIR301` or `AIR321` (currently added under AIR321). ```python from airflow.models.abstractoperator import AbstractOperator from airflow.models.abstractoperator import NotMapped from airflow.models.abstractoperator import TaskStateChangeCallback ``` ### airflow.io This import raise **ImportError** in Airflow 3.0.6. ``` from airflow.io import Properties ``` ## Test environment ```bash # run test under Airflow 3.0.6 breeze shell --python 3.12 --use-airflow-version 3.0.6 --backend postgres --auth-manager SimpleAuthManager --db-reset # run test under Airflow 3.1.5 breeze shell --python 3.12 --use-airflow-version 3.1.5 --backend postgres --auth-manager SimpleAuthManager --db-reset ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
