This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new bfec97519d5 Improve onboarding instructions for PyCharm/IntelliJ users
(#48213)
bfec97519d5 is described below
commit bfec97519d5cb29e7908318fd808d45b5a70deee
Author: Jarek Potiuk <[email protected]>
AuthorDate: Mon Mar 24 21:43:49 2025 +0100
Improve onboarding instructions for PyCharm/IntelliJ users (#48213)
---
.../contributors_quick_start_pycharm.rst | 92 ++++++++++++++-------
.../images/pycharm_add_interpreter.png | Bin 118539 -> 333057 bytes
setup_idea.py | 62 +++++++++-----
tests/sdk/__init__.py | 16 ----
tests/sdk/api/__init__.py | 16 ----
tests/sdk/execution_time/__init__.py | 16 ----
6 files changed, 101 insertions(+), 101 deletions(-)
diff --git
a/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst
b/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst
index 7d13dd1c8cd..4a676705307 100644
--- a/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst
+++ b/contributing-docs/quick-start-ide/contributors_quick_start_pycharm.rst
@@ -29,7 +29,6 @@ Setup your project
alt="Cloning github fork to Pycharm">
</div>
-
2. Paste the repository link in the URL field and submit.
.. raw:: html
@@ -39,52 +38,37 @@ Setup your project
alt="Cloning github fork to Pycharm">
</div>
-3. Configure the source root directories well as for ``task-sdk`` and
``devel-common``.
- You also have to set "source" and "tests" root directories for each
provider you want to develop (!).
+3. Synchronize local ``.venv`` virtualenv using uv
- This is important in Airflow 3.0 we split ``task-sdk``, ``devel-common``
and each provider to be separate
- distribution - each with separate ``pyproject.toml`` file, so you need to
separately
- add "src" and "tests" directories for each provider you develop to be
respectively
- "source roots" and "test roots".
+ .. code-block:: bash
- This might be improve and might be better automated in the future, but for
now you need to do it
- for each provider separately.
+ $ uv sync
- .. raw:: html
+This will create ``.venv`` virtual environment in the project root directory
and install all the dependencies of
+airflow core. If you plan to work on providers, at this time you can install
dependencies for all providers:
- <div align="center" style="padding-bottom:10px">
- <img src="images/pycharm_add_provider_sources_and_tests.png"
- alt="Adding Source Root directories to Pycharm">
- </div>
+ .. code-block:: bash
- You also need to add ``task-sdk`` sources (and ``devel-common`` in similar
way).
+ $ uv sync --all-packages
- .. raw:: html
+Or for specific provider and it's cross-provider dependencies:
- <div align="center" style="padding-bottom:10px">
- <img src="images/pycharm_add_task-sdk_sources.png"
- alt="Adding Source Root directories to Pycharm">
- </div>
-
-4. Once step 3 is done it is recommended to invalidate caches and restart
Pycharm.
+ .. code-block:: bash
- .. raw:: html
+ $ uv sync --packages apache-airflow-provider-amazon
- <div align="center" style="padding-bottom:10px">
- <img src="images/pycharm_invalidate_caches.png"
- alt="Invalidate caches and restart Pycharm">
- </div>
+Next: Configure your IDEA project.
-5. An alternative way to add source roots is to configure the ``airflow.iml``
file under ``.idea`` directory and update the
- ``module.xml`` file:
+3. The fastest way to add source roots is to configure the ``airflow.iml``
file under ``.idea`` directory and update the
+ ``module.xml`` file using the ``setup_idea.py`` script:
To setup the source roots for all the modules that exist in the project,
you can run the following command:
This needs to done on the airflow repository root directory. It overwrites
the existing ``.idea/airflow.iml`` and
- ``.idea/modules.xml`` files.
+ ``.idea/modules.xml`` files if they exist.
.. code-block:: bash
- $ python setup_idea.py
+ $ uv run setup_idea.py
Then Restart the PyCharm/IntelliJ IDEA.
@@ -102,6 +86,52 @@ Setup your project
alt="modules.xml">
</div>
+4. Alternatively, you can configure your project manually. Configure the
source root directories well
+ as for ``airflow-core`` ``task-sdk``, ``airflow-ctl`` and ``devel-common``.
You also have to set
+ "source" and "tests" root directories for each provider you want to develop
(!).
+
+ In Airflow 3.0 we split ``airflow-core``, ``task-sdk``, ``airflow-ctl``,
``devel-common``,
+ and each provider to be separate distribution - each with separate
``pyproject.toml`` file,
+ so you need to separately add ``src`` and ``tests`` directories for each
provider you develop
+ to be respectively "source roots" and "test roots".
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/pycharm_add_provider_sources_and_tests.png"
+ alt="Adding Source Root directories to Pycharm">
+ </div>
+
+ You also need to add ``task-sdk`` sources (and ``devel-common`` in similar
way).
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/pycharm_add_task-sdk_sources.png"
+ alt="Adding Source Root directories to Pycharm">
+ </div>
+
+5. Once step 3 or 4 is done you should configure python interpreter for your
PyCharm/IntelliJ to use
+ the virtualenv created by ``uv sync``.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/pycharm_add_interpreter.png"
+ alt="Configuring Python Interpreter">
+ </div>
+
+6. It is recommended to invalidate caches and restart PyCharm after setting up
the project.
+
+ .. raw:: html
+
+ <div align="center" style="padding-bottom:10px">
+ <img src="images/pycharm_invalidate_caches.png"
+ alt="Invalidate caches and restart Pycharm">
+ </div>
+
+
+
Setting up debugging
####################
diff --git
a/contributing-docs/quick-start-ide/images/pycharm_add_interpreter.png
b/contributing-docs/quick-start-ide/images/pycharm_add_interpreter.png
index 207bba32efc..70cc4f10978 100644
Binary files
a/contributing-docs/quick-start-ide/images/pycharm_add_interpreter.png and
b/contributing-docs/quick-start-ide/images/pycharm_add_interpreter.png differ
diff --git a/setup_idea.py b/setup_idea.py
index 2f8e75d2b6b..2bbb2d23bac 100644
--- a/setup_idea.py
+++ b/setup_idea.py
@@ -14,9 +14,18 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+# /// script
+# requires-python = ">=3.9"
+# dependencies = [
+# "rich>=13.6.0",
+# ]
+# ///
from __future__ import annotations
-import os
+from pathlib import Path
+
+from rich import print
+from rich.prompt import Confirm
iml_xml_template = """<?xml version="1.0" encoding="UTF-8"?>
<module type="PYTHON_MODULE" version="4">
@@ -57,43 +66,52 @@ module_xml_template = """<?xml version="1.0"
encoding="UTF-8"?>
source_root_module_patter: str = '<sourceFolder
url="file://$MODULE_DIR$/{path}" isTestSource="{status}" />'
-source_root_modules: list[str] = ["airflow-core", "airflow-ctl", "dev/breeze",
"task-sdk"]
+source_root_modules: list[str] = ["airflow-core", "airflow-ctl", "task-sdk",
"devel-common", "dev/breeze"]
all_module_paths: list[str] = []
-ROOT_DIR: str = "./providers"
+ROOT_AIRFLOW_FOLDER_PATH = Path(__file__).parent
+IDEA_FOLDER_PATH = ROOT_AIRFLOW_FOLDER_PATH / ".idea"
+AIRFLOW_IML_FILE = IDEA_FOLDER_PATH / "airflow.iml"
+MODULES_XML_FILE = IDEA_FOLDER_PATH / "modules.xml"
+
+ROOT_PROVIDERS_FOLDER_PATH = ROOT_AIRFLOW_FOLDER_PATH / "providers"
def setup_idea():
# Providers discovery
- for dirpath, _, filenames in os.walk(ROOT_DIR):
- if "pyproject.toml" in filenames:
- relative_path = os.path.relpath(dirpath, ROOT_DIR)
- source_root_modules.append(f"providers/{relative_path}")
+ for pyproject_toml_file in
ROOT_PROVIDERS_FOLDER_PATH.rglob("pyproject.toml"):
+ relative_path =
pyproject_toml_file.relative_to(ROOT_AIRFLOW_FOLDER_PATH).parent.as_posix()
+ source_root_modules.append(f"{relative_path}")
source_root_modules.sort()
for module in source_root_modules:
+ print(f"[green]Adding[/] module: [blue]{module}[/]")
all_module_paths.append(source_root_module_patter.format(path=f"{module}/src",
status="false"))
-
all_module_paths.append(source_root_module_patter.format(path=f"{module}/test",
status="true"))
-
- source_root_module_path: str = "\n\t\t".join(all_module_paths)
+
all_module_paths.append(source_root_module_patter.format(path=f"{module}/tests",
status="true"))
- base_source_root_xml: str =
iml_xml_template.format(SOURCE_ROOT_MODULE_PATH=source_root_module_path)
+ source_root_module_path = "\n\t\t".join(all_module_paths)
- with open(".idea/airflow.iml", "w") as file:
- file.write(base_source_root_xml)
+ base_source_root_xml =
iml_xml_template.format(SOURCE_ROOT_MODULE_PATH=source_root_module_path)
- with open(".idea/modules.xml", "w") as file:
- file.write(module_xml_template)
+ IDEA_FOLDER_PATH.mkdir(exist_ok=True)
+ AIRFLOW_IML_FILE.write_text(base_source_root_xml)
+ MODULES_XML_FILE.write_text(module_xml_template)
if __name__ == "__main__":
- user_input = input(
- "This script will overwrites the .idea/airflow.iml and
.idea/modules.xml files. Press Enter Y/N to continue: "
- )
- if user_input.lower() == "y":
+ print("\n[yellow]Warning!!![/] This script will update the
PyCharm/IntelliJ IDEA configuration files:\n")
+ print(f"* {AIRFLOW_IML_FILE}")
+ print(f"* {MODULES_XML_FILE}\n")
+ should_continue = Confirm.ask("Overwrite the files?")
+ if should_continue:
+ print()
setup_idea()
- print("Updated airflow.iml and modules.xml files, Now restart the
PyCharm/IntelliJ IDEA")
+ print("\n[green]Success\n")
+ print(
+ f"Updated {AIRFLOW_IML_FILE} and {MODULES_XML_FILE} files. "
+ f"Now restart the PyCharm/IntelliJ IDEA\n"
+ )
else:
- print("Not updating airflow.iml and modules.xml files")
- exit(0)
+ print("[yellow]Skipped\n")
+ print(f"Not updated {AIRFLOW_IML_FILE} and {MODULES_XML_FILE} files\n")
diff --git a/tests/sdk/__init__.py b/tests/sdk/__init__.py
deleted file mode 100644
index 13a83393a91..00000000000
--- a/tests/sdk/__init__.py
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
diff --git a/tests/sdk/api/__init__.py b/tests/sdk/api/__init__.py
deleted file mode 100644
index 13a83393a91..00000000000
--- a/tests/sdk/api/__init__.py
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
diff --git a/tests/sdk/execution_time/__init__.py
b/tests/sdk/execution_time/__init__.py
deleted file mode 100644
index 13a83393a91..00000000000
--- a/tests/sdk/execution_time/__init__.py
+++ /dev/null
@@ -1,16 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
