aritra24 commented on code in PR #36961: URL: https://github.com/apache/airflow/pull/36961#discussion_r1462698655
########## contribution-docs/README.rst: ########## @@ -0,0 +1,72 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Detailed contributing documentation +----------------------------------- + +The following documents provide detailed information about contributing to Airflow. + +* `Roles in Airflow Project <roles_in_airflow_project.rst>`__ describes + the roles in the Airflow project and how they relate to each other. + +* `How to communicate <how_to_communicate.rst>`__ + describes how to communicate with the community and how to get help. + +* `Contributors quick start <contributors_quick_start.rst>`__ describes + how to set up your development environment and make your first contribution. There are also more + detailed documents describing how to set up your development environment for specific IDE/environment: + + * `Contributors quick start for PyCharm <contributors_quick_start_pycharm.rst>`__ + * `Contributors quick start for VSCode <contributors_quick_start_vscode.rst>`__ + * `Contributors quick start for Codespaces <contributors_quick_start_codespaces.rst>`__ + * `Contributors quick start for GitPod <contributors_quick_start_gitpod.rst>`__ + +* `Creating issues and Pull requests <creating_issues_and_pull_requests.rst>`__ describes + when and how you can create issues and submit pull requests as well as describes the pull request guidelines + and conding standards. Review Comment: Coding* ########## contribution-docs/airflow_dependencies_and_extras.rst: ########## @@ -0,0 +1,210 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Airflow dependencies +==================== + +Airflow is not a standard python project. Most of the python projects fall into one of two types - +application or library. As described in +`this StackOverflow question <https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions>`_, +the decision whether to pin (freeze) dependency versions for a python project depends on the type. For +applications, dependencies should be pinned, but for libraries, they should be open. + +For application, pinning the dependencies makes it more stable to install in the future - because new +(even transitive) dependencies might cause installation to fail. For libraries - the dependencies should +be open to allow several different libraries with the same requirements to be installed at the same time. + +The problem is that Apache Airflow is a bit of both - application to install and library to be used when +you are developing your own operators and DAGs. + +This - seemingly unsolvable - puzzle is solved by having pinned constraints files. + +.. contents:: :local: + +Pinned constraint files +----------------------- + +.. note:: + + Only ``pip`` installation is officially supported. + + While it is possible to install Airflow with tools like `poetry <https://python-poetry.org/>`_ or + `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + There are known issues with ``bazel`` that might lead to circular dependencies when using it to install + Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing + the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that + newer versions of ``bazel`` will handle it. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. + + +By default when you install ``apache-airflow`` package - the dependencies are as open as possible while +still allowing the apache-airflow package to install. This means that ``apache-airflow`` package might fail to +install in case a direct or transitive dependency is released that breaks the installation. In such case +when installing ``apache-airflow``, you might need to provide additional constraints (for +example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``) + +There are several sets of constraints we keep: + +* 'constraints' - those are constraints generated by matching the current airflow version from sources Review Comment: Nit: These* are ########## contribution-docs/airflow_dependencies_and_extras.rst: ########## @@ -0,0 +1,210 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Airflow dependencies +==================== + +Airflow is not a standard python project. Most of the python projects fall into one of two types - +application or library. As described in +`this StackOverflow question <https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions>`_, +the decision whether to pin (freeze) dependency versions for a python project depends on the type. For +applications, dependencies should be pinned, but for libraries, they should be open. + +For application, pinning the dependencies makes it more stable to install in the future - because new +(even transitive) dependencies might cause installation to fail. For libraries - the dependencies should +be open to allow several different libraries with the same requirements to be installed at the same time. + +The problem is that Apache Airflow is a bit of both - application to install and library to be used when +you are developing your own operators and DAGs. + +This - seemingly unsolvable - puzzle is solved by having pinned constraints files. + +.. contents:: :local: + +Pinned constraint files +----------------------- + +.. note:: + + Only ``pip`` installation is officially supported. + + While it is possible to install Airflow with tools like `poetry <https://python-poetry.org/>`_ or + `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + There are known issues with ``bazel`` that might lead to circular dependencies when using it to install + Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing + the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that + newer versions of ``bazel`` will handle it. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. + + +By default when you install ``apache-airflow`` package - the dependencies are as open as possible while +still allowing the apache-airflow package to install. This means that ``apache-airflow`` package might fail to +install in case a direct or transitive dependency is released that breaks the installation. In such case +when installing ``apache-airflow``, you might need to provide additional constraints (for +example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``) + +There are several sets of constraints we keep: + +* 'constraints' - those are constraints generated by matching the current airflow version from sources Review Comment: A few of the follow-up lines have the same thing, if we chose to change it all of them need the same change. ########## contribution-docs/airflow_dependencies_and_extras.rst: ########## @@ -0,0 +1,210 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Airflow dependencies +==================== + +Airflow is not a standard python project. Most of the python projects fall into one of two types - +application or library. As described in +`this StackOverflow question <https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions>`_, +the decision whether to pin (freeze) dependency versions for a python project depends on the type. For +applications, dependencies should be pinned, but for libraries, they should be open. + +For application, pinning the dependencies makes it more stable to install in the future - because new +(even transitive) dependencies might cause installation to fail. For libraries - the dependencies should +be open to allow several different libraries with the same requirements to be installed at the same time. + +The problem is that Apache Airflow is a bit of both - application to install and library to be used when +you are developing your own operators and DAGs. + +This - seemingly unsolvable - puzzle is solved by having pinned constraints files. + +.. contents:: :local: + +Pinned constraint files +----------------------- + +.. note:: + + Only ``pip`` installation is officially supported. + + While it is possible to install Airflow with tools like `poetry <https://python-poetry.org/>`_ or + `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + There are known issues with ``bazel`` that might lead to circular dependencies when using it to install + Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing + the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that + newer versions of ``bazel`` will handle it. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. + + +By default when you install ``apache-airflow`` package - the dependencies are as open as possible while +still allowing the apache-airflow package to install. This means that ``apache-airflow`` package might fail to +install in case a direct or transitive dependency is released that breaks the installation. In such case +when installing ``apache-airflow``, you might need to provide additional constraints (for +example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``) + +There are several sets of constraints we keep: + +* 'constraints' - those are constraints generated by matching the current airflow version from sources + and providers that are installed from PyPI. Those are constraints used by the users who want to + install airflow with pip, they are named ``constraints-<PYTHON_MAJOR_MINOR_VERSION>.txt``. + +* "constraints-source-providers" - those are constraints generated by using providers installed from + current sources. While adding new providers their dependencies might change, so this set of providers + is the current set of the constraints for airflow and providers from the current main sources. + Those providers are used by CI system to keep "stable" set of constraints. They are named + ``constraints-source-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt`` + +* "constraints-no-providers" - those are constraints generated from only Apache Airflow, without any + providers. If you want to manage airflow separately and then add providers individually, you can + use those. Those constraints are named ``constraints-no-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt``. + +The first two can be used as constraints file when installing Apache Airflow in a repeatable way. +It can be done from the sources: + +from the PyPI package: + +.. code-block:: bash + + pip install apache-airflow[google,amazon,async]==2.2.5 \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.5/constraints-3.8.txt"; + +The last one can be used to install Airflow in "minimal" mode - i.e when bare Airflow is installed without +extras. + +When you install airflow from sources (in editable mode) you should use "constraints-source-providers" +instead (this accounts for the case when some providers have not yet been released and have conflicting +requirements). + +.. code-block:: bash + + pip install -e ".[devel]" \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.8.txt"; + + +This works also with extras - for example: Review Comment: Nit: also works with extras ########## contribution-docs/airflow_dependencies_and_extras.rst: ########## @@ -0,0 +1,210 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Airflow dependencies +==================== + +Airflow is not a standard python project. Most of the python projects fall into one of two types - +application or library. As described in +`this StackOverflow question <https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions>`_, +the decision whether to pin (freeze) dependency versions for a python project depends on the type. For +applications, dependencies should be pinned, but for libraries, they should be open. + +For application, pinning the dependencies makes it more stable to install in the future - because new +(even transitive) dependencies might cause installation to fail. For libraries - the dependencies should +be open to allow several different libraries with the same requirements to be installed at the same time. + +The problem is that Apache Airflow is a bit of both - application to install and library to be used when +you are developing your own operators and DAGs. + +This - seemingly unsolvable - puzzle is solved by having pinned constraints files. + +.. contents:: :local: + +Pinned constraint files +----------------------- + +.. note:: + + Only ``pip`` installation is officially supported. + + While it is possible to install Airflow with tools like `poetry <https://python-poetry.org/>`_ or + `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + There are known issues with ``bazel`` that might lead to circular dependencies when using it to install + Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing + the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that + newer versions of ``bazel`` will handle it. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. + + +By default when you install ``apache-airflow`` package - the dependencies are as open as possible while +still allowing the apache-airflow package to install. This means that ``apache-airflow`` package might fail to +install in case a direct or transitive dependency is released that breaks the installation. In such case +when installing ``apache-airflow``, you might need to provide additional constraints (for +example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``) + +There are several sets of constraints we keep: + +* 'constraints' - those are constraints generated by matching the current airflow version from sources + and providers that are installed from PyPI. Those are constraints used by the users who want to + install airflow with pip, they are named ``constraints-<PYTHON_MAJOR_MINOR_VERSION>.txt``. + +* "constraints-source-providers" - those are constraints generated by using providers installed from + current sources. While adding new providers their dependencies might change, so this set of providers + is the current set of the constraints for airflow and providers from the current main sources. + Those providers are used by CI system to keep "stable" set of constraints. They are named + ``constraints-source-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt`` + +* "constraints-no-providers" - those are constraints generated from only Apache Airflow, without any + providers. If you want to manage airflow separately and then add providers individually, you can + use those. Those constraints are named ``constraints-no-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt``. + +The first two can be used as constraints file when installing Apache Airflow in a repeatable way. +It can be done from the sources: + +from the PyPI package: + +.. code-block:: bash + + pip install apache-airflow[google,amazon,async]==2.2.5 \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.5/constraints-3.8.txt"; + +The last one can be used to install Airflow in "minimal" mode - i.e when bare Airflow is installed without +extras. + +When you install airflow from sources (in editable mode) you should use "constraints-source-providers" +instead (this accounts for the case when some providers have not yet been released and have conflicting +requirements). + +.. code-block:: bash + + pip install -e ".[devel]" \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.8.txt"; + + +This works also with extras - for example: Review Comment: Nit: also works with extras ########## contribution-docs/airflow_dependencies_and_extras.rst: ########## @@ -0,0 +1,210 @@ + .. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + .. http://www.apache.org/licenses/LICENSE-2.0 + + .. Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +Airflow dependencies +==================== + +Airflow is not a standard python project. Most of the python projects fall into one of two types - +application or library. As described in +`this StackOverflow question <https://stackoverflow.com/questions/28509481/should-i-pin-my-python-dependencies-versions>`_, +the decision whether to pin (freeze) dependency versions for a python project depends on the type. For +applications, dependencies should be pinned, but for libraries, they should be open. + +For application, pinning the dependencies makes it more stable to install in the future - because new +(even transitive) dependencies might cause installation to fail. For libraries - the dependencies should +be open to allow several different libraries with the same requirements to be installed at the same time. + +The problem is that Apache Airflow is a bit of both - application to install and library to be used when +you are developing your own operators and DAGs. + +This - seemingly unsolvable - puzzle is solved by having pinned constraints files. + +.. contents:: :local: + +Pinned constraint files +----------------------- + +.. note:: + + Only ``pip`` installation is officially supported. + + While it is possible to install Airflow with tools like `poetry <https://python-poetry.org/>`_ or + `pip-tools <https://pypi.org/project/pip-tools/>`_, they do not share the same workflow as + ``pip`` - especially when it comes to constraint vs. requirements management. + Installing via ``Poetry`` or ``pip-tools`` is not currently supported. + + There are known issues with ``bazel`` that might lead to circular dependencies when using it to install + Airflow. Please switch to ``pip`` if you encounter such problems. ``Bazel`` community works on fixing + the problem in `this PR <https://github.com/bazelbuild/rules_python/pull/1166>`_ so it might be that + newer versions of ``bazel`` will handle it. + + If you wish to install airflow using those tools you should use the constraint files and convert + them to appropriate format and workflow that your tool requires. + + +By default when you install ``apache-airflow`` package - the dependencies are as open as possible while +still allowing the apache-airflow package to install. This means that ``apache-airflow`` package might fail to +install in case a direct or transitive dependency is released that breaks the installation. In such case +when installing ``apache-airflow``, you might need to provide additional constraints (for +example ``pip install apache-airflow==1.10.2 Werkzeug<1.0.0``) + +There are several sets of constraints we keep: + +* 'constraints' - those are constraints generated by matching the current airflow version from sources + and providers that are installed from PyPI. Those are constraints used by the users who want to + install airflow with pip, they are named ``constraints-<PYTHON_MAJOR_MINOR_VERSION>.txt``. + +* "constraints-source-providers" - those are constraints generated by using providers installed from + current sources. While adding new providers their dependencies might change, so this set of providers + is the current set of the constraints for airflow and providers from the current main sources. + Those providers are used by CI system to keep "stable" set of constraints. They are named + ``constraints-source-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt`` + +* "constraints-no-providers" - those are constraints generated from only Apache Airflow, without any + providers. If you want to manage airflow separately and then add providers individually, you can + use those. Those constraints are named ``constraints-no-providers-<PYTHON_MAJOR_MINOR_VERSION>.txt``. + +The first two can be used as constraints file when installing Apache Airflow in a repeatable way. +It can be done from the sources: + +from the PyPI package: + +.. code-block:: bash + + pip install apache-airflow[google,amazon,async]==2.2.5 \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.5/constraints-3.8.txt"; + +The last one can be used to install Airflow in "minimal" mode - i.e when bare Airflow is installed without +extras. + +When you install airflow from sources (in editable mode) you should use "constraints-source-providers" +instead (this accounts for the case when some providers have not yet been released and have conflicting +requirements). + +.. code-block:: bash + + pip install -e ".[devel]" \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.8.txt"; + + +This works also with extras - for example: + +.. code-block:: bash + + pip install ".[ssh]" \ + --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-main/constraints-source-providers-3.8.txt"; + + +There are different set of fixed constraint files for different python major/minor versions and you should +use the right file for the right python version. + +If you want to update just airflow dependencies, without paying attention to providers, you can do it using Review Comment: Nit: update just the* airflow dependencies, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
