potiuk commented on a change in pull request #11842: URL: https://github.com/apache/airflow/pull/11842#discussion_r512844950
########## File path: UPDATING_TO_2.0.md ########## @@ -0,0 +1,1009 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +# Updating Airflow + +This file documents any backwards-incompatible changes in Airflow and +assists users migrating to a new version. + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Step 1: Upgrade to Python 3](#step-1-upgrade-to-python-3) +- [Step 2: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release)](#step-2-upgrade-to-airflow-11013-aka-our-bridge-release) +- [Step 3: Set Operators to Backport Providers](#step-3-set-operators-to-backport-providers) +- [Step 3: Upgrade Airflow DAGs](#step-3-upgrade-airflow-dags) +- [Step 4: Update system configurations](#step-4-update-system-configurations) +- [Step 5: Upgrade KubernetesExecutor settings](#step-5-upgrade-kubernetesexecutor-settings) +- [Changed Parameters for the KubernetesPodOperator](#changed-parameters-for-the-kubernetespodoperator) Review comment: Those should be in "Appendix" or "reference" section IMHO. or maybe steps should be grouped under the "Migration steps" . ########## File path: UPDATING_TO_2.0.md ########## @@ -0,0 +1,1009 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +# Updating Airflow + +This file documents any backwards-incompatible changes in Airflow and +assists users migrating to a new version. + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Step 1: Upgrade to Python 3](#step-1-upgrade-to-python-3) +- [Step 2: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release)](#step-2-upgrade-to-airflow-11013-aka-our-bridge-release) +- [Step 3: Set Operators to Backport Providers](#step-3-set-operators-to-backport-providers) +- [Step 3: Upgrade Airflow DAGs](#step-3-upgrade-airflow-dags) +- [Step 4: Update system configurations](#step-4-update-system-configurations) +- [Step 5: Upgrade KubernetesExecutor settings](#step-5-upgrade-kubernetesexecutor-settings) +- [Changed Parameters for the KubernetesPodOperator](#changed-parameters-for-the-kubernetespodoperator) +- [Migration Guide from Experimental API to Stable API v1](#migration-guide-from-experimental-api-to-stable-api-v1) +- [CLI changes in Airflow 2.0](#cli-changes-in-airflow-20) +- [Changes to Airflow Plugins](#changes-to-airflow-plugins) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> + +## Step 1: Upgrade to Python 3 + +Airflow 1.10 will be the last release series to support Python 2. Airflow 2.0.0 will only support Python 3.6 and up. + +If you have a specific task that still requires Python 2 then you can use the PythonVirtualenvOperator for this. + +For a list of breaking changes between python 2 and python, please refer to this [handy blog](https://blog.couchbase.com/tips-and-tricks-for-upgrading-from-python-2-to-python-3/) by the CouchBaseDB team. + + +## Step 2: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release) + +To minimize upgrade friction for Airflow 1.10, we are releasing a "bridge" release that has multiple critical features that can prepare users to upgrade without disrupting existing workflows. These features include: + +1. All breaking DAG and architecture changes of Airflow 2.0 have been backported to Airflow 1.10.13. This backward-compatibility does not mean +that 1.10.13 will process these DAGs the same way as Airflow 2.0. What this does mean is that all Airflow 2.0 +compatible DAGs will work in Airflow 1.10.13. Instead, this backport will give users time to modify their DAGs over time without any service +disruption. +2. We have backported the `pod_template_file` capability for the KubernetesExecutor as well as a script that will generate a `pod_template_file` +based on your `airflow.cfg` settings. To generate this file simply run the following command: + + airflow generate_pod_template -o <output file path> Review comment: ```suggestion ```shell script airflow generate_pod_template -o <output file path> ` `` ``` ########## File path: UPDATING_TO_2.0.md ########## @@ -0,0 +1,889 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +# Updating Airflow + +This file documents any backwards-incompatible changes in Airflow and +assists users migrating to a new version. + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Step 1: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release)](#step-1-upgrade-to-airflow-11013-aka-our-bridge-release) +- [Step 3: Upgrade to Python 3](#step-3-upgrade-to-python-3) +- [Step 3: Upgrade Airflow DAGs](#step-3-upgrade-airflow-dags) Review comment: We have now two Step 3's :) ########## File path: UPDATING_TO_2.0.md ########## @@ -0,0 +1,1009 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +# Updating Airflow + +This file documents any backwards-incompatible changes in Airflow and +assists users migrating to a new version. + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Step 1: Upgrade to Python 3](#step-1-upgrade-to-python-3) +- [Step 2: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release)](#step-2-upgrade-to-airflow-11013-aka-our-bridge-release) +- [Step 3: Set Operators to Backport Providers](#step-3-set-operators-to-backport-providers) +- [Step 3: Upgrade Airflow DAGs](#step-3-upgrade-airflow-dags) +- [Step 4: Update system configurations](#step-4-update-system-configurations) +- [Step 5: Upgrade KubernetesExecutor settings](#step-5-upgrade-kubernetesexecutor-settings) +- [Changed Parameters for the KubernetesPodOperator](#changed-parameters-for-the-kubernetespodoperator) +- [Migration Guide from Experimental API to Stable API v1](#migration-guide-from-experimental-api-to-stable-api-v1) +- [CLI changes in Airflow 2.0](#cli-changes-in-airflow-20) +- [Changes to Airflow Plugins](#changes-to-airflow-plugins) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> + +## Step 1: Upgrade to Python 3 + +Airflow 1.10 will be the last release series to support Python 2. Airflow 2.0.0 will only support Python 3.6 and up. + +If you have a specific task that still requires Python 2 then you can use the PythonVirtualenvOperator for this. + +For a list of breaking changes between python 2 and python, please refer to this [handy blog](https://blog.couchbase.com/tips-and-tricks-for-upgrading-from-python-2-to-python-3/) by the CouchBaseDB team. + + +## Step 2: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release) + +To minimize upgrade friction for Airflow 1.10, we are releasing a "bridge" release that has multiple critical features that can prepare users to upgrade without disrupting existing workflows. These features include: + +1. All breaking DAG and architecture changes of Airflow 2.0 have been backported to Airflow 1.10.13. This backward-compatibility does not mean +that 1.10.13 will process these DAGs the same way as Airflow 2.0. What this does mean is that all Airflow 2.0 +compatible DAGs will work in Airflow 1.10.13. Instead, this backport will give users time to modify their DAGs over time without any service +disruption. +2. We have backported the `pod_template_file` capability for the KubernetesExecutor as well as a script that will generate a `pod_template_file` +based on your `airflow.cfg` settings. To generate this file simply run the following command: + + airflow generate_pod_template -o <output file path> + Once you have performed this step, simply write out the file path to this file in the `pod_template_file` section of the `kubernetes` + section of your `airflow.cfg` +3. Airflow 1.10.13 will contain our "upgrade check" scripts. These scripts will read through your `airflow.cfg` and all of your +Dags and will give a detailed report of all changes required before upgrading. We are testing this script diligently, and our +goal is that any Airflow setup that can pass these tests will be able to upgrade to 2.0 without any issues. + + airflow upgrade_check Review comment: Same here ```shell script would be nice ########## File path: UPDATING_TO_2.0.md ########## @@ -0,0 +1,1009 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +# Updating Airflow + +This file documents any backwards-incompatible changes in Airflow and +assists users migrating to a new version. + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Step 1: Upgrade to Python 3](#step-1-upgrade-to-python-3) +- [Step 2: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release)](#step-2-upgrade-to-airflow-11013-aka-our-bridge-release) +- [Step 3: Set Operators to Backport Providers](#step-3-set-operators-to-backport-providers) +- [Step 3: Upgrade Airflow DAGs](#step-3-upgrade-airflow-dags) +- [Step 4: Update system configurations](#step-4-update-system-configurations) +- [Step 5: Upgrade KubernetesExecutor settings](#step-5-upgrade-kubernetesexecutor-settings) +- [Changed Parameters for the KubernetesPodOperator](#changed-parameters-for-the-kubernetespodoperator) +- [Migration Guide from Experimental API to Stable API v1](#migration-guide-from-experimental-api-to-stable-api-v1) +- [CLI changes in Airflow 2.0](#cli-changes-in-airflow-20) +- [Changes to Airflow Plugins](#changes-to-airflow-plugins) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> + +## Step 1: Upgrade to Python 3 + +Airflow 1.10 will be the last release series to support Python 2. Airflow 2.0.0 will only support Python 3.6 and up. Review comment: ```suggestion Airflow 1.10.x is the last release series to support Python 2. Airflow 2.0 series only supports Python 3.6 and up. ``` ########## File path: UPDATING_TO_2.0.md ########## @@ -0,0 +1,1009 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +# Updating Airflow + +This file documents any backwards-incompatible changes in Airflow and +assists users migrating to a new version. + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Step 1: Upgrade to Python 3](#step-1-upgrade-to-python-3) +- [Step 2: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release)](#step-2-upgrade-to-airflow-11013-aka-our-bridge-release) +- [Step 3: Set Operators to Backport Providers](#step-3-set-operators-to-backport-providers) +- [Step 3: Upgrade Airflow DAGs](#step-3-upgrade-airflow-dags) +- [Step 4: Update system configurations](#step-4-update-system-configurations) +- [Step 5: Upgrade KubernetesExecutor settings](#step-5-upgrade-kubernetesexecutor-settings) +- [Changed Parameters for the KubernetesPodOperator](#changed-parameters-for-the-kubernetespodoperator) +- [Migration Guide from Experimental API to Stable API v1](#migration-guide-from-experimental-api-to-stable-api-v1) +- [CLI changes in Airflow 2.0](#cli-changes-in-airflow-20) +- [Changes to Airflow Plugins](#changes-to-airflow-plugins) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> + +## Step 1: Upgrade to Python 3 + +Airflow 1.10 will be the last release series to support Python 2. Airflow 2.0.0 will only support Python 3.6 and up. + +If you have a specific task that still requires Python 2 then you can use the PythonVirtualenvOperator for this. + +For a list of breaking changes between python 2 and python, please refer to this [handy blog](https://blog.couchbase.com/tips-and-tricks-for-upgrading-from-python-2-to-python-3/) by the CouchBaseDB team. + + +## Step 2: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release) + +To minimize upgrade friction for Airflow 1.10, we are releasing a "bridge" release that has multiple critical features that can prepare users to upgrade without disrupting existing workflows. These features include: + +1. All breaking DAG and architecture changes of Airflow 2.0 have been backported to Airflow 1.10.13. This backward-compatibility does not mean +that 1.10.13 will process these DAGs the same way as Airflow 2.0. What this does mean is that all Airflow 2.0 +compatible DAGs will work in Airflow 1.10.13. Instead, this backport will give users time to modify their DAGs over time without any service +disruption. +2. We have backported the `pod_template_file` capability for the KubernetesExecutor as well as a script that will generate a `pod_template_file` +based on your `airflow.cfg` settings. To generate this file simply run the following command: + + airflow generate_pod_template -o <output file path> Review comment: can't add ``` :( ########## File path: UPDATING_TO_2.0.md ########## @@ -0,0 +1,1009 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> +# Updating Airflow + +This file documents any backwards-incompatible changes in Airflow and +assists users migrating to a new version. + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Step 1: Upgrade to Python 3](#step-1-upgrade-to-python-3) +- [Step 2: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release)](#step-2-upgrade-to-airflow-11013-aka-our-bridge-release) +- [Step 3: Set Operators to Backport Providers](#step-3-set-operators-to-backport-providers) +- [Step 3: Upgrade Airflow DAGs](#step-3-upgrade-airflow-dags) +- [Step 4: Update system configurations](#step-4-update-system-configurations) +- [Step 5: Upgrade KubernetesExecutor settings](#step-5-upgrade-kubernetesexecutor-settings) +- [Changed Parameters for the KubernetesPodOperator](#changed-parameters-for-the-kubernetespodoperator) +- [Migration Guide from Experimental API to Stable API v1](#migration-guide-from-experimental-api-to-stable-api-v1) +- [CLI changes in Airflow 2.0](#cli-changes-in-airflow-20) +- [Changes to Airflow Plugins](#changes-to-airflow-plugins) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> + +## Step 1: Upgrade to Python 3 + +Airflow 1.10 will be the last release series to support Python 2. Airflow 2.0.0 will only support Python 3.6 and up. + +If you have a specific task that still requires Python 2 then you can use the PythonVirtualenvOperator for this. + +For a list of breaking changes between python 2 and python, please refer to this [handy blog](https://blog.couchbase.com/tips-and-tricks-for-upgrading-from-python-2-to-python-3/) by the CouchBaseDB team. + + +## Step 2: Upgrade to Airflow 1.10.13 (a.k.a our "bridge" release) + +To minimize upgrade friction for Airflow 1.10, we are releasing a "bridge" release that has multiple critical features that can prepare users to upgrade without disrupting existing workflows. These features include: + +1. All breaking DAG and architecture changes of Airflow 2.0 have been backported to Airflow 1.10.13. This backward-compatibility does not mean +that 1.10.13 will process these DAGs the same way as Airflow 2.0. What this does mean is that all Airflow 2.0 +compatible DAGs will work in Airflow 1.10.13. Instead, this backport will give users time to modify their DAGs over time without any service +disruption. +2. We have backported the `pod_template_file` capability for the KubernetesExecutor as well as a script that will generate a `pod_template_file` +based on your `airflow.cfg` settings. To generate this file simply run the following command: + + airflow generate_pod_template -o <output file path> + Once you have performed this step, simply write out the file path to this file in the `pod_template_file` section of the `kubernetes` + section of your `airflow.cfg` +3. Airflow 1.10.13 will contain our "upgrade check" scripts. These scripts will read through your `airflow.cfg` and all of your +Dags and will give a detailed report of all changes required before upgrading. We are testing this script diligently, and our +goal is that any Airflow setup that can pass these tests will be able to upgrade to 2.0 without any issues. + + airflow upgrade_check + + +## Step 3: Set Operators to Backport Providers +Now that you are set up in airflow 1.10.13 with python a 3.6+ environment, you are ready to start porting your DAGs to Airfow 2.0 compliance! + +The most important step in this transition is also the easiest step to do in pieces. All Airflow 2.0 operators are backwards compatible with Airflow 1.10 +using the [backport providers](./backport-providers.rst) service. In your own time, you can transition to using these backport-providers +by pip installing the provider via `pypi` and changing the import path. + +For example: While historically you might have imported the DockerOperator in this fashion: + +```python +from airflow.operators.docker_operator import DockerOperator +``` + +You would now run this command to import the provider: +```shell script +pip install apache-airflow-backport-providers-docker +``` + +and then import the operator with this path: +```python +from airflow.providers.docker.operators.docker import DockerOperator +``` + +Note that this backport-provider is *not* the same as the actual provider in 2.0. We have tested these operators for backwards Review comment: ```suggestion Note that the backport provider packages are just backports of the provider packages compatible with Airflow 2.0. Those provider packages are installed automatically when you install airflow with extras. For example pip install airflow[docker] automatically installs the `apache-airflow-providers-docker` package. But you can manage/upgrade remove provider packages separately from the airflow core. ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
