leahecole commented on a change in pull request #19867: URL: https://github.com/apache/airflow/pull/19867#discussion_r758654751
########## File path: dev/breeze/doc/adr/0002-implement-standalone-python-command.md ########## @@ -0,0 +1,178 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [2. Implement standalone python command](#2-implement-standalone-python-command) + - [Status](#status) + - [Context](#context) + - [Decision](#decision) + - [Consequences](#consequences) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> + +# 2. Implement standalone python command + +Date: 2021-11-28 + +## Status + +Draft + +## Context + +The [Breeze](https://github.com/apache/airflow/blob/main/BREEZE.rst) is +a command line development environment for Apache Airflow that makes +it easy to setup Airflow development and test environment easily +(< 10 minutes is the goal) and enable contributors to run any subset +of tests that are executed in our CI environment easily. + +The environment has proven to be very useful (it has successfully onboarded +a number of new contributors, and it makes the development environment of +even seasoned contributors much easier as it provides a very easy +replication of the CI environment as well as very easy to setup test +environment that can be used to run: + +* Unit tests +* Integration tests +* Kubernetes/Helm tests +* System tests + +It also serves as a base for our CI execution environment. The same scripts and tools are used +in our CI (based on GitHub actions). A lot of common code and function between CI and Breeze are +shared between the CI and Breeze. All those tools are held in "ci" package. + +Unfortunately, Breeze is largely based on Bash code - for which very few people (except maybe the +Breeze creator - Jarek Potiuk, the author of this document) have any other feeling that uneasiness, +disgust and fear of it :). Since Airflow is largely based on Python, the common consensus is that +Breeze should be rewritten in Python. + +In November 2021, Outreachy sponsored two internship for two interns: @Bowrna and @edithturn were assigned to +the projects: + +* Convert Airflow Local Development environment `Breeze` - from Bash-based to Python-based +* Rewrite Github Action workflows to Python + +With @potiuk, @eladkal and @xurror as mentors. + +The long-standing issues about those two projects are (and we hope to close the projects during the +three months internship - December 2021 - March 2022): + +* https://github.com/apache/airflow/issues/12282 +* https://github.com/apache/airflow/issues/13182 + +There are a number of problems with Bash scripts: + +* They are difficult to understand, modify and debug as Bash "magic" is somewhat arcane +* They are difficult to implement complex logic with +* Navigating common code that is used from the scripts is cumbersome and lack IDE/tools support +* Default Bash on MacOS is very old (from 3.* line) and it will not be updated to a newer version + which impacts cross-platform Breeze applicability +* Bash only work for Windows well in WSL2 environment, which further undermines cross-platform Review comment: ```suggestion * Bash only works well for Windows in WSL2 environment, which further undermines cross-platform ``` small grammar nit ########## File path: pyproject.toml ########## @@ -16,5 +16,5 @@ # under the License. [tool.black] line-length = 110 -target-version = ['py36', 'py37', 'py38'] +target-version = ['py36', 'py37', 'py38', 'py39'] Review comment: Do we also want to add 3.10, or do you only want to keep it in line with what is currently supported in Airflow? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
