Omg yes. I have been screaming out for this for months. $ find scripts -name '*.sh' | xargs egrep -v '^#' | wc -l 6911
That's entirely too much bash for my liking by about an order of magnitude ;) I would also say: make breeze do less. Right now it is three major things: * A local development environment * CI runner * It's recently grown the ability to run airflow for developing dags. That is too much. Yes there is overlap, but it's just too much in one tool, and too complex as a result. Some of this should just be replaced with a docker-compose file (that uses published release images, not floating master/nightly) and users told to run that. Make it simpler, fitting a core purpose - running CI consistently should be it's only goal. -ash On Nov 11 2020, at 9:58 am, Jarek Potiuk <[email protected]> wrote: > Hello Everyone, > > TL; DR; I was thinking for quite a while on this and I think this is the > right time to raise that subject. It's been asked several times, why Breeze > is not written in something else than Bash since it is "that big" or some > people said "monstrous" :). I think it's the right time to start a "rewrite" > project with wide community involvement and Python seems to be the best > choice :). > > > While I was opposing this while we were focusing on Airflow 2.0, and there > are some good reasons why initially I started Breeze in Bash, I think with > the current state of Airflow 2.0 betas, with Airflow 2.0 fully based on > Python 3.6 and with some "stability" and "good set of features" we have in > Breeze and a good level of modularisation we achieved - it's the right time > to think about a rewrite. > > I did not raise this subject to add a distraction on top of what is already a > lot of work for 2.0, but I think having Breeze rewritten in Python could be > the "one more thing" that we could do - as a community to make 2.0 experience > even better, and one that can make the community even closer. > > I was thinking that Breeze is perfect to be split into separate smaller > pieces, describe some assumptions that we will have for its use, and turn it > into a true community effort where a lot of people will contribute and where > we will be able to simplify some of the stuff, and - most importantly - make > more people from the community know about how our CI and development > environment works and be able to solve any problems there. > > Breeze (and underlying bash libraries) are crucial, to get our CI working and > I am mostly the single point of contact (and failure!) when it comes to that > - I would love to not be one :) and I think with most of the core committers > busy with 2.0, this is also an opportunity for more of the contributors to > take their part in it (and eventually earn their rank to become committers!). > For the core committers, this is an extra opportunity to learn how the system > works, influence its design, and possibly simplify some parts of it - even if > they will be mostly focused on 2.0. > > I would like to do it well - write some assumptions in a design doc, plan the > work and split it into separate issues, and lead the effort - but I would > love if most of the work is done by others, who would then become familiar > with the whole of it. > > WDYT? Do you think it is a good idea? Do you thin k it is the right time? Are > there some people in the community who would like to take part in it? > > J. > > -- > > > Jarek Potiuk > Polidea (https://www.polidea.com/) | Principal Software Engineer > > > > > > > > M: +48 660 796 129 (tel:+48660796129) > > > > > > > > > > > > > > > > > > >
