You screamed (among many others) and I listened :). And I think the time is now to act.
I believe the scope of "Breeze 2" should be part of the design discussion, where we will hear other's opinions (especially the first time or fresh contributors). For now, my vision is quite a bit different than yours Ash :). But I do not want to start a design discussion just yet, I want to make breathing space for others to chime in. I would love to hear many voices and interests of people before we deep dive into what "Breeze 2" might look like. What I am interested in is whether: a) it's the right time b) python is the right choice c) do I have several people who would like to join and offer both - help in designing the vision for it, as well as their time to implement it. I think it is crucial that those people who will be implementing it, will be the main people who make design decisions about it, as I would love to have a strong group of people who would like to not only take part in developing it but also in maintaining it in the future. J. On Wed, Nov 11, 2020 at 11:11 AM Ash Berlin-Taylor <a...@apache.org> wrote: > Omg yes. I have been screaming out for this for months. > > $ find scripts -name '*.sh' | xargs egrep -v '^#' | wc -l > 6911 > > That's entirely too much bash for my liking by about an order of magnitude > ;) > > I would also say: make breeze do less. Right now it is three major things: > > * A local development environment > * CI runner > * It's recently grown the ability to run airflow for developing dags. > > That is too much. Yes there is overlap, but it's just too much in one > tool, and too complex as a result. Some of this should just be replaced > with a docker-compose file (that uses published release images, not > floating master/nightly) and users told to run that. > > Make it simpler, fitting a core purpose - running CI consistently should > be it's only goal. > > -ash > > On Nov 11 2020, at 9:58 am, Jarek Potiuk <jarek.pot...@polidea.com> wrote: > > Hello Everyone, > > TL; DR; I was thinking for quite a while on this and I think this is the > right time to raise that subject. It's been asked several times, why Breeze > is not written in something else than Bash since it is "that big" or some > people said "monstrous" :). I think it's the right time to start a > "rewrite" project with wide community involvement and Python seems to be > the best choice :). > > > While I was opposing this while we were focusing on Airflow 2.0, and there > are some good reasons why initially I started Breeze in Bash, I think with > the current state of Airflow 2.0 betas, with Airflow 2.0 fully based on > Python 3.6 and with some "stability" and "good set of features" we have in > Breeze and a good level of modularisation we achieved - it's the right time > to think about a rewrite. > > I did not raise this subject to add a distraction on top of what is > already a lot of work for 2.0, but I think having Breeze rewritten in > Python could be the "one more thing" that we could do - as a community to > make 2.0 experience even better, and one that can make the community even > closer. > > I was thinking that Breeze is perfect to be split into separate smaller > pieces, describe some assumptions that we will have for its use, and turn > it into a true community effort where a lot of people will contribute and > where we will be able to simplify some of the stuff, and - most importantly > - make more people from the community know about how our CI and development > environment works and be able to solve any problems there. > > Breeze (and underlying bash libraries) are crucial, to get our CI working > and I am mostly the single point of contact (and failure!) when it comes to > that - I would love to not be one :) and I think with most of the core > committers busy with 2.0, this is also an opportunity for more of the > contributors to take their part in it (and eventually earn their rank to > become committers!). For the core committers, this is an extra opportunity > to learn how the system works, influence its design, and possibly simplify > some parts of it - even if they will be mostly focused on 2.0. > > I would like to do it well - write some assumptions in a design doc, plan > the work and split it into separate issues, and lead the effort - but I > would love if most of the work is done by others, who would then become > familiar with the whole of it. > > WDYT? Do you think it is a good idea? Do you thin k it is the right time? > Are there some people in the community who would like to take part in it? > > J. > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
