It should be fine to delete them, hopefully noone is depending on them. On Jan 4, 2017 11:41 AM, "Chris Riccomini" <[email protected]> wrote:
> @Bolke, thanks for creating the branch! Your plan sounds good to me. Re: > deleting airbnb branches, I'll leave Dan/Max/Paul/Arthur/etc to comment on > that. :) > > On Wed, Jan 4, 2017 at 7:59 AM, Bolke de Bruin <[email protected]> wrote: > > > Hi Chris, > > > > I have created branch “v1-8-test”. For now I want to keep master and > > v1-8-test in sync and do not do any cherry picking. The reason for this > is > > that we have a lot of catching up to do between 1.7.1.3 and 1.8.0, next > to > > that master is (at least to me) in an unknown state. If someone has a > > better way to do this I am open to suggestions. > > > > When we release 1.8.0 I will create branch v-1-8-stable. This should > track > > point releases (e.g., 1.8.1, 1.8.2). > > > > On a side note I have deleted many old branches. This is what is left: > > > > remotes/apache/airbnb_rb1.7.1 > > remotes/apache/airbnb_rb1.7.1_2 > > remotes/apache/airbnb_rb1.7.1_3 > > remotes/apache/airbnb_rb1.7.1_4 > > remotes/apache/master > > remotes/apache/v1-8-test > > > > I would like to remove the Airbnb branches as well. Can I? Maybe leave > one > > in as it reflect 1.7.1.3? (Which one?) > > > > - Bolke > > > > > > > On 3 Jan 2017, at 20:34, Chris Riccomini <[email protected]> > wrote: > > > > > > Hey Bolke, > > > > > > Thanks for taking this on. I'm definitely up for running stuff in our > > > environments to verify everything is working. > > > > > > Can I ask that you create a 1.8 alpha 1 branch in the git repo? This > will > > > make it easier for us to track what changes are getting cherry picked > > into > > > the branch, and will also make it easier for users to pip install, if > > they > > > want to do so via github. > > > > > > Also, yea, when we switch to beta, we need to stop merging anything > other > > > than bug fixes into the release branch. > > > > > > Cheers, > > > Chris > > > > > > On Tue, Jan 3, 2017 at 10:31 AM, Dan Davydov <[email protected]. > > invalid > > >> wrote: > > > > > >> All very reasonable to me, one reason we may not have hit the bugs in > > our > > >> production is because we are running off a different merge base and > our > > >> cherries aren't 1-1 with what we are running in production (we still > > test > > >> them but we can't run them in production), that being said I don't > > think I > > >> authored the commits you are referring to so I don't have full > context. > > >> > > >> On Tue, Jan 3, 2017 at 1:27 PM, Bolke de Bruin <[email protected]> > > wrote: > > >> > > >>> Hi Dan et al, > > >>> > > >>> That sounds good to me, however I will be pretty critical of the > > changes > > >>> in the scheduler and the cleanliness of the patches. This is due to > the > > >>> fact I have been chasing quite some bugs in master that were pretty > > hard > > >> to > > >>> track down even with a debugger at hand. I’m surprised that those > > didn’t > > >>> pop up in your production or maybe I am concerned ;-). Anyways, I > hope > > >> you > > >>> understand I might be a bit picky in understanding and needing > (design) > > >>> documentation for some of the changes. > > >>> > > >>> What I would like to suggest is that for the Alpha versions we still > > >>> accept “new” features so these PRs can get in, but from Beta we will > > not > > >>> accept new features anymore. For new features in the area of the > > >> scheduler > > >>> an integration DummyDag should be supplied, so others can test the > > >>> behaviour. Does this sound ok? > > >>> > > >>> My list of open code items for a release looks now like this: > > >>> > > >>> Blockers > > >>> * one_failed not honoured > > >>> * Alex’s sensor issue > > >>> > > >>> New features: > > >>> * Schedule all pending DAGs in a single loop > > >>> * Add support for backfill true/false > > >>> * Impersonation > > >>> * CGroups > > >>> * Add Cloud Storage updated sensor > > >>> > > >>> Alpha2 I will package tomorrow. Packages are signed now by my > > apache.org > > >> < > > >>> http://apache.org/> key. Please verify and let me know if something > is > > >>> off. I’m still waiting for access to the incubating dist repository. > > >>> > > >>> Bolke > > >>> > > >>> > > >>>> On 3 Jan 2017, at 14:38, Dan Davydov <[email protected]. > INVALID> > > >>> wrote: > > >>>> > > >>>> I have also started on this effort, recently Alex Guziel and I have > > >> been > > >>>> pushing Airbnb's custom cherries onto master to get Airbnb back onto > > >>> master > > >>>> in order for us to do a release. > > >>>> > > >>>> I think it might make sense to wait for these two commits to get > > merged > > >>> in > > >>>> since they would be quite nice to have for all Airflow users and > seem > > >>> like > > >>>> they will be merged soon: > > >>>> Schedule all pending DAG runs in a single scheduler loop - > > >>>> https://github.com/apache/incubator-airflow/pull/1906 < > > >>> https://github.com/apache/incubator-airflow/pull/1906> > > >>>> Add Support for dag.backfill=(True|False) Option - > > >>>> https://github.com/apache/incubator-airflow/pull/1830 < > > >>> https://github.com/apache/incubator-airflow/pull/1830> > > >>>> Impersonation Support + Cgroups - https://github.com/apache/ < > > >>> https://github.com/apache/> > > >>>> incubator-airflow/pull/1934 (this is kind of important from the > Airbnb > > >>> side > > >>>> so that we can help test the new master without having to cherrypick > > >> this > > >>>> PR on top of it which would make the testing unreliable for others). > > >>>> > > >>>> If there are PRs that affect the core of Airflow that other > committers > > >>>> think are important to merge we could include these too. I can > commit > > >> to > > >>>> pushing out the Impersonation/Cgroups PR this week pending PR > > comments. > > >>>> What do you think Bolke? > > >>>> > > >>>> On Tue, Jan 3, 2017 at 4:26 AM, Bolke de Bruin <[email protected] > > >>> <mailto:[email protected]>> wrote: > > >>>> > > >>>>> Hey Alex, > > >>>>> > > >>>>> I have noticed the same, and it is also the reason why we have > Alpha > > >>>>> versions. For now I have noticed the following: > > >>>>> > > >>>>> * Tasks can get in limbo between scheduler and executor: > > >>>>> https://github.com/apache/incubator-airflow/pull/1948 < > > >>> https://github.com/apache/incubator-airflow/pull/1948> < > > >>>>> https://github.com/apache/incubator-airflow/pull/1948 < > > >>> https://github.com/apache/incubator-airflow/pull/1948>> > > >>>>> * Try_number not increased due to reset in LocalTaskJob: > > >>>>> https://github.com/apache/incubator-airflow/pull/1969 < > > >>> https://github.com/apache/incubator-airflow/pull/1969> < > > >>>>> https://github.com/apache/incubator-airflow/pull/1969 < > > >>> https://github.com/apache/incubator-airflow/pull/1969>> > > >>>>> * one_failed trigger not executed > > >>>>> > > >>>>> My idea is to move to a Samba style of releases eventually, but for > > >> now > > >>> I > > >>>>> would like to get master into a state that we understand and > > therefore > > >>> not > > >>>>> accept any patches that do not address any bugs. > > >>>>> > > >>>>> If you (or anyone else) can review the above PRs and add your own > as > > >>> well > > >>>>> then I can create another Alpha version. I’ll be on gitter as much > as > > >> I > > >>> can > > >>>>> so we can speed up if needed. > > >>>>> > > >>>>> - Bolke > > >>>>> > > >>>>>> On 3 Jan 2017, at 08:51, Alex Van Boxel <[email protected]> wrote: > > >>>>>> > > >>>>>> Hey Bolke, > > >>>>>> > > >>>>>> thanks for getting this moving. But I already have some blockers, > > >>> since I > > >>>>>> moved up master to this release (moved from end November to now) > > >>>>> stability > > >>>>>> has gone down (certainly on Celary). I'm trying to identify the > core > > >>>>>> problems and see if I can fix them. > > >>>>>> > > >>>>>> On Sat, Dec 31, 2016 at 9:52 PM Bolke de Bruin <[email protected] > > >>>>> <mailto:[email protected] <mailto:[email protected]>>> wrote: > > >>>>>> > > >>>>>> Dear All, > > >>>>>> > > >>>>>> On the verge of the New Year, I decided to be a little bit cheeky > > and > > >>> to > > >>>>>> make available an Airflow 1.8.0 Alpha 1. We have been talking > about > > >> it > > >>>>> for > > >>>>>> a long time now and by doing this I wanted bootstrap the process. > It > > >>>>> should > > >>>>>> by no means be considered an Apache release yet. This is for > testing > > >>>>>> purposes in the dev community around Airflow, nothing else. > > >>>>>> > > >>>>>> The build is exactly the same as the state of master (git 410736d) > > >> plus > > >>>>> the > > >>>>>> change to version “1.8.0.alpha1” in version.py. > > >>>>>> > > >>>>>> I am dedicating quite some time next week and beyond to get a > > release > > >>>>> out. > > >>>>>> Hopefully we can get some help with testing, changelog etc. To > make > > >>> this > > >>>>>> possible I would like to propose a freeze to adding new features > for > > >> at > > >>>>>> least two weeks - say until Jan 15. > > >>>>>> > > >>>>>> You can find the tar here: http://people.apache.org/~bolke/ < > > >>> http://people.apache.org/~bolke/> < > > >>>>>> http://people.apache.org/~bolke/ <http://people.apache.org/~ > bolke/> > > >> < > > >>> http://people.apache.org/~bolke/ <http://people.apache.org/~bolke/ > >>> > > . > > >>>>> It isn’t signed. Following versions > > >>>>>> will be. SHA is available. > > >>>>>> > > >>>>>> Lastly, Alpha 1 does not have the fix for retries yet. So we will > > get > > >>> an > > >>>>>> Alpha 2 :-). @Max / @Dan / @Paul: a potential fix is in > > >>>>>> https://github.com/apache/incubator-airflow/pull/1948 < > > >>> https://github.com/apache/incubator-airflow/pull/1948> < > > >>>>> https://github.com/apache/incubator-airflow/pull/1948 < > > >>> https://github.com/apache/incubator-airflow/pull/1948>> < > > >>>>>> https://github.com/apache/incubator-airflow/pull/1948 < > > >>> https://github.com/apache/incubator-airflow/pull/1948> < > > >>>>> https://github.com/apache/incubator-airflow/pull/1948 < > > >>> https://github.com/apache/incubator-airflow/pull/1948>>> , but your > > >>>>> feedback > > >>>>>> is required as it is entrenched in new processing code that you > are > > >>>>> running > > >>>>>> in production afaik - so I wonder what happens in your fork. > > >>>>>> > > >>>>>> Happy New Year! > > >>>>>> > > >>>>>> Bolke > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> -- > > >>>>>> _/ > > >>>>>> _/ Alex Van Boxel > > >>> > > >>> > > >> > > > > >
