Hey all, Perhaps we could work backwards from the goal of "I don't want AirBNB's production Airflow installation broken."
I have observed a few things, personally, that I think we could work towards fixing: 1) Instability on master 2) Lack of code reviews (until it's too late) by those that care 3) Lack of design docs For (1), I have a few thoughts. One is better test coverage (duh). Perhaps we can lay out some rules for when tests are required before a PR will get merged, etc. A second thought is related to the way we release Airflow. If you look at how the Linux Kernel [ http://stackoverflow.com/questions/3177338/how-is-linux-kernel-tested], for example, we could follow something like that. For (3), I think Jeremiah's work on the dagrun refactor ( https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=62694286) and scheduler 2.0 work is an awesome example. IMO, an large scale work should have a design doc beforehand. This will prevent things from getting committed that folks don't agree with. Cheers, Chris On Thu, May 12, 2016 at 3:14 PM, Jakob Homan <[email protected]> wrote: > Dan- > I'm afraid not, since that's just a way evading the Apache Way > rather than working towards it, as Incubation is meant to do. (Here's > a good presentation for those unfamiliar with this manta: > http://www.slideshare.net/gagravarr/the-apache-way-apachecon-2014 > You'll hear it a lot here). > The concern here is that bad code may make it into the source tree, > and eventually into a release. First, I'd say, yeah, that'll happen. > All code's reversible, so we can't guarantee it won't happen. But > there are a few things the community can do: > * Maintain a devel branch that interested parties could run off of. > This would give time for features to be tested before being merged to > master. Some policy of merging devel to master could be in place. > * Switch to a Commit-then-Review model (any committer can commit a > patch without a +1; this makes reverting bad commits easy and routine > without the drama/conflict often associated with reverts in > Review-then-Commit projects). > * Improve test coverage and utilize ASF and Github resources for testing. > > What we can't do is tie abilities/privileges/responsibilities to > companies (or only people who work for certain companies). The big > goal of Incubation is to develop a healthy community around the code > that can survive and thrive even if one group of contributors > disappear (say, if a company decides to pull people off the project). > This is why you'll often see big, big arguments around PMC/podling > diversity flare up (e.g. http://bit.ly/ASFWayDiversityArgument). > > -Jakob > > On 12 May 2016 at 14:47, Dan Davydov <[email protected]> > wrote: > > @Jakob > > What if we made it more generic, e.g. a +1 from any commiter from a > company > > that is running at a certain scale (e.g. at least X workers) and willing > to > > help stage releases in their prods until we have more comprehensive test > > coverage/an open source staging environment? This is in Airflow's best > > interests as otherwise stability will suffer. > > > > On Thu, May 12, 2016 at 1:44 PM, Chris Riccomini <[email protected]> > > wrote: > > > >> @Sid, perhaps defining a cool-off window before a scheduler change can > be > >> committed. That way, everyone that cares can have a look at it? Also, > >> having more than one +1 seems OK with me for scheduler changes. We will > >> have to decide what "scheduler change" means, though. > >> > >> On Thu, May 12, 2016 at 1:39 PM, Jakob Homan <[email protected]> wrote: > >> > >> > Hey Sid- > >> > Thanks for the discussion. It's a good chance to the new > >> > contributors to get more experience with the ASF. > >> > > >> > Unfortunately, what you propose is not possible in ASF. As a > >> > meritocracy, ASF does not recognize individual's employers (or lack > >> > thereof). Merit is earned by the individual and follows them as they > >> > move from organization to organization. This is true even for > >> > podlings. Employees of certain organizations are not given extra > >> > power over a project or vote due to their relationship with the > >> > employer. > >> > > >> > ASF does recognize that at times people will be representing their > >> > employer (with my $EMPLOYER hat on, is a common way of expressing > >> > this), but expects that everyone is acting in the best interest of the > >> > project. > >> > > >> > -Jakob > >> > > >> > On 12 May 2016 at 12:58, Siddharth Anand <[email protected]> wrote: > >> > > Hi Folks!As many of you know, Apache Airflow (incubating) came from > >> > Airbnb, where it currently still represents the largest Airflow > >> deployment. > >> > Airflow entered the Apache Incubator shortly over a month ago but > still > >> > depends on Airbnb's production deployment to vet its release > candidates. > >> As > >> > Airflow's adoption increases, we expect to leverage multiple > companies in > >> > conjunction with Apache Infra resources to vet some of the more > >> performance > >> > critical pieces of the code base (e.g. scheduler). We're not there > yet. > >> > > So, for future commits and PRs involving the scheduler (and possibly > >> > other components, e.g. executors), I propose a 2 vote system : at > least 1 > >> > vote from an Airbnb committer and at least 1 vote from a non-Airbnb > >> > committer, separate from the PR author. This will more readily > stabilize > >> > the Airbnb production system that we rely on to vet and cut releases, > >> > speeding up our release cycle. > >> > > Please share your thoughts on the matter along with a vote > for/against. > >> > > -s > >> > > >> >
