I consider https://github.com/apache/incubator-airflow/pull/1738 critical to get in before the release. Without it, plugins are entirely non-functional.
We have a staging environment running off a version close to master, I can probably push it closer to master and try to identify bugs on our end. --George Leslie-Waksman (Clover Health) On Fri, Sep 30, 2016 at 12:56 PM Bolke de Bruin <[email protected]> wrote: > We will switch to master beginning next week. I think it would be nice to > sync on git commit hashes that we are considering to be “master”. > > We have some operational concerns so from our point of view we probably > need a couple of RCs. > > > - Bolke > > > Op 30 sep. 2016, om 18:13 heeft Alex Van Boxel <[email protected]> het > volgende geschreven: > > > > I'll do the same. Nice to have the 1.8 on the horizon. > > > > On Fri, Sep 30, 2016 at 5:51 PM Chris Riccomini <[email protected]> > > wrote: > > > >>> I'm not sure how other projects do this, but I propose that we let the > >> RC settle for a week. > >> > >> +1 > >> > >>> Who's on board!? > >> > >> Me. :) > >> > >> On Fri, Sep 30, 2016 at 7:56 AM, Maxime Beauchemin > >> <[email protected]> wrote: > >>> As related note: the most people we can get to test this release > >> candidate > >>> the better. Note that Airbnb uses only a subset of the Airflow > codebase, > >>> there are many operators and features we do not use at all, so we > >> encourage > >>> everyone to get on board with testing the RC. > >>> > >>> At Airbnb, our process starts with sending the RC into our staging > >> cluster, > >>> let it settle and fix any bug over 48 hours or so. We look for changes > >>> around DagBag processing time, memory footprint on workers, smooth > >>> processing around our canaries. > >>> > >>> When the RC is ready to hit production, we take a full backup of the > >>> metadata DB, and make sure that the whole team is fully available to > >> babysit > >>> the release and hotfix any issue that may pop up, with a clear, fast > >> revert > >>> plan. > >>> > >>> As all of our tasks are assumed to be idempotent the worst case > scenarios > >>> are around delayed SLAs and wasted resources which is all manageable. > >>> > >>> I'm not sure how other projects do this, but I propose that we let the > RC > >>> settle for a week. I suggest that we reset the clock to 7 more days for > >> each > >>> increment of RC, unless the hotfix is very minor. > >>> > >>> Who's on board!? > >>> > >>> Max > >>> > >>> On Fri, Sep 30, 2016 at 7:43 AM, Maxime Beauchemin > >>> <[email protected]> wrote: > >>>> > >>>> Airbnb will cut a release candidate on the week of October 10th off of > >>>> master. > >>>> > >>>> Community, if you have outstanding PRs that you want to make this > >> release, > >>>> now is the time! Though we do not want to rush-merge anything risky > >> that may > >>>> make the release process harder. > >>>> > >>>> Max > >>>> > >>>> On Fri, Sep 30, 2016 at 3:59 AM, Maycock, Luke > >>>> <[email protected]> wrote: > >>>>> > >>>>> Hi Chris, > >>>>> > >>>>> > >>>>> We are currently running master in our environment and we'll be sure > to > >>>>> report any issues. > >>>>> > >>>>> > >>>>> Do you know when the 1.8 release is due? > >>>>> > >>>>> > >>>>> Cheers, > >>>>> > >>>>> Luke Maycock > >>>>> OLIVER WYMAN > >>>>> > >>>>> [email protected]<mailto: > >> [email protected]> > >>>>> www.oliverwyman.com<http://www.oliverwyman.com/> > >>>>> > >>>>> > >>>>> ________________________________ > >>>>> From: Chris Riccomini <[email protected]> > >>>>> Sent: 29 September 2016 18:14:02 > >>>>> To: [email protected] > >>>>> Subject: Re: Airflow Releases > >>>>> > >>>>> Hey Luke, > >>>>> > >>>>>> Is there anything we can do to help get the next release in place? > >>>>> > >>>>> One thing that would definitely help is running master somewhere in > >>>>> your environment, and reporting any issues that see. Over the next > few > >>>>> weeks, AirBNB and a few other folks will be doing the same in an > >>>>> effort to harden the 1.8 release. > >>>>> > >>>>> Cheers, > >>>>> Chris > >>>>> > >>>>> On Thu, Sep 29, 2016 at 3:08 AM, Maycock, Luke > >>>>> <[email protected]> wrote: > >>>>>> Airflow Developers, > >>>>>> > >>>>>> > >>>>>> We were looking at writing a workflow framework in Python when we > >> found > >>>>>> Airflow. We have carried out some proof of concept work for using > >> Airflow > >>>>>> and wish to continue using it as it comes with lots of great > features > >>>>>> out-of-the-box. > >>>>>> > >>>>>> > >>>>>> We have created our own fork here: > >>>>>> > >>>>>> https://github.com/owlabs/incubator-airflow > >>>>>> > >>>>>> > >>>>>> So far, the only thing we have committed back to the main repository > >> is > >>>>>> the following fix to the mssql_hook: > >>>>>> > >>>>>> https://github.com/apache/incubator-airflow/pull/1626 > >>>>>> > >>>>>> > >>>>>> Among other types of tasks, we wish to be able to run mssql tasks > >> using > >>>>>> Airflow. In order to do so, the above and below fixes are required: > >>>>>> > >>>>>> > >>>>>> https://github.com/apache/incubator-airflow/pull/1458< > >> > https://github.com/apache/incubator-airflow/pull/1458/commits/e7e655fde3c29742149d047028cbb21aecba86ed > >>> > >>>>>> > >>>>>> > >>>>>> We have created a Chef cookbook for configuring VMs with Airflow and > >>>>>> its prerequisites. As part of this cookbook, we are installing the > >> latest > >>>>>> release of Airflow. However, it appears that the latest release does > >> not > >>>>>> have the aforementioned fixes. > >>>>>> > >>>>>> Do you know when the next release of Airflow is expected? Is there > >>>>>> anything we can do to help get the next release in place? > >>>>>> > >>>>>> > >>>>>> Luke Maycock > >>>>>> OLIVER WYMAN > >>>>>> > >>>>>> [email protected]<mailto: > >> [email protected]> > >>>>>> www.oliverwyman.com<http://www.oliverwyman.com/> > >>>>>> > >>>>>> > >>>>>> ________________________________ > >>>>>> This e-mail and any attachments may be confidential or legally > >>>>>> privileged. If you received this message in error or are not the > >> intended > >>>>>> recipient, you should destroy the e-mail message and any attachments > >> or > >>>>>> copies, and you are prohibited from retaining, distributing, > >> disclosing or > >>>>>> using any information contained herein. Please inform us of the > >> erroneous > >>>>>> delivery by return e-mail. Thank you for your cooperation. > >>>>> > >>>>> ________________________________ > >>>>> This e-mail and any attachments may be confidential or legally > >>>>> privileged. If you received this message in error or are not the > >> intended > >>>>> recipient, you should destroy the e-mail message and any attachments > or > >>>>> copies, and you are prohibited from retaining, distributing, > >> disclosing or > >>>>> using any information contained herein. Please inform us of the > >> erroneous > >>>>> delivery by return e-mail. Thank you for your cooperation. > >>>> > >>>> > >>> > >> > >
