Hi Chris, I know it's annoying to have last minute commit com in, but this is a highly desirable feature for folks using GCP operators, is it possible to include AIRFLOW-1635 <https://github.com/apache/incubator-airflow/commit/b3e985a3146272ecfd3ceaaa0d8567e4e9e117d4> in? More than happy to offer help if there's something I can do. Thanks a lot.
Feng On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <[email protected]> wrote: > Hey all, > > I have cut a 1.9.0alpha0 release of Airflow. You can download it here: > > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/ > > The bin tarball can be installed with pip: > > pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz > > The goal is to have the community install and run this to expose any bugs > before we move on to official release candidates. > > Here are the outstanding blocker bugs for 1.9.0: > > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue > AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are marked > as > AIRFLOW-1055 |Bug |airflow/jobs.py:create_dag_run() exception > for > @on > AIRFLOW-1018 |Bug |Scheduler DAG processes can not log to stdout > AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() exception for > @once > AIRFLOW-976 |Bug |Mark success running task causes it to fail > > Cheers, > Chris > > On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <[email protected]> > wrote: > > > Welp. Work got in the way, so I'll cut the beta on Monday. :) > > > > On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini <[email protected]> > > wrote: > > > >> Works for me. Will try and cut a beta before end of week. > >> > >> Blockers for 1.9.0 are: > >> > >> AIRFLOW-1611 |Bug |Customize logging in Airflow > >> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue > >> AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are > marked > >> as > >> AIRFLOW-976 |Bug |Mark success running task causes it to fail > >> > >> > >> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <[email protected]> > >> wrote: > >> > >>> Hi Chris > >>> > >>> Can I suggest releasing a beta? The stable branch is only cut at RC > >>> time. Betas allow us a broader exposure. It also gives us a point of > >>> reference. > >>> > >>> In addition the list below are mostly longer standing issues that are > >>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258, > and > >>> 976 as blocker? > >>> > >>> Cheers > >>> Bolke > >>> > >>> Verstuurd vanaf mijn iPad > >>> > >>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini < > [email protected]> > >>> het volgende geschreven: > >>> > > >>> > Hey all, > >>> > > >>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, > but > >>> > seeing as there are several outstanding bugs, I'm going to delay. > Here > >>> are > >>> > the bugs that I'm tracking: > >>> > > >>> > AIRFLOW-1611 |Bug |Customize logging in Airflow > >>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue > >>> > AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are > >>> marked as > >>> > AIRFLOW-1055 |Bug |airflow/jobs.py:create_dag_run() > exception > >>> for > >>> > @on > >>> > AIRFLOW-1018 |Bug |Scheduler DAG processes can not log to > >>> stdout > >>> > AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() exception > >>> for @once > >>> > AIRFLOW-988 |Bug |SLA Miss Callbacks Are Repeated if Email > is > >>> Not > >>> > be > >>> > AIRFLOW-976 |Bug |Mark success running task causes it to > fail > >>> > > >>> > These are the priority issues. Once they're merged, I'll cut the > >>> > v1-9-stable and beta release. > >>> > > >>> > If you can help clean this up, that would be really appreciated. > >>> > > >>> > Cheers, > >>> > Chris > >>> > > >>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini < > >>> [email protected]> > >>> > wrote: > >>> > > >>> >> Marked it for 1.9.0. > >>> >> > >>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <[email protected]> > >>> wrote: > >>> >>> > >>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA > >>> callbacks > >>> >>> are not working correctly without emails... Its not a major bug, > but > >>> it > >>> >>> does cause us some annoyance in our current deployment. > >>> >>> > >>> >>> Link to Jira: > >>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988 > >>> >>> > >>> >>> Link to PR: > >>> >>> https://github.com/apache/incubator-airflow/pull/2415 > >>> >>> > >>> >>> Thanks! > >>> >>> Charlie Jones > >>> >>> > >>> >>> CHARLIE JONES > >>> >>> Data Engineer > >>> >>> [email protected] | M: 972.821.7631 > >>> >>> __________________________________________________ > >>> >>> > >>> >>> > >>> >>> Programmatic Performance.* Localized.* > >>> >>> __________________________________________________ > >>> >>> > >>> >>> 1407 Texas Street | Suite 202 | Fort Worth, TX 76102 > >>> >>> 800.840.0768 | www.simpli.fi > >>> >>> > >>> >>> > >>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini < > >>> [email protected]> > >>> >>> wrote: > >>> >>> > >>> >>>> Merged. > >>> >>>> > >>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley < > >>> >>> [email protected]> > >>> >>>> wrote: > >>> >>>> > >>> >>>>> Would it be possible to include AIRFLOW-1587? > >>> >>>>> Running dags from the UI is currently broken on the 1.9.0 branch > >>> due > >>> >>> to > >>> >>>>> this issue. > >>> >>>>> > >>> >>>>> https://github.com/apache/incubator-airflow/pull/2590 > >>> >>>>> > >>> >>>>> Thanks, > >>> >>>>> Ryan > >>> >>>>> > >>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko > >>> >>> <[email protected] > >>> >>>>> > >>> >>>>> wrote: > >>> >>>>> > >>> >>>>>> Hi All, > >>> >>>>>> > >>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release: > >>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631 > >>> >>>>>> > >>> >>>>>> Currently importing a custom logging configuration is not work > (as > >>> >>> far > >>> >>>>> as I > >>> >>>>>> know). Any feedback on the PR would also be appreciated. > >>> >>>>>> > >>> >>>>>> Cheers, Fokko > >>> >>>>>> > >>> >>>>>> > >>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini < > [email protected] > >>> >: > >>> >>>>>> > >>> >>>>>>> Done! > >>> >>>>>>> > >>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford < > >>> >>>>>>> [email protected]> wrote: > >>> >>>>>>> > >>> >>>>>>>> Can you slide the aws and emr connection type fix in? > >>> >>>>>>>> > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 < > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 < > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> > >>> >>>>>>>> > >>> >>>>>>>> It keeps the connection type from getting blanked out on edit > >>> >>> for > >>> >>>>> these > >>> >>>>>>>> types. > >>> >>>>>>>> > >>> >>>>>>>> Thanks, > >>> >>>>>>>> Mike > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini < > >>> >>>>> [email protected]> > >>> >>>>>>>> wrote: > >>> >>>>>>>>> > >>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward to > >>> >>>> these > >>> >>>>>>> fixes! > >>> >>>>>>>>> > >>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel < > >>> >>>>> [email protected] > >>> >>>>>> . > >>> >>>>>>>> invalid > >>> >>>>>>>>>> wrote: > >>> >>>>>>>>> > >>> >>>>>>>>>> Can we get this in? > >>> >>>>>>>>>> > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 > >>> >>>>>>>>>> > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ > >>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2 > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ > >>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a > >>> >>>>>>>>>> > >>> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing to > >>> >>> have > >>> >>>> in > >>> >>>>>> 1.9. > >>> >>>>>>>>>> > >>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini < > >>> >>>>>>>> [email protected]> > >>> >>>>>>>>>> wrote: > >>> >>>>>>>>>> > >>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then. > >>> >>> Initial > >>> >>>>>>> warning > >>> >>>>>>>>>>> stands, that I will start locking down what can get into > >>> >>> 1.9.0 > >>> >>>> at > >>> >>>>>>> that > >>> >>>>>>>>>>> point. > >>> >>>>>>>>>>> > >>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin < > >>> >>>>>> [email protected]> > >>> >>>>>>>>>>> wrote: > >>> >>>>>>>>>>> > >>> >>>>>>>>>>>> No vote indeed, just to gather feedback on a particular > >>> >>> fixed > >>> >>>>>> point > >>> >>>>>>> in > >>> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than to > a > >>> >>>> git > >>> >>>>>>> pull. > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> Bolke > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini < > >>> >>>>> [email protected] > >>> >>>>>>> > >>> >>>>>>>>>>> wrote: > >>> >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> I can do a beta. Is the process significantly different? > >>> >>>> IIRC, > >>> >>>>>> it's > >>> >>>>>>>>>>>>> basically the same, just no vote, right? > >>> >>>>>>>>>>>>> > >>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin < > >>> >>>>>>> [email protected]> > >>> >>>>>>>>>>>> wrote: > >>> >>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right away? > >>> >>>>> Isn’t a > >>> >>>>>>>>>> beta > >>> >>>>>>>>>>> a > >>> >>>>>>>>>>>>>> bit smarter? > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> - Bolke > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini < > >>> >>>>>> [email protected] > >>> >>>>>>>> > >>> >>>>>>>>>>>> wrote: > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> Hey all, > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to cut > >>> >>> the > >>> >>>>>> stable > >>> >>>>>>>>>>> branch > >>> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the > >>> >>> stable > >>> >>>>>> branch > >>> >>>>>>>>>> is > >>> >>>>>>>>>>>>>> cut, I > >>> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked > into > >>> >>>> the > >>> >>>>>>>>>> branch, > >>> >>>>>>>>>>>> and > >>> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the > >>> >>> release > >>> >>>>>> out. > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> Cheers, > >>> >>>>>>>>>>>>>>> Chris > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini < > >>> >>>>>>>>>>>> [email protected] > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>> wrote: > >>> >>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>>> Hey all, > >>> >>>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the > >>> >>> outstanding > >>> >>>> PRs > >>> >>>>>>> that > >>> >>>>>>>>>>> are > >>> >>>>>>>>>>>>>>>> slated to be included into 1.9.0: > >>> >>>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>>> ISSUE ID |STATUS |DESCRIPTION > >>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open |XSS Vulnerability in Variable > >>> >>>>>> endpoint > >>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open |Customize logging in Airflow > >>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened |Fix log source of local > >>> >>> loggers > >>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open |Rename the logger to log > >>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open |Fix minor LICENSE & NOTICE > >>> >>> issue > >>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and > >>> >>> unneeded > >>> >>>>> code > >>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open |HDFSOperator to operate HDFS > >>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open |airflow/jobs.py:create_dag_ru > >>> >>> n() > >>> >>>>>>>>>> exception > >>> >>>>>>>>>>>> for > >>> >>>>>>>>>>>>>>>> @on > >>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open |active_dagruns shouldn't > >>> >>> include > >>> >>>>>> paused > >>> >>>>>>>>>>> DAGs > >>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open |Scheduler DAG processes can > >>> >>> not > >>> >>>> log > >>> >>>>>> to > >>> >>>>>>>>>>> stdout > >>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open |TreeView displayed over task > >>> >>>>>> instances > >>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open |airflow/jobs.py:manage_slas() > >>> >>>>>> exception > >>> >>>>>>>>>> for > >>> >>>>>>>>>>>>>>>> @once > >>> >>>>>>>>>>>>>>>> AIRFLOW-976 |Open |Mark success running task > >>> >>> causes > >>> >>>> it > >>> >>>>>> to > >>> >>>>>>>>>> fail > >>> >>>>>>>>>>>>>>>> AIRFLOW-914 |Open |Refactor > >>> >>>>>> BackfillJobTest.test_backfill_ > >>> >>>>>>>>>>>>>> examples > >>> >>>>>>>>>>>>>>>> to > >>> >>>>>>>>>>>>>>>> AIRFLOW-913 |Open |Refactor > >>> >>>>>> tests.CoreTest.test_scheduler_ > >>> >>>>>>>>>> job > >>> >>>>>>>>>>>> to > >>> >>>>>>>>>>>>>>>> real > >>> >>>>>>>>>>>>>>>> AIRFLOW-912 |Open |Refactor tests and build > >>> >>> matrix > >>> >>>>>>>>>>>>>>>> AIRFLOW-888 |Open |Operators should not push > >>> >>> XComs > >>> >>>> by > >>> >>>>>>>>>> default > >>> >>>>>>>>>>>>>>>> AIRFLOW-828 |Open |Add maximum size for XComs > >>> >>>>>>>>>>>>>>>> AIRFLOW-825 |Open |Add Dataflow semantics > >>> >>>>>>>>>>>>>>>> AIRFLOW-788 |Open |Context unexpectedly added to > >>> >>>> hive > >>> >>>>>> conf > >>> >>>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked into > >>> >>> the > >>> >>>>>> 1.9.0 > >>> >>>>>>>>>>>> branch > >>> >>>>>>>>>>>>>>>> shortly, so if you have something you want in, please > >>> >>> set > >>> >>>>> the > >>> >>>>>>> fix > >>> >>>>>>>>>>>>>> version > >>> >>>>>>>>>>>>>>>> to 1.9.0. > >>> >>>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev > cluster, > >>> >>>> and > >>> >>>>> it > >>> >>>>>>> has > >>> >>>>>>>>>>>> been > >>> >>>>>>>>>>>>>>>> running smoothly for several days. > >>> >>>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If you > >>> >>> run > >>> >>>>>>>>>> Airflow, > >>> >>>>>>>>>>>> it's > >>> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test branch > >>> >>>>>> somewhere, > >>> >>>>>>>>>> and > >>> >>>>>>>>>>>>>> verify > >>> >>>>>>>>>>>>>>>> it's working for your workload. ** > >>> >>>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>>>> Cheers, > >>> >>>>>>>>>>>>>>>> Chris > >>> >>>>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>>> > >>> >>>>>>>>>>> > >>> >>>>>>>>>> > >>> >>>>>>>> > >>> >>>>>>>> > >>> >>>>>>> > >>> >>>>>> > >>> >>>>> > >>> >>>> > >>> >>> > >>> >> > >>> >> > >>> > >> > >> > > >
