hi Chris, We are still running pre alpha. I am a bit preoccupied with preparing for a conference and the team is readying a release of one of our core products. So it will probably will be after this week when I get my hands dirty again.
Cheers Bolke Verstuurd vanaf mijn iPad > Op 4 okt. 2017 om 23:35 heeft Chris Riccomini <[email protected]> het > volgende geschreven: > > Hey all, > > Checking in. Has anyone tested this in their environments? Stable releases > depend on community involvement. > > Cheers, > Chris > > On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <[email protected]> > wrote: > >> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0, but >> will be included in alpha1. >> >>> On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <[email protected]> wrote: >>> >>> Hi Chris, >>> >>> I know it's annoying to have last minute commit com in, but this is a >>> highly desirable feature for folks using GCP operators, is it possible to >>> include AIRFLOW-1635 >>> <https://github.com/apache/incubator-airflow/commit/b3e985a3 >>> 146272ecfd3ceaaa0d8567e4e9e117d4> >>> in? >>> More than happy to offer help if there's something I can do. >>> Thanks a lot. >>> >>> Feng >>> >>> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <[email protected]> >>> wrote: >>> >>>> Hey all, >>>> >>>> I have cut a 1.9.0alpha0 release of Airflow. You can download it here: >>>> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/ >>>> >>>> The bin tarball can be installed with pip: >>>> >>>> pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz >>>> >>>> The goal is to have the community install and run this to expose any >>> bugs >>>> before we move on to official release candidates. >>>> >>>> Here are the outstanding blocker bugs for 1.9.0: >>>> >>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue >>>> AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are >>> marked >>>> as >>>> AIRFLOW-1055 |Bug |airflow/jobs.py:create_dag_run() exception >>>> for >>>> @on >>>> AIRFLOW-1018 |Bug |Scheduler DAG processes can not log to >>> stdout >>>> AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() exception >>> for >>>> @once >>>> AIRFLOW-976 |Bug |Mark success running task causes it to fail >>>> >>>> Cheers, >>>> Chris >>>> >>>> On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <[email protected] >>>> >>>> wrote: >>>> >>>>> Welp. Work got in the way, so I'll cut the beta on Monday. :) >>>>> >>>>> On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini < >>> [email protected]> >>>>> wrote: >>>>> >>>>>> Works for me. Will try and cut a beta before end of week. >>>>>> >>>>>> Blockers for 1.9.0 are: >>>>>> >>>>>> AIRFLOW-1611 |Bug |Customize logging in Airflow >>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue >>>>>> AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are >>>> marked >>>>>> as >>>>>> AIRFLOW-976 |Bug |Mark success running task causes it to >>> fail >>>>>> >>>>>> >>>>>> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Chris >>>>>>> >>>>>>> Can I suggest releasing a beta? The stable branch is only cut at RC >>>>>>> time. Betas allow us a broader exposure. It also gives us a point of >>>>>>> reference. >>>>>>> >>>>>>> In addition the list below are mostly longer standing issues that >>> are >>>>>>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258, >>>> and >>>>>>> 976 as blocker? >>>>>>> >>>>>>> Cheers >>>>>>> Bolke >>>>>>> >>>>>>> Verstuurd vanaf mijn iPad >>>>>>> >>>>>>>> Op 28 sep. 2017 om 19:49 heeft Chris Riccomini < >>>> [email protected]> >>>>>>> het volgende geschreven: >>>>>>>> >>>>>>>> Hey all, >>>>>>>> >>>>>>>> I was planning to cut a 1.9.0 stable branch and 1.9.0 beta >>> release, >>>> but >>>>>>>> seeing as there are several outstanding bugs, I'm going to delay. >>>> Here >>>>>>> are >>>>>>>> the bugs that I'm tracking: >>>>>>>> >>>>>>>> AIRFLOW-1611 |Bug |Customize logging in Airflow >>>>>>>> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue >>>>>>>> AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are >>>>>>> marked as >>>>>>>> AIRFLOW-1055 |Bug |airflow/jobs.py:create_dag_run() >>>> exception >>>>>>> for >>>>>>>> @on >>>>>>>> AIRFLOW-1018 |Bug |Scheduler DAG processes can not log to >>>>>>> stdout >>>>>>>> AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() >>> exception >>>>>>> for @once >>>>>>>> AIRFLOW-988 |Bug |SLA Miss Callbacks Are Repeated if >>> Email >>>> is >>>>>>> Not >>>>>>>> be >>>>>>>> AIRFLOW-976 |Bug |Mark success running task causes it to >>>> fail >>>>>>>> >>>>>>>> These are the priority issues. Once they're merged, I'll cut the >>>>>>>> v1-9-stable and beta release. >>>>>>>> >>>>>>>> If you can help clean this up, that would be really appreciated. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Chris >>>>>>>> >>>>>>>> On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini < >>>>>>> [email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Marked it for 1.9.0. >>>>>>>>> >>>>>>>>>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones < >>> [email protected]> >>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA >>>>>>> callbacks >>>>>>>>>> are not working correctly without emails... Its not a major bug, >>>> but >>>>>>> it >>>>>>>>>> does cause us some annoyance in our current deployment. >>>>>>>>>> >>>>>>>>>> Link to Jira: >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-988 >>>>>>>>>> >>>>>>>>>> Link to PR: >>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2415 >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> Charlie Jones >>>>>>>>>> >>>>>>>>>> CHARLIE JONES >>>>>>>>>> Data Engineer >>>>>>>>>> [email protected] | M: 972.821.7631 >>>>>>>>>> __________________________________________________ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Programmatic Performance.* Localized.* >>>>>>>>>> __________________________________________________ >>>>>>>>>> >>>>>>>>>> 1407 Texas Street | Suite 202 | Fort Worth, TX 76102 >>>>>>>>>> 800.840.0768 | www.simpli.fi >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini < >>>>>>> [email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Merged. >>>>>>>>>>> >>>>>>>>>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley < >>>>>>>>>> [email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Would it be possible to include AIRFLOW-1587? >>>>>>>>>>>> Running dags from the UI is currently broken on the 1.9.0 >>> branch >>>>>>> due >>>>>>>>>> to >>>>>>>>>>>> this issue. >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2590 >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Ryan >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko >>>>>>>>>> <[email protected] >>>>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi All, >>>>>>>>>>>>> >>>>>>>>>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release: >>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2631 >>>>>>>>>>>>> >>>>>>>>>>>>> Currently importing a custom logging configuration is not >>> work >>>> (as >>>>>>>>>> far >>>>>>>>>>>> as I >>>>>>>>>>>>> know). Any feedback on the PR would also be appreciated. >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, Fokko >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini < >>>> [email protected] >>>>>>>> : >>>>>>>>>>>>> >>>>>>>>>>>>>> Done! >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can you slide the aws and emr connection type fix in? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 < >>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 < >>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It keeps the connection type from getting blanked out on >>> edit >>>>>>>>>> for >>>>>>>>>>>> these >>>>>>>>>>>>>>> types. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Mike >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini < >>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward >>> to >>>>>>>>>>> these >>>>>>>>>>>>>> fixes! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel < >>>>>>>>>>>> [email protected] >>>>>>>>>>>>> . >>>>>>>>>>>>>>> invalid >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can we get this in? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ >>>>>>>>>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2 >>>>>>>>>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ >>>>>>>>>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It speeds up loading times a lot, so it's a good thing to >>>>>>>>>> have >>>>>>>>>>> in >>>>>>>>>>>>> 1.9. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini < >>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then. >>>>>>>>>> Initial >>>>>>>>>>>>>> warning >>>>>>>>>>>>>>>>>> stands, that I will start locking down what can get into >>>>>>>>>> 1.9.0 >>>>>>>>>>> at >>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>> point. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin < >>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> No vote indeed, just to gather feedback on a particular >>>>>>>>>> fixed >>>>>>>>>>>>> point >>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> time. It also gives a bit more trust to a tarball than >>> to >>>> a >>>>>>>>>>> git >>>>>>>>>>>>>> pull. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Bolke >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini < >>>>>>>>>>>> [email protected] >>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I can do a beta. Is the process significantly >>> different? >>>>>>>>>>> IIRC, >>>>>>>>>>>>> it's >>>>>>>>>>>>>>>>>>>> basically the same, just no vote, right? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin < >>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right >>> away? >>>>>>>>>>>> Isn’t a >>>>>>>>>>>>>>>>> beta >>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>> bit smarter? >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> - Bolke >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini < >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I want to send out a warning that I'm planning to >>> cut >>>>>>>>>> the >>>>>>>>>>>>> stable >>>>>>>>>>>>>>>>>> branch >>>>>>>>>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the >>>>>>>>>> stable >>>>>>>>>>>>> branch >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>> cut, I >>>>>>>>>>>>>>>>>>>>>> will be locking down what commits get cherry picked >>>> into >>>>>>>>>>> the >>>>>>>>>>>>>>>>> branch, >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> will only be doing PRs that are required to get the >>>>>>>>>> release >>>>>>>>>>>>> out. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>>>>> Chris >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini < >>>>>>>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hey all, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the >>>>>>>>>> outstanding >>>>>>>>>>> PRs >>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>>>> slated to be included into 1.9.0: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> ISSUE ID |STATUS |DESCRIPTION >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1617 |Open |XSS Vulnerability in >>> Variable >>>>>>>>>>>>> endpoint >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1611 |Open |Customize logging in >>> Airflow >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened |Fix log source of local >>>>>>>>>> loggers >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1604 |Open |Rename the logger to log >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1525 |Open |Fix minor LICENSE & NOTICE >>>>>>>>>> issue >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and >>>>>>>>>> unneeded >>>>>>>>>>>> code >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1198 |Open |HDFSOperator to operate >>> HDFS >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1055 |Open >>> |airflow/jobs.py:create_dag_ru >>>>>>>>>> n() >>>>>>>>>>>>>>>>> exception >>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>> @on >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1019 |Open |active_dagruns shouldn't >>>>>>>>>> include >>>>>>>>>>>>> paused >>>>>>>>>>>>>>>>>> DAGs >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1018 |Open |Scheduler DAG processes >>> can >>>>>>>>>> not >>>>>>>>>>> log >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> stdout >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1015 |Open |TreeView displayed over >>> task >>>>>>>>>>>>> instances >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-1013 |Open >>> |airflow/jobs.py:manage_slas() >>>>>>>>>>>>> exception >>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>> @once >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-976 |Open |Mark success running task >>>>>>>>>> causes >>>>>>>>>>> it >>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> fail >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-914 |Open |Refactor >>>>>>>>>>>>> BackfillJobTest.test_backfill_ >>>>>>>>>>>>>>>>>>>>> examples >>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-913 |Open |Refactor >>>>>>>>>>>>> tests.CoreTest.test_scheduler_ >>>>>>>>>>>>>>>>> job >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>> real >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-912 |Open |Refactor tests and build >>>>>>>>>> matrix >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-888 |Open |Operators should not push >>>>>>>>>> XComs >>>>>>>>>>> by >>>>>>>>>>>>>>>>> default >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-828 |Open |Add maximum size for XComs >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-825 |Open |Add Dataflow semantics >>>>>>>>>>>>>>>>>>>>>>> AIRFLOW-788 |Open |Context unexpectedly >>> added to >>>>>>>>>>> hive >>>>>>>>>>>>> conf >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked >>> into >>>>>>>>>> the >>>>>>>>>>>>> 1.9.0 >>>>>>>>>>>>>>>>>>> branch >>>>>>>>>>>>>>>>>>>>>>> shortly, so if you have something you want in, >>> please >>>>>>>>>> set >>>>>>>>>>>> the >>>>>>>>>>>>>> fix >>>>>>>>>>>>>>>>>>>>> version >>>>>>>>>>>>>>>>>>>>>>> to 1.9.0. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev >>>> cluster, >>>>>>>>>>> and >>>>>>>>>>>> it >>>>>>>>>>>>>> has >>>>>>>>>>>>>>>>>>> been >>>>>>>>>>>>>>>>>>>>>>> running smoothly for several days. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> ** I could really use help verifying stability. If >>> you >>>>>>>>>> run >>>>>>>>>>>>>>>>> Airflow, >>>>>>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test >>> branch >>>>>>>>>>>>> somewhere, >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>> verify >>>>>>>>>>>>>>>>>>>>>>> it's working for your workload. ** >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>>>>>> Chris >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >>
