Thank you so much Chris! On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <[email protected]> wrote:
> I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0, but > will be included in alpha1. > > On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <[email protected]> wrote: > > > Hi Chris, > > > > I know it's annoying to have last minute commit com in, but this is a > > highly desirable feature for folks using GCP operators, is it possible to > > include AIRFLOW-1635 > > <https://github.com/apache/incubator-airflow/commit/ > > b3e985a3146272ecfd3ceaaa0d8567e4e9e117d4> > > in? > > More than happy to offer help if there's something I can do. > > Thanks a lot. > > > > Feng > > > > On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <[email protected]> > > wrote: > > > > > Hey all, > > > > > > I have cut a 1.9.0alpha0 release of Airflow. You can download it here: > > > > > > https://dist.apache.org/repos/dist/dev/incubator/airflow/1. > 9.0alpha0/ > > > > > > The bin tarball can be installed with pip: > > > > > > pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz > > > > > > The goal is to have the community install and run this to expose any > bugs > > > before we move on to official release candidates. > > > > > > Here are the outstanding blocker bugs for 1.9.0: > > > > > > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue > > > AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are > > marked > > > as > > > AIRFLOW-1055 |Bug |airflow/jobs.py:create_dag_run() > exception > > > for > > > @on > > > AIRFLOW-1018 |Bug |Scheduler DAG processes can not log to > > stdout > > > AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() exception > for > > > @once > > > AIRFLOW-976 |Bug |Mark success running task causes it to > fail > > > > > > Cheers, > > > Chris > > > > > > On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini < > [email protected]> > > > wrote: > > > > > > > Welp. Work got in the way, so I'll cut the beta on Monday. :) > > > > > > > > On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini < > > [email protected]> > > > > wrote: > > > > > > > >> Works for me. Will try and cut a beta before end of week. > > > >> > > > >> Blockers for 1.9.0 are: > > > >> > > > >> AIRFLOW-1611 |Bug |Customize logging in Airflow > > > >> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue > > > >> AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are > > > marked > > > >> as > > > >> AIRFLOW-976 |Bug |Mark success running task causes it to > fail > > > >> > > > >> > > > >> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <[email protected]> > > > >> wrote: > > > >> > > > >>> Hi Chris > > > >>> > > > >>> Can I suggest releasing a beta? The stable branch is only cut at RC > > > >>> time. Betas allow us a broader exposure. It also gives us a point > of > > > >>> reference. > > > >>> > > > >>> In addition the list below are mostly longer standing issues that > are > > > >>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, > 1258, > > > and > > > >>> 976 as blocker? > > > >>> > > > >>> Cheers > > > >>> Bolke > > > >>> > > > >>> Verstuurd vanaf mijn iPad > > > >>> > > > >>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini < > > > [email protected]> > > > >>> het volgende geschreven: > > > >>> > > > > >>> > Hey all, > > > >>> > > > > >>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta > release, > > > but > > > >>> > seeing as there are several outstanding bugs, I'm going to delay. > > > Here > > > >>> are > > > >>> > the bugs that I'm tracking: > > > >>> > > > > >>> > AIRFLOW-1611 |Bug |Customize logging in Airflow > > > >>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue > > > >>> > AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator > are > > > >>> marked as > > > >>> > AIRFLOW-1055 |Bug |airflow/jobs.py:create_dag_run() > > > exception > > > >>> for > > > >>> > @on > > > >>> > AIRFLOW-1018 |Bug |Scheduler DAG processes can not log to > > > >>> stdout > > > >>> > AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() > exception > > > >>> for @once > > > >>> > AIRFLOW-988 |Bug |SLA Miss Callbacks Are Repeated if > Email > > > is > > > >>> Not > > > >>> > be > > > >>> > AIRFLOW-976 |Bug |Mark success running task causes it to > > > fail > > > >>> > > > > >>> > These are the priority issues. Once they're merged, I'll cut the > > > >>> > v1-9-stable and beta release. > > > >>> > > > > >>> > If you can help clean this up, that would be really appreciated. > > > >>> > > > > >>> > Cheers, > > > >>> > Chris > > > >>> > > > > >>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini < > > > >>> [email protected]> > > > >>> > wrote: > > > >>> > > > > >>> >> Marked it for 1.9.0. > > > >>> >> > > > >>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones < > [email protected] > > > > > > >>> wrote: > > > >>> >>> > > > >>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA > > > >>> callbacks > > > >>> >>> are not working correctly without emails... Its not a major > bug, > > > but > > > >>> it > > > >>> >>> does cause us some annoyance in our current deployment. > > > >>> >>> > > > >>> >>> Link to Jira: > > > >>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988 > > > >>> >>> > > > >>> >>> Link to PR: > > > >>> >>> https://github.com/apache/incubator-airflow/pull/2415 > > > >>> >>> > > > >>> >>> Thanks! > > > >>> >>> Charlie Jones > > > >>> >>> > > > >>> >>> CHARLIE JONES > > > >>> >>> Data Engineer > > > >>> >>> [email protected] | M: 972.821.7631 > > > >>> >>> __________________________________________________ > > > >>> >>> > > > >>> >>> > > > >>> >>> Programmatic Performance.* Localized.* > > > >>> >>> __________________________________________________ > > > >>> >>> > > > >>> >>> 1407 Texas Street | Suite 202 | Fort Worth, TX 76102 > > > >>> >>> 800.840.0768 | www.simpli.fi > > > >>> >>> > > > >>> >>> > > > >>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini < > > > >>> [email protected]> > > > >>> >>> wrote: > > > >>> >>> > > > >>> >>>> Merged. > > > >>> >>>> > > > >>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley < > > > >>> >>> [email protected]> > > > >>> >>>> wrote: > > > >>> >>>> > > > >>> >>>>> Would it be possible to include AIRFLOW-1587? > > > >>> >>>>> Running dags from the UI is currently broken on the 1.9.0 > > branch > > > >>> due > > > >>> >>> to > > > >>> >>>>> this issue. > > > >>> >>>>> > > > >>> >>>>> https://github.com/apache/incubator-airflow/pull/2590 > > > >>> >>>>> > > > >>> >>>>> Thanks, > > > >>> >>>>> Ryan > > > >>> >>>>> > > > >>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko > > > >>> >>> <[email protected] > > > >>> >>>>> > > > >>> >>>>> wrote: > > > >>> >>>>> > > > >>> >>>>>> Hi All, > > > >>> >>>>>> > > > >>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release: > > > >>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631 > > > >>> >>>>>> > > > >>> >>>>>> Currently importing a custom logging configuration is not > work > > > (as > > > >>> >>> far > > > >>> >>>>> as I > > > >>> >>>>>> know). Any feedback on the PR would also be appreciated. > > > >>> >>>>>> > > > >>> >>>>>> Cheers, Fokko > > > >>> >>>>>> > > > >>> >>>>>> > > > >>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini < > > > [email protected] > > > >>> >: > > > >>> >>>>>> > > > >>> >>>>>>> Done! > > > >>> >>>>>>> > > > >>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford < > > > >>> >>>>>>> [email protected]> wrote: > > > >>> >>>>>>> > > > >>> >>>>>>>> Can you slide the aws and emr connection type fix in? > > > >>> >>>>>>>> > > > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 < > > > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> > > > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 < > > > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> > > > >>> >>>>>>>> > > > >>> >>>>>>>> It keeps the connection type from getting blanked out on > > edit > > > >>> >>> for > > > >>> >>>>> these > > > >>> >>>>>>>> types. > > > >>> >>>>>>>> > > > >>> >>>>>>>> Thanks, > > > >>> >>>>>>>> Mike > > > >>> >>>>>>>> > > > >>> >>>>>>>> > > > >>> >>>>>>>> > > > >>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini < > > > >>> >>>>> [email protected]> > > > >>> >>>>>>>> wrote: > > > >>> >>>>>>>>> > > > >>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward > > to > > > >>> >>>> these > > > >>> >>>>>>> fixes! > > > >>> >>>>>>>>> > > > >>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel < > > > >>> >>>>> [email protected] > > > >>> >>>>>> . > > > >>> >>>>>>>> invalid > > > >>> >>>>>>>>>> wrote: > > > >>> >>>>>>>>> > > > >>> >>>>>>>>>> Can we get this in? > > > >>> >>>>>>>>>> > > > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 > > > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 > > > >>> >>>>>>>>>> > > > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ > > > >>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2 > > > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ > > > >>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a > > > >>> >>>>>>>>>> > > > >>> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing > to > > > >>> >>> have > > > >>> >>>> in > > > >>> >>>>>> 1.9. > > > >>> >>>>>>>>>> > > > >>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini < > > > >>> >>>>>>>> [email protected]> > > > >>> >>>>>>>>>> wrote: > > > >>> >>>>>>>>>> > > > >>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then. > > > >>> >>> Initial > > > >>> >>>>>>> warning > > > >>> >>>>>>>>>>> stands, that I will start locking down what can get > into > > > >>> >>> 1.9.0 > > > >>> >>>> at > > > >>> >>>>>>> that > > > >>> >>>>>>>>>>> point. > > > >>> >>>>>>>>>>> > > > >>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin < > > > >>> >>>>>> [email protected]> > > > >>> >>>>>>>>>>> wrote: > > > >>> >>>>>>>>>>> > > > >>> >>>>>>>>>>>> No vote indeed, just to gather feedback on a > particular > > > >>> >>> fixed > > > >>> >>>>>> point > > > >>> >>>>>>> in > > > >>> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than > > to > > > a > > > >>> >>>> git > > > >>> >>>>>>> pull. > > > >>> >>>>>>>>>>>> > > > >>> >>>>>>>>>>>> Bolke > > > >>> >>>>>>>>>>>> > > > >>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini < > > > >>> >>>>> [email protected] > > > >>> >>>>>>> > > > >>> >>>>>>>>>>> wrote: > > > >>> >>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>> I can do a beta. Is the process significantly > > different? > > > >>> >>>> IIRC, > > > >>> >>>>>> it's > > > >>> >>>>>>>>>>>>> basically the same, just no vote, right? > > > >>> >>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin < > > > >>> >>>>>>> [email protected]> > > > >>> >>>>>>>>>>>> wrote: > > > >>> >>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right > > away? > > > >>> >>>>> Isn’t a > > > >>> >>>>>>>>>> beta > > > >>> >>>>>>>>>>> a > > > >>> >>>>>>>>>>>>>> bit smarter? > > > >>> >>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>> - Bolke > > > >>> >>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini < > > > >>> >>>>>> [email protected] > > > >>> >>>>>>>> > > > >>> >>>>>>>>>>>> wrote: > > > >>> >>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>> Hey all, > > > >>> >>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to > cut > > > >>> >>> the > > > >>> >>>>>> stable > > > >>> >>>>>>>>>>> branch > > > >>> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the > > > >>> >>> stable > > > >>> >>>>>> branch > > > >>> >>>>>>>>>> is > > > >>> >>>>>>>>>>>>>> cut, I > > > >>> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked > > > into > > > >>> >>>> the > > > >>> >>>>>>>>>> branch, > > > >>> >>>>>>>>>>>> and > > > >>> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the > > > >>> >>> release > > > >>> >>>>>> out. > > > >>> >>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>> Cheers, > > > >>> >>>>>>>>>>>>>>> Chris > > > >>> >>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini < > > > >>> >>>>>>>>>>>> [email protected] > > > >>> >>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>> wrote: > > > >>> >>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>>> Hey all, > > > >>> >>>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the > > > >>> >>> outstanding > > > >>> >>>> PRs > > > >>> >>>>>>> that > > > >>> >>>>>>>>>>> are > > > >>> >>>>>>>>>>>>>>>> slated to be included into 1.9.0: > > > >>> >>>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>>> ISSUE ID |STATUS |DESCRIPTION > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open |XSS Vulnerability in > > Variable > > > >>> >>>>>> endpoint > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open |Customize logging in > > Airflow > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened |Fix log source of local > > > >>> >>> loggers > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open |Rename the logger to log > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open |Fix minor LICENSE & > NOTICE > > > >>> >>> issue > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and > > > >>> >>> unneeded > > > >>> >>>>> code > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open |HDFSOperator to operate > > HDFS > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open > > |airflow/jobs.py:create_dag_ru > > > >>> >>> n() > > > >>> >>>>>>>>>> exception > > > >>> >>>>>>>>>>>> for > > > >>> >>>>>>>>>>>>>>>> @on > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open |active_dagruns shouldn't > > > >>> >>> include > > > >>> >>>>>> paused > > > >>> >>>>>>>>>>> DAGs > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open |Scheduler DAG processes > can > > > >>> >>> not > > > >>> >>>> log > > > >>> >>>>>> to > > > >>> >>>>>>>>>>> stdout > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open |TreeView displayed over > > task > > > >>> >>>>>> instances > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open > > |airflow/jobs.py:manage_slas() > > > >>> >>>>>> exception > > > >>> >>>>>>>>>> for > > > >>> >>>>>>>>>>>>>>>> @once > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-976 |Open |Mark success running task > > > >>> >>> causes > > > >>> >>>> it > > > >>> >>>>>> to > > > >>> >>>>>>>>>> fail > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-914 |Open |Refactor > > > >>> >>>>>> BackfillJobTest.test_backfill_ > > > >>> >>>>>>>>>>>>>> examples > > > >>> >>>>>>>>>>>>>>>> to > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-913 |Open |Refactor > > > >>> >>>>>> tests.CoreTest.test_scheduler_ > > > >>> >>>>>>>>>> job > > > >>> >>>>>>>>>>>> to > > > >>> >>>>>>>>>>>>>>>> real > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-912 |Open |Refactor tests and build > > > >>> >>> matrix > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-888 |Open |Operators should not push > > > >>> >>> XComs > > > >>> >>>> by > > > >>> >>>>>>>>>> default > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-828 |Open |Add maximum size for > XComs > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-825 |Open |Add Dataflow semantics > > > >>> >>>>>>>>>>>>>>>> AIRFLOW-788 |Open |Context unexpectedly > added > > to > > > >>> >>>> hive > > > >>> >>>>>> conf > > > >>> >>>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked > > into > > > >>> >>> the > > > >>> >>>>>> 1.9.0 > > > >>> >>>>>>>>>>>> branch > > > >>> >>>>>>>>>>>>>>>> shortly, so if you have something you want in, > > please > > > >>> >>> set > > > >>> >>>>> the > > > >>> >>>>>>> fix > > > >>> >>>>>>>>>>>>>> version > > > >>> >>>>>>>>>>>>>>>> to 1.9.0. > > > >>> >>>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev > > > cluster, > > > >>> >>>> and > > > >>> >>>>> it > > > >>> >>>>>>> has > > > >>> >>>>>>>>>>>> been > > > >>> >>>>>>>>>>>>>>>> running smoothly for several days. > > > >>> >>>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If > > you > > > >>> >>> run > > > >>> >>>>>>>>>> Airflow, > > > >>> >>>>>>>>>>>> it's > > > >>> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test > > branch > > > >>> >>>>>> somewhere, > > > >>> >>>>>>>>>> and > > > >>> >>>>>>>>>>>>>> verify > > > >>> >>>>>>>>>>>>>>>> it's working for your workload. ** > > > >>> >>>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>>>> Cheers, > > > >>> >>>>>>>>>>>>>>>> Chris > > > >>> >>>>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>>>> > > > >>> >>>>>>>>>>>> > > > >>> >>>>>>>>>>>> > > > >>> >>>>>>>>>>> > > > >>> >>>>>>>>>> > > > >>> >>>>>>>> > > > >>> >>>>>>>> > > > >>> >>>>>>> > > > >>> >>>>>> > > > >>> >>>>> > > > >>> >>>> > > > >>> >>> > > > >>> >> > > > >>> >> > > > >>> > > > >> > > > >> > > > > > > > > > >
