I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0, but will be included in alpha1.
On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <[email protected]> wrote: > Hi Chris, > > I know it's annoying to have last minute commit com in, but this is a > highly desirable feature for folks using GCP operators, is it possible to > include AIRFLOW-1635 > <https://github.com/apache/incubator-airflow/commit/ > b3e985a3146272ecfd3ceaaa0d8567e4e9e117d4> > in? > More than happy to offer help if there's something I can do. > Thanks a lot. > > Feng > > On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <[email protected]> > wrote: > > > Hey all, > > > > I have cut a 1.9.0alpha0 release of Airflow. You can download it here: > > > > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/ > > > > The bin tarball can be installed with pip: > > > > pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz > > > > The goal is to have the community install and run this to expose any bugs > > before we move on to official release candidates. > > > > Here are the outstanding blocker bugs for 1.9.0: > > > > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue > > AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are > marked > > as > > AIRFLOW-1055 |Bug |airflow/jobs.py:create_dag_run() exception > > for > > @on > > AIRFLOW-1018 |Bug |Scheduler DAG processes can not log to > stdout > > AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() exception for > > @once > > AIRFLOW-976 |Bug |Mark success running task causes it to fail > > > > Cheers, > > Chris > > > > On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <[email protected]> > > wrote: > > > > > Welp. Work got in the way, so I'll cut the beta on Monday. :) > > > > > > On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini < > [email protected]> > > > wrote: > > > > > >> Works for me. Will try and cut a beta before end of week. > > >> > > >> Blockers for 1.9.0 are: > > >> > > >> AIRFLOW-1611 |Bug |Customize logging in Airflow > > >> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue > > >> AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are > > marked > > >> as > > >> AIRFLOW-976 |Bug |Mark success running task causes it to fail > > >> > > >> > > >> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <[email protected]> > > >> wrote: > > >> > > >>> Hi Chris > > >>> > > >>> Can I suggest releasing a beta? The stable branch is only cut at RC > > >>> time. Betas allow us a broader exposure. It also gives us a point of > > >>> reference. > > >>> > > >>> In addition the list below are mostly longer standing issues that are > > >>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258, > > and > > >>> 976 as blocker? > > >>> > > >>> Cheers > > >>> Bolke > > >>> > > >>> Verstuurd vanaf mijn iPad > > >>> > > >>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini < > > [email protected]> > > >>> het volgende geschreven: > > >>> > > > >>> > Hey all, > > >>> > > > >>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta release, > > but > > >>> > seeing as there are several outstanding bugs, I'm going to delay. > > Here > > >>> are > > >>> > the bugs that I'm tracking: > > >>> > > > >>> > AIRFLOW-1611 |Bug |Customize logging in Airflow > > >>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue > > >>> > AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are > > >>> marked as > > >>> > AIRFLOW-1055 |Bug |airflow/jobs.py:create_dag_run() > > exception > > >>> for > > >>> > @on > > >>> > AIRFLOW-1018 |Bug |Scheduler DAG processes can not log to > > >>> stdout > > >>> > AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() exception > > >>> for @once > > >>> > AIRFLOW-988 |Bug |SLA Miss Callbacks Are Repeated if Email > > is > > >>> Not > > >>> > be > > >>> > AIRFLOW-976 |Bug |Mark success running task causes it to > > fail > > >>> > > > >>> > These are the priority issues. Once they're merged, I'll cut the > > >>> > v1-9-stable and beta release. > > >>> > > > >>> > If you can help clean this up, that would be really appreciated. > > >>> > > > >>> > Cheers, > > >>> > Chris > > >>> > > > >>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini < > > >>> [email protected]> > > >>> > wrote: > > >>> > > > >>> >> Marked it for 1.9.0. > > >>> >> > > >>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones <[email protected] > > > > >>> wrote: > > >>> >>> > > >>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA > > >>> callbacks > > >>> >>> are not working correctly without emails... Its not a major bug, > > but > > >>> it > > >>> >>> does cause us some annoyance in our current deployment. > > >>> >>> > > >>> >>> Link to Jira: > > >>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988 > > >>> >>> > > >>> >>> Link to PR: > > >>> >>> https://github.com/apache/incubator-airflow/pull/2415 > > >>> >>> > > >>> >>> Thanks! > > >>> >>> Charlie Jones > > >>> >>> > > >>> >>> CHARLIE JONES > > >>> >>> Data Engineer > > >>> >>> [email protected] | M: 972.821.7631 > > >>> >>> __________________________________________________ > > >>> >>> > > >>> >>> > > >>> >>> Programmatic Performance.* Localized.* > > >>> >>> __________________________________________________ > > >>> >>> > > >>> >>> 1407 Texas Street | Suite 202 | Fort Worth, TX 76102 > > >>> >>> 800.840.0768 | www.simpli.fi > > >>> >>> > > >>> >>> > > >>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini < > > >>> [email protected]> > > >>> >>> wrote: > > >>> >>> > > >>> >>>> Merged. > > >>> >>>> > > >>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley < > > >>> >>> [email protected]> > > >>> >>>> wrote: > > >>> >>>> > > >>> >>>>> Would it be possible to include AIRFLOW-1587? > > >>> >>>>> Running dags from the UI is currently broken on the 1.9.0 > branch > > >>> due > > >>> >>> to > > >>> >>>>> this issue. > > >>> >>>>> > > >>> >>>>> https://github.com/apache/incubator-airflow/pull/2590 > > >>> >>>>> > > >>> >>>>> Thanks, > > >>> >>>>> Ryan > > >>> >>>>> > > >>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko > > >>> >>> <[email protected] > > >>> >>>>> > > >>> >>>>> wrote: > > >>> >>>>> > > >>> >>>>>> Hi All, > > >>> >>>>>> > > >>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release: > > >>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631 > > >>> >>>>>> > > >>> >>>>>> Currently importing a custom logging configuration is not work > > (as > > >>> >>> far > > >>> >>>>> as I > > >>> >>>>>> know). Any feedback on the PR would also be appreciated. > > >>> >>>>>> > > >>> >>>>>> Cheers, Fokko > > >>> >>>>>> > > >>> >>>>>> > > >>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini < > > [email protected] > > >>> >: > > >>> >>>>>> > > >>> >>>>>>> Done! > > >>> >>>>>>> > > >>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford < > > >>> >>>>>>> [email protected]> wrote: > > >>> >>>>>>> > > >>> >>>>>>>> Can you slide the aws and emr connection type fix in? > > >>> >>>>>>>> > > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 < > > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> > > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 < > > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> > > >>> >>>>>>>> > > >>> >>>>>>>> It keeps the connection type from getting blanked out on > edit > > >>> >>> for > > >>> >>>>> these > > >>> >>>>>>>> types. > > >>> >>>>>>>> > > >>> >>>>>>>> Thanks, > > >>> >>>>>>>> Mike > > >>> >>>>>>>> > > >>> >>>>>>>> > > >>> >>>>>>>> > > >>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini < > > >>> >>>>> [email protected]> > > >>> >>>>>>>> wrote: > > >>> >>>>>>>>> > > >>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward > to > > >>> >>>> these > > >>> >>>>>>> fixes! > > >>> >>>>>>>>> > > >>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel < > > >>> >>>>> [email protected] > > >>> >>>>>> . > > >>> >>>>>>>> invalid > > >>> >>>>>>>>>> wrote: > > >>> >>>>>>>>> > > >>> >>>>>>>>>> Can we get this in? > > >>> >>>>>>>>>> > > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 > > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 > > >>> >>>>>>>>>> > > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ > > >>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2 > > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ > > >>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a > > >>> >>>>>>>>>> > > >>> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing to > > >>> >>> have > > >>> >>>> in > > >>> >>>>>> 1.9. > > >>> >>>>>>>>>> > > >>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini < > > >>> >>>>>>>> [email protected]> > > >>> >>>>>>>>>> wrote: > > >>> >>>>>>>>>> > > >>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then. > > >>> >>> Initial > > >>> >>>>>>> warning > > >>> >>>>>>>>>>> stands, that I will start locking down what can get into > > >>> >>> 1.9.0 > > >>> >>>> at > > >>> >>>>>>> that > > >>> >>>>>>>>>>> point. > > >>> >>>>>>>>>>> > > >>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin < > > >>> >>>>>> [email protected]> > > >>> >>>>>>>>>>> wrote: > > >>> >>>>>>>>>>> > > >>> >>>>>>>>>>>> No vote indeed, just to gather feedback on a particular > > >>> >>> fixed > > >>> >>>>>> point > > >>> >>>>>>> in > > >>> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than > to > > a > > >>> >>>> git > > >>> >>>>>>> pull. > > >>> >>>>>>>>>>>> > > >>> >>>>>>>>>>>> Bolke > > >>> >>>>>>>>>>>> > > >>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini < > > >>> >>>>> [email protected] > > >>> >>>>>>> > > >>> >>>>>>>>>>> wrote: > > >>> >>>>>>>>>>>>> > > >>> >>>>>>>>>>>>> I can do a beta. Is the process significantly > different? > > >>> >>>> IIRC, > > >>> >>>>>> it's > > >>> >>>>>>>>>>>>> basically the same, just no vote, right? > > >>> >>>>>>>>>>>>> > > >>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin < > > >>> >>>>>>> [email protected]> > > >>> >>>>>>>>>>>> wrote: > > >>> >>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right > away? > > >>> >>>>> Isn’t a > > >>> >>>>>>>>>> beta > > >>> >>>>>>>>>>> a > > >>> >>>>>>>>>>>>>> bit smarter? > > >>> >>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>> - Bolke > > >>> >>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini < > > >>> >>>>>> [email protected] > > >>> >>>>>>>> > > >>> >>>>>>>>>>>> wrote: > > >>> >>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>> Hey all, > > >>> >>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to cut > > >>> >>> the > > >>> >>>>>> stable > > >>> >>>>>>>>>>> branch > > >>> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the > > >>> >>> stable > > >>> >>>>>> branch > > >>> >>>>>>>>>> is > > >>> >>>>>>>>>>>>>> cut, I > > >>> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked > > into > > >>> >>>> the > > >>> >>>>>>>>>> branch, > > >>> >>>>>>>>>>>> and > > >>> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the > > >>> >>> release > > >>> >>>>>> out. > > >>> >>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>> Cheers, > > >>> >>>>>>>>>>>>>>> Chris > > >>> >>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini < > > >>> >>>>>>>>>>>> [email protected] > > >>> >>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>> wrote: > > >>> >>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>>> Hey all, > > >>> >>>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the > > >>> >>> outstanding > > >>> >>>> PRs > > >>> >>>>>>> that > > >>> >>>>>>>>>>> are > > >>> >>>>>>>>>>>>>>>> slated to be included into 1.9.0: > > >>> >>>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>>> ISSUE ID |STATUS |DESCRIPTION > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open |XSS Vulnerability in > Variable > > >>> >>>>>> endpoint > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open |Customize logging in > Airflow > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened |Fix log source of local > > >>> >>> loggers > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open |Rename the logger to log > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open |Fix minor LICENSE & NOTICE > > >>> >>> issue > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and > > >>> >>> unneeded > > >>> >>>>> code > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open |HDFSOperator to operate > HDFS > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open > |airflow/jobs.py:create_dag_ru > > >>> >>> n() > > >>> >>>>>>>>>> exception > > >>> >>>>>>>>>>>> for > > >>> >>>>>>>>>>>>>>>> @on > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open |active_dagruns shouldn't > > >>> >>> include > > >>> >>>>>> paused > > >>> >>>>>>>>>>> DAGs > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open |Scheduler DAG processes can > > >>> >>> not > > >>> >>>> log > > >>> >>>>>> to > > >>> >>>>>>>>>>> stdout > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open |TreeView displayed over > task > > >>> >>>>>> instances > > >>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open > |airflow/jobs.py:manage_slas() > > >>> >>>>>> exception > > >>> >>>>>>>>>> for > > >>> >>>>>>>>>>>>>>>> @once > > >>> >>>>>>>>>>>>>>>> AIRFLOW-976 |Open |Mark success running task > > >>> >>> causes > > >>> >>>> it > > >>> >>>>>> to > > >>> >>>>>>>>>> fail > > >>> >>>>>>>>>>>>>>>> AIRFLOW-914 |Open |Refactor > > >>> >>>>>> BackfillJobTest.test_backfill_ > > >>> >>>>>>>>>>>>>> examples > > >>> >>>>>>>>>>>>>>>> to > > >>> >>>>>>>>>>>>>>>> AIRFLOW-913 |Open |Refactor > > >>> >>>>>> tests.CoreTest.test_scheduler_ > > >>> >>>>>>>>>> job > > >>> >>>>>>>>>>>> to > > >>> >>>>>>>>>>>>>>>> real > > >>> >>>>>>>>>>>>>>>> AIRFLOW-912 |Open |Refactor tests and build > > >>> >>> matrix > > >>> >>>>>>>>>>>>>>>> AIRFLOW-888 |Open |Operators should not push > > >>> >>> XComs > > >>> >>>> by > > >>> >>>>>>>>>> default > > >>> >>>>>>>>>>>>>>>> AIRFLOW-828 |Open |Add maximum size for XComs > > >>> >>>>>>>>>>>>>>>> AIRFLOW-825 |Open |Add Dataflow semantics > > >>> >>>>>>>>>>>>>>>> AIRFLOW-788 |Open |Context unexpectedly added > to > > >>> >>>> hive > > >>> >>>>>> conf > > >>> >>>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked > into > > >>> >>> the > > >>> >>>>>> 1.9.0 > > >>> >>>>>>>>>>>> branch > > >>> >>>>>>>>>>>>>>>> shortly, so if you have something you want in, > please > > >>> >>> set > > >>> >>>>> the > > >>> >>>>>>> fix > > >>> >>>>>>>>>>>>>> version > > >>> >>>>>>>>>>>>>>>> to 1.9.0. > > >>> >>>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev > > cluster, > > >>> >>>> and > > >>> >>>>> it > > >>> >>>>>>> has > > >>> >>>>>>>>>>>> been > > >>> >>>>>>>>>>>>>>>> running smoothly for several days. > > >>> >>>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If > you > > >>> >>> run > > >>> >>>>>>>>>> Airflow, > > >>> >>>>>>>>>>>> it's > > >>> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test > branch > > >>> >>>>>> somewhere, > > >>> >>>>>>>>>> and > > >>> >>>>>>>>>>>>>> verify > > >>> >>>>>>>>>>>>>>>> it's working for your workload. ** > > >>> >>>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>>>> Cheers, > > >>> >>>>>>>>>>>>>>>> Chris > > >>> >>>>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>> > > >>> >>>>>>>>>>>>>> > > >>> >>>>>>>>>>>> > > >>> >>>>>>>>>>>> > > >>> >>>>>>>>>>> > > >>> >>>>>>>>>> > > >>> >>>>>>>> > > >>> >>>>>>>> > > >>> >>>>>>> > > >>> >>>>>> > > >>> >>>>> > > >>> >>>> > > >>> >>> > > >>> >> > > >>> >> > > >>> > > >> > > >> > > > > > >
