Hey all, Checking in. Has anyone tested this in their environments? Stable releases depend on community involvement.
Cheers, Chris On Tue, Oct 3, 2017 at 10:30 PM, Chris Riccomini <[email protected]> wrote: > I've added AIRFLOW-1635 to the v1-9-test branch. It's not in alpha0, but > will be included in alpha1. > > On Tue, Oct 3, 2017 at 4:13 PM, Feng Lu <[email protected]> wrote: > >> Hi Chris, >> >> I know it's annoying to have last minute commit com in, but this is a >> highly desirable feature for folks using GCP operators, is it possible to >> include AIRFLOW-1635 >> <https://github.com/apache/incubator-airflow/commit/b3e985a3 >> 146272ecfd3ceaaa0d8567e4e9e117d4> >> in? >> More than happy to offer help if there's something I can do. >> Thanks a lot. >> >> Feng >> >> On Mon, Oct 2, 2017 at 3:24 PM, Chris Riccomini <[email protected]> >> wrote: >> >> > Hey all, >> > >> > I have cut a 1.9.0alpha0 release of Airflow. You can download it here: >> > >> > https://dist.apache.org/repos/dist/dev/incubator/airflow/1.9.0alpha0/ >> > >> > The bin tarball can be installed with pip: >> > >> > pip install apache-airflow-1.9.0alpha0+incubating-bin.tar.gz >> > >> > The goal is to have the community install and run this to expose any >> bugs >> > before we move on to official release candidates. >> > >> > Here are the outstanding blocker bugs for 1.9.0: >> > >> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue >> > AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are >> marked >> > as >> > AIRFLOW-1055 |Bug |airflow/jobs.py:create_dag_run() exception >> > for >> > @on >> > AIRFLOW-1018 |Bug |Scheduler DAG processes can not log to >> stdout >> > AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() exception >> for >> > @once >> > AIRFLOW-976 |Bug |Mark success running task causes it to fail >> > >> > Cheers, >> > Chris >> > >> > On Fri, Sep 29, 2017 at 3:54 PM, Chris Riccomini <[email protected] >> > >> > wrote: >> > >> > > Welp. Work got in the way, so I'll cut the beta on Monday. :) >> > > >> > > On Thu, Sep 28, 2017 at 1:32 PM, Chris Riccomini < >> [email protected]> >> > > wrote: >> > > >> > >> Works for me. Will try and cut a beta before end of week. >> > >> >> > >> Blockers for 1.9.0 are: >> > >> >> > >> AIRFLOW-1611 |Bug |Customize logging in Airflow >> > >> AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue >> > >> AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are >> > marked >> > >> as >> > >> AIRFLOW-976 |Bug |Mark success running task causes it to >> fail >> > >> >> > >> >> > >> On Thu, Sep 28, 2017 at 1:09 PM, Bolke de Bruin <[email protected]> >> > >> wrote: >> > >> >> > >>> Hi Chris >> > >>> >> > >>> Can I suggest releasing a beta? The stable branch is only cut at RC >> > >>> time. Betas allow us a broader exposure. It also gives us a point of >> > >>> reference. >> > >>> >> > >>> In addition the list below are mostly longer standing issues that >> are >> > >>> also part of the 1.8.x branch. Maybe only consider 1611, 1525, 1258, >> > and >> > >>> 976 as blocker? >> > >>> >> > >>> Cheers >> > >>> Bolke >> > >>> >> > >>> Verstuurd vanaf mijn iPad >> > >>> >> > >>> > Op 28 sep. 2017 om 19:49 heeft Chris Riccomini < >> > [email protected]> >> > >>> het volgende geschreven: >> > >>> > >> > >>> > Hey all, >> > >>> > >> > >>> > I was planning to cut a 1.9.0 stable branch and 1.9.0 beta >> release, >> > but >> > >>> > seeing as there are several outstanding bugs, I'm going to delay. >> > Here >> > >>> are >> > >>> > the bugs that I'm tracking: >> > >>> > >> > >>> > AIRFLOW-1611 |Bug |Customize logging in Airflow >> > >>> > AIRFLOW-1525 |Improvement |Fix minor LICENSE & NOTICE issue >> > >>> > AIRFLOW-1258 |Bug |TaskInstances within SubDagOperator are >> > >>> marked as >> > >>> > AIRFLOW-1055 |Bug |airflow/jobs.py:create_dag_run() >> > exception >> > >>> for >> > >>> > @on >> > >>> > AIRFLOW-1018 |Bug |Scheduler DAG processes can not log to >> > >>> stdout >> > >>> > AIRFLOW-1013 |Bug |airflow/jobs.py:manage_slas() >> exception >> > >>> for @once >> > >>> > AIRFLOW-988 |Bug |SLA Miss Callbacks Are Repeated if >> Email >> > is >> > >>> Not >> > >>> > be >> > >>> > AIRFLOW-976 |Bug |Mark success running task causes it to >> > fail >> > >>> > >> > >>> > These are the priority issues. Once they're merged, I'll cut the >> > >>> > v1-9-stable and beta release. >> > >>> > >> > >>> > If you can help clean this up, that would be really appreciated. >> > >>> > >> > >>> > Cheers, >> > >>> > Chris >> > >>> > >> > >>> > On Thu, Sep 28, 2017 at 10:06 AM, Chris Riccomini < >> > >>> [email protected]> >> > >>> > wrote: >> > >>> > >> > >>> >> Marked it for 1.9.0. >> > >>> >> >> > >>> >>> On Thu, Sep 28, 2017 at 9:56 AM, Charlie Jones < >> [email protected]> >> > >>> wrote: >> > >>> >>> >> > >>> >>> Is there any chance we could include AIRFLOW-988 in 1.9.0? SLA >> > >>> callbacks >> > >>> >>> are not working correctly without emails... Its not a major bug, >> > but >> > >>> it >> > >>> >>> does cause us some annoyance in our current deployment. >> > >>> >>> >> > >>> >>> Link to Jira: >> > >>> >>> https://issues.apache.org/jira/browse/AIRFLOW-988 >> > >>> >>> >> > >>> >>> Link to PR: >> > >>> >>> https://github.com/apache/incubator-airflow/pull/2415 >> > >>> >>> >> > >>> >>> Thanks! >> > >>> >>> Charlie Jones >> > >>> >>> >> > >>> >>> CHARLIE JONES >> > >>> >>> Data Engineer >> > >>> >>> [email protected] | M: 972.821.7631 >> > >>> >>> __________________________________________________ >> > >>> >>> >> > >>> >>> >> > >>> >>> Programmatic Performance.* Localized.* >> > >>> >>> __________________________________________________ >> > >>> >>> >> > >>> >>> 1407 Texas Street | Suite 202 | Fort Worth, TX 76102 >> > >>> >>> 800.840.0768 | www.simpli.fi >> > >>> >>> >> > >>> >>> >> > >>> >>> On Wed, Sep 27, 2017 at 12:11 PM, Chris Riccomini < >> > >>> [email protected]> >> > >>> >>> wrote: >> > >>> >>> >> > >>> >>>> Merged. >> > >>> >>>> >> > >>> >>>> On Tue, Sep 26, 2017 at 2:25 PM, Ryan Buckley < >> > >>> >>> [email protected]> >> > >>> >>>> wrote: >> > >>> >>>> >> > >>> >>>>> Would it be possible to include AIRFLOW-1587? >> > >>> >>>>> Running dags from the UI is currently broken on the 1.9.0 >> branch >> > >>> due >> > >>> >>> to >> > >>> >>>>> this issue. >> > >>> >>>>> >> > >>> >>>>> https://github.com/apache/incubator-airflow/pull/2590 >> > >>> >>>>> >> > >>> >>>>> Thanks, >> > >>> >>>>> Ryan >> > >>> >>>>> >> > >>> >>>>> On Tue, Sep 26, 2017 at 8:14 AM, Driesprong, Fokko >> > >>> >>> <[email protected] >> > >>> >>>>> >> > >>> >>>>> wrote: >> > >>> >>>>> >> > >>> >>>>>> Hi All, >> > >>> >>>>>> >> > >>> >>>>>> I would like to include AIRFLOW-1611 in the 1.9.0 release: >> > >>> >>>>>> https://github.com/apache/incubator-airflow/pull/2631 >> > >>> >>>>>> >> > >>> >>>>>> Currently importing a custom logging configuration is not >> work >> > (as >> > >>> >>> far >> > >>> >>>>> as I >> > >>> >>>>>> know). Any feedback on the PR would also be appreciated. >> > >>> >>>>>> >> > >>> >>>>>> Cheers, Fokko >> > >>> >>>>>> >> > >>> >>>>>> >> > >>> >>>>>> 2017-09-25 23:27 GMT+02:00 Chris Riccomini < >> > [email protected] >> > >>> >: >> > >>> >>>>>> >> > >>> >>>>>>> Done! >> > >>> >>>>>>> >> > >>> >>>>>>> On Mon, Sep 25, 2017 at 5:11 AM, Michael Crawford < >> > >>> >>>>>>> [email protected]> wrote: >> > >>> >>>>>>> >> > >>> >>>>>>>> Can you slide the aws and emr connection type fix in? >> > >>> >>>>>>>> >> > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636 < >> > >>> >>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1636> >> > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626 < >> > >>> >>>>>>>> https://github.com/apache/incubator-airflow/pull/2626> >> > >>> >>>>>>>> >> > >>> >>>>>>>> It keeps the connection type from getting blanked out on >> edit >> > >>> >>> for >> > >>> >>>>> these >> > >>> >>>>>>>> types. >> > >>> >>>>>>>> >> > >>> >>>>>>>> Thanks, >> > >>> >>>>>>>> Mike >> > >>> >>>>>>>> >> > >>> >>>>>>>> >> > >>> >>>>>>>> >> > >>> >>>>>>>>> On Sep 21, 2017, at 1:27 PM, Chris Riccomini < >> > >>> >>>>> [email protected]> >> > >>> >>>>>>>> wrote: >> > >>> >>>>>>>>> >> > >>> >>>>>>>>> Absolutely. Just cherry-picked. I've been looking forward >> to >> > >>> >>>> these >> > >>> >>>>>>> fixes! >> > >>> >>>>>>>>> >> > >>> >>>>>>>>> On Wed, Sep 20, 2017 at 5:23 PM, Alex Guziel < >> > >>> >>>>> [email protected] >> > >>> >>>>>> . >> > >>> >>>>>>>> invalid >> > >>> >>>>>>>>>> wrote: >> > >>> >>>>>>>>> >> > >>> >>>>>>>>>> Can we get this in? >> > >>> >>>>>>>>>> >> > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1519 >> > >>> >>>>>>>>>> https://issues.apache.org/jira/browse/AIRFLOW-1621 >> > >>> >>>>>>>>>> >> > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ >> > >>> >>>>>>>>>> b6d2e0a46978e93e16576604624f57d1388814f2 >> > >>> >>>>>>>>>> https://github.com/apache/incubator-airflow/commit/ >> > >>> >>>>>>>>>> 656d045e90bf67ca484a3778b2a07a419bfb324a >> > >>> >>>>>>>>>> >> > >>> >>>>>>>>>> It speeds up loading times a lot, so it's a good thing to >> > >>> >>> have >> > >>> >>>> in >> > >>> >>>>>> 1.9. >> > >>> >>>>>>>>>> >> > >>> >>>>>>>>>> On Wed, Sep 20, 2017 at 11:14 AM, Chris Riccomini < >> > >>> >>>>>>>> [email protected]> >> > >>> >>>>>>>>>> wrote: >> > >>> >>>>>>>>>> >> > >>> >>>>>>>>>>> Sounds good. I'll plan on stable+beta next week, then. >> > >>> >>> Initial >> > >>> >>>>>>> warning >> > >>> >>>>>>>>>>> stands, that I will start locking down what can get into >> > >>> >>> 1.9.0 >> > >>> >>>> at >> > >>> >>>>>>> that >> > >>> >>>>>>>>>>> point. >> > >>> >>>>>>>>>>> >> > >>> >>>>>>>>>>> On Wed, Sep 20, 2017 at 11:10 AM, Bolke de Bruin < >> > >>> >>>>>> [email protected]> >> > >>> >>>>>>>>>>> wrote: >> > >>> >>>>>>>>>>> >> > >>> >>>>>>>>>>>> No vote indeed, just to gather feedback on a particular >> > >>> >>> fixed >> > >>> >>>>>> point >> > >>> >>>>>>> in >> > >>> >>>>>>>>>>>> time. It also gives a bit more trust to a tarball than >> to >> > a >> > >>> >>>> git >> > >>> >>>>>>> pull. >> > >>> >>>>>>>>>>>> >> > >>> >>>>>>>>>>>> Bolke >> > >>> >>>>>>>>>>>> >> > >>> >>>>>>>>>>>>> On 20 Sep 2017, at 20:09, Chris Riccomini < >> > >>> >>>>> [email protected] >> > >>> >>>>>>> >> > >>> >>>>>>>>>>> wrote: >> > >>> >>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>> I can do a beta. Is the process significantly >> different? >> > >>> >>>> IIRC, >> > >>> >>>>>> it's >> > >>> >>>>>>>>>>>>> basically the same, just no vote, right? >> > >>> >>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>> On Wed, Sep 20, 2017 at 10:56 AM, Bolke de Bruin < >> > >>> >>>>>>> [email protected]> >> > >>> >>>>>>>>>>>> wrote: >> > >>> >>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>> Are you sure you want to go ahead and do RCs right >> away? >> > >>> >>>>> Isn’t a >> > >>> >>>>>>>>>> beta >> > >>> >>>>>>>>>>> a >> > >>> >>>>>>>>>>>>>> bit smarter? >> > >>> >>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>> - Bolke >> > >>> >>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>> On 20 Sep 2017, at 19:41, Chris Riccomini < >> > >>> >>>>>> [email protected] >> > >>> >>>>>>>> >> > >>> >>>>>>>>>>>> wrote: >> > >>> >>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>> Hey all, >> > >>> >>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>> I want to send out a warning that I'm planning to >> cut >> > >>> >>> the >> > >>> >>>>>> stable >> > >>> >>>>>>>>>>> branch >> > >>> >>>>>>>>>>>>>>> next week, and begin the RC1 release vote. Once the >> > >>> >>> stable >> > >>> >>>>>> branch >> > >>> >>>>>>>>>> is >> > >>> >>>>>>>>>>>>>> cut, I >> > >>> >>>>>>>>>>>>>>> will be locking down what commits get cherry picked >> > into >> > >>> >>>> the >> > >>> >>>>>>>>>> branch, >> > >>> >>>>>>>>>>>> and >> > >>> >>>>>>>>>>>>>>> will only be doing PRs that are required to get the >> > >>> >>> release >> > >>> >>>>>> out. >> > >>> >>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>> Cheers, >> > >>> >>>>>>>>>>>>>>> Chris >> > >>> >>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>> On Mon, Sep 18, 2017 at 11:19 AM, Chris Riccomini < >> > >>> >>>>>>>>>>>> [email protected] >> > >>> >>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>> wrote: >> > >>> >>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>>> Hey all, >> > >>> >>>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>>> An update on the 1.9.0 release. Here are the >> > >>> >>> outstanding >> > >>> >>>> PRs >> > >>> >>>>>>> that >> > >>> >>>>>>>>>>> are >> > >>> >>>>>>>>>>>>>>>> slated to be included into 1.9.0: >> > >>> >>>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>>> ISSUE ID |STATUS |DESCRIPTION >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1617 |Open |XSS Vulnerability in >> Variable >> > >>> >>>>>> endpoint >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1611 |Open |Customize logging in >> Airflow >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1605 |Reopened |Fix log source of local >> > >>> >>> loggers >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1604 |Open |Rename the logger to log >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1525 |Open |Fix minor LICENSE & NOTICE >> > >>> >>> issue >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1499 |In Progres|Eliminate duplicate and >> > >>> >>> unneeded >> > >>> >>>>> code >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1198 |Open |HDFSOperator to operate >> HDFS >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1055 |Open >> |airflow/jobs.py:create_dag_ru >> > >>> >>> n() >> > >>> >>>>>>>>>> exception >> > >>> >>>>>>>>>>>> for >> > >>> >>>>>>>>>>>>>>>> @on >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1019 |Open |active_dagruns shouldn't >> > >>> >>> include >> > >>> >>>>>> paused >> > >>> >>>>>>>>>>> DAGs >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1018 |Open |Scheduler DAG processes >> can >> > >>> >>> not >> > >>> >>>> log >> > >>> >>>>>> to >> > >>> >>>>>>>>>>> stdout >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1015 |Open |TreeView displayed over >> task >> > >>> >>>>>> instances >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-1013 |Open >> |airflow/jobs.py:manage_slas() >> > >>> >>>>>> exception >> > >>> >>>>>>>>>> for >> > >>> >>>>>>>>>>>>>>>> @once >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-976 |Open |Mark success running task >> > >>> >>> causes >> > >>> >>>> it >> > >>> >>>>>> to >> > >>> >>>>>>>>>> fail >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-914 |Open |Refactor >> > >>> >>>>>> BackfillJobTest.test_backfill_ >> > >>> >>>>>>>>>>>>>> examples >> > >>> >>>>>>>>>>>>>>>> to >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-913 |Open |Refactor >> > >>> >>>>>> tests.CoreTest.test_scheduler_ >> > >>> >>>>>>>>>> job >> > >>> >>>>>>>>>>>> to >> > >>> >>>>>>>>>>>>>>>> real >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-912 |Open |Refactor tests and build >> > >>> >>> matrix >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-888 |Open |Operators should not push >> > >>> >>> XComs >> > >>> >>>> by >> > >>> >>>>>>>>>> default >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-828 |Open |Add maximum size for XComs >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-825 |Open |Add Dataflow semantics >> > >>> >>>>>>>>>>>>>>>> AIRFLOW-788 |Open |Context unexpectedly >> added to >> > >>> >>>> hive >> > >>> >>>>>> conf >> > >>> >>>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>>> I will be locking down what can get cherry-picked >> into >> > >>> >>> the >> > >>> >>>>>> 1.9.0 >> > >>> >>>>>>>>>>>> branch >> > >>> >>>>>>>>>>>>>>>> shortly, so if you have something you want in, >> please >> > >>> >>> set >> > >>> >>>>> the >> > >>> >>>>>>> fix >> > >>> >>>>>>>>>>>>>> version >> > >>> >>>>>>>>>>>>>>>> to 1.9.0. >> > >>> >>>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>>> We (at WePay) have deployed 1.9.0 into our dev >> > cluster, >> > >>> >>>> and >> > >>> >>>>> it >> > >>> >>>>>>> has >> > >>> >>>>>>>>>>>> been >> > >>> >>>>>>>>>>>>>>>> running smoothly for several days. >> > >>> >>>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>>> ** I could really use help verifying stability. If >> you >> > >>> >>> run >> > >>> >>>>>>>>>> Airflow, >> > >>> >>>>>>>>>>>> it's >> > >>> >>>>>>>>>>>>>>>> in your best interest to deploy the 1.9.0 test >> branch >> > >>> >>>>>> somewhere, >> > >>> >>>>>>>>>> and >> > >>> >>>>>>>>>>>>>> verify >> > >>> >>>>>>>>>>>>>>>> it's working for your workload. ** >> > >>> >>>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>>>> Cheers, >> > >>> >>>>>>>>>>>>>>>> Chris >> > >>> >>>>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>>>> >> > >>> >>>>>>>>>>>> >> > >>> >>>>>>>>>>>> >> > >>> >>>>>>>>>>> >> > >>> >>>>>>>>>> >> > >>> >>>>>>>> >> > >>> >>>>>>>> >> > >>> >>>>>>> >> > >>> >>>>>> >> > >>> >>>>> >> > >>> >>>> >> > >>> >>> >> > >>> >> >> > >>> >> >> > >>> >> > >> >> > >> >> > > >> > >> > >
