Graduation resolution passed - Airflow is a TLP

[Jakob Homan]

Hey all-
   The Board minutes haven't been published yet (probably due to
Holiday-related slowness), but I can see through the admin tool that
our Graduation resolution was approved yesterday at the meeting.
Airflow is the 199th current active Top Level Project in Apache.

Congrats all.

-Jakob

Re: [RESULT] Graduate Apache Airflow as a TLP

[Jakob Homan]

The VOTE passed in Incubator.  I'll add it the resolution to the Board
agenda, who meet this month on the 19th.  They'll approve and Airflow
will be a TLP.

-Jakob
On Wed, Dec 5, 2018 at 3:31 PM Jakob Homan  wrote:
>
> Been traveling today.  Just started the VOTE over in the general@ in 
> Incubator.
>
> -jg
> On Wed, Dec 5, 2018 at 1:26 PM Kaxil Naik  wrote:
> >
> > Hi Jakob,
> >
> > Did you raise this with IPMC, can we track it somewhere?
> >
> > Excited for graduation :-)
> >
> > Regards,
> > Kaxil
> >
> > On Tue, Dec 4, 2018, 20:04 Ash Berlin-Taylor  >>
> >> Missed my vote off that list :)
> >>
> >> > On 4 Dec 2018, at 18:17, Jakob Homan  wrote:
> >> >
> >> > I neglected to add my binding +1, so I'll do so now.
> >> >
> >> > With three days having elapsed, the VOTE is concluded successfully.
> >> >
> >> > Overall: 20 x +1 votes, 0 x -1 votes
> >> >
> >> > Binding +1 x 10: Kaxil, Tao, Bolke, Fokko, Maxime, Arthur, Hitesh,
> >> > Chris, Sid, Jakob.
> >> > Non-binding +1 x 10: Daniel, Shah, Stefan, Kevin, Marc, Sunil,
> >> > Adityan, Deng, Neelesh, Sai
> >> >
> >> > I'll use this result to start the corresponding VOTE on the IPMC.  I'm
> >> > at an offsite today, so I have limited email time.  Likely will open
> >> > the VOTE this evening.
> >> >
> >> > Thanks everyone.
> >> > -Jakob
> >> >
> >> >
> >> > On Tue, Dec 4, 2018 at 6:03 AM Bolke de Bruin  wrote:
> >> >>
> >> >> Shall we close the vote? @jakob?
> >> >>
> >> >>> On 2 Dec 2018, at 13:08, Sid Anand  wrote:
> >> >>>
> >> >>> +1 binding
> >> >>>
> >> >>> Woot! Thanks to all for this happy day!
> >> >>> -s
> >> >>>
> >> >>> On Sun, Dec 2, 2018 at 1:25 AM Sai Phanindhra  
> >> >>> wrote:
> >> >>>
> >> >>>> +1 (non binding)
> >> >>>>
> >> >>>> Excited to see this happenning.
> >> >>>>
> >> >>>> On Sat 1 Dec, 2018, 20:35  >> >>>>
> >> >>>>> +1 (binding)!
> >> >>>>>
> >> >>>>> On 30 November 2018 21:33:14 GMT, Jakob Homan  
> >> >>>>> wrote:
> >> >>>>>> Hey all!
> >> >>>>>>
> >> >>>>>> Following a very successful DISCUSS[1] regarding graduating Airflow 
> >> >>>>>> to
> >> >>>>>> Top Level Project (TLP) status, I'm starting the official VOTE.
> >> >>>>>>
> >> >>>>>> Since entering the Incubator in 2016, the community has:
> >> >>>>>> * successfully produced 7 releases
> >> >>>>>> * added 9 new committers/PPMC members
> >> >>>>>> * built a diverse group of committers from multiple different 
> >> >>>>>> employers
> >> >>>>>> * had more than 3,300 JIRA tickets opened
> >> >>>>>> * completed the project maturity model with positive responses[2]
> >> >>>>>>
> >> >>>>>> Accordingly, I believe we're ready to graduate and am calling a VOTE
> >> >>>>>> on the following graduation resolution.  This VOTE will remain open
> >> >>>>>> for at least 72 hours.  If successful, the resolution will be
> >> >>>>>> forwarded to the IPMC for its consideration.  If that VOTE is
> >> >>>>>> successful, the resolution will be voted upon by the Board at its 
> >> >>>>>> next
> >> >>>>>> monthly meeting.
> >> >>>>>>
> >> >>>>>> Everyone is encouraged to vote, even if their vote is not binding.
> >> >>>>>> We've built a nice community here, let's make sure everyone has 
> >> >>>>>> their
> >> >>>>>> voice heard.
> >> >>>>>>
> >> >>>>>> Thanks,
> >> >>>>>> Jakob
> >> >>>>>>
> >> >>>>>> [1]
> >> >>>>>>
> >> >>>&g

Re: [RESULT] Graduate Apache Airflow as a TLP

[Jakob Homan]

Been traveling today.  Just started the VOTE over in the general@ in Incubator.

-jg
On Wed, Dec 5, 2018 at 1:26 PM Kaxil Naik  wrote:
>
> Hi Jakob,
>
> Did you raise this with IPMC, can we track it somewhere?
>
> Excited for graduation :-)
>
> Regards,
> Kaxil
>
> On Tue, Dec 4, 2018, 20:04 Ash Berlin-Taylor >
>> Missed my vote off that list :)
>>
>> > On 4 Dec 2018, at 18:17, Jakob Homan  wrote:
>> >
>> > I neglected to add my binding +1, so I'll do so now.
>> >
>> > With three days having elapsed, the VOTE is concluded successfully.
>> >
>> > Overall: 20 x +1 votes, 0 x -1 votes
>> >
>> > Binding +1 x 10: Kaxil, Tao, Bolke, Fokko, Maxime, Arthur, Hitesh,
>> > Chris, Sid, Jakob.
>> > Non-binding +1 x 10: Daniel, Shah, Stefan, Kevin, Marc, Sunil,
>> > Adityan, Deng, Neelesh, Sai
>> >
>> > I'll use this result to start the corresponding VOTE on the IPMC.  I'm
>> > at an offsite today, so I have limited email time.  Likely will open
>> > the VOTE this evening.
>> >
>> > Thanks everyone.
>> > -Jakob
>> >
>> >
>> > On Tue, Dec 4, 2018 at 6:03 AM Bolke de Bruin  wrote:
>> >>
>> >> Shall we close the vote? @jakob?
>> >>
>> >>> On 2 Dec 2018, at 13:08, Sid Anand  wrote:
>> >>>
>> >>> +1 binding
>> >>>
>> >>> Woot! Thanks to all for this happy day!
>> >>> -s
>> >>>
>> >>> On Sun, Dec 2, 2018 at 1:25 AM Sai Phanindhra  
>> >>> wrote:
>> >>>
>> >>>> +1 (non binding)
>> >>>>
>> >>>> Excited to see this happenning.
>> >>>>
>> >>>> On Sat 1 Dec, 2018, 20:35 > >>>>
>> >>>>> +1 (binding)!
>> >>>>>
>> >>>>> On 30 November 2018 21:33:14 GMT, Jakob Homan  
>> >>>>> wrote:
>> >>>>>> Hey all!
>> >>>>>>
>> >>>>>> Following a very successful DISCUSS[1] regarding graduating Airflow to
>> >>>>>> Top Level Project (TLP) status, I'm starting the official VOTE.
>> >>>>>>
>> >>>>>> Since entering the Incubator in 2016, the community has:
>> >>>>>> * successfully produced 7 releases
>> >>>>>> * added 9 new committers/PPMC members
>> >>>>>> * built a diverse group of committers from multiple different 
>> >>>>>> employers
>> >>>>>> * had more than 3,300 JIRA tickets opened
>> >>>>>> * completed the project maturity model with positive responses[2]
>> >>>>>>
>> >>>>>> Accordingly, I believe we're ready to graduate and am calling a VOTE
>> >>>>>> on the following graduation resolution.  This VOTE will remain open
>> >>>>>> for at least 72 hours.  If successful, the resolution will be
>> >>>>>> forwarded to the IPMC for its consideration.  If that VOTE is
>> >>>>>> successful, the resolution will be voted upon by the Board at its next
>> >>>>>> monthly meeting.
>> >>>>>>
>> >>>>>> Everyone is encouraged to vote, even if their vote is not binding.
>> >>>>>> We've built a nice community here, let's make sure everyone has their
>> >>>>>> voice heard.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Jakob
>> >>>>>>
>> >>>>>> [1]
>> >>>>>>
>> >>>>>
>> >>>> https://lists.apache.org/thread.html/%3c0a763b0b-7d0d-4353-979a-ac6769eb0...@gmail.com%3E
>> >>>>>> [2]
>> >>>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
>> >>>>>>
>> >>>>>> 
>> >>>>>>
>> >>>>>> Establish the Apache Airflow Project
>> >>>>>>
>> >>>>>> WHEREAS, the Board of Directors deems it to be in the best
>> >>>>>> interests of the Foundation and consistent with the
>> >>>>>> Foundation's purpose to establish a Project Management
>> >>>>>> Committee charged wit

[RESULT] Graduate Apache Airflow as a TLP

[Jakob Homan]

I neglected to add my binding +1, so I'll do so now.

With three days having elapsed, the VOTE is concluded successfully.

Overall: 20 x +1 votes, 0 x -1 votes

Binding +1 x 10: Kaxil, Tao, Bolke, Fokko, Maxime, Arthur, Hitesh,
Chris, Sid, Jakob.
Non-binding +1 x 10: Daniel, Shah, Stefan, Kevin, Marc, Sunil,
Adityan, Deng, Neelesh, Sai

I'll use this result to start the corresponding VOTE on the IPMC.  I'm
at an offsite today, so I have limited email time.  Likely will open
the VOTE this evening.

Thanks everyone.
-Jakob


On Tue, Dec 4, 2018 at 6:03 AM Bolke de Bruin  wrote:
>
> Shall we close the vote? @jakob?
>
> > On 2 Dec 2018, at 13:08, Sid Anand  wrote:
> >
> > +1 binding
> >
> > Woot! Thanks to all for this happy day!
> > -s
> >
> > On Sun, Dec 2, 2018 at 1:25 AM Sai Phanindhra  wrote:
> >
> >> +1 (non binding)
> >>
> >> Excited to see this happenning.
> >>
> >> On Sat 1 Dec, 2018, 20:35  >>
> >>> +1 (binding)!
> >>>
> >>> On 30 November 2018 21:33:14 GMT, Jakob Homan  wrote:
> >>>> Hey all!
> >>>>
> >>>> Following a very successful DISCUSS[1] regarding graduating Airflow to
> >>>> Top Level Project (TLP) status, I'm starting the official VOTE.
> >>>>
> >>>> Since entering the Incubator in 2016, the community has:
> >>>>  * successfully produced 7 releases
> >>>>  * added 9 new committers/PPMC members
> >>>> * built a diverse group of committers from multiple different employers
> >>>>  * had more than 3,300 JIRA tickets opened
> >>>>  * completed the project maturity model with positive responses[2]
> >>>>
> >>>> Accordingly, I believe we're ready to graduate and am calling a VOTE
> >>>> on the following graduation resolution.  This VOTE will remain open
> >>>> for at least 72 hours.  If successful, the resolution will be
> >>>> forwarded to the IPMC for its consideration.  If that VOTE is
> >>>> successful, the resolution will be voted upon by the Board at its next
> >>>> monthly meeting.
> >>>>
> >>>> Everyone is encouraged to vote, even if their vote is not binding.
> >>>> We've built a nice community here, let's make sure everyone has their
> >>>> voice heard.
> >>>>
> >>>> Thanks,
> >>>> Jakob
> >>>>
> >>>> [1]
> >>>>
> >>>
> >> https://lists.apache.org/thread.html/%3c0a763b0b-7d0d-4353-979a-ac6769eb0...@gmail.com%3E
> >>>> [2]
> >>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
> >>>>
> >>>> 
> >>>>
> >>>> Establish the Apache Airflow Project
> >>>>
> >>>> WHEREAS, the Board of Directors deems it to be in the best
> >>>> interests of the Foundation and consistent with the
> >>>> Foundation's purpose to establish a Project Management
> >>>> Committee charged with the creation and maintenance of
> >>>> open-source software, for distribution at no charge to
> >>>> the public, related to workflow automation and scheduling
> >>>> that can be used to author and manage data pipelines.
> >>>>
> >>>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> >>>> Committee (PMC), to be known as the "Apache Airflow Project",
> >>>> be and hereby is established pursuant to Bylaws of the
> >>>> Foundation; and be it further
> >>>>
> >>>> RESOLVED, that the Apache Airflow Project be and hereby is
> >>>> responsible for the creation and maintenance of software
> >>>> related to workflow automation and scheduling that can be
> >>>> used to author and manage data pipelines; and be it further
> >>>>
> >>>> RESOLVED, that the office of "Vice President, Apache Airflow" be
> >>>> and hereby is created, the person holding such office to
> >>>> serve at the direction of the Board of Directors as the chair
> >>>> of the Apache Airflow Project, and to have primary responsibility
> >>>> for management of the projects within the scope of
> >>>> responsibility of the Apache Airflow Project; and be it further
> >>>>
> >>>> RESOLVED, that the persons listed immediately below be and
&

Re: [DISCUSS] Apache Airflow graduation from the incubator

[Jakob Homan]

I've finished the paperwork.  I don't seem to have karma to trigger
the build on Jenkins, so we'll just wait for the daily rebuild.  With
that, I've opened the VOTE thread as well.  Thanks everybody.
On Wed, Nov 28, 2018 at 5:08 PM Jakob Homan  wrote:
>
> I'll finish up the template at
> http://incubator.apache.org/projects/airflow.html tomorrow or Friday
> (I *think* you have to be an IPMC member to update it since it lives
> in the Incubator SVN).  Looks like there's no actual work to do, just
> marking stuff that has been done but not yet recorded, and verifying
> some licenses.
>
> -Jakob
>
>
>
> On Wed, Nov 28, 2018 at 2:48 PM Tao Feng  wrote:
> >
> > Sorry, just saw Kaxil's latest email. Kaxil, is there anything else I could
> > help with?
> >
> > Thanks,
> > -Tao
> >
> > On Wed, Nov 28, 2018 at 2:40 PM Tao Feng  wrote:
> >
> > > I would like to help on the documentation. Let me take a look at it. I
> > > will work Kaxil on that.
> > >
> > > On Tue, Nov 27, 2018 at 12:39 PM Bolke de Bruin  wrote:
> > >
> > >> Hi Folks,
> > >>
> > >> Thanks all for your responses and particularly Stefan for his suggestion
> > >> to use the generic Apache way to handle security issues. This seems to be
> > >> an accepted way for more projects, so I have added this to the maturity
> > >> evaluation[1] and marked is as resolved. While handling the GPL library 
> > >> can
> > >> be nicer we are already in compliance with CD30, so @Fokko and @Ash if 
> > >> you
> > >> want to help out towards graduation please spend your time elsewhere like
> > >> fixing CO50. This means adding a page to confluence that describes how to
> > >> become a committer on the project. As we are following Apache many 
> > >> examples
> > >> of other projects are around[2]
> > >>
> > >> Then there is the paperwork[3] as referred to by Jakob. This mainly
> > >> concerns filling in some items, maybe here and there creation some
> > >> documentation but I don't think much. @Kaxil, @Tao: are you willing to 
> > >> pick
> > >> this up? @Sid can you share how to edit that page?
> > >>
> > >> If we have resolved these items in my opinion we can start the voting
> > >> here and at the IPMC thereafter, targeting the board meeting of January 
> > >> for
> > >> graduation. How’s that for a New Year’s resolution?
> > >>
> > >> Cheers!
> > >> Bolke
> > >>
> > >> P.S. Would it be nice to have updated graduation web page? Maybe one of
> > >> the contributors/community members likes to take a stab at this[4]
> > >>
> > >> [1]
> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation <
> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation>
> > >> [2] https://cwiki.apache.org/confluence/display/HAWQ/Becoming+a+committer
> > >> <https://cwiki.apache.org/confluence/display/HAWQ/Becoming+a+committer>
> > >> [3] http://incubator.apache.org/projects/airflow.html <
> > >> http://incubator.apache.org/projects/airflow.html>
> > >> [4] https://airflow.apache.org/ <https://airflow.apache.org/>
> > >>
> > >>
> > >>
> > >> > On 27 Nov 2018, at 16:32, Driesprong, Fokko 
> > >> wrote:
> > >> >
> > >> > +1 from my side. Would be awesome to graduate Airflow
> > >> >
> > >> > If time allows, I'll also dive into CD30.
> > >> >
> > >> > Cheers, Fokko
> > >> >
> > >> > Op di 27 nov. 2018 om 16:21 schreef Ash Berlin-Taylor 
> > >> > :
> > >> >
> > >> >> Oarsome Bolke, thanks for starting this.
> > >> >>
> > >> >> It looks like we are closer than I thought!
> > >> >>
> > >> >> We can use those security lists (though having our own would be nice) 
> > >> >> -
> > >> >> either way we will need to make this prominent in the docs.
> > >> >>
> > >> >> Couple of points
> > >> >>
> > >> >> CS10: that github link is only visible to members of the team
> > >> >>
> > >> >> CD30: probably good as it is, we may want to do
> > >> >> https://issues.apache.org/jira/br

[VOTE] Graduate the Apache Airflow as a TLP

[Jakob Homan]

Hey all!

Following a very successful DISCUSS[1] regarding graduating Airflow to
Top Level Project (TLP) status, I'm starting the official VOTE.

Since entering the Incubator in 2016, the community has:
   * successfully produced 7 releases
   * added 9 new committers/PPMC members
   * built a diverse group of committers from multiple different employers
   * had more than 3,300 JIRA tickets opened
   * completed the project maturity model with positive responses[2]

Accordingly, I believe we're ready to graduate and am calling a VOTE
on the following graduation resolution.  This VOTE will remain open
for at least 72 hours.  If successful, the resolution will be
forwarded to the IPMC for its consideration.  If that VOTE is
successful, the resolution will be voted upon by the Board at its next
monthly meeting.

Everyone is encouraged to vote, even if their vote is not binding.
We've built a nice community here, let's make sure everyone has their
voice heard.

Thanks,
Jakob

[1] 
https://lists.apache.org/thread.html/%3c0a763b0b-7d0d-4353-979a-ac6769eb0...@gmail.com%3E
[2] https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation



Establish the Apache Airflow Project

WHEREAS, the Board of Directors deems it to be in the best
interests of the Foundation and consistent with the
Foundation's purpose to establish a Project Management
Committee charged with the creation and maintenance of
open-source software, for distribution at no charge to
the public, related to workflow automation and scheduling
that can be used to author and manage data pipelines.

NOW, THEREFORE, BE IT RESOLVED, that a Project Management
Committee (PMC), to be known as the "Apache Airflow Project",
be and hereby is established pursuant to Bylaws of the
Foundation; and be it further

RESOLVED, that the Apache Airflow Project be and hereby is
responsible for the creation and maintenance of software
related to workflow automation and scheduling that can be
used to author and manage data pipelines; and be it further

RESOLVED, that the office of "Vice President, Apache Airflow" be
and hereby is created, the person holding such office to
serve at the direction of the Board of Directors as the chair
of the Apache Airflow Project, and to have primary responsibility
for management of the projects within the scope of
responsibility of the Apache Airflow Project; and be it further

RESOLVED, that the persons listed immediately below be and
hereby are appointed to serve as the initial members of the
Apache Airflow Project:

* Alex Guziel 
* Alex Van Boxel 
* Arthur Wiedmer 
* Ash Berlin-Taylor 
* Bolke de Bruin 
* Chris Riccomini 
* Dan Davydov 
* Fokko Driesprong 
* Hitesh Shah 
* Jakob Homan 
* Jeremiah Lowin 
* Joy Gao 
* Kaxil Naik 
* Maxime Beauchemin 
* Siddharth Anand 
* Sumit Maheshwari 
* Tao Feng 

NOW, THEREFORE, BE IT FURTHER RESOLVED, that Bolke de Bruin
be appointed to the office of Vice President, Apache Airflow, to
serve in accordance with and subject to the direction of the
Board of Directors and the Bylaws of the Foundation until
death, resignation, retirement, removal or disqualification,
or until a successor is appointed; and be it further

RESOLVED, that the initial Apache Airflow PMC be and hereby is
tasked with the creation of a set of bylaws intended to
encourage open development and increased participation in the
Apache Airflow Project; and be it further

RESOLVED, that the Apache Airflow Project be and hereby
is tasked with the migration and rationalization of the Apache
Incubator Airflow podling; and be it further

RESOLVED, that all responsibilities pertaining to the Apache
Incubator Airflow podling encumbered upon the Apache Incubator
Project are hereafter discharged.

Re: [DISCUSS] Apache Airflow graduation from the incubator

[Jakob Homan]

I'll finish up the template at
http://incubator.apache.org/projects/airflow.html tomorrow or Friday
(I *think* you have to be an IPMC member to update it since it lives
in the Incubator SVN).  Looks like there's no actual work to do, just
marking stuff that has been done but not yet recorded, and verifying
some licenses.

-Jakob



On Wed, Nov 28, 2018 at 2:48 PM Tao Feng  wrote:
>
> Sorry, just saw Kaxil's latest email. Kaxil, is there anything else I could
> help with?
>
> Thanks,
> -Tao
>
> On Wed, Nov 28, 2018 at 2:40 PM Tao Feng  wrote:
>
> > I would like to help on the documentation. Let me take a look at it. I
> > will work Kaxil on that.
> >
> > On Tue, Nov 27, 2018 at 12:39 PM Bolke de Bruin  wrote:
> >
> >> Hi Folks,
> >>
> >> Thanks all for your responses and particularly Stefan for his suggestion
> >> to use the generic Apache way to handle security issues. This seems to be
> >> an accepted way for more projects, so I have added this to the maturity
> >> evaluation[1] and marked is as resolved. While handling the GPL library can
> >> be nicer we are already in compliance with CD30, so @Fokko and @Ash if you
> >> want to help out towards graduation please spend your time elsewhere like
> >> fixing CO50. This means adding a page to confluence that describes how to
> >> become a committer on the project. As we are following Apache many examples
> >> of other projects are around[2]
> >>
> >> Then there is the paperwork[3] as referred to by Jakob. This mainly
> >> concerns filling in some items, maybe here and there creation some
> >> documentation but I don't think much. @Kaxil, @Tao: are you willing to pick
> >> this up? @Sid can you share how to edit that page?
> >>
> >> If we have resolved these items in my opinion we can start the voting
> >> here and at the IPMC thereafter, targeting the board meeting of January for
> >> graduation. How’s that for a New Year’s resolution?
> >>
> >> Cheers!
> >> Bolke
> >>
> >> P.S. Would it be nice to have updated graduation web page? Maybe one of
> >> the contributors/community members likes to take a stab at this[4]
> >>
> >> [1]
> >> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation <
> >> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation>
> >> [2] https://cwiki.apache.org/confluence/display/HAWQ/Becoming+a+committer
> >> 
> >> [3] http://incubator.apache.org/projects/airflow.html <
> >> http://incubator.apache.org/projects/airflow.html>
> >> [4] https://airflow.apache.org/ 
> >>
> >>
> >>
> >> > On 27 Nov 2018, at 16:32, Driesprong, Fokko 
> >> wrote:
> >> >
> >> > +1 from my side. Would be awesome to graduate Airflow
> >> >
> >> > If time allows, I'll also dive into CD30.
> >> >
> >> > Cheers, Fokko
> >> >
> >> > Op di 27 nov. 2018 om 16:21 schreef Ash Berlin-Taylor :
> >> >
> >> >> Oarsome Bolke, thanks for starting this.
> >> >>
> >> >> It looks like we are closer than I thought!
> >> >>
> >> >> We can use those security lists (though having our own would be nice) -
> >> >> either way we will need to make this prominent in the docs.
> >> >>
> >> >> Couple of points
> >> >>
> >> >> CS10: that github link is only visible to members of the team
> >> >>
> >> >> CD30: probably good as it is, we may want to do
> >> >> https://issues.apache.org/jira/browse/AIRFLOW-3400 <
> >> >> https://issues.apache.org/jira/browse/AIRFLOW-3400> to remove the last
> >> >> niggle of the GPL env var at install time (but not a hard requirement,
> >> just
> >> >> nice)
> >> >>
> >> >> -ash
> >> >>
> >> >>> On 26 Nov 2018, at 21:10, Stefan Seelmann 
> >> >> wrote:
> >> >>>
> >> >>> I agree that Apache Airflow should graduate.
> >> >>>
> >> >>> I'm only involved since beginning of this year, but the project did
> >> two
> >> >>> releases during that time, once TLP releasing becomes easier :)
> >> >>>
> >> >>> Regarding QU30 you may consider to use the ASF wide security mailing
> >> >>> list [3] and process [4].
> >> >>>
> >> >>> Kind Regards,
> >> >>> Stefan
> >> >>>
> >> >>> [3] https://www.apache.org/security/
> >> >>> [4] https://www.apache.org/security/committers.html
> >> >>>
> >> >>>
> >> >>> On 11/26/18 8:46 PM, Bolke de Bruin wrote:
> >>  Ping!
> >> 
> >>  Sent from my iPhone
> >> 
> >> > On 24 Nov 2018, at 12:57, Bolke de Bruin  wrote:
> >> >
> >> > Hi All,
> >> >
> >> > With the Apache Airflow community healthy and growing, I think now
> >> >> would be a good time to
> >> > discuss where we stand regarding to graduation from the Incubator,
> >> and
> >> >> what requirements remains.
> >> >
> >> > Apache Airflow entered incubation around 2 years ago, since then,
> >> the
> >> >> Airflow community learned
> >> > a lot about how to do things in Apache ways. Now we are a very
> >> helpful
> >> >> and engaged community,
> >> > ready to help on all questions from the Airflow 

Re: [DISCUSS] Apache Airflow graduation from the incubator

[Jakob Homan]

With my Mentor hat on, I'm entirely confident that Airflow is ready to
graduate.  The community broadly gets the Apache Way and operates
within it.  The community is healthy and engaged.  The last couple
releases went well, with no hitches whatsoever for the last one.

The graduation process is mainly paperwork[1] and running VOTEs [2]
here and on the IPMC.  Last time I suggested this had to be done by
the Champion, which wasn't correct - anyone from the PPMC can do so.
I may have some free cycles over the next few weeks, so I'll take a
look at the check list to see what we can get out of the way, but any
of the PPMCers can also take items.  The IPMC also likes it if
Podlings go through and grade themselves on the Maturity Model[2], if
someone wants to do that.

-Jakob

[1] http://incubator.apache.org/projects/airflow.html
[2] 
http://mail-archives.apache.org/mod_mbox/airflow-dev/201809.mbox/%3CCAMdzn8vNbKQr2FF8WcJydFj16q0bgn0B42jfu5h28RS-ZQ=w...@mail.gmail.com%3E
[3] https://community.apache.org/apache-way/apache-project-maturity-model.html
On Mon, Nov 26, 2018 at 1:10 PM Stefan Seelmann  wrote:
>
> I agree that Apache Airflow should graduate.
>
> I'm only involved since beginning of this year, but the project did two
> releases during that time, once TLP releasing becomes easier :)
>
> Regarding QU30 you may consider to use the ASF wide security mailing
> list [3] and process [4].
>
> Kind Regards,
> Stefan
>
> [3] https://www.apache.org/security/
> [4] https://www.apache.org/security/committers.html
>
>
> On 11/26/18 8:46 PM, Bolke de Bruin wrote:
> > Ping!
> >
> > Sent from my iPhone
> >
> >> On 24 Nov 2018, at 12:57, Bolke de Bruin  wrote:
> >>
> >> Hi All,
> >>
> >> With the Apache Airflow community healthy and growing, I think now would 
> >> be a good time to
> >> discuss where we stand regarding to graduation from the Incubator, and 
> >> what requirements remains.
> >>
> >> Apache Airflow entered incubation around 2 years ago, since then, the 
> >> Airflow community learned
> >> a lot about how to do things in Apache ways. Now we are a very helpful and 
> >> engaged community,
> >> ready to help on all questions from the Airflow community. We delivered 
> >> multiple releases that have
> >> been increasing in quality ever since, now we can do self-driving releases 
> >> in good cadence.
> >>
> >> The community is growing, new committers and PPMC members keep joining. We 
> >> addressed almost all
> >> the maturity issues stipulated by Apache Project Maturity Model [1]. So 
> >> final requirements remain, but
> >> those just need a final nudge. Committers and contributors are invited to 
> >> verify the list and pick up the last
> >> bits (QU30, CO50). Finally (yahoo!) all the License and IP issues we can 
> >> see got resolved.
> >>
> >> Base on those, I believes it's time for us to graduate to TLP. [2] Any 
> >> thoughts?
> >> And welcome advice from Airflow Mentors?
> >>
> >> Thanks,
> >>
> >> [1] https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
> >> [2] 
> >> https://incubator.apache.org/guides/graduation.html#graduating_to_a_top_level_project
> >>  Regards,
>

Re: Fundamental change - Separate DAG name and id.

[Jakob Homan]

Sounds like this may be a good candidate for an Airflow Improvement Plan...

On 20 September 2018 at 22:45, Brian Greene
 wrote:
> Prior to using airflow for much, on first inspection, I think I may have 
> agreed with you.
>
> After a bit of use I’d agree with Fokko and others - this isn’t really a 
> problem, and separating them seems to do more harm than good related to 
> deployment.
>
> I was gonna stop there, but why?
>
> You can add a task to a dag that’s deployed and has run and still view 
> history.  The “new” task shows up white Squares in the old dags.  nobody said 
> you’re required to also rename the dag when you do so this.  If your process 
> or desire or design determines you need to rename it, well then by 
> definition... isn’t it a new thing without a history?  Airflow is 
> implementing exactly that.
>
> One could argue that renaming to reflect exact purpose is good practice.  
> Yes, I’d agree, but again following that logic if it’s a small enough change 
> to “slip in” then the name likely shouldn’t change.  If it’s big enough I 
> want to change the name then it’s a big enough change that I’m functionally 
> running something “new”, and I expect to need to account for that.  Airflow 
> is enforcing that logic by coupling the name to the deployment of what you 
> said was a new process.
>
> One might put forth that changing the name to be more descriptive In the ui 
> makes it easier for support staff.  I think perhaps if that’s your challenge 
> it’s not airflow that’s a problem.  Dags are of course documented elsewhere 
> besides their name, right?  Yeah it’s self documenting (and the graphs are 
> cool), but I have to assume there’s something besides the NAME to tell people 
> what it does.  Additionally, far more than the name is required for even an 
> operator or monitor watcher to take action - you don’t expect them to know 
> which tasks to rerun or how to troubleshoot failures just based on your “now 
> most descriptive name in the UI” do you?
>
> I spent time In an informatica shop where all the jobs were numbered.  
> Numbered.  Let’s be more exact... their NAMES were NUMBERS like 56709. 
> Terrible, but 100% worked, because while a descriptive name would have been 
> useful, the name is the thing that’s supposed to NOT CHANGE (see code of 
> Abibarshim), and all the other information can attach to that in places where 
> you write... other information.  People would curse a number “F’ing 6291 
> failed again” - everyone knew what they were talking about.. I digress.
>
>  You might decide to document “dag ID 12” or just “12” on your wiki - I’m  
> going to document “daily_sales_import”.  And when things start failing at 3am 
> it’s not my dag “56” that’s failing, it’s the sales_export dag.  But if you 
> document “12”, that’s still it’s name, and it’d better be 12 in all your 
> environments and documents.  This also means the actual db IDs from your 
> proposal are almost certainly NOT the same across your environments, making 
> the 12 unchangeable name!
>
> There are lots of languages (most of them) where the name of a thing is 
> important and hard to change.  It’s not a bad thing, and I’d assume that 
> deploying a thing by name has some significance in many systems.  Go rename a 
> class in... pick a language... tell me how that should be easier to do 
> willy-nilly so it’s easier In the UI.
>
> I suppose you could view it as a limitation, But i don’t think you’ve 
> illuminated a single use case where it’s an actual technical constraint or 
> limitation.
>
> The BEST argument against the current implementation is db performance.  It’s 
> a hogwash argument.  Basic key indexes on low cardinality string columns are 
> plenty fast for the airflow workload, and if your task load is so high 
> airflow can’t keep up or your seeing super-fast tasks and airflow db/tracking 
> latency is too much... perhaps a messaging or queue processing solution is 
> better suited to those workloads.  We see scheduler bottlenecks long before 
> the database for our “quick task” scenarios.  Additionally, reading through 
> this list you’ll find people running airflow at substantial scale - I’ve not 
> seen anyone complaining of production performance issues based on this design 
> decision.   At first I hated it.  String keys are dirty, we’re all taught 
> that as good little programmers.  Except when performance won’t be a huge 
> consideration since it’s not OLTP and easy of queryabilty is more important 
> because it’s a growing system... good decision - whoever made it.
>
> How does filename matter?  Frankly I wish the filename was REQUIRED to be the 
> dag name so people would quit confusing themselves by mismatching them !   
> We’ve renamed dag files with no issue as long as the content doesn’t change, 
> so again, not a real use case.  And really - name your stuff careful before 
> you get to prod man.
>
> I gotta ask - airflowuser - are you gonna use airflow for anything, or just 

Re: Airflow: Apache Graduation

[Jakob Homan]

The big list that *has* to be filled out and correct is here:
http://incubator.apache.org/projects/airflow.html

This is usually done by the Champion, Chris Riccomini.  Chris  - are
you still interested to do this?

After that and the maturity questionnaire are in good shape, it's:
1) Graduation DISCUSS by the community.  This thread can cover that,
although an explicit DISCUSS would help too.  Example:
https://mail-archives.apache.org/mod_mbox/samza-dev/201412.mbox/%3CCADiKvVtBOPaji22Nd3927R6Tj%2BmWpxnHH%2Bk%3DyXCQEMGf_CnAqw%40mail.gmail.com%3E
2) Graduation VOTE by the community on a resolution for the new top
level project (TLP). Example:
https://mail-archives.apache.org/mod_mbox/samza-dev/201412.mbox/%3CCADiKvVt_LK9%2BBXC8eR_gb3ALOU2KFJqDDyZOLao7rm0XjuZmcg%40mail.gmail.com%3E
- most of the text is boiler plate, but be sure not leave anything in
there that shouldn't be (Jakob said with experience...)
3) Exact same text is VOTEd on by the IPMC
4) Champion adds the resolution to the Board's next monthly agenda,
where the Board approves it.  Project is officially TLP once that is
done.
5) Champion cleans up incubator resources, creates new TLP resources.
Example: https://jira.apache.org/jira/browse/INFRA-10639

-Jakob


On 21 September 2018 at 13:49, Bolke de Bruin  wrote:
> There is a maturity check that we can do. Don't have the link ready now.
> But can look it up later.
>
> To me two things need to be more mature
>
> 1. Handling of security issues
> 2. Licensing.
>  2a having ppl aware that with new dependencies updates to dependencies the
> respective licenses need to be updated. Too often I need to fix some issues
> at release time.
>  2b versions need to be added to the licenses (version of the software)
>
> B.
>
> Op vr 21 sep. 2018 13:00 schreef Ash Berlin-Taylor :
>
>> Your guess is as good as mine on what is involved in graduation. I think
>> that we sorted the Licensing issues in 1.10.0 (even if the way we sorted it
>> was a little annoying - having to specify the environment variable at `pip
>> install` time is a littlle bit un-pythonic, but I wasn't thinking of fixing
>> that in 1.10.1)
>>
>> Some of the steps and requirements are listed here
>> graduating_to_a_top_level_project <
>> https://incubator.apache.org/guides/graduation.html>, but in summary from
>> a quick read of it I think the process is:
>>
>> - Collect some information, put it on
>> http://incubator.apache.org/projects/airflow.html <
>> http://incubator.apache.org/projects/airflow.html> for the IPMC
>> - We as a community hold a vote on if we think we're ready to graduate
>> - IPMC vote on it too
>> - propose motion for (monthly) Apache board meeting
>>
>> There might be a few more steps involves, such as drafting a Charter (we
>> would probably start with a "stock" Apache one)
>>
>> -ash
>>
>> > On 20 Sep 2018, at 18:22, Maxime Beauchemin 
>> wrote:
>> >
>> > Yeah let's make it happen! I'm happy to set some time aside to help with
>> > the final push.
>> >
>> > Max
>> >
>> > On Thu, Sep 20, 2018 at 9:53 AM Sid Anand  wrote:
>> >
>> >> Folks! (specifically Bolke, Fokko, Ash)
>> >> What's needed to graduate from Apache?
>> >>
>> >> Can we make 1.10.1 be about meeting our licensing needs to allow us to
>> >> graduate?
>> >>
>> >> -s
>> >>
>>
>>

Re: Guidelines on Contrib vs Non-contrib

[Jakob Homan]

> My understanding as a contributor is that if a hook/operator is in core, it
> means that a committer is willing to take personal responsibility to
> maintain it (or at least help maintain it), and everything else goes in
> contrib.

That's not correct.  All of the code is owned by the entire
community[1]; no one person is responsible for any of it.  There's no
silos, fiefdoms, walled gardens, etc.  If the community cannot support
a piece of code it should be deprecated and subsequently removed.

Contrib sections are almost always problematic for this reason.
Hadoop ended up abandoning its.  Because Airflow acts as a gathering
point for so many disparate technologies (databases, storage systems,
compute engines, etc.), trying to keep all of them corralled and up to
date will be very difficult.  Individual operators and hooks living in
separate repositories on github (or possibly other Apache projects),
which are then distributed by pip and installed as libraries seems
like it would scale better.

-Jakob

[1] https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie

On 15 September 2018 at 13:29, Jeff Payne  wrote:
> How many operators are added to contrib per month? Is it too many to make the 
> decision case by case? If so, then the above mentioned rule sounds fairly 
> reasonable. However, if that's the rule, shouldn't a bunch of existing 
> modules be moved from contrib to core?
>
> Get Outlook for Android
>
> 
> From: Taylor Edmiston 
> Sent: Saturday, September 15, 2018 1:13:47 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: Guidelines on Contrib vs Non-contrib
>
> My understanding as a contributor is that if a hook/operator is in core, it
> means that a committer is willing to take personal responsibility to
> maintain it (or at least help maintain it), and everything else goes in
> contrib.
>
> *Taylor Edmiston*
> Blog  | LinkedIn
>  | Stack Overflow
>  | Developer Story
> 
>
>
>
> On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik  wrote:
>
>> Hi, all (mainly contributors),
>>
>> Can we decide on a common guideline on when a hook/operator should go under
>> contrib vs core?
>>
>> Regards,
>>
>> *Kaxil Naik*
>> *Big Data Consultant *@ *Data Reply UK*
>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
>> Developer
>> *Phone: *+44 (0) 74820 88992
>> *LinkedIn*: https://www.linkedin.com/in/kaxil
>>

Re: [VOTE][RESULT] Replace with Gitter with Slack?

[Jakob Homan]

So, binding VOTEs are generally only called on releases, where PMC
members get those binding votes (and on electing new committers or PMC
members, and those votes happen in private on the private list).
Communities often call VOTEs on other things, like logos, questions of
sponsorship, large scale tech decisions, etc.  How those VOTEs are
executed (consesus [ie anyone with a binding vote can VETO] or
majority) are up to the community to decide.

The community can codify those decisions with bylaws (Here's Hadoop's:
https://hadoop.apache.org/bylaws.html).  However, extensive bylaws are
considered an anti-pattern - they can be indicative of a fractured or
unwieldy community.  This is why Incubator projects are not generally
recommended to go down this route.  Also, incubator podlings are not
separate top-level projects, and so would be governed by the Incubator
project bylaws, which as far as I can tell, don't exist.

Also, keep in mind that the bylaws of any particular project are valid
at the pleasure of the Board.  If the Board deems any part of them
invalid, which it has done even very recently, it can come in and
require the project to change them immediately.

As for the current discussion, the large number of <1 votes, is a bit
concerning.  This is indicative of lack of consensus and that further
discussion is probably warranted.  Whether or not it's an official
result is up for debate.

However, one thing worth noting is that there's no 'official' gitter
or slack channel for any ASF project.  The only official communication
medium for ASF is mailing lists.  The edict 'If it didn't happen on
the mailing list, it didn't happen' is one of the axioms of the Apache
Way [1][2].  There's also no official IRC channel or official Stack
Overflow tag or official in-person meetup.  Every member of the
community is welcome to participate in whatever forum they want, but
the only place things actually happen is on the list.  Any non-PMC
discussions can happen anywhere, but if a decision needs to occur as
part of that discussion, the discussion needs to be re-homed onto the
mailing list and made there.  Part of the PMC's job is to enforce and
model this behavior.

Speaking entirely for myself, I'm not comfortable with the invitation
model of Slack discussions.  For most of my time at Apache, irc was
the way to go for most projects.  Some had very active irc channels
and some were nearly dead.  (The #infra irc channel is a godsend
though.)

And again, this type of meta discussion is awesome.  It shows the
project is maturing and thinking through some of the edge cases of
what it means to an Apache project.

-Jakob



[1] 
https://community.apache.org/newbiefaq.html#NewbieFAQ-IsthereaCodeofConductforApacheprojects?
[2] 
https://blogs.apache.org/foundation/entry/success-at-apache-asynchronous-decision

On 8 September 2018 at 23:14, Sid Anand  wrote:
> Taking Binding votes into account :
>
> +1: 1 vote
>
>- Sid Anand
>
> 0: 2 votes
>
>- Bolke de Bruin
>- Kaxil Naik
>
> -0.5: 1 vote
>
>- Arthur Wiedmer
>
>
> Vote result is a net positive of +0.5.
>
> I counted all of the PMC/committers' votes as binding.
>
> -s
>
> On Sat, Sep 8, 2018 at 11:07 PM Arthur Wiedmer 
> wrote:
>
>> Sid,
>>
>> Erm, the next line is (emphasis mine) :
>>
>> PMC members have formally binding votes, but in general community members
>> are encouraged to vote, even if their votes are *only advisory*.
>>
>> Again, that's not to say that the community at large cannot decide to host
>> a Slack channel. I think you are more than welcome to if you want to.
>>
>> Best,
>> Arthur
>>
>> On Sat, Sep 8, 2018 at 10:59 PM Sid Anand  wrote:
>>
>> > Why doesn't every vote matter for this topic?
>> > https://www.apache.org/foundation/voting.html#binding-votes
>> >
>> > Am I misinterpreting the "Who is permitted to vote is, to some extent, a
>> > community-specific thing."?
>> >
>> > For the established Apache processes around promoting a contributor to
>> > committer/PMC, deciding on the number of +1s to allow a merge to master,
>> or
>> > voting on releases, I understand the need to follow the
>> binding/non-binding
>> > protocol. In all of those cases, the outcomes affect maintainers.
>> >
>> > My understanding is that this topic affects the entire community, where
>> > some members of the community are helping others. This seems to chug
>> along
>> > without maintainers being present. Hence, why do maintainers' votes
>> matter
>> > more here?
>> >
>> > Question for the mentors.. Jakob?
>> >
>> > -s
>> >
>> > On Sat, Sep 8, 2018 at 10:27 PM Bolke de Bruin 
>> wrote:
>> >
>> > > Committers can only vote binding. Most of the time it is tagged by,
>> > > indeed, adding "binding" to the vote.
>> > >
>> > > Sent from my iPhone
>> > >
>> > > > On 9 Sep 2018, at 07:20, Scott Halgrim > > .INVALID>
>> > > wrote:
>> > > >
>> > > > What makes a vote binding? Just putting “(binding)” after your vote?
>> > > >> On Sep 8, 2018, 10:19 PM -0700, Bolke de Bruin 

Seattle Apache Airflow Meetup ready for members; planning its first get together

[Jakob Homan]

Hey all-
   For those in the Puget Sound area, I'm happy to announce we've
formed the Seattle Apache Airflow Meetup and hope to schedule our
first get together for mid October.  Please reach out if you're
interested in submitting a talk.

https://www.meetup.com/Seattle-Apache-Airflow-Users-Group/

I imagine these meetups will be quarterly at first.  We'll likely host
the first one at OfferUp in Bellevue, but would love to alternate
between the Eastside and Seattle.  Also reach out If you're interested
hosting on either side of Lake Washington.

Thanks,
Jakob

Re: Apache Spark Interfering with Airflow Jira/PRs ??

[Jakob Homan]

Open a ticket against infra@. Either a misconfiguration or spam. 

> On Sep 2, 2018, at 12:48 PM, Kaxil Naik  wrote:
> 
> I am getting that too. Tons of emails as well about the same. Not sure the
> reason.
> 
>> On Sun, 2 Sep 2018, 20:45 Sid Anand,  wrote:
>> 
>> Fellow committers... what happened around 11a PT today? I see a flood of
>> updates to our Jiras from "Apache Spark".
>> 
>> Some examples:
>> 
>>   1.
>> 
>> https://issues.apache.org/jira/browse/AIRFLOW-2408?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=16601616#comment-16601616
>>   2. It also updated https://issues.apache.org/jira/browse/AIRFLOW-2553
>> as
>>   resolved and closed
>> https://github.com/apache/incubator-airflow/pull/3451
>>   PR
>> 
>> What's going on?
>> -s
>> 

Re: PR Review Dashboard?

[Jakob Homan]

Spark is an order of magnitude larger in terms of contributors,
codebase, issues, etc.  I don't think adding an layer of complexity
into the review process would be helpful right now.  Not everything
that works for one ASF project works for another.

I'd vote against this.

-Jakob

On 28 August 2018 at 10:39, Bolke de Bruin  wrote:
> I’m not in favor of moving to GitHub issues. While JIRA is not perfect it 
> actually moves discussion to the mailing list. With GitHub issues the stuff 
> just gets lost imho.
>
> I do like our changelogs a lot better now.
>
> B.
>
> Verstuurd vanaf mijn iPad
>
>> Op 27 aug. 2018 om 05:53 heeft Holden Karau  het 
>> volgende geschreven:
>>
>> Awesome, so I'll give this a shot with JIRA for now and if we end up moving 
>> to GH we can move it over. I've got a fork of the dashboard I've been using 
>> in Apache Beam as well as the Spark one so it shouldn't take me too long to 
>> generalize it again.
>>
>>> On Sun, Aug 26, 2018 at 8:50 PM Maxime Beauchemin 
>>>  wrote:
>>> I love the idea. I took a quick look at /databricks/spark-pr-dashboard
>>>  and that looks like a
>>> nice easy start. It would be interesting to try to make a generic project
>>> out of it that would work on top of any project/repo. Basically just
>>> refactor all of the Spark-specific code and configurations into a
>>> `config.py` (assuming there's not too much frontend Spark-specific
>>> code...).
>>>
>>> Of course this tool assumes Jira+Github which works for Airflow, but
>>> probably isn't as common of a setup to really generalize beyond Apache. It
>>> seems like by embracing Github issues and dropping Jira we could be
>>> building something much more relevant.
>>>
>>> Max
>>>
>>> On Fri, Aug 24, 2018 at 7:27 PM Holden Karau  wrote:
>>>
>>> > Update: we can do this with the dashboard code. Since this would modify
>>> > the JIRAs I’d love a sign-off on turning that feature on from someone on
>>> > the PMC (or at least a week with no PMC folks saying no).
>>> >
>>> > On Thu, Aug 23, 2018 at 8:00 AM Holden Karau  wrote:
>>> >
>>> >> I mean a few ASF projects update JIRA tickets based on PRs automatically.
>>> >> Switching from JIRA to GH issues (or back) is super painful, so I’d
>>> >> probably do more incremental improvements personally, I just don’t have 
>>> >> the
>>> >> time to do something like that.
>>> >>
>>> >> I’ll take a look at some the K8s tools (over in Beam they’re looking at
>>> >> one of the review tagging tools out of K8s) if the Spark one is too
>>> >> difficult to adapt to our use case.
>>> >>
>>> >> On Thu, Aug 23, 2018 at 2:34 AM Eamon Keane 
>>> >> wrote:
>>> >>
>>> >>> Kubernetes is a good place to look as they've invested a lot in github
>>> >>> bots
>>> >>> and label based workflows. E.g. the cherry-picking script and doc is
>>> >>> here:
>>> >>>
>>> >>>
>>> >>> https://github.com/kubernetes/kubernetes/blob/master/hack/cherry_pick_pull.sh
>>> >>>
>>> >>> https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md
>>> >>>
>>> >>> And more general overview here:
>>> >>>
>>> >>> https://github.com/kubernetes/community/tree/master/contributors/devel
>>> >>>
>>> >>> On Thu, Aug 23, 2018 at 5:11 AM Maxime Beauchemin <
>>> >>> maximebeauche...@gmail.com> wrote:
>>> >>>
>>> >>> > I've heard many times in the past about a GH/Jira syncing tool but
>>> >>> never
>>> >>> > seen it in action. Personally my vote is to move issues to GH and drop
>>> >>> > Jira. Though in the process this will break the release helper script
>>> >>> here:
>>> >>> >
>>> >>> https://github.com/apache/incubator-airflow/blob/master/dev/airflow-jira
>>> >>> >
>>> >>> > We'll be working on a Github label-driven release baking magic script
>>> >>> for
>>> >>> > Superset, maybe we could use the same tooling on both Airflow and
>>> >>> Superset.
>>> >>> > The idea is that the script would use labels like `target:apache-1.11`
>>> >>> on
>>> >>> > PRs to bake releases. The tool would take as input a base SHA and
>>> >>> release
>>> >>> > minor release number, and would craft a release branch, fetch and
>>> >>> > cherry-pick all the right commits in the right order based on labels,
>>> >>> > generate release tags (on minor versions) and output state into
>>> >>> > release-info files (listing the base, all cherries, all tags, ...). 
>>> >>> > The
>>> >>> > tricky part is resolving merge conflicts while auto-picking cherries,
>>> >>> but
>>> >>> > the script would guide the operator through it .
>>> >>> >
>>> >>> > Curious to hear about how other projects do it. I think it generally
>>> >>> > involves a lot of manual work. Let me know if you know of open source
>>> >>> > tooling to deal with release management.
>>> >>> >
>>> >>> > Max
>>> >>> >
>>> >>> > On Wed, Aug 22, 2018 at 6:25 PM Holden Karau 
>>> >>> > wrote:
>>> >>> >
>>> >>> > > Thanks for the reminder, forgot to ask at coffee but I'll ask.
>>> >>> > >
>>> >>> > > On Wed, Aug 22, 2018, 1:52 

Re: [VOTE] Airflow 1.10.0rc1

[Jakob Homan]

@Bolke - I didn't raise the concern, so I can't speak to whether or
not Sebb will be ok with that. He tends to be pretty fastidious on
this stuff and 'but some other TLP does it' hasn't gone over well
before (trust me... I've tried).  Totally up to you if you'd rather
discuss it as part of the IPMC vote or just fix it to avoid
discussion.

-jakob

On 13 July 2018 at 09:48, Ash Berlin-Taylor
 wrote:
> Cloud that be related to my ignorefile change? `airflow list_dags` still 
> shows the example dags - the output is the same for that command as on 
> v1-9-stable.
>
> Though I just noticed I'd left `self.log.info ()` in 
> there. That's going to be noisy. 
> https://github.com/apache/incubator-airflow/pull/3603 
> 
>
> -ash
>
>> On 13 Jul 2018, at 17:36, Bolke de Bruin  wrote:
>>
>> Example dags are not picked up. If you put a dag in the normal dag folder it 
>> works fine.
>>
>> Please create a jira for this @fokko. A pr would be appreciated.
>>
>> B.
>>
>> Sent from my iPhone
>>
>>> On 13 Jul 2018, at 15:46, Driesprong, Fokko  wrote:
>>>
>>> With the SequentialExecutor the webserver also acts as the scheduler
>>> (without parallelism)
>>>
>>> 2018-07-13 15:43 GMT+02:00 Carl Johan Gustavsson :
>>>
>

Re: [VOTE] Airflow 1.10.0rc1

[Jakob Homan]

+1 (binding)

* Sigs look good
* Artifact has incubating in name
* LICENSE/NOTICE/DISCLAIMER look good
   - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
other projects.
* Nit: Last release Sebb called out "Copyright 2016 and onwards" in
NOTICE as being imprecise.  This language remains.
* Spot check on license headers looks ok.
* Nit: Last release there was a request for instructions on how to
build the code.  This isn't included.

-Jakob

On 11 July 2018 at 19:50, Sid Anand  wrote:
> FYI!
> I just installed the release candidate. The first thing I noticed is a
> missing tool tip for the Null State in the Recent Tasks column on the
> landing page. Since the null globe is new to this UI, users will likely
> hover over it to inquire what it means... and will be left wanting. Of
> course, they could click on the globe, which will take them to
> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator_state_equals=null,
> which will always show an empty list, leaving them a bit more confused.
>
> -s
>
> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>  wrote:
>
>> Hi Bolke,
>>
>> (Switching email to avoid moderation on my emails.)
>>
>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>> utf-8.
>>
>> I think it is a proper test though, it is a minimal reproducible version of
>> the code that fails. And the only difference in behaviour is at 3.7 which
>> we don’t support anyway so I’m fairly sure it is broken for all supported
>> Python 3 versions.
>>
>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>> unset and I see the same failure.
>>
>> I don’t think this is a big thing though and we could release it without
>> the fix I made. I think most people run it with a sane LC_ALL, but
>> apparently we didn’t.
>> Here’s the log for the test:
>>
>> > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>> root@b99b297df111:/# locale
>> LANG=C.UTF-8
>> LANGUAGE=
>> LC_CTYPE="C.UTF-8"
>> LC_NUMERIC="C.UTF-8"
>> LC_TIME="C.UTF-8"
>> LC_COLLATE="C.UTF-8"
>> LC_MONETARY="C.UTF-8"
>> LC_MESSAGES="C.UTF-8"
>> LC_PAPER="C.UTF-8"
>> LC_NAME="C.UTF-8"
>> LC_ADDRESS="C.UTF-8"
>> LC_TELEPHONE="C.UTF-8"
>> LC_MEASUREMENT="C.UTF-8"
>> LC_IDENTIFICATION="C.UTF-8"
>> LC_ALL=
>> > unset LANG
>> root@b99b297df111:/# locale
>> LANG=
>> LANGUAGE=
>> LC_CTYPE="POSIX"
>> LC_NUMERIC="POSIX"
>> LC_TIME="POSIX"
>> LC_COLLATE="POSIX"
>> LC_MONETARY="POSIX"
>> LC_MESSAGES="POSIX"
>> LC_PAPER="POSIX"
>> LC_NAME="POSIX"
>> LC_ADDRESS="POSIX"
>> LC_TELEPHONE="POSIX"
>> LC_MEASUREMENT="POSIX"
>> LC_IDENTIFICATION="POSIX"
>> LC_ALL=
>> root@b99b297df111:/# pip install -e .[devel]
>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>> + export AIRFLOW_HOME=/root/airflow
>> + AIRFLOW_HOME=/root/airflow
>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>> + export AIRFLOW_USE_NEW_IMPORTS=1
>> + AIRFLOW_USE_NEW_IMPORTS=1
>> +++ dirname ./run_unit_tests.sh
>> ++ cd .
>> ++ pwd
>> + DIR=/airflow
>> + export PYTHONPATH=:/airflow/tests/test_utils
>> + PYTHONPATH=:/airflow/tests/test_utils
>> + nose_args=
>> + which airflow
>> + echo 'Initializing the DB'
>> Initializing the DB
>> + airflow resetdb
>> + yes
>> Traceback (most recent call last):
>>   File "/usr/local/bin/airflow", line 6, in 
>> exec(compile(open(__file__).read(), __file__, 'exec'))
>>   File "/airflow/airflow/bin/airflow", line 21, in 
>> from airflow import configuration
>>   File "/airflow/airflow/__init__.py", line 35, in 
>> from airflow import configuration as conf
>>   File "/airflow/airflow/configuration.py", line 106, in 
>> DEFAULT_CONFIG = f.read()
>>   File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>> return codecs.ascii_decode(input, self.errors)[0]
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
>> ordinal not in range(128)
>> + '[' '' ']'
>> + '[' -z '' ']'
>> + nose_args='--with-coverage --cover-erase --cover-html
>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>   --with-ignore-docstrings --rednose --with-timer -s -v
>> --logging-level=DEBUG '
>> + echo 'Starting the unit tests with the following nose arguments:
>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>> --rednose --with-timer -s -v --logging-level=DEBUG
>> Starting the unit tests with the following nose arguments: --with-coverage
>> --cover-erase --cover-html --cover-package=airflow
>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>> --rednose --with-timer -s -v --logging-level=DEBUG
>> + nosetests --with-coverage --cover-erase --cover-html
>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>> --with-ignore-docstrings 

[DISCUSS] AIP - Time for Airflow Improvement Proposals?

[Jakob Homan]

Lots of Apache projects use ?IPs - Whatever Improvement Proposal - to
document and gather consensus on large changes to the code base.  Some
examples:
   * Kafka Improvement Proposals (KIP) -
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
  * Flink Improvement Proposal (FLIP) -
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
  * Spark Improvement Proposal (SPIP) -
https://spark.apache.org/improvement-proposals.html

We've got a few changes that have been discussed, either on the
list/JIRA (good) or in private (bad -
https://incubator.apache.org/guides/committer.html#mailing_lists) that
are of a magnitude that they may benefit from some version of this
process.  Examples:
   * The in-progress plan to refactor out connectors and hooks (AIRFLOW-2732)
   * K8S deployment operator proposal
   * Initial Design for Supporting fine-grained Connection encryption


The benefits of this approach is that the design is hosted somewhere
less ephemeral and more editable than email.  It also provides a
framework for documenting and confirming consensus through the whole
community.

   What do y'all think?

-Jakob

Re: 1.10.0beta1 now available for download

[Jakob Homan]

Hey Bolke-
   I appreciate your patience; these types of discussions can
certainly drag on.  And again, I appreciate your effort to move this
release forward.

   I'd recommend bringing this discussion to the general@incubator.
There will be lots of folks there who will be ready to share an
opinion.  You're looking for a lightweight mechanism to package and
distribute some work for wider consumption in the hopes of the wider
community using it and improving it.  As a mentor, this looks at lot
to me like a release or a release process (announcing it on the
mailing list, using dist.apache.org to host the artifacts, the
artifact being given a version and modifier without a community vote,
etc.).  I'm all for finding the way forward with this; again this is a
great effort and just needs to be done within the ASF framework.

Thanks,
Jakob


On 2 May 2018 at 15:33, Bolke de Bruin <bdbr...@gmail.com> wrote:
> Hi Jakob,
>
> I’m having the feeling we are on different wave lengths and we are not 
> getting closer :-(.
>
> Remarks inline.
>
>> On 2 May 2018, at 22:56, Jakob Homan <jgho...@gmail.com> wrote:
>>
>> Hey Bolke-
>>  Stabilizing the tree has nothing to do with getting a release
>> through IPMC.  The IPMC doesn't test the code - it only verifies that
>> the required licenses and legal obligations are met, that the release
>> artifacts meet the requirements to be processed through ASF's
>> publishing infra, etc.  Minor issues like a couple missing headers are
>> also now generally let through (since it's an incubator release), to
>> be fixed in the next go around.  And honestly, if a podling that
>> believes it's ready to graduate is still having trouble making sure
>> correct license headers are applied, that's a giant red flag that
>> there may be large remaining Apache Way issues to address.  Podlings
>> demonstrating their ability to follow ASF process and operate in the
>> Apache Way is the crux of the incubation process.
>
> The header XYZ was just an example. I think we are doing reasonably
> fine on the process with a couple of hiccups here and there. And some
> discussions like this of course ;-)
>
>>
>>   Also, I entirely agree with you that it's much harder than it
>> should be to get the requisite votes from the IPMC.  I do quite a lot
>> of vote munging for all the podlings with which I'm involved and it's
>> always annoying.  The IPMC, like all of ASF, is made up of volunteers
>> and is not always as responsive as it should be.
>
> Fully understood and thank you for the effort!
>
>>
>>  The process you describe as not having merit is a large part of the
>> ASF.  Specifically, preparing a release candidate is pretty well
>> documented across ASF [1,2,3,4].  The goals you describe (stabilizing
>> a new feature like the Kubernates executor, and rallying people to try
>> the code and fix bugs) are exactly what the RC process does as well.
>> And once the community gets experience in rolling RCs and running
>> release votes, it's actually not that much work.  Lots of projects
>> have multiple releases (main branches and bug fixes) going nearly
>> constantly.
>
> Here, I don’t follow you anymore. I don’t agree that the release process
> as described mentions Release Candidates at all. It mentions shepherding
> from an initial consensus to a final distribution. It doesn’t mention how to
> do this (e.g. by having a release candidate) it just mentions output criteria.
> So it is up to the release manager in consensus with the community
> how to get to a release. It is not stipulated that we need release candidates.
>
> The different projects seem to have different ways of shepherding.
>
>>
>>  I'm not suggesting that we vote on alpha/beta releases.  I'm
>> pointing out that the goals of the alpha release as you describe them
>> match up very well with the goals of the RC process - which will need
>> to be done subsequently anyway.  I'm also saying that announcing
>> 'betas' doesn't really jive with how ASF expects artifacts to be
>> released or voted on for release (which, if an artifact is up on
>> dist.apache.org, it most definitely appears to be).  My suggestion
>> would be to take this very meritorious effort and make it official.
>> Pick a release manager, create a branch, roll an RC, ask for people to
>> stabilize it, merge bug fixes but not features to the branch, repeat
>> until an RC passes a vote.
>
> I don’t think they match. Firstly, for the beta it is not a release and we 
> don’t
> intend to make it one. Secondly, a Release Candidate has a meaning
> attached to it to our community. It says: We think it Ready for Production
> 

Re: 1.10.0beta1 now available for download

[Jakob Homan]

Hey Bolke-
  Stabilizing the tree has nothing to do with getting a release
through IPMC.  The IPMC doesn't test the code - it only verifies that
the required licenses and legal obligations are met, that the release
artifacts meet the requirements to be processed through ASF's
publishing infra, etc.  Minor issues like a couple missing headers are
also now generally let through (since it's an incubator release), to
be fixed in the next go around.  And honestly, if a podling that
believes it's ready to graduate is still having trouble making sure
correct license headers are applied, that's a giant red flag that
there may be large remaining Apache Way issues to address.  Podlings
demonstrating their ability to follow ASF process and operate in the
Apache Way is the crux of the incubation process.

   Also, I entirely agree with you that it's much harder than it
should be to get the requisite votes from the IPMC.  I do quite a lot
of vote munging for all the podlings with which I'm involved and it's
always annoying.  The IPMC, like all of ASF, is made up of volunteers
and is not always as responsive as it should be.

  The process you describe as not having merit is a large part of the
ASF.  Specifically, preparing a release candidate is pretty well
documented across ASF [1,2,3,4].  The goals you describe (stabilizing
a new feature like the Kubernates executor, and rallying people to try
the code and fix bugs) are exactly what the RC process does as well.
And once the community gets experience in rolling RCs and running
release votes, it's actually not that much work.  Lots of projects
have multiple releases (main branches and bug fixes) going nearly
constantly.

  I'm not suggesting that we vote on alpha/beta releases.  I'm
pointing out that the goals of the alpha release as you describe them
match up very well with the goals of the RC process - which will need
to be done subsequently anyway.  I'm also saying that announcing
'betas' doesn't really jive with how ASF expects artifacts to be
released or voted on for release (which, if an artifact is up on
dist.apache.org, it most definitely appears to be).  My suggestion
would be to take this very meritorious effort and make it official.
Pick a release manager, create a branch, roll an RC, ask for people to
stabilize it, merge bug fixes but not features to the branch, repeat
until an RC passes a vote.

Thanks,
Jakob

[1] https://www.google.com/search?q=preparing+an+apache+release+candidate
[2] http://www.apache.org/dev/release-publishing.html
[3] https://incubator.apache.org/guides/releasemanagement.html
[4] https://www.apache.org/foundation/voting.html

On 2 May 2018 at 10:40, Bolke de Bruin <bdbr...@gmail.com> wrote:
> Hi Jakob,
>
> This ‘release’ is not effectively a RC. We want to have the kubernetes
> executor stabilised or at least passing its own tests before we like to move
> to RC status. People also tend to rally to have some extra bugfixes in or
> some extra features when we announce “beta” status. Given the fact that
> going from 1.9 to 1.10 is a big leap I think it is important to have
> period to funnel towards a RC/Release.
>
> Gotcha on httpd. However it still seems semantics to me. I would equal
> a Spark nightly somewhat to an Airflow alpha. A snapshot somewhat
> to a beta. Ie. for Airflow ‘alphas’ and ‘betas’ are not releases, not from a
> process perspective and and not from a technical perspective.
>
> Practically, I think we need a way to stabilise the tree so we have
> a reasonable confidence we can pass a vote for ‘real release, which is a
> technical vote of confidence and a process vote of confidence. Voting
> on alphas (equivalent to a nightly) and betas would make this a very
> cumbersome process. Particularly as a podling: getting 3 votes at the IPMC
> is a tough process (I’ve been physically going around at a conference to
> obtain votes last year). If we then get a “no you can’t have a alpha because
> header XYZ is missing” it kind of defeats the purpose of having alphas
> from the process side (which you are basically saying). However, it still
> has a technical merit.
>
> What would your suggestion be? I’m really afraid of getting stuck
> in process and the process, to me currently, does not seem to have the merit
> we are looking for*. We might have a different understanding
> what we consider to be a ‘release’ though. So open to suggestions
> (also from the wider community here :) ).
>
> Cheers
> Bolke
>
> * dont misunderstand me here please, for Releases (e.g. 1.10.0 with no extra
> label) I’m quite okay.
>
>> On 1 May 2018, at 23:51, Jakob Homan <jgho...@gmail.com> wrote:
>>
>> Hey-
>>   Correct, we can publish nightlies and SNAPSHOTs, but those are not
>> releases.  Also, if a community votes to consider a release alpha or
>> beta, it may do so (From the ht

Re: 1.10.0beta1 now available for download

[Jakob Homan]

Hey-
   Correct, we can publish nightlies and SNAPSHOTs, but those are not
releases.  Also, if a community votes to consider a release alpha or
beta, it may do so (From the httpd link, "Based on the community's
confidence in the code, the potential release is tagged as alpha, beta
or general availability (GA) and the candidate and is voted in that
manner."), but this is an indicator of the technical quality of the
actual release, not the point in the release's lifecycle.

   My question is - if this  release is effectively an RC, why not
make it officially so? What's the goal of the beta compared to an RC?
As a mentor, I see an invitation for users to come and test some work
that could potentially be a release.  That's what we ask for during a
release process, along with the release manager activity, publishing
to specified locations, etc.  It would be good to demonstrate we can
do that well.

Thanks,
Jakob


On 1 May 2018 at 14:31, Bolke de Bruin <bdbr...@gmail.com> wrote:
> Hi Jakob,
>
> To be honest I’m confused now. In software land (and I assume you know)
> Alpha -> Beta -> RC -> Release is well known and so well established that I 
> would
> be surprised if anyone got confused by that. Even the oldest project from 
> Apache
> have alpha-s and beta-s (https://httpd.apache.org/dev/release.html) and 
> something
> called GA which is equal to a release I guess.
>
> If you would expect people to pick up from a git tag and build from there and 
> then report back
> to us, that doesn’t really happen. We are always having a challenge to have 
> enough test surface,
> that would diminish that surface.
>
> Other projects also “publish” other than voted upon artefacts. E.g. Spark has 
> nightly builds and SNAPSHOTS.
> A snapshot clearly has a different state than a nightly. Apache Flink state 
> that 1.4.2 is their latest stable release.
> So there seems to be a “non-stable” release as well. I did see that their git 
> repositories only mention “RC-X” tags
> or branches.
>
> Reading through https://incubator.apache.org/policy/incubation.html#releases 
> it does not mention anywhere
> that we need to have RCs. It just states that if you want to do a release you 
> need to call a vote and for distribution
> it must be at a certain location. As mentioned this is a “beta” which is not 
> a “release”. We haven’t released it either as
> it wasn’t voted upon and no vote was called. It was just made available for 
> convenience of the community.
>
> So I am not sure what is expected from us here. How do wo go though dev -> 
> test -> acc -> prod release process
> together with the community? The release process you seem to be referring is 
> only part of the last state imho. Or
> do we need to call a vote on every state change?
>
> Cheers
> Bolke
>
>
>> On 1 May 2018, at 22:47, Jakob Homan <jgho...@gmail.com> wrote:
>>
>> Hey Bolke-
>>   To be clear, I'm not suggesting anyone is trying to do anything
>> wrong.  Release wasn't mentioned, but a new tar ball with a new
>> version number with a 'beta' tag is published in some way for people
>> to come and test.  How is that different than the expected release/RC
>> process (specify a git point, offer a tar ball, add an RCx tag and
>> invite people to test that)?  Seems like a parallel process with lots
>> of similarities that could confuse both our end users and the IPMC.
>>
>> Thanks,
>> Jakob
>>
>> On 1 May 2018 at 13:08, Bolke de Bruin <bdbr...@gmail.com> wrote:
>>> Hi Jakob,
>>>
>>> Understood. But isn’t that in this case not just wording? Ie. this is a 
>>> tar-ball that we think is beyond just developer testing (alpha) but more 
>>> towards the enthusiasts (beta) but not a version of the tarball that is for 
>>> the general public to test (RC) and not a Release (release)? Ie. is the 
>>> issue in calling it a ‘release’ which in this case is just meta for a 
>>> tarball? In the original email in never mentioned the word release in 
>>> conjunction with the beta I think.
>>>
>>> Cheers
>>> Bolke
>>>
>>>
>>>> On 1 May 2018, at 22:01, Jakob Homan <jgho...@gmail.com> wrote:
>>>>
>>>> Hey all-
>>>> With my Mentor hat on, I need to point out that ASF doesn't really
>>>> have beta releases.  This work is awesome, but really needs to go
>>>> through the proper steps.  The Release Candidate process is pretty
>>>> well described:
>>>> https://incubator.apache.org/policy/incubation.html#releases.  This is
>>>> particularly important since, as was mentioned, graduation should be

Re: 1.10.0beta1 now available for download

[Jakob Homan]

Hey Bolke-
   To be clear, I'm not suggesting anyone is trying to do anything
wrong.  Release wasn't mentioned, but a new tar ball with a new
version number with a 'beta' tag is published in some way for people
to come and test.  How is that different than the expected release/RC
process (specify a git point, offer a tar ball, add an RCx tag and
invite people to test that)?  Seems like a parallel process with lots
of similarities that could confuse both our end users and the IPMC.

Thanks,
Jakob

On 1 May 2018 at 13:08, Bolke de Bruin <bdbr...@gmail.com> wrote:
> Hi Jakob,
>
> Understood. But isn’t that in this case not just wording? Ie. this is a 
> tar-ball that we think is beyond just developer testing (alpha) but more 
> towards the enthusiasts (beta) but not a version of the tarball that is for 
> the general public to test (RC) and not a Release (release)? Ie. is the issue 
> in calling it a ‘release’ which in this case is just meta for a tarball? In 
> the original email in never mentioned the word release in conjunction with 
> the beta I think.
>
> Cheers
> Bolke
>
>
>> On 1 May 2018, at 22:01, Jakob Homan <jgho...@gmail.com> wrote:
>>
>> Hey all-
>>  With my Mentor hat on, I need to point out that ASF doesn't really
>> have beta releases.  This work is awesome, but really needs to go
>> through the proper steps.  The Release Candidate process is pretty
>> well described:
>> https://incubator.apache.org/policy/incubation.html#releases.  This is
>> particularly important since, as was mentioned, graduation should be
>> imminent and this process will be heavily scrutinized.
>>
>> -Jakob
>>
>> On 1 May 2018 at 12:41, James Meickle <jmeic...@quantopian.com> wrote:
>>> Thanks for the pointer! I went through and set this up today, using Google
>>> OAuth as the RBAC provider. Overall I'm quite enthusiastic about this move,
>>> but I thought that it might be helpful to collect feedback as someone who
>>> hasn't been following the overall process and is therefore coming at it
>>> with fresh eyes.
>>>
>>> - The Flask appbuilder security documentation is poor quality (e.g.,
>>> there's some broken sentences); if Airflow is to send people there, it
>>> might be worth PRing some of the docs to at least look more professional.
>>>
>>> - There's not much documentation out there on how to properly set up an
>>> OAuth app in Google (in my case, using the G+ API). From an adoption POV,
>>> it would be good to screenshot the (current) steps in the process, and
>>> point out which values should be used in which fields on Google. For
>>> example, I had to grep the code base to find the callback URL.
>>>
>>> - The initial login UI seems over-complex: you have to click the provider
>>> icon, and then click either login or register. The standard for this
>>> workflow is that you login by clicking the desired provider's icon, and
>>> doing so will register you automatically if you aren't already. In my case
>>> I only have one provider, so this menu was even more confusing.
>>>
>>> - It was not clear to me that the "Public" role has absolutely no
>>> permissions. When I set this as the default role and registered, I could no
>>> longer access the site until I cleared cookies. I thought it was an OAuth
>>> error at first, but it turns out the Public role has fewer effective
>>> permissions than an anonymous user; this resulted in a redirect loop
>>> because I could not even view the homepage. I had to correct this in the
>>> database to be able to log in.
>>>
>>> - The roles list (at roles/list/ ) is intimidatingly large and hard to
>>> parse. For instance, I couldn't tell at a glance what "user" allows
>>> relative to "viewer". It would be good to have a narrative description of
>>> what each of these roles is intended for, and to present the list of
>>> permissions in a more clustered or diffable way. Permissions lists tend to
>>> only grow, after all.
>>>
>>> - A "Viewer" currently lacks enough access to see their own profile.
>>>
>>> - "User Statistics" (userstatschartview/chart/) uses the internal name,
>>> rather than firstname/lastname - which in my case is a `google_idnumber`
>>> name. Should probably show both names.
>>>
>>> Unrelatedly to RBAC (I think), on this branch on my sandbox instance, tasks
>>> appear to be failing with the only logs present in the UI as:
>>>
>>> [{'end_of_log': True}, {'end_of_lo

Re: 1.10.0beta1 now available for download

[Jakob Homan]

Hey all-
  With my Mentor hat on, I need to point out that ASF doesn't really
have beta releases.  This work is awesome, but really needs to go
through the proper steps.  The Release Candidate process is pretty
well described:
https://incubator.apache.org/policy/incubation.html#releases.  This is
particularly important since, as was mentioned, graduation should be
imminent and this process will be heavily scrutinized.

-Jakob

On 1 May 2018 at 12:41, James Meickle  wrote:
> Thanks for the pointer! I went through and set this up today, using Google
> OAuth as the RBAC provider. Overall I'm quite enthusiastic about this move,
> but I thought that it might be helpful to collect feedback as someone who
> hasn't been following the overall process and is therefore coming at it
> with fresh eyes.
>
> - The Flask appbuilder security documentation is poor quality (e.g.,
> there's some broken sentences); if Airflow is to send people there, it
> might be worth PRing some of the docs to at least look more professional.
>
> - There's not much documentation out there on how to properly set up an
> OAuth app in Google (in my case, using the G+ API). From an adoption POV,
> it would be good to screenshot the (current) steps in the process, and
> point out which values should be used in which fields on Google. For
> example, I had to grep the code base to find the callback URL.
>
> - The initial login UI seems over-complex: you have to click the provider
> icon, and then click either login or register. The standard for this
> workflow is that you login by clicking the desired provider's icon, and
> doing so will register you automatically if you aren't already. In my case
> I only have one provider, so this menu was even more confusing.
>
> - It was not clear to me that the "Public" role has absolutely no
> permissions. When I set this as the default role and registered, I could no
> longer access the site until I cleared cookies. I thought it was an OAuth
> error at first, but it turns out the Public role has fewer effective
> permissions than an anonymous user; this resulted in a redirect loop
> because I could not even view the homepage. I had to correct this in the
> database to be able to log in.
>
> - The roles list (at roles/list/ ) is intimidatingly large and hard to
> parse. For instance, I couldn't tell at a glance what "user" allows
> relative to "viewer". It would be good to have a narrative description of
> what each of these roles is intended for, and to present the list of
> permissions in a more clustered or diffable way. Permissions lists tend to
> only grow, after all.
>
> - A "Viewer" currently lacks enough access to see their own profile.
>
> - "User Statistics" (userstatschartview/chart/) uses the internal name,
> rather than firstname/lastname - which in my case is a `google_idnumber`
> name. Should probably show both names.
>
> Unrelatedly to RBAC (I think), on this branch on my sandbox instance, tasks
> appear to be failing with the only logs present in the UI as:
>
> [{'end_of_log': True}, {'end_of_log': True}, {'end_of_log': True},
> {'end_of_log': True}, {'end_of_log': True}, {'end_of_log': True}]
>
>
> Finally, in case anyone else wanted to test run a similar setup, here is
> the webserver_config.py that I ended up using (note that it has Jinja
> templating via Ansible):
>
> import os
> from airflow import configuration as conf
> from flask_appbuilder.security.manager import AUTH_OAUTH
> basedir = os.path.abspath(os.path.dirname(__file__))
>
> # The SQLAlchemy connection string.
> SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')
>
> # Flask-WTF flag for CSRF
> CSRF_ENABLED = True
>
> # The name to display, e.g. "Airflow Staging Sandbox"
> APP_NAME = "Airflow {{ env }} {{ app_config | capitalize }}"
>
> # Use OAuth
> AUTH_TYPE = AUTH_OAUTH
>
> # Will allow user self registration
> AUTH_USER_REGISTRATION = True
>
> # The default user self registration role
> AUTH_USER_REGISTRATION_ROLE = "{{ airflow_rbac_registration_role |
> default('Viewer') }}"
>
> # Google OAuth:
> OAUTH_PROVIDERS = [{
> # The name of the provider
> 'name': 'google',
> # The icon to use
> 'icon': 'fa-google',
> # The name of the key that the provider sends
> 'token_key': 'access_token',
> # Just in case, whitelist to only @quantopian.com emails
> 'whitelist': ['@quantopian.com'],
> # Define the remote app:
> 'remote_app': {
> 'base_url': 'https://www.googleapis.com/oauth2/v2/',
> 'access_token_url': 'https://accounts.google.com/o/oauth2/token',
> 'authorize_url': 'https://accounts.google.com/o/oauth2/auth',
> 'request_token_url': None,
> 'request_token_params': {
> # Uses the Google+ API, requestingf the 'email' and 'profile' scope
> 'scope': 'email profile'
> },
> 'consumer_key': '{{ vault_airflow_google_oauth_key }}',
> 'consumer_secret': '{{ vault_airflow_google_oauth_secret }}'
> }
> }]
>
>
>
> On Mon, Apr 30, 2018 at 12:54 PM, Jørn A Hansen 
> wrote:
>
>> On 

Re: [VOTE] Migrate to Github as primary repo (a.k.a. Gitbox)

[Jakob Homan]

> +1 (binding)
>
> For future reference, is this vote for anyone on the mailing list, or for
> those with some kind of status in the project?

Matthew - yeah, binding votes are reserved for committers or PMC
members (depending on the vote).  Everyone in the community is
encouraged to vote, and those with binding votes are expected to pay
attention to those votes that aren't binding (ie, don't vote something
through that the larger community is angry about or has noticed
significant problems with), but in the end, it's the binding votes
that actually count.  Researching a question and voting is a form of
contribution to the project, so it's never wasted.  Brett has a good
slide on the general way this works:
https://www.slideshare.net/Hadoop_Summit/the-apache-way-80377908

-Jakob

On 12 March 2018 at 10:23, Chris Riccomini  wrote:
> +1
>
> On Sat, Mar 10, 2018 at 11:18 PM, Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
>
>> +1 (binding)
>>
>> On Sat, Mar 10, 2018 at 2:03 PM, Beau Barker 
>> wrote:
>>
>> > +1 for Github.
>> >
>> > Also think that moving to Github issues would be a step in the right
>> > direction.
>> >
>> >
>> > > On 11 Mar 2018, at 05:56, Matthew Housley 
>> > wrote:
>> > >
>> > > +1 (binding)
>> > >
>> > > For future reference, is this vote for anyone on the mailing list, or
>> for
>> > > those with some kind of status in the project? I find the documentation
>> > > here a little ambiguous:
>> > > https://httpd.apache.org/dev/guidelines.html#voting
>> > > Apologies if this has been answered before.
>> > >
>> > > On Sat, Mar 10, 2018 at 8:29 AM Ash Berlin-Taylor <
>> > > ash_airflowl...@firemirror.com> wrote:
>> > >
>> > >> Consider this my +1 (binding) vote for the below proposal. This vote
>> > will
>> > >> run for 7 days (until 2018-03-17 15:30+00)
>> > >>
>> > >> **Proposal**: We switch to using GitHub as our primary repo
>> > >>
>> > >> We would still use the Apache Jira for issue/release tracking etc.
>> > >>
>> > >> Benefits:
>> > >>
>> > >> The contributors will gain write access to
>> > >> github.com/apache/incubator-airflow. This would mean we would be able
>> > to:
>> > >>
>> > >> - merge directly on github.com
>> > >> - close stale issues
>> > >> - be able to re-run Travis jobs (I think/hope)
>> > >>
>> > >> Risks:
>> > >>
>> > >> Neither of these are likely to be a problem, but the possible
>> downsides
>> > >> are:
>> > >>
>> > >> - It is still possible to commit to the ASF repo, which if it happens
>> > can
>> > >> lead to "split brain" (i.e. different views of master) which will need
>> > >> INFRA team support to fix.
>> > >>
>> > >> - Contributors will need to agree to Github terms of service. Given
>> this
>> > >> is how PRs are reviewed currently this isn't a problem for any current
>> > >> contributors. Just worth mentioning.
>> > >>
>> > >>
>> > >> If the vote passes we will need to:
>> > >>
>> > >> - Update the airflow-pr tool to work directly on github, not ASF repos
>> > >> - Update any docs that point to ASF repos (
>> > >> http://incubator.apache.org/projects/airflow.html,
>> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/
>> Committers%27+Guide
>> > -
>> > >> there might be more)
>> > >> - Ensure all committers have access. There is an self-serve process
>> for
>> > >> this (see below)
>> > >> - Open a ticket with the INFRA queue asking them to migrate the repos.
>> > (An
>> > >> example of ticket from another Apache project that did this recently
>> > >> https://issues.apache.org/jira/browse/INFRA-15983 )
>> > >>
>> > >>
>> > >> Contributor set up steps:
>> > >>
>> > >> - Go to https://id.apache.org/ and ensure you have a github username
>> > >> entered in your profile. This will send an invite to join the Apache
>> > org on
>> > >> Github: accept that.
>> > >> - Go to https://gitbox.apache.org/setup/ and link the accounts
>> > >>
>> > >> These steps can be done now no matter the outcome of the vote -- we
>> just
>> > >> won't get write access to airflow unless we migrate.
>> > >>
>> > >> Ash
>> >
>>

Re: Airflow thshirts

[Jakob Homan]

Be sure to run the new design past Brand Management too
(https://www.apache.org/foundation/marks/ - Using Apache trademarks on
merchandise).

Thanks for doing this Maxime, it's awesome.

-jg

On 26 January 2018 at 11:06, Maxime Beauchemin
<maximebeauche...@gmail.com> wrote:
> Ooops. Apparently the link didn't work either. I'll do a second try soon
> and add a feather!
>
> Max
>
> On Wed, Jan 24, 2018 at 11:45 PM, Sid Anand <san...@apache.org> wrote:
>
>> I couldn't agree more..
>>
>> add the feather :_)
>>
>> -s
>>
>> On Wed, Jan 24, 2018 at 5:49 PM, Jakob Homan <jgho...@gmail.com> wrote:
>>
>> > 
>> > *Apache* Airflow...
>> > 
>> >
>> > ...
>> >
>> > -jg
>> >
>> >
>> > On 24 January 2018 at 14:42, Trent Robbins <robbi...@gmail.com> wrote:
>> > > Thanks Max!
>> > >
>> > > Trent
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > Best,
>> > >
>> > > Trent Robbins
>> > > Strategic Consultant for Open Source Software
>> > > Tau Informatics LLC
>> > > desk: 415-404-9452
>> > > cell: 513-233-5651
>> > > tr...@tauinformatics.com
>> > > https://www.linkedin.com/in/trentrobbins
>> > >
>> > > On Wed, Jan 24, 2018 at 1:59 PM, Maxime Beauchemin <
>> > > maximebeauche...@gmail.com> wrote:
>> > >
>> > >> After a recent Meetup where many folks asked about where they could
>> get
>> > an
>> > >> Airflow tshift like the one I was wearing, Now I'm trying to get a
>> > batch of
>> > >> Airflow tshirts kickstarted on Teespring.
>> > >>
>> > >> BTW Teespring uses Airflow, making it the obvious choice for this :)
>> > >>
>> > >> Order yours here:
>> > >> https://teespring.com/teespringdirect-nl2qbp#pid=72;
>> cid=5902=front
>> > >>
>> > >> Profits (if any) will be donated to charity.
>> > >>
>> > >> Thanks,
>> > >>
>> > >> Max
>> > >>
>> >
>>

Re: Airflow thshirts

[Jakob Homan]

*Apache* Airflow...


...

-jg


On 24 January 2018 at 14:42, Trent Robbins  wrote:
> Thanks Max!
>
> Trent
>
>
>
>
>
> Best,
>
> Trent Robbins
> Strategic Consultant for Open Source Software
> Tau Informatics LLC
> desk: 415-404-9452
> cell: 513-233-5651
> tr...@tauinformatics.com
> https://www.linkedin.com/in/trentrobbins
>
> On Wed, Jan 24, 2018 at 1:59 PM, Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
>
>> After a recent Meetup where many folks asked about where they could get an
>> Airflow tshift like the one I was wearing, Now I'm trying to get a batch of
>> Airflow tshirts kickstarted on Teespring.
>>
>> BTW Teespring uses Airflow, making it the obvious choice for this :)
>>
>> Order yours here:
>> https://teespring.com/teespringdirect-nl2qbp#pid=72=5902=front
>>
>> Profits (if any) will be donated to charity.
>>
>> Thanks,
>>
>> Max
>>

[RESULT][VOTE] Release Airflow 1.9.0

[Jakob Homan]

The vote to release Airflow 1.9.0-incubating, having been open for 11
days is now closed.

There were three binding +1s and no -1 votes.

+1 (binding):
Chris Douglas
Justin Mclean
Jakob Homan

The release is approved.

Thanks to all those who voted.

-Jakob

On 29 December 2017 at 12:18, Chris Douglas <cdoug...@apache.org> wrote:
> +1 (binding) overall. Looked through the src tarball.
>
> * Checked LICENSE/NOTICE/DISCLAIMER
> * Hashes match
> * Signature matches
> * Incubator naming, no binary files (other than images)
>
> For future releases: the src tarball extracting to the current
> directory was surprising. The top-level INSTALL file could reference
> (or include) more of the quickstart under docs/. -C
>
>
> On Thu, Dec 21, 2017 at 3:39 PM, Jakob Homan <jgho...@gmail.com> wrote:
>> +1 (binding)
>>
>> * sigs/hashes look good.
>> * disclaimer / license / notice look good
>> * a few files are missing licenses, like docs/Makefile, as Justin noticed
>> * artifact naming looks good
>> * setup.py build succeeds
>>
>> Thanks,
>> Jakob
>>
>>
>> On 20 December 2017 at 13:41, Bolke de Bruin <bdbr...@gmail.com> wrote:
>>> @sebb, @henk are you now able to vote +1 or is there anything else we need 
>>> to pickup before?
>>>
>>> Cheers
>>> Bolke
>>>
>>>> On 20 Dec 2017, at 15:29, sebb <seb...@gmail.com> wrote:
>>>>
>>>> On 19 December 2017 at 18:42, Chris Riccomini <criccom...@apache.org 
>>>> <mailto:criccom...@apache.org>> wrote:
>>>>>> Please don't use the ".sha" extension ; use ".sha1", ".sha256", or
>>>>> ".sha512" instead for :
>>>>>
>>>>> Done.
>>>>>
>>>>>> If development of 1.8.0-incubating (and/or 1.8.1-incubating) has stopped
>>>>> (because superseded by 1.8.2-incubating), please remove it from
>>>>> /dist/release/incubator/airflow/.
>>>>>
>>>>> Done.
>>>>>
>>>>>> The dist.apache.org URL is not for use by the general public; it is only
>>>>> intended as the staging area for Airflow developers during a release.
>>>>>
>>>>> @sebb, this is where I'm confused. Since you guys haven't approved the
>>>>> release yet, I was under the impression that we were still staging the
>>>>> release. Are you saying I should move the rc8 artifacts over to
>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ now, before
>>>>> the IPMC has +1'd the release?
>>>>
>>>> Sorry, my bad. I misread the email.
>>>>
>>>> dist.a.o is the correct host to use for artifacts, sigs, hashes in a VOTE.
>>>>
>>>> Ideally the KEYS link should use www.a.o/dist <http://www.a.o/dist>, as 
>>>> the keys are best
>>>> maintained in dist/release rather than dist/dev (the file only needs
>>>> to be updated to add new RM keys).
>>>>
>>>>> On Tue, Dec 19, 2017 at 8:35 AM, sebb <seb...@gmail.com> wrote:
>>>>>
>>>>>> On 18 December 2017 at 20:32, Chris Riccomini <criccom...@apache.org>
>>>>>> wrote:
>>>>>>> Hello Incubator PMC’ers,
>>>>>>>
>>>>>>> The Apache Airflow community has voted and approved the proposal to
>>>>>> release
>>>>>>> Apache Airflow 1.9.0 (incubating) based on 1.9.0 Release Candidate 8. We
>>>>>>> now kindly request the Incubator PMC members to review and vote on this
>>>>>>> incubator release.
>>>>>>>
>>>>>>> Airflow is a platform to programmatically author, schedule, and monitor
>>>>>>> workflows. Use Airflow to author workflows as directed acyclic graphs
>>>>>>> (DAGs) of tasks. The airflow scheduler executes your tasks on an array 
>>>>>>> of
>>>>>>> workers while following the specified dependencies. Rich command line
>>>>>>> utilities make performing complex surgeries on DAGs a snap. The rich 
>>>>>>> user
>>>>>>> interface makes it easy to visualize pipelines running in production,
>>>>>>> monitor progress, and troubleshoot issues when needed. When workflows 
>>>>>>> are
>>>>>>> defined as code, they become more maintainable, versionable, testable

Re: [RESULT][VOTE] Airflow 1.9.0rc8

[Jakob Homan]

Sent out a call to arms...

-jg


On 27 December 2017 at 10:22, Sid Anand  wrote:
> Ugh... you're right. Hmn... what about our other 2 mentors?
>
> -s
>
> On Wed, Dec 27, 2017 at 4:14 AM, Bolke de Bruin  wrote:
>
>> Did you count 3? I only counted 2?
>>
>> Sent from my iPhone
>>
>> > On 27 Dec 2017, at 02:48, Sid Anand  wrote:
>> >
>> > I see that we received 3 +1 binding votes (assuming all IPMC members).
>> Per
>> > https://incubator.apache.org/guides/releasemanagement.html, shouldn't
>> that
>> > be sufficient?
>> >
>> > In the podling report, should I mention that we did our 4th release or
>> > still working toward it?
>> > -s
>> >
>> > On Tue, Dec 19, 2017 at 9:50 AM, Chris Riccomini 
>> > wrote:
>> >
>> >> I am waiting until incubator actually approves the release. Otherwise, I
>> >> might have to redo the tag if they reject us and we have to re-vote a
>> new
>> >> RC.
>> >>
>> >>> On Tue, Dec 19, 2017 at 8:19 AM, Bolke de Bruin 
>> wrote:
>> >>>
>> >>> Chris,
>> >>>
>> >>> Can you please tag 1.9.0 correctly (ie 1.9.0)? I see reports on gitter
>> >>> people have issues with it.
>> >>>
>> >>> Cheers,
>> >>> Bolke.
>> >>>
>>  On 18 Dec 2017, at 21:49, Chris Riccomini 
>> >> wrote:
>> 
>>  Thanks to all for helping validate it. :)
>> 
>>  On Mon, Dec 18, 2017 at 12:48 PM, Ash Berlin-Taylor <
>>  ash_airflowl...@firemirror.com> wrote:
>> 
>> > Woot! Great news, and thanks for all the time you've spent preparing
>> >> all
>> > the RCs.
>> >
>> > -ash
>> >
>> >> On 18 Dec 2017, at 20:13, Chris Riccomini 
>> >>> wrote:
>> >>
>> >> Hello,
>> >>
>> >> Apache Airflow (incubating) 1.9.0 (based on RC8) has been accepted.
>> >>
>> >> 5 “+1” binding votes received:
>> >>
>> >> - Maxime Beauchemin (binding)
>> >> - Chris Riccomini (binding)
>> >> - Bolke de Bruin (binding)
>> >> - Joy Gao (binding)
>> >> - Fokko Driesprong (binding)
>> >>
>> >> My next step is to open a thread with the IPMC.
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> On Mon, Dec 18, 2017 at 11:26 AM, Chris Riccomini <
>> >>> criccom...@apache.org
>> >>
>> >> wrote:
>> >>
>> >>> I believe so, yes.
>> >>>
>> >>> I will proceed with release. :)
>> >>>
>> >>> On Mon, Dec 18, 2017 at 11:20 AM, Bolke de Bruin <
>> bdbr...@gmail.com
>> >>>
>> >>> wrote:
>> >>>
>>  But it doesn’t stop us from releasing now right? You can just
>> >> update
>> > the
>>  tag. It is not connected to the artifacts.
>> 
>>  Bolke.
>> 
>>  Sent from my iPhone
>> 
>> > On 18 Dec 2017, at 18:51, Chris Riccomini > >
>>  wrote:
>> >
>> > https://github.com/apache/incubator-airflow/pull/2889
>> >
>> > On Mon, Dec 18, 2017 at 9:44 AM, Chris Riccomini <
>> > criccom...@apache.org
>> >
>> > wrote:
>> >
>> >> Sigh. There's a catch 22 here. If we tag an RC as 1.9.0, we'll
>> >> have
>> > to
>> >> keep editing the tag every time there's a new RC. I agree that
>>  removing the
>> >> check seems like the best course of action.
>> >>
>> >> On Sun, Dec 17, 2017 at 12:01 AM, Bolke de Bruin <
>> >>> bdbr...@gmail.com>
>> >> wrote:
>> >>
>> >>> This is just matter of setting the tag in the repo right?
>> >>>
>> >>> We should remove that check or make it not fail at least. It is
>> >>> ridiculous.
>> >>>
>> >>> B.
>> >>>
>> >>> Verstuurd vanaf mijn iPad
>> >>>
>>  Op 17 dec. 2017 om 07:32 heeft Joy Gao  het
>> > volgende
>> >>> geschreven:
>> 
>>  Ahh, tested the build on a fresh virtualenv, which succeeded
>> >> *the
>>  first
>>  time* given gitpython was not installed and it skipped the
>> > assertion
>> >>> check
>>  > >>> setup.py#L68-L73>.
>>  Build fails on re-installs :(
>> 
>> > On Fri, Dec 15, 2017 at 6:01 PM, Feng Lu
>> > > >
>> >>> wrote:
>> >
>> > +0.5 (non-binding)
>> >
>> > Looks like the version(1.9.0) and tag(1.9.0rc8) is
>> mismatched,
>> > > >>> setup.py#L87>
>> > which will cause the installation (pip install or python
>> >> setup)
>> >>> to
>> >>> error
>> > out and fail.
>> > nit: mind also updating the release log "
>> > 

Re: Podling Report Reminder - April 2017

[Jakob Homan]

On 5 April 2017 at 10:33, Bolke de Bruin  wrote:
> If you can please include our intention to graduate after the 1.8.1 release 
> (no more reports ;-).

Did I miss this discussion on the mailing list?  If so, can you point
me to it?  If it was discussed there; there may be some work to do
before graduation.  :)

Also, graduation doesn't obviate the reports.  Instead of reporting to
the IPMC, we will report to the Board.  The reports are pretty much
the same and are quarterly.

-Jakob

Re: Okta Authentication

[Jakob Homan]

Yes!  We were just about to start writing one of our own.

-Jakob

On 17 November 2016 at 14:30, Brian Yang  wrote:
> Good Afternoon,
>
> I have implemented an authentication backend for Airflow to authenticate
> against an Okta server. Would this be something that is worth adding into
> Airflow?
>
> Brian Yang

Re: Next release?

[Jakob Homan]

I'm going to set aside some time early next week to see where we are
in the process and make sure nothing's blocking us.

(Jakob announces it publicly so he can be shamed if he doesn't follow through)

On 11 November 2016 at 14:50, siddharth anand  wrote:
> Chris,
> I have scheduled a meeting with Arthur to discuss this same topic on
> Monday. Anyone who would like to join the meeting, email me and I will add
> them to the meeting. It's set for 11-11:30a on Monday.
>
> I'd like to help with this process, but I suspect we are all too new to
> know what is needed. Chris, you might be needed in some form to guide us.
> -s
>
> On Fri, Nov 11, 2016 at 2:18 PM, Chris Riccomini 
> wrote:
>
>> Hey all,
>>
>> There was a plan for someone (Maxime/Arthur?) to cut a 1.8.0 release
>> branch, and start the process. It's been many weeks since that
>> discussion, and I haven't heard anything.
>>
>> Does someone on PMC want to volunteer? I can deploy 1.8.0 and start
>> working through issues once it's cut, and I know Sid and Bolke said
>> they could as well. I don't want to do it because I've already done
>> several Apache releases before. In the interest of high availability,
>> it'd be good for someone else to go through the motions.
>>
>> Cheers,
>> Chris
>>
>> On Thu, Nov 10, 2016 at 8:57 AM, Paul Zaczkiewicz 
>> wrote:
>> > Hi,
>> >
>> > I'm a new user for airflow, and I'm wondering when the next release of
>> > airflow is. My company uses mssql servers for its backend. I very quickly
>> > ran into a bug that was fixed in May
>> > > e7e655fde3c29742149d047028cbb21aecba86ed>.
>> > I've since patched that file in my installation, but I can't help but
>> > wonder what other critical issues have been fixed since version 1.7.1.3.
>> >
>> > I've considered updating my installation to some commit on the master
>> > branch, but I'm hesitant to pick a non-stable version. Even so, how
>> would I
>> > update airflow from source?  Copy the contents of
>> > https://github.com/apache/incubator-airflow
>> > to /usr/lib/python2.7/site-packages after installing 1.7.1.3 through
>> pip?
>> >
>> > Thanks,
>> > Paul Zaczkiewicz
>>

Re: +1 on PRs!

[Jakob Homan]

Be sure to distinguish between +1 == I want this, and +1 == I approve
this change to be merged into the codebase.

Non-committers are welcome and encouraged to review patches, including
providing +1s.  The patch can't be merged on the basis of that
non-committer +1 only (a committer's +1 is still required), but this
is good experience and contribution from the non-committer.


On 11 October 2016 at 12:16, Laura Lorenz  wrote:
> I started to do this and totally got myself stuck answering bug reports 
>
> On Mon, Oct 10, 2016 at 11:33 PM, siddharth anand  wrote:
>
>> Great idea, Max.
>>
>> There is also a vote feature on JIRAs. Sometimes PRs get abandoned, whereas
>> the JIRA tends to stick around longer, sometimes even without an owner. I'm
>> not sure which is the best way, but I completely agree with the sentiment.
>>
>> -s
>>
>> On Mon, Oct 10, 2016 at 6:01 PM, Maxime Beauchemin <
>> maximebeauche...@gmail.com> wrote:
>>
>> > Airflowers,
>> >
>> > I would love if people could write simple `+1` comments on the PRs they
>> > care about.
>> >
>> > It's motivating for contributors to see that people want the features
>> they
>> > work on, and it can help committers prioritize which PRs to review and
>> > release first.
>> >
>> > It's also a great way to keep a pulse on the project, see what is coming
>> > up, and to start getting involved. Of course more involved feedback
>> > (reaction icons, comments, review) are also very welcomed.
>> >
>> > See you on Github!
>> >
>> > Max
>> >
>>

Re: Custom (mongodb) hook as a plugin

[Jakob Homan]

On 23 September 2016 at 15:06, Rob Goretsky  wrote:
> More generally, would it make sense to create an enhancement to the plugin
> architecture that allows the developer of a new hook to also specify the
> name of a new 'connection type' along with what fields would be required
> for that connection type?  The web UI would then take its existing
> hardcoded list and append these plugged-in connection types and fields to
> display...

Yeah, I filed a ticket a while ago to do so
(https://issues.apache.org/jira/browse/AIRFLOW-235)

Additionally, it would be good if the connections were not hardcoded
so that it were easier to add new connections directly (perhaps
through the config file)

-Jakob

Re: Airflow Developers Meeting - 08/03 Notes

[Jakob Homan]

ASF provides per-projects blog infrastructure:
http://www.apache.org/dev/project-blogs

-Jakob


On 23 August 2016 at 15:07, Maxime Beauchemin
 wrote:
> I'm planning on working on a minimal apache release this week. If someone
> can point to the commit or PRs I can add them to `branch-1.7.2-apache` and
> start moving towards the release.
>
> Max
>
> On Tue, Aug 23, 2016 at 11:13 AM, Alex Van Boxel  wrote:
>
>> All stuff is in master (I'm running pre-production on master), if it needs
>> to be cherry-picked to a branch I can always test it. I would be grate if
>> it's in the next release so I can start blogging about it.
>>
>> On Tue, Aug 23, 2016 at 8:05 PM Bolke de Bruin  wrote:
>>
>> > (Sorry for the late response, I was on holiday)
>> >
>> > I think the G* operators just need to be cherry picked. This will make us
>> > deviate slightly from the
>> > previous release, but makes sure we don’t have to ‘fix’ history
>> afterwards.
>> >
>> > Anyone against this?
>> >
>> > - B.
>> >
>> > > Op 8 aug. 2016, om 14:21 heeft Alex Van Boxel  het
>> > volgende geschreven:
>> > >
>> > > Sorry, Bolke. What needs to be done for Google Cloud operators/hooks?
>> > >
>> > > If you bring me up-to-speed I can do this. I'm currently working an a
>> CI
>> > > setup for the Google stuff).
>> > >
>> > > On Sat, Aug 6, 2016 at 3:56 PM Bolke de Bruin 
>> wrote:
>> > >
>> > >> I have indeed a branch with cherry picked commits. This branch is
>> > >> available in the apache repo. The branch is available as
>> > >> “branch-1.7.2-apache”. What is left to do is to decide weather we do
>> > >> upgrade the google cloud operators/hooks as part of this release as
>> the
>> > >> licenses were only added after these changes came in. I prefer this
>> > >> approach as we stay away from custom commits as much as possible then.
>> > >>
>> > >> - Bolke
>> > >>
>> > >>> Op 5 aug. 2016, om 00:21 heeft Gurer Kiratli <
>> gurer.kira...@airbnb.com
>> > .INVALID>
>> > >> het volgende geschreven:
>> > >>>
>> > >>> Agenda
>> > >>>
>> > >>>
>> > >>>  -
>> > >>>
>> > >>>  Committers sync-up: progress and plans
>> > >>>  -
>> > >>>
>> > >>> Max to do a recap since last release
>> > >>> -
>> > >>>
>> > >>> Airbnb
>> > >>> -
>> > >>>
>> > >>> Anyone else?
>> > >>> -
>> > >>>
>> > >>>  Cooperation Best Practices
>> > >>>  -
>> > >>>
>> > >>>  Solicit for feedback for Impersonation Design Review
>> > >>>  -
>> > >>>
>> > >>>  Release Schedule, Management
>> > >>>  -
>> > >>>
>> > >>>  Roadmap discussion
>> > >>>
>> > >>>
>> > >>> Attendees: Jeremiah Lowin, Chris Riccomini, Andrew Phillips(?),
>> Maxime
>> > >>> Beauchemin, Paul Yang, Dan Davydov, Xuanji Li, George Ke, Arthur
>> > Wiedmer,
>> > >>> Gurer Kiratli
>> > >>>
>> > >>> Notes
>> > >>>
>> > >>>
>> > >>>  -
>> > >>>
>> > >>>  Recap for the overall project
>> > >>>  -
>> > >>>
>> > >>> [Max]
>> > >>>
>> > >> https://gist.github.com/mistercrunch/5460483ec764e2a1cb816c6b1d6ad5a3
>> > >>> -
>> > >>>
>> > >>>  Airbnb Recap
>> > >>>  -
>> > >>>
>> > >>> [Paul] Airbnb is continuing work on features related to migrating
>> > >>> more jobs onto Airflow - DB connection scalability,
>> impersonation,
>> > >>> refreshing web UI, DAG git versioning, task resource isolation
>> > >>> -
>> > >>>
>> > >>>  Other Recaps
>> > >>>  -
>> > >>>
>> > >>> [Chris] Improvements to Import functionality. Joy did some work
>> > >>> around CLIs and mark success in collaboration with Bolke.
>> > >>> -
>> > >>>
>> > >>> [Jeremiah] Came up with the Merge tool. Focusing on the work
>> around
>> > >>> configuration.
>> > >>> -
>> > >>>
>> > >>>  Meeting cadence
>> > >>>  -
>> > >>>
>> > >>> [Max] We can have monthly meetings. Let’s start with monthly we
>> can
>> > >>> change the cadence. We should invite all contributors. We should
>> > >>> specify in
>> > >>> the invite that everyone is welcome. Currently posting this on
>> > >>> the mailing
>> > >>> list and wiki.
>> > >>> -
>> > >>>
>> > >>> *Action Item*  Gurer to schedule monthly meetings next one at
>> > >> Airbnb.
>> > >>> Airbnb is welcoming folks to come onsite
>> > >>> -
>> > >>>
>> > >>>  Blog/Documentation
>> > >>>  -
>> > >>>
>> > >>> [Max, Chris, Jeremiah] Having a blog would be useful. We can
>> > >>> structure our thoughts.
>> > >>> -
>> > >>>
>> > >>> [Andrew] Blog should have better language with screenshots and
>> all.
>> > >>> This raises the level of effort bar and this might mean that
>> > >>> nobody really
>> > >>> does it.
>> > >>> -
>> > >>>
>> > >>> [Max] Maybe use Medium? We can try it out.
>> > >>> -
>> > >>>
>> > >>> [Max] Haven’t looked at the documentation for a long time. I
>> don’t
>> > >>> really know about its quality.
>> > >>> -
>> > >>>
>> > >>> [Arthur] Looking at the email list responses, seems 

Re: Private channel being used for discussion on gitter

[Jakob Homan]

Gitter is fine and the public channel I linked to in the original
email is fine.  It's functionally no different than IRC.  The channel
just has to be open for any one rather than invite only.

-Jakob


On 22 August 2016 at 13:10, Maxime Beauchemin
<maximebeauche...@gmail.com> wrote:
> I think Gitter must have just transferred along as we moved the repo. I'm
> personally open to using IRC.
>
> Max
>
> On Mon, Aug 22, 2016 at 12:50 PM, Bolke de Bruin <bdbr...@gmail.com> wrote:
>
>> Also who is the owner of this room? It seems created by Apache so they
>> (INFRA?) can set it to public? Logs will be public then by default.
>>
>> Wasn't there some discussion on a more relaxed requirement of this
>> happening on ASF infrastructure?
>>
>> Bolke.
>>
>> Sent from my iPhone
>>
>> > On 22 aug. 2016, at 20:29, Jakob Homan <jgho...@gmail.com> wrote:
>> >
>> > Hey all-
>> >  Apparently there is a private channel being used for project
>> > discussion on gitter (the archive would be here:
>> > https://gitter.im/apache/incubator-airflow/Airflow_
>> committers/archives/all,
>> > similar to the public archive of the regular gitter channel:
>> > https://gitter.im/apache/incubator-airflow/archives/all), titled
>> > "Airflow_committers."
>> >
>> > As has been discussed before, all non-PMC level ASF communication
>> > *has* to be in a public accessible forum (preferably that is archived
>> > to an ASF list).  Even if well intentioned, these channels are
>> > exclusionary and absolutely verboten per ASF.
>> >
>> > We need to open or shut that channel down and make sure we note this
>> > on the next IPMC report...
>> >
>> > -Jakob
>>

Private channel being used for discussion on gitter

[Jakob Homan]

Hey all-
  Apparently there is a private channel being used for project
discussion on gitter (the archive would be here:
https://gitter.im/apache/incubator-airflow/Airflow_committers/archives/all,
similar to the public archive of the regular gitter channel:
https://gitter.im/apache/incubator-airflow/archives/all), titled
"Airflow_committers."

As has been discussed before, all non-PMC level ASF communication
*has* to be in a public accessible forum (preferably that is archived
to an ASF list).  Even if well intentioned, these channels are
exclusionary and absolutely verboten per ASF.

We need to open or shut that channel down and make sure we note this
on the next IPMC report...

-Jakob

Re: S3 connection

[Jakob Homan]

Hey Tyrone-
   I just set this up on 1.7.1.2 and found the documentation confusing
too.  Been meaning to improve the documentation.  To get S3 logging
configured I:

(a) Set up an S3Connection (let's call it foo) with only the extra
param set to the following json:

{ "s3_config_file": "/usr/local/airflow/.aws/credentials",
"s3_config_format": "aws" }

(b) Added a remote_log_conn_id key to the core section of airflow.cfg,
with a value of "foo" (my S3Connection name)

(c) Added a remote_base_log_folder key to the core section of
airflow.cfg, with a value of "s3://where/i/put/my/logs"

Everything worked after that.

-Jakob

On 15 June 2016 at 15:35, Tyrone Hinderson  wrote:
> @Jeremiah,
>
> http://pythonhosted.org/airflow/configuration.html#logs
>
> I used to log to s3 in 1.7.0, and my background .aws/credentials would take
> care of authenticating in the background. Now it appears that I need to set
> that "remote_log_conn_id" config field in order to continue logging to s3
> in 1.7.1.2. Rather than create the connection in the web UI (afaik,
> impractical to do programatically), I'd like to use an
> "AIRFLOW_CONN_"-style env variable. I've tried an url like
> s3://[access_key_id]:[secret_key]@[bucket].s3-[region].amazonaws.com, but
> that hasn't worked:
>
> =
> [2016-06-15 21:40:26,583] {base_hook.py:53} INFO - Using connection to:
> [bucket].s3-us-east-1.amazonaws.com 
>
> [2016-06-15 21:40:26,583] {logging.py:57} ERROR - Could not create an
> S3Hook with connection id "S3_LOGS". Please make sure that airflow[s3] is
> installed and the S3 connection exists.
>
> =
>
> It's clear that my connection exists because of the "Using connection to:"
> line. However, I fear that my connection URI string is malformed. Can you
> provide some guidance as to how I might properly form an s3 connection URI,
> since I mainly followed a mixture of wikipedia's URI format
> 
> and amazon's
> s3 URI format
> ?
>
> On Tue, May 24, 2016 at 6:03 PM Jeremiah Lowin  wrote:
>
>> Where are you seeing that an S3 connection is required? It will only be
>> accessed if you tols Airflow to send logs to S3. The config option can also
>> be null (default) or a google storage location.
>>
>> The S3 connection is a standard Airflow connection. If you would like it to
>> use environment variables or a boto config, it will -- but the connection
>> object itself must be created in Airflow. See the S3 hook for details.
>>
>>
>> On Tue, May 24, 2016 at 3:57 PM George Leslie-Waksman <
>> geo...@cloverhealth.com> wrote:
>>
>> > We ran into this issue as well. If you set the environment variable to
>> > anything random, it'll get ignored and control will pass through to
>> > .aws/credentials
>> >
>> > We used "n/a"
>> >
>> > It's kind of annoying that the s3 connection is a) required, and b)
>> poorly
>> > supported as an env var.
>> >
>> > On Tue, May 24, 2016 at 8:37 AM Tyrone Hinderson > >
>> > wrote:
>> >
>> > > I was logging to S3 in 1.7.0, but now I need to create an S3
>> "Connection"
>> > > in airflow (for remote_log_conn_id) to keep doing that in 1.7.1.2.
>> Rather
>> > > than set this "S3" connection in the UI, I'd like set a AIRFLOW_CONN_S3
>> > env
>> > > variable. What does an airlfow-friendly s3 "connection string" look
>> like?
>> > >
>> >
>>

Re: Upcoming Podling report, due July 1st

[Jakob Homan]

A lot of those tasks are specific to the Champion.  Also it's part of
the Incubator infrastucture, which generally only mentors and champion
have access to.

On 31 May 2016 at 15:39, siddharth anand <san...@apache.org> wrote:
> Any takers?
>
> -s
>
> On Tue, May 31, 2016 at 10:32 AM, Hitesh Shah <hit...@apache.org> wrote:
>
>> Additionally, it would be good if someone from the community can keep
>> http://incubator.apache.org/projects/airflow.html updated for important
>> events such as new committers, infra setup, code migration, etc.
>>
>>
>> — Hitesh
>>
>> > On May 27, 2016, at 5:35 PM, siddharth anand <san...@apache.org> wrote:
>> >
>> > Folks,
>> > I have updated the June Podling report for Airflow
>> > https://wiki.apache.org/incubator/June2016
>> >
>> > Please review and share any thoughts before June 1.
>> > -s
>> >
>> > On Wed, May 25, 2016 at 7:09 PM, siddharth anand <r39...@gmail.com>
>> wrote:
>> >
>> >> I can.
>> >>
>> >> Sent from Sid's iPhone
>> >>
>> >>> On May 25, 2016, at 7:18 PM, Jakob Homan <jgho...@gmail.com> wrote:
>> >>>
>> >>> Just a reminder that the second Podling report is imminent.  After
>> >>> three monthly reports (this is our second), we move to quarterly
>> >>> reporting.  Anyone want to volunteer?
>> >>>
>> >>> http://wiki.apache.org/incubator/June2016
>> >>>
>> >>> -Jakob
>> >>
>>
>>

Upcoming Podling report, due July 1st

[Jakob Homan]

Just a reminder that the second Podling report is imminent.  After
three monthly reports (this is our second), we move to quarterly
reporting.  Anyone want to volunteer?

http://wiki.apache.org/incubator/June2016

-Jakob

Re: [DISCUSS] Review-then-commit (RTC) vs Commit-then-review (CTR)

[Jakob Homan]

It's up to each community to decide what it's comfortable with, and
then make sure the process is applied equally to everyone.  In my
experience, 'non-minor, trivial' is pretty nebulous.  Does a
3,000-line IDE-driven refactor count as non-minor?  What about a one
line change in the heart of the scheduler that inadvertently changes a
!= to a ==?

It's generally agreed that immediate fixes for broken builds (e.g.,
missing seimicolons that break the compiler, or the Python
equivalent), can be checked in without reviews (they're usually
attached to the original ticket).

-jakob

On 20 May 2016 at 09:23, siddharth anand <r39...@gmail.com> wrote:
> Can we suggest a +1 vote for non-minor/non-trivial commits, or do all
> commits require a +1? Or is the "non-minor, non-trivial" wording too
> nebulous. t feels like this would work better the more committers we have
> given our different daily work schedules.
>
> -s
>
> On Thu, May 19, 2016 at 5:56 PM, Jakob Homan <jgho...@gmail.com> wrote:
>
>> On 18 May 2016 at 16:06, siddharth anand <r39...@gmail.com> wrote:
>> > Is there an automation we can use to ensure/enforce that PRs
>> > and JIRAs receive the +1 binding (from committers) votes needed before a
>> > merge can proceed
>>
>> Not that I know of.  Since we're using github PRs, there might be some
>> kind of workflow system that plugs into that, but I'm not aware of it.
>> Honor code works well, and reverting commits work when it doesn't...
>>

Re: [DISCUSS] Review-then-commit (RTC) vs Commit-then-review (CTR)

[Jakob Homan]

Hey-
   The consensus approval link is referring to votes (e.g. releases,
new committers, new PMCers, etc) rather than code changes.  A single
+1 is standard for code changes, as Chris describes.  However,
projects are welcome to provide more fine-grained requirements as
well.  For example, Hadoop is RTC (Review-then-commit) but also
features to be developed in branches that run on CTR
(Commit-then-review).  Such a branch can only be merged back to master
with three binding +1s, as a counterweight to the possibility of
un-reviewed changes appearing.

  Historically, ASF operated on a CTR model.  However, Hadoop and the
myriad projects that flowed from it, adopted RTC.  I'm not aware of
any big-data/Hadoop/Kafka projects that currently use CTR.  (This
comment's not an endorsement of either approach, just an observation)

   Additionally, the community can chose any time to change its
approach (via a vote).
-Jakob


On 18 May 2016 at 10:55, Chris Riccomini  wrote:
> Hey Sid,
>
>> In the RTC case, we need 3 +1 binding (a.k.a. committer) votes
>
> This sounds very high. Usually one +1 (other than the person sending the)
> is normal in an RTC scenario.
>
>> In the CTR case, we may want a separate develop branch against which to
> run integration tests and merge to master only after those tests pass
>
> I would prefer not to have a separate branch, so if we feel like that's a
> requirement for CTR, then I'd prefer RTC. If we're comfortable committing
> to master, though, then I'm fine either way. We have quite a few active
> committers, so waiting 24h for a review seems fine.
>
> Basically, I don't have a strong preference either way, except that I
> strongly prefer not to have a separate development branch. If you force me
> to pick, I'd pick RTC. I find that RTC encourages a shared understanding of
> the code, which is useful.
>
> Cheers,
> Chris
>
> On Tue, May 17, 2016 at 8:10 PM, siddharth anand  wrote:
>
>> Folks,
>> It's a good time for us to discuss whether we will adopt a RTC or CTR
>> policy towards software changes.
>>
>> In the RTC case, we need 3 +1 binding (a.k.a. committer) votes and 0
>> binding vetos for any change as RTC requires consensus approval:
>>
>>- http://www.apache.org/foundation/glossary.html#ReviewThenCommit
>>- http://www.apache.org/foundation/glossary.html#ConsensusApproval
>>
>> In the CTR case, we operate by lazy consensus, which many of the committers
>> have already exercised for the recent Committer/PPMC vote. In the CTR case,
>> we may want a separate develop branch against which to run integration
>> tests and merge to master only after those tests pass. I'm curious about
>> this approach and would like to understand how well this is supported via
>> infra tools, automation, and documentation.
>>
>>- http://www.apache.org/foundation/glossary.html#CommitThenReview
>>- http://www.apache.org/foundation/glossary.html#LazyConsensus
>>
>> I'm also curious if we need to strictly adopt one of these or whether we
>> can roll our own - e.g. a +1 binding for RTC for example.
>>
>> Mentors,
>> Any guidance here?
>>
>> -s
>>