Re: Podling Report Reminder - January 2019

[Sid Anand]

Jim,
Since our graduation vote passed in the IPMC, kindly let us know if we need
to do anything to be removed from this reminder list.

Also, are there are any assets that are "incubator-only" that we need to be
aware of and phase out? For example, our dev lists is pinned to
airflow.incubator.apache.org. I'm assuming commits@ is similar. When does
this domain expire and when/how do we move everyone over the
airflow.apache.org domains?

-s

On Fri, Dec 21, 2018 at 4:47 PM  wrote:

> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 16 January 2019, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, January 02).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Candidate names should not be made public before people are actually
> elected, so please do not include the names of potential committers or
> PPMC members in your report.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
> *   How does the podling rate their own maturity.
>
> This should be appended to the Incubator Wiki page at:
>
> https://wiki.apache.org/incubator/January2019
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC
>

Re: Incubation questionnaire

[Sid Anand]

Thanks for this link - I've shared my thoughts. Others should also feel
free to contribute. I found the questionnaire to very useful -- i.e. raises
the right questions.

-s

On Fri, Dec 21, 2018 at 5:17 PM Justin Mclean  wrote:

> Hi,
>
> Given you've just graduated, congratulations are in order and there's no
> need to fill the podling report that you just got sent a reminder about.
>
> However if you could take the time file to fill out this short
> questionnaire on your incubator journey and how you think the incubating
> process could be improved it would be greatly appreciated.
>
> https://goo.gl/forms/kDZt3bhCAxj2gfH63
>
> It's your choice if you want to fill this in or not and there are no
> compulsory questions, all feedback is anonymous unless you wish to identify
> yourself. Feedback on the form is also welcome.
>
> Any feedback received will only be shared with the Incubator PMC and be
> used to make future podlings path through the incubator easier.
>
> Thanks,
> Justin
> (V.P. Incubator)
>

Re: Graduation resolution passed - Airflow is a TLP

[Sid Anand]

YaaY!

-s

On Thu, Dec 20, 2018 at 1:13 PM Jakob Homan  wrote:

> Hey all-
>The Board minutes haven't been published yet (probably due to
> Holiday-related slowness), but I can see through the admin tool that
> our Graduation resolution was approved yesterday at the meeting.
> Airflow is the 199th current active Top Level Project in Apache.
>
> Congrats all.
>
> -Jakob
>

Re: Requesting Wiki edit permission to create AIPs

[Sid Anand]

You should be able to see your names here:
https://www.dropbox.com/s/3zn869uvrg696dc/Screenshot%202018-12-20%2011.33.38.png?dl=0

-s

On Thu, Dec 20, 2018 at 11:31 AM Sid Anand  wrote:

> Tim : Added
> Kaxil, Tao, .. and everyone .. you are all space admins and have been
> since we added you.
>
>
> https://www.dropbox.com/s/w7tbks0eciof9c5/Screenshot%202018-12-20%2011.30.59.png?dl=0
>
> -s
>
>
> On Thu, Dec 20, 2018 at 11:24 AM Sid Anand  wrote:
>
>> one sec. let me grant access to all those that need it.. stay tuned.
>> -s
>>
>> On Thu, Dec 20, 2018 at 8:41 AM Kaxil Naik  wrote:
>>
>>> Sid would probably need to grant access.
>>>
>>>
>>>
>>> On Thu, Dec 20, 2018, 05:28 Tao Feng >>
>>>> I am not an Airflow wiki admin. Could any of the admin for wiki help
>>>> this
>>>> request?
>>>>
>>>> On Wed, Dec 19, 2018 at 5:58 PM Tim Swast 
>>>> wrote:
>>>>
>>>> > Wiki ID: tswast (https://cwiki.apache.org/confluence/display/~tswast)
>>>> >
>>>> > Per
>>>> >
>>>> >
>>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
>>>> >
>>>> > *  •  **Tim Swast*
>>>> > *  •  *Software Friendliness Engineer
>>>> > *  •  *Google Cloud Developer Relations
>>>> > *  •  *Seattle, WA, USA
>>>> >
>>>>
>>>

Re: Requesting Wiki edit permission to create AIPs

[Sid Anand]

one sec. let me grant access to all those that need it.. stay tuned.
-s

On Thu, Dec 20, 2018 at 8:41 AM Kaxil Naik  wrote:

> Sid would probably need to grant access.
>
>
>
> On Thu, Dec 20, 2018, 05:28 Tao Feng 
>> I am not an Airflow wiki admin. Could any of the admin for wiki help this
>> request?
>>
>> On Wed, Dec 19, 2018 at 5:58 PM Tim Swast 
>> wrote:
>>
>> > Wiki ID: tswast (https://cwiki.apache.org/confluence/display/~tswast)
>> >
>> > Per
>> >
>> >
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
>> >
>> > *  •  **Tim Swast*
>> > *  •  *Software Friendliness Engineer
>> > *  •  *Google Cloud Developer Relations
>> > *  •  *Seattle, WA, USA
>> >
>>
>

Re: Requesting Wiki edit permission to create AIPs

[Sid Anand]

Tim : Added
Kaxil, Tao, .. and everyone .. you are all space admins and have been since
we added you.

https://www.dropbox.com/s/w7tbks0eciof9c5/Screenshot%202018-12-20%2011.30.59.png?dl=0

-s


On Thu, Dec 20, 2018 at 11:24 AM Sid Anand  wrote:

> one sec. let me grant access to all those that need it.. stay tuned.
> -s
>
> On Thu, Dec 20, 2018 at 8:41 AM Kaxil Naik  wrote:
>
>> Sid would probably need to grant access.
>>
>>
>>
>> On Thu, Dec 20, 2018, 05:28 Tao Feng >
>>> I am not an Airflow wiki admin. Could any of the admin for wiki help this
>>> request?
>>>
>>> On Wed, Dec 19, 2018 at 5:58 PM Tim Swast 
>>> wrote:
>>>
>>> > Wiki ID: tswast (https://cwiki.apache.org/confluence/display/~tswast)
>>> >
>>> > Per
>>> >
>>> >
>>> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
>>> >
>>> > *  •  **Tim Swast*
>>> > *  •  *Software Friendliness Engineer
>>> > *  •  *Google Cloud Developer Relations
>>> > *  •  *Seattle, WA, USA
>>> >
>>>
>>

Re: [RESULT] Graduate Apache Airflow as a TLP

[Sid Anand]

Hi Folks!
I just returned from a 2-week speaking tour of Australia -- such a
beautiful country! Anyway, what wonderful news to return to!

Great work team!
-s

On Sun, Dec 9, 2018 at 11:06 AM Bolke de Bruin  wrote:

> Woohoo :-)
>
> Thanks Jakob!
>
>
> Op zo 9 dec. 2018 20:01 schreef Jakob Homan 
> > The VOTE passed in Incubator.  I'll add it the resolution to the Board
> > agenda, who meet this month on the 19th.  They'll approve and Airflow
> > will be a TLP.
> >
> > -Jakob
> > On Wed, Dec 5, 2018 at 3:31 PM Jakob Homan  wrote:
> > >
> > > Been traveling today.  Just started the VOTE over in the general@ in
> > Incubator.
> > >
> > > -jg
> > > On Wed, Dec 5, 2018 at 1:26 PM Kaxil Naik  wrote:
> > > >
> > > > Hi Jakob,
> > > >
> > > > Did you raise this with IPMC, can we track it somewhere?
> > > >
> > > > Excited for graduation :-)
> > > >
> > > > Regards,
> > > > Kaxil
> > > >
> > > > On Tue, Dec 4, 2018, 20:04 Ash Berlin-Taylor  > > >>
> > > >> Missed my vote off that list :)
> > > >>
> > > >> > On 4 Dec 2018, at 18:17, Jakob Homan  wrote:
> > > >> >
> > > >> > I neglected to add my binding +1, so I'll do so now.
> > > >> >
> > > >> > With three days having elapsed, the VOTE is concluded
> successfully.
> > > >> >
> > > >> > Overall: 20 x +1 votes, 0 x -1 votes
> > > >> >
> > > >> > Binding +1 x 10: Kaxil, Tao, Bolke, Fokko, Maxime, Arthur, Hitesh,
> > > >> > Chris, Sid, Jakob.
> > > >> > Non-binding +1 x 10: Daniel, Shah, Stefan, Kevin, Marc, Sunil,
> > > >> > Adityan, Deng, Neelesh, Sai
> > > >> >
> > > >> > I'll use this result to start the corresponding VOTE on the IPMC.
> > I'm
> > > >> > at an offsite today, so I have limited email time.  Likely will
> open
> > > >> > the VOTE this evening.
> > > >> >
> > > >> > Thanks everyone.
> > > >> > -Jakob
> > > >> >
> > > >> >
> > > >> > On Tue, Dec 4, 2018 at 6:03 AM Bolke de Bruin 
> > wrote:
> > > >> >>
> > > >> >> Shall we close the vote? @jakob?
> > > >> >>
> > > >> >>> On 2 Dec 2018, at 13:08, Sid Anand  wrote:
> > > >> >>>
> > > >> >>> +1 binding
> > > >> >>>
> > > >> >>> Woot! Thanks to all for this happy day!
> > > >> >>> -s
> > > >> >>>
> > > >> >>> On Sun, Dec 2, 2018 at 1:25 AM Sai Phanindhra <
> > phani8...@gmail.com> wrote:
> > > >> >>>
> > > >> >>>> +1 (non binding)
> > > >> >>>>
> > > >> >>>> Excited to see this happenning.
> > > >> >>>>
> > > >> >>>> On Sat 1 Dec, 2018, 20:35  > > >> >>>>
> > > >> >>>>> +1 (binding)!
> > > >> >>>>>
> > > >> >>>>> On 30 November 2018 21:33:14 GMT, Jakob Homan <
> > jgho...@gmail.com> wrote:
> > > >> >>>>>> Hey all!
> > > >> >>>>>>
> > > >> >>>>>> Following a very successful DISCUSS[1] regarding graduating
> > Airflow to
> > > >> >>>>>> Top Level Project (TLP) status, I'm starting the official
> VOTE.
> > > >> >>>>>>
> > > >> >>>>>> Since entering the Incubator in 2016, the community has:
> > > >> >>>>>> * successfully produced 7 releases
> > > >> >>>>>> * added 9 new committers/PPMC members
> > > >> >>>>>> * built a diverse group of committers from multiple different
> > employers
> > > >> >>>>>> * had more than 3,300 JIRA tickets opened
> > > >> >>>>>> * completed the project maturity model with positive
> > responses[2]
> > > >> >>>>>>
> > > >> >>>>>> Accordingly, I believe we're ready to graduate and am calling
> > a VOTE
> > 

Airflow Down Under

[Sid Anand]

I met Gerard Curiel in Melbourne earlier today and we talked about starting
an Airflow community / meetup in Australia (perhaps Melbourne to start
with). As an informal survey, do respond if you are an Ozzie on this list!

This will inform the amount of effort needed to build a local community
here. Per my recent YOW!  talk, there were
about 10 members of the audience who were somewhat familiar with Apache
Airflow.

Cheers!
-s

Airflow in Australia?

[Sid Anand]

Folks!
I should have mentioned this earlier, but I'm half-way through touring
Australia for the YOW! conference 
series (Sydney,
Brisbane, Melbourne). I'm in Brisbane till Wed and then in Melbourne till
Saturday.Happy to meet up for a beer if you'd like to talk Airflow over a
beer, ping me.

Warm regards!
-s

Re: [VOTE] Graduate the Apache Airflow as a TLP

[Sid Anand]

+1 binding

Woot! Thanks to all for this happy day!
-s

On Sun, Dec 2, 2018 at 1:25 AM Sai Phanindhra  wrote:

> +1 (non binding)
>
> Excited to see this happenning.
>
> On Sat 1 Dec, 2018, 20:35 
> > +1 (binding)!
> >
> > On 30 November 2018 21:33:14 GMT, Jakob Homan  wrote:
> > >Hey all!
> > >
> > >Following a very successful DISCUSS[1] regarding graduating Airflow to
> > >Top Level Project (TLP) status, I'm starting the official VOTE.
> > >
> > >Since entering the Incubator in 2016, the community has:
> > >   * successfully produced 7 releases
> > >   * added 9 new committers/PPMC members
> > >* built a diverse group of committers from multiple different employers
> > >   * had more than 3,300 JIRA tickets opened
> > >   * completed the project maturity model with positive responses[2]
> > >
> > >Accordingly, I believe we're ready to graduate and am calling a VOTE
> > >on the following graduation resolution.  This VOTE will remain open
> > >for at least 72 hours.  If successful, the resolution will be
> > >forwarded to the IPMC for its consideration.  If that VOTE is
> > >successful, the resolution will be voted upon by the Board at its next
> > >monthly meeting.
> > >
> > >Everyone is encouraged to vote, even if their vote is not binding.
> > >We've built a nice community here, let's make sure everyone has their
> > >voice heard.
> > >
> > >Thanks,
> > >Jakob
> > >
> > >[1]
> > >
> >
> https://lists.apache.org/thread.html/%3c0a763b0b-7d0d-4353-979a-ac6769eb0...@gmail.com%3E
> > >[2]
> > >https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
> > >
> > >
> > >
> > >Establish the Apache Airflow Project
> > >
> > >WHEREAS, the Board of Directors deems it to be in the best
> > >interests of the Foundation and consistent with the
> > >Foundation's purpose to establish a Project Management
> > >Committee charged with the creation and maintenance of
> > >open-source software, for distribution at no charge to
> > >the public, related to workflow automation and scheduling
> > >that can be used to author and manage data pipelines.
> > >
> > >NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > >Committee (PMC), to be known as the "Apache Airflow Project",
> > >be and hereby is established pursuant to Bylaws of the
> > >Foundation; and be it further
> > >
> > >RESOLVED, that the Apache Airflow Project be and hereby is
> > >responsible for the creation and maintenance of software
> > >related to workflow automation and scheduling that can be
> > >used to author and manage data pipelines; and be it further
> > >
> > >RESOLVED, that the office of "Vice President, Apache Airflow" be
> > >and hereby is created, the person holding such office to
> > >serve at the direction of the Board of Directors as the chair
> > >of the Apache Airflow Project, and to have primary responsibility
> > >for management of the projects within the scope of
> > >responsibility of the Apache Airflow Project; and be it further
> > >
> > >RESOLVED, that the persons listed immediately below be and
> > >hereby are appointed to serve as the initial members of the
> > >Apache Airflow Project:
> > >
> > >* Alex Guziel 
> > >* Alex Van Boxel 
> > >* Arthur Wiedmer 
> > >* Ash Berlin-Taylor 
> > >* Bolke de Bruin 
> > >* Chris Riccomini 
> > >* Dan Davydov 
> > >* Fokko Driesprong 
> > >* Hitesh Shah 
> > >* Jakob Homan 
> > >* Jeremiah Lowin 
> > >* Joy Gao 
> > >* Kaxil Naik 
> > >* Maxime Beauchemin 
> > >* Siddharth Anand 
> > >* Sumit Maheshwari 
> > >* Tao Feng 
> > >
> > >NOW, THEREFORE, BE IT FURTHER RESOLVED, that Bolke de Bruin
> > >be appointed to the office of Vice President, Apache Airflow, to
> > >serve in accordance with and subject to the direction of the
> > >Board of Directors and the Bylaws of the Foundation until
> > >death, resignation, retirement, removal or disqualification,
> > >or until a successor is appointed; and be it further
> > >
> > >RESOLVED, that the initial Apache Airflow PMC be and hereby is
> > >tasked with the creation of a set of bylaws intended to
> > >encourage open development and increased participation in the
> > >Apache Airflow Project; and be it further
> > >
> > >RESOLVED, that the Apache Airflow Project be and hereby
> > >is tasked with the migration and rationalization of the Apache
> > >Incubator Airflow podling; and be it further
> > >
> > >RESOLVED, that all responsibilities pertaining to the Apache
> > >Incubator Airflow podling encumbered upon the Apache Incubator
> > >Project are hereafter discharged.
> >
>

Re: [DISCUSS] Apache Airflow graduation from the incubator

[Sid Anand]

Great!

-s

On Sun, Dec 2, 2018 at 5:53 AM Ash Berlin-Taylor  wrote:

> I've created two tickets to add QU30 and CO50 to our docs.
>
> (I think even if we use sec@a.o we should still add something to our docs
> saying how to do it)
>
> https://issues.apache.org/jira/browse/AIRFLOW-3430 -- Fokko: did you get
> anywhere on this?
> https://issues.apache.org/jira/browse/AIRFLOW-3431 -- I'll make a start
> on this
>
> -ash
>
> > On 30 Nov 2018, at 22:06, Bolke de Bruin  wrote:
> >
> > Thanks Jakob!
> >
> > Verstuurd vanaf mijn iPad
> >
> >> Op 30 nov. 2018 om 22:49 heeft Jakob Homan  het
> volgende geschreven:
> >>
> >> I've finished the paperwork.  I don't seem to have karma to trigger
> >> the build on Jenkins, so we'll just wait for the daily rebuild.  With
> >> that, I've opened the VOTE thread as well.  Thanks everybody.
> >>> On Wed, Nov 28, 2018 at 5:08 PM Jakob Homan  wrote:
> >>>
> >>> I'll finish up the template at
> >>> http://incubator.apache.org/projects/airflow.html tomorrow or Friday
> >>> (I *think* you have to be an IPMC member to update it since it lives
> >>> in the Incubator SVN).  Looks like there's no actual work to do, just
> >>> marking stuff that has been done but not yet recorded, and verifying
> >>> some licenses.
> >>>
> >>> -Jakob
> >>>
> >>>
> >>>
>  On Wed, Nov 28, 2018 at 2:48 PM Tao Feng  wrote:
> 
>  Sorry, just saw Kaxil's latest email. Kaxil, is there anything else I
> could
>  help with?
> 
>  Thanks,
>  -Tao
> 
> > On Wed, Nov 28, 2018 at 2:40 PM Tao Feng 
> wrote:
> >
> > I would like to help on the documentation. Let me take a look at it.
> I
> > will work Kaxil on that.
> >
> >> On Tue, Nov 27, 2018 at 12:39 PM Bolke de Bruin 
> wrote:
> >>
> >> Hi Folks,
> >>
> >> Thanks all for your responses and particularly Stefan for his
> suggestion
> >> to use the generic Apache way to handle security issues. This seems
> to be
> >> an accepted way for more projects, so I have added this to the
> maturity
> >> evaluation[1] and marked is as resolved. While handling the GPL
> library can
> >> be nicer we are already in compliance with CD30, so @Fokko and @Ash
> if you
> >> want to help out towards graduation please spend your time
> elsewhere like
> >> fixing CO50. This means adding a page to confluence that describes
> how to
> >> become a committer on the project. As we are following Apache many
> examples
> >> of other projects are around[2]
> >>
> >> Then there is the paperwork[3] as referred to by Jakob. This mainly
> >> concerns filling in some items, maybe here and there creation some
> >> documentation but I don't think much. @Kaxil, @Tao: are you willing
> to pick
> >> this up? @Sid can you share how to edit that page?
> >>
> >> If we have resolved these items in my opinion we can start the
> voting
> >> here and at the IPMC thereafter, targeting the board meeting of
> January for
> >> graduation. How’s that for a New Year’s resolution?
> >>
> >> Cheers!
> >> Bolke
> >>
> >> P.S. Would it be nice to have updated graduation web page? Maybe
> one of
> >> the contributors/community members likes to take a stab at this[4]
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation <
> >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation>
> >> [2]
> https://cwiki.apache.org/confluence/display/HAWQ/Becoming+a+committer
> >> <
> https://cwiki.apache.org/confluence/display/HAWQ/Becoming+a+committer>
> >> [3] http://incubator.apache.org/projects/airflow.html <
> >> http://incubator.apache.org/projects/airflow.html>
> >> [4] https://airflow.apache.org/ 
> >>
> >>
> >>
> >>> On 27 Nov 2018, at 16:32, Driesprong, Fokko 
> >> wrote:
> >>>
> >>> +1 from my side. Would be awesome to graduate Airflow
> >>>
> >>> If time allows, I'll also dive into CD30.
> >>>
> >>> Cheers, Fokko
> >>>
> >>> Op di 27 nov. 2018 om 16:21 schreef Ash Berlin-Taylor <
> a...@apache.org>:
> >>>
>  Oarsome Bolke, thanks for starting this.
> 
>  It looks like we are closer than I thought!
> 
>  We can use those security lists (though having our own would be
> nice) -
>  either way we will need to make this prominent in the docs.
> 
>  Couple of points
> 
>  CS10: that github link is only visible to members of the team
> 
>  CD30: probably good as it is, we may want to do
>  https://issues.apache.org/jira/browse/AIRFLOW-3400 <
>  https://issues.apache.org/jira/browse/AIRFLOW-3400> to remove
> the last
>  niggle of the GPL env var at install time (but not a hard
> requirement,
> >> just
>  nice)
> 
>  -ash
> 
> > On 26 

Re: Airflow 1.10.1 is released

[Sid Anand]

Excellent work Ash! Thanks for doing the needful!!

-s

On Wed, Nov 21, 2018 at 5:40 PM Tao Feng  wrote:

> Thanks Ash for running the release!
>
> On Wed, Nov 21, 2018 at 2:20 PM Ash Berlin-Taylor  wrote:
>
> > Dear Airflow community,
> >
> > I'm happy to announce that Airflow 1.10.1 was just released.
> >
> > The source release as well as the binary "sdist" release are available
> > here:
> >
> >
> >
> https://dist.apache.org/repos/dist/release/incubator/airflow/1.10.1-incubating/
> >
> > We also made this version available on PyPi for convenience (`pip install
> > apache-airflow`):
> >
> > https://pypi.python.org/pypi/apache-airflow
> >
> > Find the CHANGELOG here for more details:
> >
> > https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt
>

Re: Request for Permission to Create new AIP

[Sid Anand]

Done
-s

On Wed, Oct 17, 2018 at 7:56 AM Ian Davison 
wrote:

> Just requesting for my Wiki ID ianjdavison to get permission to create AIP
> on the Confluence.
>
> Thanks,
> Ian
> The content of this e-mail message and any attached files transmitted with
> it are to be treated as confidential information and are intended solely
> for the use of the individual or entity to whom they are addressed. If the
> reader of this e-mail is not the intended recipient or his or her
> authorized agent, the reader is hereby notified that any reproduction,
> distribution, or disclosure of this e-mail is prohibited. If you have
> received this e-mail in error, please notify the sender by replying to this
> message and delete this e-mail immediately.***1010data, Inc. and its
> affiliates ("1010data") are not responsible for any advice concerning the
> use of its software or services or the data manipulated by its software,
> except to the extent 1010data has specifically undertaken such
> responsibility in a validly binding contract. 1010data shall have no
> responsibility for any decision concerning the appropriate method of use or
> application of its software or services or the data therein in connection
> with any transaction. Any decision concerning how, where and when to use
> this facility remains the sole responsibility of the user. NO AGREEMENTS
> MAY BE ENTERED INTO WITH ANY 1010DATA COMPANY VIA EMAIL EXCHANGE REGARDLESS
> OF THE TITLE OF THE AUTHOR OF THE EMAILS. ALL AGREEMENTS MUST BE IN THE
> FORM OF A FORMAL WRITTEN AGREEMENT SIGNED BY THE CHIEF EXECUTIVE OFFICER,
> PRESIDENT, CHIEF OPERATING OFFICER, CHIEF FINANCIAL OFFICER OR CHIEF PEOPLE
> OFFICER OF 1010DATA AND AN AUTHORIZED REPRESENTATIVE OF THE OTHER PARTY.
> Just requesting for my Wiki ID ianjdavison to get permission to create AIP
> on the Confluence.
>
> Thanks,
> Ian
> The content of this e-mail message and any attached files transmitted with
> it are to be treated as confidential information and are intended solely
> for the use of the individual or entity to whom they are addressed. If the
> reader of this e-mail is not the intended recipient or his or her
> authorized agent, the reader is hereby notified that any reproduction,
> distribution, or disclosure of this e-mail is prohibited. If you have
> received this e-mail in error, please notify the sender by replying to this
> message and delete this e-mail immediately.***1010data, Inc. and its
> affiliates ("1010data") are not responsible for any advice concerning the
> use of its software or services or the data manipulated by its software,
> except to the extent 1010data has specifically undertaken such
> responsibility in a validly binding contract. 1010data shall have no
> responsibility for any decision concerning the appropriate method of use or
> application of its software or services or the data therein in connection
> with any transaction. Any decision concerning how, where and when to use
> this facility remains the sole responsibility of the user. NO AGREEMENTS
> MAY BE ENTERED INTO WITH ANY 1010DATA COMPANY VIA EMAIL EXCHANGE REGARDLESS
> OF THE TITLE OF THE AUTHOR OF THE EMAILS. ALL AGREEMENTS MUST BE IN THE
> FORM OF A FORMAL WRITTEN AGREEMENT SIGNED BY THE CHIEF EXECUTIVE OFFICER,
> PRESIDENT, CHIEF OPERATING OFFICER, CHIEF FINANCIAL OFFICER OR CHIEF PEOPLE
> OFFICER OF 1010DATA AND AN AUTHORIZED REPRESENTATIVE OF THE OTHER PARTY.
>

October Podling Report Updated

[Sid Anand]

Mentors, kindly sign-off!

https://wiki.apache.org/incubator/October2018#preview

Re: Airflow 1.10 Migration Duration

[Sid Anand]

I checked with our Ops guy and he mentioned that when he upgraded from
1.8.x to 1.9.x, it took a few seconds. We had 3M rows in the task_instance
table and run MySQL 5.7.

-s

On Tue, Sep 25, 2018 at 4:54 PM Matt Davis  wrote:

> Hi folks,
>
> Here at Clover we're excitedly migrating to Airflow 1.10 (thanks for
> everyone's hard work on that!). We're finding that it's taking about 2
> hours to apply all the migrations to go from Airflow 1.8 to 1.10, largely
> driven by the 10 million rows in our task_instance table. That got us
> wondering what kind of maintenance people do on their Airflow metadata
> databases. Do folks mostly put up with long migrations and generally longer
> queries, or are y'all doing periodic cleanups of your metadata DB to keep
> it fairly light?
>
> Thanks,
> Matt Davis
>

Re: It's very hard to become a committer on the project

[Sid Anand]

I will chime in here as well and essentially parrot Fokko and Ash’s
sentiments.

If your driving goal is to become a committer on an Apache project, then
OSS and Apache may not be for you. Committers are contributors,
contributors are users. So, your journey starts by using Apache Airflow to
solve a problem that you have. As you use it, you may find it doesn’t work
as you’d wish or has documentation gaps. That is when, from an informed
perspective, you’d start discussions on the mailing lists and slack, open
JIras, and submit PRs. After making contributions, you may become a
committer. We have 20 committers today and that was the journey each of us
went through, similar to the journey that others go through on other Apache
projects.

Now, as much as we can leverage the community, we do. So, we love it when
users and contributors comment on JIras, update the wiki, propose changes,
and vet each other’s PRs. And we are always looking to scale out and add
more commuters.
-s


On Thu, Sep 20, 2018 at 1:02 PM Driesprong, Fokko 
wrote:

> Some history. Recently we've moved the to Gitbox
> <https://gitbox.apache.org/> and moved away from the Apache repo itself to
> simplify the setup. Before that, the code on Github was merely a mirror of
> the Apache git repo. Before Gitbox there was a Python script that would
> like the issues to the Github PR's. Now because we've moved to Github
> itself, this piece of automation is gone and we need to get this working
> again. Also for projects like Spark there is a lot of automation going on
> between Jira and Github. Unfortunately I don't have a lot of experience
> here, but I suspect a whole bunch of hooks. For my perspective, automation
> is key here.
>
> Furthermore, as Ash is saying. It is hard for Airflow to pick up any
> arbitrary task from Jira since the usage of Airflow ranges form a wide
> range of applications, from AWS to GCP, from MySQL to Druid, from Bash to
> Slack. My advise would be the same as Ash; start using Airflow, and if you
> run into anything, raise a ticket (first check if it is already there), and
> open a PR. Like your saying; you're pretty new to Airflow, it would be best
> to get some experience first.
>
> The community (committers, contributors and users) is happy to help and
> assist, like the recent conversations on Slack. But nobody is getting paid
> to work on Airflow, and this isn't a problem, but this means that most of
> us still have a full time job on the side :) Tomorrow I'll spend time to
> clean up the old Jira's.
>
> Cheers, Fokko
>
>
> Op do 20 sep. 2018 om 11:40 schreef airflowuser
> :
>
> > >> Are you volunteering to sponsor someone's time to be able to do this?
> >
> > I am. But I have no knowledge of all the components of the project.
> > I doubt I can be much of value for this tagging task.
> >
> >
> > Sent with ProtonMail Secure Email.
> >
> > ‐‐‐ Original Message ‐‐‐
> > On Thursday, September 20, 2018 12:37 PM, Ash Berlin-Taylor <
> > a...@apache.org> wrote:
> >
> > > > Remember my basic question: I want to contribute - how on earth I can
> > find a ticket that is suitable for first time committer? Can you show me?
> > >
> > > There aren't that many feature requests in Jira, so looking there for
> > easy tickets, is as you have probably found a fruitless exercise. I'd
> > recommend using Airflow and when you come across something you want
> fixed,
> > or a feature you want added, that you open a PR for it.
> > >
> > > > Again, If decided to stay with Jira.. I highly recommend that someone
> > from the project will maintain it. Don't allow to regular users to tag
> and
> > set priorities for the tickets.. someone from the project should do it.
> > >
> > > Are you volunteering to sponsor someone's time to be able to do this?
> > >
> > > > Sent with ProtonMail Secure Email.
> > > > ‐‐‐ Original Message ‐‐‐
> > > > On Tuesday, September 18, 2018 11:57 AM, Sid Anand san...@apache.org
> > wrote:
> > > >
> > > > > Hi Folks!
> > > > > For some history, Airlfow started on GH issues. We also had a very
> > popular Google group. When we moved to Apache, we were told that Jira was
> > the way we needed to go for issue tracking because it resided on Apache
> > infrastructure. When we moved over, we had to drop 100+ GH issues on the
> > floor -- there was no way to transfer them to Jira and maintain the
> > original submitter/owner info since there was no mapping of users between
> > the 2 systems.
> > > > > Here's a pie chart of our existing issues by status:
> > > > 

Airflow: Apache Graduation

[Sid Anand]

Folks! (specifically Bolke, Fokko, Ash)
What's needed to graduate from Apache?

Can we make 1.10.1 be about meeting our licensing needs to allow us to
graduate?

-s

Re: Podling Report Reminder - October 2018

[Sid Anand]

Justin,
Done!
https://wiki.apache.org/incubator/October2018#preview

I'll raise the graduation topic.

Jakob,
Please sign off. I'm really hoping that we don't end up going back toward
monthly status reports..


-s

On Thu, Sep 20, 2018 at 1:20 AM Justin Mclean  wrote:

> Hi,
>
> > Please advise if we need to do the Oct report as well. Cheers!
>
> You in group 1 which normally reports for Jan,Apr,Jul,Oct so IMO you
> should report.
>
> You can of course choose not to report and report next month. Given the
> Feb 2018 report said nearing graduation, why don’t you try and push for
> that and that way you don’t need to report anymore. I also note you been a
> long time (coming up to 3 years?) in the Incubator and made 5 or so
> releases and voted in committers.
>
> Thanks,
> Justin

Fwd: Auto-cleaning up Stale PRs

[Sid Anand]

Folks!
Probot/Stale <https://github.com/probot/stale> is now deployed and is
managing stale PRs :
https://github.com/apache/incubator-airflow/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+label%3Astale

<https://github.com/probot/stale>
Probot/stale <https://github.com/probot/stale> is enforcing our policy file
<https://github.com/apache/incubator-airflow/blob/master/.github/stale.yml>.
It simply:

   - Looks for any PRs with no activity in 90 days (via *daysUntilStale*)
  - It then comments on the PR and marks the PR stale (using the stale
  label)
  - After 7 days (via *daysUntilClose*) of being labeled as stale, p/s
  will close the PR, again commenting on it
   - The JIRA will remain open, so that someone else can pick up the work
   at some future point
  - Notably, the JIRA assignee won't change, but folks should not be
  hindered by a pre-assigned JIRA with no movement over a long
period of time
  -- they should re-assign that JIRA to themselves
   - After being closed, the stale label remains on the PRs so we can
   assess how many were closed via this route over time.

This will at least address the management portion of cold case PRs and I
hope this will help the community better focus on active PRs (those with
active conversation on them). And of course, we hope the community can help
us curate the backlog of JIRAs.

A special thanks to Ismael from the Apache Beam community for unblocking us!

I will raise a PR shortly to drop the daysUntilStale to 45 or 60 days
-s

-- Forwarded message -
From: Sid Anand 
Date: Thu, Sep 20, 2018 at 7:56 AM
Subject: Re: Auto-cleaning up Stale PRs
To: 


Will do.

-s

On Wed, Sep 19, 2018 at 8:02 PM Dave Fisher  wrote:

> Hi -
>
> No objections from me. I do ask you let the IPMC know in your podling
> report if this results in any controversy from the creators of any
> autoclosed pull requests.
>
> Regards,
> Dave
>
> Sent from my iPhone
>
> > On Sep 19, 2018, at 6:48 PM, Sid Anand  wrote:
> >
> > Thanks Greg and Ismael!
> > -s
> >
> >> On Wed, Sep 19, 2018 at 6:39 PM Greg Stein  wrote:
> >>
> >> Hello all,
> >>
> >> The confusion here was "write access to the repository" (not allowed)
> >> versus "write access to Pull Requests" (allowed). It took the Beam folks
> >> some research to determine that GitHub *does* differentiate between
> these
> >> two write capabilities (historically, GitHub has not been very granular
> >> with permissions).
> >>
> >> So. When Airflow said Probot Stale needed write access, we took that to
> >> mean *code*.
> >>
> >> After the pointer to Beam, reminding Infra of the research Beam had done
> >> (and my enabling of Stale for them) ... we realized that Stale *is*
> >> perfectly fine because it doesn't touch the code repository.
> >>
> >> Probot Stale has been enabled for Airflow.
> >>
> >> Cheers,
> >> Greg Stein
> >> Infrastructure Administrator, ASF
> >>
> >>
> >>> On Wed, Sep 19, 2018 at 7:47 PM Sid Anand  wrote:
> >>>
> >>> Ismael,
> >>> Thanks for this pointer. I've re-opened my INFRA ticket and referenced
> >> your
> >>> Apache Beam one. Super helpful.. if we get it enabled, please collect a
> >>> beer from anyone in the Apache Airflow community!
> >>>
> >>> -s
> >>>
> >>>> On Wed, Sep 19, 2018 at 7:39 AM Ismaël Mejía 
> wrote:
> >>>>
> >>>> While I agree that autoclosing PRs can be unwelcoming. I don't see
> >>>> clearly the argument of INFRA in the ticket.
> >>>>
> >>>>> The policy of no-write-access for bots is a requirement by the
> >>>> foundation legal team. We cannot allow write access to repos without
> an
> >>>> ICLA.
> >>>>
> >>>> Labeling and closing the PR in github does not imply write-access from
> >>>> the bot into the 'real' gitbox repository, so I don't see how this can
> >>>> be an issue, or are we in a gray area (in case bot automation of
> >>>> metadata can have legal issues which I doubt since this is not part of
> >>>> the source distribution).
> >>>>
> >>>> As a precedent we had Probot/Stale enabled for Apache Beam so I
> >>>> suppose that this should be possible for Airflow too.
> >>>> https://issues.apache.org/jira/browse/INFRA-16589
> >>>>
> >>>>> On Thu, Sep 13, 2018 at 5:55 PM Sid Anand  wrote:
> >>>>&

Re: Podling Report Reminder - October 2018

[Sid Anand]

Justin,
Please advise if we need to do the Oct report as well. Cheers!
-s

On Wed, Sep 19, 2018 at 5:53 PM Justin Mclean 
wrote:

> Hi,
>
> The board meeting was only a few hours ago so September wouldn't show up
> yet on that link.
>
> Thanks,
> Justin
>
> On Thu., 20 Sep. 2018, 10:43 am Sid Anand,  wrote:
>
> > Hi Justin!
> > Well, it was signed off by Jakob (Mentor):
> > https://wiki.apache.org/incubator/September2018
> >
> > Why was this missed in the board minutes?
> >
> > This is super annoying. I keep filling these out and either they are not
> > signed off or they get missed in other processes.
> >
> > In August, I filled it out but no mentors signed off :
> > https://wiki.apache.org/incubator/August2018
> > -s
> >
> >
> > On Wed, Sep 19, 2018 at 4:42 PM Justin Mclean 
> > wrote:
> >
> >> I don't see a report for September here
> >>
> >> https://whimsy.apache.org/board/minutes/Airflow.html
> >>
> >> On Thu., 20 Sep. 2018, 9:33 am Sid Anand,  wrote:
> >>
> >>> Hi Jim,
> >>> Apache Airflow just submitted this in September and it was signed off
> by
> >>> 1 mentor. Why is another one needed so soon given that we have been
> >>> incubating for more than 2 years? My understanding is that reporting
> needs
> >>> to be quarterly.
> >>>
> >>> -s
> >>>
> >>> On Wed, Sep 19, 2018 at 2:58 PM  wrote:
> >>>
> >>>> Dear podling,
> >>>>
> >>>> This email was sent by an automated system on behalf of the Apache
> >>>> Incubator PMC. It is an initial reminder to give you plenty of time to
> >>>> prepare your quarterly board report.
> >>>>
> >>>> The board meeting is scheduled for Wed, 17 October 2018, 10:30 am PDT.
> >>>> The report for your podling will form a part of the Incubator PMC
> >>>> report. The Incubator PMC requires your report to be submitted 2 weeks
> >>>> before the board meeting, to allow sufficient time for review and
> >>>> submission (Wed, October 03).
> >>>>
> >>>> Please submit your report with sufficient time to allow the Incubator
> >>>> PMC, and subsequently board members to review and digest. Again, the
> >>>> very latest you should submit your report is 2 weeks prior to the
> board
> >>>> meeting.
> >>>>
> >>>> Candidate names should not be made public before people are actually
> >>>> elected, so please do not include the names of potential committers or
> >>>> PPMC members in your report.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> The Apache Incubator PMC
> >>>>
> >>>> Submitting your Report
> >>>>
> >>>> --
> >>>>
> >>>> Your report should contain the following:
> >>>>
> >>>> *   Your project name
> >>>> *   A brief description of your project, which assumes no knowledge of
> >>>> the project or necessarily of its field
> >>>> *   A list of the three most important issues to address in the move
> >>>> towards graduation.
> >>>> *   Any issues that the Incubator PMC or ASF Board might wish/need to
> be
> >>>> aware of
> >>>> *   How has the community developed since the last report
> >>>> *   How has the project developed since the last report.
> >>>> *   How does the podling rate their own maturity.
> >>>>
> >>>> This should be appended to the Incubator Wiki page at:
> >>>>
> >>>> https://wiki.apache.org/incubator/October2018
> >>>>
> >>>> Note: This is manually populated. You may need to wait a little before
> >>>> this page is created from a template.
> >>>>
> >>>> Mentors
> >>>> ---
> >>>>
> >>>> Mentors should review reports for their project(s) and sign them off
> on
> >>>> the Incubator wiki page. Signing off reports shows that you are
> >>>> following the project - projects that are not signed may raise alarms
> >>>> for the Incubator PMC.
> >>>>
> >>>> Incubator PMC
> >>>>
> >>>
>

Fwd: Auto-cleaning up Stale PRs

[Sid Anand]

Re-opening https://issues.apache.org/jira/browse/INFRA-17005
-s

-- Forwarded message -
From: Sid Anand 
Date: Wed, Sep 19, 2018 at 5:47 PM
Subject: Re: Auto-cleaning up Stale PRs
To: 


Ismael,
Thanks for this pointer. I've re-opened my INFRA ticket and referenced your
Apache Beam one. Super helpful.. if we get it enabled, please collect a
beer from anyone in the Apache Airflow community!

-s

On Wed, Sep 19, 2018 at 7:39 AM Ismaël Mejía  wrote:

> While I agree that autoclosing PRs can be unwelcoming. I don't see
> clearly the argument of INFRA in the ticket.
>
> > The policy of no-write-access for bots is a requirement by the
> foundation legal team. We cannot allow write access to repos without an
> ICLA.
>
> Labeling and closing the PR in github does not imply write-access from
> the bot into the 'real' gitbox repository, so I don't see how this can
> be an issue, or are we in a gray area (in case bot automation of
> metadata can have legal issues which I doubt since this is not part of
> the source distribution).
>
> As a precedent we had Probot/Stale enabled for Apache Beam so I
> suppose that this should be possible for Airflow too.
> https://issues.apache.org/jira/browse/INFRA-16589
>
> On Thu, Sep 13, 2018 at 5:55 PM Sid Anand  wrote:
> >
> > Apache Airflow has, at any point, >200 PRs open. During the slower summer
> > months, we've been merging 100-200 PRs a month. We have been growing the
> > community -- we have <600 contributors, ~200 companies using it, and 20+
> > committers. A person is promoted to "Committer" in recognition for work
> > he/she has done without an expectation of future work in maintaining the
> > code base. Hence, minting new committers doesn't always translate into
> > greater bench strength where merging PRs is concerned. That said, we are
> > actively adding new committers. The last 4-5 committers we added have
> been
> > super active maintainers, so the coverage on PRs and questions has been
> > getting better.
> >
> > There are many causes of Cold-case PRs:
> >
> >1. Submitter is not actively responding
> >   1. One example is that we requested tests and they were never
> written
> >   2. Discussion ensued on the PR and the submitter did not accept the
> >   community's feedback
> >2. Committers didn't get to it in a timely manner and after a while
> the
> >engagement fell
> >
> > We are in a better position now to handle (2) -- this was not the case a
> > year ago. We're at least able to keep up with our in-flow of PRs
> > week-to-week, but are still having challenges with the
> > previously-established backlog. But, (1) is also a contributor to stale
> PRs.
> >
> > We do have a lot of stale PRs to manually handle -- I spent all of Summer
> > 2017 pinging submitters of old PRs and I find myself in the same position
> > now.
> >
> > Probot/stale is a useful tool. It has legitimate use-cases. A policy
> > reflects the health/mentality/approaches of the community. A tool like
> this
> > enforces the policy. Let's not overlook adoption of what would be a very
> > useful tool to the community due to a meta conversation about policy. I
> > think everyone on this list cares about growing a healthy and vibrant
> > community. We also care about being efficient with our spare time.  This
> > tools can help us manage both.
> >
> > Also, I am not suggesting that we close JIRA, just stale PRs. JIRAs need
> to
> > be kept open so we don't lose visibility of bugs/features/etc... This
> tool
> > doesn't handle JIRA closing anyway.
> >
> > -s
> >
> > On Thu, Sep 13, 2018 at 1:37 AM Mark Thomas  wrote:
> >
> > > On 12/09/18 19:16, Sid Anand wrote:
> > > > A stale PR is defined by a policy -- for example, 60 days without any
> > > > movement on the PR.
> > >
> > > Automatically closing such issues is not going to do anything to aid
> > > community building and is likely to actively damage such efforts.
> > >
> > > > Stale PRs would be bad experiences in general for community members,
> but
> > > > after no movement for 60 days, this is just about cleaning up PRs
> that
> > > are
> > > > not getting feedback from the committers or PR submitters.
> > >
> > > That is the wrong solution the problem.
> > >
> > > If reporters of issues are not responding to questions and there is
> > > genuinely nothing the community can do to progress the issue without
> > > their input then closing the issue is fair enough. 

Re: Podling Report Reminder - October 2018

[Sid Anand]

Hi Jim,
Apache Airflow just submitted this in September and it was signed off by 1
mentor. Why is another one needed so soon given that we have been
incubating for more than 2 years? My understanding is that reporting needs
to be quarterly.

-s

On Wed, Sep 19, 2018 at 2:58 PM  wrote:

> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 17 October 2018, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, October 03).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Candidate names should not be made public before people are actually
> elected, so please do not include the names of potential committers or
> PPMC members in your report.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
> *   How does the podling rate their own maturity.
>
> This should be appended to the Incubator Wiki page at:
>
> https://wiki.apache.org/incubator/October2018
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC
>

Apache Airflow use-cases for QCon SF?

[Sid Anand]

Hi Folks!
Along with being a committer on Airflow, I'm also a co-chair for QCon. If
you are unfamiliar with it,  QCon is global family of
practitioner-conferences with annual events in SF, NY, London, Shanghai,
Beijing, and Sao Paolo. Ours is a "for senior architect/developer by expert
practitioner" conference. We steer clear of vendor and marketing pitches --
we invite speakers who build software. Most of our speakers are invited,
though we do find some gems through our CFP process. We cater to early
adopters and share stories that highlight the challenges of being first in
a market, open sourcing a new framework/language, tackling an old problem
with a new approach, handling immense scale/growth (in traffic, company
growth, or both), etc...

QCon SF , which is Nov 5-7, has 18 tracks spread over
3 days, with 5 talks in each track. The talks are 50 minutes in length.

I'm currently running 2 data tracks:

   - https://qconsf.com/sf2018/track/emerging-trends-data-engineering
   - https://qconsf.com/sf2018/track/applied-ai-machine-learning

If you have an interesting use-case for Apache Airflow (e.g. in building ML
pipelines) and have some experience giving talks, email me directly. We
cover hotel stay at the Hyatt Regency (SF), provide a travel stipend, and
also provide most meals.

-s

Re: About 1.10 Release Timeline

[Sid Anand]

Send an email on the 1.10.1 email thread to have your PR included... Ash is
working on that release.

-s

On Tue, Sep 18, 2018 at 5:01 PM Yu, Yang  wrote:

> Hi everyone,
>
> Amazon SageMaker is working on developing SageMaker training, tuning,
> batch inference and endpoint inference operators in Airflow. The first
> three are already PRed and merged. I will send PR for the last one this
> week.
>
> I have seen current merged codes are picked in the 1.10 test branch. I was
> wondering if there is a timeline for release of 1.10?
>
> Thanks,
> Yang
>
>
>

Re: It's very hard to become a committer on the project

[Sid Anand]

Hi Folks!
For some history, Airlfow started on GH issues. We also had a very popular
Google group. When we moved to Apache, we were told that Jira was the way
we needed to go for issue tracking because it resided on Apache
infrastructure. When we moved over, we had to drop 100+ GH issues on the
floor -- there was no way to transfer them to Jira and maintain the
original submitter/owner info since there was no mapping of users between
the 2 systems.

Here's a pie chart of our existing issues by status:
https://issues.apache.org/jira/secure/ConfigureReport.jspa?projectOrFilterId=project-12320023=statuses=12320023=com.atlassian.jira.jira-core-reports-plugin%3Apie-report_token=A5KQ-2QAV-T4JA-FDED|a85ff737799378265f90bab4f1456b5e2811a507|lin=Next


I'm attaching a screen shot as well.

[image: Screenshot 2018-09-16 11.28.57.png]

I think we all agree that there is better integration between GH PRs and GH
Issues than between GH PRs and Jira issues.

There are some practical matters to consider:

   - For the 1100-1200 unclosed/unresolved issues, how will we transfer
   them to GH or will we drop those on the floor? How would we map submitters
   between the 2 systems, and how would we transfer the content/comments,etc...
   - For the existing closed PRs (>3k), whose PRs reference JIRA, we'd need
   to keep JIRA around in read-only mode so we could reference the bug/feature
   details, but somehow disallow new JIRA creations, lest some people continue
   to use it to create new issues
   - I'm assuming the GH issue naming would not conflict with that of JIRA
   naming in commit message subjects and PRs. In other words,
   incubator-airlow-1 vs AIRFLOW-1 or airflow-1 vs AIRFLOW-1 or possibly
   conflict at AIRFLOW-1? Once we graduate, I'm pretty sure the incubator name
   will be dropped, so there may be a naming conflict.

In the end, these are 2 different tools. The issues you raise are mainly
around governance.

If you folks would like to propose a new means to manage the JIRAs, can you
outline a solution on Wiki and drop a link into an email on this list? We
can then raise a vote.

IMHO, our community would scale the best if more people picked up
responsibilities such as these. Grooming/Organizing JIRAs doesn't need to
be a responsibility owned by the maintainers. Anyone can take the lead on
discussions, etc...

-s

On Mon, Sep 17, 2018 at 2:09 AM Sumit Maheshwari 
wrote:

> Strong +1 for moving to GitHub from Jira.
>
> On Mon, Sep 17, 2018 at 12:35 PM George Leslie-Waksman 
> wrote:
>
> > Are there Apache rules preventing us from switching to GitHub Issues?
> >
> > That seems like it might better fit much of Airflow's user base.
> >
> >
> > On Sun, Sep 16, 2018, 9:21 AM Jeff Payne  wrote:
> >
> > > I agree that Jira could be better utilized. I read the original
> > > conversation on the mailing list about how Jira should be used (or if
> it
> > > should be used at all) and I'm still unclear about why it was picked
> over
> > > just using github issues. It refers to a dashboard, which I've yet to
> > > investigate, but Jira is much more than just dashboards.
> > >
> > > If this project is going to use Jira, then:
> > >
> > > 1) It would be great to see moderation and labeling of the Jira issues
> by
> > > the main contributors to make it easier for people to break into
> > > contributing.
> > > 2) It would also be nice if the initial conversation of whether or not
> an
> > > issue warrants development at all happened on the Jira issue, or at
> least
> > > some acknowledgement by the main contributors.
> > > 3) Larger enhancements and efforts or vague suggestions still get
> > > discussed on the dev mailing list before a Jira is even opened, but
> after
> > > that, the discussion moves to the Jira, with a link back to the mailing
> > > list email for reference.
> > > 4) The discussion on the PR is only concerned with HOW the change/fix
> is
> > > implemented.
> > >
> > > Get Outlook for Android
> > >
> > > 
> > > From: James Meickle 
> > > Sent: Sunday, September 16, 2018 7:46:58 AM
> > > To: d...@airflow.apache.org
> > > Subject: Re: It's very hard to become a committer on the project
> > >
> > > Definitely agree with this. I'm not always opposed to JIRA for
> projects,
> > > but the way it's being used for this project makes it very hard to
> break
> > > into contributing. The split between GH and JIRA is also painful since
> > > there's no automatic integration of them.
> > >
> > > On Sun, Sep 16, 2018 at 9:29 AM airflowuser
> > >  wrote:
> > >
> > > > Hello all,
> > > >
> > > > I'm struggling finding tickets to address and while discussing it on
> > chat
> > > > others reported they had the same problem 

Re: Visual Correlation in Superset

[Sid Anand]

Wrong dev list :-)
-s

On Wed, Sep 12, 2018 at 8:55 AM Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> The main thing that "scoped filters" and "visual correlation" have in
> common is the fact that they imply some sort of cross chart communication.
> Christine I tagged you here so you'd see the diagram that Grace shared.
>
> Let's take this to dev@. I think we need an independent SIP for chart
> messaging (pub-sub). Filtering events may or may not be pubsub events. I'd
> vote for thinking of them independently for now.
>
> Max
>
> On Tue, Sep 11, 2018 at 2:22 PM Christine Chambers 
> wrote:
>
> > It is a bit too soon to call the implementation. Still in the process of
> > writing a design for scoped filters though I do like where Grace is going
> > with the idea. :)
> >
> > I see visual correlation (VC) as an extreme case of crossfilter
> >  in which the "crossed" section is
> > a single grain on the time series (based on the aggregation level of the
> > time series). We can think of scoped filters as what can be used to back
> > the VC (i.e. the actual slicing). My thought is a scoped filter can be
> > created in multiple ways (at least two):
> > 1) descriptively through the UI (the MVP IMO)
> > 2) implicitly through interactions like the one in VC (e.g. hovering over
> > a grain of time)
> >
> > The model I have forming in my head is a pubsub one where scoped filters
> > get published by the ways mentioned above and interested parties (e.g.
> > charts in dashboards) subscribe to them. I'm hesitant to pin down the
> role
> > `refreshExcept` plays right now. My gut feeling is I'll refactor/rewrite
> a
> > large portion of it. :)
> >
> > All of these said, Jamshed, I don't think scoped filters will solve the
> > particular hover interaction issue you're running into. Assuming scoped
> > filters were implemented today, I'd expect the "show tooltips in other
> > charts" bit to be callback functions each chart in the dashboard provides
> > (these callback functions will take a scoped filter as one of their
> > arguments). Btw, nvd3 is very imperative, to make it play nice with
> > redux/react, instead of reaching in for the inner nvd3 object, you could
> > consider adding a react wrapper around it and provide a callback function
> > to show tooltips.
> >
> > What's your timeline like for implementing VC? Trying to figure out
> if/how
> > I can help you with the slicing bit.
> >
> > Cheers,
> > Christine
> >
> > On Tue, Sep 11, 2018 at 12:59 PM Grace Guo  wrote:
> >
> >> BTW, use scoped filter to implement visual correlation in dashboard is
> >> just my thought…I didn’t discuss with Christine :)
> >> This feature can totally independent of scoped filter too. I am open to
> >> all proposals.
> >>
> >> - Grace
> >>
> >>
> >> On Sep 11, 2018, at 12:44 PM, Jamshed Rahman 
> wrote:
> >>
> >> Thanks for sending the Scoped Filter info. I'm not sure if this is
> inline
> >> with what I'm trying to do though. The "Chart" list objects available in
> >> Dashboard.jsx is of type ChartPropShape. They don't have references to
> the
> >> inner nvd3 object, which is required for hover/tooltip rendering. I will
> >> need to access to /Chart/Chart.jsx objects for the nvd3 reference.
> >>
> >> Christine, any thoughts? :-)
> >>
> >> Jamshed
> >>
> >> On Mon, Sep 10, 2018 at 6:19 PM Maxime Beauchemin <
> >> maximebeauche...@gmail.com> wrote:
> >>
> >>> We should take this to dev@ btw
> >>>
> >>> Max
> >>>
> >>> On Mon, Sep 10, 2018 at 6:18 PM Maxime Beauchemin <
> >>> maximebeauche...@gmail.com> wrote:
> >>>
>  + cc Christine Chambers  who's currently working
>  on scoped filters
> 
>  Max
> 
>  On Mon, Sep 10, 2018 at 4:37 PM Grace Guo 
> wrote:
> 
> >
> >
> > Here is my thought that use *Scoped filter* to resolve this problem.
> >
> > *refreshExcept *hold the logic to find the list of charts that should
> > be affected by filter change.
> >
> > Thanks!
> >
> > - Grace
> >
> > On Sep 10, 2018, at 12:01 PM, Jamshed Rahman 
> > wrote:
> >
> > That was also my idea but I need some help understanding how charts
> > can subscribe to dashboard events using the current react/redux
> model. Is
> > there an example of a chart action subscribed to dashboard state? If
> not,
> > where would be the right place to put it? Sorry I'm still learning
> > react/redux and trying to figure this out so any direction would be
> helpful.
> >
> > On Sun, Sep 9, 2018 at 8:42 PM Maxime Beauchemin <
> > maximebeauche...@gmail.com> wrote:
> >
> >> The cross communication between charts on a dashboard is currently
> >> limited to filtering events.
> >>
> >> We need a much better pub-sub approach to widen what's possible
> >> there. You can picture a pub event "time-hover" and some
> visualizations
> >> could handle this type of event.
> >>
> >> Max
> >>
> 

Re: Cold-case PRs

[Sid Anand]

Looks cool Max.

Filed an Infra ticket:
https://issues.apache.org/jira/browse/INFRA-17005

Created a Jira to add a policy file: .github/stale.yml

https://issues.apache.org/jira/browse/AIRFLOW-3040

& raised a PR : https://github.com/apache/incubator-airflow/pull/3883

We can manage the Jira backlog separately.
-s

On Mon, Sep 10, 2018 at 8:38 PM Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> It doesn't deal with Jiras, just PRs and GH issues (which we don't use...)
>
> Max
>
> On Mon, Sep 10, 2018 at 6:58 PM Sid Anand  wrote:
>
> > Max,
> > How do these manage the JIRAs?
> >
> > -s
> >
> > On Mon, Sep 10, 2018 at 6:14 PM Maxime Beauchemin <
> > maximebeauche...@gmail.com> wrote:
> >
> > > I've used https://github.com/bstriner/github-bot-close-inactive-issues
> > in
> > > the past to auto-close issues / PRs based on a policy around
> inactivity.
> > It
> > > worked alright.
> > >
> > > There's also https://github.com/probot/stale which seems to be one of
> > the
> > > leading solutions, but it may require an Apache INFRA ticket to make
> the
> > > integration work (if they'll allow it).
> > >
> > > Max
> > >
> > > On Mon, Sep 10, 2018 at 5:21 PM Sid Anand  wrote:
> > >
> > > > Folks!
> > > > It's around that time (over 200 PRs with some 2 years old) when we
> need
> > > to
> > > > consider closing some abandoned PRs. We have a lot more active
> > > maintainers
> > > > now today than we had 1 or 2 years ago, we are better able to keep up
> > > with
> > > > new PR demand, but it's still not perfect.
> > > >
> > > > I've created a GitHub Label : *Potential_Cold_Case_PR.*
> > > >
> > > > I've started labeling some old PRs likewise and pinging the
> submitters.
> > > > Anyone in the community can help with labeling and pinging
> submitters.
> > If
> > > > we (maintainers) find the PR abandoned (i.e. no updates from
> submitters
> > > for
> > > > a few days), we can close the PRs.
> > > >
> > > > Additionally, if the JIRAs themselves are obsolete and no longer
> apply,
> > > we
> > > > can consider closing the JIRAs as obsolete.
> > > >
> > > > Here's one example :
> > > > https://github.com/apache/incubator-airflow/pulls/vijaysbhat
> > > >
> > > > -s
> > > >
> > >
> >
>

Re: [VOTE][RESULT] Replace with Gitter with Slack?

[Sid Anand]

Thanks for the context.

So, I created the slack workspace and it has ~ 100 people on it now and it
pretty active.

What's pretty cool is that some folks created regional channels (e.g.
singapore, boston, etc...), which is a good way to seed future meetups.

At some point, we may want to retire the Gitter channel.
-s

On Sat, Sep 8, 2018 at 11:45 PM Jakob Homan  wrote:

> So, binding VOTEs are generally only called on releases, where PMC
> members get those binding votes (and on electing new committers or PMC
> members, and those votes happen in private on the private list).
> Communities often call VOTEs on other things, like logos, questions of
> sponsorship, large scale tech decisions, etc.  How those VOTEs are
> executed (consesus [ie anyone with a binding vote can VETO] or
> majority) are up to the community to decide.
>
> The community can codify those decisions with bylaws (Here's Hadoop's:
> https://hadoop.apache.org/bylaws.html).  However, extensive bylaws are
> considered an anti-pattern - they can be indicative of a fractured or
> unwieldy community.  This is why Incubator projects are not generally
> recommended to go down this route.  Also, incubator podlings are not
> separate top-level projects, and so would be governed by the Incubator
> project bylaws, which as far as I can tell, don't exist.
>
> Also, keep in mind that the bylaws of any particular project are valid
> at the pleasure of the Board.  If the Board deems any part of them
> invalid, which it has done even very recently, it can come in and
> require the project to change them immediately.
>
> As for the current discussion, the large number of <1 votes, is a bit
> concerning.  This is indicative of lack of consensus and that further
> discussion is probably warranted.  Whether or not it's an official
> result is up for debate.
>
> However, one thing worth noting is that there's no 'official' gitter
> or slack channel for any ASF project.  The only official communication
> medium for ASF is mailing lists.  The edict 'If it didn't happen on
> the mailing list, it didn't happen' is one of the axioms of the Apache
> Way [1][2].  There's also no official IRC channel or official Stack
> Overflow tag or official in-person meetup.  Every member of the
> community is welcome to participate in whatever forum they want, but
> the only place things actually happen is on the list.  Any non-PMC
> discussions can happen anywhere, but if a decision needs to occur as
> part of that discussion, the discussion needs to be re-homed onto the
> mailing list and made there.  Part of the PMC's job is to enforce and
> model this behavior.
>
> Speaking entirely for myself, I'm not comfortable with the invitation
> model of Slack discussions.  For most of my time at Apache, irc was
> the way to go for most projects.  Some had very active irc channels
> and some were nearly dead.  (The #infra irc channel is a godsend
> though.)
>
> And again, this type of meta discussion is awesome.  It shows the
> project is maturing and thinking through some of the edge cases of
> what it means to an Apache project.
>
> -Jakob
>
>
>
> [1]
> https://community.apache.org/newbiefaq.html#NewbieFAQ-IsthereaCodeofConductforApacheprojects
> ?
> [2]
> https://blogs.apache.org/foundation/entry/success-at-apache-asynchronous-decision
>
> On 8 September 2018 at 23:14, Sid Anand  wrote:
> > Taking Binding votes into account :
> >
> > +1: 1 vote
> >
> >- Sid Anand
> >
> > 0: 2 votes
> >
> >- Bolke de Bruin
> >- Kaxil Naik
> >
> > -0.5: 1 vote
> >
> >- Arthur Wiedmer
> >
> >
> > Vote result is a net positive of +0.5.
> >
> > I counted all of the PMC/committers' votes as binding.
> >
> > -s
> >
> > On Sat, Sep 8, 2018 at 11:07 PM Arthur Wiedmer  >
> > wrote:
> >
> >> Sid,
> >>
> >> Erm, the next line is (emphasis mine) :
> >>
> >> PMC members have formally binding votes, but in general community
> members
> >> are encouraged to vote, even if their votes are *only advisory*.
> >>
> >> Again, that's not to say that the community at large cannot decide to
> host
> >> a Slack channel. I think you are more than welcome to if you want to.
> >>
> >> Best,
> >> Arthur
> >>
> >> On Sat, Sep 8, 2018 at 10:59 PM Sid Anand  wrote:
> >>
> >> > Why doesn't every vote matter for this topic?
> >> > https://www.apache.org/foundation/voting.html#binding-votes
> >> >
> >> > Am I misinterpreting the "Who is permitted to vote is, to some
> extent, a
> >>

Cold-case PRs

[Sid Anand]

Folks!
It's around that time (over 200 PRs with some 2 years old) when we need to
consider closing some abandoned PRs. We have a lot more active maintainers
now today than we had 1 or 2 years ago, we are better able to keep up with
new PR demand, but it's still not perfect.

I've created a GitHub Label : *Potential_Cold_Case_PR.*

I've started labeling some old PRs likewise and pinging the submitters.
Anyone in the community can help with labeling and pinging submitters. If
we (maintainers) find the PR abandoned (i.e. no updates from submitters for
a few days), we can close the PRs.

Additionally, if the JIRAs themselves are obsolete and no longer apply, we
can consider closing the JIRAs as obsolete.

Here's one example :
https://github.com/apache/incubator-airflow/pulls/vijaysbhat

-s

Inaugural London Apache Airflow Meetup on Sept 20!

[Sid Anand]

Folks!
Ben Marengo from Just Eat and Ash Berlin-Taylor (Fellow PPMC/Committer)
will be speaking at the upcoming inaugural London Apache Airflow Meetup!

Links:

   - https://twitter.com/ApacheAirflow/status/1039250401080487939
   - https://www.meetup.com/London-Apache-Airflow-Meetup/events/253076795/
   -
   
https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Sept10,2018

-s

Re: [VOTE][RESULT] Replace with Gitter with Slack?

[Sid Anand]

Why doesn't every vote matter for this topic?
https://www.apache.org/foundation/voting.html#binding-votes

Am I misinterpreting the "Who is permitted to vote is, to some extent, a
community-specific thing."?

For the established Apache processes around promoting a contributor to
committer/PMC, deciding on the number of +1s to allow a merge to master, or
voting on releases, I understand the need to follow the binding/non-binding
protocol. In all of those cases, the outcomes affect maintainers.

My understanding is that this topic affects the entire community, where
some members of the community are helping others. This seems to chug along
without maintainers being present. Hence, why do maintainers' votes matter
more here?

Question for the mentors.. Jakob?

-s

On Sat, Sep 8, 2018 at 10:27 PM Bolke de Bruin  wrote:

> Committers can only vote binding. Most of the time it is tagged by,
> indeed, adding "binding" to the vote.
>
> Sent from my iPhone
>
> > On 9 Sep 2018, at 07:20, Scott Halgrim 
> wrote:
> >
> > What makes a vote binding? Just putting “(binding)” after your vote?
> >> On Sep 8, 2018, 10:19 PM -0700, Bolke de Bruin ,
> wrote:
> >> Sorry sid, only binding votes directly count to the vote. Please update
> the
> >> result accordingly.
> >>
> >> Btw did you ask on gitter itself?
> >>
> >>
> >>
> >> Op zo 9 sep. 2018 03:47 schreef Sid Anand :
> >>
> >>> +1: 12 votes
> >>>
> >>> - Sid Anand
> >>> - Steve Carpenter
> >>> - Beau Barker
> >>> - Marc Bollinger
> >>> - Pedro Machado
> >>> - Scott Halgrim
> >>> - Eamon Keane
> >>> - Adam Boscarino
> >>> - Daniel Cohen
> >>> - Chandu Kavar
> >>> - William Horton
> >>> - Ben Gregory
> >>>
> >>> 0: 2 votes
> >>>
> >>> - Bolke de Bruin
> >>> - Kaxil Naik
> >>>
> >>>
> >>> -0.5: 1 vote
> >>>
> >>> - Arthur Wiedmer
> >>>
> >>> -1: 4 votes
> >>>
> >>> - Shah Altaf
> >>> - James Meickle
> >>> - Ravi Kotecha
> >>> - airflowuser (??)
> >>>
> >>> The vote concludes with 12 votes for and 4.5 votes against. The
> community
> >>> has opted to move to Slack from Gitter.
> >>>
> >>> -s
> >>>
>

Re: Seattle Apache Airflow Meetup

[Sid Anand]

Whoops.. ignore this thread... i see Jakob already kicked of the email
thread!
-s

On Sat, Sep 8, 2018 at 10:08 PM Sid Anand  wrote:

> Folks!
> We welcome the 7th & latest Apache Airflow meetup, this one in Seattle!
> https://www.meetup.com/Seattle-Apache-Airflow-Users-Group/
>
> You can respond to this list to get involved!
> -s
>

Seattle Apache Airflow Meetup

[Sid Anand]

Folks!
We welcome the 7th & latest Apache Airflow meetup, this one in Seattle!
https://www.meetup.com/Seattle-Apache-Airflow-Users-Group/

You can respond to this list to get involved!
-s

[VOTE][RESULT] Replace with Gitter with Slack?

[Sid Anand]

+1: 12 votes

   - Sid Anand
   - Steve Carpenter
   - Beau Barker
   - Marc Bollinger
   - Pedro Machado
   - Scott Halgrim
   - Eamon Keane
   - Adam Boscarino
   - Daniel Cohen
   - Chandu Kavar
   - William Horton
   - Ben Gregory

0: 2 votes

   - Bolke de Bruin
   - Kaxil Naik


-0.5: 1 vote

   - Arthur Wiedmer

-1: 4 votes

   - Shah Altaf
   - James Meickle
   - Ravi Kotecha
   - airflowuser (??)

The vote concludes with 12 votes for and 4.5 votes against. The community
has opted to move to Slack from Gitter.

-s

Re: [VOTE] Replace with Gitter with Slack?

[Sid Anand]

BTW, I hope my last email didn't kill the vote and this is just due to
weekend traffic winding down, but feel free to vote if you haven't. The
vote will be open for another day.

-s

On Thu, Sep 6, 2018 at 11:34 AM Sid Anand  wrote:

> Good points Ben.
>
> Gitter has Communities and Rooms. Currently, we seem to be represented in
> 2 Gitter communities and a few rooms:
>
>- Airflow Developers (This was our first Gitter community)
>- 2 rooms
>  - Dev
>  - Lobby
>   - Apache
>   - incubator-airflow
>
> The Airflow Developers community seems to be the one we had before joining
> Apache. It's a black hole in the sense that we can't create new rooms under
> it, we don't know who owns it (possibly Airbnb), & people show up once in a
> while, ask a question, and don't get a response. This one should definitely
> be shut down.
>
> The other community is the Apache one, which is where our Github gitter
> badge on the Readme.md points. We have a single room there called
> incubator-airflow. This is what you refer to.. essentially, it's impossible
> to have more than one conversation at a time. It's jarring. Committers are
> allowed to create new rooms under this community, but that functionality is
> not open to the general public. I just had one of my colleagues try it out
> -- as he's not an Apache committer, he was not able to create a room under
> Apache.
>
> This is not community friendly. In Slack, any one can create a channel, so
> it supports multiple concurrent conversations.
>
> Message retention will be arguably/slightly better on Slack than Gitter.
> Slack will support multiple conversations and has a good desktop client.
> Gitter does not require you to register a new account, allow you to use
> your GitHub account, but as mentioned, the registration is still pretty
> lightweight.
>
> -s
>
>
> On Thu, Sep 6, 2018 at 9:55 AM Ben Gregory  wrote:
>
>> +1 to Slack over Gitter but understand the concerns of disruption to
>> community and message retention.
>>
>> IMO - Gitter's main disadvantage is forcing all conversations through one
>> channel which means following any one issue is confusing and inefficient.
>> I've stopped checking it so much precisely because multiple conversations
>> happening at once make it difficult to follow, with or without retention.
>> And based on the number of questions that get asked over and over again, I
>> would be surprised if people are actually looking through old messages
>> prior to posting their own as much as might be expected.
>>
>> Would this project qualify for a Non-profit discount? Free for 250 users
>> and 85% off after that on a standard plan. Unsure as to the Apache
>> Foundation's legal status and how this would fall under it.
>> https://get.slack.help/hc/en-us/articles/204368833-Slack-for-Nonprofits
>>
>> On Thu, Sep 6, 2018 at 10:20 AM Sid Anand  wrote:
>>
>> > Some good points.
>> >
>> > @Arthur Wiedmer  The Apache mailing list is
>> > always the official record and Google indexes it well, so that any
>> previous
>> > discussions on the mailing lists are available to future generations. I
>> > originally started this thread with the thought of having everyone on
>> the
>> > mailing list, but then I heard that people need some sort of near
>> real-time
>> > support in an try-fail-ask iterative fashion --  something that is not
>> met
>> > by the async nature of mailing lists. This might be for more people
>> setting
>> > up Airflow for the first time and hitting some stumbling blocks. We
>> don't
>> > want walled gardens for sure, but there seems to be a legitimate ask for
>> > near real-time collaboration, so the discussion now is just Gitter vs
>> > Slack. Slack does require registering, all be it pretty light-weight,
>> but
>> > offers a nice desktop client.
>> >
>> > @Bolke de Bruin  I'm concerned about the disruption
>> to
>> > the existing community as well. I'm hoping the vote here reveals that
>> > concern in numbers. Also, why can't we shutdown the Gitter channel? At
>> the
>> > least, we can stop pointing to it.
>> >
>> >
>> > On Thu, Sep 6, 2018 at 7:55 AM William Horton
>> >  wrote:
>> >
>> > > +1
>> > >
>> > > *William Horton*
>> > > Backend Engineer
>> > >
>> > > 90 Fifth Avenue, 3rd Floor
>> > > New York NY 10011
>> > > m: 404.432.8480
>> > >
>> > >
>> > &

Re: [VOTE] Replace with Gitter with Slack?

[Sid Anand]

Good points Ben.

Gitter has Communities and Rooms. Currently, we seem to be represented in 2
Gitter communities and a few rooms:

   - Airflow Developers (This was our first Gitter community)
   - 2 rooms
 - Dev
 - Lobby
  - Apache
  - incubator-airflow

The Airflow Developers community seems to be the one we had before joining
Apache. It's a black hole in the sense that we can't create new rooms under
it, we don't know who owns it (possibly Airbnb), & people show up once in a
while, ask a question, and don't get a response. This one should definitely
be shut down.

The other community is the Apache one, which is where our Github gitter
badge on the Readme.md points. We have a single room there called
incubator-airflow. This is what you refer to.. essentially, it's impossible
to have more than one conversation at a time. It's jarring. Committers are
allowed to create new rooms under this community, but that functionality is
not open to the general public. I just had one of my colleagues try it out
-- as he's not an Apache committer, he was not able to create a room under
Apache.

This is not community friendly. In Slack, any one can create a channel, so
it supports multiple concurrent conversations.

Message retention will be arguably/slightly better on Slack than Gitter.
Slack will support multiple conversations and has a good desktop client.
Gitter does not require you to register a new account, allow you to use
your GitHub account, but as mentioned, the registration is still pretty
lightweight.

-s


On Thu, Sep 6, 2018 at 9:55 AM Ben Gregory  wrote:

> +1 to Slack over Gitter but understand the concerns of disruption to
> community and message retention.
>
> IMO - Gitter's main disadvantage is forcing all conversations through one
> channel which means following any one issue is confusing and inefficient.
> I've stopped checking it so much precisely because multiple conversations
> happening at once make it difficult to follow, with or without retention.
> And based on the number of questions that get asked over and over again, I
> would be surprised if people are actually looking through old messages
> prior to posting their own as much as might be expected.
>
> Would this project qualify for a Non-profit discount? Free for 250 users
> and 85% off after that on a standard plan. Unsure as to the Apache
> Foundation's legal status and how this would fall under it.
> https://get.slack.help/hc/en-us/articles/204368833-Slack-for-Nonprofits
>
> On Thu, Sep 6, 2018 at 10:20 AM Sid Anand  wrote:
>
> > Some good points.
> >
> > @Arthur Wiedmer  The Apache mailing list is
> > always the official record and Google indexes it well, so that any
> previous
> > discussions on the mailing lists are available to future generations. I
> > originally started this thread with the thought of having everyone on the
> > mailing list, but then I heard that people need some sort of near
> real-time
> > support in an try-fail-ask iterative fashion --  something that is not
> met
> > by the async nature of mailing lists. This might be for more people
> setting
> > up Airflow for the first time and hitting some stumbling blocks. We don't
> > want walled gardens for sure, but there seems to be a legitimate ask for
> > near real-time collaboration, so the discussion now is just Gitter vs
> > Slack. Slack does require registering, all be it pretty light-weight, but
> > offers a nice desktop client.
> >
> > @Bolke de Bruin  I'm concerned about the disruption
> to
> > the existing community as well. I'm hoping the vote here reveals that
> > concern in numbers. Also, why can't we shutdown the Gitter channel? At
> the
> > least, we can stop pointing to it.
> >
> >
> > On Thu, Sep 6, 2018 at 7:55 AM William Horton
> >  wrote:
> >
> > > +1
> > >
> > > *William Horton*
> > > Backend Engineer
> > >
> > > 90 Fifth Avenue, 3rd Floor
> > > New York NY 10011
> > > m: 404.432.8480
> > >
> > >
> > >
> > >
> > > On Thu, Sep 6, 2018 at 10:03 AM, Ravi Kotecha 
> > > wrote:
> > >
> > > > -1
> > > >
> > > > On Thu, Sep 6, 2018 at 2:01 PM Chandu Kavar 
> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Thu, Sep 6, 2018, 10:30 AM Sid Anand  wrote:
> > > > >
> > > > > > Hi Folks!
> > > > > > In the Apache tradition, I'd like to ask the community to vote on
> > > > > replacing
> > > > > > Gitter with Slack.
> > > > > >
> > > > > > For more information about wh

Re: [VOTE] Replace with Gitter with Slack?

[Sid Anand]

Some good points.

@Arthur Wiedmer  The Apache mailing list is
always the official record and Google indexes it well, so that any previous
discussions on the mailing lists are available to future generations. I
originally started this thread with the thought of having everyone on the
mailing list, but then I heard that people need some sort of near real-time
support in an try-fail-ask iterative fashion --  something that is not met
by the async nature of mailing lists. This might be for more people setting
up Airflow for the first time and hitting some stumbling blocks. We don't
want walled gardens for sure, but there seems to be a legitimate ask for
near real-time collaboration, so the discussion now is just Gitter vs
Slack. Slack does require registering, all be it pretty light-weight, but
offers a nice desktop client.

@Bolke de Bruin  I'm concerned about the disruption to
the existing community as well. I'm hoping the vote here reveals that
concern in numbers. Also, why can't we shutdown the Gitter channel? At the
least, we can stop pointing to it.


On Thu, Sep 6, 2018 at 7:55 AM William Horton
 wrote:

> +1
>
> *William Horton*
> Backend Engineer
>
> 90 Fifth Avenue, 3rd Floor
> New York NY 10011
> m: 404.432.8480
>
>
>
>
> On Thu, Sep 6, 2018 at 10:03 AM, Ravi Kotecha 
> wrote:
>
> > -1
> >
> > On Thu, Sep 6, 2018 at 2:01 PM Chandu Kavar  wrote:
> >
> > > +1
> > >
> > > On Thu, Sep 6, 2018, 10:30 AM Sid Anand  wrote:
> > >
> > > > Hi Folks!
> > > > In the Apache tradition, I'd like to ask the community to vote on
> > > replacing
> > > > Gitter with Slack.
> > > >
> > > > For more information about what was recently discussed, refer to
> > > >
> > > >
> > > https://lists.apache.org/thread.html/8eeb6c46ec431b9158f87022ceaa5e
> > ed8dbaaf082c887dae55f86f96@%3Cdev.airflow.apache.org%3E
> > > >
> > > > If you would like to replace Gitter with Slack, vote +1. If you want
> to
> > > > keep things they way they are, vote -1. You can also vote 0 if you
> > don't
> > > > care either way because you wouldn't use either much, preferring to
> use
> > > the
> > > > mailing list instead, which is highly encouraged as it is Apache's
> > > official
> > > > record.
> > > >
> > > > The vote will be open for 72 hours and will expire at 8p PT this
> > > Saturday.
> > > > -s
> > > >
> > > > P.S. If the community votes for Slack, we could create our own
> > workspace
> > > > (e.g. airflow.slack.com).
> > > > P.P.S. In general, anyone in the community can launch a vote like
> this
> > > from
> > > > time to time. There is no binding/non-binding distinction since we
> are
> > > not
> > > > running an official Apache vote.
> > > >
> > >
> >
>

Re: [VOTE] Replace with Gitter with Slack?

[Sid Anand]

+1

On Wed, Sep 5, 2018 at 7:38 PM Steve Carpenter 
wrote:

> +1
>
> On Wed, Sep 5, 2018 at 8:30 PM Sid Anand  wrote:
>
> > Hi Folks!
> > In the Apache tradition, I'd like to ask the community to vote on
> replacing
> > Gitter with Slack.
> >
> > For more information about what was recently discussed, refer to
> >
> >
> https://lists.apache.org/thread.html/8eeb6c46ec431b9158f87022ceaa5eed8dbaaf082c887dae55f86f96@%3Cdev.airflow.apache.org%3E
> >
> > If you would like to replace Gitter with Slack, vote +1. If you want to
> > keep things they way they are, vote -1. You can also vote 0 if you don't
> > care either way because you wouldn't use either much, preferring to use
> the
> > mailing list instead, which is highly encouraged as it is Apache's
> official
> > record.
> >
> > The vote will be open for 72 hours and will expire at 8p PT this
> Saturday.
> > -s
> >
> > P.S. If the community votes for Slack, we could create our own workspace
> > (e.g. airflow.slack.com).
> > P.P.S. In general, anyone in the community can launch a vote like this
> from
> > time to time. There is no binding/non-binding distinction since we are
> not
> > running an official Apache vote.
> >
> --
>
> Steven Carpenter / Data Engineer
> steve.carpen...@ibotta.com
>
> [image: Twitter]  <https://htmlsig.com/t/01CQQWSK> [image: Facebook]
> <https://htmlsig.com/t/01CN5W1N> [image: Instagram]
> <https://htmlsig.com/t/01CR7E6D>
>

[VOTE] Replace with Gitter with Slack?

[Sid Anand]

Hi Folks!
In the Apache tradition, I'd like to ask the community to vote on replacing
Gitter with Slack.

For more information about what was recently discussed, refer to
https://lists.apache.org/thread.html/8eeb6c46ec431b9158f87022ceaa5eed8dbaaf082c887dae55f86f96@%3Cdev.airflow.apache.org%3E

If you would like to replace Gitter with Slack, vote +1. If you want to
keep things they way they are, vote -1. You can also vote 0 if you don't
care either way because you wouldn't use either much, preferring to use the
mailing list instead, which is highly encouraged as it is Apache's official
record.

The vote will be open for 72 hours and will expire at 8p PT this Saturday.
-s

P.S. If the community votes for Slack, we could create our own workspace
(e.g. airflow.slack.com).
P.P.S. In general, anyone in the community can launch a vote like this from
time to time. There is no binding/non-binding distinction since we are not
running an official Apache vote.

Re: Call for fixes for Airflow 1.10.1

[Sid Anand]

airflowuser  Can you pick a more descriptive email handle? We are all
Airflow users, so perhaps pick an email handle that better identifies you,
especially if you report bugs or raise PRs.

To do this, you simply send an email to
dev-subscr...@airflow.incubator.apache.org with a handle that is somewhat
more unique than airflowuser.
-s

On Wed, Sep 5, 2018 at 8:55 AM Bolke de Bruin  wrote:

> You should push these to v1-10-test not to stable. Only once we start
> cutting RCs you should push to -stable. See the docs. This ensures a stable
> “stable”branch.
>
> Cheers
> Bolke.
>
> > On 3 Sep 2018, at 14:20, Ash Berlin-Taylor  wrote:
> >
> > Hi everyone,
> >
> > I'm starting the process of gathering fixes for a 1.10.1. So far the
> list of issues I have that we should pull in are
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20AIRFLOW%20AND%20fixVersion%20%3D%201.10.1%20ORDER%20BY%20key%20ASC
> (reproduces below)
> >
> > I will start pushing these as cherry-picked commits to the v1-10-stable
> branch today.
> >
> > If you have something that is not in the list below let me know. I'd
> like to keep this to bug fixes against 1.10.0 only if possible.
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2145 Deadlock after
> clearing a running task
> > https://github.com/apache/incubator-airflow/pull/3657
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2476 update tabulate dep
> to 0.8.2
> > https://github.com/apache/incubator-airflow/pull/3835
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2778 Bad Import in
> collect_dag in DagBag
> > https://github.com/apache/incubator-airflow/pull/3624
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2900 Show code for
> packaged DAGs
> > https://github.com/apache/incubator-airflow/pull/3749
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2949 Add syntax highlight
> for single quote strings
> > https://github.com/apache/incubator-airflow/pull/3795
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2984 Cannot convert
> naive_datetime when task has a naive start_date/end_date
> > https://github.com/apache/incubator-airflow/pull/3822
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2979 Deprecated Celery
> Option not in Options list
> > https://github.com/apache/incubator-airflow/pull/3832
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2989 No Parameter to
> change bootDiskType for DataprocClusterCreateOperator
> > https://github.com/apache/incubator-airflow/pull/3825
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2990 Docstrings for
> Hooks/Operators are in incorrect format
> > https://github.com/apache/incubator-airflow/pull/3820
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2994 flatten_results in
> BigQueryOperator/BigQueryHook should default to None
> > https://github.com/apache/incubator-airflow/pull/3829
> >
> >
> > In addition to those PRs which are already marked with Fix Version of
> 1.10.1 I think we should also pull in these:
> >
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2713 Rename async
> variable for Python 3.7.0 compatibility
> > https://github.com/apache/incubator-airflow/pull/3561
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2895 Prevent scheduler
> from spamming heartbeats/logs
> > https://github.com/apache/incubator-airflow/pull/3747
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2921 A trivial
> incorrectness in CeleryExecutor()
> > https://github.com/apache/incubator-airflow/pull/3773
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2866 Missing CSRF Token
> Error on Web RBAC UI Create/Update Operations
> > https://github.com/apache/incubator-airflow/pull/3804
> >
> >
> > https://issues.apache.org/jira/browse/AIRFLOW-2951
> > https://github.com/apache/incubator-airflow/pull/3798 Update dag_run
> table end_date when state change
> > (though as written it has a few other deps to cherry pick in, so will
> see about this one)
> >
>
>

Re: Use Docker for running Airflow tests

[Sid Anand]

Any chance you can document a flow that ties the CI build artifacts to this
repo and how we would use them? Would our build times go down?

-s

On Sun, Aug 19, 2018 at 11:43 AM Driesprong, Fokko 
wrote:

> Hi all,
>
> Gerardo is doing some awesome work on Dockerizing the CI pipeline. The PR
> is still open here: https://github.com/apache/incubator-airflow/pull/3393
>
> To make this work, and not store any Docker images in private repositories,
> I've created a new repository which stores the Docker image which contains
> a lot of the dependencies we use to test against. The repository is here:
> https://github.com/apache/incubator-airflow-ci
> For most of the tests, we use mocks, but sometimes we also use the actual
> service, for example for some Hadoop, Kerberos, Hive functionality. The
> images are build in the Dockerhub service:
> https://hub.docker.com/r/airflowci/incubator-airflow-ci/builds/
> I've created it for now. I'm open for another name, but Airflow was already
> taken, and hyphens are not allowed.
>
> This change will greatly decouple our dependency on Travis and will make it
> much easier to test the code locally. Instead of setting up tox
> environments, we can just use Docker instead. I'm really enthusiastic about
> the change, and really thankful for Gerardo to put all this effort in
> there.
>
> Recently Holden Karau did a lifestream
> , and did her first PR. I
> think the most important learning from it was that testing was not as
> trivial as it should be. Therefore I think we should see if we can merge
> the PR from Gerardo soon.
>
> Cheers, Fokko
>

A Tale of 2 UIs

[Sid Anand]

Hi Folks!
@verdan did some work to modularlize / modernize the UI for the rbac_ui in
https://github.com/apache/incubator-airflow/pull/3572

In another commit (https://github.com/apache/incubator-airflow/pull/3641),
@verdan added eslint support to the rbac_ui

*Question 1*
What's keeping us from moving over the new UI and deprecating the old one?

*Question 2*
It looks like the instructions for building the rbac UI is
https://github.com/apache/incubator-airflow/blob/master/CONTRIBUTING.md#setting-up-the-node--npm-javascript-environment-only-for-www_rbac


Verdan, you may want to update
https://cwiki.apache.org/confluence/display/AIRFLOW/Releasing+Airflow

-s

Re: Podling Report Reminder - August 2018

[Sid Anand]

https://wiki.apache.org/incubator/September2018#preview

Done, Mentors, please sign off!
-s

On Tue, Sep 4, 2018 at 9:23 AM Sid Anand  wrote:

> Ugh.. Justin, thanks for the reminder. I will put one together.
>
> Mentors,
> I'm assuming my update won't be in vain. Please chime in here if you will
> be able to sign off today.
>
> -s
>
> On Mon, Sep 3, 2018 at 3:01 PM Justin Mclean  wrote:
>
>> HI,
>>
>> Yes it does take it seriously. If a report is not signed off, it's
>> normally asked that the podling report again next month.
>>
>> I did have you down as reporting again this month (and that report is due
>> in a day), but for some reason the email reminders don't seem to have
>> reached your list. I'll leave it up to you if you want to report and have
>> mentors sign off on that report or not.
>>
>> Thanks,
>> Justin
>>
>

Re: Podling Report Reminder - August 2018

[Sid Anand]

Ugh.. Justin, thanks for the reminder. I will put one together.

Mentors,
I'm assuming my update won't be in vain. Please chime in here if you will
be able to sign off today.

-s

On Mon, Sep 3, 2018 at 3:01 PM Justin Mclean  wrote:

> HI,
>
> Yes it does take it seriously. If a report is not signed off, it's
> normally asked that the podling report again next month.
>
> I did have you down as reporting again this month (and that report is due
> in a day), but for some reason the email reminders don't seem to have
> reached your list. I'll leave it up to you if you want to report and have
> mentors sign off on that report or not.
>
> Thanks,
> Justin
>

Apache Spark Interfering with Airflow Jira/PRs ??

[Sid Anand]

Fellow committers... what happened around 11a PT today? I see a flood of
updates to our Jiras from "Apache Spark".

Some examples:

   1.
   
https://issues.apache.org/jira/browse/AIRFLOW-2408?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=16601616#comment-16601616
   2. It also updated https://issues.apache.org/jira/browse/AIRFLOW-2553 as
   resolved and closed https://github.com/apache/incubator-airflow/pull/3451
   PR

What's going on?
-s

Re: Retiring Airflow Gitter?

[Sid Anand]

Right..

Slack has awesome desktop & mobile clients. Creating an account on Slack is
very lightweight, and I don't see that being a major hurdle IMHO, though
Gitter accepts 3rd party auth (e.g. GitHub, etc).

It turns out that https://the-asf.slack.com/ already exists and has a
channel called #airflow. If we do use this and if at some point the ASF
decides to upgrade the workspace to a standard account, we would benefit.
The negative is that the 10K message retention limit is on the entire
workspace, not the channel.

I'll submit an Apache-style VOTE on this matter and let the community
decide.

I do hope that whatever the community decides doesn't turn off engagement.
-s

On Sun, Sep 2, 2018 at 11:23 AM Scott Halgrim
 wrote:

> Data point of one here, but I prefer Slack to Gitter because I have Slack
> open all day anyway. The only time I think to open Gitter is when I have a
> question about Airflow
> On Sep 1, 2018, 2:13 PM -0700, Shah Altaf , wrote:
> > Worth considering what the benefits vs problems are for Gitter vs Slack;
> > i.e. what is Slack solving that Gitter fails at, and what will you lose
> for
> > this exchange.
> >
> > IMO there is a non-quantifiable benefit of the openness of Gitter rooms,
> in
> > that Gitter conversation snippets appear in search results, with an
> '?at='
> > querystring so you can jump straight to where the conversation occurred.
> >
> > So while Gitter's search is lacking, there is that, *added* benefit of
> the
> > free, persistent chat.
> >
> > And as mentioned earlier, Gitter works with your existing accounts.
> >
> >
> >
> >
> >
> >
> >
> > On Sat, Sep 1, 2018 at 9:26 PM Sid Anand  wrote:
> >
> > > When we first entered the incubator ~2.5 years ago and we were
> considering
> > > Gitter (Slack was not an option back then), Jakob Homan (our mentor)
> > > mentioned that the the ASF mailing lists were really the place where
> things
> > > needed to be documented. We killed our google group and GH issues and
> moved
> > > into the mailing lists and Jira. We tried Gitter to give people a
> chance to
> > > get instant answers, though as I mentioned at the top of this thread,
> it
> > > had a lot of issues.
> > >
> > > I'm glad to hear that the community is gaining some benefit from a
> > > collaborative instant messaging solution, though it does appear from
> this
> > > thread that the community wants to move to Slack. As a 4-year user of
> > > Slack, I agree Slack is a pretty awesome platform, but I've mostly used
> > > paid versions at my employers. Once I get some guidance on how to set
> it up
> > > within Apache guidelines, if the ASF has any thoughts on the matter,
> I'll
> > > proceed to transition. Having 2 parallel solutions will be confusing
> IMHO.
> > > -s
> > >
> > >
> > >
> > > On Sat, Sep 1, 2018 at 2:30 AM Ash Berlin-Taylor 
> wrote:
> > >
> > > > I'm not a fan of slack for open source work - it's a walled garden,
> > > > signing up is a hurdle (where you have to use a work around of a
> heroku
> > > > app), not to mention that the client is just so memory hungry!
> > > >
> > > > I'm just a curmudgeon who still likes IRC mainly. So long as I can
> > > install
> > > > https://slack.com/apps/A7DL60U5D-irccloud/ <
> > > > https://blog.irccloud.com/slack-integration/> I won't object ;)
> > > >
> > > > (I am not a fan of Gitter either, but I am constantly logged in via
> their
> > > > IRC gateway. I'm usually the only PPMC responding in there, Bolke
> pops up
> > > > from time-to-time too)
> > > >
> > > > -ash
> > > >
> > > > > On 1 Sep 2018, at 02:40, Sid Anand  wrote:
> > > > >
> > > > > Great feedback. There is an overwhelming interest in moving Gitter
> to
> > > > > Slack. Just curious: for the folks who set up Slack for other
> Apache
> > > > > projects, did you go via an Apache Infra ticket?
> > > > >
> > > > > -s
> > > > >
> > > > > On Fri, Aug 31, 2018 at 4:32 PM James Meickle
> > > > >  wrote:
> > > > >
> > > > > > I am in the gitter chat most work days and there's always
> activity.
> > > > > >
> > > > > > I would be fine with switching to permanent retention slack for
> > > > > > searchability but don't see the point of switching without that
>

Re: Retiring Airflow Gitter?

[Sid Anand]

When we first entered the incubator ~2.5 years ago and we were considering
Gitter (Slack was not an option back then), Jakob Homan (our mentor)
mentioned that the the ASF mailing lists were really the place where things
needed to be documented. We killed our google group and GH issues and moved
into the mailing lists and Jira. We tried Gitter to give people a chance to
get instant answers, though as I mentioned at the top of this thread, it
had a lot of issues.

I'm glad to hear that the community is gaining some benefit from a
collaborative instant messaging solution, though it does appear from this
thread that the community wants to move to Slack. As a 4-year user of
Slack, I agree Slack is a pretty awesome platform, but I've mostly used
paid versions at my employers. Once I get some guidance on how to set it up
within Apache guidelines, if the ASF has any thoughts on the matter, I'll
proceed to transition. Having 2 parallel solutions will be confusing IMHO.
-s



On Sat, Sep 1, 2018 at 2:30 AM Ash Berlin-Taylor  wrote:

> I'm not a fan of slack for open source work - it's a walled garden,
> signing up is a hurdle (where you have to use a work around of a heroku
> app), not to mention that the client is just so memory hungry!
>
> I'm just a curmudgeon who still likes IRC mainly. So long as I can install
> https://slack.com/apps/A7DL60U5D-irccloud/ <
> https://blog.irccloud.com/slack-integration/> I won't object ;)
>
> (I am not a fan of Gitter either, but I am constantly logged in via their
> IRC gateway. I'm usually the only PPMC responding in there, Bolke pops up
> from time-to-time too)
>
> -ash
>
> > On 1 Sep 2018, at 02:40, Sid Anand  wrote:
> >
> > Great feedback. There is an overwhelming interest in moving Gitter to
> > Slack. Just curious: for the folks who set up Slack for other Apache
> > projects, did you go via an Apache Infra ticket?
> >
> > -s
> >
> > On Fri, Aug 31, 2018 at 4:32 PM James Meickle
> >  wrote:
> >
> >> I am in the gitter chat most work days and there's always activity.
> >>
> >> I would be fine with switching to permanent retention slack for
> >> searchability but don't see the point of switching without that feature.
> >>
> >> On Fri, Aug 31, 2018, 12:59 Sid Anand  wrote:
> >>
> >>> For a while now, we have had an Airflow Gitter account. Though this
> >> seemed
> >>> like a good idea initially, I'd like to hear from the community if
> anyone
> >>> gets value of out it. I don't believe any of the committers spend any
> >> time
> >>> on Gitter.
> >>>
> >>> Early on, the initial committers tried to be available on it, but soon
> >>> found it impossible to be available on all the timezones in which we
> had
> >>> users. Furthermore, Gitter notoriously sucks at making previously
> >> answered
> >>> questions discoverable. Also, the single-threaded nature of Gitter
> >>> essentially makes it confusing to debug/discuss more than on topic at a
> >>> time.
> >>>
> >>> The community seems to be humming along by relying on the Apache
> mailing
> >>> lists, which don't suffer the downside listed above. Hence, as newbies
> >> join
> >>> Apache Airflow, they likely hop onto Gitter. Are they getting value
> from
> >>> it? If not, perhaps we are doing them a disservice and should consider
> >> just
> >>> deleting it.
> >>>
> >>> Thoughts welcome.
> >>> -s
> >>>
> >>
>
>

Re: Retiring Airflow Gitter?

[Sid Anand]

Great feedback. There is an overwhelming interest in moving Gitter to
Slack. Just curious: for the folks who set up Slack for other Apache
projects, did you go via an Apache Infra ticket?

-s

On Fri, Aug 31, 2018 at 4:32 PM James Meickle
 wrote:

> I am in the gitter chat most work days and there's always activity.
>
> I would be fine with switching to permanent retention slack for
> searchability but don't see the point of switching without that feature.
>
> On Fri, Aug 31, 2018, 12:59 Sid Anand  wrote:
>
> > For a while now, we have had an Airflow Gitter account. Though this
> seemed
> > like a good idea initially, I'd like to hear from the community if anyone
> > gets value of out it. I don't believe any of the committers spend any
> time
> > on Gitter.
> >
> > Early on, the initial committers tried to be available on it, but soon
> > found it impossible to be available on all the timezones in which we had
> > users. Furthermore, Gitter notoriously sucks at making previously
> answered
> > questions discoverable. Also, the single-threaded nature of Gitter
> > essentially makes it confusing to debug/discuss more than on topic at a
> > time.
> >
> > The community seems to be humming along by relying on the Apache mailing
> > lists, which don't suffer the downside listed above. Hence, as newbies
> join
> > Apache Airflow, they likely hop onto Gitter. Are they getting value from
> > it? If not, perhaps we are doing them a disservice and should consider
> just
> > deleting it.
> >
> > Thoughts welcome.
> > -s
> >
>

Retiring Airflow Gitter?

[Sid Anand]

For a while now, we have had an Airflow Gitter account. Though this seemed
like a good idea initially, I'd like to hear from the community if anyone
gets value of out it. I don't believe any of the committers spend any time
on Gitter.

Early on, the initial committers tried to be available on it, but soon
found it impossible to be available on all the timezones in which we had
users. Furthermore, Gitter notoriously sucks at making previously answered
questions discoverable. Also, the single-threaded nature of Gitter
essentially makes it confusing to debug/discuss more than on topic at a
time.

The community seems to be humming along by relying on the Apache mailing
lists, which don't suffer the downside listed above. Hence, as newbies join
Apache Airflow, they likely hop onto Gitter. Are they getting value from
it? If not, perhaps we are doing them a disservice and should consider just
deleting it.

Thoughts welcome.
-s

Re: [PPMCS] Podling project logos wanted

[Sid Anand]

To provide a bit more context, one improvement is for the logo to read
Apache Airflow. I like the font & styling used in Apache Airavata. If you
have a creative streak, feel free to raise a PR with your logo -- once
merged, we can update the Apache logo page.

Looking in
https://github.com/apache/incubator-airflow/tree/master/airflow/www/static,
we don't appear to have a logo yet in our repo. We have icons, but no
logos.

-s


On Tue, Aug 28, 2018 at 10:19 AM Sid Anand  wrote:

> Hi Folks!
>
> Firstly, I didn't even know we had a logo.
> http://www.apache.org/logos/#incubating
>
> It's not super great looking. Anyone want to take a crack at a better one?
> @max ?
>
> -s
>
> -- Forwarded message -
> From: Daniel Gruno 
> Date: Tue, Aug 28, 2018 at 3:10 AM
> Subject: [PPMCS] Podling project logos wanted
> To: 
>
>
> Hello, beautiful Apache podlings!
>
> I am looking for podling project logos for use in the central logo site,
> located at http://www.apache.org/logos/
>
> Assuming your project does not yet have their logo there yet ( you can
> check for podlings at http://www.apache.org/logos/#incubating ), if you
> could please commit your logo(s) or have me assist you in this, that
> would be awesome. The process for getting your logo(s) displayed on the
> site is detailed at http://www.apache.org/logos/about.html
>
> With warm regards,
> Daniel.
>
> PS: While this was sent to the PPMCS, please feel free to share it with
> your dev@ lists, if you need assistance from the broader community.
>
> PPS: do not use 'incubator-' in your logo names, just call it nemo.eps,
> netbeans.svg or whatever your podling DNS name is.
>

Fwd: [PPMCS] Podling project logos wanted

[Sid Anand]

Hi Folks!

Firstly, I didn't even know we had a logo.
http://www.apache.org/logos/#incubating

It's not super great looking. Anyone want to take a crack at a better one?
@max ?

-s

-- Forwarded message -
From: Daniel Gruno 
Date: Tue, Aug 28, 2018 at 3:10 AM
Subject: [PPMCS] Podling project logos wanted
To: 


Hello, beautiful Apache podlings!

I am looking for podling project logos for use in the central logo site,
located at http://www.apache.org/logos/

Assuming your project does not yet have their logo there yet ( you can
check for podlings at http://www.apache.org/logos/#incubating ), if you
could please commit your logo(s) or have me assist you in this, that
would be awesome. The process for getting your logo(s) displayed on the
site is detailed at http://www.apache.org/logos/about.html

With warm regards,
Daniel.

PS: While this was sent to the PPMCS, please feel free to share it with
your dev@ lists, if you need assistance from the broader community.

PPS: do not use 'incubator-' in your logo names, just call it nemo.eps,
netbeans.svg or whatever your podling DNS name is.

Re: Python 3.6 Support for Airflow 1.10.0

[Sid Anand]

I'm +1 on going to 3.7 -- I'm running 3.6 myself.

Regarding dropping Python2 support, with almost 200 companies using
Airflow, I'd want to be very careful that we don't put any of them at a
disadvantage. For example, my former employer (a small startup) is running
on Python2 -- after I left, they don't have anyone actively maintaining it
at the company. Easing upgrades for such cases will keep them using Airflow.

It would be good to hold a survey that we promote beyond daily readers of
this mailing list and raise this as an AIP, since it's a major change.
Let's not rush it.

-s

On Tue, Aug 28, 2018 at 9:24 AM Naik Kaxil  wrote:

> We should definitely support 3.7. I left comments on the PR @tedmiston
> regarding the same. Python 2.7 will be dropped in 2020, so I guess we
> should start planning about it. Not really 100% sure though that we should
> drop it in Airflow 2.0
>
> On 28/08/2018, 17:08, "Taylor Edmiston"  wrote:
>
> I am onboard with dropping Python 2.x support.  Django officially
> dropped
> Python 2.x support with their 2.0 release since December 2017.
>
> *Taylor Edmiston*
> Blog  | CV
>  | LinkedIn
>  | AngelList
>  | Stack Overflow
> 
>
>
>
> On Tue, Aug 28, 2018 at 12:03 PM Ash Berlin-Taylor 
> wrote:
>
> > Supporting 3.7 is absolutely something we should do - it just got
> released
> > while we were already mid-way through the release process of 1.10 and
> > didn't want the scope creep.
> >
> > I'm happy to release a 1.10.1 that supports Py 3.7. The only issue
> I've
> > seen so far is around the use of `async` as a keyword. both in
> >
> > A perhaps bigger question: What are people's thoughts on dropping
> support
> > for Python2? This wouldn't happen before 2.0 at the earliest if we
> did it.
> > Probably something to raise an AIP for.
> >
> > -ash
> >
> > >
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
> On 28 Aug 2018, at 16:56, Taylor Edmiston  wrote:
> > >
> > > We are also running on 3.6 for some time.
> > >
> > > I put a quick branch together adding / upgrading to 3.6 in all of
> the
> > > places.  CI is still running so I may expect some test failures but
> > > hopefully nothing major.  I would be happy to merge this into
> Kaxil's
> > > current #3815 or as a follow-on PR.  I'll paste this back onto his
> PR as
> > > well.
> > >
> > > https://github.com/apache/incubator-airflow/pull/3816
> > >
> > > I think it's important for the project to officially support
> Python 3.6
> > > latest especially since 3.7 is out now.  While we're on the topic,
> does
> > > anyone else have thoughts on supporting 3.7 (perhaps unofficially
> to
> > > start)?  I wouldn't mind starting to get that ball rolling.
> > >
> > > *Taylor Edmiston*
> > > Blog  | CV
> > >  | LinkedIn
> > >  | AngelList
> > >  | Stack Overflow
> > > 
> > >
> > >
> > >
> > > On Tue, Aug 28, 2018 at 9:29 AM Adam Boscarino
> > >  wrote:
> > >
> > >> fwiw, we run Airflow on Python 3.6.
> > >>
> > >> On Tue, Aug 28, 2018 at 8:30 AM Naik Kaxil 
> wrote:
> > >>
> > >>> To provide more context to the issue:
> > >>>
> > >>>
> > >>>
> > >>> PyPI shows that Airflow is supported on Py2.7, 3.4 and 3.5 :
> > >>> https://pypi.org/project/apache-airflow/
> > >>>
> > >>>
> > >>>
> > >>> This is picked from setup.py:
> > >>>
> > >>>
> > >>>
> > >>
> >
> https://github.com/apache/incubator-airflow/blob/26e0d449737e8671000f671d820a9537f23f345a/setup.py#L367
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> So, should we update setup.py to include 3.6 as well?
> > >>>
> > >>>
> > >>>
> > >>> @bolke – Thughts?
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Kaxil Naik
> > >>>
> > >>> Data Reply
> > >>> 2nd Floor, Nova South
> > >>> 160 Victoria Street, Westminster
> > >>> London SW1E 5LB - UK
> > >>> phone: +44 (0)20 7730 6000
> > >>> k.n...@reply.com
> > >>> www.reply.com
> > >>>
> > >>> [image: Data Reply]
> > >>>
> > >>> *From: *Naik Kaxil 
> > >>> *Reply-To: *"dev@airflow.incubator.apache.org" <
> > >>> dev@airflow.incubator.apache.org>
> > >>> *Date: *Tuesday, 28 August 2018 at 13:27
> > >>> *To: *"dev@airflow.incubator.apache.org" <
> > >> 

Re: [RESULT][VOTE] Release Airflow 1.10.0

[Sid Anand]

@max (mine is r39132)

On Mon, Aug 27, 2018 at 10:13 AM Driesprong, Fokko 
wrote:

> Thanks for picking this up Naik! Did not have the time today to upload the
> artifacts.
>
> Cheers, Fokko
>
> Op ma 27 aug. 2018 om 18:05 schreef Naik Kaxil :
>
> > I have upload it on PyPi as well and will update the documentation now.
> >
> > On 27/08/2018, 00:32, "Arthur Wiedmer" 
> wrote:
> >
> > Done for Bolke, Fokko and kaxil.
> >
> > Best,
> > Arthur
> >
> > On Sun, Aug 26, 2018 at 3:08 AM Driesprong, Fokko
>  > >
> > wrote:
> >
> > > Gentle ping! Would be awesome to get 1.10 on Pypi :-)
> > >
> > > Op wo 22 aug. 2018 om 23:43 schreef Naik Kaxil :
> > >
> > > > Mine is "kaxil"
> > > >
> > > > On 22/08/2018, 16:18, "Bolke de Bruin" 
> wrote:
> > > >
> > > > @max
> > > >
> > > > Mine is "bolke"
> > > >
> > > > Cheers
> > > >
> > > > B.
> > > >
> > > > Sent from my iPhone
> > > >
> > > > >
> > > >
> > > > Kaxil Naik
> > > >
> > > > Data Reply
> > > > 2nd Floor, Nova South
> > > > 160 Victoria Street, Westminster
> > > > London SW1E 5LB - UK
> > > > phone: +44 (0)20 7730 6000
> > > > k.n...@reply.com
> > > > www.reply.com
> > > >
> >
> > Kaxil Naik
> >
> > Data Reply
> > 2nd Floor, Nova South
> > 160 Victoria Street, Westminster
> > London SW1E 5LB - UK
> > phone: +44 (0)20 7730 6000
> > k.n...@reply.com
> > www.reply.com
> > On 22 Aug 2018, at 16:13, Driesprong, Fokko 
> > > wrote:
> > > > >
> > > > > Certainly:
> > > > https://github.com/apache/incubator-airflow/releases/tag/1.10.0
> > > > >
> > > > > Cheers, Fokko
> > > > >
> > > > > Op wo 22 aug. 2018 om 15:18 schreef Ash Berlin-Taylor <
> > > > a...@apache.org>:
> > > > >
> > > > >> Could you push the git tag too please Fokko/Bolke?
> > > > >>
> > > > >> -ash
> > > > >>
> > > > >>> On 22 Aug 2018, at 08:16, Driesprong, Fokko
> >  > > >
> > > > >> wrote:
> > > > >>>
> > > > >>> Thanks Max,
> > > > >>>
> > > > >>> My PyPI ID is Fokko
> > > > >>>
> > > > >>> Cheers, Fokko
> > > > >>>
> > > > >>> 2018-08-21 22:49 GMT+02:00 Maxime Beauchemin <
> > > > maximebeauche...@gmail.com
> > > > >>> :
> > > > >>>
> > > >  I can, what's your PyPI ID?
> > > > 
> > > >  Max
> > > > 
> > > >  On Mon, Aug 20, 2018 at 2:11 PM Driesprong, Fokko
> > > >  > > > >>>
> > > >  wrote:
> > > > 
> > > > > Thanks Bolke!
> > > > >
> > > > > I've just pushed the artifacts to Apache Dist:
> > > > >
> > > > > https://dist.apache.org/repos/dist/release/incubator/
> > > >  airflow/1.10.0-incubating/
> > > > >
> > > > > I don't have any access to pypi, this means that I'm
> not
> > able
> > > to
> > > > upload
> > > >  the
> > > > > artifacts over there. Anyone in the position to grand
> me
> > access
> > > > or
> > > > >> upload
> > > > > it to pypi?
> > > > >
> > > > > Thanks! Cheers, Fokko
> > > > >
> > > > > 2018-08-20 20:06 GMT+02:00 Bolke de Bruin <
> > bdbr...@gmail.com>:
> > > > >
> > > > >> Hi Guys and Gals,
> > > > >>
> > > > >> The vote has passed! Apache Airflow 1.10.0 is
> official.
> > > > >>
> > > > >> As I am AFK for a while can one of the committers
> please
> > > rename
> > > >  according
> > > > >> to the release docs and push it to the relevant
> > locations
> > > (pypi
> > > > and
> > > > > Apache
> > > > >> dist)?
> > > > >>
> > > > >> Oh and maybe start a quick 1.10.1?
> > > > >>
> > > > >> Cheers
> > > > >> Bolke
> > > > >>
> > > > >> Sent from my iPhone
> > > > >>
> > > > >> Begin forwarded message:
> > > > >>
> > > > >>> From: Bolke de Bruin 
> > > > >>> Date: 20 August 2018 at 20:00:28 CEST
> > > > >>> To: gene...@incubator.apache.org,
> > > > dev@airflow.incubator.apache.org
> > > > >>> Subject: [RESULT][VOTE] Release Airflow 1.10.0
> > > > >>>
> > > > >>> The vote to release Airflow 1.10.0-incubating, having
> > been
> > > > open for 8
> > > > >>> days is now closed.
> > > > >>>
> > > > >>> There were three binding +1s and no -1 votes.
> > > > >>>
> > > > >>> +1 (binding):
> > > > >>> Justin Mclean
> > > > >>> 

Re: Amsterdam Apache Airflow meetup

[Sid Anand]

Fokko,
I added the meetup to

   - https://twitter.com/ApacheAirflow/status/1033951217091674117
   -
   
https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements#Announcements-Aug26,2018


On Sat, Aug 25, 2018 at 12:39 AM Driesprong, Fokko 
wrote:

> Hi all,
>
> Due a cancellation there is a speaker slot available at the Amsterdam
> Meetup the 12th of September:
> https://www.meetup.com/Amsterdam-Airflow-meetup/events/253673642/
>
> If you're interested in speaking let me know. It can be quite broad, from
> how you're using Airflow within your organisation, to new features that
> you're working on. Let me know if you're interested!
>
> Cheers, Fokko
>

Re: Airflow Meetup in India

[Sid Anand]

"if you build it, they will come" .. (Field of Dreams reference). Why not
try some promotion and see what happens? Bangalore is a pretty big tech
hub, so it seems like a good place to start vis-a-vis driving awareness and
adoption. PP does have a newly-established large presence there -- I'd be
happy to check if they could provide space for a meetup.

-s

On Sun, Aug 26, 2018 at 10:21 AM Sai Phanindhra  wrote:

> Hi sumit,
> Thanks for the info. Will there be any scope for meetup or shall we try
> having few introductary sessions so as to reach bigger audience. Clearly
> there may not be many indians/ residents of bangalore in this mail list.
> Shall we few series of airflow meetups.
>
> On Sun 26 Aug, 2018, 22:34 Sumit Maheshwari, 
> wrote:
>
> > Hi Sai,
> >
> > Got a total of 13 unique folks responded to poll. Out of that 3 folks are
> > interested in speaking as well.
> >
> >
> > On Sat, Aug 25, 2018 at 3:09 PM Sai Phanindhra 
> > wrote:
> >
> > > Hey Sumit,
> > > can you tell us how many are interested in attending and how many are
> > > interested in being a speaker for airflow meetup
> > >
> > > On Tue, 21 Aug 2018 at 19:45, Sumit Maheshwari  >
> > > wrote:
> > >
> > > > @Vaibhav, I guess if there are atleast 20 or plus people are
> > interested,
> > > > then only it makes sense to conduct a meetup.
> > > >
> > > > @Sai, have updated the form.
> > > >
> > > >
> > > > On Tue, Aug 21, 2018 at 7:31 PM Sai Phanindhra 
> > > > wrote:
> > > >
> > > > > Sumit, can you update the form? It still shows past months. Add
> > future
> > > > > months where we can plan a airflow meetup.
> > > > >
> > > > > On Tue 21 Aug, 2018, 18:01 Sumit Maheshwari, <
> sumeet.ma...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Yes, the poll was conducted around March end.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 21, 2018 at 5:49 PM Sai Phanindhra <
> > phani8...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Forms says preferred month is may, june or july. Is the form
> > > > outdated?
> > > > > > >
> > > > > > > On Tue 21 Aug, 2018, 17:43 Sumit Maheshwari, <
> > > sumeet.ma...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Sure, here it is https://goo.gl/forms/2NF6HtUwpOOTvi373
> > > > > > > >
> > > > > > > > On Tue, Aug 21, 2018 at 1:56 PM ramandu...@gmail.com <
> > > > > > > ramandu...@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Sumit,
> > > > > > > > > We work for Adobe and will try to have it in bangalore
> based
> > on
> > > > the
> > > > > > > > > participation.
> > > > > > > > > Would it be possible for you to  share the poll link.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Raman Gupta
> > > > > > > > >
> > > > > > > > > On 2018/08/21 06:38:37, Sumit Maheshwari <
> msu...@apache.org>
> > > > > wrote:
> > > > > > > > > > Hi Raman,
> > > > > > > > > >
> > > > > > > > > > Folks from Qubole certainly join/talk if the meetup is
> > > > happening
> > > > > in
> > > > > > > > > > Bangalore. However, sometime back I did a poll about the
> > same
> > > > and
> > > > > > got
> > > > > > > > > only
> > > > > > > > > > 7 responses, so have dropped the idea of conducting.
> > > > > > > > > >
> > > > > > > > > > Let us know if you get enough traction this time.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Sumit
> > > > > > > > > >
> > > > > > > > > > On Tue, Aug 21, 2018 at 10:28 AM ramandu...@gmail.com <
> > > > > > > > > ramandu...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi All,
> > > > > > > > > > > Do we have airflow meetup(s) in India. We are based out
> > of
> > > > > india
> > > > > > > and
> > > > > > > > > are
> > > > > > > > > > > using Apache Airflow as an orchestration engine to
> author
> > > and
> > > > > > > manage
> > > > > > > > > > > 1000(s) of multi-step workflows.
> > > > > > > > > > > Would be interested in joining/conducting Airflow
> Meetup
> > in
> > > > > > India.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Raman Gupta
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Sai Phanindhra,
> > > Ph: +91 9043258999
> > >
> >
>

Re: Podling Report Reminder - August 2018

[Sid Anand]

Mentors,
Please remember to sign off on the Podling report-- we missed last month.
Not sure how strictly the incubator takes this.

-s

On Mon, Aug 13, 2018 at 5:33 PM Sid Anand  wrote:

> Mentors,
> Kindly sign off on the podling report!
> -s
>
> -- Forwarded message -
> From: Justin Mclean 
> Date: Sat, Aug 11, 2018 at 7:55 PM
> Subject: Re: Podling Report Reminder - August 2018
> To: 
>
>
> Hi,
>
> Thanks for your report for August but it still needs sign off by at least
> one of your mentors can you please make sure this happens before teh board
> meets this week.
>
> Thanks,
> Justin
>

Fwd: Podling Report Reminder - August 2018

[Sid Anand]

Mentors,
Kindly sign off on the podling report!
-s

-- Forwarded message -
From: Justin Mclean 
Date: Sat, Aug 11, 2018 at 7:55 PM
Subject: Re: Podling Report Reminder - August 2018
To: 


Hi,

Thanks for your report for August but it still needs sign off by at least
one of your mentors can you please make sure this happens before teh board
meets this week.

Thanks,
Justin

Re: CVE-2017-12614 XSS Vulnerability in Airflow < 1.9

[Sid Anand]

Thx for sending this out Ash!
-s

On Mon, Aug 6, 2018 at 2:29 AM Ash Berlin-Taylor  wrote:

> CVE-2017-12614: Apache Reflected Reflected XSS Vulnerability
>
> Vendor: The Apache Software Foundation:
>
> Versions Affected: < 1.9
>
> Description:
> It was noticed an XSS in certain 404 pages that could be exploited to
> perform an XSS attack. Chrome will detect this as a reflected XSS attempt
> and prevent the page from loading. Firefox and other browsers don't, and
> are vulnerable to this attack.
>
> Mitigation:
> The fix for this is to upgrade to Apache Airflow 1.9.0 or above
>
> Credit:
> This issue was discovered by Seth Long at Credit Karma

Re: Deploy Airflow on Kubernetes using Airflow Operator

[Sid Anand]

Neeto!

-s

On Fri, Aug 3, 2018 at 2:36 PM Barni Seetharaman 
wrote:

> Hi
>
> We at Google just open-sourced a Kubernetes custom controller (also called
> operator) to make deploying and managing Airflow on kubernetes simple.
> The operator pattern is a power abstraction in kubernetes.
> Please watch this repo (in the process of adding docs) for further updates.
>
> https://github.com/GoogleCloudPlatform/airflow-operator
>
> Do reach out if you have any questions.
>
> Also created a channel in kubernetes slack  (#airflow-operator)
>  for any
> discussions specific to Airflow on Kubernetes (including Daniel's
> Kubernetes Executor, Kuberenetes operator and this custom controller also
> called kuberntes airflow operator).
>
> regs
> Barni
>

Re: Airflow-pr merge fails!

[Sid Anand]

Awesome.. This works now. Bumps on the road to progress are always welcome.

Honestly, the only remaining issues are 1) the notifications and 2) how to
auto-close JIRAs.
-s

On Thu, Aug 2, 2018 at 12:53 AM Ash Berlin-Taylor 
wrote:

> I had to change my git remote to use the SSH version - it turns out I had
> already done that when I was testing the pr tool changes against my fork.
>
> https://github.com/apache/incubator-airflow/pull/3680 opened that will
> correct the remote. Or run this command
>
> git remote set-url github g...@github.com:apache/incubator-airflow
>
> Sorry for all the bumps!
>
> Ash
>
> > On 1 Aug 2018, at 21:47, Sid Anand  wrote:
> >
> > Ash,
> > Per https://github.com/apache/incubator-airflow/pull/3413
> >
> > I tried running the dev/airflow-pr merge and it failed at the Github
> > authentication point.
> >
> > The local merge is complete (PR_TOOL_MERGE_PR_3663_MASTER).
> >
> > Push to Gitbox (github)? [y/N]: y
> >
> >>> Running command: git push github PR_TOOL_MERGE_PR_3663_MASTER:master
> >
> > Username for 'https://github.com': r39132
> >
> > Password for 'https://r39...@github.com':
> >
> > remote: Invalid username or password.
> >
> > fatal: Authentication failed for '
> > https://github.com/apache/incubator-airflow.git/'
> >
> > Since we have enabled 2fa, would this approach still work?
> >
> > I did remove my previous git remotes and did rerun dev/airflow-pr
> > setup_git_remotes
> >
> > Any ideas?
> > -s
>
>

Re: Podling Report Reminder - August 2018

[Sid Anand]

Bolke?

-s

On Sun, Aug 5, 2018 at 3:27 PM Naik Kaxil  wrote:

> Just reviewed it and looks good to me. Just one thing, do we just need to
> resolve the license issue for graduation?
>
> Regards,
> Kaxil
>
>
>
> On 05/08/2018, 22:15, "Sid Anand"  wrote:
>
> Done.
>
> Folks,
> Please review and correct if anything is off:
> https://wiki.apache.org/incubator/August2018#preview
>
> @Mentors, please sign off!!
>
> If you can't access it, here's the Airflow section:
>
> AirflowAirflow is a workflow automation and
> scheduling system that can be used toauthor and manage data
> pipelines.Airflow has been incubating since 2016-03-31.Three most
> important issues to address in the move towards graduation:  1.Once we
> make a release with the licensing fix we will move forward with
> graduation.  2.  3.Any issues that the Incubator PMC (IPMC) or ASF
> Board wish/need to beaware of? NoneHow has the community developed
> since the last report?1. Since our last podling report 5 months ago
> (i.e. between March   28 & August 5, inclusive), we grew our
> contributors from 427 to 5122. Since our last podling report 5 months
> ago (i.e. between March   28 & August 5, inclusive), we resolved 554
> pull requests (currently at 2834 closed   PRs)3. Since being accepted
> into the incubator, the number of companies officially   using Apache
> Airflow has risen from 30 to 183, 34 new from the last podling
> report 5 months ago.How has the project developed since the last
> report?See above : 554 PRs resolved, 85 new contributors, & 34 new
> companies  officially using it. We have also added 2 new committers :
> Kaxil Naik on May 7 & Tao Feng on Aug 3.How would you assess the
> podling's maturity?Please feel free to add your own commentary.  [ ]
> Initial setup  [ ] Working towards first release  [ ] Community
> building  [x] Nearing graduation  [ ] Other:Date of last release:
> 2017-12-15When were the last committers or PPMC members elected?2 new
> committers : Kaxil Naik on May 7 & Tao Feng on Aug 3Signed-off-by:  [
> ](airflow) Chris Nauroth Comments:  [ ](airflow) Hitesh Shah
> Comments:  [ ](airflow) Jakob Homan Comments:
>
>
> On Sun, Aug 5, 2018 at 1:45 PM Sid Anand  wrote:
>
> > I'll do this now! Will send an update shortly to the Dev DL!
> >
> > -s
> >
> > On Fri, Aug 3, 2018 at 9:34 PM  wrote:
> >
> >> Dear podling,
> >>
> >> This email was sent by an automated system on behalf of the Apache
> >> Incubator PMC. It is an initial reminder to give you plenty of time
> to
> >> prepare your quarterly board report.
> >>
> >> The board meeting is scheduled for Wed, 15 August 2018, 10:30 am
> PDT.
> >> The report for your podling will form a part of the Incubator PMC
> >> report. The Incubator PMC requires your report to be submitted 2
> weeks
> >> before the board meeting, to allow sufficient time for review and
> >> submission (Wed, August 01).
> >>
> >> Please submit your report with sufficient time to allow the
> Incubator
> >> PMC, and subsequently board members to review and digest. Again, the
> >> very latest you should submit your report is 2 weeks prior to the
> board
> >> meeting.
> >>
> >> Candidate names should not be made public before people are actually
> >> elected, so please do not include the names of potential committers
> or
> >> PPMC members in your report.
> >>
> >> Thanks,
> >>
> >> The Apache Incubator PMC
> >>
> >> Submitting your Report
> >>
> >> --
> >>
> >> Your report should contain the following:
> >>
> >> *   Your project name
> >> *   A brief description of your project, which assumes no knowledge
> of
> >> the project or necessarily of its field
> >> *   A list of the three most important issues to address in the move
> >> towards graduation.
> >> *   Any issues that the Incubator PMC or ASF Board might wish/need
> to be
> >> aware of
> >> *   How has the community developed since the last report
> >> *   How has the project developed since the last report.
> >> *   How does the podling rate their own maturity.
> >>

Re: Airflow committers (a list)

[Sid Anand]

Nice. Thx Kaxil.
-s

On Sun, Aug 5, 2018 at 3:57 PM Naik Kaxil  wrote:

> Last time I updated to add my name, it was reflected the next day.
>
> I had added the instructions here when I joined PPMC:
> https://cwiki.apache.org/confluence/display/AIRFLOW/Committers%27+Guide#Committers'Guide-AddingNewCommitters/PMCMemberstotheAirflowIncubationStatusPage
>
> Jenkins job runs daily at around 01:39 AM for the entire incubator site:
> https://builds.apache.org/view/H-L/view/Incubator/job/Incubator%20Site/
>
> Regards,
> Kaxil
>
> On 05/08/2018, 21:44, "Sid Anand"  wrote:
>
> What are the instructions to add committers to this page?
> -s
>
> To update http://incubator.apache.org/projects/airflow.html, you need
> to
> check out the incubator SVN repo and modifying
> content/projects/airflow.xml
>
> I SVN'committed the change below for tfeng but am not sure when the
> html
> page gets rebuilt and redeployed to the site. Some CI/CD process I
> believe.
>
> sianand@LM-SJN-21002367:~/Projects/incubator/content/projects $ svn
> diff
>
> Index: airflow.xml
>
> ===
>
> --- airflow.xml (revision 1837468)
>
> +++ airflow.xml (working copy)
>
> @@ -180,6 +180,11 @@
>
>Kaxil Naik
>
>  
>
>  
>
> +  .
>
> +  tfeng
>
> +  Tao Feng
>
> +
>
> +
>
>Extra
>
>.
>
>.
>
> Every time we add a committer, we need to update this page. Here's some
> info on the SVN repo so you can check it out for future reference:
>
> sianand@LM-SJN-21002367:~/Projects/incubator/content/projects $ svn
> info
>
> Path: .
>
> Working Copy Root Path: /Users/sianand/Projects/incubator
>
> URL:
>
> https://svn.apache.org/repos/asf/incubator/public/trunk/content/projects
>
> Relative URL: ^/incubator/public/trunk/content/projects
>
> Repository Root: https://svn.apache.org/repos/asf
>
> Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
>
> Revision: 1837468
>
> Node Kind: directory
>
> Schedule: normal
>
> Last Changed Author: mck
>
> Last Changed Rev: 1837396
>
> Last Changed Date: 2018-08-03 19:07:54 -0700 (Fri, 03 Aug 2018)
>
> On Sun, Aug 5, 2018 at 1:39 PM Ash Berlin-Taylor <
> ash_airflowl...@firemirror.com> wrote:
>
> >   - https://github.com/orgs/apache/teams/airflow-committers/members
> <
> > https://github.com/orgs/apache/teams/airflow-committers/members>
> >
> > This one at least is populated automatically via a 30-minutely cron
> job.
> >
> >
> > > On 5 Aug 2018, at 21:35, Sid Anand  wrote:
> > >
> > > Committers/Mentors,
> > > We have several places where committers are listed:
> > >
> > >   - https://whimsy.apache.org/roster/ppmc/airflow
> > >   - http://incubator.apache.org/projects/airflow.html
> > >   - Mentors, is the committer list here meant to be current?
> > >  - Kaxil, I added tfeng to this page (done via SVN).. though
> I'm not
> > >  sure when the actual publishing happens to this site. I
> believe
> > it's done
> > >  via some CI/CD process now.
> > >   -
> https://github.com/orgs/apache/teams/airflow-committers/members
> > >   - This is needed for the Gitbox integration we are using now.
> Does it
> > >  get populated automatically from whimsy?
> > >
> > > When we promote a contributor to committer/PPMC, do we need to
> update
> > all 3
> > > of these places?
> > >
> > > FYI, there is a PR to also add this list to the README so the
> community
> > can
> > > see who the committers are :
> > > https://github.com/apache/incubator-airflow/pull/3699
> > >
> > > Currently, the committer section of the README points to a wiki
> page that
> > > displays all of these links.
> > >
> > > And if you wanted more options, GitHub's CODEOWNERS file provides
> some
> > > interesting functionality, though I don't think we need it:
> > > https://blog.github.com/2017-07-06-introducing-code-owners/
> > >
> > > -s
> >
> >
>
>
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>

Re: Podling Report Reminder - August 2018

[Sid Anand]

Done.

Folks,
Please review and correct if anything is off:
https://wiki.apache.org/incubator/August2018#preview

@Mentors, please sign off!!

If you can't access it, here's the Airflow section:

AirflowAirflow is a workflow automation and
scheduling system that can be used toauthor and manage data
pipelines.Airflow has been incubating since 2016-03-31.Three most
important issues to address in the move towards graduation:  1.Once we
make a release with the licensing fix we will move forward with
graduation.  2.  3.Any issues that the Incubator PMC (IPMC) or ASF
Board wish/need to beaware of? NoneHow has the community developed
since the last report?1. Since our last podling report 5 months ago
(i.e. between March   28 & August 5, inclusive), we grew our
contributors from 427 to 5122. Since our last podling report 5 months
ago (i.e. between March   28 & August 5, inclusive), we resolved 554
pull requests (currently at 2834 closed   PRs)3. Since being accepted
into the incubator, the number of companies officially   using Apache
Airflow has risen from 30 to 183, 34 new from the last podling
report 5 months ago.How has the project developed since the last
report?See above : 554 PRs resolved, 85 new contributors, & 34 new
companies  officially using it. We have also added 2 new committers :
Kaxil Naik on May 7 & Tao Feng on Aug 3.How would you assess the
podling's maturity?Please feel free to add your own commentary.  [ ]
Initial setup  [ ] Working towards first release  [ ] Community
building  [x] Nearing graduation  [ ] Other:Date of last release:
2017-12-15When were the last committers or PPMC members elected?2 new
committers : Kaxil Naik on May 7 & Tao Feng on Aug 3Signed-off-by:  [
](airflow) Chris Nauroth Comments:  [ ](airflow) Hitesh Shah
Comments:  [ ](airflow) Jakob Homan Comments:


On Sun, Aug 5, 2018 at 1:45 PM Sid Anand  wrote:

> I'll do this now! Will send an update shortly to the Dev DL!
>
> -s
>
> On Fri, Aug 3, 2018 at 9:34 PM  wrote:
>
>> Dear podling,
>>
>> This email was sent by an automated system on behalf of the Apache
>> Incubator PMC. It is an initial reminder to give you plenty of time to
>> prepare your quarterly board report.
>>
>> The board meeting is scheduled for Wed, 15 August 2018, 10:30 am PDT.
>> The report for your podling will form a part of the Incubator PMC
>> report. The Incubator PMC requires your report to be submitted 2 weeks
>> before the board meeting, to allow sufficient time for review and
>> submission (Wed, August 01).
>>
>> Please submit your report with sufficient time to allow the Incubator
>> PMC, and subsequently board members to review and digest. Again, the
>> very latest you should submit your report is 2 weeks prior to the board
>> meeting.
>>
>> Candidate names should not be made public before people are actually
>> elected, so please do not include the names of potential committers or
>> PPMC members in your report.
>>
>> Thanks,
>>
>> The Apache Incubator PMC
>>
>> Submitting your Report
>>
>> --
>>
>> Your report should contain the following:
>>
>> *   Your project name
>> *   A brief description of your project, which assumes no knowledge of
>> the project or necessarily of its field
>> *   A list of the three most important issues to address in the move
>> towards graduation.
>> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
>> aware of
>> *   How has the community developed since the last report
>> *   How has the project developed since the last report.
>> *   How does the podling rate their own maturity.
>>
>> This should be appended to the Incubator Wiki page at:
>>
>> https://wiki.apache.org/incubator/August2018
>>
>> Note: This is manually populated. You may need to wait a little before
>> this page is created from a template.
>>
>> Mentors
>> ---
>>
>> Mentors should review reports for their project(s) and sign them off on
>> the Incubator wiki page. Signing off reports shows that you are
>> following the project - projects that are not signed may raise alarms
>> for the Incubator PMC.
>>
>> Incubator PMC
>>
>

Re: Podling Report Reminder - August 2018

[Sid Anand]

I'll do this now! Will send an update shortly to the Dev DL!

-s

On Fri, Aug 3, 2018 at 9:34 PM  wrote:

> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 15 August 2018, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, August 01).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Candidate names should not be made public before people are actually
> elected, so please do not include the names of potential committers or
> PPMC members in your report.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
> *   How does the podling rate their own maturity.
>
> This should be appended to the Incubator Wiki page at:
>
> https://wiki.apache.org/incubator/August2018
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC
>

Re: Apache Airflow welcome new committer/PMC member : Feng Tao (a.k.a. feng-tao)

[Sid Anand]

Welcome aboard Tao!
-s

On Sat, Aug 4, 2018 at 5:40 PM Tao Feng  wrote:

> Thanks everyone!
>
> On Fri, Aug 3, 2018 at 3:02 PM, Grant Nicholas <
> grantnicholas2...@u.northwestern.edu> wrote:
>
> > Congrats Feng!
> >
> > On Fri, Aug 3, 2018 at 12:35 PM Maxime Beauchemin <
> > maximebeauche...@gmail.com> wrote:
> >
> > > Well deserved, welcome aboard!
> > >
> > > On Fri, Aug 3, 2018 at 9:07 AM Mark Grover <
> grover.markgro...@gmail.com>
> > > wrote:
> > >
> > > > Congrats Tao!
> > > >
> > > > On Fri, Aug 3, 2018, 08:52 Jin Chang  wrote:
> > > >
> > > > > Congrats, Tao!!
> > > > >
> > > > > On Fri, Aug 3, 2018 at 8:20 AM Taylor Edmiston <
> tedmis...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Congratulations, Feng!
> > > > > >
> > > > > > *Taylor Edmiston*
> > > > > > Blog  | CV
> > > > > >  | LinkedIn
> > > > > >  | AngelList
> > > > > >  | Stack Overflow
> > > > > > 
> > > > > >
> > > > > >
> > > > > > On Fri, Aug 3, 2018 at 7:31 AM, Driesprong, Fokko
> > >  > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Welcome Feng! Awesome to have you on board!
> > > > > > >
> > > > > > > 2018-08-03 10:41 GMT+02:00 Naik Kaxil :
> > > > > > >
> > > > > > > > Hi Airflow'ers,
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Please join the Apache Airflow PMC in welcoming its newest
> > member
> > > > and
> > > > > > > >
> > > > > > > > co-committer, Feng Tao (a.k.a. feng-tao<
> > > > https://github.com/feng-tao
> > > > > >).
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Welcome Feng, great to have you on board!
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > >
> > > > > > > > Kaxil
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Kaxil Naik
> > > > > > > >
> > > > > > > > Data Reply
> > > > > > > > 2nd Floor, Nova South
> > > > > > > > 160 Victoria Street, Westminster
> > > > > > > >  > > > > > > Westminster+%0D%0ALondon+SW1E+5LB+-+UK=gmail=g>
> > > > > > > > London SW1E 5LB - UK
> > > > > > > > phone: +44 (0)20 7730 6000
> > > > > > > > k.n...@reply.com
> > > > > > > > www.reply.com
> > > > > > > >
> > > > > > > > [image: Data Reply]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Airflow committers (a list)

[Sid Anand]

What are the instructions to add committers to this page?
-s

To update http://incubator.apache.org/projects/airflow.html, you need to
check out the incubator SVN repo and modifying content/projects/airflow.xml

I SVN'committed the change below for tfeng but am not sure when the html
page gets rebuilt and redeployed to the site. Some CI/CD process I believe.

sianand@LM-SJN-21002367:~/Projects/incubator/content/projects $ svn diff

Index: airflow.xml

===

--- airflow.xml (revision 1837468)

+++ airflow.xml (working copy)

@@ -180,6 +180,11 @@

   Kaxil Naik

 

 

+  .

+  tfeng

+  Tao Feng

+

+

   Extra

   .

   .

Every time we add a committer, we need to update this page. Here's some
info on the SVN repo so you can check it out for future reference:

sianand@LM-SJN-21002367:~/Projects/incubator/content/projects $ svn info

Path: .

Working Copy Root Path: /Users/sianand/Projects/incubator

URL:
https://svn.apache.org/repos/asf/incubator/public/trunk/content/projects

Relative URL: ^/incubator/public/trunk/content/projects

Repository Root: https://svn.apache.org/repos/asf

Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68

Revision: 1837468

Node Kind: directory

Schedule: normal

Last Changed Author: mck

Last Changed Rev: 1837396

Last Changed Date: 2018-08-03 19:07:54 -0700 (Fri, 03 Aug 2018)

On Sun, Aug 5, 2018 at 1:39 PM Ash Berlin-Taylor <
ash_airflowl...@firemirror.com> wrote:

>   - https://github.com/orgs/apache/teams/airflow-committers/members <
> https://github.com/orgs/apache/teams/airflow-committers/members>
>
> This one at least is populated automatically via a 30-minutely cron job.
>
>
> > On 5 Aug 2018, at 21:35, Sid Anand  wrote:
> >
> > Committers/Mentors,
> > We have several places where committers are listed:
> >
> >   - https://whimsy.apache.org/roster/ppmc/airflow
> >   - http://incubator.apache.org/projects/airflow.html
> >   - Mentors, is the committer list here meant to be current?
> >  - Kaxil, I added tfeng to this page (done via SVN).. though I'm not
> >  sure when the actual publishing happens to this site. I believe
> it's done
> >  via some CI/CD process now.
> >   - https://github.com/orgs/apache/teams/airflow-committers/members
> >   - This is needed for the Gitbox integration we are using now. Does it
> >  get populated automatically from whimsy?
> >
> > When we promote a contributor to committer/PPMC, do we need to update
> all 3
> > of these places?
> >
> > FYI, there is a PR to also add this list to the README so the community
> can
> > see who the committers are :
> > https://github.com/apache/incubator-airflow/pull/3699
> >
> > Currently, the committer section of the README points to a wiki page that
> > displays all of these links.
> >
> > And if you wanted more options, GitHub's CODEOWNERS file provides some
> > interesting functionality, though I don't think we need it:
> > https://blog.github.com/2017-07-06-introducing-code-owners/
> >
> > -s
>
>

Re: Apache Airflow 1.10.0b4

[Sid Anand]

Yes, is this documented anywhere?
-s

On Fri, Jul 6, 2018 at 1:22 PM Ash Berlin-Taylor <
ash_airflowl...@firemirror.com> wrote:

> If there are fixes we want to get in should we merge them into the
> 1-10-test branch?
>
> I'm not sure what the release/branching workflow is yet, and I don't think
> it's written in the wiki anywere (or I'm bad at finding it)
>
> -ash
>
>
> > On 1 Jul 2018, at 11:03, Bolke de Bruin  wrote:
> >
> > Hi All,
> >
> > I have created a sdist package that is available at:
> >
> >
> http://people.apache.org/~bolke/apache-airflow-1.10.0b4+incubating.tar.gz
>  >
> >
> > In order to distinguish it from an actual (apache) release it is:
> >
> > 1. Marked as beta (python package managers do not install beta versions
> by default - PEP 440)
> > 2. It is not signed
> > 3. It is not at an official apache distribution location
> >
> > You can also put something like this in a requirements.txt file:
> >
> > git+
> >
> https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3
> <
> https://github.com/apache/incubator-airflow@v1-10-test#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3
> >]
> > <
> https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> <
> https://github.com/rodrigc/incubator-airflow@master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,ldap,mysql,postgres,redis,slack,s3]
> >>
> >
> > and then "pip install -r requirements.txt”.
> >
> > I hope that after this beta we can go to RC and start voting on 1.10
> (really)
> >
> > Cheers
> > Bolke
>
>

Re: Apache Git Services

[Sid Anand]

So, apparently, we should no longer see comments via Gitbox?

On Wed, Aug 1, 2018 at 3:40 AM Michał Niemiec 
wrote:

> I find the experience similar - it's already classified as spam and makes
> it bit difficult to use.
>
> Any chance this could become separate list maybe?
>
> Regards
> Michał Niemiec
>
>
> On 01/08/2018, 12:07, "Victor Noagbodji" <
> vnoagbo...@amplify-analytics.com> wrote:
>
> Hey people, what is Apache Git Services? And why are we all receiving
> notifications (even those by bots) sent by that service? Where can I turn
> those off? They are (for me at least) ruining the mailing list experience...
>
>

Airflow-pr merge fails!

[Sid Anand]

Ash,
Per https://github.com/apache/incubator-airflow/pull/3413

I tried running the dev/airflow-pr merge and it failed at the Github
authentication point.

The local merge is complete (PR_TOOL_MERGE_PR_3663_MASTER).

Push to Gitbox (github)? [y/N]: y

>> Running command: git push github PR_TOOL_MERGE_PR_3663_MASTER:master

Username for 'https://github.com': r39132

Password for 'https://r39...@github.com':

remote: Invalid username or password.

fatal: Authentication failed for '
https://github.com/apache/incubator-airflow.git/'

Since we have enabled 2fa, would this approach still work?

I did remove my previous git remotes and did rerun dev/airflow-pr
setup_git_remotes

Any ideas?
-s

Re: We've migrated to Github to repo!

[Sid Anand]

 for a
> > > >>>> merge-commit free `git log` and having a linear branch history in
> > > master
> > > >>>> that aligns with when things were introduced to the branch.
> > > >>>>
> > > >>>> It's possible to disable some of these options from the repo (only
> > if
> > > >>>> you're an Admin, meaning we'd have to involve INFRA to change
> that).
> > > But
> > > >>>> it's good to have options for the cases I mentioned above.
> > > >>>>
> > > >>>> So committers, use "Squash and Merge"! It matches our previous
> > process
> > > >>> when
> > > >>>> using the defaults in the now defunct `scripts/airflow-pr`
> > > >>>>
> > > >>>> [I'm really hoping I'm not starting a merge vs rebase workflow
> > debate
> > > >>>> here...]
> > > >>>>
> > > >>>> Max
> > > >>>>
> > > >>>> On Tue, Jul 31, 2018 at 12:37 PM Driesprong, Fokko
> > >  > > >>>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi Max,
> > > >>>>>
> > > >>>>> You're right. I just started plowing though my mailbox and
> merged a
> > > >>>> commit
> > > >>>>> without squash and merge, but it changes history as you mention.
> > > >>>>> Nice thing of Github is if you change it, it remembers your
> > > preference
> > > >>>>> which is Squash and Merge :-)
> > > >>>>>
> > > >>>>> Love the Gitbox so far, great work!
> > > >>>>>
> > > >>>>> Cheers, Fokko
> > > >>>>>
> > > >>>>> 2018-07-31 21:34 GMT+02:00 Maxime Beauchemin <
> > > >>> maximebeauche...@gmail.com
> > > >>>>> :
> > > >>>>>
> > > >>>>>> "Squash & Merge" (the default) does the right thing (squashes
> the
> > > >>>>> multiple
> > > >>>>>> commit and replays the resulting commit on top of master), we
> > should
> > > >>>> use
> > > >>>>>> that most of the times. We'd only want to merge if we wanted to
> > > >>>> preserve
> > > >>>>>> history from within the PR (multiple collaborators or multiple
> > > >>>> important
> > > >>>>>> commits that we want to keep detailed in master for instance).
> > > >>>>>>
> > > >>>>>> I'm not sure how to verify whether the `master` branch is
> > protected
> > > >>> on
> > > >>>>> this
> > > >>>>>> setup (without pushing to it as a test, which I'd rather not
> do).
> > We
> > > >>>>> should
> > > >>>>>> make sure that it is though as changing history on master can
> > cause
> > > >>> all
> > > >>>>>> sorts of problems.
> > > >>>>>>
> > > >>>>>> Max
> > > >>>>>>
> > > >>>>>> On Tue, Jul 31, 2018 at 9:21 AM Sid Anand 
> > > wrote:
> > > >>>>>>
> > > >>>>>>> The other benefit of using Option 3 over Option 1 is that you
> > > >>>> maintain
> > > >>>>>> the
> > > >>>>>>> history of who committed and who authored in one line in the
> Git
> > > >>>> log--
> > > >>>>>> i.e.
> > > >>>>>>> "bob33 authored and ashb committed 3 hours ago" instead of just
> > > >>> "ashb
> > > >>>>>>> committed" for a merge commit followed by the commit(s) from
> > bob33.
> > > >>>>>>>
> > > >>>>>>> On Tue, Jul 31, 2018 at 9:11 AM Sid Anand 
> > > >>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Ash,
> > > >>>>>>>> This is pretty cool. I just merged one PR from GH directly.
> > > >>>>>>>>
> > > >>>>>>>> Interestingly, I still used the `dev/airflow-pr work_local` to
> > > >>> test
> > > >>>>> out
> > > >>>>>>>> the PR, but merging directly in the GitHub UI afterwards
> > > >>> definitely
> > > >>>>>>> avoided
> > > >>>>>>>> my needing to do another `dev/airflow-pr merge` CLI command.
> > > >>>>>>>>
> > > >>>>>>>> There are 3 options in the UI: The default is "Create a merge
> > > >>>> commit"
> > > >>>>>>>> (Option 1). I think the ones we want is the "Rebase & Merge"
> > > >>>> (Option
> > > >>>>>> 3),
> > > >>>>>>>> which requires that PR submitters squash their commits.
> > > >>> Otherwise,
> > > >>>> we
> > > >>>>>>> could
> > > >>>>>>>> use "Squash & Merge" (Option 2), though I am not clear if
> Squash
> > > >>> &
> > > >>>>>> Merge
> > > >>>>>>> is
> > > >>>>>>>> more like option 1 or option 3.
> > > >>>>>>>>
> > > >>>>>>>> -s
> > > >>>>>>>>
> > > >>>>>>>> On Mon, Jul 30, 2018 at 7:19 PM Andrew Phillips <
> > > >>>>> aphill...@qrmedia.com
> > > >>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>>> We should ask Apache infra to send the GH notifs to another
> > > >>>>> mailing
> > > >>>>>>>>>> list.
> > > >>>>>>>>>
> > > >>>>>>>>> Over at jclouds, we created a "notifications@" list for this
> > > >>>>> purpose
> > > >>>>>>>>> (well, actually we renamed "issues@" to "notifications@"),
> and
> > > >>>> send
> > > >>>>>>>>> messages there:
> > > >>>>>>>>>
> > > >>>>>>>>> https://issues.apache.org/jira/browse/INFRA-7180
> > > >>>>>>>>>
> > > >>> https://mail-archives.apache.org/mod_mbox/jclouds-notifications/
> > > >>>>>>>>>
> > > >>>>>>>>> Regards
> > > >>>>>>>>>
> > > >>>>>>>>> ap
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >> --
> > > >>
> > > >> Sergei
> > > >
> > >
> > > --
> >
> > Sergei
> >
>

Re: How to handle Group filtering in RBAC?

[Sid Anand]

This looks like a great feature.

Another piece of functionality that would be nice on the UI is to filter
dags by owner, not just by the person logged in. So, if I wanted to see
Bob's or Bill's DAGs in the main landing page, by typing in Bob or Bill's
user names, the DAG view would filters DAGs shown.

On Tue, Jul 31, 2018 at 3:18 AM Luangsay Sourygna 
wrote:

> Hi all,
>
>
> I've just tested the last version of Aiflow with the new RBAC feature. It
> looks great.
>
> However, I miss the possibility to do "Group filtering".
>
> That is: I only want users to be able to auto register if they belong to a
> specific (ldap) group.
>
>
> I have been able to add this feature by modifying a bit the code:
>
>
> [root@18bc31801cc3 airflow]# diff
>
> /usr/lib/python2.7/site-packages/Flask_AppBuilder-1.11.1-py2.7.egg/flask_appbuilder/security/manager.py
> /tmp/manager.py
>
> 290,297d289
>
> < def auth_ldap_group_field(self):
>
> < return self.appbuilder.get_app.config['AUTH_LDAP_GROUP_FIELD']
>
> <
>
> < @property
>
> < def auth_ldap_allowed_group_auto_registration_field(self):
>
> < return
> self.appbuilder.get_app.config['AUTH_LDAP_ALLOWED_GROUP_AUTO_REGISTRATION']
>
> <
>
> < @property
>
> 610d601
>
> <  self.auth_ldap_group_field,
>
> 706,708d697
>
> < if not
> self.auth_ldap_allowed_group_auto_registration_field in
> ldap_user_info[self.auth_ldap_group_field]:
>
> < log.warning(username + " cannot auto register
> because it does not belong to the group " +
> self.auth_ldap_allowed_group_auto_registration_field)
>
> < return None
>
>
>
>
> However, before proposing to push this upstream, I would like to know if
> some of you have some better ideas about how to handle this group problem?
>
> What is more, in my changes I modify Flask and not Airflow, so maybe I
> should look at another way to solve my requirement.
>
>
> Regards,
>
>
> Sourygna
>

Re: We've migrated to Github to repo!

[Sid Anand]

I've also updated https://issues.apache.org/jira/browse/INFRA-16602 with
some questions for pono!

-s

On Tue, Jul 31, 2018 at 5:56 PM Sid Anand  wrote:

> I've opened the following ticket to disable the "Merge Commits" option in
> the Merge Button, leaving the other 2 options available (e.g. Squash and
> Merge, Rebase and Merge).
>
> https://issues.apache.org/jira/browse/INFRA-16852
> -s
>
> On Tue, Jul 31, 2018 at 4:51 PM Sid Anand  wrote:
>
>> If squash & merge does a rebase under the hood, then I agree that is the
>> way to go.
>>
>> -s
>>
>> On Tue, Jul 31, 2018 at 4:50 PM Sid Anand  wrote:
>>
>>> So, in GitHub, we can disable certain options. So under settings -->
>>> options, there is a section called "Merge Button", with 3 options: "Allow
>>> merge commits", "Allow squash merging", and "Allow rebase merging". You can
>>> see this on your fork but not on the
>>> https://github.com/apache/incubator-airflow.git since we (committers)
>>> are not admins.
>>>
>>> The default option is "Allow merge commits", but as Fokko mentions,
>>> GitHub remembers your preference once you executed one of the merge
>>> strategies on your most recent PR merge. Maybe we should just file a ticket
>>> to disable "Allow merge commits" and just allow the other 2?
>>>
>>> [image: Screenshot 2018-07-31 16.37.14.png]
>>>
>>>
>>> Generally speaking, in the old apache hosted setup (i.e.
>>>
>>> https://git-wip-us.apache.org/repos/asf/incubator-airflow.git) , the
>>> master branch was not protected -- force push was allowed, so I'm guessing
>>> the same would be true here? In the old setup, that was not recommended for
>>> many obvious reasons and some not-so-obvious ones. The non-obvious one was
>>> that a force push to apache broke the mirroring to
>>>
>>> https://github.com/apache/incubator-airflow.git
>>>
>>>
>>> How does the new system work? Does it replicate from
>>> https://github.com/apache/incubator-airflow.git to
>>> https://git-wip-us.apache.org/repos/asf/incubator-airflow.git?
>>>
>>>
>>> The old system replicated in the other direction? I suspect we don't
>>> want some committers to use the old way and some to use the new way since
>>> the replication directions oppose one another.
>>>
>>> On Tue, Jul 31, 2018 at 12:50 PM Maxime Beauchemin <
>>> maximebeauche...@gmail.com> wrote:
>>>
>>>> What I meant by changing history is mutating one or many SHAs in the
>>>> branch, an operation that would require force-pushing, which merging
>>>> doesn't do. Personally I prefer "Squash & Merge" as it makes for a
>>>> merge-commit free `git log` and having a linear branch history in master
>>>> that aligns with when things were introduced to the branch.
>>>>
>>>> It's possible to disable some of these options from the repo (only if
>>>> you're an Admin, meaning we'd have to involve INFRA to change that). But
>>>> it's good to have options for the cases I mentioned above.
>>>>
>>>> So committers, use "Squash and Merge"! It matches our previous process
>>>> when
>>>> using the defaults in the now defunct `scripts/airflow-pr`
>>>>
>>>> [I'm really hoping I'm not starting a merge vs rebase workflow debate
>>>> here...]
>>>>
>>>> Max
>>>>
>>>> On Tue, Jul 31, 2018 at 12:37 PM Driesprong, Fokko >>> >
>>>> wrote:
>>>>
>>>> > Hi Max,
>>>> >
>>>> > You're right. I just started plowing though my mailbox and merged a
>>>> commit
>>>> > without squash and merge, but it changes history as you mention.
>>>> > Nice thing of Github is if you change it, it remembers your preference
>>>> > which is Squash and Merge :-)
>>>> >
>>>> > Love the Gitbox so far, great work!
>>>> >
>>>> > Cheers, Fokko
>>>> >
>>>> > 2018-07-31 21:34 GMT+02:00 Maxime Beauchemin <
>>>> maximebeauche...@gmail.com>:
>>>> >
>>>> > > "Squash & Merge" (the default) does the right thing (squashes the
>>>> > multiple
>>>> > > commit and replays the resulting commit on top of master), we
>>>> should use

Re: We've migrated to Github to repo!

[Sid Anand]

I've opened the following ticket to disable the "Merge Commits" option in
the Merge Button, leaving the other 2 options available (e.g. Squash and
Merge, Rebase and Merge).

https://issues.apache.org/jira/browse/INFRA-16852
-s

On Tue, Jul 31, 2018 at 4:51 PM Sid Anand  wrote:

> If squash & merge does a rebase under the hood, then I agree that is the
> way to go.
>
> -s
>
> On Tue, Jul 31, 2018 at 4:50 PM Sid Anand  wrote:
>
>> So, in GitHub, we can disable certain options. So under settings -->
>> options, there is a section called "Merge Button", with 3 options: "Allow
>> merge commits", "Allow squash merging", and "Allow rebase merging". You can
>> see this on your fork but not on the
>> https://github.com/apache/incubator-airflow.git since we (committers)
>> are not admins.
>>
>> The default option is "Allow merge commits", but as Fokko mentions,
>> GitHub remembers your preference once you executed one of the merge
>> strategies on your most recent PR merge. Maybe we should just file a ticket
>> to disable "Allow merge commits" and just allow the other 2?
>>
>> [image: Screenshot 2018-07-31 16.37.14.png]
>>
>>
>> Generally speaking, in the old apache hosted setup (i.e.
>>
>> https://git-wip-us.apache.org/repos/asf/incubator-airflow.git) , the
>> master branch was not protected -- force push was allowed, so I'm guessing
>> the same would be true here? In the old setup, that was not recommended for
>> many obvious reasons and some not-so-obvious ones. The non-obvious one was
>> that a force push to apache broke the mirroring to
>>
>> https://github.com/apache/incubator-airflow.git
>>
>>
>> How does the new system work? Does it replicate from
>> https://github.com/apache/incubator-airflow.git to
>> https://git-wip-us.apache.org/repos/asf/incubator-airflow.git?
>>
>>
>> The old system replicated in the other direction? I suspect we don't want
>> some committers to use the old way and some to use the new way since the
>> replication directions oppose one another.
>>
>> On Tue, Jul 31, 2018 at 12:50 PM Maxime Beauchemin <
>> maximebeauche...@gmail.com> wrote:
>>
>>> What I meant by changing history is mutating one or many SHAs in the
>>> branch, an operation that would require force-pushing, which merging
>>> doesn't do. Personally I prefer "Squash & Merge" as it makes for a
>>> merge-commit free `git log` and having a linear branch history in master
>>> that aligns with when things were introduced to the branch.
>>>
>>> It's possible to disable some of these options from the repo (only if
>>> you're an Admin, meaning we'd have to involve INFRA to change that). But
>>> it's good to have options for the cases I mentioned above.
>>>
>>> So committers, use "Squash and Merge"! It matches our previous process
>>> when
>>> using the defaults in the now defunct `scripts/airflow-pr`
>>>
>>> [I'm really hoping I'm not starting a merge vs rebase workflow debate
>>> here...]
>>>
>>> Max
>>>
>>> On Tue, Jul 31, 2018 at 12:37 PM Driesprong, Fokko >> >
>>> wrote:
>>>
>>> > Hi Max,
>>> >
>>> > You're right. I just started plowing though my mailbox and merged a
>>> commit
>>> > without squash and merge, but it changes history as you mention.
>>> > Nice thing of Github is if you change it, it remembers your preference
>>> > which is Squash and Merge :-)
>>> >
>>> > Love the Gitbox so far, great work!
>>> >
>>> > Cheers, Fokko
>>> >
>>> > 2018-07-31 21:34 GMT+02:00 Maxime Beauchemin <
>>> maximebeauche...@gmail.com>:
>>> >
>>> > > "Squash & Merge" (the default) does the right thing (squashes the
>>> > multiple
>>> > > commit and replays the resulting commit on top of master), we should
>>> use
>>> > > that most of the times. We'd only want to merge if we wanted to
>>> preserve
>>> > > history from within the PR (multiple collaborators or multiple
>>> important
>>> > > commits that we want to keep detailed in master for instance).
>>> > >
>>> > > I'm not sure how to verify whether the `master` branch is protected
>>> on
>>> > this
>>> > > setup (without pushing to it as a test, which I'd rath

Re: We've migrated to Github to repo!

[Sid Anand]

If squash & merge does a rebase under the hood, then I agree that is the
way to go.

-s

On Tue, Jul 31, 2018 at 4:50 PM Sid Anand  wrote:

> So, in GitHub, we can disable certain options. So under settings -->
> options, there is a section called "Merge Button", with 3 options: "Allow
> merge commits", "Allow squash merging", and "Allow rebase merging". You can
> see this on your fork but not on the
> https://github.com/apache/incubator-airflow.git since we (committers) are
> not admins.
>
> The default option is "Allow merge commits", but as Fokko mentions, GitHub
> remembers your preference once you executed one of the merge strategies on
> your most recent PR merge. Maybe we should just file a ticket to disable
> "Allow merge commits" and just allow the other 2?
>
> [image: Screenshot 2018-07-31 16.37.14.png]
>
>
> Generally speaking, in the old apache hosted setup (i.e.
>
> https://git-wip-us.apache.org/repos/asf/incubator-airflow.git) , the
> master branch was not protected -- force push was allowed, so I'm guessing
> the same would be true here? In the old setup, that was not recommended for
> many obvious reasons and some not-so-obvious ones. The non-obvious one was
> that a force push to apache broke the mirroring to
>
> https://github.com/apache/incubator-airflow.git
>
>
> How does the new system work? Does it replicate from
> https://github.com/apache/incubator-airflow.git to
> https://git-wip-us.apache.org/repos/asf/incubator-airflow.git?
>
>
> The old system replicated in the other direction? I suspect we don't want
> some committers to use the old way and some to use the new way since the
> replication directions oppose one another.
>
> On Tue, Jul 31, 2018 at 12:50 PM Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
>
>> What I meant by changing history is mutating one or many SHAs in the
>> branch, an operation that would require force-pushing, which merging
>> doesn't do. Personally I prefer "Squash & Merge" as it makes for a
>> merge-commit free `git log` and having a linear branch history in master
>> that aligns with when things were introduced to the branch.
>>
>> It's possible to disable some of these options from the repo (only if
>> you're an Admin, meaning we'd have to involve INFRA to change that). But
>> it's good to have options for the cases I mentioned above.
>>
>> So committers, use "Squash and Merge"! It matches our previous process
>> when
>> using the defaults in the now defunct `scripts/airflow-pr`
>>
>> [I'm really hoping I'm not starting a merge vs rebase workflow debate
>> here...]
>>
>> Max
>>
>> On Tue, Jul 31, 2018 at 12:37 PM Driesprong, Fokko 
>> wrote:
>>
>> > Hi Max,
>> >
>> > You're right. I just started plowing though my mailbox and merged a
>> commit
>> > without squash and merge, but it changes history as you mention.
>> > Nice thing of Github is if you change it, it remembers your preference
>> > which is Squash and Merge :-)
>> >
>> > Love the Gitbox so far, great work!
>> >
>> > Cheers, Fokko
>> >
>> > 2018-07-31 21:34 GMT+02:00 Maxime Beauchemin <
>> maximebeauche...@gmail.com>:
>> >
>> > > "Squash & Merge" (the default) does the right thing (squashes the
>> > multiple
>> > > commit and replays the resulting commit on top of master), we should
>> use
>> > > that most of the times. We'd only want to merge if we wanted to
>> preserve
>> > > history from within the PR (multiple collaborators or multiple
>> important
>> > > commits that we want to keep detailed in master for instance).
>> > >
>> > > I'm not sure how to verify whether the `master` branch is protected on
>> > this
>> > > setup (without pushing to it as a test, which I'd rather not do). We
>> > should
>> > > make sure that it is though as changing history on master can cause
>> all
>> > > sorts of problems.
>> > >
>> > > Max
>> > >
>> > > On Tue, Jul 31, 2018 at 9:21 AM Sid Anand  wrote:
>> > >
>> > > > The other benefit of using Option 3 over Option 1 is that you
>> maintain
>> > > the
>> > > > history of who committed and who authored in one line in the Git
>> log--
>> > > i.e.
>> > > > "bob33 authored and ashb committed 3 hours ago" instead of just
>> "ashb
>> &

Re: We've migrated to Github to repo!

[Sid Anand]

So, in GitHub, we can disable certain options. So under settings -->
options, there is a section called "Merge Button", with 3 options: "Allow
merge commits", "Allow squash merging", and "Allow rebase merging". You can
see this on your fork but not on the
https://github.com/apache/incubator-airflow.git since we (committers) are
not admins.

The default option is "Allow merge commits", but as Fokko mentions, GitHub
remembers your preference once you executed one of the merge strategies on
your most recent PR merge. Maybe we should just file a ticket to disable
"Allow merge commits" and just allow the other 2?

[image: Screenshot 2018-07-31 16.37.14.png]


Generally speaking, in the old apache hosted setup (i.e.

https://git-wip-us.apache.org/repos/asf/incubator-airflow.git) , the master
branch was not protected -- force push was allowed, so I'm guessing the
same would be true here? In the old setup, that was not recommended for
many obvious reasons and some not-so-obvious ones. The non-obvious one was
that a force push to apache broke the mirroring to

https://github.com/apache/incubator-airflow.git


How does the new system work? Does it replicate from
https://github.com/apache/incubator-airflow.git to
https://git-wip-us.apache.org/repos/asf/incubator-airflow.git?


The old system replicated in the other direction? I suspect we don't want
some committers to use the old way and some to use the new way since the
replication directions oppose one another.

On Tue, Jul 31, 2018 at 12:50 PM Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> What I meant by changing history is mutating one or many SHAs in the
> branch, an operation that would require force-pushing, which merging
> doesn't do. Personally I prefer "Squash & Merge" as it makes for a
> merge-commit free `git log` and having a linear branch history in master
> that aligns with when things were introduced to the branch.
>
> It's possible to disable some of these options from the repo (only if
> you're an Admin, meaning we'd have to involve INFRA to change that). But
> it's good to have options for the cases I mentioned above.
>
> So committers, use "Squash and Merge"! It matches our previous process when
> using the defaults in the now defunct `scripts/airflow-pr`
>
> [I'm really hoping I'm not starting a merge vs rebase workflow debate
> here...]
>
> Max
>
> On Tue, Jul 31, 2018 at 12:37 PM Driesprong, Fokko 
> wrote:
>
> > Hi Max,
> >
> > You're right. I just started plowing though my mailbox and merged a
> commit
> > without squash and merge, but it changes history as you mention.
> > Nice thing of Github is if you change it, it remembers your preference
> > which is Squash and Merge :-)
> >
> > Love the Gitbox so far, great work!
> >
> > Cheers, Fokko
> >
> > 2018-07-31 21:34 GMT+02:00 Maxime Beauchemin  >:
> >
> > > "Squash & Merge" (the default) does the right thing (squashes the
> > multiple
> > > commit and replays the resulting commit on top of master), we should
> use
> > > that most of the times. We'd only want to merge if we wanted to
> preserve
> > > history from within the PR (multiple collaborators or multiple
> important
> > > commits that we want to keep detailed in master for instance).
> > >
> > > I'm not sure how to verify whether the `master` branch is protected on
> > this
> > > setup (without pushing to it as a test, which I'd rather not do). We
> > should
> > > make sure that it is though as changing history on master can cause all
> > > sorts of problems.
> > >
> > > Max
> > >
> > > On Tue, Jul 31, 2018 at 9:21 AM Sid Anand  wrote:
> > >
> > > > The other benefit of using Option 3 over Option 1 is that you
> maintain
> > > the
> > > > history of who committed and who authored in one line in the Git
> log--
> > > i.e.
> > > > "bob33 authored and ashb committed 3 hours ago" instead of just "ashb
> > > > committed" for a merge commit followed by the commit(s) from bob33.
> > > >
> > > > On Tue, Jul 31, 2018 at 9:11 AM Sid Anand  wrote:
> > > >
> > > > > Ash,
> > > > > This is pretty cool. I just merged one PR from GH directly.
> > > > >
> > > > > Interestingly, I still used the `dev/airflow-pr work_local` to test
> > out
> > > > > the PR, but merging directly in the GitHub UI afterwards definitely
> > > > avoided
> > > > > my needing to do another `dev/airflow-pr merge` CLI comm

Re: We've migrated to Github to repo!

[Sid Anand]

The other benefit of using Option 3 over Option 1 is that you maintain the
history of who committed and who authored in one line in the Git log-- i.e.
"bob33 authored and ashb committed 3 hours ago" instead of just "ashb
committed" for a merge commit followed by the commit(s) from bob33.

On Tue, Jul 31, 2018 at 9:11 AM Sid Anand  wrote:

> Ash,
> This is pretty cool. I just merged one PR from GH directly.
>
> Interestingly, I still used the `dev/airflow-pr work_local` to test out
> the PR, but merging directly in the GitHub UI afterwards definitely avoided
> my needing to do another `dev/airflow-pr merge` CLI command.
>
> There are 3 options in the UI: The default is "Create a merge commit"
> (Option 1). I think the ones we want is the "Rebase & Merge" (Option 3),
> which requires that PR submitters squash their commits. Otherwise, we could
> use "Squash & Merge" (Option 2), though I am not clear if Squash & Merge is
> more like option 1 or option 3.
>
> -s
>
> On Mon, Jul 30, 2018 at 7:19 PM Andrew Phillips 
> wrote:
>
>> > We should ask Apache infra to send the GH notifs to another mailing
>> > list.
>>
>> Over at jclouds, we created a "notifications@" list for this purpose
>> (well, actually we renamed "issues@" to "notifications@"), and send
>> messages there:
>>
>> https://issues.apache.org/jira/browse/INFRA-7180
>> https://mail-archives.apache.org/mod_mbox/jclouds-notifications/
>>
>> Regards
>>
>> ap
>>
>

Re: We've migrated to Github to repo!

[Sid Anand]

Ash,
This is pretty cool. I just merged one PR from GH directly.

Interestingly, I still used the `dev/airflow-pr work_local` to test out the
PR, but merging directly in the GitHub UI afterwards definitely avoided my
needing to do another `dev/airflow-pr merge` CLI command.

There are 3 options in the UI: The default is "Create a merge commit"
(Option 1). I think the ones we want is the "Rebase & Merge" (Option 3),
which requires that PR submitters squash their commits. Otherwise, we could
use "Squash & Merge" (Option 2), though I am not clear if Squash & Merge is
more like option 1 or option 3.

-s

On Mon, Jul 30, 2018 at 7:19 PM Andrew Phillips 
wrote:

> > We should ask Apache infra to send the GH notifs to another mailing
> > list.
>
> Over at jclouds, we created a "notifications@" list for this purpose
> (well, actually we renamed "issues@" to "notifications@"), and send
> messages there:
>
> https://issues.apache.org/jira/browse/INFRA-7180
> https://mail-archives.apache.org/mod_mbox/jclouds-notifications/
>
> Regards
>
> ap
>

Re: Podling Report Reminder - July 2018

[Sid Anand]

My Bad... totally missed this. Will catch the next one.
-s

On Wed, Jun 27, 2018 at 2:41 PM  wrote:

> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 18 July 2018, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, July 04).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Candidate names should not be made public before people are actually
> elected, so please do not include the names of potential committers or
> PPMC members in your report.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
> *   How does the podling rate their own maturity.
>
> This should be appended to the Incubator Wiki page at:
>
> https://wiki.apache.org/incubator/July2018
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC
>

Re: Kerberos and Airflow

[Sid Anand]

+1 Kerberos is important for us (PayPal)... happy to help on this effort as
well.
-s

On Fri, Jul 27, 2018 at 8:41 AM Hitesh Shah  wrote:

> Hi Taylor
>
> +1 on upstreaming this. It would be great if you can submit a pull request
> to enhance the apache airflow docs.
>
> thanks
> Hitesh
>
>
> On Thu, Jul 26, 2018 at 2:32 PM Taylor Edmiston 
> wrote:
>
> > While we're on the topic, I'd love any feedback from Bolke or others
> who've
> > used Kerberos with Airflow on this quick guide I put together yesterday.
> > It's similar to what's in the Airflow docs but instead all on one page
> > and slightly
> > expanded.
> >
> >
> >
> https://github.com/astronomerio/airflow-guides/blob/master/guides/kerberos.md
> > (or web version )
> >
> > One thing I'd like to add is a minimal example of how to Kerberize a
> hook.
> >
> > I'd be happy to upstream this as well if it's useful (maybe a Concepts >
> > Additional Functionality > Kerberos page?)
> >
> > Best,
> > Taylor
> >
> >
> > *Taylor Edmiston*
> > Blog  | CV
> >  | LinkedIn
> >  | AngelList
> >  | Stack Overflow
> > 
> >
> >
> > On Thu, Jul 26, 2018 at 5:18 PM, Driesprong, Fokko  >
> > wrote:
> >
> > > Hi Ry,
> > >
> > > You should ask Bolke de Bruin. He's really experienced with Kerberos
> and
> > he
> > > did also the implementation for Airflow. Beside that he worked also on
> > > implementing Kerberos in Ambari. Just want to let you know.
> > >
> > > Cheers, Fokko
> > >
> > > Op do 26 jul. 2018 om 23:03 schreef Ry Walker 
> > >
> > > > Hi everyone -
> > > >
> > > > We have several bigCo's who are considering using Airflow asking into
> > its
> > > > support for Kerberos.
> > > >
> > > > We're going to work on a proof-of-concept next week, will likely
> > record a
> > > > screencast on it.
> > > >
> > > > For now, we're looking for any anecdotal information from
> organizations
> > > who
> > > > are using Kerberos with Airflow, if anyone would be willing to share
> > > their
> > > > experiences here, or reply to me personally, it would be greatly
> > > > appreciated!
> > > >
> > > > -Ry
> > > >
> > > > --
> > > >
> > > > *Ry Walker* | CEO, Astronomer  |
> > > 513.417.2163 |
> > > > @rywalker  | LinkedIn
> > > > 
> > > >
> > >
> >
>

Re: Catchup By default = False vs LatestOnlyOperator

[Sid Anand]

I will +1 James comment and add to it. At Agari, one of our DAGs had as a
final step the sending of an alert. The alerts only made sense when the DAG
was current. But, sometimes, we did need to recompute some metrics based on
historical data, but not alert on them. The LatestOnlyOperator was a good
fit for this case.

George/Ben,
It would be great to document this discussion -- i.e. when to use one over
another.

-s


On Mon, Jul 23, 2018 at 2:03 PM George Leslie-Waksman 
wrote:

> Ok, not so fringe; I'm glad it's working well for your use case, James.
>
> I retract my suggestion of deprecation.
>
> On Mon, Jul 23, 2018 at 12:58 PM James Meickle
>  wrote:
>
> > We use LatestOnlyOperator in production. Generally our data is available
> on
> > a regular schedule, and we update production services with it as soon as
> it
> > is available; we might occasionally want to re-run historical days, in
> > which case we want to run the same DAG but without interacting with live
> > production services at all.
> >
> > On Mon, Jul 23, 2018 at 2:18 PM, George Leslie-Waksman <
> waks...@gmail.com>
> > wrote:
> >
> > > As the author of LatestOnlyOperator, the goal was as a stopgap until
> > > catchup=False landed.
> > >
> > > There are some (very) fringe use cases where you might still want
> > > LatestOnlyOperator but in almost all cases what you want is probably
> > > catchup=False.
> > >
> > > The situations where LatestOnlyOperator is still useful are where you
> > want
> > > to run most of your DAG for every schedule interval but you want some
> of
> > > the tasks to run only on the latest run (not catching up, not
> > backfilling).
> > >
> > > It may be best to deprecate LatestOnlyOperator at this point to avoid
> > > confusion.
> > >
> > > --George
> > >
> > > On Sat, Jul 21, 2018 at 7:34 PM Ben Tallman 
> wrote:
> > >
> > > > As the author of catch-up, the idea is that in many cases your data
> > > > doesn't "window" nicely and you want instead to just run as if it
> were
> > a
> > > > brilliant Cron...
> > > >
> > > > Ben
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On Jul 20, 2018, at 11:39 PM, Shah Altaf 
> wrote:
> > > > >
> > > > > Hi my understanding is: if you use the LatestOnlyOperator then when
> > you
> > > > run
> > > > > the DAG for the first time you'll see a whole bunch of DAG runs
> > queued
> > > > up,
> > > > > and in each run the LatestOnlyOperator will cause the rest of the
> DAG
> > > run
> > > > > to be skipped.  Only the latest DAG will run in 'full'.
> > > > >
> > > > > With catchup = False, you should just get just the latest DAG run.
> > > > >
> > > > >
> > > > > On Fri, Jul 20, 2018 at 10:58 PM Shubham Gupta <
> > > > shubham180695...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> -- Forwarded message -
> > > > >> From: Shubham Gupta 
> > > > >> Date: Fri, Jul 20, 2018 at 2:38 PM
> > > > >> Subject: Catchup By default = False vs LatestOnlyOperator
> > > > >> To: 
> > > > >>
> > > > >>
> > > > >> Hi!
> > > > >>
> > > > >> Can someone please explain the difference b/w catchup by default =
> > > False
> > > > >> and LatestOnlyOperator?
> > > > >>
> > > > >> Regarding
> > > > >> Shubham Gupta
> > > > >>
> > > >
> > >
> >
>

Re: [RESULT][VOTE] Airflow 1.10.0rc2

[Sid Anand]

Great work Bolke!
-s

On Wed, Jul 18, 2018 at 11:47 AM Chris Riccomini 
wrote:

> Thank you for running this one, Bolke. Sincerely!
>
> On Wed, Jul 18, 2018 at 11:30 AM Bolke de Bruin  wrote:
>
> > Hello,
> >
> > (Just making sure this works with the IPMC)
> >
> > Apache Airflow (incubating) 1.10.0 (based on RC2) has been accepted.
> >
> > 4 “+1” binding votes received:
> >
> > - Joy Gao (binding)
> > - Naik Kaxil (binding)
> > - Bolke de Bruin (binding)
> > - Fokko Driesprong (binding)
> >
> > My next step is to open a thread with the IPMC.
> >
> > Cheers,
> > Bolke
> >
> > > On 18 Jul 2018, at 01:04, Joy Gao  wrote:
> > >
> > > Tested this with new installment of Airflow in python 2.7/3.6, mainly
> > > focusing on:
> > > - simple DAGs running with local executor
> > > - log generation / rendering
> > > - fernet key generation on installment
> > > - rbac/non-rbac UI view rendering
> > >
> > > Voting: +1 (binding)
> > >
> > >
> > >
> > >
> > > On Mon, Jul 16, 2018 at 1:20 AM, Naik Kaxil  wrote:
> > >
> > >> I have run tests on python 2.7 and 3.5 and it works fine.
> > >>
> > >> My vote: +1 (binding)
> > >>
> > >> Thanks @bolke.
> > >>
> > >> Regards,
> > >> K
> > >>
> > >> On 16/07/2018, 09:03, "fo...@driesprongen.nl on behalf of
> Driesprong,
> > >> Fokko" 
> wrote:
> > >>
> > >>My vote is ​+1 (binding)​
> > >>
> > >>Cheers, Fokko
> > >>
> > >>2018-07-16 9:52 GMT+02:00 Driesprong, Fokko  >:
> > >>
> > >>> Awesome Bolke!
> > >>>
> > >>> I don't have a production Airflow at hand right now, but I've ran
> > >> some
> > >>> simple tests against Python 2.7, 3.5, 3.6 and it all looked fine.
> > >>>
> > >>> +1 From my side
> > >>>
> > >>> Cheers, Fokko
> > >>>
> > >>> 2018-07-15 20:05 GMT+02:00 Bolke de Bruin :
> > >>>
> >  Hey all,
> > 
> >  I have cut Airflow 1.10.0 RC2. This email is calling a vote on the
> >  release,
> >  which will last for 72 hours. Consider this my (binding) +1.
> > 
> >  Airflow 1.10.0 RC 2 is available at:
> > 
> >  https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc2/
> > >> <
> >  https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc2/
> > >>>
> > 
> >  apache-airflow-1.10.0rc2+incubating-source.tar.gz is a source
> > >> release
> >  that
> >  comes with INSTALL instructions.
> >  apache-airflow-1.10.0rc2+incubating-bin.tar.gz is the binary Python
> >  "sdist"
> >  release.
> > 
> >  Public keys are available at:
> > 
> >  https://dist.apache.org/repos/dist/release/incubator/airflow/ <
> >  https://dist.apache.org/repos/dist/release/incubator/airflow/>
> > 
> >  The amount of JIRAs fixed is over 700. Please have a look at the
> >  changelog.
> >  Since RC2 the following has been fixed:
> > 
> >  * [AIRFLOW-1729][AIRFLOW-2797][AIRFLOW-2729] Ignore whole
> > >> directories in
> >  .airflowignore
> >  * [AIRFLOW-2739] Always read default configuration files as utf-8
> >  * [AIRFLOW-2752] Log using logging instead of stdout
> >  * [AIRFLOW-1729][AIRFLOW-XXX] Remove extra debug log at info level
> > 
> >  Please note that the version number excludes the `rcX` string as
> > >> well
> >  as the "+incubating" string, so it's now simply 1.10.0. This will
> > >> allow us
> >  to rename the artifact without modifying the artifact checksums
> > >> when we
> >  actually release.
> > 
> > 
> >  Cheers,
> >  Bolke
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> Kaxil Naik
> > >>
> > >> Data Reply
> > >> 2nd Floor, Nova South
> > >> 160 Victoria Street, Westminster
> > >> London SW1E 5LB - UK
> > >> phone: +44 (0)20 7730 6000
> > >> k.n...@reply.com
> > >> www.reply.com
> > >>
> >
> >
>

Re: Staled Infra ticket to migrate to github - can we chase anyone?

[Sid Anand]

Ash,
You could mention this on the general list ... gene...@incubator.apache.org.
People are very active on that list.

You'd first need to subscribe by sending an email to
general-subscr...@incubator.apache.org
-s

On Wed, Jul 18, 2018 at 2:09 AM Ash Berlin-Taylor <
ash_airflowl...@firemirror.com> wrote:

> Hi everyone,
>
> A while ago (~6 weeks) I opened
> https://issues.apache.org/jira/browse/INFRA-16602 <
> https://issues.apache.org/jira/browse/INFRA-16602> asking the Apache
> Infra team to migrate us to github as the primary repo.
>
> Sadly nothing has happened in the time, and I tried commenting asking for
> an ETA but haven't heard anything. Does anyone if there's anything we can
> do to get this ticket moving along? I see there is at least one other
> project migrations that have been requested and done since (
> https://issues.apache.org/jira/browse/INFRA-16615 <
> https://issues.apache.org/jira/browse/INFRA-16615> for example).
>
> Any ideas?
>
> Thanks,
> Ash

Re: Slides / Airflow presentation OpenRDay

[Sid Anand]

Germain,
Thx for sharing.

I think the proper link is :
https://fr.slideshare.net/GermainTanguy/apache-airflow-at-dailymotion

Also, would you like to raise a PR to update the README's list of companies
using Airflow?
https://github.com/apache/incubator-airflow
-s

On Tue, Jul 10, 2018 at 3:06 AM Germain TANGUY <
germain.tan...@dailymotion.com> wrote:

> Hi airflow community,
>
> At my company, we have been using airflow in production for the past two
> years. A few days ago we did a talk about airflow (focus on our day to day
> use and how data engineer and data analyst work together).
>
> I just wanted to share with you the slides:
> https://fr.slideshare.net/GermainTanguy/apache-airflow-at-<
> https://fr.slideshare.net/GermainTanguy/apache-airflow-at-dailymotion>dailymotion
> and YouTube link: https://www.youtube.com/watch?v=NEtmrJWZbXQ
>
> The talk was in French but the slides are in English ;).
>
> Best regards,
>
> Germain.
> Data engineer @ Dailymotion.
>
>
>
>

Re: [VOTE] Airflow 1.10.0rc1

[Sid Anand]

FYI!
I just installed the release candidate. The first thing I noticed is a
missing tool tip for the Null State in the Recent Tasks column on the
landing page. Since the null globe is new to this UI, users will likely
hover over it to inquire what it means... and will be left wanting. Of
course, they could click on the globe, which will take them to
http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator_state_equals=null,
which will always show an empty list, leaving them a bit more confused.

-s

On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
 wrote:

> Hi Bolke,
>
> (Switching email to avoid moderation on my emails.)
>
> The normal Airflow test suite does not fail as it uses a LC_ALL set to
> utf-8.
>
> I think it is a proper test though, it is a minimal reproducible version of
> the code that fails. And the only difference in behaviour is at 3.7 which
> we don’t support anyway so I’m fairly sure it is broken for all supported
> Python 3 versions.
>
> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
> unset and I see the same failure.
>
> I don’t think this is a big thing though and we could release it without
> the fix I made. I think most people run it with a sane LC_ALL, but
> apparently we didn’t.
> Here’s the log for the test:
>
> > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
> root@b99b297df111:/# locale
> LANG=C.UTF-8
> LANGUAGE=
> LC_CTYPE="C.UTF-8"
> LC_NUMERIC="C.UTF-8"
> LC_TIME="C.UTF-8"
> LC_COLLATE="C.UTF-8"
> LC_MONETARY="C.UTF-8"
> LC_MESSAGES="C.UTF-8"
> LC_PAPER="C.UTF-8"
> LC_NAME="C.UTF-8"
> LC_ADDRESS="C.UTF-8"
> LC_TELEPHONE="C.UTF-8"
> LC_MEASUREMENT="C.UTF-8"
> LC_IDENTIFICATION="C.UTF-8"
> LC_ALL=
> > unset LANG
> root@b99b297df111:/# locale
> LANG=
> LANGUAGE=
> LC_CTYPE="POSIX"
> LC_NUMERIC="POSIX"
> LC_TIME="POSIX"
> LC_COLLATE="POSIX"
> LC_MONETARY="POSIX"
> LC_MESSAGES="POSIX"
> LC_PAPER="POSIX"
> LC_NAME="POSIX"
> LC_ADDRESS="POSIX"
> LC_TELEPHONE="POSIX"
> LC_MEASUREMENT="POSIX"
> LC_IDENTIFICATION="POSIX"
> LC_ALL=
> root@b99b297df111:/# pip install -e .[devel]
> root@b99b297df111:/airflow# ./run_unit_tests.sh
> + export AIRFLOW_HOME=/root/airflow
> + AIRFLOW_HOME=/root/airflow
> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
> + AIRFLOW__CORE__UNIT_TEST_MODE=True
> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
> + export AIRFLOW_USE_NEW_IMPORTS=1
> + AIRFLOW_USE_NEW_IMPORTS=1
> +++ dirname ./run_unit_tests.sh
> ++ cd .
> ++ pwd
> + DIR=/airflow
> + export PYTHONPATH=:/airflow/tests/test_utils
> + PYTHONPATH=:/airflow/tests/test_utils
> + nose_args=
> + which airflow
> + echo 'Initializing the DB'
> Initializing the DB
> + airflow resetdb
> + yes
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 6, in 
> exec(compile(open(__file__).read(), __file__, 'exec'))
>   File "/airflow/airflow/bin/airflow", line 21, in 
> from airflow import configuration
>   File "/airflow/airflow/__init__.py", line 35, in 
> from airflow import configuration as conf
>   File "/airflow/airflow/configuration.py", line 106, in 
> DEFAULT_CONFIG = f.read()
>   File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
> return codecs.ascii_decode(input, self.errors)[0]
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
> ordinal not in range(128)
> + '[' '' ']'
> + '[' -z '' ']'
> + nose_args='--with-coverage --cover-erase --cover-html
> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>   --with-ignore-docstrings --rednose --with-timer -s -v
> --logging-level=DEBUG '
> + echo 'Starting the unit tests with the following nose arguments:
> --with-coverage' --cover-erase --cover-html --cover-package=airflow
> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> --rednose --with-timer -s -v --logging-level=DEBUG
> Starting the unit tests with the following nose arguments: --with-coverage
> --cover-erase --cover-html --cover-package=airflow
> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
> --rednose --with-timer -s -v --logging-level=DEBUG
> + nosetests --with-coverage --cover-erase --cover-html
> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
> nose.plugins.cover: ERROR: Coverage not available: unable to import
> coverage module
> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> position 21082: ordinal not in range(128)) ... ERROR
> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
> position 21082: ordinal not in range(128)) ... ERROR
> ==
> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
> in position 21082: ordinal not in range(128))
> 

Re: [DISCUSS] AIP - Time for Airflow Improvement Proposals?

[Sid Anand]

+1

On Tue, Jul 10, 2018 at 1:02 PM George Leslie-Waksman
 wrote:

> +1
>
> On Tue, Jul 10, 2018 at 11:50 AM Jakob Homan  wrote:
>
> > Lots of Apache projects use ?IPs - Whatever Improvement Proposal - to
> > document and gather consensus on large changes to the code base.  Some
> > examples:
> >* Kafka Improvement Proposals (KIP) -
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >   * Flink Improvement Proposal (FLIP) -
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >   * Spark Improvement Proposal (SPIP) -
> > https://spark.apache.org/improvement-proposals.html
> >
> > We've got a few changes that have been discussed, either on the
> > list/JIRA (good) or in private (bad -
> > https://incubator.apache.org/guides/committer.html#mailing_lists) that
> > are of a magnitude that they may benefit from some version of this
> > process.  Examples:
> >* The in-progress plan to refactor out connectors and hooks
> > (AIRFLOW-2732)
> >* K8S deployment operator proposal
> >* Initial Design for Supporting fine-grained Connection encryption
> >
> >
> > The benefits of this approach is that the design is hosted somewhere
> > less ephemeral and more editable than email.  It also provides a
> > framework for documenting and confirming consensus through the whole
> > community.
> >
> >What do y'all think?
> >
> > -Jakob
> >
>

Re: Airflow London Meetup Group

[Sid Anand]

Good to hear.. QCon committees start working about 6 months before the
event. So, in October, make sure you reach out to me to let me know.

FYI, I've just started work on the QCon SF committee (Nov 2017)...
-s

On Sat, Jun 2, 2018 at 5:17 AM, Naik Kaxil  wrote:

> I would surely try to give a talk at next Qcon event. And would love to
> see you at the meetup whenever you visit London.
>
> Regards,
> Kaxil
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> <https://maps.google.com/?q=160+Victoria+Street,+Westminster+%0D%0ALondon+SW1E+5LB+-+UK=gmail=g>
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>
> [image: Data Reply]
> --
> *From:* Sid Anand 
> *Sent:* Friday, June 1, 2018 5:57:23 PM
> *To:* dev@airflow.incubator.apache.org
> *Subject:* Re: Airflow London Meetup Group
>
> I usually visit London in March for QCon London... Aside from hoping to see
> an Airflow talk at QCon London someday soon, I'd also like to meet the
> Airflow community in London town. Thx for starting this!
>
> -s
>
> On Fri, Jun 1, 2018 at 9:56 AM, Sid Anand  wrote:
>
> > Woot!
> >
> > I've updated :
> >
> >- https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> >- https://twitter.com/ApacheAirflow/status/1002594373849120769
> >- https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements
>
> >
> >
> > On Fri, Jun 1, 2018 at 5:42 AM, Naik Kaxil  wrote:
> >
> >> We have just created the Airflow London meetup and hope to grow the
> >> Airflow community in London.
> >>
> >> Meetup URL: https://www.meetup.com/London-Apache-Airflow-Meetup/
> >>
> >> Looking forward to meet Airflow users in London.
> >>
> >> Regards,
> >> Kaxil
> >>
> >> On 01/06/2018, 10:31, "Tautvydas" 
> wrote:
> >>
> >> I'd be interested in Airflow meetup as well.
> >>
> >> Cheers,
> >> Tautvydas
> >>
> >> On 1 June 2018 at 08:55, Christian Barra 
> wrote:
> >>
> >> > >
> >> > > Hi there,
> >> >
> >> >
> >> > > >
> >> > > > We(me and my colleagues at Data Reply UK) are planning to
> start
> >> a
> >> > meetup
> >> > > for Airflow users. Me & one of my friend already gave a talk at
> >> PyData
> >> > > London last month and people showed huge interest in it.
> >> > > >
> >> > >
> >> >
> >> > >
> >> > slightly unrelated but this year EuroPython is going to be in
> >> Edinburgh
> >> > (July).
> >> >
> >> > What about organizing a sprint there?
> >> >
> >>
> >>
> >>
> >>
> >>
> >>
> >> Kaxil Naik
> >>
> >> Data Reply
> >> 2nd Floor, Nova South
> >> 160 Victoria Street, Westminster
> <https://maps.google.com/?q=160+Victoria+Street,+Westminster+%0D%0A+London+SW1E+5LB+-+UK=gmail=g>
> >> London SW1E 5LB - UK
> <https://maps.google.com/?q=160+Victoria+Street,+Westminster+%0D%0A+London+SW1E+5LB+-+UK=gmail=g>
> >> phone: +44 (0)20 7730 6000
> >> k.n...@reply.com
> >> www.reply.com
> >>
> >
> >
>
>
>

Adding Kaxil & Arthur as co-mods on our email lists

[Sid Anand]

https://issues.apache.org/jira/browse/INFRA-16621

If others are interested, let me know.
-s

Re: Adding moderators to the email lists for an incubating project

[Sid Anand]

I did look at the committer SVN repo and those instructions seemed to get
me no where.
-s

On Fri, Jun 1, 2018 at 4:28 PM, Sid Anand  wrote:

> Which committers would like to be added as dev and commits list moderators
> for Airflow? If interested, let me know on this thread and I will open an
> infra ticket. I get between 5-10 emails a day that require moderation.
>
> -s
>
> -- Forwarded message --
> From: Jakob Homan 
> Date: Fri, Jun 1, 2018 at 3:49 PM
> Subject: Re: Adding moderators to the email lists for an incubating project
> To: gene...@incubator.apache.org
>
>
> Sid-
>   Adding new moderators is through an infra ticket:
> https://apache.org/dev/committers.html#mailing-list-moderators
>
>I think the resources there for describing who are the current
> moderators may be out of date.
> -jg
>
>
> On 1 June 2018 at 14:55, Sid Anand  wrote:
> > Folks!
> > I seem to be one of the few committers on Airflow's email moderator list.
> > I'd like to add other committers who have voiced interest in moderating
> the
> > lists. How can I add others and view the current list of moderators?
> >
> > -s (Sid)
>
> -
> To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> For additional commands, e-mail: general-h...@incubator.apache.org
>
>
>

Fwd: Adding moderators to the email lists for an incubating project

[Sid Anand]

Which committers would like to be added as dev and commits list moderators
for Airflow? If interested, let me know on this thread and I will open an
infra ticket. I get between 5-10 emails a day that require moderation.

-s

-- Forwarded message --
From: Jakob Homan 
Date: Fri, Jun 1, 2018 at 3:49 PM
Subject: Re: Adding moderators to the email lists for an incubating project
To: gene...@incubator.apache.org


Sid-
  Adding new moderators is through an infra ticket:
https://apache.org/dev/committers.html#mailing-list-moderators

   I think the resources there for describing who are the current
moderators may be out of date.
-jg


On 1 June 2018 at 14:55, Sid Anand  wrote:
> Folks!
> I seem to be one of the few committers on Airflow's email moderator list.
> I'd like to add other committers who have voiced interest in moderating
the
> lists. How can I add others and view the current list of moderators?
>
> -s (Sid)

-
To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
For additional commands, e-mail: general-h...@incubator.apache.org

Re: Airflow London Meetup Group

[Sid Anand]

I usually visit London in March for QCon London... Aside from hoping to see
an Airflow talk at QCon London someday soon, I'd also like to meet the
Airflow community in London town. Thx for starting this!

-s

On Fri, Jun 1, 2018 at 9:56 AM, Sid Anand  wrote:

> Woot!
>
> I've updated :
>
>- https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
>- https://twitter.com/ApacheAirflow/status/1002594373849120769
>- https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements
>
>
> On Fri, Jun 1, 2018 at 5:42 AM, Naik Kaxil  wrote:
>
>> We have just created the Airflow London meetup and hope to grow the
>> Airflow community in London.
>>
>> Meetup URL: https://www.meetup.com/London-Apache-Airflow-Meetup/
>>
>> Looking forward to meet Airflow users in London.
>>
>> Regards,
>> Kaxil
>>
>> On 01/06/2018, 10:31, "Tautvydas"  wrote:
>>
>> I'd be interested in Airflow meetup as well.
>>
>> Cheers,
>> Tautvydas
>>
>> On 1 June 2018 at 08:55, Christian Barra  wrote:
>>
>> > >
>> > > Hi there,
>> >
>> >
>> > > >
>> > > > We(me and my colleagues at Data Reply UK) are planning to start
>> a
>> > meetup
>> > > for Airflow users. Me & one of my friend already gave a talk at
>> PyData
>> > > London last month and people showed huge interest in it.
>> > > >
>> > >
>> >
>> > >
>> > slightly unrelated but this year EuroPython is going to be in
>> Edinburgh
>> > (July).
>> >
>> > What about organizing a sprint there?
>> >
>>
>>
>>
>>
>>
>>
>> Kaxil Naik
>>
>> Data Reply
>> 2nd Floor, Nova South
>> 160 Victoria Street, Westminster
>> London SW1E 5LB - UK
>> phone: +44 (0)20 7730 6000
>> k.n...@reply.com
>> www.reply.com
>>
>
>

Re: Airflow London Meetup Group

[Sid Anand]

Woot!

I've updated :

   - https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
   - https://twitter.com/ApacheAirflow/status/1002594373849120769
   - https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements


On Fri, Jun 1, 2018 at 5:42 AM, Naik Kaxil  wrote:

> We have just created the Airflow London meetup and hope to grow the
> Airflow community in London.
>
> Meetup URL: https://www.meetup.com/London-Apache-Airflow-Meetup/
>
> Looking forward to meet Airflow users in London.
>
> Regards,
> Kaxil
>
> On 01/06/2018, 10:31, "Tautvydas"  wrote:
>
> I'd be interested in Airflow meetup as well.
>
> Cheers,
> Tautvydas
>
> On 1 June 2018 at 08:55, Christian Barra  wrote:
>
> > >
> > > Hi there,
> >
> >
> > > >
> > > > We(me and my colleagues at Data Reply UK) are planning to start a
> > meetup
> > > for Airflow users. Me & one of my friend already gave a talk at
> PyData
> > > London last month and people showed huge interest in it.
> > > >
> > >
> >
> > >
> > slightly unrelated but this year EuroPython is going to be in
> Edinburgh
> > (July).
> >
> > What about organizing a sprint there?
> >
>
>
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>

Re: Moving to Github? Re: Merging PRs, closing Jira tickets (a.k.a New Committer) guide?

[Sid Anand]

;>>> com> wrote:
> >>>>>>
> >>>>>> Ah that would explain why I don't have a button :)
> >>>>>>
> >>>>>> Is this Apache policy, or is it possible for committers to be
> granted
> >>>>> permission to do this? Having this permission would also let us click
> >> the
> >>>>> "rerun tests" button in Travis which would be nice.
> >>>>>>
> >>>>>> Is it worth opening an INFRA ticket asking for this, or is it not
> >>>>> possible?
> >>>>>>
> >>>>>> -ash
> >>>>>>
> >>>>>>> On 6 Mar 2018, at 08:25, Driesprong, Fokko <fo...@driesprong.frl>
> >>>>> wrote:
> >>>>>>>
> >>>>>>> Hi Ash,
> >>>>>>>
> >>>>>>> As a committer we don't have any rights on the Github itself. The
> >> Github
> >>>>>>> repo is just a sync of the apache repo. Unfortunately, therefore we
> >>>>> don't
> >>>>>>> have the right to close any PR.
> >>>>>>>
> >>>>>>> Cheers, Fokko
> >>>>>>>
> >>>>>>> 2018-03-06 0:49 GMT+01:00 Ash Berlin-Taylor <
> >>>>> ash_airflowl...@firemirror.com>
> >>>>>>> :
> >>>>>>>
> >>>>>>>> I've merged two PRs now, and the second one seemed to be better
> >>>>> (though I
> >>>>>>>> did have some trouble with the tool not merging properly and I
> >> needed
> >>>>> to
> >>>>>>>> manually coax git. Hmm)
> >>>>>>>>
> >>>>>>>> Jira: I _think_ that previously I could only comment on Jira
> issues.
> >>>>> With
> >>>>>>>> the new permissions I can now do more (as those of you subscribed
> to
> >>>>> the
> >>>>>>>> commit list will see) -- I started going through old Jira tickets
> >> and
> >>>>>>>> closing ones that are no longer an issue or that were fixed+merged
> >> but
> >>>>> not
> >>>>>>>> closed.
> >>>>>>>>
> >>>>>>>> Github: I don't have a button to close PRs in Github - Is that
> >>>>> expected?
> >>>>>>>>
> >>>>>>>> -ash
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On 4 Mar 2018, at 00:59, Sid Anand <san...@apache.org> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi Ash,
> >>>>>>>>> Welcome aboard.
> >>>>>>>>>
> >>>>>>>>> Firstly, I'm moving this conversation over to the dev list -- the
> >>>>> first
> >>>>>>>>> lesson we all learned at the insistence of the incubator mentors
> >> was
> >>>>> to
> >>>>>>>> use
> >>>>>>>>> the private list for voting and discussion on PMC matters. They
> >>>>> require
> >>>>>>>>> that all information-oriented discussions be routed to the
> >> dev-list.
> >>>>>>>>>
> >>>>>>>>> *Merging PRs:*
> >>>>>>>>> To merge a PR, you can use a very handy tool written by @jlowin
> >>>>>>>>> <https://github.com/jlowin> : https://github.com/apache/
> >>>>>>>>> incubator-airflow/blob/master/dev/airflow-pr
> >>>>>>>>>
> >>>>>>>>> It's pretty self-explanatory. Essentially, you follow the
> >>>>> instructions in
> >>>>>>>>> the tool -- the result is that the PR will be merged to Apache
> >>>>>>>>> <https://git-wip-us.apache.org/repos/asf/incubator-airflow.git>
> >> and
> >>>>> then
> >>>>>>>>> synced (mirrored) to https://github.com/apache/incubator-airflow
> .
> >> The
> >>>>>>>>> Apache mirror runs on apache infrastructure and is the source of
> >> truth
> >>>>>>&

Re: Chicago Airflow user meetup group

[Sid Anand]

The 4th one is not required. You are free to add anyone you wish as admins.
As a best practice, ensure there are enough admins to avoid a dead/unowned
meetup.

-s

On Tue, May 22, 2018 at 9:13 AM, Victor Noagbodji <
vnoagbo...@amplify-nation.com> wrote:

> I am curious about this one. Why should committers be added as admins?
>
> > On May 22, 2018, at 12:08 PM, Sid Anand <san...@apache.org> wrote:
> >
> > That's great news and we welcome the new Meetup. Here are a few pointers:
> >
> >   1. Add your new meet-up on :
> >   https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
> >   2. Add the new Meetup to
> >   https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements &
> >   follow the guidelines on the link above to invite speakers and
> attendees to
> >   your meet-up.
> >  1. Once you have done 1 & 2, we can tweet your new Meetup on the
> >  Apache Airflow twitter account
> >   3. To do 1&2, you need to give me your cwiki id so that I can give your
> >   edit perms
> >   4. Finally, add some of the committers to the meetup as admins
> >  1. You can add me (https://www.meetup.com/members/10527138/) and
> any
> >  other committers that voice interest!
> >
> >
> >
> > -s
> >
> > On Mon, May 21, 2018 at 9:26 PM, Trevor Beuthel <tbeut...@gmail.com>
> wrote:
> >
> >> Hi there,
> >>
> >> My name's Trevor Beuthel and I would like to start an airflow meetup
> group
> >> in Chicago.
> >>
> >> I work at a company called Vibes here in Chicago, and we just rolled
> out a
> >> decent sized migration to airflow that took us a little under a year to
> >> complete. We use many of the built in features of airflow and also
> created
> >> many plugins to satisfy our needs.
> >>
> >> I've been speaking at a few events with Snowflake, and there seems to
> be a
> >> lot of interest in airflow out here. A handful of companies that I've
> >> talked to are actually using it, many are experimenting with it, and
> much
> >> more have heard of it and are interested.
> >>
> >> Let me know if I have your blessing to create the group and the steps I
> >> should take to do so.
> >>
> >> Thanks,
> >> Trevor
> >>
>
>

Re: Chicago Airflow user meetup group

[Sid Anand]

That's great news and we welcome the new Meetup. Here are a few pointers:

   1. Add your new meet-up on :
   https://cwiki.apache.org/confluence/display/AIRFLOW/Meetups
   2. Add the new Meetup to
   https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements &
   follow the guidelines on the link above to invite speakers and attendees to
   your meet-up.
  1. Once you have done 1 & 2, we can tweet your new Meetup on the
  Apache Airflow twitter account
   3. To do 1&2, you need to give me your cwiki id so that I can give your
   edit perms
   4. Finally, add some of the committers to the meetup as admins
  1. You can add me (https://www.meetup.com/members/10527138/) and any
  other committers that voice interest!



-s

On Mon, May 21, 2018 at 9:26 PM, Trevor Beuthel  wrote:

> Hi there,
>
> My name's Trevor Beuthel and I would like to start an airflow meetup group
> in Chicago.
>
> I work at a company called Vibes here in Chicago, and we just rolled out a
> decent sized migration to airflow that took us a little under a year to
> complete. We use many of the built in features of airflow and also created
> many plugins to satisfy our needs.
>
> I've been speaking at a few events with Snowflake, and there seems to be a
> lot of interest in airflow out here. A handful of companies that I've
> talked to are actually using it, many are experimenting with it, and much
> more have heard of it and are interested.
>
> Let me know if I have your blessing to create the group and the steps I
> should take to do so.
>
> Thanks,
> Trevor
>

moderators of dev@airflow.incubator.apache.org?

[Sid Anand]

Mentors,
How do I get the list of moderators of dev@airflow.incubator.apache.org?

I currently moderate the email, but I'm wondering whom else is a moderator
and how I can add a few more committers/PMC members to the moderator list.

-s

Re: 1.10.0beta1 now available for download

[Sid Anand]

Awesome!
-s

On Mon, Apr 23, 2018 at 11:06 AM, Bolke de Bruin  wrote:

> Hi All,
>
> I am really happy that Fokko and I have created the v1-10-test branch and
> subsequently build the first beta of Apache Airflow 1.10!
>
> It is available for testing here:
>
> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0beta1/
>
> Highlights include:
>
> * New RBAC web interface in beta
> * Timezone support
> * First class kubernetes operator
> * Experimental kubernetes executor
> * Documentation improvements
> * Performance optimizations for large DAGs
> * many GCP and S3 integration improvements
> * many new operators
> * many many many bug fixes
>
> We are aiming for a fully compliant Apache release so we should be able to
> kick off the graduation process after this release. I hope you help us out
> getting there!
>
> Kind regards,
>
> Bolke & Fokko

Re: Give it up for Fokko!

[Sid Anand]

+100

On Fri, Apr 13, 2018 at 4:08 PM, Alex Tronchin-James 949-412-7220 <
alex.n.ja...@gmail.com> wrote:

> Bravo!!! Bien fait!
>
> On Fri, Apr 13, 2018 at 3:54 PM Joy Gao  wrote:
>
> >  
> >
> > On Fri, Apr 13, 2018 at 11:47 AM, Naik Kaxil  wrote:
> >
> > > Couldn't agree more. Thanks Fokko
> > >
> > > On 13/04/2018, 17:56, "Maxime Beauchemin"  >
> > > wrote:
> > >
> > > Hey all,
> > >
> > > I wanted to point out the amazing work that Fokko is doing,
> > > reviewing/merging PRs and doing fantastic committer & maintainer
> > work.
> > > It
> > > takes a variety of contributions to make projects like Airflow
> > thrive,
> > > but
> > > without this kind of involvement it wouldn't be possible to keep
> > > shipping
> > > better versions of the product steadily.
> > >
> > > Cheers to that!
> > >
> > > Max
> > >
> > >
> > >
> > >
> > >
> > >
> > > Kaxil Naik
> > >
> > > Data Reply
> > > 38 Grosvenor Gardens
> > > London SW1W 0EB - UK
> > > phone: +44 (0)20 7730 6000 <+44%2020%207730%206000>
> > > k.n...@reply.com
> > > www.reply.com
> > >
> >
>

Re: Slides / Presentations for Airflow

[Sid Anand]

Great question.. currently, we don't.

After meet-ups, usually the host of the meetup sends an email to this DL
with a link to slides and videos -- we update
https://cwiki.apache.org/confluence/display/AIRFLOW/Announcements and share
it on Twitter.

WePay is very good about sharing videos of the Airflow meet-ups they host
within a short time -- I suspect we will be hearing from Joy, Chris R., et
al shortly.

BTW, Apache Beam folks speak very often... maybe on average once or twice a
month.. and all of their talks are not up on their site. It's a best effort
thing.

-s

On Wed, Apr 11, 2018 at 1:56 AM, Naik Kaxil  wrote:

> Hi all,
>
>
>
> Do we maintain slides or presentation materials like Apache Beam does ?
> https://beam.apache.org/contribute/presentation-materials/
>
>
> Or if anyone has just presented somewhere can I get the link to slides and
> videos if possible?
>
>
>
> Regards,
>
> Kaxil
>
>
> Kaxil Naik
>
> Data Reply
> 38 Grosvenor Gardens
> 
> London SW1W 0EB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>
> [image: Data Reply]
>

Re: Readthedocs

[Sid Anand]

Grazzi Art!
-s

On Tue, Apr 10, 2018 at 3:40 PM, Arthur Wiedmer <arthur.wied...@gmail.com>
wrote:

> Done, Sid!
>
> On Tue, Apr 10, 2018 at 3:34 PM, Sid Anand <san...@apache.org> wrote:
>
> > Kaxil,
> > The same request for you. My user id is r39132.
> >
> > -s
> >
> > On Tue, Apr 10, 2018 at 5:21 AM, Driesprong, Fokko <fo...@driesprong.frl
> >
> > wrote:
> >
> > > Thanks Kaxil!
> > >
> > > Cheers, Fokko
> > >
> > > 2018-04-10 14:19 GMT+02:00 Naik Kaxil <k.n...@reply.com>:
> > >
> > > > Added you.
> > > >
> > > > https://readthedocs.org/projects/airflow/
> > > >
> > > > Regards,
> > > > Kaxil
> > > >
> > > >
> > > > On 10/04/2018, 13:04, "fo...@driesprongen.nl on behalf of
> Driesprong,
> > > > Fokko" <fo...@driesprongen.nl on behalf of fo...@driesprong.frl>
> > wrote:
> > > >
> > > > Thanks Kaxil,
> > > >
> > > > Thanks for the quick response! My username is Fokko
> > > >
> > > > Cheers, Fokko
> > > >
> > > > 2018-04-10 12:56 GMT+02:00 Naik Kaxil <k.n...@reply.com>:
> > > >
> > > > > Additionally, I have also updated the docs to reflect the
> latest
> > > > change.
> > > > >
> > > > > https://airflow.readthedocs.io/en/latest/index.html
> > > > >
> > > > > https://airflow.readthedocs.io/en/latest/start.html
> > > > >
> > > > > Regards,
> > > > > Kaxil
> > > > >
> > > > > On 10/04/2018, 11:54, "Naik Kaxil" <k.n...@reply.com> wrote:
> > > > >
> > > > > Can you create an account on readthedocs and give me your
> > user
> > > > id?
> > > > > I'll add you to it.
> > > > >
> > > > > Also,
> > > > >
> > > > > You can run the "sphinx-build -b html .
> ~/Desktop/airflowdoc"
> > > > command
> > > > > on your local machine to generate the docs and
> > > > "incubator-airflow/docs/_build/html'
> > > > > directory to see the generated docs.
> > > > >
> > > > > Regards,
> > > > > Kaxil
> > > > >
> > > > >
> > > > > On 10/04/2018, 08:05, "fo...@driesprongen.nl on behalf of
> > > > Driesprong,
> > > > > Fokko" <fo...@driesprongen.nl on behalf of
> fo...@driesprong.frl>
> > > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Would it be possible to give me access to the
> Readthedocs
> > > > > environment. I'm
> > > > > working on improving the docs a bit,
> > > > > <https://github.com/apache/incubator-airflow/pull/3201
> >
> > > but
> > > > I
> > > > > can't see the
> > > > > output of the docs generation.
> > > > >
> > > > > Cheers, Fokko
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Kaxil Naik
> > > > >
> > > > > Data Reply
> > > > > 38 Grosvenor Gardens
> > > > > London SW1W 0EB - UK
> > > > > phone: +44 (0)20 7730 6000
> > > > > k.n...@reply.com
> > > > > www.reply.com
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Kaxil Naik
> > > > >
> > > > > Data Reply
> > > > > 38 Grosvenor Gardens
> > > > > London SW1W 0EB - UK
> > > > > phone: +44 (0)20 7730 6000
> > > > > k.n...@reply.com
> > > > > www.reply.com
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Kaxil Naik
> > > >
> > > > Data Reply
> > > > 38 Grosvenor Gardens
> > > > London SW1W 0EB - UK
> > > > phone: +44 (0)20 7730 6000
> > > > k.n...@reply.com
> > > > www.reply.com
> > > >
> > >
> >
>

Re: Readthedocs

[Sid Anand]

Kaxil,
The same request for you. My user id is r39132.

-s

On Tue, Apr 10, 2018 at 5:21 AM, Driesprong, Fokko 
wrote:

> Thanks Kaxil!
>
> Cheers, Fokko
>
> 2018-04-10 14:19 GMT+02:00 Naik Kaxil :
>
> > Added you.
> >
> > https://readthedocs.org/projects/airflow/
> >
> > Regards,
> > Kaxil
> >
> >
> > On 10/04/2018, 13:04, "fo...@driesprongen.nl on behalf of Driesprong,
> > Fokko"  wrote:
> >
> > Thanks Kaxil,
> >
> > Thanks for the quick response! My username is Fokko
> >
> > Cheers, Fokko
> >
> > 2018-04-10 12:56 GMT+02:00 Naik Kaxil :
> >
> > > Additionally, I have also updated the docs to reflect the latest
> > change.
> > >
> > > https://airflow.readthedocs.io/en/latest/index.html
> > >
> > > https://airflow.readthedocs.io/en/latest/start.html
> > >
> > > Regards,
> > > Kaxil
> > >
> > > On 10/04/2018, 11:54, "Naik Kaxil"  wrote:
> > >
> > > Can you create an account on readthedocs and give me your user
> > id?
> > > I'll add you to it.
> > >
> > > Also,
> > >
> > > You can run the "sphinx-build -b html . ~/Desktop/airflowdoc"
> > command
> > > on your local machine to generate the docs and
> > "incubator-airflow/docs/_build/html'
> > > directory to see the generated docs.
> > >
> > > Regards,
> > > Kaxil
> > >
> > >
> > > On 10/04/2018, 08:05, "fo...@driesprongen.nl on behalf of
> > Driesprong,
> > > Fokko" 
> > wrote:
> > >
> > > Hi all,
> > >
> > > Would it be possible to give me access to the Readthedocs
> > > environment. I'm
> > > working on improving the docs a bit,
> > > 
> but
> > I
> > > can't see the
> > > output of the docs generation.
> > >
> > > Cheers, Fokko
> > >
> > >
> > >
> > >
> > >
> > >
> > > Kaxil Naik
> > >
> > > Data Reply
> > > 38 Grosvenor Gardens
> > > London SW1W 0EB - UK
> > > phone: +44 (0)20 7730 6000
> > > k.n...@reply.com
> > > www.reply.com
> > >
> > >
> > >
> > >
> > >
> > >
> > > Kaxil Naik
> > >
> > > Data Reply
> > > 38 Grosvenor Gardens
> > > London SW1W 0EB - UK
> > > phone: +44 (0)20 7730 6000
> > > k.n...@reply.com
> > > www.reply.com
> > >
> >
> >
> >
> >
> >
> >
> > Kaxil Naik
> >
> > Data Reply
> > 38 Grosvenor Gardens
> > London SW1W 0EB - UK
> > phone: +44 (0)20 7730 6000
> > k.n...@reply.com
> > www.reply.com
> >
>