Re: Graduation resolution passed - Airflow is a TLP

[Kaxil Naik]

Congratulations all.

On Thu, Dec 20, 2018, 21:41 Driesprong, Fokko  Awesome! Congrats!
>
> Cheers, Fokko
>
> Op do 20 dec. 2018 om 22:40 schreef Sid Anand 
>
> > YaaY!
> >
> > -s
> >
> > On Thu, Dec 20, 2018 at 1:13 PM Jakob Homan  wrote:
> >
> > > Hey all-
> > >The Board minutes haven't been published yet (probably due to
> > > Holiday-related slowness), but I can see through the admin tool that
> > > our Graduation resolution was approved yesterday at the meeting.
> > > Airflow is the 199th current active Top Level Project in Apache.
> > >
> > > Congrats all.
> > >
> > > -Jakob
> > >
> >
>

Re: Requesting Wiki edit permission to create AIPs

[Kaxil Naik]

Sid would probably need to grant access.



On Thu, Dec 20, 2018, 05:28 Tao Feng  I am not an Airflow wiki admin. Could any of the admin for wiki help this
> request?
>
> On Wed, Dec 19, 2018 at 5:58 PM Tim Swast 
> wrote:
>
> > Wiki ID: tswast (https://cwiki.apache.org/confluence/display/~tswast)
> >
> > Per
> >
> >
> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals
> >
> > *  •  **Tim Swast*
> > *  •  *Software Friendliness Engineer
> > *  •  *Google Cloud Developer Relations
> > *  •  *Seattle, WA, USA
> >
>

Re: Call for fixes for Airflow 1.10.2

[Kaxil Naik]

Sure, I am waiting for final comments from Ash on it, if he has any. If not
I will pick it up and merge to master + cherry-pick to the test branch

On Thu, Dec 13, 2018 at 2:51 PM Thomas Brockmeier <
thomas.brockme...@randstadgroep.nl> wrote:

> Hi Kaxil,
>
> If possible, I would like to get AIRFLOW-1552 in:
> https://issues.apache.org/jira/browse/AIRFLOW-1552
> https://github.com/apache/incubator-airflow/pull/4276
>
> Thanks,
> Thomas
>
> On 2018/11/28 22:40:55, Kaxil Naik  wrote:
> > Hi everyone,>
> >
> > I'm starting the process of gathering fixes for a 1.10.2. So far the
> list>
> > of issues I have that we should pull in are>
> > *
>
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> >
>
> > <
>
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> >*>
>
> >
> > I will start pushing these as cherry-picked commits to the v1-10-test>
> > branch today.>
> >
> > *Kaxil Naik*>
> > *Big Data Consultant *@ *Data Reply UK*>
> > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> Neo4j>
> > Developer>
> > *Phone: *+44 (0) 74820 88992>
> > *LinkedIn*: https://www.linkedin.com/in/kaxil>
> >
>
> --
> The information contained in this e-mail communication is solely intended
> for the person/legal person to whom it has been sent, and as it may
> contain
> information of a personal or confidential nature, it may not be made
> public
> by virtue of law, regulations or agreement. If someone other than the
> intended recipient should receive or come into possession of this e-mail
> communication, he/she will not be entitled to read, disseminate, disclose
> or duplicate it. If you are not the intended recipient, you are requested
> to inform the sender of this e-mail message of this immediately, and to
> destroy the original e-mail communication. Neither Randstad Holding nv nor
> its subsidiaries accept any liability for incorrect and incomplete
> transmission or delayed receipt of this e-mail.
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: [RESULT] Graduate Apache Airflow as a TLP

[Kaxil Naik]

Awesome, thanks @Jakob, that is a great news. A good Christmas present ;)

On Sun, Dec 9, 2018, 19:18 Sid Anand  Hi Folks!
> I just returned from a 2-week speaking tour of Australia -- such a
> beautiful country! Anyway, what wonderful news to return to!
>
> Great work team!
> -s
>
> On Sun, Dec 9, 2018 at 11:06 AM Bolke de Bruin  wrote:
>
> > Woohoo :-)
> >
> > Thanks Jakob!
> >
> >
> > Op zo 9 dec. 2018 20:01 schreef Jakob Homan  >
> > > The VOTE passed in Incubator.  I'll add it the resolution to the Board
> > > agenda, who meet this month on the 19th.  They'll approve and Airflow
> > > will be a TLP.
> > >
> > > -Jakob
> > > On Wed, Dec 5, 2018 at 3:31 PM Jakob Homan  wrote:
> > > >
> > > > Been traveling today.  Just started the VOTE over in the general@ in
> > > Incubator.
> > > >
> > > > -jg
> > > > On Wed, Dec 5, 2018 at 1:26 PM Kaxil Naik 
> wrote:
> > > > >
> > > > > Hi Jakob,
> > > > >
> > > > > Did you raise this with IPMC, can we track it somewhere?
> > > > >
> > > > > Excited for graduation :-)
> > > > >
> > > > > Regards,
> > > > > Kaxil
> > > > >
> > > > > On Tue, Dec 4, 2018, 20:04 Ash Berlin-Taylor  wrote:
> > > > >>
> > > > >> Missed my vote off that list :)
> > > > >>
> > > > >> > On 4 Dec 2018, at 18:17, Jakob Homan  wrote:
> > > > >> >
> > > > >> > I neglected to add my binding +1, so I'll do so now.
> > > > >> >
> > > > >> > With three days having elapsed, the VOTE is concluded
> > successfully.
> > > > >> >
> > > > >> > Overall: 20 x +1 votes, 0 x -1 votes
> > > > >> >
> > > > >> > Binding +1 x 10: Kaxil, Tao, Bolke, Fokko, Maxime, Arthur,
> Hitesh,
> > > > >> > Chris, Sid, Jakob.
> > > > >> > Non-binding +1 x 10: Daniel, Shah, Stefan, Kevin, Marc, Sunil,
> > > > >> > Adityan, Deng, Neelesh, Sai
> > > > >> >
> > > > >> > I'll use this result to start the corresponding VOTE on the
> IPMC.
> > > I'm
> > > > >> > at an offsite today, so I have limited email time.  Likely will
> > open
> > > > >> > the VOTE this evening.
> > > > >> >
> > > > >> > Thanks everyone.
> > > > >> > -Jakob
> > > > >> >
> > > > >> >
> > > > >> > On Tue, Dec 4, 2018 at 6:03 AM Bolke de Bruin <
> bdbr...@gmail.com>
> > > wrote:
> > > > >> >>
> > > > >> >> Shall we close the vote? @jakob?
> > > > >> >>
> > > > >> >>> On 2 Dec 2018, at 13:08, Sid Anand  wrote:
> > > > >> >>>
> > > > >> >>> +1 binding
> > > > >> >>>
> > > > >> >>> Woot! Thanks to all for this happy day!
> > > > >> >>> -s
> > > > >> >>>
> > > > >> >>> On Sun, Dec 2, 2018 at 1:25 AM Sai Phanindhra <
> > > phani8...@gmail.com> wrote:
> > > > >> >>>
> > > > >> >>>> +1 (non binding)
> > > > >> >>>>
> > > > >> >>>> Excited to see this happenning.
> > > > >> >>>>
> > > > >> >>>> On Sat 1 Dec, 2018, 20:35  > > > >> >>>>
> > > > >> >>>>> +1 (binding)!
> > > > >> >>>>>
> > > > >> >>>>> On 30 November 2018 21:33:14 GMT, Jakob Homan <
> > > jgho...@gmail.com> wrote:
> > > > >> >>>>>> Hey all!
> > > > >> >>>>>>
> > > > >> >>>>>> Following a very successful DISCUSS[1] regarding graduating
> > > Airflow to
> > > > >> >>>>>> Top Level Project (TLP) status, I'm starting the official
> > VOTE.
> > > > >> >>>>>>
> > > > >> >>>>>> Since entering the Incubator in 2016, the community has:
> > > > >> >>&g

Re: [RESULT] Graduate Apache Airflow as a TLP

[Kaxil Naik]

Hi Jakob,

Did you raise this with IPMC, can we track it somewhere?

Excited for graduation :-)

Regards,
Kaxil

On Tue, Dec 4, 2018, 20:04 Ash Berlin-Taylor  Missed my vote off that list :)
>
> > On 4 Dec 2018, at 18:17, Jakob Homan  wrote:
> >
> > I neglected to add my binding +1, so I'll do so now.
> >
> > With three days having elapsed, the VOTE is concluded successfully.
> >
> > Overall: 20 x +1 votes, 0 x -1 votes
> >
> > Binding +1 x 10: Kaxil, Tao, Bolke, Fokko, Maxime, Arthur, Hitesh,
> > Chris, Sid, Jakob.
> > Non-binding +1 x 10: Daniel, Shah, Stefan, Kevin, Marc, Sunil,
> > Adityan, Deng, Neelesh, Sai
> >
> > I'll use this result to start the corresponding VOTE on the IPMC.  I'm
> > at an offsite today, so I have limited email time.  Likely will open
> > the VOTE this evening.
> >
> > Thanks everyone.
> > -Jakob
> >
> >
> > On Tue, Dec 4, 2018 at 6:03 AM Bolke de Bruin  wrote:
> >>
> >> Shall we close the vote? @jakob?
> >>
> >>> On 2 Dec 2018, at 13:08, Sid Anand  wrote:
> >>>
> >>> +1 binding
> >>>
> >>> Woot! Thanks to all for this happy day!
> >>> -s
> >>>
> >>> On Sun, Dec 2, 2018 at 1:25 AM Sai Phanindhra 
> wrote:
> >>>
> >>>> +1 (non binding)
> >>>>
> >>>> Excited to see this happenning.
> >>>>
> >>>> On Sat 1 Dec, 2018, 20:35  >>>>
> >>>>> +1 (binding)!
> >>>>>
> >>>>> On 30 November 2018 21:33:14 GMT, Jakob Homan 
> wrote:
> >>>>>> Hey all!
> >>>>>>
> >>>>>> Following a very successful DISCUSS[1] regarding graduating Airflow
> to
> >>>>>> Top Level Project (TLP) status, I'm starting the official VOTE.
> >>>>>>
> >>>>>> Since entering the Incubator in 2016, the community has:
> >>>>>> * successfully produced 7 releases
> >>>>>> * added 9 new committers/PPMC members
> >>>>>> * built a diverse group of committers from multiple different
> employers
> >>>>>> * had more than 3,300 JIRA tickets opened
> >>>>>> * completed the project maturity model with positive responses[2]
> >>>>>>
> >>>>>> Accordingly, I believe we're ready to graduate and am calling a VOTE
> >>>>>> on the following graduation resolution.  This VOTE will remain open
> >>>>>> for at least 72 hours.  If successful, the resolution will be
> >>>>>> forwarded to the IPMC for its consideration.  If that VOTE is
> >>>>>> successful, the resolution will be voted upon by the Board at its
> next
> >>>>>> monthly meeting.
> >>>>>>
> >>>>>> Everyone is encouraged to vote, even if their vote is not binding.
> >>>>>> We've built a nice community here, let's make sure everyone has
> their
> >>>>>> voice heard.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Jakob
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>>
> >>>>
> https://lists.apache.org/thread.html/%3c0a763b0b-7d0d-4353-979a-ac6769eb0...@gmail.com%3E
> >>>>>> [2]
> >>>>>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
> >>>>>>
> >>>>>> 
> >>>>>>
> >>>>>> Establish the Apache Airflow Project
> >>>>>>
> >>>>>> WHEREAS, the Board of Directors deems it to be in the best
> >>>>>> interests of the Foundation and consistent with the
> >>>>>> Foundation's purpose to establish a Project Management
> >>>>>> Committee charged with the creation and maintenance of
> >>>>>> open-source software, for distribution at no charge to
> >>>>>> the public, related to workflow automation and scheduling
> >>>>>> that can be used to author and manage data pipelines.
> >>>>>>
> >>>>>> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> >>>>>> Committee (PMC), to be known as the "Apache Airflow Project",
> >>>>>> be and hereby is established pursuant to By

Re: Call for fixes for Airflow 1.10.2

[Kaxil Naik]

Added a comment on the PR. Happy to merge it once that is addressed :)

On Thu, Nov 29, 2018 at 10:47 PM Kevin Yang  wrote:

> Hi Kaxil,
> Ty for running 1.10.2, that's pretty fast. Can you please also include the
> following bug fix in 1.10.2 please?
>
> AIRFLOW-3414 <https://issues.apache.org/jira/browse/AIRFLOW-3414>: Fix
> reload_module in DagFileProcessorAgent
>
> Cheers,
> Kevin Y
>
> On Wed, Nov 28, 2018 at 2:41 PM Kaxil Naik  wrote:
>
> > Hi everyone,
> >
> > I'm starting the process of gathering fixes for a 1.10.2. So far the list
> > of issues I have that we should pull in are
> > *
> >
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> > <
> >
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> > >*
> >
> > I will start pushing these as cherry-picked commits to the v1-10-test
> > branch today.
> >
> > *Kaxil Naik*
> > *Big Data Consultant *@ *Data Reply UK*
> > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> > Developer
> > *Phone: *+44 (0) 74820 88992
> > *LinkedIn*: https://www.linkedin.com/in/kaxil
> >
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: Call for fixes for Airflow 1.10.2

[Kaxil Naik]

Added.

On Sat, Dec 1, 2018 at 2:25 AM Deng Xiaodong  wrote:

> Thanks Kaxil.
>
> Not sure if 1.10.2 is focusing on fixing 1.10.0/1.10.1, or also open for
> new patches/features?
>
> The current `airflow flower` (until 1.10.1) doesn’t come with any
> authentication. This may make essential information exposed to unintended
> users in an untrusted environment.
>
> Earlier I submitted a PR to add HTTP basic authentication for Flower in
> Airflow. This feature helps enhance the the whole application slightly in
> terms of security.
>
> Please refer to:
> PR 4166 (merged in master):
> https://github.com/apache/incubator-airflow/pull/4166 <
> https://github.com/apache/incubator-airflow/pull/4166>
> JIRA: https://issues.apache.org/jira/browse/AIRFLOW-3323 <
> https://issues.apache.org/jira/browse/AIRFLOW-3323>
>
> Thanks.
>
>
> XD
>
>
> > On 30 Nov 2018, at 6:47 AM, Kevin Yang  wrote:
> >
> > Hi Kaxil,
> > Ty for running 1.10.2, that's pretty fast. Can you please also include
> the
> > following bug fix in 1.10.2 please?
> >
> > AIRFLOW-3414 <https://issues.apache.org/jira/browse/AIRFLOW-3414>: Fix
> > reload_module in DagFileProcessorAgent
> >
> > Cheers,
> > Kevin Y
> >
> > On Wed, Nov 28, 2018 at 2:41 PM Kaxil Naik  wrote:
> >
> >> Hi everyone,
> >>
> >> I'm starting the process of gathering fixes for a 1.10.2. So far the
> list
> >> of issues I have that we should pull in are
> >> *
> >>
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> >> <
> >>
> https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
> >>> *
> >>
> >> I will start pushing these as cherry-picked commits to the v1-10-test
> >> branch today.
> >>
> >> *Kaxil Naik*
> >> *Big Data Consultant *@ *Data Reply UK*
> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> Neo4j
> >> Developer
> >> *Phone: *+44 (0) 74820 88992
> >> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>
>
>

-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: [DISCUSS] Apache Airflow graduation from the incubator

[Kaxil Naik]

;> >>
> > > >> >> We can use those security lists (though having our own would be
> nice) -
> > > >> >> either way we will need to make this prominent in the docs.
> > > >> >>
> > > >> >> Couple of points
> > > >> >>
> > > >> >> CS10: that github link is only visible to members of the team
> > > >> >>
> > > >> >> CD30: probably good as it is, we may want to do
> > > >> >> https://issues.apache.org/jira/browse/AIRFLOW-3400 <
> > > >> >> https://issues.apache.org/jira/browse/AIRFLOW-3400> to remove
> the last
> > > >> >> niggle of the GPL env var at install time (but not a hard
> requirement,
> > > >> just
> > > >> >> nice)
> > > >> >>
> > > >> >> -ash
> > > >> >>
> > > >> >>> On 26 Nov 2018, at 21:10, Stefan Seelmann <
> m...@stefan-seelmann.de>
> > > >> >> wrote:
> > > >> >>>
> > > >> >>> I agree that Apache Airflow should graduate.
> > > >> >>>
> > > >> >>> I'm only involved since beginning of this year, but the project
> did
> > > >> two
> > > >> >>> releases during that time, once TLP releasing becomes easier :)
> > > >> >>>
> > > >> >>> Regarding QU30 you may consider to use the ASF wide security
> mailing
> > > >> >>> list [3] and process [4].
> > > >> >>>
> > > >> >>> Kind Regards,
> > > >> >>> Stefan
> > > >> >>>
> > > >> >>> [3] https://www.apache.org/security/
> > > >> >>> [4] https://www.apache.org/security/committers.html
> > > >> >>>
> > > >> >>>
> > > >> >>> On 11/26/18 8:46 PM, Bolke de Bruin wrote:
> > > >> >>>> Ping!
> > > >> >>>>
> > > >> >>>> Sent from my iPhone
> > > >> >>>>
> > > >> >>>>> On 24 Nov 2018, at 12:57, Bolke de Bruin 
> wrote:
> > > >> >>>>>
> > > >> >>>>> Hi All,
> > > >> >>>>>
> > > >> >>>>> With the Apache Airflow community healthy and growing, I
> think now
> > > >> >> would be a good time to
> > > >> >>>>> discuss where we stand regarding to graduation from the
> Incubator,
> > > >> and
> > > >> >> what requirements remains.
> > > >> >>>>>
> > > >> >>>>> Apache Airflow entered incubation around 2 years ago, since
> then,
> > > >> the
> > > >> >> Airflow community learned
> > > >> >>>>> a lot about how to do things in Apache ways. Now we are a very
> > > >> helpful
> > > >> >> and engaged community,
> > > >> >>>>> ready to help on all questions from the Airflow community. We
> > > >> >> delivered multiple releases that have
> > > >> >>>>> been increasing in quality ever since, now we can do
> self-driving
> > > >> >> releases in good cadence.
> > > >> >>>>>
> > > >> >>>>> The community is growing, new committers and PPMC members keep
> > > >> >> joining. We addressed almost all
> > > >> >>>>> the maturity issues stipulated by Apache Project Maturity
> Model [1].
> > > >> >> So final requirements remain, but
> > > >> >>>>> those just need a final nudge. Committers and contributors are
> > > >> invited
> > > >> >> to verify the list and pick up the last
> > > >> >>>>> bits (QU30, CO50). Finally (yahoo!) all the License and IP
> issues we
> > > >> >> can see got resolved.
> > > >> >>>>>
> > > >> >>>>> Base on those, I believes it's time for us to graduate to
> TLP. [2]
> > > >> Any
> > > >> >> thoughts?
> > > >> >>>>> And welcome advice from Airflow Mentors?
> > > >> >>>>>
> > > >> >>>>> Thanks,
> > > >> >>>>>
> > > >> >>>>> [1]
> > > >> >>
> > > >>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
> > > >> >>>>> [2]
> > > >> >>
> > > >>
> https://incubator.apache.org/guides/graduation.html#graduating_to_a_top_level_project
> > > >> >> Regards,
> > > >> >>>
> > > >> >>
> > > >> >>
> > > >>
> > > >>
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: [VOTE] Graduate the Apache Airflow as a TLP

[Kaxil Naik]

+1 (binding)

On Fri, Nov 30, 2018 at 9:39 PM Daniel Huang  wrote:

> +1 (non-binding) Thanks for everyone's hard work!
>
> On Fri, Nov 30, 2018 at 1:33 PM Jakob Homan  wrote:
>
> > Hey all!
> >
> > Following a very successful DISCUSS[1] regarding graduating Airflow to
> > Top Level Project (TLP) status, I'm starting the official VOTE.
> >
> > Since entering the Incubator in 2016, the community has:
> >* successfully produced 7 releases
> >* added 9 new committers/PPMC members
> >* built a diverse group of committers from multiple different
> employers
> >* had more than 3,300 JIRA tickets opened
> >* completed the project maturity model with positive responses[2]
> >
> > Accordingly, I believe we're ready to graduate and am calling a VOTE
> > on the following graduation resolution.  This VOTE will remain open
> > for at least 72 hours.  If successful, the resolution will be
> > forwarded to the IPMC for its consideration.  If that VOTE is
> > successful, the resolution will be voted upon by the Board at its next
> > monthly meeting.
> >
> > Everyone is encouraged to vote, even if their vote is not binding.
> > We've built a nice community here, let's make sure everyone has their
> > voice heard.
> >
> > Thanks,
> > Jakob
> >
> > [1]
> >
> https://lists.apache.org/thread.html/%3c0a763b0b-7d0d-4353-979a-ac6769eb0...@gmail.com%3E
> > [2]
> > https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
> >
> > 
> >
> > Establish the Apache Airflow Project
> >
> > WHEREAS, the Board of Directors deems it to be in the best
> > interests of the Foundation and consistent with the
> > Foundation's purpose to establish a Project Management
> > Committee charged with the creation and maintenance of
> > open-source software, for distribution at no charge to
> > the public, related to workflow automation and scheduling
> > that can be used to author and manage data pipelines.
> >
> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > Committee (PMC), to be known as the "Apache Airflow Project",
> > be and hereby is established pursuant to Bylaws of the
> > Foundation; and be it further
> >
> > RESOLVED, that the Apache Airflow Project be and hereby is
> > responsible for the creation and maintenance of software
> > related to workflow automation and scheduling that can be
> > used to author and manage data pipelines; and be it further
> >
> > RESOLVED, that the office of "Vice President, Apache Airflow" be
> > and hereby is created, the person holding such office to
> > serve at the direction of the Board of Directors as the chair
> > of the Apache Airflow Project, and to have primary responsibility
> > for management of the projects within the scope of
> > responsibility of the Apache Airflow Project; and be it further
> >
> > RESOLVED, that the persons listed immediately below be and
> > hereby are appointed to serve as the initial members of the
> > Apache Airflow Project:
> >
> > * Alex Guziel 
> > * Alex Van Boxel 
> > * Arthur Wiedmer 
> > * Ash Berlin-Taylor 
> > * Bolke de Bruin 
> > * Chris Riccomini 
> > * Dan Davydov 
> > * Fokko Driesprong 
> > * Hitesh Shah 
> > * Jakob Homan 
> > * Jeremiah Lowin 
> > * Joy Gao 
> > * Kaxil Naik 
> > * Maxime Beauchemin 
> > * Siddharth Anand 
> > * Sumit Maheshwari 
> > * Tao Feng 
> >
> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Bolke de Bruin
> > be appointed to the office of Vice President, Apache Airflow, to
> > serve in accordance with and subject to the direction of the
> > Board of Directors and the Bylaws of the Foundation until
> > death, resignation, retirement, removal or disqualification,
> > or until a successor is appointed; and be it further
> >
> > RESOLVED, that the initial Apache Airflow PMC be and hereby is
> > tasked with the creation of a set of bylaws intended to
> > encourage open development and increased participation in the
> > Apache Airflow Project; and be it further
> >
> > RESOLVED, that the Apache Airflow Project be and hereby
> > is tasked with the migration and rationalization of the Apache
> > Incubator Airflow podling; and be it further
> >
> > RESOLVED, that all responsibilities pertaining to the Apache
> > Incubator Airflow podling encumbered upon the Apache Incubator
> > Project are hereafter discharged.
> >
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Call for fixes for Airflow 1.10.2

[Kaxil Naik]

Hi everyone,

I'm starting the process of gathering fixes for a 1.10.2. So far the list
of issues I have that we should pull in are
*https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2
<https://issues.apache.org/jira/browse/AIRFLOW-3384?jql=project%20%3D%20AIRFLOW%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%201.10.2>*

I will start pushing these as cherry-picked commits to the v1-10-test
branch today.

*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: [DISCUSS] Apache Airflow graduation from the incubator

[Kaxil Naik]

I have updated http://incubator.apache.org/projects/airflow.html with news
(all our announcements since 2016), added links, add links to incubation
status reports.

@Jakob Homan  - Let me know if we need to update
anything else.

Regards,
Kaxil

On Wed, Nov 28, 2018 at 8:00 PM Kaxil Naik  wrote:

> Definitely willing to pick up the paperwork at
> http://incubator.apache.org/projects/airflow.html .
>
> @Jakob Homan   Is there any guide on what needs to be
> filled out?
>
>
>
> On Wed, Nov 28, 2018 at 6:04 PM Bolke de Bruin  wrote:
>
>> Ping!
>>
>> Verstuurd vanaf mijn iPad
>>
>> > Op 27 nov. 2018 om 21:39 heeft Bolke de Bruin  het
>> volgende geschreven:
>> >
>> > Hi Folks,
>> >
>> > Thanks all for your responses and particularly Stefan for his
>> suggestion to use the generic Apache way to handle security issues. This
>> seems to be an accepted way for more projects, so I have added this to the
>> maturity evaluation[1] and marked is as resolved. While handling the GPL
>> library can be nicer we are already in compliance with CD30, so @Fokko and
>> @Ash if you want to help out towards graduation please spend your time
>> elsewhere like fixing CO50. This means adding a page to confluence that
>> describes how to become a committer on the project. As we are following
>> Apache many examples of other projects are around[2]
>> >
>> > Then there is the paperwork[3] as referred to by Jakob. This mainly
>> concerns filling in some items, maybe here and there creation some
>> documentation but I don't think much. @Kaxil, @Tao: are you willing to pick
>> this up? @Sid can you share how to edit that page?
>> >
>> > If we have resolved these items in my opinion we can start the voting
>> here and at the IPMC thereafter, targeting the board meeting of January for
>> graduation. How’s that for a New Year’s resolution?
>> >
>> > Cheers!
>> > Bolke
>> >
>> > P.S. Would it be nice to have updated graduation web page? Maybe one of
>> the contributors/community members likes to take a stab at this[4]
>> >
>> > [1]
>> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
>> > [2]
>> https://cwiki.apache.org/confluence/display/HAWQ/Becoming+a+committer
>> > [3] http://incubator.apache.org/projects/airflow.html
>> > [4] https://airflow.apache.org/
>> >
>> >
>> >
>> >> On 27 Nov 2018, at 16:32, Driesprong, Fokko 
>> wrote:
>> >>
>> >> +1 from my side. Would be awesome to graduate Airflow
>> >>
>> >> If time allows, I'll also dive into CD30.
>> >>
>> >> Cheers, Fokko
>> >>
>> >> Op di 27 nov. 2018 om 16:21 schreef Ash Berlin-Taylor > >:
>> >>
>> >>> Oarsome Bolke, thanks for starting this.
>> >>>
>> >>> It looks like we are closer than I thought!
>> >>>
>> >>> We can use those security lists (though having our own would be nice)
>> -
>> >>> either way we will need to make this prominent in the docs.
>> >>>
>> >>> Couple of points
>> >>>
>> >>> CS10: that github link is only visible to members of the team
>> >>>
>> >>> CD30: probably good as it is, we may want to do
>> >>> https://issues.apache.org/jira/browse/AIRFLOW-3400 <
>> >>> https://issues.apache.org/jira/browse/AIRFLOW-3400> to remove the
>> last
>> >>> niggle of the GPL env var at install time (but not a hard
>> requirement, just
>> >>> nice)
>> >>>
>> >>> -ash
>> >>>
>> >>>> On 26 Nov 2018, at 21:10, Stefan Seelmann 
>> >>> wrote:
>> >>>>
>> >>>> I agree that Apache Airflow should graduate.
>> >>>>
>> >>>> I'm only involved since beginning of this year, but the project did
>> two
>> >>>> releases during that time, once TLP releasing becomes easier :)
>> >>>>
>> >>>> Regarding QU30 you may consider to use the ASF wide security mailing
>> >>>> list [3] and process [4].
>> >>>>
>> >>>> Kind Regards,
>> >>>> Stefan
>> >>>>
>> >>>> [3] https://www.apache.org/security/
>> >>>> [4] https://www.apache.org/security/committers.html
>> >>>>
>> >>>>
>

Re: [DISCUSS] Apache Airflow graduation from the incubator

[Kaxil Naik]

flow entered incubation around 2 years ago, since then,
> the
> >>> Airflow community learned
> >>>>>> a lot about how to do things in Apache ways. Now we are a very
> helpful
> >>> and engaged community,
> >>>>>> ready to help on all questions from the Airflow community. We
> >>> delivered multiple releases that have
> >>>>>> been increasing in quality ever since, now we can do self-driving
> >>> releases in good cadence.
> >>>>>>
> >>>>>> The community is growing, new committers and PPMC members keep
> >>> joining. We addressed almost all
> >>>>>> the maturity issues stipulated by Apache Project Maturity Model [1].
> >>> So final requirements remain, but
> >>>>>> those just need a final nudge. Committers and contributors are
> invited
> >>> to verify the list and pick up the last
> >>>>>> bits (QU30, CO50). Finally (yahoo!) all the License and IP issues we
> >>> can see got resolved.
> >>>>>>
> >>>>>> Base on those, I believes it's time for us to graduate to TLP. [2]
> Any
> >>> thoughts?
> >>>>>> And welcome advice from Airflow Mentors?
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> [1]
> >>>
> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
> >>>>>> [2]
> >>>
> https://incubator.apache.org/guides/graduation.html#graduating_to_a_top_level_project
> >>> Regards,
> >>>>
> >>>
> >>>
> >
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: [DISCUSS] Apache Airflow graduation from the incubator

[Kaxil Naik]

It is about time :) for Airflow to be a TLP.

I will take a look at pending issues but I agree with that it is the
correct time for us to graduate to a TLP.

Regards,
Kaxil

On Mon, Nov 26, 2018 at 7:46 PM Bolke de Bruin  wrote:

> Ping!
>
> Sent from my iPhone
>
> > On 24 Nov 2018, at 12:57, Bolke de Bruin  wrote:
> >
> > Hi All,
> >
> > With the Apache Airflow community healthy and growing, I think now would
> be a good time to
> > discuss where we stand regarding to graduation from the Incubator, and
> what requirements remains.
> >
> > Apache Airflow entered incubation around 2 years ago, since then, the
> Airflow community learned
> > a lot about how to do things in Apache ways. Now we are a very helpful
> and engaged community,
> > ready to help on all questions from the Airflow community. We delivered
> multiple releases that have
> > been increasing in quality ever since, now we can do self-driving
> releases in good cadence.
> >
> > The community is growing, new committers and PPMC members keep joining.
> We addressed almost all
> > the maturity issues stipulated by Apache Project Maturity Model [1]. So
> final requirements remain, but
> > those just need a final nudge. Committers and contributors are invited
> to verify the list and pick up the last
> > bits (QU30, CO50). Finally (yahoo!) all the License and IP issues we can
> see got resolved.
> >
> > Base on those, I believes it's time for us to graduate to TLP. [2] Any
> thoughts?
> > And welcome advice from Airflow Mentors?
> >
> > Thanks,
> >
> > [1]
> https://cwiki.apache.org/confluence/display/AIRFLOW/Maturity+Evaluation
> > [2]
> https://incubator.apache.org/guides/graduation.html#graduating_to_a_top_level_project
> Regards,
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: Remove airflow from pypi

[Kaxil Naik]

Thanks Dan

On Fri, Nov 23, 2018 at 3:44 PM Dan Davydov 
wrote:

> This could potentially break builds for some users but I feel the pros
> mentioned outweigh this, I went ahead and deleted it.
>
> On Fri, Nov 23, 2018 at 10:18 AM Bolke de Bruin  wrote:
>
> > Agree! This is even a security issue.
> >
> > Sent from my iPhone
> >
> > > On 23 Nov 2018, at 15:29, Driesprong, Fokko 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I think we should remove airflow <https://pypi.org/project/airflow/>
> > (not
> > > apache-airflow) from Pypi. I still get questions from people who
> > > accidentally install Airflow 1.8.0. I see this is maintained
> > > by mistercrunch, artwr, aeon. Anyone any objections?
> > >
> > > Cheers, Fokko
> >
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: Remove airflow from pypi

[Kaxil Naik]

+1 to remove it. I know people installing it accidentally.

On Fri, Nov 23, 2018, 14:29 Driesprong, Fokko  Hi all,
>
> I think we should remove airflow  (not
> apache-airflow) from Pypi. I still get questions from people who
> accidentally install Airflow 1.8.0. I see this is maintained
> by mistercrunch, artwr, aeon. Anyone any objections?
>
> Cheers, Fokko
>

Re: Airflow 1.10.1 is released

[Kaxil Naik]

 Great stuff Ash.

On Wed, Nov 21, 2018, 22:20 Ash Berlin-Taylor  Dear Airflow community,
>
> I'm happy to announce that Airflow 1.10.1 was just released.
>
> The source release as well as the binary "sdist" release are available
> here:
>
>
> https://dist.apache.org/repos/dist/release/incubator/airflow/1.10.1-incubating/
>
> We also made this version available on PyPi for convenience (`pip install
> apache-airflow`):
>
> https://pypi.python.org/pypi/apache-airflow
>
> Find the CHANGELOG here for more details:
>
> https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt

Re: Official JIRA integration with Github

[Kaxil Naik]

That's a great news. We can surely raise an Apache Infra ticket to see if
this integration is possible for us.

On Sun, Oct 7, 2018 at 2:00 PM Jarek Potiuk 
wrote:

> Github just announced the official support for JIRA integraton
>
> https://blog.github.com/2018-10-04-announcing-the-new-github-and-jira-software-cloud-integration/
> .
>
> Is it possible to enable (and configure) it for Apache Airflow's JIRA? It
> could help with automation of issue workflows in JIRA and has a lot of
> other useful features.
>
> J.
>
> --
>
> *Jarek Potiuk, Principal Software Engineer*
> Mobile: +48 660 796 129
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Airflow Docs - RTD vs Apache Site

[Kaxil Naik]

Hi all,

Continuing discussion from Slack, many users have had the problem with
looking at a wrong version of the documentation. Currently, our docs on
apache.airflow.org don't properly state version. Although we have specified
this info on our Github readme and confluence, there has still been lots of
confusion among the new users who try to google for the docs and are
pointed to airflow.apache.org site which doesn't have version info.

The problem currently with a.a.o site is it needs to be manually built and
only has stable version docs. We can do 2 things if we don't want to
redirect a.a.o with RTD: (1) Maintain History on our static a.a.o site (2)
Point a.a.o site to RTD docs, so a.a.o would point to RTD docs i.e. add the
domain to RTD site

Ash has also suggested another approach:

> Apache Infra run a jenkins instance (or other build bot type things) that
> we might be able to use for autobuilding docs if we want?



Let's discuss this and decide on a single-approach that is user-friendly.

NB: I will be busy for a month, hence won't be able to actively help with
this, so please feel free to contribute/commit after an approach is
finalized.

Regards,
Kaxil

Re: Travis CI tests failing in master

[Kaxil Naik]

Haha. Well, what can I say? The solution was changing `pip` tp `pip2`.
Banging my head against the wall. I just read yesterday that it wasn't
working and now ... well well...

On Fri, Sep 28, 2018 at 10:02 AM Ash Berlin-Taylor  wrote:

> Looks like someone has beaten you to the punch
> https://github.com/apache/incubator-airflow/pull/3968#pullrequestreview-159772413
> <
> https://github.com/apache/incubator-airflow/pull/3968#pullrequestreview-159772413
> >
>
> :)
>
> -ash
>
> > On 28 Sep 2018, at 09:59, Kaxil Naik  wrote:
> >
> > The problem is 80% solved. The only failing error are related to
> > TestPythonVirtualenvOperator .
> > More details are discussed at
> > https://github.com/apache/incubator-airflow-ci/pull/3
> >
> > It is because protocol=3 is used instead of 2.
> >
> > Can someone take a dig to resolve this?
> >
> > Regards,
> > Kaxil
> >
> >
> > On Thu, Sep 27, 2018 at 11:36 AM Jarek Potiuk 
> > wrote:
> >
> >> I see also that the solution is on it's way for the first problem as
> well
> >> :). https://github.com/apache/incubator-airflow/pull/3962
> >>
> >> On Thu, Sep 27, 2018 at 10:47 AM Jarek Potiuk  >
> >> wrote:
> >>
> >>> Seem that the second problem is already solved in this PR:
> >>> https://github.com/apache/incubator-airflow-ci/pull/3 - hopefully will
> >> be
> >>> merged soon :).
> >>>
> >>> On Thu, Sep 27, 2018 at 9:57 AM airflowuser
> >>>  wrote:
> >>>
> >>>> Also I see this a lot:
> >>>> 1) ERROR: Failure: ProgrammingError
> >> ((_mysql_exceptions.ProgrammingError)
> >>>> (1146, "Table 'airflow.task_instance' doesn't exist") [SQL: u'DELETE
> >> FROM
> >>>> task_instance WHERE task_instance.dag_id = %s'] [parameters:
> >>>> ('unit_tests',)])
> >>>>
> >>>>
> >>>> Sent with ProtonMail Secure Email.
> >>>>
> >>>> ‐‐‐ Original Message ‐‐‐
> >>>> On Thursday, September 27, 2018 9:20 AM, Jarek Potiuk <
> >>>> jarek.pot...@polidea.com> wrote:
> >>>>
> >>>>> Hello Everyone,
> >>>>>
> >>>>> Seems that since yesterday the builds started to fail in Travis CI
> >>>> badly.
> >>>>> And we need some urgent actions to fix it otherwise everyone
> >> developing
> >>>>> Airflow is affected now.
> >>>>>
> >>>>> I am a bit fresh in Airflow so I am not sure how to handle those
> >>>> problems
> >>>>> in an "emergency" way (and I am not sure if I am the best to propose
> >>>>> solutions), so it would be great that more experienced people could
> >>>> help to
> >>>>> solve it.
> >>>>>
> >>>>> It was not caused by any change - builds that worked before, started
> >> to
> >>>>> fail now. The nature of those errors suggests transitive dependencies
> >>>>> problems (they are all errors during installation of requirements).
> >>>>>
> >>>>> Currently the master of airflow-incubator is broken. Actually the
> >>>> problem
> >>>>> has worsened - now we seem to have two distinct problems depending on
> >>>>> python requirement.
> >>>>>
> >>>>> Error 1: Flask appbuilder
> >>>>>
> >>>>> Yesterday we had just this one (for both python 2.7 and 3.5):
> >>>>> "flask-appbuilder 1.11.1 has requirement cli

Re: execution_date - can we stop the confusion?

[Kaxil Naik]

This has been clearly documented as Bolke stated. It is an integral part of
Airflow and a user learning Airflow needs to learn this. If you think it in
an ETL perspective it completely makes sense. Also, if you can use you real
name than "airflowuser" would be good, your preference though.

Also, bear in mind that there is always a learning curve for any project or
tool. A user should take time to learn the tool and understand it properly.

On Wed, 26 Sep 2018, 16:34 Maxime Beauchemin, 
wrote:

> I think if you have a functional mindset (as in "functional data
> engineering
> <
> https://medium.com/@maximebeauchemin/functional-data-engineering-a-modern-paradigm-for-batch-data-processing-2327ec32c42a
> >")
> as opposed to a cron mindset, using the left bound of the time interval
> makes a lot of sense. Things like your daily table partition keys align
> with your Airflow execution_date.
>
> The main thing is that whatever we do we cannot break backwards
> compatibility. Offering both views (left bound/right bound), as it's been
> proposed before, either as an environment setting or a user personal
> preference is even more confusing to me personally. Users would have to
> switch context as they help each other or change environments.
>
> Also note that your intuition may differ from other people's intuition, and
> that "unlearning" something is way harder than learning something.
>
> My personal take on this is to make this a rite of passage. This is just
> one of the many thing you have to learn when learning Airflow.
>
> Max
>
> On Wed, Sep 26, 2018 at 8:18 AM Sam Elamin 
> wrote:
>
> > Hi Bolke
> >
> > Speaking as a consultant who is constantly training other teams how to
> use
> > airflow, I do frequently see this confusion.
> > Another one is how the batch_date is always batch_date + interval or as
> the
> > docs make it quite clear
> >
> > "*Let’s Repeat That* The scheduler runs your job one schedule_interval
> > AFTER
> > the start date, at the END of the period."
> >
> > Renaming it would make it simpler for newbies, but essentially they will
> > need to understand how Airflow behaves, execution_date being the batch
> > execution date not the run_date of the DAG
> >
> > I am actually in the process of writing a blog post
> > 
> > about this which I could use peoples feedback
> >
> > If it helps, I find that explaining how backfills work and why they are
> > important will drive home what the execution_date is :)
> >
> >
> > Regards
> > Sam
> >
> >
> >
> > On Wed, Sep 26, 2018 at 4:10 PM Bolke de Bruin 
> wrote:
> >
> > > I dont think this makes sense and I dont that think anyone had a real
> > > issue with this. Execution date has been clearly documented  and is
> part
> > of
> > > the core principles of airflow. Renaming will create more confusion.
> > >
> > > Please note that I do think that as an anonymous user you cannot speak
> > for
> > > any "new airflow user". That is a contradiction to me.
> > >
> > > Thanks
> > > Bolke
> > >
> > > Sent from my iPhone
> > >
> > > > On 26 Sep 2018, at 07:59, airflowuser  > .INVALID>
> > > wrote:
> > > >
> > > > One of the most annoying, hard to understand and against all common
> > > sense is the execution_date behavior. I assume that any new Airflow
> user
> > > has been struggling with it.
> > > > The amount of questions with answers referring to :
> > > https://airflow.apache.org/scheduler.html?scheduling-triggers  is
> > > uncountable.
> > > >
> > > > Most people mistakenly think that execution_date is the datetime
> which
> > > the DAG started to run.
> > > >
> > > > I suggest the following changes:
> > > > 1. Renaming the execution_date to something else like: run_stamped
> > >  This name won't cause people to get confused.
> > > > 2. Adding a new variable which indicated the actual datetime when the
> > > DAG run was generated. call it execution_start_date. People seem to
> want
> > > the information when the DAG actually started to be executed/run.
> > > >
> > > > This is only naming changes. No need to actual change the behavior -
> > > This will only make things simpler as when user encounter  run_stamped
> > he
> > > won't be confused by the name like execution_date
> > >
> >
>

Re: [jira] [Assigned] (AIRFLOW-3099) Errors raised when some blocs are missing in airflow.cfg

[Kaxil Naik]

Yes, it is restoring assignees. It's fine with me :)

On Sat, Sep 22, 2018 at 1:03 AM Holden Karau  wrote:

> Thanks for the heads up :) So it should automatically restore the
> assignee, which it looks like it did, and it's set up to only do this for
> new PRs (no back-fills). I can turn it all the way off if it's annoying to
> folks but otherwise it will automatically switch JIRAs to in-progress as
> folks make PRs for them so when we go to look for JIRAs which are not being
> worked on with the filters in JIRA it works :)
>
> On Fri, Sep 21, 2018 at 3:34 PM Kaxil Naik  wrote:
>
>> Hi Holden Karau,
>>
>> Not sure on what is happening, but seems like the dashboard you are
>> working on is setting assignee. Pinging you just in-case if you aren't
>> aware :)
>>
>> -- Forwarded message -
>> From: Holden Karau's magical unicorn (JIRA) 
>> Date: Fri, Sep 21, 2018 at 11:31 PM
>> Subject: [jira] [Assigned] (AIRFLOW-3099) Errors raised when some blocs
>> are missing in airflow.cfg
>> To: 
>>
>>
>>
>>  [
>> https://issues.apache.org/jira/browse/AIRFLOW-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
>> ]
>>
>> Holden Karau's magical unicorn reassigned AIRFLOW-3099:
>> ---
>>
>> Assignee: Holden Karau's magical unicorn  (was: Kaxil Naik)
>>
>> > Errors raised when some blocs are missing in airflow.cfg
>> > 
>> >
>> > Key: AIRFLOW-3099
>> > URL: https://issues.apache.org/jira/browse/AIRFLOW-3099
>> > Project: Apache Airflow
>> >  Issue Type: Bug
>> >  Components: configuration
>> >Affects Versions: 1.10.0
>> >Reporter: Christophe
>> >Assignee: Holden Karau's magical unicorn
>> >Priority: Minor
>> > Fix For: 1.10.1
>> >
>> >
>> > When we upgrade from a version of airflow to another one and new config
>> bloc are available or if we delete useless blocs, lots of errors are raised
>> if we don't have some blocs in airflow.cfg file.
>> > We need to avoid these errors for non-required blocs.
>> >
>> > For the record logs (not exhaustive):
>> >
>> > {noformat}
>> > [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option
>> sasl_enabled does not exist in the config!
>> > [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option
>> host does not exist in the config!
>> > [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option
>> port does not exist in the config!
>> > [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option
>> username does not exist in the config!
>> > [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option
>> password does not exist in the config!
>> > [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section hive Option
>> default_hive_mapred_queue does not exist in the config!
>> > [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
>> master does not exist in the config!
>> > [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
>> framework_name does not exist in the config!
>> > [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
>> task_cpu does not exist in the config!
>> > [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
>> task_memory does not exist in the config!
>> > [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
>> checkpoint does not exist in the config!
>> > [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
>> authenticate does not exist in the config!
>> > [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section elasticsearch
>> Option elasticsearch_host does not exist in the config!
>> > [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section elasticsearch
>> Option elasticsearch_log_id_template does not exist in the config!
>> > [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section elasticsearch
>> Option elasticsearch_end_of_log_mark does not exist in the config!
>> > [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section kubernetes
>> Option worker_container_repository does not exist in the config!
>> > [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section kubernetes
>> Option worker_container_tag does not exist in the config!
>> > [2018-09-21 10:

Fwd: [jira] [Assigned] (AIRFLOW-3099) Errors raised when some blocs are missing in airflow.cfg

[Kaxil Naik]

Hi Holden Karau,

Not sure on what is happening, but seems like the dashboard you are working
on is setting assignee. Pinging you just in-case if you aren't aware :)

-- Forwarded message -
From: Holden Karau's magical unicorn (JIRA) 
Date: Fri, Sep 21, 2018 at 11:31 PM
Subject: [jira] [Assigned] (AIRFLOW-3099) Errors raised when some blocs are
missing in airflow.cfg
To: 



 [
https://issues.apache.org/jira/browse/AIRFLOW-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Holden Karau's magical unicorn reassigned AIRFLOW-3099:
---

Assignee: Holden Karau's magical unicorn  (was: Kaxil Naik)

> Errors raised when some blocs are missing in airflow.cfg
> 
>
> Key: AIRFLOW-3099
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3099
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: configuration
>Affects Versions: 1.10.0
>Reporter: Christophe
>Assignee: Holden Karau's magical unicorn
>Priority: Minor
> Fix For: 1.10.1
>
>
> When we upgrade from a version of airflow to another one and new config
bloc are available or if we delete useless blocs, lots of errors are raised
if we don't have some blocs in airflow.cfg file.
> We need to avoid these errors for non-required blocs.
>
> For the record logs (not exhaustive):
>
> {noformat}
> [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option
sasl_enabled does not exist in the config!
> [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option host
does not exist in the config!
> [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option port
does not exist in the config!
> [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option
username does not exist in the config!
> [2018-09-21 10:49:37,727] {cli.py:464} ERROR - Section atlas Option
password does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section hive Option
default_hive_mapred_queue does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
master does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
framework_name does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
task_cpu does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
task_memory does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
checkpoint does not exist in the config!
> [2018-09-21 10:49:37,728] {cli.py:464} ERROR - Section mesos Option
authenticate does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section elasticsearch
Option elasticsearch_host does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section elasticsearch
Option elasticsearch_log_id_template does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section elasticsearch
Option elasticsearch_end_of_log_mark does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section kubernetes Option
worker_container_repository does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section kubernetes Option
worker_container_tag does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section kubernetes Option
delete_worker_pods does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section kubernetes Option
namespace does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section kubernetes Option
airflow_configmap does not exist in the config!
> [2018-09-21 10:49:37,729] {cli.py:464} ERROR - Section kubernetes Option
dags_volume_subpath does not exist in the config!
> [2018-09-21 10:49:37,730] {cli.py:464} ERROR - Section kubernetes Option
dags_volume_claim does not exist in the config!
> [2018-09-21 10:49:37,730] {cli.py:464} ERROR - Section kubernetes Option
logs_volume_subpath does not exist in the config!
> [2018-09-21 10:49:37,730] {cli.py:464} ERROR - Section kubernetes Option
logs_volume_claim does not exist in the config!
> [2018-09-21 10:49:37,730] {cli.py:464} ERROR - Section kubernetes Option
git_repo does not exist in the config!
> [2018-09-21 10:49:37,730] {cli.py:464} ERROR - Section kubernetes Option
git_branch does not exist in the config!
> [2018-09-21 10:49:37,730] {cli.py:464} ERROR - Section kubernetes Option
git_user does not exist in the config!
> [2018-09-21 10:49:37,730] {cli.py:464} ERROR - Section kubernetes Option
git_password does not exist in the config!
> [201

Re: Guidelines on Contrib vs Non-contrib

[Kaxil Naik]

+1 for moving contribs into each of their separate repos, we have
been talking about these for several months now. This should go into
Airflow 2.0. I will draft an AIP (Airflow Improvement Proposal) with some
proposed steps at the end of this week. If anyone wants to draft it before
that, feel free to do it.

On Tue, Sep 18, 2018 at 7:15 AM Jeff Payne  wrote:

> George, did you mean "move towards keeping non-core supported code
> outside of the core project/repo?"
>
>
> [image: 1500660321366_bombora_logo-org-small] <http://bombora.com/>
> jeff payne | technical lead & senior data engineer
> 257 Park Ave S | 6th Floor | New York, NY 10010
> m: +1 916-273-0938| e: jpa...@bombora.com| @bomboradata
> <https://twitter.com/bomboradata>
>
> *Get our free report on using Intent Data for content and Account-Based
> Marketing*
> <http://bombora.com/powering-content-marketing-intent-data/?utm_source=email_medium=signature_campaign=contentreport>
> [image: 1500660332982_2017]
>
> --
> *From:* George Leslie-Waksman 
> *Sent:* Monday, September 17, 2018 4:50:39 PM
> *To:* dev@airflow.incubator.apache.org
> *Subject:* Re: Guidelines on Contrib vs Non-contrib
>
> Given we have a plugin system, could we alternatively move away from
> keeping non-core supported code outside of the core project/repo?
>
> It would hugely decrease the surface area of the main repository and
> testing infrastructure to get most of the contrib code out to its own
> place.
>
> Further, it would decrease the committer burden of having to approve/merge
> code that is not supposed to be their responsibility.
>
> On Mon, Sep 17, 2018 at 4:37 PM Tim Swast 
> wrote:
>
> > > Individual operators and hooks living in separate repositories on
> github
> > (or possibly other Apache projects), which are then distributed by pip
> and
> > installed as libraries seems like it would scale better.
> >
> > Pandas did this about a year ago, and it's seemed to have worked well.
> For
> > example, pandas.read_gbq is a very thin wrapper around
> pandas_gbq.read_gbq
> > (distributed as a separate package). It has made it easier for me to
> track
> > issues corresponding to my area of expertise.
> >
> > On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan  wrote:
> >
> > > > My understanding as a contributor is that if a hook/operator is in
> > core,
> > > it
> > > > means that a committer is willing to take personal responsibility to
> > > > maintain it (or at least help maintain it), and everything else goes
> in
> > > > contrib.
> > >
> > > That's not correct.  All of the code is owned by the entire
> > > community[1]; no one person is responsible for any of it.  There's no
> > > silos, fiefdoms, walled gardens, etc.  If the community cannot support
> > > a piece of code it should be deprecated and subsequently removed.
> > >
> > > Contrib sections are almost always problematic for this reason.
> > > Hadoop ended up abandoning its.  Because Airflow acts as a gathering
> > > point for so many disparate technologies (databases, storage systems,
> > > compute engines, etc.), trying to keep all of them corralled and up to
> > > date will be very difficult.  Individual operators and hooks living in
> > > separate repositories on github (or possibly other Apache projects),
> > > which are then distributed by pip and installed as libraries seems
> > > like it would scale better.
> > >
> > > -Jakob
> > >
> > > [1]
> https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie
> > >
> > > On 15 September 2018 at 13:29, Jeff Payne  wrote:
> > > > How many operators are added to contrib per month? Is it too many to
> > > make the decision case by case? If so, then the above mentioned rule
> > sounds
> > > fairly reasonable. However, if that's the rule, shouldn't a bunch of
> > > existing modules be moved from contrib to core?
> > > >
> > > > Get Outlook for Android<https://aka.ms/ghei36>
> > > >
> > > > 
> > > > From: Taylor Edmiston 
> > > > Sent: Saturday, September 15, 2018 1:13:47 PM
> > > > To: dev@airflow.incubator.apache.org
> > > > Subject: Re: Guidelines on Contrib vs Non-contrib
> > > >
> > > > My understanding as a contributor is that if a hook/operator is in
> > core,
> > > it
> > > > means that a committer is willing to take personal respo

Guidelines on Contrib vs Non-contrib

[Kaxil Naik]

Hi, all (mainly contributors),

Can we decide on a common guideline on when a hook/operator should go under
contrib vs core?

Regards,

*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: Call for fixes for Airflow 1.10.1

[Kaxil Naik]

I have cherry-picked the fix for this issue on top of 1.10-test branch
along-with 16 other commits (the ones I listed earlier in the mail + their
dependent commits)

Regards,
Kaxil

On Fri, Sep 14, 2018 at 6:07 AM Gerardo Curiel  wrote:

> Hello,
>
> I wonder if there is still time to wait for a fix for "BigQuery hook does
> not allow specifying both the partition field name and table name at the
> same time": https://issues.apache.org/jira/browse/AIRFLOW-2772.
>
> The BigQueryHook is taking some liberties and implementing some client-side
> logic that shouldn't be there.
>
>
> On Wed, Sep 12, 2018 at 6:59 PM Driesprong, Fokko 
> wrote:
>
> > Hi Ash,
> >
> > I've cherry-picked two commits on top of 1.10-test branch:
> >
>
> Cheers,
>
> --
> Gerardo Curiel // https://gerar.do
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: [VOTE] Replace with Gitter with Slack?

[Kaxil Naik]

0 (binding) I prefer using StackOverflow + Mailing list which are
searchable in Google.

And as someone said before the free slack version has only 10,000 messages
that are searchable.

If ASF provides a paid version, than I am happy to move to Slack - as it is
for sure better than gitter.

Regards,
Kaxil

On Thu, 6 Sep 2018, 09:05 Eamon Keane,  wrote:

> +1
>
> On Thu, Sep 6, 2018 at 8:28 AM Bolke de Bruin  wrote:
>
> > 0 (binding) abstaining from the vote.
> >
> > I am not convinced we gain anything except disruption for an existing
> > community. We cannot even force people to move away only make it “less”
> > official. My question would be do we get more attendance of contributors
> > and committers if we move to slack? Otherwise I don’t see the benefit.
> >
> > B.
> >
> > Verstuurd vanaf mijn iPad
> >
> > > Op 6 sep. 2018 om 09:04 heeft airflowuser  .INVALID>
> > het volgende geschreven:
> > >
> > > -1
> > >
> > >
> > > Sent with ProtonMail Secure Email.
> > >
> > > ‐‐‐ Original Message ‐‐‐
> > >> On September 6, 2018 5:30 AM, Sid Anand  wrote:
> > >>
> > >> Hi Folks!
> > >> In the Apache tradition, I'd like to ask the community to vote on
> > replacing
> > >> Gitter with Slack.
> > >>
> > >> For more information about what was recently discussed, refer to
> > >>
> >
> https://lists.apache.org/thread.html/8eeb6c46ec431b9158f87022ceaa5eed8dbaaf082c887dae55f86f96@
> > 
> > >>
> > >> If you would like to replace Gitter with Slack, vote +1. If you want
> to
> > >> keep things they way they are, vote -1. You can also vote 0 if you
> don't
> > >> care either way because you wouldn't use either much, preferring to
> use
> > the
> > >> mailing list instead, which is highly encouraged as it is Apache's
> > official
> > >> record.
> > >>
> > >> The vote will be open for 72 hours and will expire at 8p PT this
> > Saturday.
> > >> -s
> > >>
> > >> P.S. If the community votes for Slack, we could create our own
> workspace
> > >> (e.g. airflow.slack.com).
> > >> P.P.S. In general, anyone in the community can launch a vote like this
> > from
> > >> time to time. There is no binding/non-binding distinction since we are
> > not
> > >> running an official Apache vote.
> > >
> > >
> >
>

Re: Call for fixes for Airflow 1.10.1

[Kaxil Naik]

ator/BigQueryHook should default to None
> > > https://github.com/apache/incubator-airflow/pull/3829
> > >
> > >
> > > In addition to those PRs which are already marked with Fix Version of
> > > 1.10.1 I think we should also pull in these:
> > >
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2713 Rename async
> variable
> > > for Python 3.7.0 compatibility
> > > https://github.com/apache/incubator-airflow/pull/3561
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2895 Prevent scheduler
> > from
> > > spamming heartbeats/logs
> > > https://github.com/apache/incubator-airflow/pull/3747
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2921 A trivial
> > > incorrectness in CeleryExecutor()
> > > https://github.com/apache/incubator-airflow/pull/3773
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2866 Missing CSRF Token
> > > Error on Web RBAC UI Create/Update Operations
> > > https://github.com/apache/incubator-airflow/pull/3804
> > >
> > >
> > > https://issues.apache.org/jira/browse/AIRFLOW-2951
> > > https://github.com/apache/incubator-airflow/pull/3798 Update dag_run
> > > table end_date when state change
> > > (though as written it has a few other deps to cherry pick in, so will
> see
> > > about this one)
> > >
> > >
> >
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: Apache Spark Interfering with Airflow Jira/PRs ??

[Kaxil Naik]

I am getting that too. Tons of emails as well about the same. Not sure the
reason.

On Sun, 2 Sep 2018, 20:45 Sid Anand,  wrote:

> Fellow committers... what happened around 11a PT today? I see a flood of
> updates to our Jiras from "Apache Spark".
>
> Some examples:
>
>1.
>
> https://issues.apache.org/jira/browse/AIRFLOW-2408?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel=16601616#comment-16601616
>2. It also updated https://issues.apache.org/jira/browse/AIRFLOW-2553
> as
>resolved and closed
> https://github.com/apache/incubator-airflow/pull/3451
>PR
>
> What's going on?
> -s
>

Re: Add git tag for 1.10

[Kaxil Naik]

We already do have it:
https://github.com/apache/incubator-airflow/releases/tag/1.10.0

On Fri, 31 Aug 2018, 06:23 Beau Barker,  wrote:

> Can we please tag the final v1.10 commit?
>

Re: Missing operators in the docs

[Kaxil Naik]

@ashb Yes, we are mocking out some of the Google Cloud modules in
docs/conf.py .

On Thu, 30 Aug 2018, 11:20 Ash Berlin-Taylor,  wrote:

> There's a setting available for sphinx projects where imports/modules can
> be mocked - it might be worth exploring
> http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html?highlight=autodoc_mock_imports#confval-autodoc_mock_imports
> <
> http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html?highlight=autodoc_mock_imports#confval-autodoc_mock_imports>
> for some of the harder-to-install modules?
>
> -ash
>
> > On 29 Aug 2018, at 21:53, Kaxil Naik  wrote:
> >
> > I have fixed the issue on https://airflow.apache.org/ , added a comment
> on
> > confluence as well. Will try to fix ReadTheDocs environment (Have opened
> a
> > Jira for it) as it can't install all the dependencies that depend on C
> > Modules (
> >
> https://read-the-docs.readthedocs.io/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules
> > )
> >
> > Regards,
> > Kaxil
> >
> > On Wed, Aug 29, 2018 at 9:15 PM Kaxil Naik  wrote:
> >
> >> Will have a look and resolve it.
> >>
> >> On Wed, Aug 29, 2018 at 8:25 PM Maxime Beauchemin <
> >> maximebeauche...@gmail.com> wrote:
> >>
> >>> Looks like both.
> >>>
> >>> On Wed, Aug 29, 2018 at 12:18 PM Kaxil Naik 
> wrote:
> >>>
> >>>> Hi Max,
> >>>>
> >>>> Did you see that on readthedocs or airflow.apache one?
> >>>>
> >>>> On Wed, 29 Aug 2018, 20:15 Maxime Beauchemin, <
> >>> maximebeauche...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hey committers,
> >>>>>
> >>>>> I noticed that some of the operators are missing from the API
> >>> reference
> >>>>> part of the docs (HiveOperator for instance). I'm guessing a
> committer
> >>>>> generated / pushed the docs with some libs missing and that the
> >>> operators
> >>>>> depending on those missing libs got skipped.
> >>>>>
> >>>>> We may have to improve the doc-generation wiki page or make a
> >>> bulletproof
> >>>>> shell script that ensures all libs are installed prior to generating
> >>> the
> >>>>> docs.
> >>>>>
> >>>>> Max
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> --
> >> *Kaxil Naik*
> >> *Big Data Consultant *@ *Data Reply UK*
> >> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark &
> Neo4j
> >> Developer
> >> *Phone: *+44 (0) 74820 88992
> >> *LinkedIn*: https://www.linkedin.com/in/kaxil
> >>
> >
> >
> > --
> > *Kaxil Naik*
> > *Big Data Consultant *@ *Data Reply UK*
> > *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> > Developer
> > *Phone: *+44 (0) 74820 88992
> > *LinkedIn*: https://www.linkedin.com/in/kaxil
>
>

Re: Missing operators in the docs

[Kaxil Naik]

I have fixed the issue on https://airflow.apache.org/ , added a comment on
confluence as well. Will try to fix ReadTheDocs environment (Have opened a
Jira for it) as it can't install all the dependencies that depend on C
Modules (
https://read-the-docs.readthedocs.io/en/latest/faq.html#i-get-import-errors-on-libraries-that-depend-on-c-modules
)

Regards,
Kaxil

On Wed, Aug 29, 2018 at 9:15 PM Kaxil Naik  wrote:

> Will have a look and resolve it.
>
> On Wed, Aug 29, 2018 at 8:25 PM Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
>
>> Looks like both.
>>
>> On Wed, Aug 29, 2018 at 12:18 PM Kaxil Naik  wrote:
>>
>> > Hi Max,
>> >
>> > Did you see that on readthedocs or airflow.apache one?
>> >
>> > On Wed, 29 Aug 2018, 20:15 Maxime Beauchemin, <
>> maximebeauche...@gmail.com>
>> > wrote:
>> >
>> > > Hey committers,
>> > >
>> > > I noticed that some of the operators are missing from the API
>> reference
>> > > part of the docs (HiveOperator for instance). I'm guessing a committer
>> > > generated / pushed the docs with some libs missing and that the
>> operators
>> > > depending on those missing libs got skipped.
>> > >
>> > > We may have to improve the doc-generation wiki page or make a
>> bulletproof
>> > > shell script that ensures all libs are installed prior to generating
>> the
>> > > docs.
>> > >
>> > > Max
>> > >
>> >
>>
>
>
> --
> *Kaxil Naik*
> *Big Data Consultant *@ *Data Reply UK*
> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
> Developer
> *Phone: *+44 (0) 74820 88992
> *LinkedIn*: https://www.linkedin.com/in/kaxil
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: Missing operators in the docs

[Kaxil Naik]

Will have a look and resolve it.

On Wed, Aug 29, 2018 at 8:25 PM Maxime Beauchemin <
maximebeauche...@gmail.com> wrote:

> Looks like both.
>
> On Wed, Aug 29, 2018 at 12:18 PM Kaxil Naik  wrote:
>
> > Hi Max,
> >
> > Did you see that on readthedocs or airflow.apache one?
> >
> > On Wed, 29 Aug 2018, 20:15 Maxime Beauchemin, <
> maximebeauche...@gmail.com>
> > wrote:
> >
> > > Hey committers,
> > >
> > > I noticed that some of the operators are missing from the API reference
> > > part of the docs (HiveOperator for instance). I'm guessing a committer
> > > generated / pushed the docs with some libs missing and that the
> operators
> > > depending on those missing libs got skipped.
> > >
> > > We may have to improve the doc-generation wiki page or make a
> bulletproof
> > > shell script that ensures all libs are installed prior to generating
> the
> > > docs.
> > >
> > > Max
> > >
> >
>


-- 
*Kaxil Naik*
*Big Data Consultant *@ *Data Reply UK*
*Certified *Google Cloud Data Engineer | *Certified* Apache Spark & Neo4j
Developer
*Phone: *+44 (0) 74820 88992
*LinkedIn*: https://www.linkedin.com/in/kaxil

Re: Missing operators in the docs

[Kaxil Naik]

Hi Max,

Did you see that on readthedocs or airflow.apache one?

On Wed, 29 Aug 2018, 20:15 Maxime Beauchemin, 
wrote:

> Hey committers,
>
> I noticed that some of the operators are missing from the API reference
> part of the docs (HiveOperator for instance). I'm guessing a committer
> generated / pushed the docs with some libs missing and that the operators
> depending on those missing libs got skipped.
>
> We may have to improve the doc-generation wiki page or make a bulletproof
> shell script that ensures all libs are installed prior to generating the
> docs.
>
> Max
>