Re: Airflow Documentation - Readthedocs

[Driesprong, Fokko]

Vo


Op zo 11 feb. 2018 om 01:31 schreef Maxime Beauchemin <
maximebeauche...@gmail.com>

> I just hit the "Destroy" button.
>
> Max
>
> On Sat, Feb 10, 2018 at 12:53 PM, Naik Kaxil  wrote:
>
> > From what I see the package owners at PyPi are mistercrunch, aoen, artwr
> .
> > Will it be possible for one of you to take care of this?
> >
> >
> > On 10/02/2018, 20:26, "Ash Berlin-Taylor" <
> ash_airflowl...@firemirror.com>
> > wrote:
> >
> > Relatedly: can we remove the docs for the old "airflow" dist on pypi?
> >
> > If someone has permissions to manage the on pypi, can you go to
> > https://pypi.python.org/pypi?%3Aaction=pkg_edit=airflow <
> > https://pypi.python.org/pypi?:action=pkg_edit=airflow> and hit the
> > "Destroy Documentation" button:
> >
> > > If you would like to DESTROY any existing documentation hosted at
> > http://pythonhosted.org/airflow Use this button, There is no undo.
> > This should hopefully mean fewer people find the older version of the
> > docs form Google etc.
> >
> >
> >
> > -ash
> >
> >
> >
> > > On 9 Feb 2018, at 22:35, Naik Kaxil  wrote:
> > >
> > > Thanks Andy,
> > >
> > > It fails because of missing '.readthedocs.yml' (
> > https://github.com/apache/incubator-airflow/blob/master/.readthedocs.yml
> )
> > file that forces it to use 'pip' to install packages and dependencies
> > instead of using 'setup.py' file. I uploaded that file in my fork for
> > V1.8-Stable branch and it passes there. I am not sure whether we should
> do
> > that or ignore V1.8 version.
> > >
> > > On 10/02/2018, 00:46, "Andy Hadjigeorgiou" 
> > wrote:
> > >
> > >Absolutely - I did it for 1.9 stable and 1.8 stable. 1.8.2
> > building failed (
> > >https://readthedocs.org/projects/airflow/builds/6724340/)
> > >
> > >
> > >On Fri, Feb 9, 2018 at 12:56 PM, Naik Kaxil 
> > wrote:
> > >
> > >> Thanks Andy for doing that. Can you please do the same for version
> > 1.9 and
> > >> 1.8.2 if possible? If Arthur adds me I will do it but if you can
> > spare some
> > >> time to do that, it would be great.
> > >>
> > >>
> > >>
> > >> On 09/02/2018, 22:58, "Andy Hadjigeorgiou" 
> > wrote:
> > >>
> > >>Absolutely, thanks!
> > >>
> > >>Updated the docs, build passing, Will think about a proper
> > cadence for
> > >> it.
> > >>
> > >>- Andy
> > >>
> > >>On Fri, Feb 9, 2018 at 11:16 AM, Arthur Wiedmer <
> > >> arthur.wied...@gmail.com>
> > >>wrote:
> > >>
> > >>> Done.
> > >>>
> > >>> Thanks for the help Andy!
> > >>>
> > >>> On Fri, Feb 9, 2018 at 6:56 AM, Andy Hadjigeorgiou <
> > >> andyxha...@gmail.com>
> > >>> wrote:
> > >>>
> >  My username is andyxhadji, email is andyxha...@gmail.com.
> > 
> >  Thanks for the help Arthur!
> > 
> >  - Andy
> > 
> >  On Fri, Feb 9, 2018 at 9:46 AM, Arthur Wiedmer <
> > >> arthur.wied...@gmail.com
> > 
> >  wrote:
> > 
> > > Someone needs to create an account on the site, I can then add
> > >> them as
> > >>> a
> > > maintainer for the project.
> > >
> > > Best,
> > > Arthur
> > >
> > > On Sun, Feb 4, 2018 at 8:14 AM, Bolke de Bruin <
> > >> bdbr...@gmail.com>
> >  wrote:
> > >
> > >> Perfectly fine to me. It would be highly appreciated if this
> > >> could be
> > >> picked up so it can be made opart of the release process.
> > >>
> > >> @Arthur @max: can you share / make available credentials?
> > >>
> > >> Cheers
> > >> Bolke
> > >>
> > >>
> > >>> On 4 Feb 2018, at 15:16, Naik Kaxil 
> > >> wrote:
> > >>>
> > >>> Yes, the same happened to me. One of my colleague wanted to
> > >> use a
> > > Google
> > >> Cloud Storage operator but he was looking at the old docs so
> he
> >  couldn't
> > >> find one.
> > >>>
> > >>> I suggest we should have documentation at 2 places:
> > >>> 1) PythonHosted - For latest stable airflow version on PyPI
> > >>> 2) Readthedocs - For up-to-date documentation of Airflow on
> > >> GitHub
> > >>>
> > >>> Regards,
> > >>> Kaxil
> > >>>
> > >>> On 04/02/2018, 13:47, "fo...@driesprongen.nl on behalf of
> >  Driesprong,
> > >> Fokko"  > >> fo...@driesprong.frl>
> >  wrote:
> > >>>
> > >>>   This is a good question, I see that artwr and Maxime are
> > >> owners
> > >>> of
> > > the
> > >>>   project, maybe they can fix the build? Or add some more
> > >> 

Re: Airflow Documentation - Readthedocs

[Maxime Beauchemin]

I just hit the "Destroy" button.

Max

On Sat, Feb 10, 2018 at 12:53 PM, Naik Kaxil  wrote:

> From what I see the package owners at PyPi are mistercrunch, aoen, artwr .
> Will it be possible for one of you to take care of this?
>
>
> On 10/02/2018, 20:26, "Ash Berlin-Taylor" 
> wrote:
>
> Relatedly: can we remove the docs for the old "airflow" dist on pypi?
>
> If someone has permissions to manage the on pypi, can you go to
> https://pypi.python.org/pypi?%3Aaction=pkg_edit=airflow <
> https://pypi.python.org/pypi?:action=pkg_edit=airflow> and hit the
> "Destroy Documentation" button:
>
> > If you would like to DESTROY any existing documentation hosted at
> http://pythonhosted.org/airflow Use this button, There is no undo.
> This should hopefully mean fewer people find the older version of the
> docs form Google etc.
>
>
>
> -ash
>
>
>
> > On 9 Feb 2018, at 22:35, Naik Kaxil  wrote:
> >
> > Thanks Andy,
> >
> > It fails because of missing '.readthedocs.yml' (
> https://github.com/apache/incubator-airflow/blob/master/.readthedocs.yml)
> file that forces it to use 'pip' to install packages and dependencies
> instead of using 'setup.py' file. I uploaded that file in my fork for
> V1.8-Stable branch and it passes there. I am not sure whether we should do
> that or ignore V1.8 version.
> >
> > On 10/02/2018, 00:46, "Andy Hadjigeorgiou" 
> wrote:
> >
> >Absolutely - I did it for 1.9 stable and 1.8 stable. 1.8.2
> building failed (
> >https://readthedocs.org/projects/airflow/builds/6724340/)
> >
> >
> >On Fri, Feb 9, 2018 at 12:56 PM, Naik Kaxil 
> wrote:
> >
> >> Thanks Andy for doing that. Can you please do the same for version
> 1.9 and
> >> 1.8.2 if possible? If Arthur adds me I will do it but if you can
> spare some
> >> time to do that, it would be great.
> >>
> >>
> >>
> >> On 09/02/2018, 22:58, "Andy Hadjigeorgiou" 
> wrote:
> >>
> >>Absolutely, thanks!
> >>
> >>Updated the docs, build passing, Will think about a proper
> cadence for
> >> it.
> >>
> >>- Andy
> >>
> >>On Fri, Feb 9, 2018 at 11:16 AM, Arthur Wiedmer <
> >> arthur.wied...@gmail.com>
> >>wrote:
> >>
> >>> Done.
> >>>
> >>> Thanks for the help Andy!
> >>>
> >>> On Fri, Feb 9, 2018 at 6:56 AM, Andy Hadjigeorgiou <
> >> andyxha...@gmail.com>
> >>> wrote:
> >>>
>  My username is andyxhadji, email is andyxha...@gmail.com.
> 
>  Thanks for the help Arthur!
> 
>  - Andy
> 
>  On Fri, Feb 9, 2018 at 9:46 AM, Arthur Wiedmer <
> >> arthur.wied...@gmail.com
> 
>  wrote:
> 
> > Someone needs to create an account on the site, I can then add
> >> them as
> >>> a
> > maintainer for the project.
> >
> > Best,
> > Arthur
> >
> > On Sun, Feb 4, 2018 at 8:14 AM, Bolke de Bruin <
> >> bdbr...@gmail.com>
>  wrote:
> >
> >> Perfectly fine to me. It would be highly appreciated if this
> >> could be
> >> picked up so it can be made opart of the release process.
> >>
> >> @Arthur @max: can you share / make available credentials?
> >>
> >> Cheers
> >> Bolke
> >>
> >>
> >>> On 4 Feb 2018, at 15:16, Naik Kaxil 
> >> wrote:
> >>>
> >>> Yes, the same happened to me. One of my colleague wanted to
> >> use a
> > Google
> >> Cloud Storage operator but he was looking at the old docs so he
>  couldn't
> >> find one.
> >>>
> >>> I suggest we should have documentation at 2 places:
> >>> 1) PythonHosted - For latest stable airflow version on PyPI
> >>> 2) Readthedocs - For up-to-date documentation of Airflow on
> >> GitHub
> >>>
> >>> Regards,
> >>> Kaxil
> >>>
> >>> On 04/02/2018, 13:47, "fo...@driesprongen.nl on behalf of
>  Driesprong,
> >> Fokko"  >> fo...@driesprong.frl>
>  wrote:
> >>>
> >>>   This is a good question, I see that artwr and Maxime are
> >> owners
> >>> of
> > the
> >>>   project, maybe they can fix the build? Or add some more
> >> Airflow
> >> committers
> >>>   as readthedocs project owners.
> >>>
> >>>   I get quite some questions from colleagues because they
> >> are
>  reading
> >> old
> >>>   docs :)
> >>>
> >>>   Cheers, Fokko
> >>>
> >>>   2018-02-04 14:32 GMT+01:00 Naik Kaxil :
> >>>
> 

Re: Failing Tasks Indication

[Taylor Edmiston]

Like Sean mentioned, bash (in scripts) swallows exit status codes from
commands by default.

https://vaneyckt.io/posts/safer_bash_scripts_with_set_euxo_pipefail/

Not sure if this fits your use case but the PythonOperator with your
existing code would work as expected.

By the way, for Boolean comparisons in Python it's preferred to use is over
==.

https://stackoverflow.com/a/27276841/149428

Best,
Taylor

On Thu, Feb 8, 2018 at 2:52 PM Sean Fern  wrote:

> Is the bash script wrapping this python command running with "set -e" or
> "set -o errexit"?
>
> On Tue, Feb 6, 2018 at 4:27 AM, Johan de Braak 
> wrote:
>
> > Hi,
> >
> >
> >
> > I am trying to make airflow mark tasks as failed when an exception is
> > raised. I have saved a python script to disk which I would like to run
> with
> > the BashOperator like this:
> >
> >
> >
> > import sys
> >
> > def myfunc(a):
> >
> > if a == True:
> >
> > return
> >
> > else:
> >
> > raise Exception
> >
> >
> >
> > if __name__ == '__main__':
> >
> > try:
> >
> > myfunc(False)
> >
> > except:
> >
> > sys.exit(1)
> >
> >
> >
> > When running code like this, I would expect that I would see a red circle
> > (indicating failure) in my overview since the except part is excecuted.
> > However, I see dark green indicating that the task was a succes. Could
> > anyone explain how to make the tasks indicate it when they failed?
> >
> >
> >
> > Thanks in advance,
> >
> >
> >
> > *Johan de Braak*
> >
> > Data Analyst
> >
> >
> >
> > [image: Description: Beschrijving: Description: Description:
> > M:\Project_Acquisition\ACQUISITION DOCUMENTS\09. Huisstijl en
> > plaatjes\Huisstijl\LOGO ELEAF High Res.jpg]
> >
> >
> > *johan.de.br...@eleaf.com  *
> > eLEAF Competence Center • *www.eleaf.com *
> >
> > This message is intended for the exclusive use of the person(s) mentioned
> > as recipient(s) and may contain personal and/or confidential information.
> > If you have received this message in error, please notify the sender and
> > delete this message from your system immediately. Directly or indirectly
> > copying, disclosing, distributing, printing, publicizing and/or in any
> way
> > using this message or any part thereof by any means is strictly
> prohibited
> > if you are not the intended recipient(s). This message and any associated
> > attachments cannot be deemed as legally binding under any circumstances
> > without the express written consent from eLEAF©. eLEAF© is not
> responsible
> > for any loss and/or damages arising from any errors and/or omissions in
> its
> > e-mail messages. Reference to the own general terms and conditions by the
> > other party is explicitly rejected by eLEAF©. Please read our general
> terms
> > and condition on http://www.eleaf.com/legal.
> >
> > P You save 200 ml of water, 2g of CO2 and 2g of wood per page: Please
> > print only if it is really necessary.
> >
> >
> >
>
-- 
*Taylor Edmiston*
TEdmiston.com  | Blog

Stack Overflow CV  | LinkedIn
 | AngelList

Re: Airflow Documentation - Readthedocs

[Ash Berlin-Taylor]

Relatedly: can we remove the docs for the old "airflow" dist on pypi?

If someone has permissions to manage the on pypi, can you go to 
https://pypi.python.org/pypi?%3Aaction=pkg_edit=airflow 
 and hit the 
"Destroy Documentation" button:

> If you would like to DESTROY any existing documentation hosted at 
> http://pythonhosted.org/airflow Use this button, There is no undo.
This should hopefully mean fewer people find the older version of the docs form 
Google etc.



-ash 



> On 9 Feb 2018, at 22:35, Naik Kaxil  wrote:
> 
> Thanks Andy, 
> 
> It fails because of missing '.readthedocs.yml' 
> (https://github.com/apache/incubator-airflow/blob/master/.readthedocs.yml) 
> file that forces it to use 'pip' to install packages and dependencies instead 
> of using 'setup.py' file. I uploaded that file in my fork for V1.8-Stable 
> branch and it passes there. I am not sure whether we should do that or ignore 
> V1.8 version.
> 
> On 10/02/2018, 00:46, "Andy Hadjigeorgiou"  wrote:
> 
>Absolutely - I did it for 1.9 stable and 1.8 stable. 1.8.2 building failed 
> (
>https://readthedocs.org/projects/airflow/builds/6724340/)
> 
> 
>On Fri, Feb 9, 2018 at 12:56 PM, Naik Kaxil  wrote:
> 
>> Thanks Andy for doing that. Can you please do the same for version 1.9 and
>> 1.8.2 if possible? If Arthur adds me I will do it but if you can spare some
>> time to do that, it would be great.
>> 
>> 
>> 
>> On 09/02/2018, 22:58, "Andy Hadjigeorgiou"  wrote:
>> 
>>Absolutely, thanks!
>> 
>>Updated the docs, build passing, Will think about a proper cadence for
>> it.
>> 
>>- Andy
>> 
>>On Fri, Feb 9, 2018 at 11:16 AM, Arthur Wiedmer <
>> arthur.wied...@gmail.com>
>>wrote:
>> 
>>> Done.
>>> 
>>> Thanks for the help Andy!
>>> 
>>> On Fri, Feb 9, 2018 at 6:56 AM, Andy Hadjigeorgiou <
>> andyxha...@gmail.com>
>>> wrote:
>>> 
 My username is andyxhadji, email is andyxha...@gmail.com.
 
 Thanks for the help Arthur!
 
 - Andy
 
 On Fri, Feb 9, 2018 at 9:46 AM, Arthur Wiedmer <
>> arthur.wied...@gmail.com
 
 wrote:
 
> Someone needs to create an account on the site, I can then add
>> them as
>>> a
> maintainer for the project.
> 
> Best,
> Arthur
> 
> On Sun, Feb 4, 2018 at 8:14 AM, Bolke de Bruin <
>> bdbr...@gmail.com>
 wrote:
> 
>> Perfectly fine to me. It would be highly appreciated if this
>> could be
>> picked up so it can be made opart of the release process.
>> 
>> @Arthur @max: can you share / make available credentials?
>> 
>> Cheers
>> Bolke
>> 
>> 
>>> On 4 Feb 2018, at 15:16, Naik Kaxil 
>> wrote:
>>> 
>>> Yes, the same happened to me. One of my colleague wanted to
>> use a
> Google
>> Cloud Storage operator but he was looking at the old docs so he
 couldn't
>> find one.
>>> 
>>> I suggest we should have documentation at 2 places:
>>> 1) PythonHosted - For latest stable airflow version on PyPI
>>> 2) Readthedocs - For up-to-date documentation of Airflow on
>> GitHub
>>> 
>>> Regards,
>>> Kaxil
>>> 
>>> On 04/02/2018, 13:47, "fo...@driesprongen.nl on behalf of
 Driesprong,
>> Fokko" > fo...@driesprong.frl>
 wrote:
>>> 
>>>   This is a good question, I see that artwr and Maxime are
>> owners
>>> of
> the
>>>   project, maybe they can fix the build? Or add some more
>> Airflow
>> committers
>>>   as readthedocs project owners.
>>> 
>>>   I get quite some questions from colleagues because they
>> are
 reading
>> old
>>>   docs :)
>>> 
>>>   Cheers, Fokko
>>> 
>>>   2018-02-04 14:32 GMT+01:00 Naik Kaxil :
>>> 
 Hi guys,
 
 
 
 Are we still using http://airflow.readthedocs.io/ for
>> latest
 documentation?
 
 
 
 I see that the last build was 2 months ago which failed..
 
 
 
 http://readthedocs.org/projects/airflow/builds/
 
 
 
 The documentation at http://pythonhosted.org/airflow/ will
>> only
>>> be
> for
 the latest airflow version at PyPI (1.9 for now).
 
 
 
 It would be good to have documentation of the GitHub
>> version at
 readthedocs.
 
 
 
 Regards,
 
 Kaxil
 
 
 Kaxil Naik
 
 Data Reply
 38 Grosvenor Gardens
 > London+SW1W+0EB+-+UK=gmail=g>
 London SW1W 0EB - UK
 phone: +44 (0)20 7730 6000 <+44%2020%207730%206000>
 k.n...@reply.com
 www.reply.com