Re: BranchPythonOperator skips 'join' branch in all cases

[Joseph Napolitano]

Great, glad you figured it out!

Cheers

On Mon, Sep 19, 2016 at 3:29 PM, Nikita Tovstoles  wrote:

> You're right, Joe. The key is to set a trigger_rule on joining task. see
> updated gist and thanks!
> https://gist.github.com/dukehoops/dae9c45c2035d50e41fee7c7d75a50dd
>
> On Mon, Sep 19, 2016 at 12:16 PM, Joseph Napolitano <
> joseph.napolit...@blueapron.com.invalid> wrote:
>
> > My guess is that this line:
> > [one, two, branch_skip_upload] >> join >> finish
> >
> > Requires all 3 tasks to complete. This may or may not be true--I haven't
> > needed to rejoin branched tasks yet. Given this though, all 3 of these
> > cannot complete. There may be a way to say "at least 1" has to complete,
> in
> > which case you'll have to join one & two into the same path.
> >
> > Hope that helps until somebody can confirm or deny.
> >
> > Cheers
> >
> > On Mon, Sep 19, 2016 at 3:07 PM, Joseph Napolitano <
> > joseph.napolit...@blueapron.com> wrote:
> >
> > > Can you confirm that "return 'branch_upload'" is "tabbed" over?  It's
> on
> > > the left margin in your Gist.
> > >
> > > On Mon, Sep 19, 2016 at 3:02 PM, Nikita Tovstoles <
> > > nik...@stripe.com.invalid> wrote:
> > >
> > >> Hi, folks:
> > >>
> > >> Airflow novice here trying to build a simple workflow where an
> upstream
> > >> task decides whether to follow an 'upload' branch or proceed directly
> to
> > >> subsequent join >> finish tasks. Regardless of whether branch callable
> > >> returns 'branch_skip_upload' or 'branch_skip_upload', tasks 'join',
> > >> 'finish' are skipped - and I want these two tasks to always execute.
> > >>
> > >> The source is here:
> > >> https://gist.github.com/dukehoops/dae9c45c2035d50e41fee7c7d75a50dd
> > >>
> > >> What am I doing incorrectly?
> > >>
> > >> Thank you.
> > >>
> > >> --
> > >> -nikita
> > >>
> > >
> > >
> > >
> > > --
> > > *Joe Napolitano *| Sr. Data Engineer
> > > www.blueapron.com | 5 Crosby Street, New York, NY 10013
> > >
> >
> >
> >
> > --
> > *Joe Napolitano *| Sr. Data Engineer
> > www.blueapron.com | 5 Crosby Street, New York, NY 10013
> >
>
>
>
> --
> -nikita
>



-- 
*Joe Napolitano *| Sr. Data Engineer
www.blueapron.com | 5 Crosby Street, New York, NY 10013

Re: BranchPythonOperator skips 'join' branch in all cases

[Nikita Tovstoles]

You're right, Joe. The key is to set a trigger_rule on joining task. see
updated gist and thanks!
https://gist.github.com/dukehoops/dae9c45c2035d50e41fee7c7d75a50dd

On Mon, Sep 19, 2016 at 12:16 PM, Joseph Napolitano <
joseph.napolit...@blueapron.com.invalid> wrote:

> My guess is that this line:
> [one, two, branch_skip_upload] >> join >> finish
>
> Requires all 3 tasks to complete. This may or may not be true--I haven't
> needed to rejoin branched tasks yet. Given this though, all 3 of these
> cannot complete. There may be a way to say "at least 1" has to complete, in
> which case you'll have to join one & two into the same path.
>
> Hope that helps until somebody can confirm or deny.
>
> Cheers
>
> On Mon, Sep 19, 2016 at 3:07 PM, Joseph Napolitano <
> joseph.napolit...@blueapron.com> wrote:
>
> > Can you confirm that "return 'branch_upload'" is "tabbed" over?  It's on
> > the left margin in your Gist.
> >
> > On Mon, Sep 19, 2016 at 3:02 PM, Nikita Tovstoles <
> > nik...@stripe.com.invalid> wrote:
> >
> >> Hi, folks:
> >>
> >> Airflow novice here trying to build a simple workflow where an upstream
> >> task decides whether to follow an 'upload' branch or proceed directly to
> >> subsequent join >> finish tasks. Regardless of whether branch callable
> >> returns 'branch_skip_upload' or 'branch_skip_upload', tasks 'join',
> >> 'finish' are skipped - and I want these two tasks to always execute.
> >>
> >> The source is here:
> >> https://gist.github.com/dukehoops/dae9c45c2035d50e41fee7c7d75a50dd
> >>
> >> What am I doing incorrectly?
> >>
> >> Thank you.
> >>
> >> --
> >> -nikita
> >>
> >
> >
> >
> > --
> > *Joe Napolitano *| Sr. Data Engineer
> > www.blueapron.com | 5 Crosby Street, New York, NY 10013
> >
>
>
>
> --
> *Joe Napolitano *| Sr. Data Engineer
> www.blueapron.com | 5 Crosby Street, New York, NY 10013
>



-- 
-nikita

Re: BranchPythonOperator skips 'join' branch in all cases

[Joseph Napolitano]

My guess is that this line:
[one, two, branch_skip_upload] >> join >> finish

Requires all 3 tasks to complete. This may or may not be true--I haven't
needed to rejoin branched tasks yet. Given this though, all 3 of these
cannot complete. There may be a way to say "at least 1" has to complete, in
which case you'll have to join one & two into the same path.

Hope that helps until somebody can confirm or deny.

Cheers

On Mon, Sep 19, 2016 at 3:07 PM, Joseph Napolitano <
joseph.napolit...@blueapron.com> wrote:

> Can you confirm that "return 'branch_upload'" is "tabbed" over?  It's on
> the left margin in your Gist.
>
> On Mon, Sep 19, 2016 at 3:02 PM, Nikita Tovstoles <
> nik...@stripe.com.invalid> wrote:
>
>> Hi, folks:
>>
>> Airflow novice here trying to build a simple workflow where an upstream
>> task decides whether to follow an 'upload' branch or proceed directly to
>> subsequent join >> finish tasks. Regardless of whether branch callable
>> returns 'branch_skip_upload' or 'branch_skip_upload', tasks 'join',
>> 'finish' are skipped - and I want these two tasks to always execute.
>>
>> The source is here:
>> https://gist.github.com/dukehoops/dae9c45c2035d50e41fee7c7d75a50dd
>>
>> What am I doing incorrectly?
>>
>> Thank you.
>>
>> --
>> -nikita
>>
>
>
>
> --
> *Joe Napolitano *| Sr. Data Engineer
> www.blueapron.com | 5 Crosby Street, New York, NY 10013
>



-- 
*Joe Napolitano *| Sr. Data Engineer
www.blueapron.com | 5 Crosby Street, New York, NY 10013

Re: BranchPythonOperator skips 'join' branch in all cases

[Joseph Napolitano]

Can you confirm that "return 'branch_upload'" is "tabbed" over?  It's on
the left margin in your Gist.

On Mon, Sep 19, 2016 at 3:02 PM, Nikita Tovstoles  wrote:

> Hi, folks:
>
> Airflow novice here trying to build a simple workflow where an upstream
> task decides whether to follow an 'upload' branch or proceed directly to
> subsequent join >> finish tasks. Regardless of whether branch callable
> returns 'branch_skip_upload' or 'branch_skip_upload', tasks 'join',
> 'finish' are skipped - and I want these two tasks to always execute.
>
> The source is here:
> https://gist.github.com/dukehoops/dae9c45c2035d50e41fee7c7d75a50dd
>
> What am I doing incorrectly?
>
> Thank you.
>
> --
> -nikita
>



-- 
*Joe Napolitano *| Sr. Data Engineer
www.blueapron.com | 5 Crosby Street, New York, NY 10013