> It handles everything. Now if you want to send a Slack message from a
> PythonOperator
you need to initialize a hook, find the right function to invoke etc.
> And to Elad point " "I know there is an operator that does X, so I will
just use it inside the python function I invoke from the
I am really torn on that one to be honest.
I am OK with the error (with the note that it will likely break a lot of
workflows), I am ok with the warning as well as a softer way of letting the
user know they are doing it wrong).
But ultimately, I'd really want we (re) consider if we cannot make it
The reason users are sure they can use operators like that is that it has
worked for a long time - hell I even wrote a custom nested operator in the past
(pre 2.0 admittedly).
So this pr should only be a warning by default, or a config option to warn but
not error
Alternatively do we just
Just to add to the discussion - a discussion raised today
https://github.com/apache/airflow/discussions/38311 where the user is sure
that they can use operators in such a way as described above, and even used
the term "nested operator".
I think getting
The issue here is not just about decorators it happens also with regular
operators (operator inside operator) and also with operator inside
on_x_callback
For example:
https://stackoverflow.com/questions/64291042/airflow-call-a-operator-inside-a-function/
I see that we have already (thanks David!) a PR:
https://github.com/apache/airflow/pull/37937 to forbid this use (which is
cool and I am glad my discussion had some ripple effect :D ).
I am quite happy to get this one merged once it passes tests/reviews, but I
would still want to explore future
One wrinkle to the have cake and eat it too approach is deferrable
operators. It doesn't seem it would be very practical to resume back into
the operator that is nested inside a taskflow function. One solution would
be to run the trigger in process like we currently do with `dag.test()`.
That
I guess PR to propose something for that would be a good start :). I I
am fine - for now with forbidding it "as-is" and then maybe next step
once we do it is to figure out a nicer way of doing it than Hooks.
Maybe it can be done in parallel.
On Fri, Mar 1, 2024 at 9:19 PM Blain David wrote:
>
>
It's certainly possible to check from where a python method is being called
using traceback.
I do think prohibiting the execute method of an operator being called manually
would be a good idea, I've also came accross this in multiple DAG's and this is
ugly and looks like a hack.
Maybe in the
I can't see which problem is solved by allowing running one operator inside
another.
*Propagate templated fields?*
In most cases it could be changed for a particular task or entire operator
by monkey patching, which in this case more safe rather than run operator
in operator. Or even available
Yeah. I kinda like (and see it emerging from the discussion) that we can
(which I love) have cake and eat it too :). Means - I think we can have
both 1) and 2) ...
1) We should raise an error if someone uses AnOperator in task context (the
way TP described it would work nicely) - making calling
I think manually using *execute*, *poke*, *choose_branch* and etc. should
be considered as an antipattern, these methods only should be invoked by
Airflow Worker.
You never know what should be done before, especially if it about
deferrable operator or sensors in reschedule mode
On Tue, 27 Feb
Interesting! Jarek and I had a bit of a discussion on slack a couple of
months ago in the context of Operators Need to Die (if I recall correctly).
The idea behind Operators is that you have a vetted way of executing a
certain logic apart from the boilerplate (i.e. execute, post_execute) that
This kind of ties back to Bolke’s Operator Must Die manifesto—you shouldn’t use
the operator class here at all in this situation, but the corresponding hook
instead.
Regarding preventing this anti-pattern, I can think of two ways:
Add a check in BaseOperator to detect whether __init__ is
Or (and here things get interesting and I am looking forward to discussions
it can bring) alternatively - maybe we could make the pattern **work**
actually If we could dynamically instrument such `execute` methods
and do all the pre/post processing on the operator, maybe it could be a
useful
Hello here,
I have seen recently at least a few times that our users started to use a
strange pattern
@task(task_id='some_id', provide_context=True)
def some_dummy_task(**context):
ti = context['ti']
cmd2 = 'echo "pushing 2"'
dbt_task = BashOperator(task_id='some_dummy_task_id',
16 matches
Mail list logo
