potiuk commented on issue #8792:
URL: https://github.com/apache/airflow/issues/8792#issuecomment-1473647642
> Any operator can be resulted in failure. Currently as far as I can tell we
don't push to xcom information about the failure.
If we want to do this we should probably make it generic (as much as we
can). Personally I would prefer this push not to be the default behavior.
TL;DR; I would be on having the push_on_failure as an option - but only for
the few "generic" operators we have - KPO/Docker/Bash etc.
I thought a bit on that and this is very much philosophical issue :). Whie I
have no "only one good" solution, intuitively I think pushing xcom on failure
should be added as option for "generic" operators - like
KPO/Docker/Bash/Python, but we shuold not do it for "specific" operator.
Let me explain my line of thoughts - maybe that will lead
Generally speaking default behaviour for "regular" operator is that they
push to xcom whatever is returned by execute() (unless do_xcom_push is false).
That's the current semantics. And when there is a failure we CANNOT push
anything because the execute () method does not return anything, unless we
change this semantics.
Also doing so is a bit superfluous if you consider that the author of the
operator might choose to do it on their own:
```
def execute():
try:
....
except Exception():
Xcom.push(...)
raise
```
And they will be able to put a message if they really want to push something
on failure. And it's up to the author of the operator to define the behaviour.
As a user, when you get an operator that does specific thing, it is generally
"closed" - it does what it tells you, you have not much freedom there as a
user, the author already made some decisions for you. Of course as a user you
can extend such operator and then you can change the behaviour and add similar
try/except wrapper.
The thing with KPO (and few other generic operators like Docker, Bash,
Python) is that this is generic operator - and as a user you have more freedom
to decide what and where happens - by providing a bash script, image,
pod_template etc. - and this is where you also might get to decide what to do
in case of the failure. But ... you cannot really (as a user) currently make a
decision whether to push it or not now (without extending the operator) - so
suddenly the "generic" operators are not as generic any more. Yes you can
extend it -but "generic" operator's philsophy is that they shoud not need to be
extended, whereas for the "specific" operators, "extension" is the only way of
changing the behaoviour that hte author of the operator made decisions on.
I am not super strong on it, but wanted to explain not only what I think but
also what led me to thinking this is the best approach. Maybe this will be good
for others to think that through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
