dstandish commented on code in PR #31201:
URL: https://github.com/apache/airflow/pull/31201#discussion_r1204807909
##########
airflow/providers/apache/livy/operators/livy.py:
##########
@@ -207,4 +207,5 @@ def execute_complete(self, context: Context, event:
dict[str, Any]) -> Any:
self.task_id,
event["response"],
)
+ context["ti"].xcom_push(key="app_id",
value=self.get_hook().get_batch(event["batch_id"])["appId"])
Review Comment:
i think it's fine to wait until needed.
but i thought that's what @bdsoha was saying here:
> It was very useful, as I used in to retrieve application logs from yarn
following a tasks execution.
re the "wasted record" concern, i am not concerned with that. we're talking
about "exceptional" circumstances, when the creation fails right? well, that
isn't what happens ordinarily. so you're talking about a small number of
cases. but, if it was successful the record would be created anyway and you
it's certainly no worse. in the long run, all airflow databases need cleanup /
purging of old records. and that's why we added a helper command for that.
i think @potiuk's concern is about consistency. consistency is good but
also we can make steps in a direction without necessarily always having to
update all operators to fit a certain convention. and there is inevitably
variation in behavior between operators, each of which may be tailored to
different services and different use cases, so one always has to know what the
behavior is in order to use it properly. and example dags and docs are there
to help with that.
related note: xcoms are cleared when tasks are cleared so it doesn't have an
impact there.
so, i'm not saying you need to push xcom immediately just saying i don't see
a probem with doing so if there's a good use case for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
