tokoko commented on issue #35297:
URL: https://github.com/apache/airflow/issues/35297#issuecomment-1843745172
@blag my 2 cents, I think dataset event extras would make much more reliable
solution for state persistence for lots of use cases, for example if single
logical dataset can be updated from multiple dags. Also if you're using
data-aware scheduling, using xcoms would mean that you have to additionally
track down the dag from which dataset was last updated, which sort of defeats
the whole purpose of datasets imho.
@mpgreg If you are really looking for a solution.. today. one (ugly) way to
do it would be to call `setattr(context['ti'].task, 'outlet_extra', payload)`
from somewhere in the `execute` or maybe `post_execute` methods. And then you
have to define your own implementation for DatasetManager (and set
AIRFLOW__CORE__DATASET_MANAGER_CLASS) something like this:
```
class ExtraDatasetManager(DatasetManager):
def register_dataset_change(
self, *, task_instance, dataset: Dataset, extra=None, session:
Session, **kwargs
) -> None:
if not extra and hasattr(task_instance.task, 'outlet_extra'):
extra = getattr(task_instance.task, 'outlet_extra')
super().register_dataset_change(task_instance=task_instance,
dataset=dataset,extra=extra, session=session, **kwargs)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
