uranusjr opened a new pull request, #39367:
URL: https://github.com/apache/airflow/pull/39367
This is kind of the other side of `dataset_events` implemented in #38481.
The `inlet_events` context key allows the task to access past events associated
with a dataset that’s defined in the task’s `inlets`, like this:
```python
@task(inlets=my_ds)
def my_task(inlet_events):
last_event_timestamp = inlet_events[my_ds][-1].timestamp
```
Note that `inlets` is not logically related to using the dataset to schedule
a DAG. An inlet dataset may or may not also be present in the DAG’s schedule.
Subsequently, events accessed from `inlet_events` do not contain any logical
filtering—all past events are simply returned with a list-like interface.
This PR implements the basic structure including a lazy list-like structure
(that queries the database on-demand). I plan to add more changes in future PRs
after this is merged (all targeting 2.10):
* Rename `dataset_events` to `outlet_events` for consistency.
* Allow slicing syntax e.g. `inlet_events[ds][:-3]`.
* Some refactor to consolidate other list-like and lazy db access interfaces
we provide elsewhere, most significantly LazyXComAccess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
