hussein-awala commented on PR #70225:
URL: https://github.com/apache/airflow/pull/70225#issuecomment-5059278192
> Thanks! I'll take a deeper look, but I'd first like to understand why we
need a sensor like this.
>
> I thought we could always use a custom `PartitionMapper` to achieve
something similar. Are there any use cases I'm missing? Thanks!
@Lee-W Good point, but they're at different layers. `PartitionMapper` is
scheduler-side: it decides whether/how a run is *created* from
partition-aligned events, and its only count gate (`MinimumCount`/`WaitForAll`)
works over a declared partition-key window on a `PartitionedAssetTimetable`.
`AssetEventSensor` is a task-level primitive: it waits *inside an
already-running Dag* (cron/manual/any trigger) until events matching arbitrary
filters reach a count, then passes them downstream via XCom. Two things a
mapper can't do:
1. Mid-Dag gating in a non-asset-scheduled Dag (e.g. wait for ≥3 events on
today's partition before continuing).
2. Filter on `extra`/time-range and reuse the payload (e.g. wait for N
events with `extra={"status":"validated"}`, then `process_result` them for
downstream tasks), mappers only see `partition_key`.
We can say it's a wrapper for the `inlet_events` but with the sensor
features (wait for the events, max wait, soft fail, ...)
So complementary, not redundant. Can add this to the docs if useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
