Hi Shahar, Thanks for working on this! I reviewed both the AIP and the draft PR and I think the proposal fills a valid gap. The overall approach seems to go in the right direction. I added a few comments on the AIP and the PR.
On the live scheduling question, I would prefer no overlap. The backfill could lock the key's open pending row and link it to its own run, so live scheduling never creates a second run for it. I've added more details in the AIP comment. Christos On Mon, Nov 16, 2026 at 11:22 AM Shahar Epstein via dev < [email protected]> wrote: > Hello everyone, > > I'd like to start a discussion about a new AIP: > AIP-129 Backfill for Asset Consumers with Dated Partitions > https://cwiki.apache.org/confluence/x/HQiWGw > A draft proof of concept is available at > https://github.com/apache/airflow/pull/77152. > > This is the first in a series of AIPs that I plan in order to strengthen > asset-based scheduling in Airflow, starting with backfill. Without proper > backfill support, users have to rerun producers or trigger runs by hand, > which can be expensive. > > Today, a Dag scheduled on partitioned assets cannot be backfilled, even > when the input partitions already exist. For example, a new weekly report > consumer should be able to build past reports from existing daily sales > partitions without rerunning the sales producer. > > The proposal extends the existing backfill mechanism to supported > RollupMapper and IdentityMapper consumers: > - Select consumer partitions by date range or explicit keys. > - Use the newest retained asset event for each required input, rather than > replaying historical events. > - Create at most one consumer run per key per request, and report missing > inputs for incomplete partitions. > - Reuse the existing backfill controls, without rerunning producers or > changing live scheduling. > > Identity mapping is included for one-to-one partitioned pipelines, so that > moving a daily consumer from cron to an asset dependency does not mean > losing > backfill support. The AIP describes its date-resolution and historical-data > limitations. > > The first version is deliberately narrow. It refuses, among others, hourly > rollups, mixed rollup and identity inputs, OR conditions, and any consumer > when multi-team is enabled. > > I'd particularly welcome feedback on a specific design choice: > The proposal leaves live scheduling independent. A pending live row can > therefore produce another run for a key that a backfill has already > processed, > even without a new event arriving afterward. Is that trade-off acceptable > for the first version, or should skipping pending keys or coordinated > handling be required? > > Feedback on the scope and the other proposed defaults is welcome too. > > > Thanks, > Shahar >
