Thanks Shahar, this looks interesting. As you noted this was identified as future work in a couple of earlier AIPs and this seems like a logical follow-up.
Constance, TP, Daniel, and others, I would really appreciate your perspective on this. Best regards, Vikram On Wed, Nov 18, 2026 at 11:00 AM Christos Bisias <[email protected]> wrote: > Hi Shahar, > > Thanks for working on this! I reviewed both the AIP and the draft PR and I > think the proposal fills a valid gap. The overall approach seems to go in > the right direction. I added a few comments on the AIP and the PR. > > On the live scheduling question, I would prefer no overlap. The backfill > could lock the key's open pending row and link it to its own run, so live > scheduling never creates a second run for it. I've added more details in > the AIP comment. > > Christos > > On Mon, Nov 16, 2026 at 11:22 AM Shahar Epstein via dev < > [email protected]> wrote: > > > Hello everyone, > > > > I'd like to start a discussion about a new AIP: > > AIP-129 Backfill for Asset Consumers with Dated Partitions > > https://cwiki.apache.org/confluence/x/HQiWGw > > A draft proof of concept is available at > > https://github.com/apache/airflow/pull/77152. > > > > This is the first in a series of AIPs that I plan in order to strengthen > > asset-based scheduling in Airflow, starting with backfill. Without proper > > backfill support, users have to rerun producers or trigger runs by hand, > > which can be expensive. > > > > Today, a Dag scheduled on partitioned assets cannot be backfilled, even > > when the input partitions already exist. For example, a new weekly report > > consumer should be able to build past reports from existing daily sales > > partitions without rerunning the sales producer. > > > > The proposal extends the existing backfill mechanism to supported > > RollupMapper and IdentityMapper consumers: > > - Select consumer partitions by date range or explicit keys. > > - Use the newest retained asset event for each required input, rather > than > > replaying historical events. > > - Create at most one consumer run per key per request, and report missing > > inputs for incomplete partitions. > > - Reuse the existing backfill controls, without rerunning producers or > > changing live scheduling. > > > > Identity mapping is included for one-to-one partitioned pipelines, so > that > > moving a daily consumer from cron to an asset dependency does not mean > > losing > > backfill support. The AIP describes its date-resolution and > historical-data > > limitations. > > > > The first version is deliberately narrow. It refuses, among others, > hourly > > rollups, mixed rollup and identity inputs, OR conditions, and any > consumer > > when multi-team is enabled. > > > > I'd particularly welcome feedback on a specific design choice: > > The proposal leaves live scheduling independent. A pending live row can > > therefore produce another run for a key that a backfill has already > > processed, > > even without a new event arriving afterward. Is that trade-off acceptable > > for the first version, or should skipping pending keys or coordinated > > handling be required? > > > > Feedback on the scope and the other proposed defaults is welcome too. > > > > > > Thanks, > > Shahar > > >
