Thanks Shahar, this looks interesting.
As you noted this was identified as future work in a couple of earlier AIPs
and this seems like a logical follow-up.

Constance, TP, Daniel, and others, I would really appreciate your
perspective on this.

Best regards,
Vikram


On Wed, Nov 18, 2026 at 11:00 AM Christos Bisias <[email protected]>
wrote:

> Hi Shahar,
>
> Thanks for working on this! I reviewed both the AIP and the draft PR and I
> think the proposal fills a valid gap. The overall approach seems to go in
> the right direction. I added a few comments on the AIP and the PR.
>
> On the live scheduling question, I would prefer no overlap. The backfill
> could lock the key's open pending row and link it to its own run, so live
> scheduling never creates a second run for it. I've added more details in
> the AIP comment.
>
> Christos
>
> On Mon, Nov 16, 2026 at 11:22 AM Shahar Epstein via dev <
> [email protected]> wrote:
>
> > Hello everyone,
> >
> > I'd like to start a discussion about a new AIP:
> > AIP-129 Backfill for Asset Consumers with Dated Partitions
> > https://cwiki.apache.org/confluence/x/HQiWGw
> > A draft proof of concept is available at
> > https://github.com/apache/airflow/pull/77152.
> >
> > This is the first in a series of AIPs that I plan in order to strengthen
> > asset-based scheduling in Airflow, starting with backfill. Without proper
> > backfill support, users have to rerun producers or trigger runs by hand,
> > which can be expensive.
> >
> > Today, a Dag scheduled on partitioned assets cannot be backfilled, even
> > when the input partitions already exist. For example, a new weekly report
> > consumer should be able to build past reports from existing daily sales
> > partitions without rerunning the sales producer.
> >
> > The proposal extends the existing backfill mechanism to supported
> > RollupMapper and IdentityMapper consumers:
> > - Select consumer partitions by date range or explicit keys.
> > - Use the newest retained asset event for each required input, rather
> than
> > replaying historical events.
> > - Create at most one consumer run per key per request, and report missing
> > inputs for incomplete partitions.
> > - Reuse the existing backfill controls, without rerunning producers or
> > changing live scheduling.
> >
> > Identity mapping is included for one-to-one partitioned pipelines, so
> that
> > moving a daily consumer from cron to an asset dependency does not mean
> > losing
> > backfill support. The AIP describes its date-resolution and
> historical-data
> > limitations.
> >
> > The first version is deliberately narrow. It refuses, among others,
> hourly
> > rollups, mixed rollup and identity inputs, OR conditions, and any
> consumer
> > when multi-team is enabled.
> >
> > I'd particularly welcome feedback on a specific design choice:
> > The proposal leaves live scheduling independent. A pending live row can
> > therefore produce another run for a key that a backfill has already
> > processed,
> > even without a new event arriving afterward. Is that trade-off acceptable
> > for the first version, or should skipping pending keys or coordinated
> > handling be required?
> >
> > Feedback on the scope and the other proposed defaults is welcome too.
> >
> >
> > Thanks,
> > Shahar
> >
>

Reply via email to