Hi Shahar,

Thanks for working on this! I reviewed both the AIP and the draft PR and I
think the proposal fills a valid gap. The overall approach seems to go in
the right direction. I added a few comments on the AIP and the PR.

On the live scheduling question, I would prefer no overlap. The backfill
could lock the key's open pending row and link it to its own run, so live
scheduling never creates a second run for it. I've added more details in
the AIP comment.

Christos

On Mon, Nov 16, 2026 at 11:22 AM Shahar Epstein via dev <
[email protected]> wrote:

> Hello everyone,
>
> I'd like to start a discussion about a new AIP:
> AIP-129 Backfill for Asset Consumers with Dated Partitions
> https://cwiki.apache.org/confluence/x/HQiWGw
> A draft proof of concept is available at
> https://github.com/apache/airflow/pull/77152.
>
> This is the first in a series of AIPs that I plan in order to strengthen
> asset-based scheduling in Airflow, starting with backfill. Without proper
> backfill support, users have to rerun producers or trigger runs by hand,
> which can be expensive.
>
> Today, a Dag scheduled on partitioned assets cannot be backfilled, even
> when the input partitions already exist. For example, a new weekly report
> consumer should be able to build past reports from existing daily sales
> partitions without rerunning the sales producer.
>
> The proposal extends the existing backfill mechanism to supported
> RollupMapper and IdentityMapper consumers:
> - Select consumer partitions by date range or explicit keys.
> - Use the newest retained asset event for each required input, rather than
> replaying historical events.
> - Create at most one consumer run per key per request, and report missing
> inputs for incomplete partitions.
> - Reuse the existing backfill controls, without rerunning producers or
> changing live scheduling.
>
> Identity mapping is included for one-to-one partitioned pipelines, so that
> moving a daily consumer from cron to an asset dependency does not mean
> losing
> backfill support. The AIP describes its date-resolution and historical-data
> limitations.
>
> The first version is deliberately narrow. It refuses, among others, hourly
> rollups, mixed rollup and identity inputs, OR conditions, and any consumer
> when multi-team is enabled.
>
> I'd particularly welcome feedback on a specific design choice:
> The proposal leaves live scheduling independent. A pending live row can
> therefore produce another run for a key that a backfill has already
> processed,
> even without a new event arriving afterward. Is that trade-off acceptable
> for the first version, or should skipping pending keys or coordinated
> handling be required?
>
> Feedback on the scope and the other proposed defaults is welcome too.
>
>
> Thanks,
> Shahar
>

Reply via email to