Hello everyone, I'd like to start a discussion about a new AIP: AIP-129 Backfill for Asset Consumers with Dated Partitions https://cwiki.apache.org/confluence/x/HQiWGw A draft proof of concept is available at https://github.com/apache/airflow/pull/77152.
This is the first in a series of AIPs that I plan in order to strengthen asset-based scheduling in Airflow, starting with backfill. Without proper backfill support, users have to rerun producers or trigger runs by hand, which can be expensive. Today, a Dag scheduled on partitioned assets cannot be backfilled, even when the input partitions already exist. For example, a new weekly report consumer should be able to build past reports from existing daily sales partitions without rerunning the sales producer. The proposal extends the existing backfill mechanism to supported RollupMapper and IdentityMapper consumers: - Select consumer partitions by date range or explicit keys. - Use the newest retained asset event for each required input, rather than replaying historical events. - Create at most one consumer run per key per request, and report missing inputs for incomplete partitions. - Reuse the existing backfill controls, without rerunning producers or changing live scheduling. Identity mapping is included for one-to-one partitioned pipelines, so that moving a daily consumer from cron to an asset dependency does not mean losing backfill support. The AIP describes its date-resolution and historical-data limitations. The first version is deliberately narrow. It refuses, among others, hourly rollups, mixed rollup and identity inputs, OR conditions, and any consumer when multi-team is enabled. I'd particularly welcome feedback on a specific design choice: The proposal leaves live scheduling independent. A pending live row can therefore produce another run for a key that a backfill has already processed, even without a new event arriving afterward. Is that trade-off acceptable for the first version, or should skipping pending keys or coordinated handling be required? Feedback on the scope and the other proposed defaults is welcome too. Thanks, Shahar
