shahar1 opened a new pull request, #77152:
URL: https://github.com/apache/airflow/pull/77152

   Proof of concept for [AIP-129: Backfill for Asset Consumers with Dated 
Partitions](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462817309/AIP-129+Backfill+for+Asset+Consumers+with+Dated+Partitions).
   
   This draft makes the implementation available for the AIP discussion. It is 
not ready to merge and does not imply approval of the proposal.
   
   Allow a supported `PartitionedAssetTimetable` consumer to backfill 
partitions from existing asset events without rerunning producers. Support 
calendar rollups and identity-mapped consumers, select the newest event for 
each required input, and retain existing partitioned-backfill reprocessing 
behavior.
   
   The branch includes internal Python support for explicit partition keys and 
detailed planning results. The REST routes support date-range creation and the 
existing dry run.
   
   ### Validation
   
   Rebased onto `upstream/main` at `8a016b710ff`.
   Using Breeze with Python 3.11, ran:
   
   ```bash
   breeze run --backend sqlite pytest 
airflow-core/tests/unit/models/test_backfill_consumer.py 
airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_backfills.py::TestConsumerBackfill
 airflow-core/tests/unit/models/test_backfill.py -q
   breeze run --backend postgres pytest 
airflow-core/tests/unit/models/test_backfill_consumer.py 
airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_backfills.py::TestConsumerBackfill
 airflow-core/tests/unit/models/test_backfill.py -q
   breeze run --backend mysql pytest 
airflow-core/tests/unit/models/test_backfill_consumer.py 
airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_backfills.py::TestConsumerBackfill
 airflow-core/tests/unit/models/test_backfill.py -q
   ```
   
   - SQLite: 163 passed, 2 skipped.
   - PostgreSQL 14: 165 passed.
   - MySQL 8.0: 165 passed.
   - Fast pre-commit hooks and manual-stage hooks passed, excluding the 
repository-prescribed interactive/dev-server hooks.
   
   These are targeted results, not a claim that the full CI matrix or 
documentation builds passed.
   
   ### Remaining work and known limits
   
   - Add explicit-key request fields and CLI support.
   - Expose detailed per-key results through the REST API and UI.
   - Filter detailed preview asset information according to the caller's 
permissions before exposing it.
   - Add configuration metadata for the candidate-key limit.
   - Complete the supported-schedule restrictions and design decisions 
documented in the AIP.
   - Live scheduling remains independent and can create another run for the 
same key.
   - The existing backfill lookup can prefer an older finished run over a 
queued run with no `start_date`; this draft does not fix that pre-existing 
behavior.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes — GitHub Copilot (Copilot SDK in VS Code; review, validation and 
PR preparation)
   
   Generated-by: GitHub Copilot (Copilot SDK in VS Code) following [the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   ---
   Drafted-by: GitHub Copilot (Copilot SDK in VS Code) (no human review before 
posting)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to