aoelvp94 opened a new pull request, #73015:
URL: https://github.com/apache/airflow/pull/73015
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
closes: #73010
`S3DagBundle` stages the bundle by downloading the prefix one object at a
time (`S3Hook.sync_to_local_dir`), so a bundle of N small files costs ~N GET
(+N HEAD) sequential requests on every component that stages it. Under
KubernetesExecutor every task instance is a fresh pod with an empty bundle
directory, which puts the full staging cost on the critical path of every task
start (~20+ s for a 400-file bundle against real S3), and even a steady-state
"nothing changed" refresh still pays a full per-object walk.
This PR adds an opt-in `archive_key` kwarg to `S3DagBundle`: when set,
`refresh()` stages the bundle from a single `.tar.gz` object instead — one
`HEAD` (skipped staging entirely when the ETag matches the last staged
archive), one `GET`, a safe unpack (`tarfile` with `filter="data"`), and an
atomic rename swap so a partially staged tree is never observable. On any
archive problem (missing object, corrupt tar, unpack error) staging falls back
to the existing per-object sync of `prefix`, so the worst case is exactly the
current behavior. Publishing the archive is the deployer's job (documented:
`tar -C dags . | aws s3 cp` in the same CI that syncs the prefix).
Measured with the reproduction environment from the issue (fully local:
Airflow 3 + MinIO, identical 400-file / ~3 MiB file set —
https://github.com/aoelvp94/airflow-s3-dag-bundle-staging-demo):
| staging strategy | wall-clock (median) | requests served |
|---|---:|---:|
| per-object sync | 15.8 s | 802 |
| single archive | 1.5 s | 5 |
Design notes:
* One class, opt-in: no `archive_key` ⇒ nothing changes. `prefix` keeps its
meaning as the fallback source, and `_initialize()` skips the prefix-existence
check only when the archive exists (archive-only buckets work; missing archives
warn and validate/stage from the prefix as today).
* The local ETag marker lives inside the bundle directory, so a fallback
per-object sync (`delete_stale=True`) removes it — a later re-published archive
can never be wrongly skipped against a stale marker.
* Deletions need no reconciliation logic in the archive path: the unpacked
tree replaces the previous one wholesale.
* The staging strategy is provider-agnostic; if there is interest, the same
shape could be extracted for the GCS bundle (and the proposed Azure Blob bundle
in #67016) in a follow-up.
Covered by 6 new unit tests (moto): archive staging, ETag skip + re-staging
on change, archive-only bucket, fallback on corrupt and on missing archive, and
marker cleanup after fallback. Docs updated in
`providers/amazon/docs/bundles/index.rst`. `prek` static checks pass locally.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
— all code, tests, and docs were reviewed by the author, and static checks
plus the provider unit test suite were run locally.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
* Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information. Note: commit author/co-author name and email in commits
become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License
Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment:
`{pr_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
You can add this file in a follow-up commit after the PR is created so you
know the PR number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
