uranusjr commented on code in PR #73015:
URL: https://github.com/apache/airflow/pull/73015#discussion_r4410484234
##########
providers/amazon/src/airflow/providers/amazon/aws/bundles/s3.py:
##########
@@ -37,23 +63,40 @@ class S3DagBundle(BaseDagBundle):
:param bucket_name: The name of the S3 bucket containing the Dag files.
:param prefix: Optional subdirectory within the S3 bucket where the Dags
are stored.
A trailing slash is optional.
- If None, Dags are assumed to be at the root of the bucket
(Optional).
+ If None, Dags are assumed to be at the root of the bucket.
Mutually exclusive
+ with ``archive_key`` (Optional).
+ :param archive_key: Optional S3 key of a ``.tar.gz`` archive holding the
Dag files. When set,
+ the bundle is staged by downloading this single object and
unpacking it
+ locally, instead of downloading every object under a
prefix one at a time.
+ The archive is then the only source of the Dags:
``archive_key`` is mutually
+ exclusive with ``prefix``, so there is no second copy to
keep in sync and no
+ fallback to a per-object sync. Members are extracted
relative to the bundle
+ root, e.g. created with ``tar -C <dags_dir> -czf
dags.tar.gz .`` (Optional).
"""
supports_versioning = False
+ archive_etag_marker = ".airflow_bundle_archive_etag"
Review Comment:
This would be written into the Dags folder itself and visible to the Dag
processor and any user listing the bundle. Consider keeping it next to the
bundle dir instead (e.g. in the staging root or `_tracking`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
