This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new a12a4a5d93 gcs to gcs deprecation removal (#39726)
a12a4a5d93 is described below
commit a12a4a5d93f97f6de0be40339421d71bb4b0da05
Author: Bowrna <[email protected]>
AuthorDate: Mon May 27 15:13:04 2024 +0530
gcs to gcs deprecation removal (#39726)
---
.../operators/transfer/gcs_to_gcs.rst | 28 +++------------------
tests/always/test_example_dags.py | 1 -
.../google/cloud/gcs/example_gcs_to_gcs.py | 29 +++-------------------
3 files changed, 7 insertions(+), 51 deletions(-)
diff --git
a/docs/apache-airflow-providers-google/operators/transfer/gcs_to_gcs.rst
b/docs/apache-airflow-providers-google/operators/transfer/gcs_to_gcs.rst
index 0fe146c200..6e49dc4229 100644
--- a/docs/apache-airflow-providers-google/operators/transfer/gcs_to_gcs.rst
+++ b/docs/apache-airflow-providers-google/operators/transfer/gcs_to_gcs.rst
@@ -116,28 +116,11 @@ Copy multiple files
There are several ways to copy multiple files, various examples of which are
presented following.
-.. exampleinclude::
/../../tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py
- :language: python
- :dedent: 4
- :start-after: [START howto_operator_gcs_to_gcs_wildcard]
- :end-before: [END howto_operator_gcs_to_gcs_wildcard]
-
-The ``source_object`` value may contain one wild card, denoted as "*". All
files matching the wild card expression will
-be copied. In this example, all root level files ending with ``.txt`` in
``BUCKET_1_SRC`` will be copied to the ``data``
-folder in ``BUCKET_1_DST``, with file names unchanged.
-
-.. exampleinclude::
/../../tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py
- :language: python
- :dedent: 4
- :start-after: [START howto_operator_gcs_to_gcs_delimiter]
- :end-before: [END howto_operator_gcs_to_gcs_delimiter]
-
-For source_objects with no wildcard, all files in source_objects would be
listed, using provided delimiter if any.
-Then copy files from source_objects to destination_object and rename each
source file.
-
As previously stated, the ``delimiter`` field, as well as utilizing a wildcard
(``*``) in the source object(s),
are both deprecated. Thus, it is not recommended to use them - but to utilize
``match_glob`` instead, as follows:
+The following example would copy the files that matches the glob pattern in
``data/`` folder from ``BUCKET_1_SRC`` GCS bucket to the ``backup/`` folder in
``BUCKET_1_DST`` bucket.
+
.. exampleinclude::
/../../tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py
:language: python
:dedent: 4
@@ -153,9 +136,7 @@ the ``BUCKET_1_SRC`` GCS bucket to the ``backup/`` folder
in ``BUCKET_1_DST`` bu
:start-after: [START howto_operator_gcs_to_gcs_without_wildcard]
:end-before: [END howto_operator_gcs_to_gcs_without_wildcard]
-The delimiter field may be specified to select any source files starting with
``source_object`` and ending with the
-value supplied to ``delimiter``. This example uses the ``delimiter`` value to
implement the same functionality as the
-prior example.
+
.. exampleinclude::
/../../tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py
:language: python
@@ -164,8 +145,7 @@ prior example.
:end-before: [END howto_operator_gcs_to_gcs_list]
Lastly, files may be copied by omitting the ``source_object`` argument and
instead supplying a list to ``source_objects``
-argument. In this example, ``OBJECT_1`` and ``OBJECT_2`` will be copied from
``BUCKET_1_SRC`` to ``BUCKET_1_DST``. Instead
-of specific file names, the list can contain one or more wild card
expressions, each with no more than one wild card.
+argument. In this example, ``OBJECT_1`` and ``OBJECT_2`` will be copied from
``BUCKET_1_SRC`` to ``BUCKET_1_DST``.
Supplying a list of size 1 functions the same as supplying a value to the
``source_object`` argument.
Move single file
diff --git a/tests/always/test_example_dags.py
b/tests/always/test_example_dags.py
index 8801743205..88db9a0556 100644
--- a/tests/always/test_example_dags.py
+++ b/tests/always/test_example_dags.py
@@ -51,7 +51,6 @@ IGNORE_AIRFLOW_PROVIDER_DEPRECATION_WARNING: tuple[str, ...]
= (
"tests/system/providers/google/cloud/bigquery/example_bigquery_operations.py",
"tests/system/providers/google/cloud/dataproc/example_dataproc_gke.py",
"tests/system/providers/google/cloud/gcs/example_gcs_sensor.py",
- "tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py",
"tests/system/providers/google/cloud/kubernetes_engine/example_kubernetes_engine.py",
"tests/system/providers/google/cloud/kubernetes_engine/example_kubernetes_engine_async.py",
"tests/system/providers/google/cloud/kubernetes_engine/example_kubernetes_engine_job.py",
diff --git a/tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py
b/tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py
index 5885ae2834..746f877678 100644
--- a/tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py
+++ b/tests/system/providers/google/cloud/gcs/example_gcs_to_gcs.py
@@ -166,19 +166,9 @@ with DAG(
)
# [END howto_operator_gcs_to_gcs_single_file]
- # [START howto_operator_gcs_to_gcs_wildcard]
- copy_files_with_wildcard = GCSToGCSOperator(
- task_id="copy_files_with_wildcard",
- source_bucket=BUCKET_NAME_SRC,
- source_object="data/*.txt",
- destination_bucket=BUCKET_NAME_DST,
- destination_object="backup/",
- )
- # [END howto_operator_gcs_to_gcs_wildcard]
-
# [START howto_operator_gcs_to_gcs_without_wildcard]
- copy_files_without_wildcard = GCSToGCSOperator(
- task_id="copy_files_without_wildcard",
+ copy_files = GCSToGCSOperator(
+ task_id="copy_files",
source_bucket=BUCKET_NAME_SRC,
source_object="subdir/",
destination_bucket=BUCKET_NAME_DST,
@@ -186,17 +176,6 @@ with DAG(
)
# [END howto_operator_gcs_to_gcs_without_wildcard]
- # [START howto_operator_gcs_to_gcs_delimiter]
- copy_files_with_delimiter = GCSToGCSOperator(
- task_id="copy_files_with_delimiter",
- source_bucket=BUCKET_NAME_SRC,
- source_object="data/",
- destination_bucket=BUCKET_NAME_DST,
- destination_object="backup/",
- delimiter=".txt",
- )
- # [END howto_operator_gcs_to_gcs_delimiter]
-
# [START howto_operator_gcs_to_gcs_match_glob]
copy_files_with_match_glob = GCSToGCSOperator(
task_id="copy_files_with_match_glob",
@@ -264,9 +243,7 @@ with DAG(
sync_to_subdirectory,
sync_from_subdirectory,
copy_single_file,
- copy_files_with_wildcard,
- copy_files_without_wildcard,
- copy_files_with_delimiter,
+ copy_files,
copy_files_with_match_glob,
copy_files_with_list,
move_single_file,
