moiseenkov opened a new pull request, #32486:
URL: https://github.com/apache/airflow/pull/32486
This PR fixes the following case.
The goal is to copy files with prefix `source/foo.txt` to the folder `dest/`
within a single GCS bucket.
1. Create a GCS bucket and upload two files to source directory like this:
```
gs://my-bucket/source/foo.txt
gs://my-bucket/source/foo.txt.abc
gs://my-bucket/source/foo.txt/subfolder/file.txt
```
2. Upload the following DAG to a Cloud Composer environment:
```python
from airflow import DAG
from airflow.providers.google.cloud.transfers.gcs_to_gcs import
GCSToGCSOperator
from datetime import datetime
with DAG(
dag_id="gcs_to_gcs_fail_example",
schedule_interval=None,
catchup=False,
start_date=datetime(2021,1,1)
) as dag:
copy_file = GCSToGCSOperator(
task_id="copy_file",
source_bucket="my-bucket",
source_object="source/foo.txt",
destination_object="dest/",
)
copy_file
```
3. Run the DAG
**Expected bucket state:**
```
gs://my-bucket/source/foo.txt
gs://my-bucket/source/foo.txt.abc
gs://my-bucket/source/foo.txt/subfolder/file.txt
gs://my-bucket/dest/foo.txt
gs://my-bucket/dest/foo.txt.abc
gs://my-bucket/dest/foo.txt/subfolder/file.txt
```
**Actual (incorrect) bucket state:**
```
gs://my-bucket/source/foo.txt
gs://my-bucket/source/foo.txt.abc
gs://my-bucket/source/foo.txt/subfolder/file.txt
gs://my-bucket/dest/source/foo.txt
gs://my-bucket/dest/source/foo.txt.abc
gs://my-bucket/dest/source/foo.txt/subfolder/file.txt
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
