[jira] [Created] (AIRFLOW-2348) GoogleCloudStorageToGoogleCloudStorageOperator copies full source path

[Christos Soulios (JIRA)]

Christos Soulios created AIRFLOW-2348:
-----------------------------------------

             Summary: GoogleCloudStorageToGoogleCloudStorageOperator copies 
full source path
                 Key: AIRFLOW-2348
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2348
             Project: Apache Airflow
          Issue Type: Bug
          Components: gcp
    Affects Versions: 2.0.0
            Reporter: Christos Soulios


I use the GoogleCloudStorageToGoogleCloudStorageOperator to copy a directory 
recursively from source dir to target dir. When files are copied, the full 
source dir hierarchy is included. 

For example, I want to copy the following files to /path/to/destination

{{/path/to/source/a/file1.txt}}

{{/path/to/source/a/file2.txt}}

{{/path/to/source/b/file3.txt}}

 

When running:
{{stage_files = GoogleCloudStorageToGoogleCloudStorageOperator(}}
{{ task_id='task',}}
{{ source_bucket='gs_bucket',}}
{{ source_object='/path/to/source/*',}}
{{destination_object='/path/to/destination')}}

I get the following result:

{{/path/to/destination/path/to/source/a/file1.txt}}

{{/path/to/destination/path/to/source/a/file2.txt}}

{{/path/to/destination/path/to/source/b/file3.txt}}

 

I would expect the the operator behavior is similar to gsutil and would return 
the following result:

{{/path/to/destination/a/file1.txt}}

{{/path/to/destination/a/file2.txt}}

{{/path/to/destination/b/file3.txt}}

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)