rodrigoWizeLine edited a comment on issue #4766: [AIRFLOW-XXX] add prefix to file match in GCS_TO_S3 operator URL: https://github.com/apache/airflow/pull/4766#issuecomment-470125966 The #4546 only takes out the subfolder of the result returned array My change, look into the right subfolder for the specific file and not the whole bucket Take this as example My_bucket --Subfolder1 -----myfile1 -----myfile2 --Subfolder2 -----myfile3 -----myfile4 Right now the function will return My_Bucket/Subfolder1/ My_Bucket/Subfolder1/myfile1 My_Bucket/Subfolder1/myfile2 My_Bucket/Subfolder2/ My_Bucket/Subfolder2/myfile3 My_Bucket/Subfolder2/myfile4 So if you are looking into a inner subfolder will look into the whole bucket and if is too big it will take forever just to compare Also right now it will never match because you are trying to sync the full path in GCS and just the file name.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
