[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831528#comment-16831528 ] ASF GitHub Bot commented on AIRFLOW-3720: - ashb commented on pull request #4546: AIRFLOW-3720 URL: https://github.com/apache/airflow/pull/4546 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 1.10.4, 2.0.0 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829723#comment-16829723 ] ASF subversion and git services commented on AIRFLOW-3720: -- Commit 247d0cb3ebe89791b7e6e817669f1ca13470041b in airflow's branch refs/heads/v1-10-stable from Rodrigo Chaparro Plata Hernandez [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=247d0cb ] [AIRFLOW-3720] Fix missmatch while comparing GCS and S3 files (#4766) (cherry picked from commit 60b9023ed92b31a75dbdf8b33ce7e9c2bc3637d1) > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 1.10.4, 2.0.0 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829401#comment-16829401 ] ASF subversion and git services commented on AIRFLOW-3720: -- Commit 247d0cb3ebe89791b7e6e817669f1ca13470041b in airflow's branch refs/heads/v1-10-test from Rodrigo Chaparro Plata Hernandez [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=247d0cb ] [AIRFLOW-3720] Fix missmatch while comparing GCS and S3 files (#4766) (cherry picked from commit 60b9023ed92b31a75dbdf8b33ce7e9c2bc3637d1) > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 1.10.4, 2.0.0 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827917#comment-16827917 ] ASF subversion and git services commented on AIRFLOW-3720: -- Commit 3d6f1ae7e47644d5fb7823a6fd1a9f861e1c077b in airflow's branch refs/heads/v1-10-stable from Rodrigo Chaparro Plata Hernandez [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=3d6f1ae ] [AIRFLOW-3720] Fix missmatch while comparing GCS and S3 files (#4766) (cherry picked from commit 60b9023ed92b31a75dbdf8b33ce7e9c2bc3637d1) > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 1.10.4, 2.0.0 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827922#comment-16827922 ] ASF subversion and git services commented on AIRFLOW-3720: -- Commit fcfc8e963c78f31e71369246a5eb67099b2dd4a7 in airflow's branch refs/heads/v1-10-test from Rodrigo Chaparro Plata Hernandez [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=fcfc8e9 ] [AIRFLOW-3720] Fix missmatch while comparing GCS and S3 files (#4766) (cherry picked from commit 60b9023ed92b31a75dbdf8b33ce7e9c2bc3637d1) > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 1.10.4, 2.0.0 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827716#comment-16827716 ] ASF subversion and git services commented on AIRFLOW-3720: -- Commit 667acabfea726a1db47f5b6bcdf63b9ffab8c833 in airflow's branch refs/heads/v1-10-test from Rodrigo Chaparro Plata Hernandez [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=667acab ] [AIRFLOW-3720] Fix missmatch while comparing GCS and S3 files (#4766) > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 2.0.0 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827719#comment-16827719 ] ASF subversion and git services commented on AIRFLOW-3720: -- Commit f449b2951edc2709bb71af4fdae1685241fe15a5 in airflow's branch refs/heads/v1-10-stable from Rodrigo Chaparro Plata Hernandez [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f449b29 ] [AIRFLOW-3720] Fix missmatch while comparing GCS and S3 files (#4766) > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 1.10.4, 2.0.0 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827713#comment-16827713 ] ASF subversion and git services commented on AIRFLOW-3720: -- Commit 60b9023ed92b31a75dbdf8b33ce7e9c2bc3637d1 in airflow's branch refs/heads/master from Rodrigo Chaparro Plata Hernandez [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=60b9023 ] [AIRFLOW-3720] Fix missmatch while comparing GCS and S3 files (#4766) > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 2.0.0 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827711#comment-16827711 ] ASF GitHub Bot commented on AIRFLOW-3720: - potiuk commented on pull request #4766: [AIRFLOW-3720] Add prefix to file match in GCS_TO_S3 operator to avoid missmatch URL: https://github.com/apache/airflow/pull/4766 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: aws >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 2.0.0 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[
https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785426#comment-16785426
]
jack commented on AIRFLOW-3720:
---
[~Turkel] I'm wondering why this issue wasn't raised when
[https://github.com/apache/airflow/pull/4371] was discussed.
The code changed to:
{code:java}
files = list(set(files) - set(existing_files)){code}
Can you give example how it doesn't work for you but works for others? Am I
missing something here?
> GoogleCloudStorageToS3Operator - incorrect folder compare
> --
>
> Key: AIRFLOW-3720
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3720
> Project: Apache Airflow
> Issue Type: Bug
> Components: boto3
>Affects Versions: 1.10.0
>Reporter: Chaim
>Assignee: Chaim
>Priority: Major
> Fix For: 2.0.0
>
>
> the code that compares folders from gcp to s3 is incorrect.
> the code is:
> files = set(files) - set(existing_files)
> but the list from gcp has a "/" to the name, for example: "myfolder/", while
> in s3 it does not have "/" so the folder is "myfolder"
> the result is that the code tries to recopy the folder name but fails since
> it already exists
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784801#comment-16784801 ] jack commented on AIRFLOW-3720: --- [~yohei] didnt you encounter a similar issue? i remember you fixed it didnt you? > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: boto3 >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 2.0.0 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744957#comment-16744957 ] ASF GitHub Bot commented on AIRFLOW-3720: - chaimt commented on pull request #4546: AIRFLOW-3720 URL: https://github.com/apache/airflow/pull/4546 Make sure you have checked _all_ steps below. ### Jira - [ ] My PR addresses the following [Airflow Jira] - https://issues.apache.org/jira/browse/AIRFLOW-3720 ### Description - [ ] When comparing google storage to s3 buckets, folder names are not compared correctly This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: boto3 >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 1.10.2 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-3720) GoogleCloudStorageToS3Operator - incorrect folder compare
[ https://issues.apache.org/jira/browse/AIRFLOW-3720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744793#comment-16744793 ] jack commented on AIRFLOW-3720: --- The bug was fixed in 1.10.2 [https://github.com/apache/airflow/pull/4371] > GoogleCloudStorageToS3Operator - incorrect folder compare > -- > > Key: AIRFLOW-3720 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3720 > Project: Apache Airflow > Issue Type: Bug > Components: boto3 >Affects Versions: 1.10.0 >Reporter: Chaim >Assignee: Chaim >Priority: Major > Fix For: 1.10.2 > > > the code that compares folders from gcp to s3 is incorrect. > the code is: > files = set(files) - set(existing_files) > but the list from gcp has a "/" to the name, for example: "myfolder/", while > in s3 it does not have "/" so the folder is "myfolder" > the result is that the code tries to recopy the folder name but fails since > it already exists -- This message was sent by Atlassian JIRA (v7.6.3#76005)
