[jira] [Commented] (AIRFLOW-6546) add GDriveToGcsOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015377#comment-17015377 ] Kamil Bregula commented on AIRFLOW-6546: It seems that the cycles have already been blocked in the API. Hurra Do you want to work on this ticket? I have prepared a piece of code that can be helpful for you. [https://pastebin.com/hDAiQPUZ] If you want to write a copy operator, I'd write two methods in hook - list_files_recursive. It should also support filtering using wildcard. Another method can download a file to the specified directory. Operators can use this methods and NamedTemporaryFile to transfer files to GCS. > add GDriveToGcsOperator > --- > > Key: AIRFLOW-6546 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6546 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.6 >Reporter: lovk korm >Priority: Major > > There is GcsToGDriveOperator but there isn't the equivalent in the other > direction > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6546) add GDriveToGcsOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015256#comment-17015256 ] Kamil Bregula commented on AIRFLOW-6546: I think that wildcard support is very helpful. I would like all operators that support the GCP/Google platform to have a similar set of features. This will facilitate their easier use. Maybe we should skip directories if we detect a cycle? > add GDriveToGcsOperator > --- > > Key: AIRFLOW-6546 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6546 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.6 >Reporter: lovk korm >Priority: Major > > There is GcsToGDriveOperator but there isn't the equivalent in the other > direction > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6546) add GDriveToGcsOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015206#comment-17015206 ] lovk korm commented on AIRFLOW-6546: [~kamil.bregula] I see that now. I think that your concern is for a wider case. If I understand correctly support moving a single file should be ok? when working with Google Drive usually you don't need to move hundreds of fails in scheduled jobs.. the need is more singular (copy one file every day) I found this [https://medium.com/@philipplies/transferring-data-from-google-drive-to-google-cloud-storage-using-google-colab-96e088a8c041 ]so maybe i'll just do what it says but I do think that the operator is needed even if it's functionality is limited. > add GDriveToGcsOperator > --- > > Key: AIRFLOW-6546 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6546 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.6 >Reporter: lovk korm >Priority: Major > > There is GcsToGDriveOperator but there isn't the equivalent in the other > direction > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-6546) add GDriveToGcsOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17014240#comment-17014240 ] Kamil Bregula commented on AIRFLOW-6546: Do you want to work on it or is it a feature request? We've already discussed these operators before. https://issues.apache.org/jira/browse/AIRFLOW-4758 > add GDriveToGcsOperator > --- > > Key: AIRFLOW-6546 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6546 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.6 >Reporter: lovk korm >Priority: Major > > There is GcsToGDriveOperator but there isn't the equivalent in the other > direction > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
