[jira] [Updated] (AIRFLOW-3236) Create AzureDataLakeStorageListOperator

2018-10-24 Thread Brandon Kvarda (JIRA) 


 [ 
https://issues.apache.org/jira/browse/AIRFLOW-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Brandon Kvarda updated AIRFLOW-3236:

Priority: Minor  (was: Major)

> Create AzureDataLakeStorageListOperator
> ---
>
> Key: AIRFLOW-3236
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3236
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Brandon Kvarda
>Assignee: Brandon Kvarda
>Priority: Minor
>
> Creates an Operator that is similar to GoogleCloudStorageListOperator and 
> S3ListOperator that returns a list of files at some specified path. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3236) Create AzureDataLakeStorageListOperator

2018-10-20 Thread Brandon Kvarda (JIRA) 
Brandon Kvarda created AIRFLOW-3236:
---

 Summary: Create AzureDataLakeStorageListOperator
 Key: AIRFLOW-3236
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3236
 Project: Apache Airflow
  Issue Type: Improvement
  Components: operators
Reporter: Brandon Kvarda
Assignee: Brandon Kvarda


Creates an Operator that is similar to GoogleCloudStorageListOperator and 
S3ListOperator that returns a list of files at some specified path. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3235) Add list function in AzureDataLakeHook

2018-10-19 Thread Brandon Kvarda (JIRA) 
Brandon Kvarda created AIRFLOW-3235:
---

 Summary: Add list function in AzureDataLakeHook
 Key: AIRFLOW-3235
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3235
 Project: Apache Airflow
  Issue Type: Improvement
  Components: hooks
Reporter: Brandon Kvarda
Assignee: Brandon Kvarda


AzureDataLakeHook could use a list function similar to those of both the GCS 
and S3 hooks. This could be the basis for an ADLS list operator (again, which 
both GCS and S3 have). 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3213) Create ADLS to GCS operator

2018-10-15 Thread Brandon Kvarda (JIRA) 
Brandon Kvarda created AIRFLOW-3213:
---

 Summary: Create ADLS to GCS operator 
 Key: AIRFLOW-3213
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3213
 Project: Apache Airflow
  Issue Type: Improvement
  Components: gcp, operators
Reporter: Brandon Kvarda
Assignee: Brandon Kvarda


Create ADLS to GCS operator that supports copying of files from ADLS to GCS



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AIRFLOW-3155) Add ability to filter by a last modified time in GoogleCloudStorageToGoogleCloudStorageOperator

2018-10-03 Thread Brandon Kvarda (JIRA) 
Brandon Kvarda created AIRFLOW-3155:
---

 Summary: Add ability to filter by a last modified time in 
GoogleCloudStorageToGoogleCloudStorageOperator
 Key: AIRFLOW-3155
 URL: https://issues.apache.org/jira/browse/AIRFLOW-3155
 Project: Apache Airflow
  Issue Type: Improvement
  Components: gcp
Affects Versions: 2.0.0
Reporter: Brandon Kvarda
Assignee: Brandon Kvarda


Currently the GoogleCloudStorageToGoogleCloudStorageOperator doesn't support 
filtering objects based on a last modified time/date. This would add the 
ability to further filter source object(s) to copy/move based on a last 
modified time threshold (for example, if the objects were updated after the 
last run at 10:00 yesterday, then copy/move them; otherwise, do not.) 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)