[
https://issues.apache.org/jira/browse/AIRFLOW-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manish Zope reassigned AIRFLOW-5612:
------------------------------------
Assignee: Manish Zope
> Add ability to actually do things with created and modified date in
> GoogleCloudStorageHook
> ------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-5612
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5612
> Project: Apache Airflow
> Issue Type: Improvement
> Components: gcp
> Affects Versions: 1.10.5
> Reporter: Joel Croteau
> Assignee: Manish Zope
> Priority: Major
>
> {{GoogleCloudStorageHook}} seems to support only a very small subset of the
> actual GCS API. In particular, the only thing it allows you to do with the
> date of an object is check if the metadata was updated after a specified time
> using {{is_updated_after}}. First of all, this only looks at the metadata
> update date, which is probably not what is wanted for most purposes, as
> {{timeCreated}} is generally what conveys useful information. Second of all,
> it seems rather arbitrary to only allow me to compare if the updated time is
> greater than some other time, and not just give me the time and let me make
> my own inferences. In particular, for a scheduled workflow with a potential
> backfill, I would like to check for a creation date with both a minimum and
> maximum value, which this doesn't allow.
>
> Also, tangentially, if you want to get multiple pieces of information on an
> object, using {{GoogleCloudStorageHook}} will necessitate a separate call to
> {{objects().get()}} for every piece of information, even though everything is
> returned by the one call. Would it not make more sense to be able to return
> an object structure with all of the needed information in it?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
