[
https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16827920#comment-16827920
]
ASF subversion and git services commented on AIRFLOW-4397:
----------------------------------------------------------
Commit 97c31a1cb45d4ecee672b5257e055ca4f9effd95 in airflow's branch
refs/heads/v1-10-test from Jacob Ferriero
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=97c31a1 ]
[AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166)
* [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor
This commit add a GoogleCloudStorageUploadSessionCompleteSensor
to address the use case of accepting files from a third party vendor
who refuses to send a success indicator when providing data files
into a bucket and waiting until an inactivity period has passed to
indicate the end of an upload session.
(cherry picked from commit 2bb79197cdba49b43a5a821674af1f11c0279d75)
> Add GCSUploadSessionCompleteSensor
> ----------------------------------
>
> Key: AIRFLOW-4397
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4397
> Project: Apache Airflow
> Issue Type: New Feature
> Components: contrib
> Reporter: Jacob Ferriero
> Assignee: Jacob Ferriero
> Priority: Minor
> Labels: beginner, newbie
> Fix For: 1.10.4, 2.0.0
>
>
> I'd like to contribute a Sensor for Google Cloud Storage that can poke a
> bucket until there has been sufficient time without a new file drop. Often
> times there are cases where a third party vendor drops data to a bucket but
> don't send a success flag when they are done. This sensor would allow you to
> poke every n minutes to check if more files have been added since the last
> poke, and if there had been `inactivity_period` minutes without a new file
> drop, return `True`. This could allow SLA misses if data did not arrive by an
> expected time, and have a configurable deadline past which the sensor would
> fail. Optionally the user could specify a minimum number of files for the
> sensor to succeed. This would be my first time contributing to an OSS
> project, so please let me know if this is not the appropriate place to start.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
