[
https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16824799#comment-16824799
]
ASF GitHub Bot commented on AIRFLOW-4397:
-----------------------------------------
jaketf commented on pull request #5166: [AIRFLOW-4397] Add
GCSUploadSessionCompleteSensor
URL: https://github.com/apache/airflow/pull/5166
This commit add a GoogleCloudStorageUploadSessionCompleteSensor
to address the use case of accepting files from a third party vendor
who refuses to send a success indicator when providing data files
into a bucket and waiting until an inactivity period has passed to
indicate the end of an upload session.
Make sure you have checked _all_ steps below.
### Jira
- [x] My PR addresses the following
[Airflow-4937](https://issues.apache.org/jira/browse/AIRFLOW-4397) issues and
references them in the PR title.
### Description
- [x] This PR add a GCS sensor to poke a bucket until some inactivity
period has passed with no new files added to the bucket.
### Tests
- [x] My PR adds the following unit tests:
`tests/contrib/sensors/test_gcs_upload_session_sensor.py`
### Commits
- [x] My commits all reference Jira issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
1. Subject is limited to 50 characters (not including Jira issue reference)
1. Subject does not end with a period
1. Subject uses the imperative mood ("add", not "adding")
1. Body wraps at 72 characters
1. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds documentation that describes
how to use it.
- All the public functions and the classes in the PR contain docstrings
that explain what it does
- If you implement backwards incompatible changes, please leave a note in
the [Updating.md](https://github.com/apache/airflow/blob/master/UPDATING.md) so
we can assign it to a appropriate release
### Code Quality
- [x] Passes `flake8`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Add GCSUploadSessionCompleteSensor
> ----------------------------------
>
> Key: AIRFLOW-4397
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4397
> Project: Apache Airflow
> Issue Type: New Feature
> Components: contrib
> Reporter: Jacob Ferriero
> Assignee: Jacob Ferriero
> Priority: Minor
> Labels: beginner, newbie
>
> I'd like to contribute a Sensor for Google Cloud Storage that can poke a
> bucket until there has been sufficient time without a new file drop. Often
> times there are cases where a third party vendor drops data to a bucket but
> don't send a success flag when they are done. This sensor would allow you to
> poke every n minutes to check if more files have been added since the last
> poke, and if there had been `inactivity_period` minutes without a new file
> drop, return `True`. This could allow SLA misses if data did not arrive by an
> expected time, and have a configurable deadline past which the sensor would
> fail. Optionally the user could specify a minimum number of files for the
> sensor to succeed. This would be my first time contributing to an OSS
> project, so please let me know if this is not the appropriate place to start.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
