[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830919#comment-16830919 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit 370987dfb7b54dce33c13f66535df0567f44fc6c in airflow's branch refs/heads/v1-10-stable from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=370987d ] [AIRFLOW-4397] add integrations docs manually for gcs sensors (#5204) (cherry picked from commit 7eb7abe18513008bf7dc00b61a9ae05062ea5bb2) > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830899#comment-16830899 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit 370987dfb7b54dce33c13f66535df0567f44fc6c in airflow's branch refs/heads/v1-10-test from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=370987d ] [AIRFLOW-4397] add integrations docs manually for gcs sensors (#5204) (cherry picked from commit 7eb7abe18513008bf7dc00b61a9ae05062ea5bb2) > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830896#comment-16830896 ] ASF GitHub Bot commented on AIRFLOW-4397: - potiuk commented on pull request #5204: [AIRFLOW-4397] add integrations docs manually for gcs sensors URL: https://github.com/apache/airflow/pull/5204 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829906#comment-16829906 ] ASF GitHub Bot commented on AIRFLOW-4397: - jaketf commented on pull request #5204: [AIRFLOW-4397] add integrations docs manually for gcs sensors URL: https://github.com/apache/airflow/pull/5204 This PR adds documentation for GCS sensors and replaces #5201. @potiuk for review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829722#comment-16829722 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit f328674914b08579d0406880e825831232fd5da9 in airflow's branch refs/heads/v1-10-stable from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f328674 ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. (cherry picked from commit 2bb79197cdba49b43a5a821674af1f11c0279d75) > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829721#comment-16829721 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit f328674914b08579d0406880e825831232fd5da9 in airflow's branch refs/heads/v1-10-stable from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f328674 ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. (cherry picked from commit 2bb79197cdba49b43a5a821674af1f11c0279d75) > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829400#comment-16829400 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit f328674914b08579d0406880e825831232fd5da9 in airflow's branch refs/heads/v1-10-test from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f328674 ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. (cherry picked from commit 2bb79197cdba49b43a5a821674af1f11c0279d75) > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829399#comment-16829399 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit f328674914b08579d0406880e825831232fd5da9 in airflow's branch refs/heads/v1-10-test from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f328674 ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. (cherry picked from commit 2bb79197cdba49b43a5a821674af1f11c0279d75) > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827921#comment-16827921 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit 97c31a1cb45d4ecee672b5257e055ca4f9effd95 in airflow's branch refs/heads/v1-10-test from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=97c31a1 ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. (cherry picked from commit 2bb79197cdba49b43a5a821674af1f11c0279d75) > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827920#comment-16827920 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit 97c31a1cb45d4ecee672b5257e055ca4f9effd95 in airflow's branch refs/heads/v1-10-test from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=97c31a1 ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. (cherry picked from commit 2bb79197cdba49b43a5a821674af1f11c0279d75) > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827916#comment-16827916 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit f5f07de46bdc493dae60a479c90a95c8361b8827 in airflow's branch refs/heads/v1-10-stable from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f5f07de ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. (cherry picked from commit 2bb79197cdba49b43a5a821674af1f11c0279d75) > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827915#comment-16827915 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit f5f07de46bdc493dae60a479c90a95c8361b8827 in airflow's branch refs/heads/v1-10-stable from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=f5f07de ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. (cherry picked from commit 2bb79197cdba49b43a5a821674af1f11c0279d75) > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 1.10.4, 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827718#comment-16827718 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit a31c02f06b3e6e7b443335464e8f195d8f9d1fe2 in airflow's branch refs/heads/v1-10-stable from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=a31c02f ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827717#comment-16827717 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit a31c02f06b3e6e7b443335464e8f195d8f9d1fe2 in airflow's branch refs/heads/v1-10-stable from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=a31c02f ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827715#comment-16827715 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit 5c94474dccf55044d85f9e3c65b0fa5e76b15b7d in airflow's branch refs/heads/v1-10-test from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5c94474 ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827714#comment-16827714 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit 5c94474dccf55044d85f9e3c65b0fa5e76b15b7d in airflow's branch refs/heads/v1-10-test from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=5c94474 ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > Fix For: 2.0.0 > > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827709#comment-16827709 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit 2bb79197cdba49b43a5a821674af1f11c0279d75 in airflow's branch refs/heads/master from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2bb7919 ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827707#comment-16827707 ] ASF GitHub Bot commented on AIRFLOW-4397: - potiuk commented on pull request #5166: [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor URL: https://github.com/apache/airflow/pull/5166 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-4397) Add GCSUploadSessionCompleteSensor
[ https://issues.apache.org/jira/browse/AIRFLOW-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16827708#comment-16827708 ] ASF subversion and git services commented on AIRFLOW-4397: -- Commit 2bb79197cdba49b43a5a821674af1f11c0279d75 in airflow's branch refs/heads/master from Jacob Ferriero [ https://gitbox.apache.org/repos/asf?p=airflow.git;h=2bb7919 ] [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor (#5166) * [AIRFLOW-4397] Add GCSUploadSessionCompleteSensor This commit add a GoogleCloudStorageUploadSessionCompleteSensor to address the use case of accepting files from a third party vendor who refuses to send a success indicator when providing data files into a bucket and waiting until an inactivity period has passed to indicate the end of an upload session. > Add GCSUploadSessionCompleteSensor > -- > > Key: AIRFLOW-4397 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4397 > Project: Apache Airflow > Issue Type: New Feature > Components: contrib >Reporter: Jacob Ferriero >Assignee: Jacob Ferriero >Priority: Minor > Labels: beginner, newbie > > I'd like to contribute a Sensor for Google Cloud Storage that can poke a > bucket until there has been sufficient time without a new file drop. Often > times there are cases where a third party vendor drops data to a bucket but > don't send a success flag when they are done. This sensor would allow you to > poke every n minutes to check if more files have been added since the last > poke, and if there had been `inactivity_period` minutes without a new file > drop, return `True`. This could allow SLA misses if data did not arrive by an > expected time, and have a configurable deadline past which the sensor would > fail. Optionally the user could specify a minimum number of files for the > sensor to succeed. This would be my first time contributing to an OSS > project, so please let me know if this is not the appropriate place to start. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
