[jira] [Commented] (AIRFLOW-2073) FileSensor always return True
[ https://issues.apache.org/jira/browse/AIRFLOW-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357547#comment-16357547 ] ASF subversion and git services commented on AIRFLOW-2073: -- Commit 4751abf8acad766cb576ecfe3a333d68cc693b8c in incubator-airflow's branch refs/heads/master from [~ashb] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4751abf ] [AIRFLOW-2073] Make FileSensor fail when the file doesn't exist Previously it never returned False because os.walk never failed. It also wasn't clear the behaviour of this sensor should have when given a directory. I think the most useful behaviour would be to wait for any files to exist, rather than returning as soon as the directory itself exists. Closes #3017 from ashb/AIRFLOW-2073-file-sensor- never-failed > FileSensor always return True > - > > Key: AIRFLOW-2073 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2073 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.9.0 > Environment: Ubuntu 16.04 >Reporter: Pierre Payet >Priority: Trivial > Fix For: 1.10.0 > > > When using a FileSensor, the path is tested with os.walk. However, this > function never raise an error if the path does not exist and the poke will > always return True. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2073) FileSensor always return True
[ https://issues.apache.org/jira/browse/AIRFLOW-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357548#comment-16357548 ] ASF subversion and git services commented on AIRFLOW-2073: -- Commit 4751abf8acad766cb576ecfe3a333d68cc693b8c in incubator-airflow's branch refs/heads/master from [~ashb] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=4751abf ] [AIRFLOW-2073] Make FileSensor fail when the file doesn't exist Previously it never returned False because os.walk never failed. It also wasn't clear the behaviour of this sensor should have when given a directory. I think the most useful behaviour would be to wait for any files to exist, rather than returning as soon as the directory itself exists. Closes #3017 from ashb/AIRFLOW-2073-file-sensor- never-failed > FileSensor always return True > - > > Key: AIRFLOW-2073 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2073 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.9.0 > Environment: Ubuntu 16.04 >Reporter: Pierre Payet >Priority: Trivial > Fix For: 1.10.0 > > > When using a FileSensor, the path is tested with os.walk. However, this > function never raise an error if the path does not exist and the poke will > always return True. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2073) FileSensor always return True
[ https://issues.apache.org/jira/browse/AIRFLOW-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356957#comment-16356957 ] Ash Berlin-Taylor commented on AIRFLOW-2073: PR opened https://github.com/apache/incubator-airflow/pull/3017 > FileSensor always return True > - > > Key: AIRFLOW-2073 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2073 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.9.0 > Environment: Ubuntu 16.04 >Reporter: Pierre Payet >Priority: Trivial > > When using a FileSensor, the path is tested with os.walk. However, this > function never raise an error if the path does not exist and the poke will > always return True. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2073) FileSensor always return True
[
https://issues.apache.org/jira/browse/AIRFLOW-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356919#comment-16356919
]
Ash Berlin-Taylor commented on AIRFLOW-2073:
Yeah okay, this Sensor is currently totally broken, and the correct behaviour
is unclear.
{{os.walk}} basically never fails. {{[f for f in walk('/i/dont/exist')]}}
never fails and always evaluates to an empty list.
Using {{os.exists(full_path)}} might be one solution here, but it's not clear
what the behaviour of this sensor is meant to be when given a directory. If
it's given a directory is it meant to wait for any files to appear inside the
directory?
> FileSensor always return True
> -
>
> Key: AIRFLOW-2073
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2073
> Project: Apache Airflow
> Issue Type: Bug
> Components: contrib
>Affects Versions: 1.9.0
> Environment: Ubuntu 16.04
>Reporter: Pierre Payet
>Priority: Trivial
>
> When using a FileSensor, the path is tested with os.walk. However, this
> function never raise an error if the path does not exist and the poke will
> always return True.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2073) FileSensor always return True
[ https://issues.apache.org/jira/browse/AIRFLOW-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356890#comment-16356890 ] Yuliya Volkova commented on AIRFLOW-2073: - You right, I had been delete my comment, understand that I was wrong > FileSensor always return True > - > > Key: AIRFLOW-2073 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2073 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.9.0 > Environment: Ubuntu 16.04 >Reporter: Pierre Payet >Priority: Trivial > > When using a FileSensor, the path is tested with os.walk. However, this > function never raise an error if the path does not exist and the poke will > always return True. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2073) FileSensor always return True
[ https://issues.apache.org/jira/browse/AIRFLOW-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356881#comment-16356881 ] Ash Berlin-Taylor commented on AIRFLOW-2073: FileSensor in 1.9.0 lives here https://github.com/apache/incubator-airflow/blob/1.9.0/airflow/contrib/operators/fs_operator.py > FileSensor always return True > - > > Key: AIRFLOW-2073 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2073 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.9.0 > Environment: Ubuntu 16.04 >Reporter: Pierre Payet >Priority: Trivial > > When using a FileSensor, the path is tested with os.walk. However, this > function never raise an error if the path does not exist and the poke will > always return True. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2073) FileSensor always return True
[ https://issues.apache.org/jira/browse/AIRFLOW-2073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356845#comment-16356845 ] Yuliya Volkova commented on AIRFLOW-2073: - [~tikok], hello, can you clarify which version of airflow are you using? Seems like you talk about version from master branch, because inside stable release 1.9.0 there are no FileSensor [https://github.com/apache/incubator-airflow/tree/d760d63e1a141a43a4a43daee9abd54cf11c894b/airflow/contrib/sensors] , but it exist on master branch > FileSensor always return True > - > > Key: AIRFLOW-2073 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2073 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Affects Versions: 1.9.0 > Environment: Ubuntu 16.04 >Reporter: Pierre Payet >Priority: Trivial > > When using a FileSensor, the path is tested with os.walk. However, this > function never raise an error if the path does not exist and the poke will > always return True. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
