[
https://issues.apache.org/jira/browse/AIRFLOW-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17310748#comment-17310748
]
ASF GitHub Bot commented on AIRFLOW-4085:
-----------------------------------------
NBardelot edited a comment on pull request #5358:
URL: https://github.com/apache/airflow/pull/5358#issuecomment-809495082
FYI this behaviour is inconsistent.
If anyone like me comes to this PR after looking at the glob() behaviour,
this cannot work for hooks that make use of a remote FS (like SFTPHook for
example). As the Python documentation states that glob() uses a mix of
os.scandir() and fnmatch.fnmatch() which make the code proposed here only
adapted to a local FS.
Thus, trying to use a path with a glob pattern and a hook to a remote FS
will have a random effect: either you're lucky and the glob() will not find the
equivalent path locally and just return that the path does not exist (the
sensor will never trigger); or in a worse case scenario you might trigger the
sensor for a file that exists locally but not on the remote FS as expected...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Support wildcards in FileSensor
> -------------------------------
>
> Key: AIRFLOW-4085
> URL: https://issues.apache.org/jira/browse/AIRFLOW-4085
> Project: Apache Airflow
> Issue Type: Improvement
> Components: contrib
> Reporter: Matthew Livesey
> Priority: Minor
> Fix For: 2.0.0
>
>
> When using FileSensor, it is necessary to specify the exact path of the file
> or folder to be sensed.
> There are use cases where the exact file name may not be known. For example
> if a file is delivered daily, but includes the full timestamp of file
> creation in the name, such as
> {code:java}
> my_data_2019-03-01T13_45_03.csv
> {code}
> I would like the FileSensor to sense with a file path such as
> {code:java}
> my_data_2019-03-01*.csv{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
