utkarsharma2 commented on code in PR #37117:
URL: https://github.com/apache/airflow/pull/37117#discussion_r1475839636
##########
airflow/providers/sftp/sensors/sftp.py:
##########
@@ -119,3 +124,50 @@ def poke(self, context: Context) -> PokeReturnValue | bool:
xcom_value={"files_found": files_found,
"decorator_return_value": callable_return},
)
return True
+
+ def execute(self, context: Context) -> Any:
+ # Unlike other async sensors, we do not follow the pattern of calling
the synchronous self.poke()
+ # method before deferring here. This is due to the current limitations
we have in the synchronous
+ # SFTPHook methods. The limitations are discovered while being worked
upon the ticket
+ # https://github.com/astronomer/astronomer-providers/issues/1021. They
are as follows:
+ #
+ # For file_pattern sensing, the hook implements list_directory()
method which returns a list of
+ # filenames only without the attributes like modified time which is
required for the file_pattern
+ # sensing when newer_than is supplied. This leads to intermittent
failures potentially due to
+ # throttling by the SFTP server as the hook makes multiple calls to
the server to get the
+ # attributes for each of the files in the directory.This limitation is
resolved here by instead
+ # calling the read_directory() method which returns a list of files
along with their attributes
+ # in a single call. We can add back the call to self.poke() before
deferring once the above
+ # limitations are resolved in the
Review Comment:
```suggestion
# limitations are resolved in the sync sensor.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
