yuseok89 commented on code in PR #73269:
URL: https://github.com/apache/airflow/pull/73269#discussion_r4045114697
##########
providers/amazon/src/airflow/providers/amazon/aws/transfers/ftp_to_s3.py:
##########
@@ -135,17 +135,17 @@ def execute(self, context: Context):
path=self.ftp_path,
)
- if self.ftp_filenames == "*":
+ ftp_prefix: str = self.ftp_filenames
+ if ftp_prefix == "*":
files = list_dir
else:
- ftp_filename: str = self.ftp_filenames
- files = [f for f in list_dir if ftp_filename in f]
+ files = [f for f in list_dir if f.startswith(ftp_prefix)]
Review Comment:
Good points. I agree that the FTP operator should use the basename for
matching and replacement while retaining the original path for retrieval. I
also agree that changing the matching behavior immediately could surprise
existing users and should be noted in the Amazon provider changelog.
The `startswith` change was intended to resolve the mismatch between the
documented prefix semantics and the current substring matching.
Would it make sense to preserve substring matching for now, emit a
deprecation warning for non-prefix matches, clarify the behavior for nested S3
keys, and switch to prefix-only matching after the deprecation period?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
