wegmatho opened a new issue, #24321: URL: https://github.com/apache/airflow/issues/24321
### Apache Airflow Provider(s) amazon ### Versions of Apache Airflow Providers 3.4.0 ### Apache Airflow version 2.3.2 (latest released) ### Operating System Debian GNU/Linux 10 ### Deployment Docker-Compose ### Deployment details _No response_ ### What happened For patterns like "*.zip" the S3KeySensor succeeds for all files, does not take full pattern into account i.e. the ".zip" part). Bug introduced in https://github.com/apache/airflow/pull/22737 ### What you think should happen instead Full pattern match as in version 3.3.0 (in S3KeySensor poke()): ``` ... if self.wildcard_match: return self.get_hook().check_for_wildcard_key(self.bucket_key, self.bucket_name) ... ``` alternatively the files obtained by `files = self.get_hook().get_file_metadata(prefix, bucket_name)` which only match the prefix should be further filtered. ### How to reproduce create a DAG with a key sensor task containing wildcard and suffix, e.g. the following task should succeed only if any ZIP-files are available in "my-bucket", but succeeds for all instead: `S3KeySensor(task_id="wait_for_file", bucket_name="my-bucket", bucket_key="*.zip", wildcard_match=True)` ### Anything else Not directly part of this issue but at the same time I would suggest to include additional file attributes in method _check_key, e.g. the actual key of the files. This way more filters, e.g. exclude specific keys, could be implemented by using the check_fn. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
