o-nikolas commented on code in PR #31640:
URL: https://github.com/apache/airflow/pull/31640#discussion_r1212142795
##########
airflow/providers/amazon/aws/hooks/s3.py:
##########
@@ -425,7 +427,10 @@ def _is_in_period(input_date: datetime) -> bool:
:return: a list of matched keys
"""
- prefix = prefix or ""
+ _prefix = prefix or ""
+ wildcard_prefix = _prefix
+ if apply_wildcard and "*" in _prefix:
+ wildcard_prefix = _prefix.split("*", 1)[0]
Review Comment:
When I run this locally with "*a" as an example, the `[0]'th` item is just
an empty string, don't you want the `[1]'st` index?:
```python
In [1]: _prefix = "*a"
In [2]: _prefix.split("*", 1)[0]
Out[2]: ''
In [3]: _prefix.split("*", 1)
Out[3]: ['', 'a']
```
##########
airflow/providers/amazon/aws/operators/s3.py:
##########
@@ -683,7 +683,12 @@ def execute(self, context: Context):
self.delimiter,
)
- return hook.list_keys(bucket_name=self.bucket, prefix=self.prefix,
delimiter=self.delimiter)
+ return hook.list_keys(
+ bucket_name=self.bucket,
+ prefix=self.prefix,
+ delimiter=self.delimiter,
+ apply_wildcard=True,
Review Comment:
This will change the behaviour of this operator with no way to disable it.
Can you add a new field to the operator itself for applying wildcards and set
the default to False.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
