uranusjr commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r726573093
##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -67,18 +67,15 @@ def __init__(
super().__init__(**kwargs)
# Parse
self.bucket_name = bucket_name
- self.prefix = prefix
+ self.prefix = [prefix] if isinstance(prefix, str) else prefix
self.delimiter = delimiter
- self.full_url = "s3://" + bucket_name + '/' + prefix
self.aws_conn_id = aws_conn_id
self.verify = verify
self.hook: Optional[S3Hook] = None
- def poke(self, context):
+ def poke(self, context: Dict[str, Any]):
self.log.info('Poking for prefix : %s in bucket s3://%s', self.prefix,
self.bucket_name)
- return self.get_hook().check_for_prefix(
- prefix=self.prefix, delimiter=self.delimiter,
bucket_name=self.bucket_name
- )
+ return all(self._check_for_prefix(prefix) for prefix in self.prefix)
Review comment:
I just realised this would cause some `_check_for_prefix` to not be
called if any of the prefixes fail the check, because `all` returns eagerly.
Not sure if that would be an issue in practice.
Also we should've changed the attribute from `prefix` to `prefixes` 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
