anaynayak commented on a change in pull request #18807:
URL: https://github.com/apache/airflow/pull/18807#discussion_r737505894
##########
File path: airflow/providers/amazon/aws/sensors/s3_prefix.py
##########
@@ -67,18 +67,15 @@ def __init__(
super().__init__(**kwargs)
# Parse
self.bucket_name = bucket_name
- self.prefix = prefix
+ self.prefix = [prefix] if isinstance(prefix, str) else prefix
self.delimiter = delimiter
- self.full_url = "s3://" + bucket_name + '/' + prefix
self.aws_conn_id = aws_conn_id
self.verify = verify
self.hook: Optional[S3Hook] = None
- def poke(self, context):
+ def poke(self, context: Dict[str, Any]):
self.log.info('Poking for prefix : %s in bucket s3://%s', self.prefix,
self.bucket_name)
- return self.get_hook().check_for_prefix(
- prefix=self.prefix, delimiter=self.delimiter,
bucket_name=self.bucket_name
- )
+ return all(self._check_for_prefix(prefix) for prefix in self.prefix)
Review comment:
Agree that with the removal of `full_url` the backward compatibility has
gone away. For the name change, the worry was more for people passing keyword
arguments. Wasn't worried about the access to attributes. I can create a small
PR to rename to _prefixes if that isn't an issue. Let me know if that makes
sense.
I now understand what you were referring to with the eager return of `all` .
Personally I think it helps avoid a few extra calls to s3 and so should be
fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
