curiousjazz77 commented on issue #8464: URL: https://github.com/apache/airflow/pull/8464#issuecomment-617739434
> I have two questions that I want to raise: > > 1. If we have this operator do we need the S3ListOperator ? > This raises a question that i asked a while ago but didn't have the item to check it. > Why the S3Hook has `list_prefixes` & `list_keys` functions? Is there a use case for `list_keys` that isn't covered by `list_prefixes`? > > The way I see it `list_prefixes` include subfolders while list_keys exclude them. So can't `list_prefixes` just have additional parameter to set it? > > The fact that we have both `list_prefixes` & `list_keys` resulted in having two sensors S3KeySensor & S3PrefixSensor. Now it will also result in having two operators. > > 1. Maybe worth assembling all the S3 operators (S3ListOperator , S3DeleteObjectsOperator , etc.. ) into a single file similar to [gcs.py](https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/gcs.py)? > > WDYT? Thank you for raising these questions. > I have two questions that I want to raise: > > 1. If we have this operator do we need the S3ListOperator ? > This raises a question that i asked a while ago but didn't have the item to check it. > Why the S3Hook has `list_prefixes` & `list_keys` functions? Is there a use case for `list_keys` that isn't covered by `list_prefixes`? > > The way I see it `list_prefixes` include subfolders while list_keys exclude them. So can't `list_prefixes` just have additional parameter to set it? > > The fact that we have both `list_prefixes` & `list_keys` resulted in having two sensors S3KeySensor & S3PrefixSensor. Now it will also result in having two operators. > > 1. Maybe worth assembling all the S3 operators (S3ListOperator , S3DeleteObjectsOperator , etc.. ) into a single file similar to [gcs.py](https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/operators/gcs.py)? > > WDYT? I like this train of thought. For your first point, I wholeheartedly agree. It would be optimal if there was one function that worked like the AWS cli. Reorganizing s3-related operators in one file as sounds good to me. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
