SadmiB commented on issue #28879:
URL: https://github.com/apache/airflow/issues/28879#issuecomment-1380097679
Yes it is:
```python3
class DatasetDownloader:
"""Datasets downloader from s3 ml bucket to local directory"""
def __init__(self, config):
self.config = config
# set up s3 instance
self.s3 = self._get_s3_instance()
self.logger = self._configure_logging()
#self.cvat = self._get_cvat_instance(cvat_auth, cvat_host)
def download_folder(self, data_dir):
# download the datasets from s3 bucket
self.s3.s3_download_folder(self.config.S3_BUCKET, data_dir)
def download_raw_weights(self):
# download raw weights from s3
self.s3.s3_download_file(self.config.S3_BUCKET,
self.config.RAW_WEIGTHS_PATH,
self.config.RAW_WEIGTHS_PATH)
@staticmethod
def _get_s3_instance():
s3 = S3()
return s3
@staticmethod
def _configure_logging():
# To be improved
logger = logging.getLogger(__name__)
return logger
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
