uranusjr commented on code in PR #26886:
URL: https://github.com/apache/airflow/pull/26886#discussion_r996862613
##########
airflow/providers/amazon/aws/hooks/s3.py:
##########
@@ -902,14 +924,30 @@ def download_file(self, key: str, bucket_name: str | None
= None, local_path: st
else:
raise e
- with NamedTemporaryFile(dir=local_path, prefix='airflow_tmp_',
delete=False) as local_tmp_file:
+ if preserve_file_name:
+ local_dir = local_path if local_path else gettempdir()
+ subdir = f"airflow_tmp_dir_{uuid4().hex[0:8]}" if
use_autogenerated_subdir else ""
+ filename_in_s3 = s3_obj.key.rsplit('/', 1)[-1]
+ file_path = Path(local_dir, subdir, filename_in_s3)
+
+ if file_path.is_file():
+ self.log.error("file '%s' already exists. Failing the task and
not overwriting it", file_path)
+ raise FileExistsError
+
+ file_path.parent.mkdir(exist_ok=True, parents=True)
+
+ file = open(file_path, 'wb')
+ else:
+ file = NamedTemporaryFile(dir=local_path, prefix='airflow_tmp_',
delete=False) # type: ignore
Review Comment:
```suggestion
file: TextIO = open(file_path, 'wb')
else:
file = NamedTemporaryFile(dir=local_path, prefix='airflow_tmp_',
delete=False)
```
This should work (with `from typing import TextIO`)
##########
airflow/providers/amazon/aws/hooks/s3.py:
##########
@@ -902,14 +924,30 @@ def download_file(self, key: str, bucket_name: str | None
= None, local_path: st
else:
raise e
- with NamedTemporaryFile(dir=local_path, prefix='airflow_tmp_',
delete=False) as local_tmp_file:
+ if preserve_file_name:
+ local_dir = local_path if local_path else gettempdir()
+ subdir = f"airflow_tmp_dir_{uuid4().hex[0:8]}" if
use_autogenerated_subdir else ""
+ filename_in_s3 = s3_obj.key.rsplit('/', 1)[-1]
+ file_path = Path(local_dir, subdir, filename_in_s3)
+
+ if file_path.is_file():
+ self.log.error("file '%s' already exists. Failing the task and
not overwriting it", file_path)
+ raise FileExistsError
+
+ file_path.parent.mkdir(exist_ok=True, parents=True)
+
+ file = open(file_path, 'wb')
+ else:
+ file = NamedTemporaryFile(dir=local_path, prefix='airflow_tmp_',
delete=False) # type: ignore
Review Comment:
```suggestion
file: BinaryIO = open(file_path, 'wb')
else:
file = NamedTemporaryFile(dir=local_path, prefix='airflow_tmp_',
delete=False)
```
This should work (with `from typing import BinaryIO`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
