alexkruc commented on code in PR #26886:
URL: https://github.com/apache/airflow/pull/26886#discussion_r997025873
##########
airflow/providers/amazon/aws/hooks/s3.py:
##########
@@ -902,14 +924,30 @@ def download_file(self, key: str, bucket_name: str | None
= None, local_path: st
else:
raise e
- with NamedTemporaryFile(dir=local_path, prefix='airflow_tmp_',
delete=False) as local_tmp_file:
+ if preserve_file_name:
+ local_dir = local_path if local_path else gettempdir()
+ subdir = f"airflow_tmp_dir_{uuid4().hex[0:8]}" if
use_autogenerated_subdir else ""
+ filename_in_s3 = s3_obj.key.rsplit('/', 1)[-1]
+ file_path = Path(local_dir, subdir, filename_in_s3)
+
+ if file_path.is_file():
+ self.log.error("file '%s' already exists. Failing the task and
not overwriting it", file_path)
+ raise FileExistsError
+
+ file_path.parent.mkdir(exist_ok=True, parents=True)
+
+ file = open(file_path, 'wb')
+ else:
+ file = NamedTemporaryFile(dir=local_path, prefix='airflow_tmp_',
delete=False) # type: ignore
Review Comment:
The pre-commit checks are failing on this with this error:
```shell
Run mypy for
providers.................................................................Failed
- hook id: run-mypy
- exit code: 1
airflow/providers/amazon/aws/hooks/s3.py:939: error: Incompatible types in
assignment (expression has type "BufferedWriter", variable has type "TextIO")
[assignment]
file: TextIO = open(file_path, 'wb')
^
airflow/providers/amazon/aws/hooks/s3.py:941: error: Incompatible types in
assignment (expression has type "_TemporaryFileWrapper[bytes]", variable has
type "TextIO") [assignment]
file = NamedTemporaryFile(dir=local_path, prefix='airflow_...
```
I've added the `# type: ignore` section because `NamedTemporaryFile` is
returning a file-like interface, but not an actual file..
([link](https://github.com/python/cpython/blob/main/Lib/tempfile.py#L539))
Do you think it's ok to ignore the types check of `mypy` in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
