molcay commented on code in PR #72953:
URL: https://github.com/apache/airflow/pull/72953#discussion_r4132332549
##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -1071,8 +1076,10 @@ def _upload(upload_file: Path):
blob = bucket.blob(blob_name=upload_file_name,
chunk_size=self.chunk_size)
- blob.upload_from_filename(
- filename=str(upload_file),
+ self._run_with_attempts(
+ lambda:
blob.upload_from_filename(filename=str(upload_file)),
+ num_attempts=self.upload_num_attempts,
+ description=f"Upload of {upload_file_name} to
gs://{self.destination_bucket}",
)
return upload_file_name
Review Comment:
Same goes here, we can again use `tenacity.retry` decorator.
##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -996,7 +997,11 @@ def _download(blob_name: str):
)
destination_file.parent.mkdir(parents=True, exist_ok=True)
- blob.download_to_filename(filename=str(destination_file))
+ self._run_with_attempts(
+ lambda:
blob.download_to_filename(filename=str(destination_file)),
+ num_attempts=self.download_num_attempts,
+ description=f"Download of
gs://{self.source_bucket}/{blob_name}",
+ )
Review Comment:
Since we already have a private function here; maybe we can use the
`tenacity.retry` decorator to have a consistence across the provider.
```python
@tenacity.retry(
stop=tenacity.stop_after_attempt(5),
wait=tenacity.wait_exponential(multiplier=1, max=10),
retry=tenacity.retry_if_exception_type(ValueError),
)
def _download(blob_name: str):
...
```
> of course you may need to change the arguments of the decorator.
##########
providers/google/src/airflow/providers/google/cloud/operators/gcs.py:
##########
@@ -1101,6 +1108,28 @@ def _upload(upload_file: Path):
return files_uploaded
+ def _run_with_attempts(
Review Comment:
If we use the `tenacity.retry` decorator; we can totally avoid this function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
