potiuk commented on code in PR #799:
URL: https://github.com/apache/airflow-site/pull/799#discussion_r1226347887
##########
post-docs/add-back-references.py:
##########
@@ -40,11 +41,14 @@ class GenerationType(enum.Enum):
def download_file(url):
- filedata = urlopen(url)
- datatowrite = filedata.read()
+ try:
+ filedata = urlopen(url)
+ datatowrite = filedata.read()
- with open('redirects.txt', 'wb') as f:
- f.write(datatowrite)
+ with open('redirects.txt', 'wb') as f:
+ f.write(datatowrite)
+ except URLError as e:
+ log.warning(e)
Review Comment:
(I think what is going to happen with this version is that will keep on
using the redirects.txt retrieved from previous successfully downloaded
redirect.txt ? I guess it would be great to do two things:
1) make sure that the redirects.txt is named differently depending on a
provider (just in case of programming mistakes)
2) store them somewhere where they will be deleted (tmp dir)
3) if download fails, skip the whole `generate`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
