Package: internetarchive
Version: 1.8.1-1+deb10u1
Severity: important
This is a bit of a follow up on #950289, I've retried uploading the folder and
ran out of file descriptors again, this time because the https connections
aren't closed.
The folder I'm testing with has ~10.000 files and fails midway.
Using lsof -nPi shows a large number of open connections in CLOSE_WAIT state:
[...]
ia 20254 user 469u IPv4 99476489 0t0 TCP
X.X.X.X:XXX->207.241.239.10:443 (CLOSE_WAIT)
ia 20254 user 470u IPv4 99476491 0t0 TCP
X.X.X.X:XXX->207.241.239.10:443 (CLOSE_WAIT)
ia 20254 user 471u IPv4 99476522 0t0 TCP
X.X.X.X:XXX->207.241.239.10:443 (CLOSE_WAIT)
ia 20254 user 472u IPv4 99476528 0t0 TCP
X.X.X.X:XXX->207.241.239.10:443 (CLOSE_WAIT)
ia 20254 user 473u IPv4 99476553 0t0 TCP
X.X.X.X:XXX:40406->207.241.239.10:443 (ESTABLISHED)
It eventually crashes with:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 321,
in ssl_wrap_socket
OSError: [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line
600, in urlopen
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line
343, in _make_request
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line
841, in _validate_conn
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line
344, in connect
File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 323,
in ssl_wrap_socket
urllib3.exceptions.SSLError: [Errno 24] Too many open files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449,
in send
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line
638, in urlopen
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line
398, in increment
urllib3.exceptions.MaxRetryError:
HTTPSConnectionPool(host='s3.us.archive.org', port=443): Max retries exceeded
with url: /redacted (Caused by SSLError(OSError(24, 'Too many open files')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/ia", line 11, in <module>
File "/usr/lib/python3/dist-packages/internetarchive/cli/ia.py", line
159, in main
File
"/usr/lib/python3/dist-packages/internetarchive/cli/ia_upload.py", line 225, in
main
File
"/usr/lib/python3/dist-packages/internetarchive/cli/ia_upload.py", line 86, in
_upload_files
File "/usr/lib/python3/dist-packages/internetarchive/item.py", line
828, in upload
File "/usr/lib/python3/dist-packages/internetarchive/item.py", line
710, in upload_file
File "/usr/lib/python3/dist-packages/internetarchive/session.py",
line 370, in send
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646,
in send
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514,
in send
requests.exceptions.SSLError:
HTTPSConnectionPool(host='s3.us.archive.org', port=443): Max retries exceeded
with url: /redacted (Caused by SSLError(OSError(24, 'Too many open files')))
I'm currently short on time so I'm having trouble fully disecting this bug and
test if sid is also affected. I've reported it upstream to ask for assistance:
https://github.com/jjjake/internetarchive/issues/336
