Hi Nikolaus
On Thu 05-Nov-2015 at 07:57:52PM -0800, Nikolaus Rath wrote:
>
> Where did you look for further information? I presume not in
> https://bitbucket.org/nikratio/s3ql/wiki/FAQ#!what-does-the-transport-endpoint-not-connected-error-mean?
Thanks, as you guessed I hadn't recently read that.
> You are experiencing regular file
> system crashes, and you have created a *script* to work around the
> issue?
I'm afraid so, as I said it happens a lot, limiting the threads to "1" hasn't
solved it either -- there was an a "Transport endpoint is not connected" error
again last night while rsyncing a 276M MySQL dump, mount.log entries follow.
> I would drop everything and use a different file system until the
> problem was fixed.
By different filesystem do you mean a different bucket or something
other than s3ql?
This looks this the relevant part of the mount.log:
2015-11-06 00:18:10.725 30000:MainThread s3ql.mount.main: Autodetected 65501
file descriptors available for cache entries
2015-11-06 00:18:13.665 30000:MainThread s3ql.mount.get_metadata: Using cached
metadata.
2015-11-06 00:18:13.667 30000:MainThread s3ql.mount.main: Setting cache size to
8726 MB
2015-11-06 00:18:13.695 30000:MainThread s3ql.mount.main: Mounting filesystem...
2015-11-06 00:18:13.720 30005:MainThread s3ql.daemonize.detach_process_context:
Daemonizing, new PID is 30006
2015-11-06 01:38:11.538 30006:Thread-3 root.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/lib/s3ql/s3ql/mount.py", line 64, in run_with_except_hook
run_old(*args, **kw)
File "/usr/lib/python3.4/threading.py", line 859, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/s3ql/s3ql/block_cache.py", line 404, in _upload_loop
self._do_upload(*tmp)
File "/usr/lib/s3ql/s3ql/block_cache.py", line 431, in _do_upload
% obj_id).get_obj_size()
File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
return method(*a, **kw)
File "/usr/lib/s3ql/s3ql/backends/common.py", line 337, in perform_write
return fn(fh)
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 551, in __exit__
self.close()
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 545, in close
self.fh.close()
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 710, in close
self.fh.close()
File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
return method(*a, **kw)
File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 910, in close
headers=self.headers, body=self.fh)
File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 510, in _do_request
self._parse_error_response(resp)
File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 544, in _parse_error_response
raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'),
resp.headers)
s3ql.backends.s3c.S3Error: ServiceUnavailable: Please reduce your request rate.
2015-11-06 01:38:15.738 30006:CommitThread s3ql.mount.exchook: Unhandled
top-level exception during shutdown (will not be re-raised)
2015-11-06 01:38:15.738 30006:CommitThread root.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/lib/s3ql/s3ql/database.py", line 143, in get_row
row = next(res)
StopIteration
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/s3ql/s3ql/block_cache.py", line 535, in upload
block_id = self.db.get_val('SELECT id FROM blocks WHERE hash=?', (hash_,))
File "/usr/lib/s3ql/s3ql/database.py", line 127, in get_val
return self.get_row(*a, **kw)[0]
File "/usr/lib/s3ql/s3ql/database.py", line 145, in get_row
raise NoSuchRowError()
s3ql.database.NoSuchRowError: Query produced 0 result rows
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/s3ql/s3ql/mount.py", line 64, in run_with_except_hook
run_old(*args, **kw)
File "/usr/lib/s3ql/s3ql/mount.py", line 754, in run
self.block_cache.upload(el)
File "/usr/lib/s3ql/s3ql/block_cache.py", line 554, in upload
self._queue_upload((el, obj_id))
File "/usr/lib/s3ql/s3ql/block_cache.py", line 597, in _queue_upload
raise NoWorkerThreads('no upload threads')
s3ql.block_cache.NoWorkerThreads: no upload threads
2015-11-06 01:38:21.388 30006:MainThread s3ql.block_cache.destroy: Unable to
flush cache, no upload threads left alive
2015-11-06 01:38:26.413 30006:MainThread s3ql.mount.unmount: Unmounting file
system...
2015-11-06 01:38:26.503 30006:MainThread root.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/lib/s3ql/s3ql/mount.py", line 225, in main
raise exc_inst.with_traceback(exc_tb)
File "/usr/lib/s3ql/s3ql/mount.py", line 64, in run_with_except_hook
run_old(*args, **kw)
File "/usr/lib/python3.4/threading.py", line 859, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/s3ql/s3ql/block_cache.py", line 404, in _upload_loop
self._do_upload(*tmp)
File "/usr/lib/s3ql/s3ql/block_cache.py", line 431, in _do_upload
% obj_id).get_obj_size()
File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
return method(*a, **kw)
File "/usr/lib/s3ql/s3ql/backends/common.py", line 337, in perform_write
return fn(fh)
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 551, in __exit__
self.close()
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 545, in close
self.fh.close()
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 710, in close
self.fh.close()
File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
return method(*a, **kw)
File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 910, in close
headers=self.headers, body=self.fh)
File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 510, in _do_request
self._parse_error_response(resp)
File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 544, in _parse_error_response
raise get_S3Error(tree.findtext('Code'), tree.findtext('Message'),
resp.headers)
s3ql.backends.s3c.S3Error: ServiceUnavailable: Please reduce your request rate.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/bin/mount.s3ql", line 9, in <module>
load_entry_point('s3ql==2.15', 'console_scripts', 'mount.s3ql')()
File "/usr/lib/s3ql/s3ql/mount.py", line 229, in main
unmount_clean = True
File "/usr/lib/python3.4/contextlib.py", line 336, in __exit__
raise exc_details[1]
File "/usr/lib/python3.4/contextlib.py", line 321, in __exit__
if cb(*exc_details):
File "/usr/lib/python3.4/contextlib.py", line 267, in _exit_wrapper
callback(*args, **kwds)
File "/usr/lib/s3ql/s3ql/block_cache.py", line 390, in destroy
os.rmdir(self.path)
OSError: [Errno 39] Directory not empty:
'/root/.s3ql/s3c:=2F=2Fs.qstack.advania.com:443=2Fcrin1=2F-cache'
Any suggestings about how to reduce these errors would be appreciated.
All the best
Chris
--
Webarchitects Co-operative
http://webarchitects.coop/
+44 114 276 9709
@webarchcoop
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
