Nikolaus, thanks for the prompt response On Tuesday, February 17, 2015 at 12:19:26 AM UTC-2, Nikolaus Rath wrote:
> Hello, > > > > I use s3ql for quite some time as my apache webroot, which serves > > about 300gb of files and some php applications. I had few problems > ^^^ > > you already said that. SCNR. > ;) > with high I/O scenarios in the past (using the 1.x series), where the > > mount daemon would die under high load, but managed to solve them all > > with fsck.s3ql after rebooting. > > Actually that doesn't sound like a solution at all. mount.s3ql should > never crash, now matter how high the load is. Can you still reproduce > that? If so, it'd be great if you could post the backtrace. > This time it happened while I was performing 2 full backups with lots of small files, from my mounted s3qlfs to another s3 bucket via duplicity. I've experienced this when apache was under high load. When mount.s3ql stops, all other processes start to wait for io, increasing the load constantly, I can try to force this behaviour on another bucket after I restore this one. [...] > Btw, which version is it now? Your subject says 2.12. Filesystem was created on 2.11, I tried upgrading to 2.13 but couldn't due to db revision upgrade, so I had to compile 2.12 to try fsck/upgrade my volume - /usr/local/bin/fsck.s3ql --debug s3://my-bucket/home > > Now fsck.s3ql is hanging at ..processed 99500 objects so far.. > > fsck.log (with --debug) shows a lot of HEAD requests, It seems to be > > running a verify for every block (about 1400000). > > That should not happen. Please post more context for the logfile. What's > the last message before the HEAD requests are starting? > As this should not happen, I stopped fsck and ran it again to check the logs, the output doesn't show any errors https://gist.github.com/guigouz/7a6a624d97d12918b3f6 The Exception I get after hitting ctrl-c 2015-02-17 04:09:29.478 19239:MainThread root.excepthook: Uncaught top-level exception: Traceback (most recent call last): File "/usr/local/bin/fsck.s3ql", line 9, in <module> load_entry_point('s3ql==2.12', 'console_scripts', 'fsck.s3ql')() File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/fsck.py", line 1255, in main fsck.check() File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/fsck.py", line 89, in check self.check_objects_id() File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/fsck.py", line 925, in check_objects_id if ('s3ql_data_%d' % obj_id) in self.backend: File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/backends/common.py", line 199, in __contains__ return self.contains(key) File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/backends/comprenc.py", line 265, in contains return self.backend.contains(key) File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/backends/common.py", line 354, in contains self.lookup(key) File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/backends/common.py", line 46, in wrapped return method(*a, **kw) File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/backends/s3c.py", line 253, in lookup resp = self._do_request('HEAD', '/%s%s' % (self.prefix, key)) File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/backends/s3c.py", line 407, in _do_request query_string=query_string, body=body) File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/backends/s3c.py", line 666, in _send_request return read_response() File "/usr/local/lib/python3.4/dist-packages/s3ql-2.12-py3.4-linux-x86_64.egg/s3ql/backends/s3c.py", line 628, in read_response resp = self.conn.read_response() File "/usr/lib/python3/dist-packages/dugong/__init__.py", line 708, in read_response return eval_coroutine(self.co_read_response(), self.timeout) File "/usr/lib/python3/dist-packages/dugong/__init__.py", line 1396, in eval_coroutine if not next(crt).poll(timeout=timeout): File "/usr/lib/python3/dist-packages/dugong/__init__.py", line 115, in poll return bool(poll.poll(timeout*1000)) # convert to ms KeyboardInterrupt Is there any other debugging info I may provide ? > I also tried > > downloading a metadata backup, but it hangs at the same point, here's > > an excerpt of the ongoing log: > > What do you mean with "at the same point"? Downloading metadata should > not issue any requests for data objects at all. > fsck hangs at ..processed 99500 objects so far.., no matter which db backup I use Thanks, Guilherme -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
