tring to upload 100K files i'm getting corruption data in s3ql, in some
point s3ql crash because hubic close connections and i lose all data from
last meta backup :/
this is part of the log
2015-12-10 16:48:54.503 28772:Thread-14 s3ql.backends.common.wrapped: Had
to retry 535 times over the last 60 seconds, server or network problem?
2015-12-10 16:48:54.539 28772:Thread-22 s3ql.backends.common.wrapped: Had
to retry 536 times over the last 60 seconds, server or network problem?
2015-12-10 16:48:54.545 28772:Thread-19 s3ql.backends.common.wrapped: Had
to retry 537 times over the last 60 seconds, server or network problem?
2015-12-10 16:48:54.591 28772:Thread-8 s3ql.backends.common.wrapped: Had to
retry 538 times over the last 60 seconds, server or network problem?
2015-12-10 16:49:31.752 28772:MainThread s3ql.mount.main: FUSE main loop
terminated.
2015-12-10 16:49:31.762 28772:MainThread s3ql.mount.unmount: Unmounting
file system...
2015-12-10 16:49:32.153 28772:MainThread
s3ql.metadata.dump_and_upload_metadata: Dumping metadata...
2015-12-10 16:49:32.153 28772:MainThread s3ql.metadata.dump_metadata:
..objects..
2015-12-10 16:49:32.165 28772:MainThread s3ql.metadata.dump_metadata:
..blocks..
2015-12-10 16:49:32.186 28772:MainThread s3ql.metadata.dump_metadata:
..inodes..
2015-12-10 16:49:32.223 28772:MainThread s3ql.metadata.dump_metadata:
..inode_blocks..
2015-12-10 16:49:32.241 28772:MainThread s3ql.metadata.dump_metadata:
..symlink_targets..
2015-12-10 16:49:32.242 28772:MainThread s3ql.metadata.dump_metadata:
..names..
2015-12-10 16:49:32.257 28772:MainThread s3ql.metadata.dump_metadata:
..contents..
2015-12-10 16:49:32.275 28772:MainThread s3ql.metadata.dump_metadata:
..ext_attributes..
2015-12-10 16:49:32.276 28772:MainThread s3ql.metadata.upload_metadata:
Compressing and uploading metadata...
2015-12-10 16:49:33.234 28772:MainThread s3ql.metadata.upload_metadata:
Wrote 987 KiB of compressed metadata.
2015-12-10 16:49:33.234 28772:MainThread s3ql.metadata.upload_metadata:
Cycling metadata backups...
2015-12-10 16:49:33.234 28772:MainThread s3ql.metadata.cycle_metadata:
Backing up old metadata...
2015-12-10 16:49:39.172 28772:MainThread s3ql.mount.main: Cleaning up local
metadata...
2015-12-10 16:49:39.298 28772:MainThread s3ql.mount.main: All done.
2015-12-10 16:49:59.295 35486:MainThread s3ql.mount.get_metadata: Using
cached metadata.
2015-12-10 16:49:59.296 35486:MainThread s3ql.mount.main: Mounting
filesystem...
2015-12-10 16:49:59.301 35490:MainThread
s3ql.daemonize.detach_process_context: Daemonizing, new PID is 35491
2015-12-10 17:17:20.900 35491:Thread-3 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.911 35491:Thread-14 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.914 35491:Thread-5 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.920 35491:Thread-19 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.940 35491:Thread-17 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.944 35491:Thread-7 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.952 35491:Thread-12 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.955 35491:Thread-22 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.957 35491:Thread-6 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.962 35491:Thread-11 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.973 35491:Thread-4 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:20.979 35491:Thread-18 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:21.004 35491:Thread-21 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:21.011 35491:Thread-10 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:21.014 35491:Thread-8 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:21.037 35491:Thread-9 s3ql.backends.swift._do_request:
OpenStack auth token seems to have expired, requesting new one.
2015-12-10 17:17:21.920 35491:Thread-14 s3ql.backends.common.wrapped:
Encountered HTTPError (509 unused), retrying ObjectW.close (attempt 3)...
2015-12-10 17:17:22.344 35491:Thread-3 s3ql.backends.common.wrapped:
Encountered HTTPError (509 unused), retrying ObjectW.close (attempt 4)...
2015-12-10 17:17:22.347 35491:Thread-17 s3ql.backends.common.wrapped:
Encountered HTTPError (509 unused), retrying ObjectW.close (attempt 4)...
2015-12-10 17:17:22.352 35491:Thread-14 s3ql.backends.common.wrapped:
Encountered HTTPError (509 unused), retrying ObjectW.close (attempt 4)...
2015-12-10 17:17:22.356 35491:Thread-7 s3ql.backends.common.wrapped:
Encountered HTTPError (509 unused), retrying ObjectW.close (attempt 4)...
2015-12-10 17:17:22.499 35491:Thread-18 root.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/lib/s3ql/s3ql/mount.py", line 64, in run_with_except_hook
run_old(*args, **kw)
File "/usr/lib/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/s3ql/s3ql/block_cache.py", line 404, in _upload_loop
self._do_upload(*tmp)
File "/usr/lib/s3ql/s3ql/block_cache.py", line 431, in _do_upload
% obj_id).get_obj_size()
File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
return method(*a, **kw)
File "/usr/lib/s3ql/s3ql/backends/common.py", line 337, in perform_write
return fn(fh)
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 716, in __exit__
self.close()
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 710, in close
self.fh.close()
File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
return method(*a, **kw)
File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 910, in close
headers=self.headers, body=self.fh)
File "/usr/lib/s3ql/s3ql/backends/swift.py", line 210, in _do_request
self.conn = self._get_conn()
File "/usr/lib/s3ql/s3ql/backends/swift.py", line 167, in _get_conn
raise AuthorizationError(resp.reason)
s3ql.backends.common.AuthorizationError: Access denied. Server said:
Authorization Required
2015-12-10 17:17:22.536 35491:Thread-22 s3ql.backends.common.wrapped:
Encountered HTTPError (509 unused), retrying ObjectW.close (attempt 5)...
2015-12-10 17:17:22.540 35491:Thread-21 s3ql.backends.common.wrapped:
Encountered HTTPError (509 unused), retrying ObjectW.close (attempt 5)...
2015-12-10 17:17:22.552 35491:Thread-17 s3ql.backends.common.wrapped:
Encountered HTTPError (509 unused), retrying ObjectW.close (attempt 5)...
2015-12-10 17:17:22.554 35491:Thread-3 s3ql.backends.common.wrapped:
Encountered HTTPError (509 unused), retrying ObjectW.close (attempt 5)...
2015-12-10 17:17:22.560 35491:Thread-14 s3ql.backends.common.wrapped:
Encountered HTTPError (509 unused), retrying ObjectW.close (attempt 5)...
2015-12-10 17:17:22.664 35491:Thread-19 s3ql.mount.exchook: Unhandled
top-level exception during shutdown (will not be re-raised)
Traceback (most recent call last):
File "/usr/lib/s3ql/s3ql/mount.py", line 64, in run_with_except_hook
run_old(*args, **kw)
File "/usr/lib/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/s3ql/s3ql/block_cache.py", line 404, in _upload_loop
self._do_upload(*tmp)
File "/usr/lib/s3ql/s3ql/block_cache.py", line 431, in _do_upload
% obj_id).get_obj_size()
File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
return method(*a, **kw)
File "/usr/lib/s3ql/s3ql/backends/common.py", line 337, in perform_write
return fn(fh)
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 716, in __exit__
self.close()
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 710, in close
self.fh.close()
File "/usr/lib/s3ql/s3ql/backends/common.py", line 107, in wrapped
return method(*a, **kw)
File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 910, in close
headers=self.headers, body=self.fh)
File "/usr/lib/s3ql/s3ql/backends/swift.py", line 210, in _do_request
self.conn = self._get_conn()
File "/usr/lib/s3ql/s3ql/backends/swift.py", line 167, in _get_conn
raise AuthorizationError(resp.reason)
s3ql.backends.common.AuthorizationError: Access denied. Server said:
Authorization Required
2015-12-10 17:17:22.666 35491:Thread-10 s3ql.mount.exchook: Unhandled
top-level exception during shutdown (will not be re-raised)
2015-12-10 17:16 GMT+01:00 Nikolaus Rath <[email protected]>:
> On Dec 10 2015, Daniel Jagszent <[email protected]> wrote:
> >>> > [...] At the end the filesystem had 16 million directory
> >>> > entries and 1.5 million inodes (Burp uses hard links excessively) and
> >>> > the sqlite database that S3QL uses to store the filesystem structure
> was
> >>> > 1.2 GB uncompressed.
> >>
> >> This is not unreasonable though. Note that ext4 would require at least 5
> >> GB of metadata as well - just to store the inodes (assuming 4096 bytes
> >> inode size). That's not yet counting directory entry *names*.
> >
> > Sure. The size of the sqlite database is reasonable for so many
> > inodes/directory entries. But I suspect that ext4 will scale better in
> > terms of execution time for normal operations like e.g. file system
> > stats (|df|). S3QL needs to do several full table scans
> > <
> https://bitbucket.org/nikratio/s3ql/src/default/src/s3ql/fs.py?fileviewer=file-view-default#fs.py-916:918
> >
> > for that and this will take its time for tables that big (In my case
> > approx. 10 seconds).
>
> True. But this could be fixed easily by keeping the relevant numbers in
> memory and updating them on write/delete. This would be rather simple,
> it's just that no one ever complained about the time required for df or
> s3qlstat before. Personally don't use it more than once every few weeks,
> so the performance never bothered me.
>
> >> However, I have plans in the drawer to fix this at some point. The idea
> >> is to handle reads/writes for blocks that are already cached entirely at
> >> the C level. This will allow concurrenty *and* at the same time boost
> >> single-threaded performance as well. Just need to find the time...
> >
> > That sounds great! (More performance always does :) )
> > Am I right in assuming that this will speed up read/write syscalls but
> > not stuff that solely works on the database?
>
> Yes
>
> > (like opendir or the attr
> > and xattr calls)
>
> xattr requires database access and would stay slow.
> opendir and getattr don't necessarily act on the database (getattr
> results are cached), but they still wouldn't be speed up because S3QL
> itself is mostly single threaded.
>
>
> Best,
> -Nikolaus
>
> --
> GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
> Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
>
> »Time flies like an arrow, fruit flies like a Banana.«
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "s3ql" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/s3ql/hSyJvnXN0zs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
