I had an fsck of an s3ql filesystem on amazon s3 fail due to a full disk on
the local machine. When I try to fsck the file system again, I get these
errors:
Backing up old metadata...
Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
retrying call to Backend.copy for the 3-th time...
Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
retrying call to Backend.copy for the 4-th time...
Is this related to the recent s3 outage? I think the outage is over, but I
still get these errors. Any advice much appreciated. I've attached the full
fsck.log.
Yours
Joseph
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
2016-01-12 17:06:51.595 15867:MainThread (name)s.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/bin/fsck.s3ql", line 9, in <module>
load_entry_point('s3ql==2.11.1', 'console_scripts', 'fsck.s3ql')()
File "/usr/lib/s3ql/s3ql/fsck.py", line 1081, in main
backend = get_backend(options)
File "/usr/lib/s3ql/s3ql/common.py", line 288, in get_backend
return get_backend_factory(options, plain)()
File "/usr/lib/s3ql/s3ql/common.py", line 348, in get_backend_factory
backend_login = getpass("Enter backend login: ")
File "/usr/lib/python3.4/getpass.py", line 78, in unix_getpass
passwd = _raw_input(prompt, stream, input=input)
File "/usr/lib/python3.4/getpass.py", line 147, in _raw_input
line = input.readline()
KeyboardInterrupt
2016-01-12 17:07:17.721 15913:MainThread (name)s.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/bin/fsck.s3ql", line 9, in <module>
load_entry_point('s3ql==2.11.1', 'console_scripts', 'fsck.s3ql')()
File "/usr/lib/s3ql/s3ql/fsck.py", line 1081, in main
backend = get_backend(options)
File "/usr/lib/s3ql/s3ql/common.py", line 288, in get_backend
return get_backend_factory(options, plain)()
File "/usr/lib/s3ql/s3ql/common.py", line 348, in get_backend_factory
backend_login = getpass("Enter backend login: ")
File "/usr/lib/python3.4/getpass.py", line 78, in unix_getpass
passwd = _raw_input(prompt, stream, input=input)
File "/usr/lib/python3.4/getpass.py", line 147, in _raw_input
line = input.readline()
KeyboardInterrupt
2016-01-12 17:12:05.275 16723:MainThread (name)s.main: Starting fsck of
s3://echo-maher-org-uk-backup/
2016-01-12 17:12:06.829 16723:MainThread (name)s.main: Using cached metadata.
2016-01-12 17:12:06.838 16723:MainThread (name)s.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/bin/fsck.s3ql", line 9, in <module>
load_entry_point('s3ql==2.11.1', 'console_scripts', 'fsck.s3ql')()
File "/usr/lib/s3ql/s3ql/fsck.py", line 1099, in main
db = Connection(cachepath + '.db')
File "/usr/lib/s3ql/s3ql/database.py", line 71, in __init__
cur.execute(s)
File "src/cursor.c", line 990, in APSWCursor_execute.sqlite3_prepare
File "src/statementcache.c", line 386, in sqlite3_prepare
apsw.BusyError: BusyError: database is locked
2016-01-12 17:14:46.307 16818:MainThread (name)s.main: Starting fsck of
s3://echo-maher-org-uk-backup/
2016-01-12 17:14:47.695 16818:MainThread (name)s.main: Using cached metadata.
2016-01-12 17:14:47.975 16818:MainThread (name)s.main: Remote metadata is
outdated.
2016-01-12 17:14:47.975 16818:MainThread (name)s.main: Checking DB integrity...
2016-01-12 17:15:00.871 16818:MainThread (name)s.check: Creating temporary
extra indices...
2016-01-12 17:15:01.230 16818:MainThread (name)s.check_lof: Checking
lost+found...
2016-01-12 17:15:01.230 16818:MainThread (name)s.check_cache: Checking cached
objects...
2016-01-12 17:15:01.231 16818:MainThread (name)s.log_error: Committing block 0
of inode 94194 to backend
2016-01-12 17:15:01.231 16818:MainThread (name)s.log_error: Committing block 0
of inode 112690 to backend
2016-01-12 17:15:01.239 16818:MainThread (name)s.log_error: Committing block 0
of inode 114066 to backend
2016-01-12 17:15:01.250 16818:MainThread (name)s.log_error: Committing block 0
of inode 123563 to backend
2016-01-12 17:15:01.270 16818:MainThread (name)s.log_error: Committing block 0
of inode 128525 to backend
2016-01-12 17:15:01.271 16818:MainThread (name)s.log_error: Committing block 0
of inode 130562 to backend
2016-01-12 17:15:01.277 16818:MainThread (name)s.log_error: Committing block 0
of inode 131888 to backend
2016-01-12 17:15:01.287 16818:MainThread (name)s.check_names_refcount: Checking
names (refcounts)...
2016-01-12 17:15:01.909 16818:MainThread (name)s.check_contents_name: Checking
contents (names)...
2016-01-12 17:15:01.989 16818:MainThread (name)s.check_contents_inode: Checking
contents (inodes)...
2016-01-12 17:15:02.067 16818:MainThread (name)s.check_contents_parent_inode:
Checking contents (parent inodes)...
2016-01-12 17:15:02.085 16818:MainThread (name)s.check_objects_refcount:
Checking objects (reference counts)...
2016-01-12 17:15:02.297 16818:MainThread (name)s.check_objects_id: Checking
objects (backend)...
2016-01-12 17:16:41.469 16818:MainThread (name)s.log_error: object 88384 only
exists in table but not in backend, deleting
2016-01-12 17:16:41.470 16818:MainThread (name)s.log_error: File may lack data,
moved to /lost+found:
b'/current/linode/home/math/Maildir/.sent-mail.2011/cur/1325454927.M190181P31201V000000000000CA00I000E8619_369.li12-186,S=2133:2,'
2016-01-12 17:16:41.652 16818:MainThread (name)s.log_error: object 105779 only
exists in table but not in backend, deleting
2016-01-12 17:16:41.652 16818:MainThread (name)s.log_error: File may lack data,
moved to /lost+found:
b'/current/linode/home/suzanne/Maildir/.Other.Property.Rent
Info.Lettingsupermarket/cur/1405605902.M297041P16840V000000000000CA00I00000000000A6102_17.li12-186,S=13157:2,RS'
2016-01-12 17:16:41.831 16818:MainThread (name)s.log_error: object 106943 only
exists in table but not in backend, deleting
2016-01-12 17:16:41.832 16818:MainThread (name)s.log_error: File may lack data,
moved to /lost+found:
b'/current/linode/home/suzanne/Maildir/.Other.TAX/cur/1273003917.M421475P29822V000000000000CA00I0003D1D4_43.li12-186,S=26110:2,S'
2016-01-12 17:16:42.012 16818:MainThread (name)s.log_error: object 120534 only
exists in table but not in backend, deleting
2016-01-12 17:16:42.012 16818:MainThread (name)s.log_error: File may lack data,
moved to /lost+found:
b'/current/linode/home/tonyandkat/Maildir/.Sent/cur/1293143310.M284600P9941V000000000000CA00I00024FD0_2.li12-186,S=2019:2,S'
2016-01-12 17:16:42.191 16818:MainThread (name)s.log_error: object 122552 only
exists in table but not in backend, deleting
2016-01-12 17:16:42.191 16818:MainThread (name)s.log_error: File may lack data,
moved to /lost+found:
b'/current/linode/home/tonyandkat/Maildir/.spam/cur/1386269544.H639904P4658.li12-186.members.linode.com:2,'
2016-01-12 17:16:42.375 16818:MainThread (name)s.log_error: object 123448 only
exists in table but not in backend, deleting
2016-01-12 17:16:42.376 16818:MainThread (name)s.log_error: File may lack data,
moved to /lost+found:
b'/current/linode/home/tonyandkat/Maildir/cur/1358106985.H862918P22655.li12-186.members.linode.com:2,S'
2016-01-12 17:16:42.385 16818:MainThread (name)s.check_objects_size: Checking
objects (sizes)...
2016-01-12 17:16:42.396 16818:MainThread (name)s.log_error: Object 115906 has
no size information, retrieving from backend...
2016-01-12 17:16:42.598 16818:MainThread (name)s.check_blocks_obj_id: Checking
blocks (referenced objects)...
2016-01-12 17:16:42.667 16818:MainThread (name)s.check_blocks_refcount:
Checking blocks (refcounts)...
2016-01-12 17:16:42.940 16818:MainThread (name)s.check_inode_blocks_block_id:
Checking inode-block mapping (blocks)...
2016-01-12 17:16:43.011 16818:MainThread (name)s.check_inode_blocks_inode:
Checking inode-block mapping (inodes)...
2016-01-12 17:16:43.081 16818:MainThread (name)s.check_inodes_refcount:
Checking inodes (refcounts)...
2016-01-12 17:16:43.377 16818:MainThread (name)s.check_inodes_size: Checking
inodes (sizes)...
2016-01-12 17:16:43.757 16818:MainThread (name)s.check_ext_attributes_name:
Checking extended attributes (names)...
2016-01-12 17:16:43.757 16818:MainThread (name)s.check_ext_attributes_inode:
Checking extended attributes (inodes)...
2016-01-12 17:16:43.757 16818:MainThread (name)s.check_symlinks_inode: Checking
symlinks (inodes)...
2016-01-12 17:16:43.759 16818:MainThread (name)s.check_loops: Checking
directory reachability...
2016-01-12 17:16:44.024 16818:MainThread (name)s.check_unix: Checking unix
conventions...
2016-01-12 17:16:46.465 16818:MainThread (name)s.check_foreign_keys: Checking
referential integrity...
2016-01-12 17:16:46.837 16818:MainThread (name)s.check: Dropping temporary
indices...
2016-01-12 17:16:46.852 16818:MainThread (name)s.main: Dumping metadata...
2016-01-12 17:16:46.853 16818:MainThread (name)s.dump_metadata: ..objects..
2016-01-12 17:16:46.926 16818:MainThread (name)s.dump_metadata: ..blocks..
2016-01-12 17:16:47.054 16818:MainThread (name)s.dump_metadata: ..inodes..
2016-01-12 17:16:47.313 16818:MainThread (name)s.dump_metadata: ..inode_blocks..
2016-01-12 17:16:47.453 16818:MainThread (name)s.dump_metadata:
..symlink_targets..
2016-01-12 17:16:47.456 16818:MainThread (name)s.dump_metadata: ..names..
2016-01-12 17:16:47.562 16818:MainThread (name)s.dump_metadata: ..contents..
2016-01-12 17:16:47.710 16818:MainThread (name)s.dump_metadata:
..ext_attributes..
2016-01-12 17:16:47.711 16818:MainThread (name)s.main: Compressing and
uploading metadata...
2016-01-12 17:16:55.433 16818:MainThread (name)s.main: Wrote 8 MiB of
compressed metadata.
2016-01-12 17:16:55.433 16818:MainThread (name)s.main: Cycling metadata
backups...
2016-01-12 17:16:55.433 16818:MainThread (name)s.cycle_metadata: Backing up old
metadata...
2016-01-12 17:16:58.639 16818:MainThread (name)s.main: Cleaning up local
metadata...
2016-01-12 17:16:59.628 16818:MainThread (name)s.main: Completed fsck of
s3://echo-maher-org-uk-backup/
2016-01-12 22:12:44.622 32343:MainThread (name)s.main: Starting fsck of
s3://echo-maher-org-uk-backup/
2016-01-12 22:12:45.969 32343:MainThread (name)s.main: Using cached metadata.
2016-01-12 22:12:46.148 32343:MainThread (name)s.main: File system is marked as
clean. Use --force to force checking.
2017-02-03 22:12:35.254 12168:MainThread (name)s.main: Starting fsck of
s3://echo-maher-org-uk-backup/
2017-02-03 22:12:36.674 12168:MainThread (name)s.main: Using cached metadata.
2017-02-03 22:12:36.888 12168:MainThread (name)s.main: Checking DB integrity...
2017-02-03 22:14:57.989 12168:MainThread (name)s.check: Creating temporary
extra indices...
2017-02-03 22:18:23.319 12168:MainThread (name)s.check_lof: Checking
lost+found...
2017-02-03 22:18:23.320 12168:MainThread (name)s.check_cache: Checking cached
objects...
2017-02-03 22:18:23.320 12168:MainThread (name)s.check_names_refcount: Checking
names (refcounts)...
2017-02-03 22:18:36.409 12168:MainThread (name)s.check_contents_name: Checking
contents (names)...
2017-02-03 22:19:12.734 12168:MainThread (name)s.check_contents_inode: Checking
contents (inodes)...
2017-02-03 22:19:57.551 12168:MainThread (name)s.check_contents_parent_inode:
Checking contents (parent inodes)...
2017-02-03 22:20:08.731 12168:MainThread (name)s.check_objects_refcount:
Checking objects (reference counts)...
2017-02-03 22:20:11.932 12168:MainThread (name)s.check_objects_id: Checking
objects (backend)...
2017-02-03 22:39:24.043 12168:MainThread (name)s.check_objects_size: Checking
objects (sizes)...
2017-02-03 22:39:24.182 12168:MainThread (name)s.check_blocks_obj_id: Checking
blocks (referenced objects)...
2017-02-03 22:39:25.200 12168:MainThread (name)s.check_blocks_refcount:
Checking blocks (refcounts)...
2017-02-03 22:40:21.034 12168:MainThread (name)s.check_inode_blocks_block_id:
Checking inode-block mapping (blocks)...
2017-02-03 22:40:51.788 12168:MainThread (name)s.check_inode_blocks_inode:
Checking inode-block mapping (inodes)...
2017-02-03 22:41:28.583 12168:MainThread (name)s.check_inodes_refcount:
Checking inodes (refcounts)...
2017-02-03 22:44:02.732 12168:MainThread (name)s.check_inodes_size: Checking
inodes (sizes)...
2017-02-03 22:47:20.663 12168:MainThread (name)s.check_ext_attributes_name:
Checking extended attributes (names)...
2017-02-03 22:47:20.663 12168:MainThread (name)s.check_ext_attributes_inode:
Checking extended attributes (inodes)...
2017-02-03 22:47:20.664 12168:MainThread (name)s.check_symlinks_inode: Checking
symlinks (inodes)...
2017-02-03 22:47:22.542 12168:MainThread (name)s.check_loops: Checking
directory reachability...
2017-02-03 22:54:32.143 12168:MainThread (name)s.check_unix: Checking unix
conventions...
2017-02-03 23:12:49.117 12168:MainThread (name)s.check_foreign_keys: Checking
referential integrity...
2017-02-03 23:14:46.385 12168:MainThread (name)s.check: Dropping temporary
indices...
2017-02-03 23:14:51.216 12168:MainThread (name)s.main: Dumping metadata...
2017-02-03 23:14:51.217 12168:MainThread (name)s.dump_metadata: ..objects..
2017-02-03 23:14:52.154 12168:MainThread (name)s.dump_metadata: ..blocks..
2017-02-03 23:14:53.761 12168:MainThread (name)s.dump_metadata: ..inodes..
2017-02-03 23:16:42.591 12168:MainThread (name)s.dump_metadata: ..inode_blocks..
2017-02-03 23:17:47.711 12168:MainThread (name)s.dump_metadata:
..symlink_targets..
2017-02-03 23:17:49.035 12168:MainThread (name)s.dump_metadata: ..names..
2017-02-03 23:17:50.542 12168:MainThread (name)s.dump_metadata: ..contents..
2017-02-03 23:20:00.678 12168:MainThread (name)s.dump_metadata:
..ext_attributes..
2017-02-03 23:20:01.173 12168:MainThread (name)s.main: Compressing and
uploading metadata...
2017-02-03 23:20:58.707 12168:MainThread (name)s.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/bin/fsck.s3ql", line 9, in <module>
load_entry_point('s3ql==2.11.1', 'console_scripts', 'fsck.s3ql')()
File "/usr/lib/s3ql/s3ql/fsck.py", line 1221, in main
is_compressed=True)
File "/usr/lib/s3ql/s3ql/backends/common.py", line 46, in wrapped
return method(*a, **kw)
File "/usr/lib/s3ql/s3ql/backends/common.py", line 258, in perform_write
return fn(fh)
File "/usr/lib/s3ql/s3ql/fsck.py", line 1216, in do_write
stream_write_bz2(fh, obj_fh)
File "/usr/lib/s3ql/s3ql/common.py", line 82, in stream_write_bz2
ofh.write(buf)
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 618, in write
self.fh.write(buf2)
File "/usr/lib/s3ql/s3ql/backends/s3c.py", line 821, in write
self.fh.write(buf)
OSError: [Errno 28] No space left on device
2017-03-01 08:04:16.594 5268:MainThread (name)s.main: Starting fsck of
s3://echo-maher-org-uk-backup/
2017-03-01 08:04:18.021 5268:MainThread (name)s.main: Using cached metadata.
2017-03-01 08:04:18.220 5268:MainThread (name)s.main: Remote metadata is
outdated.
2017-03-01 08:04:18.220 5268:MainThread (name)s.main: Checking DB integrity...
2017-03-01 08:08:13.371 5268:MainThread (name)s.check: Creating temporary extra
indices...
2017-03-01 08:11:38.452 5268:MainThread (name)s.check_lof: Checking
lost+found...
2017-03-01 08:11:38.484 5268:MainThread (name)s.check_cache: Checking cached
objects...
2017-03-01 08:11:38.485 5268:MainThread (name)s.check_names_refcount: Checking
names (refcounts)...
2017-03-01 08:11:51.164 5268:MainThread (name)s.check_contents_name: Checking
contents (names)...
2017-03-01 08:12:28.503 5268:MainThread (name)s.check_contents_inode: Checking
contents (inodes)...
2017-03-01 08:13:19.878 5268:MainThread (name)s.check_contents_parent_inode:
Checking contents (parent inodes)...
2017-03-01 08:13:31.882 5268:MainThread (name)s.check_objects_refcount:
Checking objects (reference counts)...
2017-03-01 08:13:35.285 5268:MainThread (name)s.check_objects_id: Checking
objects (backend)...
2017-03-01 08:33:08.444 5268:MainThread (name)s.check_objects_size: Checking
objects (sizes)...
2017-03-01 08:33:08.583 5268:MainThread (name)s.check_blocks_obj_id: Checking
blocks (referenced objects)...
2017-03-01 08:33:09.668 5268:MainThread (name)s.check_blocks_refcount: Checking
blocks (refcounts)...
2017-03-01 08:34:08.241 5268:MainThread (name)s.check_inode_blocks_block_id:
Checking inode-block mapping (blocks)...
2017-03-01 08:34:39.683 5268:MainThread (name)s.check_inode_blocks_inode:
Checking inode-block mapping (inodes)...
2017-03-01 08:35:17.482 5268:MainThread (name)s.check_inodes_refcount: Checking
inodes (refcounts)...
2017-03-01 08:37:56.016 5268:MainThread (name)s.check_inodes_size: Checking
inodes (sizes)...
2017-03-01 08:41:07.064 5268:MainThread (name)s.check_ext_attributes_name:
Checking extended attributes (names)...
2017-03-01 08:41:07.081 5268:MainThread (name)s.check_ext_attributes_inode:
Checking extended attributes (inodes)...
2017-03-01 08:41:07.081 5268:MainThread (name)s.check_symlinks_inode: Checking
symlinks (inodes)...
2017-03-01 08:41:09.646 5268:MainThread (name)s.check_loops: Checking directory
reachability...
2017-03-01 08:51:21.538 5268:MainThread (name)s.check_unix: Checking unix
conventions...
2017-03-01 09:10:19.679 5268:MainThread (name)s.check_foreign_keys: Checking
referential integrity...
2017-03-01 09:12:37.253 5268:MainThread (name)s.check: Dropping temporary
indices...
2017-03-01 09:12:45.189 5268:MainThread (name)s.main: Dumping metadata...
2017-03-01 09:12:45.228 5268:MainThread (name)s.dump_metadata: ..objects..
2017-03-01 09:12:46.152 5268:MainThread (name)s.dump_metadata: ..blocks..
2017-03-01 09:12:49.065 5268:MainThread (name)s.dump_metadata: ..inodes..
2017-03-01 09:14:35.974 5268:MainThread (name)s.dump_metadata: ..inode_blocks..
2017-03-01 09:15:47.985 5268:MainThread (name)s.dump_metadata:
..symlink_targets..
2017-03-01 09:15:49.262 5268:MainThread (name)s.dump_metadata: ..names..
2017-03-01 09:15:50.615 5268:MainThread (name)s.dump_metadata: ..contents..
2017-03-01 09:18:33.257 5268:MainThread (name)s.dump_metadata:
..ext_attributes..
2017-03-01 09:18:33.258 5268:MainThread (name)s.main: Compressing and uploading
metadata...
2017-03-01 09:25:34.897 5268:MainThread (name)s.main: Wrote 582 MiB of
compressed metadata.
2017-03-01 09:25:34.897 5268:MainThread (name)s.main: Cycling metadata
backups...
2017-03-01 09:25:34.897 5268:MainThread (name)s.cycle_metadata: Backing up old
metadata...
2017-03-01 09:26:25.286 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 09:27:05.626 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 09:27:16.224 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 09:27:57.258 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 09:28:39.553 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 09:28:50.160 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 09:29:00.873 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 5-th time...
2017-03-01 09:29:11.757 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 6-th time...
2017-03-01 09:29:22.927 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 7-th time...
2017-03-01 09:29:34.846 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 8-th time...
2017-03-01 09:29:47.944 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 9-th time...
2017-03-01 09:30:03.702 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 10-th time...
2017-03-01 09:30:24.482 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 11-th time...
2017-03-01 09:30:55.522 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 12-th time...
2017-03-01 09:31:47.048 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 13-th time...
2017-03-01 09:33:19.593 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 14-th time...
2017-03-01 09:36:14.070 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 15-th time...
2017-03-01 09:41:24.729 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 16-th time...
2017-03-01 09:46:35.483 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 17-th time...
2017-03-01 09:52:17.283 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 09:52:28.214 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 09:52:39.158 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 5-th time...
2017-03-01 09:52:50.098 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 6-th time...
2017-03-01 09:53:01.503 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 7-th time...
2017-03-01 09:53:13.411 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 8-th time...
2017-03-01 09:53:26.532 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 9-th time...
2017-03-01 09:53:42.198 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 10-th time...
2017-03-01 09:54:02.993 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 11-th time...
2017-03-01 09:54:34.334 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 12-th time...
2017-03-01 09:55:25.868 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 13-th time...
2017-03-01 09:56:58.468 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 14-th time...
2017-03-01 09:59:52.926 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 15-th time...
2017-03-01 10:05:03.578 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 16-th time...
2017-03-01 10:10:14.526 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 17-th time...
2017-03-01 10:15:25.395 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 18-th time...
2017-03-01 10:20:36.011 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 19-th time...
2017-03-01 10:25:46.684 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 20-th time...
2017-03-01 10:30:57.437 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 21-th time...
2017-03-01 10:36:08.065 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 22-th time...
2017-03-01 10:41:18.695 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 23-th time...
2017-03-01 10:46:29.888 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 24-th time...
2017-03-01 10:51:40.507 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 25-th time...
2017-03-01 10:56:51.188 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 26-th time...
2017-03-01 11:02:01.835 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 27-th time...
2017-03-01 11:07:12.715 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 28-th time...
2017-03-01 11:12:23.468 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 29-th time...
2017-03-01 11:17:34.426 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 30-th time...
2017-03-01 11:22:45.059 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 31-th time...
2017-03-01 11:27:56.012 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 32-th time...
2017-03-01 11:33:06.762 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 33-th time...
2017-03-01 11:38:17.515 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 34-th time...
2017-03-01 11:43:28.139 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 35-th time...
2017-03-01 11:48:38.996 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 36-th time...
2017-03-01 11:53:49.610 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 37-th time...
2017-03-01 11:59:00.642 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 38-th time...
2017-03-01 12:04:11.287 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 39-th time...
2017-03-01 12:09:22.186 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 40-th time...
2017-03-01 12:14:32.928 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 41-th time...
2017-03-01 12:19:43.889 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 42-th time...
2017-03-01 12:24:54.510 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 43-th time...
2017-03-01 12:30:05.507 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 44-th time...
2017-03-01 12:35:47.109 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 12:35:57.978 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 12:36:08.665 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 5-th time...
2017-03-01 12:36:19.524 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 6-th time...
2017-03-01 12:36:31.010 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 7-th time...
2017-03-01 12:36:42.826 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 8-th time...
2017-03-01 12:36:55.915 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 9-th time...
2017-03-01 12:37:11.577 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 10-th time...
2017-03-01 12:37:32.340 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 11-th time...
2017-03-01 12:38:03.374 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 12-th time...
2017-03-01 12:38:55.030 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 13-th time...
2017-03-01 12:40:27.501 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 14-th time...
2017-03-01 12:43:21.906 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 15-th time...
2017-03-01 12:48:32.559 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 16-th time...
2017-03-01 12:53:43.193 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 17-th time...
2017-03-01 12:58:53.845 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 18-th time...
2017-03-01 13:04:04.748 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 19-th time...
2017-03-01 13:09:15.466 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 20-th time...
2017-03-01 13:14:26.107 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 21-th time...
2017-03-01 13:19:36.737 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 22-th time...
2017-03-01 13:24:47.383 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 23-th time...
2017-03-01 13:29:58.009 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 24-th time...
2017-03-01 13:35:09.007 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 25-th time...
2017-03-01 13:40:19.648 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 26-th time...
2017-03-01 13:45:30.231 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 27-th time...
2017-03-01 13:50:41.190 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 28-th time...
2017-03-01 13:55:52.044 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 29-th time...
2017-03-01 14:01:02.679 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 30-th time...
2017-03-01 14:06:13.348 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 31-th time...
2017-03-01 14:11:24.229 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 32-th time...
2017-03-01 14:17:05.936 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 14:17:16.793 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 14:17:27.503 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 5-th time...
2017-03-01 14:17:38.338 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 6-th time...
2017-03-01 14:17:49.497 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 7-th time...
2017-03-01 14:18:01.329 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 8-th time...
2017-03-01 14:18:14.426 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 9-th time...
2017-03-01 14:18:30.087 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 10-th time...
2017-03-01 14:18:50.967 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 11-th time...
2017-03-01 14:19:22.018 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 12-th time...
2017-03-01 14:20:13.680 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 13-th time...
2017-03-01 14:21:46.209 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 14-th time...
2017-03-01 14:24:40.661 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 15-th time...
2017-03-01 14:29:51.536 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 16-th time...
2017-03-01 14:35:02.166 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 17-th time...
2017-03-01 14:40:13.059 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 18-th time...
2017-03-01 14:45:23.940 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 19-th time...
2017-03-01 14:50:34.571 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 20-th time...
2017-03-01 14:55:45.200 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 21-th time...
2017-03-01 15:00:56.469 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 22-th time...
2017-03-01 15:06:07.083 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 23-th time...
2017-03-01 15:11:17.747 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 24-th time...
2017-03-01 15:16:28.661 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 25-th time...
2017-03-01 15:21:39.281 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 26-th time...
2017-03-01 15:26:49.917 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 27-th time...
2017-03-01 15:32:00.854 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 28-th time...
2017-03-01 15:37:11.499 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 29-th time...
2017-03-01 15:42:22.492 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 30-th time...
2017-03-01 15:47:33.365 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 31-th time...
2017-03-01 15:52:43.994 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 32-th time...
2017-03-01 15:57:54.940 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 33-th time...
2017-03-01 16:03:05.897 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 34-th time...
2017-03-01 16:08:16.629 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 35-th time...
2017-03-01 16:13:27.400 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 36-th time...
2017-03-01 16:18:38.087 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 37-th time...
2017-03-01 16:23:49.065 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 38-th time...
2017-03-01 16:28:59.700 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 39-th time...
2017-03-01 16:34:10.341 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 40-th time...
2017-03-01 16:39:21.258 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 41-th time...
2017-03-01 16:44:32.200 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 42-th time...
2017-03-01 16:49:43.107 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 43-th time...
2017-03-01 16:54:53.758 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 44-th time...
2017-03-01 17:00:04.634 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 45-th time...
2017-03-01 17:05:15.833 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 46-th time...
2017-03-01 17:10:26.471 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 47-th time...
2017-03-01 17:15:37.365 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 48-th time...
2017-03-01 17:20:48.313 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 49-th time...
2017-03-01 17:25:59.048 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 50-th time...
2017-03-01 17:31:09.627 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 51-th time...
2017-03-01 17:36:20.543 5268:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 52-th time...
2017-03-01 17:38:56.008 5268:MainThread (name)s.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/bin/fsck.s3ql", line 9, in <module>
load_entry_point('s3ql==2.11.1', 'console_scripts', 'fsck.s3ql')()
File "/usr/lib/s3ql/s3ql/fsck.py", line 1224, in main
cycle_metadata(backend)
File "/usr/lib/s3ql/s3ql/metadata.py", line 121, in cycle_metadata
cycle_fn("s3ql_metadata_bak_%d" % i, "s3ql_metadata_bak_%d" % (i + 1))
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 290, in copy
self._copy_or_rename(src, dest, rename=False, metadata=metadata)
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 323, in _copy_or_rename
self.backend.copy(src, dest, metadata=meta_raw)
File "/usr/lib/s3ql/s3ql/backends/s3.py", line 93, in copy
extra_headers=extra_headers)
File "/usr/lib/s3ql/s3ql/backends/common.py", line 72, in wrapped
time.sleep(interval)
KeyboardInterrupt
2017-03-01 17:39:03.083 30959:MainThread (name)s.main: Starting fsck of
s3://echo-maher-org-uk-backup/
2017-03-01 17:39:04.877 30959:MainThread (name)s.main: Using cached metadata.
2017-03-01 17:39:05.145 30959:MainThread (name)s.main: Remote metadata is
outdated.
2017-03-01 17:39:05.145 30959:MainThread (name)s.main: Checking DB integrity...
2017-03-01 17:42:44.396 30959:MainThread (name)s.check: Creating temporary
extra indices...
2017-03-01 17:46:00.239 30959:MainThread (name)s.check_lof: Checking
lost+found...
2017-03-01 17:46:00.347 30959:MainThread (name)s.check_cache: Checking cached
objects...
2017-03-01 17:46:00.348 30959:MainThread (name)s.check_names_refcount: Checking
names (refcounts)...
2017-03-01 17:46:13.222 30959:MainThread (name)s.check_contents_name: Checking
contents (names)...
2017-03-01 17:46:50.209 30959:MainThread (name)s.check_contents_inode: Checking
contents (inodes)...
2017-03-01 17:47:42.285 30959:MainThread (name)s.check_contents_parent_inode:
Checking contents (parent inodes)...
2017-03-01 17:47:54.273 30959:MainThread (name)s.check_objects_refcount:
Checking objects (reference counts)...
2017-03-01 17:47:57.603 30959:MainThread (name)s.check_objects_id: Checking
objects (backend)...
2017-03-01 18:07:00.972 30959:MainThread (name)s.check_objects_size: Checking
objects (sizes)...
2017-03-01 18:07:01.110 30959:MainThread (name)s.check_blocks_obj_id: Checking
blocks (referenced objects)...
2017-03-01 18:07:02.188 30959:MainThread (name)s.check_blocks_refcount:
Checking blocks (refcounts)...
2017-03-01 18:07:55.422 30959:MainThread (name)s.check_inode_blocks_block_id:
Checking inode-block mapping (blocks)...
2017-03-01 18:08:26.448 30959:MainThread (name)s.check_inode_blocks_inode:
Checking inode-block mapping (inodes)...
2017-03-01 18:09:03.653 30959:MainThread (name)s.check_inodes_refcount:
Checking inodes (refcounts)...
2017-03-01 18:11:39.773 30959:MainThread (name)s.check_inodes_size: Checking
inodes (sizes)...
2017-03-01 18:14:50.972 30959:MainThread (name)s.check_ext_attributes_name:
Checking extended attributes (names)...
2017-03-01 18:14:50.972 30959:MainThread (name)s.check_ext_attributes_inode:
Checking extended attributes (inodes)...
2017-03-01 18:14:50.972 30959:MainThread (name)s.check_symlinks_inode: Checking
symlinks (inodes)...
2017-03-01 18:14:53.136 30959:MainThread (name)s.check_loops: Checking
directory reachability...
2017-03-01 18:22:02.689 30959:MainThread (name)s.check_unix: Checking unix
conventions...
2017-03-01 18:39:20.474 30959:MainThread (name)s.check_foreign_keys: Checking
referential integrity...
2017-03-01 18:41:19.529 30959:MainThread (name)s.check: Dropping temporary
indices...
2017-03-01 18:41:24.368 30959:MainThread (name)s.main: Dumping metadata...
2017-03-01 18:41:24.369 30959:MainThread (name)s.dump_metadata: ..objects..
2017-03-01 18:41:25.470 30959:MainThread (name)s.dump_metadata: ..blocks..
2017-03-01 18:41:27.049 30959:MainThread (name)s.dump_metadata: ..inodes..
2017-03-01 18:43:12.815 30959:MainThread (name)s.dump_metadata: ..inode_blocks..
2017-03-01 18:44:13.827 30959:MainThread (name)s.dump_metadata:
..symlink_targets..
2017-03-01 18:44:15.096 30959:MainThread (name)s.dump_metadata: ..names..
2017-03-01 18:44:16.197 30959:MainThread (name)s.dump_metadata: ..contents..
2017-03-01 18:45:38.645 30959:MainThread (name)s.dump_metadata:
..ext_attributes..
2017-03-01 18:45:38.646 30959:MainThread (name)s.main: Compressing and
uploading metadata...
2017-03-01 18:52:43.541 30959:MainThread (name)s.main: Wrote 582 MiB of
compressed metadata.
2017-03-01 18:52:43.541 30959:MainThread (name)s.main: Cycling metadata
backups...
2017-03-01 18:52:43.541 30959:MainThread (name)s.cycle_metadata: Backing up old
metadata...
2017-03-01 18:55:02.788 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 18:55:43.003 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 18:55:53.612 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 18:56:56.500 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 18:57:07.125 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 18:57:17.803 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 5-th time...
2017-03-01 18:57:28.657 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 6-th time...
2017-03-01 18:57:39.840 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 7-th time...
2017-03-01 18:57:51.665 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 8-th time...
2017-03-01 18:58:04.766 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 9-th time...
2017-03-01 18:58:20.412 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 10-th time...
2017-03-01 18:58:41.214 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 11-th time...
2017-03-01 18:59:12.239 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 12-th time...
2017-03-01 19:00:03.877 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 13-th time...
2017-03-01 19:01:36.417 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 14-th time...
2017-03-01 19:04:30.890 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 15-th time...
2017-03-01 19:09:41.855 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 16-th time...
2017-03-01 19:14:52.509 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 17-th time...
2017-03-01 19:20:03.390 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 18-th time...
2017-03-01 19:25:14.625 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 19-th time...
2017-03-01 19:30:25.528 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 20-th time...
2017-03-01 19:35:36.808 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 21-th time...
2017-03-01 19:40:47.561 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 22-th time...
2017-03-01 19:45:58.206 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 23-th time...
2017-03-01 19:51:09.023 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 24-th time...
2017-03-01 19:56:19.963 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 25-th time...
2017-03-01 20:01:30.909 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 26-th time...
2017-03-01 20:06:41.538 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 27-th time...
2017-03-01 20:11:52.402 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 28-th time...
2017-03-01 20:17:03.137 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 29-th time...
2017-03-01 20:22:14.017 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 30-th time...
2017-03-01 20:27:25.012 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 31-th time...
2017-03-01 20:32:35.661 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 32-th time...
2017-03-01 20:37:46.266 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 33-th time...
2017-03-01 20:42:57.163 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 34-th time...
2017-03-01 20:48:08.043 30959:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 35-th time...
2017-03-01 20:49:02.965 30959:MainThread (name)s.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File "/usr/bin/fsck.s3ql", line 9, in <module>
load_entry_point('s3ql==2.11.1', 'console_scripts', 'fsck.s3ql')()
File "/usr/lib/s3ql/s3ql/fsck.py", line 1224, in main
cycle_metadata(backend)
File "/usr/lib/s3ql/s3ql/metadata.py", line 121, in cycle_metadata
cycle_fn("s3ql_metadata_bak_%d" % i, "s3ql_metadata_bak_%d" % (i + 1))
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 290, in copy
self._copy_or_rename(src, dest, rename=False, metadata=metadata)
File "/usr/lib/s3ql/s3ql/backends/comprenc.py", line 323, in _copy_or_rename
self.backend.copy(src, dest, metadata=meta_raw)
File "/usr/lib/s3ql/s3ql/backends/s3.py", line 93, in copy
extra_headers=extra_headers)
File "/usr/lib/s3ql/s3ql/backends/common.py", line 72, in wrapped
time.sleep(interval)
KeyboardInterrupt
2017-03-01 20:49:09.150 28468:MainThread (name)s.main: Starting fsck of
s3://echo-maher-org-uk-backup/
2017-03-01 20:49:10.908 28468:MainThread (name)s.main: Using cached metadata.
2017-03-01 20:49:11.147 28468:MainThread (name)s.main: Remote metadata is
outdated.
2017-03-01 20:49:11.147 28468:MainThread (name)s.main: Checking DB integrity...
2017-03-01 20:51:34.902 28468:MainThread (name)s.check: Creating temporary
extra indices...
2017-03-01 20:54:50.699 28468:MainThread (name)s.check_lof: Checking
lost+found...
2017-03-01 20:54:50.699 28468:MainThread (name)s.check_cache: Checking cached
objects...
2017-03-01 20:54:50.700 28468:MainThread (name)s.check_names_refcount: Checking
names (refcounts)...
2017-03-01 20:55:03.432 28468:MainThread (name)s.check_contents_name: Checking
contents (names)...
2017-03-01 20:55:40.897 28468:MainThread (name)s.check_contents_inode: Checking
contents (inodes)...
2017-03-01 20:56:26.339 28468:MainThread (name)s.check_contents_parent_inode:
Checking contents (parent inodes)...
2017-03-01 20:56:51.969 28468:MainThread (name)s.check_objects_refcount:
Checking objects (reference counts)...
2017-03-01 20:56:55.126 28468:MainThread (name)s.check_objects_id: Checking
objects (backend)...
2017-03-01 21:15:49.410 28468:MainThread (name)s.check_objects_size: Checking
objects (sizes)...
2017-03-01 21:15:49.546 28468:MainThread (name)s.check_blocks_obj_id: Checking
blocks (referenced objects)...
2017-03-01 21:15:50.595 28468:MainThread (name)s.check_blocks_refcount:
Checking blocks (refcounts)...
2017-03-01 21:16:43.767 28468:MainThread (name)s.check_inode_blocks_block_id:
Checking inode-block mapping (blocks)...
2017-03-01 21:17:14.927 28468:MainThread (name)s.check_inode_blocks_inode:
Checking inode-block mapping (inodes)...
2017-03-01 21:17:52.398 28468:MainThread (name)s.check_inodes_refcount:
Checking inodes (refcounts)...
2017-03-01 21:20:31.529 28468:MainThread (name)s.check_inodes_size: Checking
inodes (sizes)...
2017-03-01 21:23:47.454 28468:MainThread (name)s.check_ext_attributes_name:
Checking extended attributes (names)...
2017-03-01 21:23:47.454 28468:MainThread (name)s.check_ext_attributes_inode:
Checking extended attributes (inodes)...
2017-03-01 21:23:47.454 28468:MainThread (name)s.check_symlinks_inode: Checking
symlinks (inodes)...
2017-03-01 21:23:51.448 28468:MainThread (name)s.check_loops: Checking
directory reachability...
2017-03-01 21:31:24.083 28468:MainThread (name)s.check_unix: Checking unix
conventions...
2017-03-01 21:50:28.126 28468:MainThread (name)s.check_foreign_keys: Checking
referential integrity...
2017-03-01 21:53:34.158 28468:MainThread (name)s.check: Dropping temporary
indices...
2017-03-01 21:53:39.740 28468:MainThread (name)s.main: Dumping metadata...
2017-03-01 21:53:39.741 28468:MainThread (name)s.dump_metadata: ..objects..
2017-03-01 21:53:40.697 28468:MainThread (name)s.dump_metadata: ..blocks..
2017-03-01 21:53:44.292 28468:MainThread (name)s.dump_metadata: ..inodes..
2017-03-01 21:55:32.455 28468:MainThread (name)s.dump_metadata: ..inode_blocks..
2017-03-01 21:56:39.891 28468:MainThread (name)s.dump_metadata:
..symlink_targets..
2017-03-01 21:56:41.345 28468:MainThread (name)s.dump_metadata: ..names..
2017-03-01 21:56:42.858 28468:MainThread (name)s.dump_metadata: ..contents..
2017-03-01 21:58:52.688 28468:MainThread (name)s.dump_metadata:
..ext_attributes..
2017-03-01 21:58:52.689 28468:MainThread (name)s.main: Compressing and
uploading metadata...
2017-03-01 22:06:01.030 28468:MainThread (name)s.main: Wrote 582 MiB of
compressed metadata.
2017-03-01 22:06:01.030 28468:MainThread (name)s.main: Cycling metadata
backups...
2017-03-01 22:06:01.031 28468:MainThread (name)s.cycle_metadata: Backing up old
metadata...
2017-03-01 22:06:41.655 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 22:06:52.505 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 22:07:43.619 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 22:07:54.236 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 22:08:04.948 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 5-th time...
2017-03-01 22:09:06.288 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 22:09:17.021 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 22:09:27.699 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 5-th time...
2017-03-01 22:09:38.887 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 6-th time...
2017-03-01 22:09:50.086 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 7-th time...
2017-03-01 22:10:01.890 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 8-th time...
2017-03-01 22:10:14.988 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 9-th time...
2017-03-01 22:10:30.654 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 10-th time...
2017-03-01 22:10:51.465 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 11-th time...
2017-03-01 22:11:22.500 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 12-th time...
2017-03-01 22:12:44.753 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 22:13:26.276 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 3-th time...
2017-03-01 22:13:36.892 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 4-th time...
2017-03-01 22:13:47.579 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 5-th time...
2017-03-01 22:13:58.436 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 6-th time...
2017-03-01 22:14:09.727 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 7-th time...
2017-03-01 22:14:21.556 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 8-th time...
2017-03-01 22:14:34.634 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 9-th time...
2017-03-01 22:14:50.297 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 10-th time...
2017-03-01 22:15:11.093 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 11-th time...
2017-03-01 22:15:42.109 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 12-th time...
2017-03-01 22:16:33.628 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 13-th time...
2017-03-01 22:18:06.176 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 14-th time...
2017-03-01 22:21:00.672 28468:MainThread (name)s.wrapped: Encountered
ConnectionTimedOut exception (send/recv timeout exceeded), retrying call to
Backend.copy for the 15-th time...
