I'm wondering someone can help me out with an error I'm receiving on S3QL.
I am running it on Ubuntu, with the S3QL PPA added, and recently (today)
did an apt-get dist-upgrade of my instance. Included in that was updating
s3ql to version 1.19-1~precise1. I am running Ubuntu Precise (12.04.4),
and the backported trusty kernel (3.13.0-36-generic #63~precise1-Ubuntu SMP
Thu Sep 4 22:28:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux). Other than
that, there is nothing out of the ordinary installed or configured. This
is a server in AWS.
After I rebooted (also updated my kernel) my s3 filesystem failed to
mount. I tried to run a mount manually and received the following error:
~/.s3ql # mount.s3ql --debug all --upstart --allow-other --cachesize
16777216 --cachedir /mnt/tmpdisk/S3QL-cache --compress none --authfile
/root/.s3ql/authinfo2 --metadata-upload-interval 600
s3://secops-data/PMS/POC_ /mnt/pms_store --nfs
Using 8 upload threads.
Ignoring locally cached metadata (outdated).
Backend reports that fs is still mounted elsewhere, aborting.
I've seen this before, so I then tried to run an fsck.s3ql. Usually that
works for me, but now I have received the following error when running that
script:
~/.s3ql # fsck.s3ql --force s3://secops-data/PMS/POC_
Starting fsck of s3://secops-data/PMS/POC_
Ignoring locally cached metadata (outdated).
Backend reports that file system is still mounted elsewhere. Either
the file system has not been unmounted cleanly or the data has not yet
propagated through the backend. In the later case, waiting for a while
should fix the problem, in the former case you should try to run fsck
on the computer where the file system has been mounted most recently.
Enter "continue" to use the outdated data anyway:
> continue
Downloading and decompressing metadata...
Reading metadata...
..objects..
..blocks..
..inodes..
..inode_blocks..
..symlink_targets..
..names..
Uncaught top-level exception:
Traceback (most recent call last):
File "/usr/bin/fsck.s3ql", line 9, in <module>
load_entry_point('s3ql==1.19', 'console_scripts', 'fsck.s3ql')()
File "/usr/lib/s3ql/s3ql/fsck.py", line 1177, in main
db = restore_metadata(tmpfh, cachepath + '.db')
File "/usr/lib/s3ql/s3ql/metadata.py", line 95, in restore_metadata
load_table(table, columns, db=db, fh=fh)
File "deltadump.pyx", line 411, in s3ql.deltadump.load_table (src/s3ql
/deltadump.c:6257)
File "deltadump.pyx", line 513, in s3ql.deltadump.load_table (src/s3ql
/deltadump.c:6123)
File "deltadump.pyx", line 162, in s3ql.deltadump.fread (src/s3ql
/deltadump.c:1887)
IOError: [Errno 14] Bad address
I am including the logs that I received when I tried to run the mount
command:
2014-09-24 12:53:18.603 [4143] MainThread: [mount] Using 8 upload threads.
2014-09-24 12:53:18.610 [4143] MainThread: [backend] Connecting to
secops-data.s3.amazonaws.com...
2014-09-24 12:53:18.610 [4143] MainThread: [backends.s3c] _do_request():
start with parameters ('GET', '/PMS/POC_s3ql_passphrase', None, None, None,
None)
2014-09-24 12:53:18.610 [4143] MainThread: [backends.s3c] _send_request():
processing request for /PMS/POC_s3ql_passphrase
2014-09-24 12:53:18.658 [4143] MainThread: [backends.s3c] _do_request():
request-id: 35F09F37A8AF9CCA
2014-09-24 12:53:18.658 [4143] MainThread: [backends.s3c] _do_request():
start with parameters ('GET', '/PMS/POC_s3ql_passphrase', None, None, None,
None)
2014-09-24 12:53:18.658 [4143] MainThread: [backends.s3c] _send_request():
processing request for /PMS/POC_s3ql_passphrase
2014-09-24 12:53:18.685 [4143] MainThread: [backends.s3c] _do_request():
request-id: ADFB2494A50DB595
2014-09-24 12:53:18.685 [4143] MainThread: [backend] Connecting to
secops-data.s3.amazonaws.com...
2014-09-24 12:53:18.685 [4143] MainThread: [backends.s3c]
list(s3ql_seq_no_): start
2014-09-24 12:53:18.685 [4143] MainThread: [backends.s3c]
list(PMS/POC_s3ql_seq_no_): requesting with marker=
2014-09-24 12:53:18.686 [4143] MainThread: [backends.s3c] _do_request():
start with parameters ('GET', '/', None, {'marker': '', 'prefix':
'PMS/POC_s3ql_seq_no_', 'max-keys': 1000}, None, None)
2014-09-24 12:53:18.686 [4143] MainThread: [backends.s3c] _send_request():
processing request for /?marker=&prefix=PMS%2FPOC_s3ql_seq_no_&max-keys=1000
2014-09-24 12:53:18.828 [4143] MainThread: [backends.s3c] _do_request():
request-id: F8FF5D95E1E342E5
2014-09-24 12:53:18.829 [4143] MainThread: [backends.s3c]
lookup(s3ql_seq_no_18)
2014-09-24 12:53:18.829 [4143] MainThread: [backends.s3c] _do_request():
start with parameters ('HEAD', '/PMS/POC_s3ql_seq_no_18', None, None, None,
None)
2014-09-24 12:53:18.829 [4143] MainThread: [backends.s3c] _send_request():
processing request for /PMS/POC_s3ql_seq_no_18
2014-09-24 12:53:18.853 [4143] MainThread: [backends.s3c] _do_request():
request-id: 58EF7E317DC83C36
2014-09-24 12:53:18.853 [4143] MainThread: [backends.s3c]
lookup(s3ql_seq_no_18)
2014-09-24 12:53:18.853 [4143] MainThread: [backends.s3c] _do_request():
start with parameters ('HEAD', '/PMS/POC_s3ql_seq_no_18', None, None, None,
None)
2014-09-24 12:53:18.853 [4143] MainThread: [backends.s3c] _send_request():
processing request for /PMS/POC_s3ql_seq_no_18
2014-09-24 12:53:18.875 [4143] MainThread: [backends.s3c] _do_request():
request-id: C03AF8D52D9B90C8
2014-09-24 12:53:18.875 [4143] MainThread: [backends.s3c]
lookup(s3ql_seq_no_19)
2014-09-24 12:53:18.875 [4143] MainThread: [backends.s3c] _do_request():
start with parameters ('HEAD', '/PMS/POC_s3ql_seq_no_19', None, None, None,
None)
2014-09-24 12:53:18.876 [4143] MainThread: [backends.s3c] _send_request():
processing request for /PMS/POC_s3ql_seq_no_19
2014-09-24 12:53:18.900 [4143] MainThread: [backends.s3c] _do_request():
request-id: 607C738A1B12EB0E
2014-09-24 12:53:18.900 [4143] MainThread: [mount] Ignoring locally cached
metadata (outdated).
2014-09-24 12:53:18.900 [4143] MainThread: [backends.s3c]
lookup(s3ql_metadata)
2014-09-24 12:53:18.901 [4143] MainThread: [backends.s3c] _do_request():
start with parameters ('HEAD', '/PMS/POC_s3ql_metadata', None, None, None,
None)
2014-09-24 12:53:18.901 [4143] MainThread: [backends.s3c] _send_request():
processing request for /PMS/POC_s3ql_metadata
2014-09-24 12:53:18.983 [4143] MainThread: [backends.s3c] _do_request():
request-id: 7353C5A55662EA0B
2014-09-24 12:53:18.983 [4143] MainThread: [root] Backend reports that fs
is still mounted elsewhere, aborting.
Took a look on Google, and didn't see any similar issue like this that's
been encountered before. If it's possibly an S3 bucket - side issue
(corrupt or missing data), that's fine, I'd just like to know what exactly
is going on. Thanks in advance to everyone who takes a look and can offer
help...
- Alfonso
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
