Hey Nikolaus,
Thank you for responding. I did send you this issue via e-mail, but you
recommended I posted here for a wider audience, so that may be where you're
remembering it from.
Actually I figured out the issue, looked like it was corrupted metadata
backup files in AWS S3. I was able to restore one of the older files
(s3ql_metadata_bak_1) to s3ql_metadata using s3cmd, and that allowed me to
run a fsck.s3ql on the volume. After that I was able to remount the
volume. I did lose some files, but were able to recover them in the
/lost+found/ directory of the volume.
Thank you again for your response.
On Saturday, October 4, 2014 3:05:03 PM UTC-4, Alfonso Brown wrote:
>
> I'm wondering someone can help me out with an error I'm receiving on S3QL.
>
> I am running it on Ubuntu, with the S3QL PPA added, and recently (today)
> did an apt-get dist-upgrade of my instance. Included in that was updating
> s3ql to version 1.19-1~precise1. I am running Ubuntu Precise (12.04.4),
> and the backported trusty kernel (3.13.0-36-generic #63~precise1-Ubuntu
> SMP Thu Sep 4 22:28:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux). Other
> than that, there is nothing out of the ordinary installed or configured.
> This is a server in AWS.
>
> After I rebooted (also updated my kernel) my s3 filesystem failed to
> mount. I tried to run a mount manually and received the following error:
>
> ~/.s3ql # mount.s3ql --debug all --upstart --allow-other --cachesize
> 16777216 --cachedir /mnt/tmpdisk/S3QL-cache --compress none --authfile
> /root/.s3ql/authinfo2 --metadata-upload-interval 600
> s3://secops-data/PMS/POC_ /mnt/pms_store --nfs
> Using 8 upload threads.
> Ignoring locally cached metadata (outdated).
> Backend reports that fs is still mounted elsewhere, aborting.
>
> I've seen this before, so I then tried to run an fsck.s3ql. Usually that
> works for me, but now I have received the following error when running that
> script:
>
> ~/.s3ql # fsck.s3ql --force s3://secops-data/PMS/POC_
> Starting fsck of s3://secops-data/PMS/POC_
> Ignoring locally cached metadata (outdated).
> Backend reports that file system is still mounted elsewhere. Either
> the file system has not been unmounted cleanly or the data has not yet
> propagated through the backend. In the later case, waiting for a while
> should fix the problem, in the former case you should try to run fsck
> on the computer where the file system has been mounted most recently.
> Enter "continue" to use the outdated data anyway:
> > continue
> Downloading and decompressing metadata...
> Reading metadata...
> ..objects..
> ..blocks..
> ..inodes..
> ..inode_blocks..
> ..symlink_targets..
> ..names..
> Uncaught top-level exception:
> Traceback (most recent call last):
> File "/usr/bin/fsck.s3ql", line 9, in <module>
> load_entry_point('s3ql==1.19', 'console_scripts', 'fsck.s3ql')()
> File "/usr/lib/s3ql/s3ql/fsck.py", line 1177, in main
> db = restore_metadata(tmpfh, cachepath + '.db')
> File "/usr/lib/s3ql/s3ql/metadata.py", line 95, in restore_metadata
> load_table(table, columns, db=db, fh=fh)
> File "deltadump.pyx", line 411, in s3ql.deltadump.load_table (src/s3ql
> /deltadump.c:6257)
> File "deltadump.pyx", line 513, in s3ql.deltadump.load_table (src/s3ql
> /deltadump.c:6123)
> File "deltadump.pyx", line 162, in s3ql.deltadump.fread (src/s3ql
> /deltadump.c:1887)
> IOError: [Errno 14] Bad address
>
> I am including the logs that I received when I tried to run the mount
> command:
>
> 2014-09-24 12:53:18.603 [4143] MainThread: [mount] Using 8 upload threads.
> 2014-09-24 12:53:18.610 [4143] MainThread: [backend] Connecting to
> secops-data.s3.amazonaws.com...
> 2014-09-24 12:53:18.610 [4143] MainThread: [backends.s3c] _do_request():
> start with parameters ('GET', '/PMS/POC_s3ql_passphrase', None, None, None,
> None)
> 2014-09-24 12:53:18.610 [4143] MainThread: [backends.s3c] _send_request():
> processing request for /PMS/POC_s3ql_passphrase
> 2014-09-24 12:53:18.658 [4143] MainThread: [backends.s3c] _do_request():
> request-id: 35F09F37A8AF9CCA
> 2014-09-24 12:53:18.658 [4143] MainThread: [backends.s3c] _do_request():
> start with parameters ('GET', '/PMS/POC_s3ql_passphrase', None, None, None,
> None)
> 2014-09-24 12:53:18.658 [4143] MainThread: [backends.s3c] _send_request():
> processing request for /PMS/POC_s3ql_passphrase
> 2014-09-24 12:53:18.685 [4143] MainThread: [backends.s3c] _do_request():
> request-id: ADFB2494A50DB595
> 2014-09-24 12:53:18.685 [4143] MainThread: [backend] Connecting to
> secops-data.s3.amazonaws.com...
> 2014-09-24 12:53:18.685 [4143] MainThread: [backends.s3c]
> list(s3ql_seq_no_): start
> 2014-09-24 12:53:18.685 [4143] MainThread: [backends.s3c]
> list(PMS/POC_s3ql_seq_no_): requesting with marker=
> 2014-09-24 12:53:18.686 [4143] MainThread: [backends.s3c] _do_request():
> start with parameters ('GET', '/', None, {'marker': '', 'prefix':
> 'PMS/POC_s3ql_seq_no_', 'max-keys': 1000}, None, None)
> 2014-09-24 12:53:18.686 [4143] MainThread: [backends.s3c] _send_request():
> processing request for /?marker=&prefix=PMS%2FPOC_s3ql_seq_no_&max-keys=1000
> 2014-09-24 12:53:18.828 [4143] MainThread: [backends.s3c] _do_request():
> request-id: F8FF5D95E1E342E5
> 2014-09-24 12:53:18.829 [4143] MainThread: [backends.s3c]
> lookup(s3ql_seq_no_18)
> 2014-09-24 12:53:18.829 [4143] MainThread: [backends.s3c] _do_request():
> start with parameters ('HEAD', '/PMS/POC_s3ql_seq_no_18', None, None, None,
> None)
> 2014-09-24 12:53:18.829 [4143] MainThread: [backends.s3c] _send_request():
> processing request for /PMS/POC_s3ql_seq_no_18
> 2014-09-24 12:53:18.853 [4143] MainThread: [backends.s3c] _do_request():
> request-id: 58EF7E317DC83C36
> 2014-09-24 12:53:18.853 [4143] MainThread: [backends.s3c]
> lookup(s3ql_seq_no_18)
> 2014-09-24 12:53:18.853 [4143] MainThread: [backends.s3c] _do_request():
> start with parameters ('HEAD', '/PMS/POC_s3ql_seq_no_18', None, None, None,
> None)
> 2014-09-24 12:53:18.853 [4143] MainThread: [backends.s3c] _send_request():
> processing request for /PMS/POC_s3ql_seq_no_18
> 2014-09-24 12:53:18.875 [4143] MainThread: [backends.s3c] _do_request():
> request-id: C03AF8D52D9B90C8
> 2014-09-24 12:53:18.875 [4143] MainThread: [backends.s3c]
> lookup(s3ql_seq_no_19)
> 2014-09-24 12:53:18.875 [4143] MainThread: [backends.s3c] _do_request():
> start with parameters ('HEAD', '/PMS/POC_s3ql_seq_no_19', None, None, None,
> None)
> 2014-09-24 12:53:18.876 [4143] MainThread: [backends.s3c] _send_request():
> processing request for /PMS/POC_s3ql_seq_no_19
> 2014-09-24 12:53:18.900 [4143] MainThread: [backends.s3c] _do_request():
> request-id: 607C738A1B12EB0E
> 2014-09-24 12:53:18.900 [4143] MainThread: [mount] Ignoring locally cached
> metadata (outdated).
> 2014-09-24 12:53:18.900 [4143] MainThread: [backends.s3c]
> lookup(s3ql_metadata)
> 2014-09-24 12:53:18.901 [4143] MainThread: [backends.s3c] _do_request():
> start with parameters ('HEAD', '/PMS/POC_s3ql_metadata', None, None, None,
> None)
> 2014-09-24 12:53:18.901 [4143] MainThread: [backends.s3c] _send_request():
> processing request for /PMS/POC_s3ql_metadata
> 2014-09-24 12:53:18.983 [4143] MainThread: [backends.s3c] _do_request():
> request-id: 7353C5A55662EA0B
> 2014-09-24 12:53:18.983 [4143] MainThread: [root] Backend reports that fs
> is still mounted elsewhere, aborting.
>
> Took a look on Google, and didn't see any similar issue like this that's
> been encountered before. If it's possibly an S3 bucket - side issue
> (corrupt or missing data), that's fine, I'd just like to know what exactly
> is going on. Thanks in advance to everyone who takes a look and can offer
> help...
>
> - Alfonso
>
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
