On 10/19/2014 08:00 AM, Rich B wrote:
> Hello,
>
> I'm trying to use S3QL with Google Storage, but I'm consistently
> running into a problem where blocks from the cache are not making it
> to the backend. I'm running on Ubuntu 14.04 and using S3QL from the
> PPA (version 2.11.1+dfsg-1~trusty1).
>
> My mount command looks like this:
>
> $ mount.s3ql --threads 4 --metadata-upload-interval 7200 --allow-other
> --debug --cachedir /data/s3ql-cache --cachesize 1048567
> gs://MyBucket/MyPrefix /mnt/MyBucket
>
> Here is a sample from the mount log sometime after copying 200+ files
> (0.5-3MB each) into the mount point:
>
> 2014-10-19 09:57:42.902 28314:Dummy-23 (name)s.statfs: statfs(): start
> 2014-10-19 09:57:44.365 28314:CommitThread (name)s.put: timeout, returning
> 2014-10-19 09:57:44.365 28314:CommitThread (name)s.put: waiting for
> reader..
> 2014-10-19 09:57:49.382 28314:CommitThread (name)s.put: timeout, returning
> 2014-10-19 09:57:49.382 28314:CommitThread (name)s.put: waiting for
> reader..
> 2014-10-19 09:57:52.902 28314:Dummy-20 (name)s.statfs: statfs(): start
> 2014-10-19 09:57:53.237 28314:Thread-6 (name)s.close:
> ObjectW(s3ql_data_4).close(): start
> 2014-10-19 09:57:53.238 28314:Thread-6 (name)s._do_request: preparing
> PUT /MyPrefixs3ql_data_4?None, qs=None
> 2014-10-19 09:57:53.239 28314:Thread-6 (name)s._send_request:
> _send_request(): PUT /MyPrefixs3ql_data_4
> 2014-10-19 09:57:53.351 28314:Thread-5 (name)s.close:
> ObjectW(s3ql_data_3).close(): start
> 2014-10-19 09:57:53.352 28314:Thread-5 (name)s._do_request: preparing
> PUT /MyPrefixs3ql_data_3?None, qs=None
> 2014-10-19 09:57:53.353 28314:Thread-5 (name)s._send_request:
> _send_request(): PUT /MyPrefixs3ql_data_3
> 2014-10-19 09:57:54.383 28314:CommitThread (name)s.put: timeout, returning
> 2014-10-19 09:57:54.383 28314:CommitThread (name)s.put: waiting for
> reader..
> 2014-10-19 09:57:54.702 28314:Thread-3 (name)s.close:
> ObjectW(s3ql_data_1).close(): start
> 2014-10-19 09:57:54.702 28314:Thread-3 (name)s._do_request: preparing
> PUT /MyPrefixs3ql_data_1?None, qs=None
> 2014-10-19 09:57:54.703 28314:Thread-3 (name)s._send_request:
> _send_request(): PUT /MyPrefixs3ql_data_1
> 2014-10-19 09:57:59.384 28314:CommitThread (name)s.put: timeout, returning
> 2014-10-19 09:57:59.384 28314:CommitThread (name)s.put: waiting for
> reader..
> 2014-10-19 09:57:59.793 28314:Thread-4 (name)s.close:
> ObjectW(s3ql_data_2).close(): start
> 2014-10-19 09:57:59.794 28314:Thread-4 (name)s._do_request: preparing
> PUT /MyPrefixs3ql_data_2?None, qs=None
> 2014-10-19 09:57:59.795 28314:Thread-4 (name)s._send_request:
> _send_request(): PUT /MyPrefixs3ql_data_2
> 2014-10-19 09:58:02.902 28314:Dummy-24 (name)s.statfs: statfs(): start
> 2014-10-19 09:58:03.506 28314:Thread-6 (name)s.wrapped: Encountered
> ConnectionTimedOut exception (send/recv timeout exceeded), retrying
> call to ObjectW.close for the 33-th time...
> 2014-10-19 09:58:03.631 28314:Thread-5 (name)s.wrapped: Encountered
> ConnectionTimedOut exception (send/recv timeout exceeded), retrying
> call to ObjectW.close for the 33-th time...
> 2014-10-19 09:58:04.385 28314:CommitThread (name)s.put: timeout, returning
> 2014-10-19 09:58:04.385 28314:CommitThread (name)s.put: waiting for
> reader..
> 2014-10-19 09:58:05.282 28314:Thread-3 (name)s.wrapped: Encountered
> ConnectionTimedOut exception (send/recv timeout exceeded), retrying
> call to ObjectW.close for the 33-th time...
> 2014-10-19 09:58:09.386 28314:CommitThread (name)s.put: timeout, returning
> 2014-10-19 09:58:09.386 28314:CommitThread (name)s.put: waiting for
> reader..
> 2014-10-19 09:58:10.943 28314:Thread-4 (name)s.wrapped: Encountered
> ConnectionTimedOut exception (send/recv timeout exceeded), retrying
> call to ObjectW.close for the 33-th time...
> 2014-10-19 09:58:12.902 28314:Dummy-19 (name)s.statfs: statfs(): start
> 2014-10-19 09:58:14.387 28314:CommitThread (name)s.put: timeout, returning
> 2014-10-19 09:58:14.387 28314:CommitThread (name)s.put: waiting for
> reader..
> 2014-10-19 09:58:19.388 28314:CommitThread (name)s.put: timeout, returning
> 2014-10-19 09:58:19.388 28314:CommitThread (name)s.put: waiting for
> reader..
> 2014-10-19 09:58:22.902 28314:Dummy-18 (name)s.statfs: statfs(): start
> 2014-10-19 09:58:24.389 28314:CommitThread (name)s.put: timeout, returning
> 2014-10-19 09:58:24.389 28314:CommitThread (name)s.put: waiting for
> reader..
Could you give some more context? In particular, I'd like to see the
logs from the 32st and 1st retry.
> If I manually kill mount.s3ql and umount the filesystem, then run
> fsck.s3ql, the cache does get uploaded (with some difficulty):
>
> $ fsck.s3ql --cachedir /data/s3ql-cache
> gs://MyBucket/MyPrefix
>
>
> Starting fsck of gs://MyBucket/MyPrefix
> Using cached metadata.
> Remote metadata is outdated.
> Checking DB integrity...
> Creating temporary extra indices...
> Checking lost+found...
> Checking cached objects...
> Committing block 0 of inode 143 to backend
> Committing block 0 of inode 289 to backend
> Committing block 0 of inode 212 to backend
> Committing block 0 of inode 338 to backend
> Committing block 0 of inode 334 to backend
> Committing block 0 of inode 154 to backend
> Committing block 0 of inode 350 to backend
> Committing block 0 of inode 354 to backend
> Committing block 0 of inode 224 to backend
> Committing block 0 of inode 215 to backend
> Committing block 0 of inode 221 to backend
> Committing block 0 of inode 339 to backend
> Committing block 0 of inode 278 to backend
> Committing block 0 of inode 178 to backend
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 3-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 4-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 5-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 6-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 7-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 8-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 9-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 10-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 11-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 12-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 13-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 14-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 15-th time...
> Encountered ConnectionTimedOut exception (send/recv timeout exceeded),
> retrying call to ObjectW.close for the 16-th time...
> 2014-10-19 10:45:45.960 4472:MainThread (name)s.log_error: Committing
> block 0 of inode 344 to backend
> 2014-10-19 10:45:54.484 4472:MainThread (name)s.log_error: Committing
> block 0 of inode 240 to backend
>
>
> I'm stumped. Has anyone got an idea of what might be causing my problem?
Do you have the same problem when you store data in Amazon S3 from the
same system?
Do you have the same problem when you store data in Google Storage, but
from a different system with a different internet connection?
If you have the necessary SSL-fu, you could also do a traffic capture
using Wireshark. That's gonna be a bit tricky, but it should tell us a
lot more.
Best,
-Nikolaus
--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
»Time flies like an arrow, fruit flies like a Banana.«
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to s3ql+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
