I'm using a similar setup. Mount ACD locally with acd_cli and then use an
s3ql volume inside the mounted ACD path. The reason for using ACD rather
than S3 is very simple: cost.
Currently, I'm using 166 GB on ACD and I expect this to grow by a factor of
2 or 3 within the next weeks. One gets unlimited storage on ACD for $60 per
year, i.e. $5 per month. $5/mo buys about 167 GB of standard S3 storage
(infrequent access or glacier would be cheaper, but not more than about 700
GB) and that's not counting any request cost. So ACD is a lot cheaper once
one has more than a few hundred GB.
Here's what I've found with acd_cli + s3ql.
1) Stability is markedly improved by running acd_cli in single-thread mode.
Without this, I had times when even the metadata backup rotation would fail
every time. Unfortunately, this cuts my bandwidth by a factor of 2 or 3.
2) S3QL is actually very good with data integrity, even when it crashes.
Multiple times, I copied 10's of GBs to my S3QL volume (it all goes into
the cache) and then the S3QL file system crashes after uploading a few GB
(see one backtrace below). However, the mount.s3ql process will keep going,
uploading all the data until all the data is uploaded (I can monitor
network traffic and the ACD web interface). Once all the data is uploaded,
I can kill mount.s3ql and run fsck.s3ql, which cleans everything up.
Sometimes a few data blocks got corrupted and so some files get moved to
lost+found. I can then copy those files again into the mounted S3QL volume,
which is very fast because thanks to deduplication only the missing data
blocks need to be uploaded. Then I can delete the file in lost+found.
3) If I run acd_cli in multi-threaded mode (default) then S3QL crashes very
quickly, but it still keeps uploading all the data as described above,
however this upload after the crash seems to be done in a single-threaded
manner, which again reduces the bandwidth. So I've decided to run acd_cli
in single-thread mode since the upload will the single-threaded after the
crash anyway and this way I can at least delay the crash for a substantial
amount of time and metadata backups will only work in single-threaded mode.
Here's the backtrace (from mount.log) of the error I usually see:
2016-10-24 17:58:46.920 7693:Thread-5 root.excepthook: Uncaught top-level
exception:
Traceback (most recent call last):
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.20-py3.5-linux-x86_64.egg/s3ql/mount.py",
line 64, in run_with_except_hook
run_old(*args, **kw)
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.20-py3.5-linux-x86_64.egg/s3ql/block_cache.py",
line 405, in _upload_loop
self._do_upload(*tmp)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.20-py3.5-linux-x86_64.egg/s3ql/block_cache.py",
line 432, in _do_upload
% obj_id).get_obj_size()
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.20-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 108, in wrapped
return method(*a, **kw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.20-py3.5-linux-x86_64.egg/s3ql/backends/common.py",
line 339, in perform_write
with self.open_write(key, metadata, is_compressed) as fh:
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.20-py3.5-linux-x86_64.egg/s3ql/backends/comprenc.py",
line 228, in open_write
fh = self.backend.open_write(key, meta_raw)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.20-py3.5-linux-x86_64.egg/s3ql/backends/local.py",
line 102, in open_write
dest = ObjectW(tmpname)
File
"/usr/local/lib/python3.5/dist-packages/s3ql-2.20-py3.5-linux-x86_64.egg/s3ql/backends/local.py",
line 304, in __init__
self.fh = open(name, 'wb', buffering=0)
OSError: [Errno 14] Bad address:
'/home/jlippuner/cloud_storage/ACD/remote/s3ql_backup/s3ql_data_/698/s3ql_data_69888#7693-140501390976768.tmp'
I wonder how hard it would be to add a native ACD interface to S3QL...
Best,
Jonas
On Saturday, October 15, 2016 at 2:16:42 PM UTC-7, Nikolaus Rath wrote:
>
> On Oct 15 2016, Mike Beaubien <[email protected] <javascript:>> wrote:
> > Hi,
> >
> > I'm doing the usual workaround to get s3ql on top of acd using unionfs
> to
> > merge a local s3ql file system in RW mode with some additional data
> files
> > stored in ACD and mounted through acd_cli in RO.
>
> I very much hope thata this is not truly a "usual" configuration. Why
> don't you use the S3 backend?
>
> >
> > It works well when it works, but occasionally s3ql will crash with an
> error
> > like:
> >
> [..]
> > buf = fh.read(9)
> > OSError: [Errno 70] Communication error on send
> >
> > It's probably just some temporary error for whatever network reason. Is
> > there anyway to get s3ql to ignore and retry on these errors?
>
> Not in practice, no. If S3QL attempts to read data from a file, and the
> operating system returns an error, then this typically means that
> something is seriosly wrong and needs immediate attention. Crashing is
> the best course of action to make sure the problem is noticed. There is
> no way for S3QL to determine that in this case the problem is caused by
> an apparently buggy acd_cli. And even if there was a way, it would not
> be feasible for S3QL to anticipate and handle them for all the system
> calls that could possibly fail.
>
> Best,
> -Nikolaus
> --
> GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
> Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
>
> »Time flies like an arrow, fruit flies like a Banana.«
>
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
