Not to take anything from the marvellous s3ql, but have you considered
options like Ceph or Riak? (or openstack)
Obviously at the simplest they can be used as the endpoints for an s3ql
system, but for example Ceph has a filesystem option and integrating
either as an object store is possible?
Integrating directly allows you full control over multi-thread access
semantics. I would recommend thinking about what you need from your
webapp. You may find it beneficial to look at other solutions, if only
to gain experience?
Good luck
Ed W
On 17/10/2014 15:18, [email protected] wrote:
Hi Everyone.
We're looking at this solution as part of a web app we're building, I
wanted to ask the community what their experience is with benchmarks,
how many reads/writes can this system manage at what point will it
need to be scaled out. Although this seems to have solved all the
issues we're going to come up against, I'm worried that outgrowing it
will be a difficult process if we're moving terabytes of data in a few
years time because we need to increase the speed...
This brings me to my next question, I know Nikolaus has mentioned on a
few occasions that this doesn't scale above one mount.s3ql for writes,
although you can add a few extra read mounts through sshfs, as
mentioned here, would this improve read performance or would all reads
go still go through the one mount (ie creating a bottleneck and not
improving performance...)
Thinking about clustering and performance it seems as though the
system is built off the back of a sqlite db, would this be possible to
separate from the system thereby making it possible to have multiple
upload/download points all write to same db??
I'm just thinking aloud on this but I'd appreciate hearing anyone
else's feedback on using this system before we commit to it.
Kind regards
Giles
--
You received this message because you are subscribed to the Google
Groups "s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected]
<mailto:[email protected]>.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"s3ql" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
