Hello,
On 2013-05-06 07:11, Hoang Tuan Viet wrote:
1. I have not heard about Piler scalability yet.
Could I store 1 billion emails with Piler? How about the performance
of Piler and MySQL?
I have no experience with such a load, however I think that a billion
emails should be distributed among several piler hosts. Note that the
storage requirements depend greatly on your specific emails (average
message size, how much dedup and compression can help, number of users,
etc.)
The demo site has 1.26 million emails (from ubuntu-bugs@ and
linux-kernel@
lists + 500k enron test emails). They occupy 6.5 GB on the disk + 4.2
GB
sphinx index files + 3.7 GB mysql files ~= 15 GB total, ~12 GB / one
million emails. It yields to 12 TB for 1 billion emails.
Note that these emails are usually small (average 6kB) with few
attachments
only, and thus compression and dedup can help little. For real world
scenarios,
I'd count with 30-40 TB disk space for a billion emails.
Sphinx supports distributed indices, so you can spread the load among
several
hosts. It's also a good practice to create several index entries for
sphinx
even if you use a single computer only. On my low end desktop pc where
the demo
resides, it takes 7-8 minutes until the delta index is merged to the
1.3 GB big
main index.
For mysql you have to store 1 billion entries in the metadata table,
5-6 billion
entries in the rcpt table, and cca. 200 million entries in the
attachment table.
Even though mysql supports partitioning, and other fancy features, I
would definitely
use a mysql cluster to store that much data.
From 0.1.23 piler supports a server_id variable to allow you combining
several (<255)
piler hosts into a single view, so you may decide to put only 100
million emails to 1
node.
End users access piler via the web based gui, and you are allowed to
create a machine
solely for the purpose of serving web clients only (having no piler, no
mysql, no
sphinx, just a usual LAMP installation). If your users use the archive
heavily then
you may install more than 1 LAMP computers to act as the gui only.
I didn't mention any redundancy (other than the mysql cluster), but you
should definitely
have some sort of protection against losing a node. It depends on your
environment. Eg.
if you have a redundant storage your nodes may use, then you are
protected from data loss,
so if a node is down, you may replace it easily.
Note that piler also supports exporting your emails to a secondary
storage. It's also a good
practice to use the pilerexport utility to dump all the emails of the
previous day to a
secondary storage, eg. a tape or cheap sata raid array, etc.
Perhaps the other list members have much greater piler installations,
and they share their
experiences.
2. I have checked that there are number of files in piler/store that
is close to number of emails. Do you store 1 email in 1 file?
Yes. If a message has attachments, then they are stored as separate
files,
eg. 1 email with two attached files are 3 files under the store
directory.
Janos
