Janos, I would like to put my personal experience into this question as well.
I have been running piler over 6 days now, with a daily average of 10k emails. Average message size is 0.2MB, metadata is 2k, and index size is 6k. Average size per day is 2,402.7MB. I have received 68,640 emails, 1,758 of them have been marked as duplicate. My current archive size is 16,721.1MB, which is just over 16GB. I am running a bit overkill on the virtual machine - Ubuntu 12.04.2 LTS, 8 vCPUs, 8GB RAM, 80GB mounted on the root FS, and 2.4TB on the /var FS. According to piler estimates, I will run out of space for archiving in 2 years, 8 months, the usage trend is increasing, of course. I'm sure that these numbers are typical of a single domain archive, we do have a lot of emails with attachments and several outbound only addresses. We're archiving everything that is sent or received through our Exchange server. Steve Miller -----Original Message----- From: Janos SUTO [mailto:[email protected]] Sent: Monday, May 06, 2013 4:17 AM To: [email protected] Subject: Re: Piler scalability Hello, On 2013-05-06 07:11, Hoang Tuan Viet wrote: > > 1. I have not heard about Piler scalability yet. > > Could I store 1 billion emails with Piler? How about the performance > of Piler and MySQL? I have no experience with such a load, however I think that a billion emails should be distributed among several piler hosts. Note that the storage requirements depend greatly on your specific emails (average message size, how much dedup and compression can help, number of users, etc.) The demo site has 1.26 million emails (from ubuntu-bugs@ and linux-kernel@ lists + 500k enron test emails). They occupy 6.5 GB on the disk + 4.2 GB sphinx index files + 3.7 GB mysql files ~= 15 GB total, ~12 GB / one million emails. It yields to 12 TB for 1 billion emails. Note that these emails are usually small (average 6kB) with few attachments only, and thus compression and dedup can help little. For real world scenarios, I'd count with 30-40 TB disk space for a billion emails. Sphinx supports distributed indices, so you can spread the load among several hosts. It's also a good practice to create several index entries for sphinx even if you use a single computer only. On my low end desktop pc where the demo resides, it takes 7-8 minutes until the delta index is merged to the 1.3 GB big main index. For mysql you have to store 1 billion entries in the metadata table, 5-6 billion entries in the rcpt table, and cca. 200 million entries in the attachment table. Even though mysql supports partitioning, and other fancy features, I would definitely use a mysql cluster to store that much data. From 0.1.23 piler supports a server_id variable to allow you combining several (<255) piler hosts into a single view, so you may decide to put only 100 million emails to 1 node. End users access piler via the web based gui, and you are allowed to create a machine solely for the purpose of serving web clients only (having no piler, no mysql, no sphinx, just a usual LAMP installation). If your users use the archive heavily then you may install more than 1 LAMP computers to act as the gui only. I didn't mention any redundancy (other than the mysql cluster), but you should definitely have some sort of protection against losing a node. It depends on your environment. Eg. if you have a redundant storage your nodes may use, then you are protected from data loss, so if a node is down, you may replace it easily. Note that piler also supports exporting your emails to a secondary storage. It's also a good practice to use the pilerexport utility to dump all the emails of the previous day to a secondary storage, eg. a tape or cheap sata raid array, etc. Perhaps the other list members have much greater piler installations, and they share their experiences. > 2. I have checked that there are number of files in piler/store that > is close to number of emails. Do you store 1 email in 1 file? Yes. If a message has attachments, then they are stored as separate files, eg. 1 email with two attached files are 3 files under the store directory. Janos ------------------------------------------------------------------------------------------- This email is the property of NACA and/or its relevant affiliate and may contain confidential and privileged material for the sole use of the intended recipient(s). Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive for the recipient), please contact the sender and delete all copies of the message. Any views expressed in this message are those of the individual sender, except where the sender specifies, and with authority, states them to be the views of NACA.
