On 2014-06-15 17:27, Mistina Michal wrote:
Hello Janos.
Finally I've found following conversation, where the most part is
explained:
https://bitbucket.org/jsuto/piler/issue/195/how-can-i-backup-piler-where-is-the-body
yeah, some people have a hard time to figure out what an issue tracker
is for ;-)
So here it is (briefly) what happens with the message.
The usual way is to use the piler daemon, an smtp server to receive a
copy of emails
from the mail server, but the message flow is the same if you prefer
archiving with
the pilerimport utility.
The piler daemon writes the message to a temp file in /var/piler/tmp,
then it parses
the email, and extracts the attachments to separate files.
The next stage is deduplication. Piler checks if the message-id of the
email is already
in the metadata table. If so, then it's a duplicate, and piler discards
the message along
with logging.
Before storing the message files, piler compresses (via zlib) then
encrypts them (using the
blowfish algorithm with the help of the piler.key file, and the "iv"
string in piler.conf),
and finally writes them to the /var/piler/store/00 directory. Note that
piler relies on
the openssl libraries both to encrypt / decrypt the messages, and
provide the starttls
smtp extension. I know, openssl has given most of us a heart attack
recently, but that's
what we have at the moment. I hope the libressl project will be
available soon :-)
Piler also checks if the given attachment is already present in the
store by looking up
the attachment table, and searching for the sha256 sum of the
attachment. If there's a
match, then only a pointer is created in the attachment table, and the
given attachment
is discraded. Note that to restore a deduplicated message, you need the
attachment table.
Just a sidenote: the attachment table could have been eliminated if I
used a hard link
to the already stored attachment, and not an sql table. Too bad that it
came to my mind
yesterday. Sigh. Probably I'll fix or change this after releasing the
next stable version.
In this case I'll give you a utility to create the necessary hard
links. We'll see. Anyway
such an improvement would allow you to recreate both metadata and rcpt
tables from scratch
even if your mysql server melts down, and probably give you some safe
mode
The parser also extracts max. ~64 kB textual data from the email. In
case of some common
file types (eg. pdf, txt, doc, xls, ppt, ...) the parser runs external
helper applications.
OK, it's time to fill some sql tables with meaningful data. I described
the purpose of table
attachment, and here is another: rcpt. A single email may have several
recipients, so I store
all of them in the rcpt table. The key is the serial number of the
message piler assigns to
the given message.
And we have the metadata table which stores the some timestamps, sender
address, message size
message-id, a generated piler-id, number of attachments, some
checksums, etc. You may think of
the metadata table as the history of your emails. Whenever piler
archives a message it also
keeps tracks in the sql database.
Although piler works ok archiving emails, it's pretty unpleasant to
make a backup of the archive,
since it has zillions of subdirs and (mostly) small files. However
notice that you have dirs like
527, 528, 529, 52a, ... under /var/piler/store/00. Each of these dirs
contains ~12 days of emails,
and these values (eg. 52a) come from the timestamp, so if you have 52b,
then you can be sure that
52a, and previous dirs (ie. 529, 528, ...) won't change, so you don't
have to rsync these older
dirs all the time, you can concentrate on the actually directory being
written under /var/piler/store/00.
So to sum it up: the emails are in the filesystem
(/var/piler/store/00/...), and some essential
metadata are in the sql database. So make sure your tables are healthy
all the time.
Piler uses sphinx search to provide some searching capabilities. Sphinx
search reads the sph_index
table to index new emails. The sph_index table has similar entries as
the metadata table allowing
you to search on dates, subject, body, sender, recipient, and size.
Note that when the sphinx indexer
is done with sph_index it deletes all processed entries from it.
Every while there are some piler users who run "indexer --all" because
they think it might be a good
idea. Actually it's not. Piler uses such a sphinx setup that this
command practically resets the sphinx
indices, making the sphinx data empty. Don't worry your emails are not
affected however you have to
reindex all emails because of this false step. So unless you are told
so, do not execute "indexer --all"
other then install time.
Note that the piler daemon (or pilerimport) has nothing to do with
sphinx. Piler just writes to the sph_index
table, then forgets about it. So piler can work properly even if
searchd is not running.
Also note that the indexer process runs from crontab in every 30 mins,
and creates / updates data in the
/var/piler/sphinx directory.
When the email is processed by piler, it returns a 250 OK
400000....4784784 or something like message to the
mail server telling it that the email is archived properly. (Actually a
250 OK message means that the sending
mail server can release the given message from its mail queue, and
forget about it).
OK, we have archived and indexed the emails. But hwo do you present the
archive to the users? By a usual LAMP
server application, the gui. The gui is a web application that lets you
search in the archive. It gives you
some input fields you can type your search criteria, hit the search
button, the the gui returns you some matching
emails.
Note that the gui requires to access to both searchd (sphinx search
daemon) and the mysql server. The search query
is parsed by the gui, and a sphinx sql query is assembled to pull the
matching email ids (remember the serial "id"
column in the metadata table?). Then the gui reads the metadata
belonging to the returned unique serial ids, then
presents these data to the browser.
Of course you need a web server (be it apache, nginx, etc.) to use the
gui, and a recent php installation. 5.4 is
known to work with the piler gui.
If you have some questions left, don't hesitate to ask.
Best regards,
Janos
