Hello Janos,

nice to hear, that the development goes on for the future requests and "ideas".
I use piler since 5 Yeras - and i can say - it does what it should do.

Only one thing: I still have the problem to delete some special mails of the 
archive out of the web gui. I dont know how to do - there is no option, no flag 
to set.


-----Urspr√ľngliche Nachricht-----
Von: Janos SUTO [mailto:s...@acts.hu]
Gesendet: Montag, 17. Oktober 2016 22:37
An: Piler User
Betreff: piler 1.2.0 is out!

Dear piler users,

I've just released the latest stable version of piler. Actually it has been 
released a while ago, I just had no time to write this email.

Notice that it's 1.2.x, and not 1.1.x, that is there're some minor 
incompatibilities you must be aware of. I've compiled a RELEASE_NOTES file 
which describes some of the changes.

The most important change is that I've moved all piler related configs to 
${sysconfdir}/piler directory (with the default options it's 

It means that whatever you had in /usr/local/etc, that must be moved to 
/usr/local/etc/piler, eg. /usr/local/etc/piler.conf -> 
/usr/local/etc/piler/piler.conf, etc.

I've decided to put the sphinx config file to  ${sysconfdir}/piler.
Debian and Ubuntu ship
a sphinx package which enabled a periodic indexer --all cron job, which 
practically destroys the sphinx indices, and despite both the install docs and 
the FAQ warn about it, many piler users fell for this debian 'trick'.

To match the new path, I've updated the rc.searchd file, and the indexer shell 
scripts as well.

If you upgraded, then be sure to run the util/db-upgrade-1.1.0-vs-1.2.0.sql 
script. If you have questions about the upgrade procedure, don't hesitate to 
ask. I recommend you to run pilerconf after the upgrade, and check if you get 
the values in piler.conf back.
If so, then the config
files are at the proper new location.

What next? I have three interesting topics in my head. One of them is high 
Currently your most basic option is to setup two archives (even in different 
locations), and have your mail server send copies of emails to both archives. 
Then you have two independent archives with the same content. Either of them 
goes down, your archived emails are still accessible.

However it's not that elegant, and while this approach may work out for you, it 
can be improved. Mysql supports a cluster mode. Sphinx data can be replicated 
easily (think about rsync), however replicating the stored millions of files is 
not that easy. I've seen some replicating object stores, eg. swift from 
openstack or ambry from linkedin. I think they could be used to replicate the 
stored encrypted and gzipped files.

Another idea in my head is zstandard. It's facebook's new compressing algorithm 
which outperforms gzip in every way. Fortunately it can read gzipped data 
(=your already stored emails will be readable in the future), and new emails 
can be compressed with zstandard's new algorithm offering better speed and 

The 3rd thing in my head is a non forking version of piler. An o365 user 
reported a problem that he got lots of NDRs of undeliverable emails. It turned 
out that o365 has no means of flow control, so in case of a spike in the email 
volume the default
10 piler workers are not enough to handle the emails delivered in paralel. 
After a trial and error approach it required 40 piler workers to serve the load.

A non forking piler smtp server would solve the problem by only receiving the 
emails, and amazingly fast. With such a processing model it can receive 100 or 
even more smtp sessions simultaneously very effectively. Then we need a few 
workers that actually processing the stored emails, ie. parsing, indexing, 
encrypting and storing.

I'm investigating the poll() mechanism at the moment. I've been told to use 
epoll, because it's much more efficient than select() or poll(), however epoll 
is Linux only.
So if anyone used piler on freebsd, solaris, etc. other than Linux, then it 
would be a problem. So before picking either poll or epoll, let me know if 
someone uses piler on a Unix flavor other than Linux.

Let me know what you think.



Reply via email to