Dear piler users,

I've just released the latest stable version of piler. Actually it has been released
a while ago, I just had no time to write this email.

Notice that it's 1.2.x, and not 1.1.x, that is there're some minor incompatibilities you must be aware of. I've compiled a RELEASE_NOTES file which describes some of the

The most important change is that I've moved all piler related configs to ${sysconfdir}/piler
directory (with the default options it's /usr/local/etc/piler).

It means that whatever you had in /usr/local/etc, that must be moved to /usr/local/etc/piler,
eg. /usr/local/etc/piler.conf -> /usr/local/etc/piler/piler.conf, etc.

I've decided to put the sphinx config file to ${sysconfdir}/piler. Debian and Ubuntu ship a sphinx package which enabled a periodic indexer --all cron job, which practically destroys the sphinx indices, and despite both the install docs and the FAQ warn about it, many piler
users fell for this debian 'trick'.

To match the new path, I've updated the rc.searchd file, and the indexer shell scripts as well.

If you upgraded, then be sure to run the util/db-upgrade-1.1.0-vs-1.2.0.sql script. If you have questions about the upgrade procedure, don't hesitate to ask. I recommend you to run pilerconf after the upgrade, and check if you get the values in piler.conf back. If so, then the config
files are at the proper new location.

What next? I have three interesting topics in my head. One of them is high availability. Currently your most basic option is to setup two archives (even in different locations), and have your mail server send copies of emails to both archives. Then you have two independent archives with the same content. Either of them goes down, your archived
emails are still accessible.

However it's not that elegant, and while this approach may work out for you, it can be improved. Mysql supports a cluster mode. Sphinx data can be replicated easily (think about rsync), however replicating the stored millions of files is not that easy. I've seen some replicating object stores, eg. swift from openstack or ambry from linkedin. I think they
could be used to replicate the stored encrypted and gzipped files.

Another idea in my head is zstandard. It's facebook's new compressing algorithm which outperforms gzip in every way. Fortunately it can read gzipped data (=your already stored emails will be readable in the future), and new emails can be compressed with
zstandard's new algorithm offering better speed and compression.

The 3rd thing in my head is a non forking version of piler. An o365 user reported a problem that he got lots of NDRs of undeliverable emails. It turned out that o365 has no means of flow control, so in case of a spike in the email volume the default 10 piler workers are not enough to handle the emails delivered in paralel. After a trial and error approach it required 40 piler workers to serve the load.

A non forking piler smtp server would solve the problem by only receiving the emails, and amazingly fast. With such a processing model it can receive 100 or even more smtp sessions simultaneously very effectively. Then we need a few workers that actually processing the stored emails, ie. parsing, indexing, encrypting and

I'm investigating the poll() mechanism at the moment. I've been told to use epoll, because it's much more efficient than select() or poll(), however epoll is Linux only. So if anyone used piler on freebsd, solaris, etc. other than Linux, then it would be a problem. So before picking either poll or epoll, let me know if someone
uses piler on a Unix flavor other than Linux.

Let me know what you think.


Reply via email to