Re: Rentention-rules and pilerpurge
Hello Marina, On 2018-11-05 13:35, Frau Marina Diezler wrote: I turned on periodic purge in the backend so that calling pilerpurge is allowed to delete anything at all. We created some retention rules, for example all mails sent from our gitlab-instance on another host should be deleted after 30 days. We added some archiving rules saying that everything mared as spam or containing \*SPAM\* in the subject should be ignored and that the existing spam should be deleted via retention rule following the same identification rules and giving a retention time of 1 day. But I can still see "fresh" Spam coming into the archive and the old spam-messages still being held where they are. note that the rules (both archiving and retention) affects only new incoming emails. Already stored emails are not affected. When I started pilerpurge it deleted only about 100 messages which is way to few. The retention rules should meet thousands of messages and in addition all emails older than 3561 days should be deleted too but the amount of pilerpurge'd mails so far wasn't so big. If I try to pilertest an arbitrary message then I never get readable infos but always something like this: root@piler:/var/piler/store/00/5a1/3d/38# pilertest 40005a105109082cef0400aae4733d38.m locale: de_DE.UTF-8 build: 955 parsing... post parsing... message-id: null / 74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b from: * ()* to: * ()* reference: ** subject: ** body: ** sent: 1541417598, delivered-date: 0 hdr len: 0 body digest: d10a0fcad8f1fe6dee72f70a4d9888604ddfaf3e98b0309ebe23ecabdcd6c542 rules check: (null) folder: 0 retention period: 1856863998 attachments: direction: 0 spam: 0 NOT IN mydomains Is this supposed to be like this? no. Anything in /var/piler/store is an encrypted and compressed file, pilertest can't handle it. Instead use pilerget to rertieve it, eg. pilerget 40005a105109082cef0400aae4733d38 > /tmp/1.eml and then analyze it with pilertest: pilertest /tmp/1.eml Do you have any idea what I should do so that more messages are being purged? Especially the retention period seems suspect I think... For already archived spam, you should identify them in the metadata table, eg. check the subject field. Then set the retained column for such messages only to the past, eg. a yesterday timestamp, and pilerpurge will take care of them. Note that such a late job will be slow and pretty ineffective. As you stated above: it's best to specify both archiving and retention rules before the 1st message arrives. And another question is: Do I understand this correctly: purged mails still show up forever in the browser backend and only disappear if I rebuild the index via command line? Is there a way to find out in No. Once pilerpurge has done, it sets the 'deleted' column to 1 which would participate for the sphinx k-list (kill list) to suppress the already purged emails from the gui. Of course, rebuilding the index is another option since the reindex tool honors the deleted column. Janos
Rentention-rules and pilerpurge
Hello, about one year ago I set up a piler instance for our emails - I found out a lot by myself, but now I do need some help please. As I don't dare at the moment to update the system we still use piler 1.3.0. After installation on a debian jessie server I tried to follow all the steps and archiving everything worked fine. I imported old emails back down to the year 2001 from pst-files via pilerimport, all was good so far and archiving worked/works. As you can imagine our archive is filling up with mails, spam messages are archived as well as notifications from gitlab for example, there is a lot of overhead from other resources too. I know that we should have had these rule before but there was no time to create them (the lack of manpower stood against the urge to create an archive at all). Now we slowly come to the point where we want to create archiving and retention rules but things aren't working as I expected. I turned on periodic purge in the backend so that calling pilerpurge is allowed to delete anything at all. We created some retention rules, for example all mails sent from our gitlab-instance on another host should be deleted after 30 days. We added some archiving rules saying that everything mared as spam or containing \*SPAM\* in the subject should be ignored and that the existing spam should be deleted via retention rule following the same identification rules and giving a retention time of 1 day. But I can still see "fresh" Spam coming into the archive and the old spam-messages still being held where they are. When I started pilerpurge it deleted only about 100 messages which is way to few. The retention rules should meet thousands of messages and in addition all emails older than 3561 days should be deleted too but the amount of pilerpurge'd mails so far wasn't so big. If I try to pilertest an arbitrary message then I never get readable infos but always something like this: root@piler :/var/piler/store/00/5a1/3d/38# pilertest 40005a105109082cef0400aae4733d38.m locale: de_DE.UTF-8 build: 955 parsing... post parsing... message-id: null / 74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b from: * ()* to: * ()* reference: ** subject: ** body: ** sent: 1541417598, delivered-date: 0 hdr len: 0 body digest: d10a0fcad8f1fe6dee72f70a4d9888604ddfaf3e98b0309ebe23ecabdcd6c542 rules check: (null) folder: 0 retention period: 1856863998 attachments: direction: 0 spam: 0 NOT IN mydomains Is this supposed to be like this? Do you have any idea what I should do so that more messages are being purged? Especially the retention period seems suspect I think... And another question is: Do I understand this correctly: purged mails still show up forever in the browser backend and only disappear if I rebuild the index via command line? Is there a way to find out in which folder in the pilerstore at /var/piler/store/00/ the message is stored so I can check there against what the browser backend says? Thanks for your help, any hints are appreciated. Regards, Marina
