That appears to have taken care of it. Thanks,
Jeff On Fri, Apr 25, 2014 at 2:23 PM, Janos SUTO <[email protected]> wrote: > > I've just introduced this fix. Please download the latest master branch, > and upgrade piler, then let me know if it solves the duplicate issue. > > Janos > > > > On 2014-04-25 19:26, Janos SUTO wrote: >> >> Hello Jeff, >> >> On 2014-04-25 17:25, Jeff Larsen wrote: >>> >>> Whenever I do a search in mail-piler, I see duplicates for most messages. >>> We have 5 mail domains: Headquarters (domain.com), plus 4 subdomains >>> (a.domain.com, b.domain.com, etc.) >>> All 5 mail servers have always_bcc = [email protected] which is >>> delivered to mail-piler. However, the subdomains route their mail to >>> the outside world via the parent domain mail server, so it is passing >>> through 2 servers that have always_bcc enabled. >>> Looking at the headers of the archived messages, I see one that has >>> Recieved: ... for <[email protected]> and the duplicate has >>> Received: ... for <[email protected]>. Both messages have the same >>> Message-Id. >>> Now, this could be a postfix deficiency as we are on a rather old >>> postfix version (upgrades are in the works), but I would think that >>> mail-piler could be smart enough to filter out duplicates with the >>> same Message-Id. >>> Is there anything I can reconfigure to address this issue? >> >> >> yes. The problem lies at 2 levels: >> >> #1: your postfix configuration causes some messages to always_bcc'ed >> twice to piler >> >> #2: piler fails to identifies duplicates >> >> >> The solution can be to fix #1 and/or #2. If it's possible, it's worth to >> reconfigure your postfix layout to eliminate multiple always_bcc for the >> very same message. Again it's not a postfix issue, it's your local >> "speciality". >> >> Regarding piler: it has deduplication detection to prevent the same >> message-id >> to get into the archive multiple times. To do that piler inserts a record >> to the >> metadata table (this is where the message-id is stored). The problem is >> that >> (currently) this sql insert takes place at the end of the message >> processing >> to ensure that all previous stuff is in its place. This whole process >> takes >> some time, and if a duplicate arrives before this final step is completed, >> then because the sql table scheme allows it, a duplicate record may be >> created. >> >> So the solution on the piler side can be either to fix the sql scheme and >> make >> the message_id column unqiue, so the second insert will fail. Another >> possible >> solution can be creating a hash value of the message_id (eg. >> "msg_id_hash") >> during the parsing, then immediately create a lock file for that >> message-id hash. >> >> This value is removed automatically after processing the message. >> Then if two dups >> arrive at the same time, only one of them can create the "lockfile" >> (this lockfile >> is a mere 0-byte length file). >> >> This lock file workaround has the advantage of that we can keep the >> current message >> processing workflow. >> >> >> Best regards, >> Janos > >
