I've just introduced this fix. Please download the latest master
branch,
and upgrade piler, then let me know if it solves the duplicate issue.
Janos
On 2014-04-25 19:26, Janos SUTO wrote:
Hello Jeff,
On 2014-04-25 17:25, Jeff Larsen wrote:
Whenever I do a search in mail-piler, I see duplicates for most
messages.
We have 5 mail domains: Headquarters (domain.com), plus 4 subdomains
(a.domain.com, b.domain.com, etc.)
All 5 mail servers have always_bcc = [email protected] which is
delivered to mail-piler. However, the subdomains route their mail to
the outside world via the parent domain mail server, so it is passing
through 2 servers that have always_bcc enabled.
Looking at the headers of the archived messages, I see one that has
Recieved: ... for <[email protected]> and the duplicate has
Received: ... for <[email protected]>. Both messages have the same
Message-Id.
Now, this could be a postfix deficiency as we are on a rather old
postfix version (upgrades are in the works), but I would think that
mail-piler could be smart enough to filter out duplicates with the
same Message-Id.
Is there anything I can reconfigure to address this issue?
yes. The problem lies at 2 levels:
#1: your postfix configuration causes some messages to always_bcc'ed
twice to piler
#2: piler fails to identifies duplicates
The solution can be to fix #1 and/or #2. If it's possible, it's worth
to
reconfigure your postfix layout to eliminate multiple always_bcc for
the
very same message. Again it's not a postfix issue, it's your local
"speciality".
Regarding piler: it has deduplication detection to prevent the same
message-id
to get into the archive multiple times. To do that piler inserts a
record to the
metadata table (this is where the message-id is stored). The problem
is that
(currently) this sql insert takes place at the end of the message
processing
to ensure that all previous stuff is in its place. This whole process
takes
some time, and if a duplicate arrives before this final step is
completed,
then because the sql table scheme allows it, a duplicate record may be
created.
So the solution on the piler side can be either to fix the sql scheme
and make
the message_id column unqiue, so the second insert will fail. Another
possible
solution can be creating a hash value of the message_id (eg.
"msg_id_hash")
during the parsing, then immediately create a lock file for that
message-id hash.
This value is removed automatically after processing the message.
Then if two dups
arrive at the same time, only one of them can create the "lockfile"
(this lockfile
is a mere 0-byte length file).
This lock file workaround has the advantage of that we can keep the
current message
processing workflow.
Best regards,
Janos
