Konstantin Ryabitsev <konstan...@linuxfoundation.org> wrote:
> On 02/09/18 13:17, Eric Wong wrote:
> > In addition to the git object_id (blob SHA-1) and Message-Id
> > header; it seems necessary to introduce an in-between identifier
> > for deduplicating which isn't as loose as Message-Id or as
> > strict as object_id: content_id
> > I think a hash of the following raw headers + raw body will
> > suffice:
> > Subject, From, Date, Message-Id, References, To, Cc,
> > In-Reply-To, MIME-Version, Content-Type,
> > Content-Disposition, Content-Transfer-Encoding
> That's similar to what ARC/DKIM do. E.g. in my mailbox the message has
> the following headers sealed:
Right, I checked typical DKIM signatures in case I missed some
> If we trim the arc- and list-specific headers, that's:
OK, that's another missing header I noticed from the archives I got.
> I'm not sure we should care about content-transfer-encoding, because
> that can be mangled by intermediate MTAs (at least that used to happen
> all the time in the past -- not sure if it's still the case).
MIME-Version and Content-* can be no-ops in common cases;
so maybe they're not worth hashing for content_id...
> > List-Id, X-Mailing-List should be left out so different
> > readers/lists can share spam removals in cross posts.
> Note, that mailing lists that modify the Subject header (e.g. to add
> [mailinglist] identifier) will also be impacted similarly.
Right; but I don't think Subject mangling is done for vger
lists. I know fuse-devel has it, but that's hosted on SF.net
The content_id would also get tripped up by the vger unsubscribe
signature which AFAIK only existed before 2016. I filtered that
sig out with PublicInbox::Filter::Vger for the git@vger import
and will likely do that for LKML.
There might be some crossposted messages from other lists which
have similar signatures which also get into the archives. I
know it happened when getting messages from gmane.
> > (*) I noticed the first Received: header (last hop) is missing
> > from the cregit sources; but the first remaining Received:
> > header also includes the identity of the recipient in more
> > recent mails...
> I specifically sanitized all Received: headers that didn't say "by
> vger.kernel.org" because these are donated by individual users and I
> didn't want to expose their potentially private info.
Ah, but yes, it's still there in the remaining Received. I only
noticed Willy's address and his address/involvement is hardly
secret; but there might be others I haven't noticed, yet...
I also noticed DKIM-Signature being missing, too. I'm not sure
what else was removed (along with Precedence).