Re: [arch-general] ML is being sent to Spam by Gmail

[Ilya Boka via arch-general]

For me it's fixed now :)

On Mon, Jul 4, 2016 at 7:40 PM, Florian Pritz via arch-general
 wrote:
> On 16.06.2016 03:46, Eli Schwartz via arch-general wrote:
>> On 06/08/2016 03:45 PM, Florian Pritz via arch-general wrote:
>>> Now that is something I totally missed when I read their guidelines, but
>>> it certainly sounds like it could be the cause of these problems. I have
>>> just switched the server to deliver mails to gmail.com (only that
>>> domain, no others) via IPv4. Please tell me if you see any improvement
>>> during the next few days.
>>
>> So far so good. A week later and I haven't seen any more Arch ML emails
>> going to spam.
>
> Since we didn't have too much traffic on the list in that week, what's
> the situation like now? Have you seen any more list traffic go to spam
> or is this fixed now?
>
> Florian
>
>

Re: [arch-general] ML is being sent to Spam by Gmail

[Florian Pritz via arch-general]

On 16.06.2016 03:46, Eli Schwartz via arch-general wrote:
> On 06/08/2016 03:45 PM, Florian Pritz via arch-general wrote:
>> Now that is something I totally missed when I read their guidelines, but
>> it certainly sounds like it could be the cause of these problems. I have
>> just switched the server to deliver mails to gmail.com (only that
>> domain, no others) via IPv4. Please tell me if you see any improvement
>> during the next few days.
> 
> So far so good. A week later and I haven't seen any more Arch ML emails
> going to spam.

Since we didn't have too much traffic on the list in that week, what's
the situation like now? Have you seen any more list traffic go to spam
or is this fixed now?

Florian




signature.asc
Description: OpenPGP digital signature

Re: [arch-general] ML is being sent to Spam by Gmail

[Eli Schwartz via arch-general]

On 06/08/2016 03:45 PM, Florian Pritz via arch-general wrote:
> Now that is something I totally missed when I read their guidelines, but
> it certainly sounds like it could be the cause of these problems. I have
> just switched the server to deliver mails to gmail.com (only that
> domain, no others) via IPv4. Please tell me if you see any improvement
> during the next few days.

So far so good. A week later and I haven't seen any more Arch ML emails
going to spam.

Thanks! :)

-- 
Eli Schwartz

Re: [arch-general] ML is being sent to Spam by Gmail

[Uwe Koloska]

Am 08.06.2016 um 06:26 schrieb Eli Schwartz via arch-general:
> Recently a (disproportionately) large number of emails from the various
> Arch Linux mailing lists are getting rerouted to my Spam folder in
> Gmail.
> 
> Is anyone else seeing this behavior change?

I don't use a gmail address for arch mailinglists, but I see two problems:

1. Problems with DNS proxy (e.g. FritzBox)
--

The Tunderbird DKIM Checker plugin says:
  Ungültig (Kein DKIM Schlüssel im DNS Server gefunden)
  Invalid (No DKIM key found in DNS server)

AFAIK this is because the FritzBox doesn't return the TXT entry:

  dig luna2._domainkey.archlinux.org TXT

gives

;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 29290
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

while asking my local DNS proxy server

but with explicitly giving the DNS-server

  dig @8.8.8.8 luna2._domainkey.archlinux.org TXT

the correct record is returned


2. Additional Google DKIM header


Mails from gmail sender addresses have an additional Google DKIM header:

  X-Google-DKIM-Signature:

and maybe this is checked by gmail and considered when calculating the
spam level.


Regards
Uwe

Re: [arch-general] ML is being sent to Spam by Gmail

[Florian Pritz via arch-general]

On 08.06.2016 17:18, Eli Schwartz via arch-general wrote:
> In one, I see:
> Authentication-Results: luna.archlinux.org;
> dkim=fail reason="signature verification failed" (2048-bit key)
> header.d=dray-be.20150623.gappssmtp.com
> header.i=@dray-be.20150623.gappssmtp.com header.b=lZvs/tYM

That's weird, but it's difficult to tell what part of the mail changed.
I'm also not sure when that header is being generated. Opendkim doesn't
seem to verify mails consistently for me and so far I haven't managed to
figure out why because debugging features also don't seem to work properly.

> In another, I see the line:
> Authentication-Results: luna.archlinux.org; dkim=none

Well, that's entirely fine. Not everyone supports DKIM.

> I noticed that. It is slightly frustrating since most things seem to
> come from the mailing list rather than the original sender...

That is unfortunate indeed, but there is really no good solutions to
this apart from not changing the mail in any way which is also not a
good solution for us given people rely on the way our lists behave
(subject prefix, reply-to).

Mailman has an option to wrap messages into a new message which might
fix this problem, but I'd first have to investigate client support and
test that feature.

As I posted in another part of this thread, I've switched gmail.com
delivery to IPv4. Let's see if that changes anything.

Florian



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] ML is being sent to Spam by Gmail

[Florian Pritz via arch-general]

On 08.06.2016 11:18, Maarten de Vries wrote:
> ​Gmail imposes more strict checks on email coming in over IPv6, with​
> the rationale that IPv6 enabled machines are more modern and thus should
> be configured properly for newer verification techniques.

Now that is something I totally missed when I read their guidelines, but
it certainly sounds like it could be the cause of these problems. I have
just switched the server to deliver mails to gmail.com (only that
domain, no others) via IPv4. Please tell me if you see any improvement
during the next few days.



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] ML is being sent to Spam by Gmail

[Ben Oliver via arch-general]

I have been meaning to write here about this issue too, 70% of ML stuff has
been going straight to spam.

A way to quickly create filters is to click the little drop down arrow at
the top
of the message, next to the 'to Foo'. Next to the 'mailing list:' title,
there's a button
that says 'filter messages from this mailing list'.

Re: [arch-general] ML is being sent to Spam by Gmail

[Eli Schwartz via arch-general]

On 06/08/2016 04:55 AM, Florian Pritz via arch-general wrote:
> On 08.06.2016 06:26, Eli Schwartz via arch-general wrote:
>> dkim header remarks indicate either failed or missing dkim sigs for
>> those messages.
> 
> That's weird. For me the signatures very just fine. Could you show me
> the exact error you get (assuming there is one)?


In one, I see:
Authentication-Results: luna.archlinux.org;
dkim=fail reason="signature verification failed" (2048-bit key)
header.d=dray-be.20150623.gappssmtp.com
header.i=@dray-be.20150623.gappssmtp.com header.b=lZvs/tYM

In another, I see the line:
Authentication-Results: luna.archlinux.org; dkim=none

> Maybe also run the full mail (source, including headers) through
> `opendkim-testmsg` (part of the opendkim package)? If there is no error,
> the mail verifies fine.

I did try that, by saving the email(s) from Thunderbird. It matches what
I'd expect from the Authentication-Results header.

[eschwartz@arch ~]$ cat /tmp/Re\:\ \[aur-general\]\ afraiddns-git\
PKGBUILD\:\ review\ and\ criticism\ welcome.eml | opendkim-testmsg
opendkim-testmsg: dkim_eom(): Bad signature

[eschwartz@arch ~]$ cat /tmp/\[aur-general\]\ Updating\ MATE\
Packages.eml |opendkim-testmsg
opendkim-testmsg: dkim_chunk(): No signature


> Also missing DKIM signatures are not our fault and FWIW a missing
> signature should not cause mails to go to spam. An invalid signature
> also shouldn't unless there is a DMARC policy for that domain that
> states so. archlinux.org currently doesn't publish a DMARC policy so the
> default of letting everything through applies.

Correlation does not imply causation. On the other hand, I know very
little about mail protocols so I am just floating suggestions and hoping
someone who knows more than I can find the answer. :)
I did not check every email.

> We do change the From address of any mail that uses DMARC though and
> resign the mail with our key so that signatures for those mails are
> valid. Since we change the From address, the DMARC policy of the
> original sender no longer matters.

I noticed that. It is slightly frustrating since most things seem to
come from the mailing list rather than the original sender...

> I don't know if gmail provides any information as to why they classify a
> specific mail as spam, but if they do, please show me. If they do not,
> please send me all the headers of one mail that has been delivered to
> spam so I can check them for possible problems.

"Why is this message in Spam? It's similar to messages that were
detected by our spam filters.  Learn more"

So, no. :(

Pastebins of the two emails I have referenced here:
http://pastebin.com/raw/v249MDTC
http://pastebin.com/raw/gW4gJtXZ

> Do the mails that go to spam have anything in common? Do they all
> contain links or are they all sent by users from a specific domain? For
> mails where we change the From address, the old address appears is added
> to CC so check that as well.
> 
> Florian
> 

Well, none of them have had the From address changed (although there are
some which do not get changed which were not sent to Spam, so I don't
know if that means anything).

And some users have sent emails that go to Spam as well as emails that
do not go to Spam.
Two users whose email went to spam use gmail.com addresses.

-- 
Eli Schwartz

Re: [arch-general] ML is being sent to Spam by Gmail

[Dragon ryu via arch-general]

also, word checking as well.
2016/06/08 18:18 "Maarten de Vries via arch-general" <
arch-general@archlinux.org>:

> On 8 June 2016 at 10:55, Florian Pritz via arch-general <
> arch-general@archlinux.org> wrote:
>
> > On 08.06.2016 06:26, Eli Schwartz via arch-general wrote:
> > > dkim header remarks indicate either failed or missing dkim sigs for
> > > those messages.
> >
> > That's weird. For me the signatures very just fine. Could you show me
> > the exact error you get (assuming there is one)?
> >
> > Maybe also run the full mail (source, including headers) through
> > `opendkim-testmsg` (part of the opendkim package)? If there is no error,
> > the mail verifies fine.
> >
> > Also missing DKIM signatures are not our fault and FWIW a missing
> > signature should not cause mails to go to spam. An invalid signature
> > also shouldn't unless there is a DMARC policy for that domain that
> > states so. archlinux.org currently doesn't publish a DMARC policy so the
> > default of letting everything through applies.
> >
> > We do change the From address of any mail that uses DMARC though and
> > resign the mail with our key so that signatures for those mails are
> > valid. Since we change the From address, the DMARC policy of the
> > original sender no longer matters.
> >
> > I don't know if gmail provides any information as to why they classify a
> > specific mail as spam, but if they do, please show me. If they do not,
> > please send me all the headers of one mail that has been delivered to
> > spam so I can check them for possible problems.
> >
>
> ​Gmail imposes more strict checks on email coming in over IPv6, with​ the
> rationale that IPv6 enabled machines are more modern and thus should be
> configured properly for newer verification techniques. Of course, for a
> mailinglist this does not fly since it does not generate the messages
> itself. See also [1], specifically "Additional guidelines for IPv6".
>
> So... A stupid but possibly pragmatic approach is to use IPv4 when relaying
> email to gmail.
>
> [1] https://support.google.com/mail/answer/81126
>

Re: [arch-general] ML is being sent to Spam by Gmail

[Maarten de Vries via arch-general]

On 8 June 2016 at 10:55, Florian Pritz via arch-general <
arch-general@archlinux.org> wrote:

> On 08.06.2016 06:26, Eli Schwartz via arch-general wrote:
> > dkim header remarks indicate either failed or missing dkim sigs for
> > those messages.
>
> That's weird. For me the signatures very just fine. Could you show me
> the exact error you get (assuming there is one)?
>
> Maybe also run the full mail (source, including headers) through
> `opendkim-testmsg` (part of the opendkim package)? If there is no error,
> the mail verifies fine.
>
> Also missing DKIM signatures are not our fault and FWIW a missing
> signature should not cause mails to go to spam. An invalid signature
> also shouldn't unless there is a DMARC policy for that domain that
> states so. archlinux.org currently doesn't publish a DMARC policy so the
> default of letting everything through applies.
>
> We do change the From address of any mail that uses DMARC though and
> resign the mail with our key so that signatures for those mails are
> valid. Since we change the From address, the DMARC policy of the
> original sender no longer matters.
>
> I don't know if gmail provides any information as to why they classify a
> specific mail as spam, but if they do, please show me. If they do not,
> please send me all the headers of one mail that has been delivered to
> spam so I can check them for possible problems.
>

​Gmail imposes more strict checks on email coming in over IPv6, with​ the
rationale that IPv6 enabled machines are more modern and thus should be
configured properly for newer verification techniques. Of course, for a
mailinglist this does not fly since it does not generate the messages
itself. See also [1], specifically "Additional guidelines for IPv6".

So... A stupid but possibly pragmatic approach is to use IPv4 when relaying
email to gmail.

[1] https://support.google.com/mail/answer/81126

Re: [arch-general] ML is being sent to Spam by Gmail

[G. Schlisio]

Am 08.06.2016 um 10:34 schrieb Pierre Neidhardt via arch-general:

> On 16-06-08 14:49:14, Syrone Wong via arch-general wrote:

>> On Wed, Jun 8, 2016 at 2:29 PM, Sri Krishna via arch-general

dear pierre, syrone and Srikrishna, please respect the new community
code of conduct and stop top posting [0].
thank you!

[0] https://wiki.archlinux.org/index.php/Code_of_conduct#Top_posting



signature.asc
Description: OpenPGP digital signature

Re: [arch-general] ML is being sent to Spam by Gmail

[Pierre Neidhardt via arch-general]

Hi,

Same issue here: For the last few weeks I got plenty of e-mails from the Arch
MLs sent to spam.

Since I've tweaked Gmail's filters it seems to be OK now.

On 16-06-08 14:49:14, Syrone Wong via arch-general wrote:
> Hi,
> 
> You can add a filter in gmail, set
> `list:"arch-general@archlinux.org"` to match the exact email address
> and click "Never send it to Spam".
> 
> Enjoy.
> Best Regards,
> Syrone Wong
> 
> 
> On Wed, Jun 8, 2016 at 2:29 PM, Sri Krishna via arch-general
>  wrote:
> > I am also seeing many more emails from this mailing list go to the spam
> > folder in GMail (a few every week).
> >
> > I'm don't know how to check the DKIM information though.
> >
> > Srikrishna Sekhar
> >
> >
> > On 8 June 2016 at 11:34, Henrik Danielsson via arch-general <
> > arch-general@archlinux.org> wrote:
> >
> >> 2016-06-08 6:26 GMT+02:00 Eli Schwartz via arch-general <
> >> arch-general@archlinux.org>:
> >> >
> >> >
> >> > dkim header remarks indicate either failed or missing dkim sigs for
> >> > those messages.
> >> >
> >> > Is anyone else seeing this behavior change?
> >> >
> >> I didn't look very long but i could only find one message from this list in
> >> my Gmail spam folder. (About a missing GPG key 5 days ago.)
> >> The DKIM result on that one was just "none", but it got a -1.3 (minimum 5
> >> needed) spam rating so I guess that's why it was shot down.
> >> At work we have had problems with regular messages (including those from
> >> Google) ending up being tagged spam though.
> >>

Re: [arch-general] ML is being sent to Spam by Gmail

[Syrone Wong via arch-general]

Hi,

You can add a filter in gmail, set
`list:"arch-general@archlinux.org"` to match the exact email address
and click "Never send it to Spam".

Enjoy.
Best Regards,
Syrone Wong


On Wed, Jun 8, 2016 at 2:29 PM, Sri Krishna via arch-general
 wrote:
> I am also seeing many more emails from this mailing list go to the spam
> folder in GMail (a few every week).
>
> I'm don't know how to check the DKIM information though.
>
> Srikrishna Sekhar
>
>
> On 8 June 2016 at 11:34, Henrik Danielsson via arch-general <
> arch-general@archlinux.org> wrote:
>
>> 2016-06-08 6:26 GMT+02:00 Eli Schwartz via arch-general <
>> arch-general@archlinux.org>:
>> >
>> >
>> > dkim header remarks indicate either failed or missing dkim sigs for
>> > those messages.
>> >
>> > Is anyone else seeing this behavior change?
>> >
>> I didn't look very long but i could only find one message from this list in
>> my Gmail spam folder. (About a missing GPG key 5 days ago.)
>> The DKIM result on that one was just "none", but it got a -1.3 (minimum 5
>> needed) spam rating so I guess that's why it was shot down.
>> At work we have had problems with regular messages (including those from
>> Google) ending up being tagged spam though.
>>

Re: [arch-general] ML is being sent to Spam by Gmail

[Sri Krishna via arch-general]

I am also seeing many more emails from this mailing list go to the spam
folder in GMail (a few every week).

I'm don't know how to check the DKIM information though.

Srikrishna Sekhar


On 8 June 2016 at 11:34, Henrik Danielsson via arch-general <
arch-general@archlinux.org> wrote:

> 2016-06-08 6:26 GMT+02:00 Eli Schwartz via arch-general <
> arch-general@archlinux.org>:
> >
> >
> > dkim header remarks indicate either failed or missing dkim sigs for
> > those messages.
> >
> > Is anyone else seeing this behavior change?
> >
> I didn't look very long but i could only find one message from this list in
> my Gmail spam folder. (About a missing GPG key 5 days ago.)
> The DKIM result on that one was just "none", but it got a -1.3 (minimum 5
> needed) spam rating so I guess that's why it was shot down.
> At work we have had problems with regular messages (including those from
> Google) ending up being tagged spam though.
>

Re: [arch-general] ML is being sent to Spam by Gmail

[Henrik Danielsson via arch-general]

2016-06-08 6:26 GMT+02:00 Eli Schwartz via arch-general <
arch-general@archlinux.org>:
>
>
> dkim header remarks indicate either failed or missing dkim sigs for
> those messages.
>
> Is anyone else seeing this behavior change?
>
I didn't look very long but i could only find one message from this list in
my Gmail spam folder. (About a missing GPG key 5 days ago.)
The DKIM result on that one was just "none", but it got a -1.3 (minimum 5
needed) spam rating so I guess that's why it was shot down.
At work we have had problems with regular messages (including those from
Google) ending up being tagged spam though.

[arch-general] ML is being sent to Spam by Gmail

[Eli Schwartz via arch-general]

Recently a (disproportionately) large number of emails from the various
Arch Linux mailing lists are getting rerouted to my Spam folder in
Gmail. It seems to have started roughly around the time of this change:
[arch-devops] Finally fixed DKIM (I hope)[1]

dkim header remarks indicate either failed or missing dkim sigs for
those messages.

Is anyone else seeing this behavior change?
@Florian Pritz, whatever you did doesn't seem to be working, in fact it
is worse. :(


[1] https://lists.archlinux.org/pipermail/arch-devops/2016-May/66.html

-- 
Eli Schwartz