Now I'm in a position where the powers that be have requested that TLS be
disabled because of inbound problems from gmail. Apparently, gmail users
who send 25mb+ files have been getting this error more frequently than I
thought.
This is an automatically generated Delivery Status Notification
THIS IS A WARNING MESSAGE ONLY.
YOU DO NOT NEED TO RESEND YOUR MESSAGE.
Delivery to the following recipient has been delayed:
ouru...@ourcharity.org
Message will be retried for 1 more day(s)
Technical details of temporary failure:
Missed upload deadline (899.99s) (state SENT_MESSAGE)
One of the major donors got this today, which raised the flag with the
directors. Makes testing really tough....
I might be able to test this for a little bit after hours this weekend.
On Thu, Aug 4, 2016 at 3:32 AM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:
> debug such a connection
>
> set debugCode to:
>
> $Con{$fh}->{mailfrom} =~ /\@gmail\.com/ && $Con{$fh}->{SIZE} > 1024000
>
> 1024000 can be larger
>
> Thomas
>
>
>
>
>
> Von: K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum: 03.08.2016 19:08
> Betreff: Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> watching the SMTP Connections GUI, it looks like google starts out pretty
> fast for the first 2mb or so, but then really slows down. Might there be
> something with memory handling on my end?
>
> after x seconds: total bytes transferred
> 10 seconds: 1,400,000 bytes
> 30 seconds: 2,600,000 bytes
> 55 seconds: 3,800,000 bytes
> 90 seconds: 5,300,000 bytes
> 160 seconds: 7,500,000 bytes
>
> Hit 1.4mb in the first 10 seconds, but then slows to a rate of about 2mb a
> minute, sometimes slower. Does this help you figure out what might be
> going on with gmail?
>
>
>
>
> On Tue, Aug 2, 2016 at 10:40 PM, K Post <nntp.p...@gmail.com> wrote:
>
> > activestate just published net::ssleay 1.77 in their repository. Doesn't
> > seem to make any difference in terms of speed. Capping out at about 2mb
> a
> > minute with TLS.
> >
> > the ssleay.dll that is in c:\perl\site\lib\auto\Net\SSLeay appears to
> have
> > been updated by the ppm. ASSP in infostats still says:
> > OpenSSL 1.0.2h
> > OpenSSL-lib 1.0.2g Mar 2016
> >
> > Is that first line my c:\openssl installation from Shining Light (I
> don't
> > know anywhere else that 1.0.2h is installed)?
> > and OpenSSL-lib is the ssleay.dll that is seen in the
> > c:\perl\sit\lib\auto\net\ssleay folder?
> >
> > Does it matter that there's also a ssleay.dll in c:\openssl that is
> surely
> > 1.0.2h?
> >
> > Still, I ask all these questions, but it's only gmail that's giving me a
> > headache. Other senders all seem fine so far, no nearly as fast as
> without
> > TLS. For example, I just sent the same 11mb file that google takes
> about 7
> > minutes to send via Outlook.com and it only took 35 seconds.
> >
> > thanks again
> >
> >
> >
> >
> >
> > On Tue, Aug 2, 2016 at 9:44 PM, K Post <nntp.p...@gmail.com> wrote:
> >
> >> scratch that Bob. I'm still closer to 1.5-2mb per minute despite the
> >> tweaks.
> >>
> >> On Tue, Aug 2, 2016 at 9:36 PM, K Post <nntp.p...@gmail.com> wrote:
> >>
> >>> Thanks Thomas, but what OpenSSL should I be using? I really don't
> think
> >>> this is the problem, but I might as well eliminate it. I've got
> >>> activestate's perl 5.20 installed and net::ssleay from the activestate
> >>> ppm. However,the OpenSSL binaries that I have (I'm talking about the
> FULL
> >>> openssl installation in c:\openssl) not the dll files that net::ssleay
> >>> >might< have, is 1.0.2h from Shiining LIght (
> >>> slproweb.com/products/Win32OpenSSL.html)
> >>>
> >>> ASSP says net::ssleay is OpenSSL 1.0.2g - apparently it hasn't been
> >>> compiled using 1.0.2h yet. That the readme from net::ssleay talks
> >>> specifically about shining light and that it's best to roll your own
> >>> worries me.
> >>>
> >>> And Bob,
> >>> Thanks for testing this out. 3MB in 25 seconds is about what I'm
> >>> generally seeing now that I've tweaked the performance settings of
> ASSP,
> >>> but without TLS, we can receive a 10mb attachment in just a few
> seconds
> >>> thanks to a fast line. Curious, if you disable TLS temporarily and
> send
> >>> yourself that same 3mb attachment from gmail, how long does it take?
> >>>
> >>>
> >>>
> >>> On Tue, Aug 2, 2016 at 2:04 PM, Thomas Eckardt <
> >>> thomas.ecka...@thockar.com> wrote:
> >>>
> >>>> >Having looked through the Net:SSLEAY readme, there's a bunch that
> >>>> suggests
> >>>> >that it's best to compile your own net:ssleay and OpenSSL on the
> same
> >>>> >machine with the same settings.
> >>>>
> >>>> This will be the case, if you use the PPM from ActiveState. Perl and
> all
> >>>> modules are compiled with the same compiler and header files.
> >>>> Net::SSLeay
> >>>> is compiled static, means it contains all required openssl code.
> >>>>
> >>>> >I'd love to find the time to give this a go,
> >>>> You'll find something better to do, than to compile this module on
> >>>> windows.
> >>>>
> >>>>
> >>>> Thomas
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Von: K Post <nntp.p...@gmail.com>
> >>>> An: ASSP development mailing list
> <assp-test@lists.sourceforge.net>
> >>>> Datum: 02.08.2016 19:42
> >>>> Betreff: Re: [Assp-test] Inbound TLS from gmail.com addresses
> /
> >>>> servers
> >>>>
> >>>>
> >>>>
> >>>> Having looked through the Net:SSLEAY readme, there's a bunch that
> >>>> suggests
> >>>> that it's best to compile your own net:ssleay and OpenSSL on the same
> >>>> machine with the same settings. I've not done that, and never have
> (nor
> >>>> do
> >>>> I have the skillset to do much more than run a simple make command).
> >>>> I'd
> >>>> love to find the time to give this a go, but what do you all think -
> >>>> could
> >>>> this be it? Why would gmail.com always be bad, but others not (that
> >>>> I've
> >>>> seen)?
> >>>>
> >>>> On Tue, Aug 2, 2016 at 1:22 PM, Thomas Eckardt
> >>>> <thomas.ecka...@thockar.com>
> >>>> wrote:
> >>>>
> >>>> > >How do you know the type of encryption that gmail is using?
> >>>> >
> >>>> > You'll find it in the Received header line written by assp.
> >>>> >
> >>>> > >I have SSLDebug set to level 3,
> >>>> >
> >>>> > This helps not much. Most of the SSL-debug output goes to NUL.
> >>>> > But if there were errors in SSL - you would see them in the
> maillog.
> >>>> >
> >>>> > >I changed EnableHighPerformace to "very high,"
> >>>> > I don't recommend to do this. This cuts the cycle time (poll/select
> >>>> wait
> >>>> > time) in the workers to a minmum. Even if assp is idle - if this is
> >>>> set,
> >>>> > it will permanently poll the sockets and will produce unwanted CPU
> >>>> > workload. I know 'EnableHighPerformace' sounds magic, but it is
> more
> >>>> > implemented to tweak exceptional environments.
> >>>> > How ever, if your host accepts this workload - it is fine.
> >>>> >
> >>>> > >Anything else I should try tweaking?
> >>>> >
> >>>> > Don't try to much. Tweak (if) one by one step. Use the
> >>>> > 'notes/confighistory.txt' - the old and new values are recoded
> there.
> >>>> >
> >>>> > I have an idea about the gmail problem. It may be the case, that
> they
> >>>> > request SSL rehandshakes more or less often depending on the used
> >>>> > certificate and/or cipher to raise the security of the connection.
> >>>> Such
> >>>> a
> >>>> > behavior would slow down the SSL speed - BUT, now the bad news,
> this
> >>>> is
> >>>> a
> >>>> > client request (made my gmail). Perl's Net::SSLeay has no easy way
> to
> >>>> > ignore these requests. The only way would be to pipe all SSL
> packest
> >>>> > through an assp routine (this is possible), which would drop the
> >>>> > renegotiation requests. Such a code will slow down ALL SSL traffic
> >>>> > dramaticaly, if written in pure perl.
> >>>> >
> >>>> > >We are using a 2048bit certificate. It's a wildcard (*.
> >>>> ourcharity.org)
> >>>> > >cert, but I don't think that has anything to do with it.
> >>>> >
> >>>> > Who knows? But to exclude this, you may use an innocent selfcert
> >>>> > certificate and key - create it with openssl - for a while.
> >>>> > BTW. assp will create such certificate and keys, if the
> 'assp/certs'
> >>>> > folder is empty at startup. :):)
> >>>> >
> >>>> > Thomas
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > Von: K Post <nntp.p...@gmail.com>
> >>>> > An: ASSP development mailing list <
> >>>> assp-test@lists.sourceforge.net>
> >>>> > Datum: 02.08.2016 18:34
> >>>> > Betreff: Re: [Assp-test] Inbound TLS from gmail.com
> addresses
> >>>> /
> >>>> > servers
> >>>> >
> >>>> >
> >>>> >
> >>>> > Thanks for chiming in Thomas with such a detailed response.
> >>>> >
> >>>> > First, when Google gives up, it gives a message like:
> >>>> >
> >>>> > Technical details of temporary failure:
> >>>> >
> >>>> > Missed upload deadline (899.97s) (state SENT_MESSAGE)
> >>>> >
> >>>> > So it's 15 minutes that it'll try to send a file for. At under 2mb
> a
> >>>> > minute, anything over about 25megs (considering overhead) will
> >>>> ultimately
> >>>> > fail. No good since lots of gmail users send us large files.
> >>>> >
> >>>> >
> >>>> > We're on a 100mbit line, both directions, but I'd happily take a
> 9.1
> >>>> mb
> >>>> > attachment sent over TLS taking 2 minutes. I suspect when i find
> out
> >>>> what
> >>>> > the problem is that it'll be MUCh faster than that.
> >>>> >
> >>>> > We are using a 2048bit certificate. It's a wildcard (*.
> >>>> ourcharity.org)
> >>>> > cert, but I don't think that has anything to do with it.
> >>>> >
> >>>> > We're using local storage on the Hypver-V host, RAID 10 with 4
> 7200rpm
> >>>> SAS
> >>>> > drives. It's not the fasted disk array, but it seems fine. I
> can't
> >>>> see
> >>>> > slow disks impacting TLS like this if non-TLS connections fly.
> >>>> >
> >>>> > The hyper-v host is a dual processor, 2.6ghz, 6 core each, 12mb
> cache.
> >>>> > I've got a total of 10 cores assigned to the ASSP guest.
> >>>> >
> >>>> > I have SSLDebug set to level 3, but I don't see anything in the
> >>>> maillog.
> >>>> > How do you know the type of encryption that gmail is using? It
> would
> >>>> be
> >>>> > nice to compare how gmail is connecting vs outlook.com which seems
> >>>> much
> >>>> > faster (though not super fast)
> >>>> >
> >>>> > I've got SSL_Version set to
> >>>> > SSLv23:!SSLv3:!SSLv2
> >>>> >
> >>>> > and
> >>>> > SSL_Cipher_List set to
> >>>> >
> >>>> >
> >>>>
> >>>>
> kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:RC4:!aNULL:!eNULL:!
> LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!kECDH:!CAMELLIA128:!IDEA:!SEED
> >>>> >
> >>>> > my unscientific test of changing the cipher list to the default
> >>>> doesn't
> >>>> > seem to make a difference.
> >>>> >
> >>>> > MinPollTime is 1, I think it always has been.
> >>>> > I changed EnableHighPerformace to "very high," changed thread cycle
> >>>> time
> >>>> > to
> >>>> > 1000, maintenance thread cycle time to 2000, and
> >>>> rebuildthreadcycletime
> >>>> to
> >>>> > 15. That definitely made a difference in the rebuild time, almost
> >>>> halving
> >>>> > it (not that I really care about that though).
> >>>> >
> >>>> > Anything else I should try tweaking? I don't care if there's high
> CPU
> >>>> > usage, we have reasonable processing power to spare.
> >>>> >
> >>>> > Thank you
> >>>> >
> >>>> > On Tue, Aug 2, 2016 at 12:02 PM, Thomas Eckardt
> >>>> > <thomas.ecka...@thockar.com>
> >>>> > wrote:
> >>>> >
> >>>> > > I just made simlar tests with my gmail account. I can't reproduce
> >>>> this
> >>>> > > behavior related to gmail.com.
> >>>> > >
> >>>> > > I've sent a 9.1MB attachment in 133 seconds. Gmail used
> >>>> SMTPS(TLSv1_2
> >>>> > > ECDHE-RSA-AES256-GCM-SHA384)- which is commonly used by many
> >>>> > > clients/servers.
> >>>> > > Sender was mail-qt0-f181.google.com ([209.85.216.181]
> >>>> > > helo=mail-qt0-f181.google.com)
> >>>> > > My line speed is 16MB/s inbound and 4MB/s outbound.
> >>>> > >
> >>>> > > I saw many faster SMTPS connections but also many slower - this
> may
> >>>> > depend
> >>>> > > on the usage of my ISP connection.
> >>>> > >
> >>>> > > 133 seconds for such a mail is acceptable (I think).
> >>>> > >
> >>>> > > SSLv2/3:!SSLv3:!SSLv2
> >>>> > > DEFAULT:!aNULL:!RC4:!MD5
> >>>> > >
> >>>> > > are my SSL settings - not very strong - I know :):)
> >>>> > >
> >>>> > > the privat key used is 2048 Bit long
> >>>> > >
> >>>> > > In front of assp is the ISP-router and a pfsense 2.3.2 with snort
> >>>> > 3.2.9.1
> >>>> > > . Snort is configured the very hard way, except the SMTP rules
> are a
> >>>> bit
> >>>> > > more weak, because I need some spam.
> >>>> > > ASSP is running on a 4 Core 6GB W2K3 enterprise with an absolute
> >>>> > uptodate
> >>>> > > ActivePerl 5.16.3 - using all Plugins, features and a replicated
> >>>> MySQL
> >>>> > > 5.6.
> >>>> > > Domain based mail routing (in- and out-bound) is done by
> hmailserver
> >>>> > > 5.6.4-B2283.
> >>>> > > All components are configured to use SSL/TLS when ever this is
> >>>> possible.
> >>>> > > For testing purposes I use a FreeBSD 10.2 with Perl 5.20 and ASSP
> -
> >>>> it
> >>>> > > runs the same way stable like the production system.
> >>>> > >
> >>>> > > You see - nothing magic, but maintenained (except the nice old
> W2K3
> >>>> -
> >>>> > but
> >>>> > > it works like a swiss made watch with an ETA 7750).
> >>>> > >
> >>>> > > I really don't know what I can do to fix up the SSL/TLS problems.
> >>>> > >
> >>>> > > Only to be complete:
> >>>> > > Backend for the mail environment and LDAP stuff is a Domino
> >>>> 9.0.1FP6.
> >>>> > > All the stuff above (and very much more) is running on a single
> >>>> VMWare
> >>>> > > vSphere 5.5 ( 8x 2.66GHz 48GB / x3650M2).
> >>>> > > Backups are done with EMC-Networker + EBR + DataDomain-VE, stored
> >>>> at a
> >>>> > > QNAP 419P+
> >>>> > >
> >>>> > > Thomas
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Von: K Post <nntp.p...@gmail.com>
> >>>> > > An: ASSP development mailing list
> >>>> <assp-test@lists.sourceforge.net>
> >>>> > > Datum: 02.08.2016 00:07
> >>>> > > Betreff: [Assp-test] Inbound TLS from gmail.com addresses
> /
> >>>> > servers
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > I originally thought that we had a problem with all TLS inbound
> >>>> email.
> >>>> > As
> >>>> > > it turns out, my conclusion appears to have been wrong.
> >>>> > >
> >>>> > >
> >>>> > > - There are some SLOW servers outside that are just plain slow
> >>>> > (nothing
> >>>> > > I can do there),
> >>>> > >
> >>>> > > - TLS seems to work reasonably fast with most inbound mail,
> >>>> though
> >>>> > > significantly slower than without TLS (5 seconds for an 11mb
> >>>> file
> >>>> > > without
> >>>> > > tls, vs 45 seconds with TLS on)
> >>>> > >
> >>>> > > - GMAIL.com inbound TLS emails are SLOW, no matter what
> settings
> >>>> I
> >>>> > > tweak
> >>>> > >
> >>>> > >
> >>>> > > With inbound gmail.com message. if I have TLS off, an 11mb
> >>>> attachment
> >>>> is
> >>>> > > delivered through ASSP in under 5 seconds. With TLS on it takes
> >>>> close
> >>>> > to
> >>>> > > 10 minutes, which gets close to gmail's limit.
> >>>> > >
> >>>> > > I've tested with Outlook.com and that same 11mb attachment comes
> in
> >>>> > > through
> >>>> > > ASSP with TLS on in about 45 seconds.
> >>>> > >
> >>>> > > Sending a 30mb attachment from gmail FAILS because it takes too
> >>>> long.
> >>>> > > gmail
> >>>> > > will try for I believe 10 minutes to send a message, then it
> quits
> >>>> and
> >>>> > > retries. After a couple tries, it sends an NDR.
> >>>> > >
> >>>> > > This is a Windows 2012 R2 server, latest ASSP dev, OpenSSL 1.0.2h
> >>>> > > installed
> >>>> > > from slproweb.com/products/Win32OpenSSL.html (though I've also
> >>>> tried
> >>>> > with
> >>>> > > the OpenSSL I downloaded a while back from the ASSP sourceforge
> >>>> site.
> >>>> > > net::ssleay 1.74 (openssl 1.0.2g). I'm almost certain that the
> >>>> OpenSSL
> >>>> > > installation is not used by ASSP, but I've not been able to get
> >>>> > > confirmation of that here.
> >>>> > >
> >>>> > > Just updated IO::Socket::SSL to 2.033.
> >>>> > > Net::SMTP:SSL 1.02.
> >>>> > >
> >>>> > > CPU usage as reported by assp is 4.78%. It's not on the fastest
> >>>> machine
> >>>> > > in
> >>>> > > the world (it's a hypver-v guest on a decent machine), but it
> seems
> >>>> > speedy
> >>>> > > enough. 24gb ram. We've got similar physical hosts running
> >>>> Exchange
> >>>> as
> >>>> > a
> >>>> > > guest without any speed issues whatsoever.
> >>>> > >
> >>>> > > Any other info I can provide to help figure this out?
> >>>> > >
> >>>> > > Disabling TLS for any gmail inbound mail isn't a feasible option,
> >>>> plus
> >>>> I
> >>>> > > don't know if it really is just google, or just the way that
> google
> >>>> > > connects which others might too...
> >>>> > >
> >>>> > > Thank you all.
> >>>> > >
> >>>> > >
> >>>> >
> >>>> >
> >>>>
> >>>>
> ------------------------------------------------------------
> ------------------
> >>>> > > _______________________________________________
> >>>> > > Assp-test mailing list
> >>>> > > Assp-test@lists.sourceforge.net
> >>>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > DISCLAIMER:
> >>>> > > *******************************************************
> >>>> > > This email and any files transmitted with it may be confidential,
> >>>> > legally
> >>>> > > privileged and protected in law and are intended solely for the
> use
> >>>> of
> >>>> > the
> >>>> > >
> >>>> > > individual to whom it is addressed.
> >>>> > > This email was multiple times scanned for viruses. There should
> be
> >>>> no
> >>>> > > known virus in this email!
> >>>> > > *******************************************************
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> >
> >>>> >
> >>>>
> >>>>
> ------------------------------------------------------------
> ------------------
> >>>> > >
> >>>> > > _______________________________________________
> >>>> > > Assp-test mailing list
> >>>> > > Assp-test@lists.sourceforge.net
> >>>> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> >>>> > >
> >>>> > >
> >>>> >
> >>>> >
> >>>>
> >>>>
> ------------------------------------------------------------
> ------------------
> >>>> > _______________________________________________
> >>>> > Assp-test mailing list
> >>>> > Assp-test@lists.sourceforge.net
> >>>> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > DISCLAIMER:
> >>>> > *******************************************************
> >>>> > This email and any files transmitted with it may be confidential,
> >>>> legally
> >>>> > privileged and protected in law and are intended solely for the use
> of
> >>>> the
> >>>> >
> >>>> > individual to whom it is addressed.
> >>>> > This email was multiple times scanned for viruses. There should be
> no
> >>>> > known virus in this email!
> >>>> > *******************************************************
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>>
> >>>>
> ------------------------------------------------------------
> ------------------
> >>>> >
> >>>> > _______________________________________________
> >>>> > Assp-test mailing list
> >>>> > Assp-test@lists.sourceforge.net
> >>>> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >>>> >
> >>>> >
> >>>>
> >>>>
> ------------------------------------------------------------
> ------------------
> >>>> _______________________________________________
> >>>> Assp-test mailing list
> >>>> Assp-test@lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/assp-test
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> DISCLAIMER:
> >>>> *******************************************************
> >>>> This email and any files transmitted with it may be confidential,
> >>>> legally
> >>>> privileged and protected in law and are intended solely for the use
> of
> >>>> the
> >>>>
> >>>> individual to whom it is addressed.
> >>>> This email was multiple times scanned for viruses. There should be no
> >>>> known virus in this email!
> >>>> *******************************************************
> >>>>
> >>>>
> >>>>
> >>>>
> ------------------------------------------------------------
> ------------------
> >>>>
> >>>> _______________________________________________
> >>>> Assp-test mailing list
> >>>> Assp-test@lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/assp-test
> >>>>
> >>>>
> >>>
> >>
> >
> ------------------------------------------------------------
> ------------------
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
>
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
------------------------------------------------------------------------------
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
