very good news, and some not so good news with 16273. I watched this same 11MB attachment come across using the Shutdown_list (about 15,000,000 bytes after all said and done). It transferred in about 40 seconds. As in OUTSTANDING SPEED. As in, whatever you did has made a HUGE HUGE difference. (looking at the code, nice work. clever)
HOWEVER, I had to revert as the email never seemed to complete. *It then just sat with the idle / damping time increasing indefinitely. * All SSL connections did this too, fast transfer,then idle forever. I didn't have the time to test non-SSL connections. I tried restarting for safe measure just to be sure. Same problem. Once I went back to 16271, normal behavior resumed. Seems close, but not quite right.... On Thu, Sep 29, 2016 at 10:25 AM, cw <colin.war...@gmail.com> wrote: > Hi Thomas, > I moved up to 16270 following this thread of discussion but then had a day > working away. I've come back to huge issues with delays, mails not going > through and many, many of these in the logs: > > Info: unable to detect any running worker for a new connection - wait (max > 30 seconds) > > When I say many, I have over 21,000 lines in today's log file. I also found > the GUI unresponsive or not connecting at all and ASSP restarting quite > regularly. > > I've dropped back to 16256 and things are instantly better. Do you think > going up to 16273 might improve things over 16270 or am I better holding > off for now? > All the best, > Colin. > > On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt < > thomas.ecka...@thockar.com> > wrote: > > > I just released 2.5.2 build 16273 at CVS test folder > > > > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/ > > > > This release should make a very large difference for SSL/TLS mails sent > by > > hosts that uses small SSL-frame size. > > > > Tell me your test results. > > > > > > Thomas > > > > > > > > > > > > Von: K Post <nntp.p...@gmail.com> > > An: ASSP development mailing list <assp-test@lists.sourceforge.net> > > Datum: 28.09.2016 19:42 > > Betreff: Re: [Assp-test] Inbound TLS from gmail.com addresses / > > servers > > > > > > > > But I want a postman driving a Ferarri with monster truck tires that can > > roll over the traffic (and if wishes are being granted, I'd prefer the > car > > in a deep blue instead of classic red). > > > > We regularly see people attaching large files or a bunch of smaller ones > > that add up to a big email, I'm talking lots and lots of different people > > from outside the organization sending to us, and this happens on a daily > > basis. It's especially popular with photos and huge scans multi-page > > 600dpi (which people don't understand can be done at low resolution). > > Often it's people sending in scanned official documents for us to review > > an > > help them. They're not our staff, they're the people we help. They have > > a > > tendency of not following any instructions, and ignore the fact that we > > have a web based system for the process. We can't control it and the > > powers that be don't want us lowering the 30 MB threshold across the > > board. Lot of these people use gmail.com addresses and google allows > for > > up to 25 MB - https://support.google.com/mail/answer/6584 > > > > I think it's really interesting that google seems to use this inefficient > > small packet size for SSL, allows for 25MB emails, is a big proponent of > > SSL, and at the same time doesn't allow mails to take more than 15 > minutes > > to transfer. Now that you've made things >much< more efficient on the > > ASSP > > side, I'm hoping that all will be okay. I just get annoyed by > > inefficiency. > > > > > > I'm going to tryrunning with npSize of zero, the no queuing size set very > > high and see how that goes. I want to insure that even the biggest > emails > > are scanned for malware before hitting the inbox. > > > > > > I've never said this before, but it seems like google's doing it wrong. > I > > think we've exhausted our options here. If anyone has a Google > > engineering > > friend, or a friend of a friend, this might be a good time to have a > > little > > chat in an effort to reach someone who can either explain why they're use > > a > > 1.4 kB frame size for ssl packets, but a normal much bigger size for > > non-encrypted traffic or to get them to see an error in their ways and > fix > > this! > > > > Thank you again for all of the discussion and explanation and of course > > the > > code changes which have made a huge difference. > > > > On Wed, Sep 28, 2016 at 12:59 PM, Thomas Eckardt > > <thomas.ecka...@thockar.com > > > wrote: > > > > > >I'm still afraid of running into frequent problems with 25mb > > attachments > > > though. > > > > > > 25MB - I don't know anyone who allows this. Sending a link to a > download > > > is much smarter. > > > ASSP_AFC has an option to do this for your outgoing mails! - > > > ASSP_AFCWebScript > > > > > > >Can someone else run a debug test with Gmail and TLS to see if you're > > > seeing these tiny packets too? > > > > > > I've done it - same behavior - 1400 byte per SSL frame from gmail. > > > > > > > > > >Would it be an easy modification to show the negotiated cipher in the > > > > > > ASSP does this for years now - simply look in to the received header > > line > > > added by assp. > > > > > > -- -- -- Received: from vs10241.internet1.de ([158.181.49.77] > > > helo=vs10241.internet1.de) > > > by xxxxxx with SMTPS(TLSv1_2 ECDHE-RSA-AES256-GCM-SHA384) > > (2.5.2); > > > 24 Jul 2016 05:34:12 +0200 > > > > > > if you set 'ConnectionLog' at least to verbose, you'll see the > > > SSL-negotiation results in maillog.txt. > > > > > > info: started TLS-SSL session for client $cliIP - using > > > $Con{$ssls}->{sslversion} , $Con{$ssls}->{sslcipher} > > > > > > >what other options are there? > > > > > > Set 'EnableHighPerformance' to very high. > > > > > > >Are you sure that negotiating a lesser cipher with Gmail wouldn't have > > > them switch to sending larger SSL packets? > > > > > > Yes, I'm 100% sure. > > > > > > An SSL peer is free to use any SSL frame size between 1 byte and 16kB. > > > There is no SSL-negotiation parameter for min or max frame size. > > > If gmail.com sends 1.4kB frames, they are free to do it this way. > There > > is > > > no technical way to force them to send larger frames. > > > > > > >If neverQueue is set to 12MB, am I correct in saying that we're open > to > > a > > > targeted exploit, say an .exe in a .zip as long as the email is more > > than > > > 12MB? > > > > > > Yes, you are correct. > > > > > > >or is that just after the whole message is received? > > > > > > Yes, after the complete mail is received. > > > > > > It is like it is - if the postman with his Ferrari is in a traffic jam, > > > you'll have to wait longer for your letter or you'll get it the next > > day. > > > > > > > > > Thomas > > > > > > > > > > > > Von: K Post <nntp.p...@gmail.com> > > > An: ASSP development mailing list <assp-test@lists.sourceforge.net > > > > > Datum: 28.09.2016 17:03 > > > Betreff: Re: [Assp-test] Inbound TLS from gmail.com addresses / > > > servers > > > > > > > > > > > > Thanks again for the detailed explanation. > > > > > > 10% faster is terrific, and it's more than 50% faster since before > > 16270. > > > I'm still afraid of running into frequent problems with 25mb > attachments > > > though. Yes, that's too big for email in our opinion, but Gmail allows > > it > > > as do most ISP's these days. It's not terribly unusual for us to have > a > > a > > > person try to send and email with four or five 4 MB photos - and we > > can't > > > stop that trend. That'll translate into a message somewhere in the > > range > > > of 22-25 MB. I'm not sure if we're fast enough to >consistently< > > receive > > > these. And on top of that, if it's an outside big wig, emailing an > > inside > > > big wig, they don't have 15 minutes to waste waiting for the big email > > to > > > arrive. Then they yell at me, which is why I keep asking more of these > > > questions -- and there's a lot of them here (sorry) > > > > > > > > > *Are you sure that negotiating a lesser cipher with Gmail wouldn't have > > > them switch to sending larger SSL packets? * I have a solid high level > > > understanding of encryption, but once you get down the the packet > level, > > > I'm a bit out of my depth, despite what I've learned from your > excellent > > > explanations. > > > > > > > > > Can someone else run a debug test with Gmail and TLS to see if you're > > > seeing these tiny packets too? I keep going back to *not being able to > > > believe that Google doing something that was any slower than necessary. > > * > > > They > > > try to optimize everything and are huge proponents of SSL everywhere. > If > > > they're sending tons of tiny un-optimized packets, there's got to be a > > > reason or cause. I can't imagine that it's a default and deliberate > > > setting. I tried calling US support, but this is a network engineering > > > type of issue - no chance of me getting through to someone there who > > even > > > understands the question. > > > > > > The npSize option and neverQueueSize, only affects things after the > > whole > > > message is received right? * It's not like it's doing something > > > significantly extra at each SSL cylce is it except using up RAM > right??* > > > My unscientific tests are shower marginally slower speeds when > > receiving > > > large email from Google with TLS on and neverQueueSize set crazy high, > > so > > > maybe I'm wrong. Being that this same message through outlook.com > only > > > takes 30 seconds, even with npSize set to 0 (unlimited) and > > neverQueueSize > > > = 999,000,000, I'm wondering if the processing power and ram > > availability > > > (24 GB) on my assp box combined with low usage (only averaging 5000 > > > messages a day) might be enough to have neverQueueSize really high to > > > always scan everything. I'll play around there, but if neverQueueSize > > is > > > 99MB, will things like ASSP_AFCDetectSpamAttachRe be slow WHILE the > > > message > > > is transferring, or is that just after the whole message is received? > > > > > > If neverQueue is set to 12MB, am I correct in saying that we're open to > > a > > > targeted exploit, say an .exe in a .zip as long as the email is more > > than > > > 12MB? If so, that seems like a big hole, and something that I > > personally > > > would be willing to sacrifice speed/processing power to close. > > > > > > Suggestion, note in the gui under npSize that 0 really sets this to a > > > maximum of 12,000,000 unless neverQueueSize is set higher in > > > CorrectASSPcfg.pm. I've got that correct, right?) > > > > > > I forgot that DKIM signing can be done over the body too! We don't do > > > that, but I forgot that others might. Maybe we have npSize only ignore > > > DKIM if it's done over the body? DKIM should be fast if it's just the > > > typical headers that are used right? We have npSize set to zero > > though, > > > so this doesn't really impact me. > > > > > > Would it be an easy modification to show the negotiated cipher in the > > > maillog or in the email body, if nothing else, just to have more info? > > > X-ASSP-Cipher: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384 > > > for example or in the received line like Microsoft does: > > > > > > Received: from __________ (x.x.x.x) by ______ > > > (y.y.y.y) with Microsoft SMTP Server (version=TLS1_2, > > > cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) > > > > > > Or like Gmail does > > > Received: from smtp.ourcharity.org (smtp.ourcharity.org. [x.x.x.x]) > > > by mx.google.com with ESMTPS id > > > l1si54144332.11.2016.09.28.07.51.21 > > > for <nntp.p...@gmail.com> > > > (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 > > bits=128/128); > > > Wed, 28 Sep 2016 07:24:22 -0700 (PDT) > > > > > > It's a little more data (and certainly your time to code this), but > > could > > > be useful if we need it in the future. > > > > > > and:: > > > > > > Again Ken, the SSL parameters are NOT the problem on your system. Your > > > > > > debug log shows a socket read time of max ~ 0.5 milliseconds (typical > > ~0.3 > > > > > > ms) for SSL. This read operation includes the time required for the > > > > > > decryption of the data. This is very very fast! > > > > > > It is simply the amazing count (10.400) of read and process operations > > > > > > (cycles) required by assp for such a mail, that causes the overall slow > > > > > > mail receive. > > > > > > in summary, if there's not an ASSP setting that I can tweak to have > > google > > > start sending larger SSL packets like Outlook.com and everyone else > I've > > > seen does, what other options are there? Are you suggesting that ALL > > ASSP > > > installations are incapable of receiving large TLS emails from Google > > > regardless of the installation's processing power, RAM, and bandwidth? > > > > > > > > > > > > On Wed, Sep 28, 2016 at 4:26 AM, Thomas Eckardt > > > <thomas.ecka...@thockar.com> > > > wrote: > > > > > > > >The same email now took only 269 seconds. > > > > > > > > This is OK for googles behavior - the 1440 byte SSL frame. Around > 300s > > > was > > > > expected by me for this mail - hmm.. it is 10% better - nice! > > > > > > > > Take the following math. > > > > > > > > mail size = 15000 kB > > > > google frame size = 1.44 kB > > > > > > > > required assp loop cycles = mail size / frame size = 10.400 > > > > > > > > mail size = 15000 kB > > > > outlook.com frame size = 16 kB > > > > > > > > required assp loop cycles = mail size / frame size = 940 > > > > > > > > >I really wonder if I need to tweak my cipher_list. > > > > > > > > No! > > > > Again Ken, the SSL parameters are NOT the problem on your system. > Your > > > > debug log shows a socket read time of max ~ 0.5 milliseconds (typical > > > ~0.3 > > > > ms) for SSL. This read operation includes the time required for the > > > > decryption of the data. This is very very fast! > > > > It is simply the amazing count (10.400) of read and process > operations > > > > (cycles) required by assp for such a mail, that causes the overall > > slow > > > > mail receive. > > > > > > > > >Is ClamAV scanning skipped too? > > > > Yes. > > > > > > > > >Could the plugins be run on the full mail after receipt, regardless > > of > > > > size? > > > > > > > > Override the config parameters. Keep in mind that 'npSize' may also > > > > involved in skipping or processing some mail body checks. > > > > > > > > >Isn't DKIM checking just a one time thing and not intensive? > > > > > > > > The full DKIM check is very intensive. It requires to calculate an > > > RSA/SHA > > > > hash over the complete mail. > > > > DCC and Razor are doing something similar. > > > > ASSP_AFC would make all checks (ClamAV, FileScan, content checks with > > > > several regular expression, decompression of attachments ....) for > the > > > > complete mail. It parses the complete mail at once with Email::MIME. > > > This > > > > requires a huge amount of memory. Not a big deal on a 64bit OS with > > 8GB > > > > RAM and several CPU cores - who can !? > > > > All these can be done for any mail size, if the system is able to > > > process > > > > the amount of data fast enough. > > > > On most havy load systems, the 12.000.000 will be to large and may > > lead > > > in > > > > to stucking workers. > > > > > > > > ASSP has this set of config parameters - change them to your need - > > try > > > - > > > > and if does not work switch back - nothing more easy. > > > > > > > > Thomas > > > > > > > > > > > > > > > > Von: K Post <nntp.p...@gmail.com> > > > > An: ASSP development mailing list > > <assp-test@lists.sourceforge.net> > > > > Datum: 27.09.2016 21:08 > > > > Betreff: Re: [Assp-test] Inbound TLS from gmail.com > addresses / > > > > servers > > > > > > > > > > > > > > > > Our primary internet connection went down again (nothing to do with > > > ASSP) > > > > which gave me the opportunity to replace 16270 with 16271. Nothing > > like > > > > making lemonade out of lemons... > > > > > > > > The same email now took only 269 seconds. That's about 15x longer > > than > > > > with TLS off, but WOW that's way better than it was before. > > > > I also tried with the blank cipher list, no notable difference > > > > And with the SSL buffers set to 0 (64 MB), again without a speed > > > > difference. > > > > > > > > SO- you've made a real difference here!! Is there more optimization > > to > > > > be > > > > made? > > > > > > > > The rub is that the exact same message sent through Outlook.com to > us, > > > > took > > > > exactly 30 seconds, just a 50% overhead when compared to the 19 > > seconds > > > > for > > > > a non-TLS message of the same size, instead of a 1500% overhead for > > > > encryption when receiving from google. > > > > > > > > *Is there some magical debug switch that I could turn on to see what > > > > encryption Outlook.com is using and compare that to what Google's > > > > connection to us with?* I think prohibiting whatever the slow cipher > > > that > > > > google's using (probably a really strong one) might make the final > bit > > > of > > > > difference. > > > > > > > > I'm breathing so much easier now!! Thank you. > > > > > > > > > > > > On Tue, Sep 27, 2016 at 2:08 PM, K Post <nntp.p...@gmail.com> wrote: > > > > > > > > > Despite all the problems we have with personalities and policies > in > > > our > > > > > organization, the infrastructure is pretty solid. MTU's are set > > > > correctly, > > > > > no fragmentation, no jitter. There's low latency across the > board, > > > and > > > > > really low bandwidth usage. If we sent 1000 mails a day, it's a > > lot. > > > > > > > > > > ...and yes, they even pay their bills, though not me very well :) > > > > > > > > > > I think it's which SSL algorithm is being used that's at least > > > partially > > > > > to blame. I have: > > > > > SSL_Version: SSLv23:!SSLv3:!SSLv2 > > > > > SSL_Cipher_list: kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:RC4 > > > > > :!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!kECDH:! > > > > > CAMELLIA128:!IDEA:!SEED > > > > > > > > > > I tried the default wih SSL_Cipher_List blank before, I don't think > > > > there > > > > > was a difference (but I've played with so many settings, I really > > > don't > > > > > remember) > > > > > > > > > > And last, on the SSL buffer size. If set to zero in the gui, on > > > windows > > > > > 2012, it says in green that it's set to 64 MB. I follow what > you're > > > > saying > > > > > about it readying 4x 16 Kb without a loop cycle. Is that a good or > > > bad > > > > > thing though? > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------ > > > > ------------------ > > > > _______________________________________________ > > > > Assp-test mailing list > > > > Assp-test@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > > > > > > > > > > > > > > > > DISCLAIMER: > > > > ******************************************************* > > > > This email and any files transmitted with it may be confidential, > > > legally > > > > privileged and protected in law and are intended solely for the use > of > > > the > > > > > > > > individual to whom it is addressed. > > > > This email was multiple times scanned for viruses. There should be no > > > > known virus in this email! > > > > ******************************************************* > > > > > > > > > > > > ------------------------------------------------------------ > > > > ------------------ > > > > > > > > _______________________________________________ > > > > Assp-test mailing list > > > > Assp-test@lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > > > > > > > ------------------------------------------------------------ > > > ------------------ > > > _______________________________________________ > > > Assp-test mailing list > > > Assp-test@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > > > > > > > > > > > DISCLAIMER: > > > ******************************************************* > > > This email and any files transmitted with it may be confidential, > > legally > > > privileged and protected in law and are intended solely for the use of > > the > > > > > > individual to whom it is addressed. > > > This email was multiple times scanned for viruses. There should be no > > > known virus in this email! > > > ******************************************************* > > > > > > > > > ------------------------------------------------------------ > > > ------------------ > > > > > > _______________________________________________ > > > Assp-test mailing list > > > Assp-test@lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > > > > ------------------------------------------------------------ > > ------------------ > > _______________________________________________ > > Assp-test mailing list > > Assp-test@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > > > > > > DISCLAIMER: > > ******************************************************* > > This email and any files transmitted with it may be confidential, legally > > privileged and protected in law and are intended solely for the use of > the > > > > individual to whom it is addressed. > > This email was multiple times scanned for viruses. There should be no > > known virus in this email! > > ******************************************************* > > > > > > ------------------------------------------------------------ > > ------------------ > > > > _______________________________________________ > > Assp-test mailing list > > Assp-test@lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/assp-test > > > > > > ------------------------------------------------------------ > ------------------ > > _______________________________________________ > Assp-test mailing list > Assp-test@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/assp-test > >
------------------------------------------------------------------------------
_______________________________________________ Assp-test mailing list Assp-test@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/assp-test
