Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[cw]

Hi Thomas,

I've got 16277 running straight away. Oddly I've not been able to get
connected to the GUI on either server since upgrading I just get timeouts.

I'm not entirely convinced this build is working properly for me. I'm
seeing only one email every few minutes rather than the constant output to
the logfiles.

Unfortunately I only saw that stuck message once and it cleared itself
quite quickly/before ASSP decided it was shutting itself down. Most of the
time there are no stuck workers and the status page shows green.



On Mon, Oct 3, 2016 at 2:05 PM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:

> >action: header (Content-Disposition -attr)
>
> 2.5.4 16277 will no longer use this output to state the worker
>
> please post the line produced with the current build
>
> "action: header (Content-Disposition -attr)" may be stated by a virus or
> attachment check, done in the body check of assp.pl or the ASSP_AFC plugin
>
> Thomas
>
>
>
>
>
> Von:cw <colin.war...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  30.09.2016 17:02
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Mixed results on this. So far no problems with running workers being
> logged
> but the GUI has become incredibly unresponsive. By unresponsive I mean I
> waited a good couple of minutes for the shutdown_list page to load.
> The dot on the main page is red yet the workers page is all green.
> Scratch that, it has refreshed again and I have a worker stuck:
> Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
> filename name (stuck)
> 30s later and it is healthy again..
>
> On the server I haven't upgraded the shutdown_list page comes up within
> seconds. I'm not sure whether to leave it running or whether this is
> evidence of the same kind of unresponsiveness that cause me to have to
> roll
> back earlier this week.
>
> On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote:
>
> > I wish I'd spotted this before writing out the other message. I'll give
> it
> > a test now for you.
> >
> > On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
> > thomas.ecka...@thockar.com> wrote:
> >
> >> Collin, this should no longer happen using the updated 2.5.2 16274_1 at
> >> CVS /test
> >>
> >> Thomas
> >>
> >>
> >>
> >> Von:cw <colin.war...@gmail.com>
> >> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> >> Datum:  29.09.2016 16:40
> >> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> >> servers
> >>
> >>
> >>
> >> Hi Thomas,
> >> I moved up to 16270 following this thread of discussion but then had a
> day
> >> working away. I've come back to huge issues with delays, mails not
> going
> >> through and many, many of these in the logs:
> >>
> >> Info: unable to detect any running worker for a new connection - wait
> (max
> >> 30 seconds)
> >>
> >> When I say many, I have over 21,000 lines in today's log file. I also
> >> found
> >> the GUI unresponsive or not connecting at all and ASSP restarting quite
> >> regularly.
> >>
> >> I've dropped back to 16256 and things are instantly better. Do you
> think
> >> going up to 16273 might improve things over 16270 or am I better
> holding
> >> off for now?
> >> All the best,
> >> Colin.
> >>
> >> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
> >> <thomas.ecka...@thockar.com>
> >> wrote:
> >>
> >> > I just released 2.5.2 build 16273 at CVS test folder
> >> >
> >> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
> >> >
> >> > This release should make a very large difference for SSL/TLS mails
> sent
> >> by
> >> > hosts that uses small SSL-frame size.
> >> >
> >> > Tell me your test results.
> >> >
> >> >
> >> > Thomas
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Von:K Post <nntp.p...@gmail.com>
> >> > An: ASSP development mailing list
> <assp-test@lists.sourceforge.net>
> >> > Datum:  28.09.2016 19:42
> >> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses
> /
> >> > servers
> >> >
> >> >
> >> >
> >> &g

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>action: header (Content-Disposition -attr)

2.5.4 16277 will no longer use this output to state the worker

please post the line produced with the current build

"action: header (Content-Disposition -attr)" may be stated by a virus or 
attachment check, done in the body check of assp.pl or the ASSP_AFC plugin

Thomas





Von:cw <colin.war...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  30.09.2016 17:02
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Mixed results on this. So far no problems with running workers being 
logged
but the GUI has become incredibly unresponsive. By unresponsive I mean I
waited a good couple of minutes for the shutdown_list page to load.
The dot on the main page is red yet the workers page is all green.
Scratch that, it has refreshed again and I have a worker stuck:
Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
filename name (stuck)
30s later and it is healthy again..

On the server I haven't upgraded the shutdown_list page comes up within
seconds. I'm not sure whether to leave it running or whether this is
evidence of the same kind of unresponsiveness that cause me to have to 
roll
back earlier this week.

On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote:

> I wish I'd spotted this before writing out the other message. I'll give 
it
> a test now for you.
>
> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
> thomas.ecka...@thockar.com> wrote:
>
>> Collin, this should no longer happen using the updated 2.5.2 16274_1 at
>> CVS /test
>>
>> Thomas
>>
>>
>>
>> Von:cw <colin.war...@gmail.com>
>> An:     ASSP development mailing list <assp-test@lists.sourceforge.net>
>> Datum:  29.09.2016 16:40
>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>> servers
>>
>>
>>
>> Hi Thomas,
>> I moved up to 16270 following this thread of discussion but then had a 
day
>> working away. I've come back to huge issues with delays, mails not 
going
>> through and many, many of these in the logs:
>>
>> Info: unable to detect any running worker for a new connection - wait 
(max
>> 30 seconds)
>>
>> When I say many, I have over 21,000 lines in today's log file. I also
>> found
>> the GUI unresponsive or not connecting at all and ASSP restarting quite
>> regularly.
>>
>> I've dropped back to 16256 and things are instantly better. Do you 
think
>> going up to 16273 might improve things over 16270 or am I better 
holding
>> off for now?
>> All the best,
>> Colin.
>>
>> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
>> <thomas.ecka...@thockar.com>
>> wrote:
>>
>> > I just released 2.5.2 build 16273 at CVS test folder
>> >
>> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>> >
>> > This release should make a very large difference for SSL/TLS mails 
sent
>> by
>> > hosts that uses small SSL-frame size.
>> >
>> > Tell me your test results.
>> >
>> >
>> > Thomas
>> >
>> >
>> >
>> >
>> >
>> > Von:K Post <nntp.p...@gmail.com>
>> > An: ASSP development mailing list 
<assp-test@lists.sourceforge.net>
>> > Datum:  28.09.2016 19:42
>> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses 
/
>> > servers
>> >
>> >
>> >
>> > But I want a postman driving a Ferarri with monster truck tires that 
can
>> > roll over the traffic (and if wishes are being granted, I'd prefer 
the
>> car
>> > in a deep blue instead of classic red).
>> >
>> > We regularly see people attaching large files or a bunch of smaller 
ones
>> > that add up to a big email, I'm talking lots and lots of different
>> people
>> > from outside the organization sending to us, and this happens on a 
daily
>> > basis.  It's especially popular with photos and huge scans multi-page
>> > 600dpi (which people don't understand can be done at low resolution).
>> > Often it's people sending in scanned official documents for us to 
review
>> > an
>> > help them.  They're not our staff, they're the people we help.  They
>> have
>> > a
>> > tendency of not following any instructions, and ignore the fact that 
we
>> > have a web based system for the process.  We can't control it and the
>> > powers that be don't want us lowering the 30 MB threshold across the
>> > board.  Lot of these 

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[cw]

Hi Thomas,

I've put 16275 on one of my servers this morning. I'm seeing the no running
workers error every 10-20 minutes causing ASSP to shut down. I've sat there
with the worker status page open and I'm not seeing any workers getting
stuck that coincide with this problem. In fact when the error is scrolling
past in the logs the workers are happily carrying on with whatever emails
they are processing.

Whilst typing, worker 5 has just gone to Bayes(OK_Run) Stuck with a timer
of around 240 before it cleared. This didn't trigger the fault just like
the other stuck message didn't really coincide and cleared itself.

I've upgraded the second server and so far am not seeing the delays that I
did with 16270. I'll keep an eye on it. Sure wish I could figure out what
is causing the no running workers error though.

All the best,
Colin

On Sun, Oct 2, 2016 at 7:44 PM, K Post <nntp.p...@gmail.com> wrote:

> It's been about a full day now with 75.  I see nothing but greatness.
>
> On Sun, Oct 2, 2016 at 2:59 AM, Thomas Eckardt <thomas.ecka...@thockar.com
> >
> wrote:
>
> > >then I see 9 seconds of idle/damping,
> >
> > If the complete mail is queued, this time is required to do the post
> > checks and conversions (charset conversion, plugins level 2, DKIM) and to
> > send the mail data to your MTA.
> >
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:K Post <nntp.p...@gmail.com>
> > An:     ASSP development mailing list <assp-test@lists.sourceforge.net>
> > Datum:  01.10.2016 22:01
> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> > servers
> >
> >
> >
> > I've been testing 16275 for a couple hours now.  FAST FAST FAST and
> stable
> > so far.   I see no errors in the maillog.
> >
> > Some notes / questions
> >
> > 1)  11MB attachment (14 MB email after encoding) transfers from google,
> > total time 25-29 seconds.  It seems that the whole email transfers in
> > about
> > 20 seconds, then I see 9 seconds of idle/damping,   I've never before
> seen
> > (or noticed I suppose) the idle column grow under normal circumstances.
> > Did you change the way messages are processed, in terms of doing
> > scans/analysis post transfer?  This idle counter doesn't matter to me,
> > speed's great, but I want to insure there's nothing wrong - and I'm
> > curious
> > if something changed on this front.
> >
> > 2) I see no notable difference in speed when I disable TLS for google
> now.
> > This is TERRIFIC.  There's certainly processing overhead for SSL, but
> your
> > code is so fast now that it's not noticeable.
> >
> > 3) Any point in adding a column to shutdown_list saying what the task at
> > hand is? AFC?  ClamAV?  Etc?
> >
> > For reference - we resumed this discussion August 1 (there were previous
> > attempts at getting this resolved, but they didn't go anywhere).  About 2
> > months later, 70 or so messages in this thread, a ton of your time,
> > dedication, and brainpower, we went from a gmail TLS message totaling 14
> > MB
> > taking upwards of 13 minutes and being unreliable now take only 25
> seconds
> > and be rock solid!  I don't know how you figured all of this out, put up
> > with my persistence, or had the time and brainpower to spare, but you
> seem
> > to have conquered this terrible problem!
> >
> > Tausend dank!  Ich bin Ihnen sehr dankbar für.
> >
> >
> >
> >
> > On Sat, Oct 1, 2016 at 11:23 AM, Thomas Eckardt
> > <thomas.ecka...@thockar.com>
> > wrote:
> >
> > > Ken, 2.5.2 build 16275 is in CVS /test
> > >
> > > Switching on the 'SSL_read_ahead' flag was a nice idea, but this caused
> > > some sockets - some times - at any state - to become unreadable.
> > > I've done extensive test on this - but I was not able to find a
> > > reproduceable reason for the behavior.
> > > I removed this code in build 16275. The read ahead is still available
> > for
> > > TLS/SSL and it is used per default by assp. The mechanism is a bit
> > > different like in build 16273/74.
> > > Every TLS/SSL socket gets additionally 50 milliseconds to read and
> > decode
> > > a much as available data from the underlying BIO
> > >
> > > On my nice old slow system, I was able to receive a 20MB mail from
> gmail
> > > in 110 seconds.
> > > Even for yahoo (they use 4096 byte SSL frames) it makes a big
> > difference.
> > >
> > > Thomas
> > >
> > >
> > >
> > >
> >

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

It's been about a full day now with 75.  I see nothing but greatness.

On Sun, Oct 2, 2016 at 2:59 AM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:

> >then I see 9 seconds of idle/damping,
>
> If the complete mail is queued, this time is required to do the post
> checks and conversions (charset conversion, plugins level 2, DKIM) and to
> send the mail data to your MTA.
>
>
> Thomas
>
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  01.10.2016 22:01
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> I've been testing 16275 for a couple hours now.  FAST FAST FAST and stable
> so far.   I see no errors in the maillog.
>
> Some notes / questions
>
> 1)  11MB attachment (14 MB email after encoding) transfers from google,
> total time 25-29 seconds.  It seems that the whole email transfers in
> about
> 20 seconds, then I see 9 seconds of idle/damping,   I've never before seen
> (or noticed I suppose) the idle column grow under normal circumstances.
> Did you change the way messages are processed, in terms of doing
> scans/analysis post transfer?  This idle counter doesn't matter to me,
> speed's great, but I want to insure there's nothing wrong - and I'm
> curious
> if something changed on this front.
>
> 2) I see no notable difference in speed when I disable TLS for google now.
> This is TERRIFIC.  There's certainly processing overhead for SSL, but your
> code is so fast now that it's not noticeable.
>
> 3) Any point in adding a column to shutdown_list saying what the task at
> hand is? AFC?  ClamAV?  Etc?
>
> For reference - we resumed this discussion August 1 (there were previous
> attempts at getting this resolved, but they didn't go anywhere).  About 2
> months later, 70 or so messages in this thread, a ton of your time,
> dedication, and brainpower, we went from a gmail TLS message totaling 14
> MB
> taking upwards of 13 minutes and being unreliable now take only 25 seconds
> and be rock solid!  I don't know how you figured all of this out, put up
> with my persistence, or had the time and brainpower to spare, but you seem
> to have conquered this terrible problem!
>
> Tausend dank!  Ich bin Ihnen sehr dankbar für.
>
>
>
>
> On Sat, Oct 1, 2016 at 11:23 AM, Thomas Eckardt
> <thomas.ecka...@thockar.com>
> wrote:
>
> > Ken, 2.5.2 build 16275 is in CVS /test
> >
> > Switching on the 'SSL_read_ahead' flag was a nice idea, but this caused
> > some sockets - some times - at any state - to become unreadable.
> > I've done extensive test on this - but I was not able to find a
> > reproduceable reason for the behavior.
> > I removed this code in build 16275. The read ahead is still available
> for
> > TLS/SSL and it is used per default by assp. The mechanism is a bit
> > different like in build 16273/74.
> > Every TLS/SSL socket gets additionally 50 milliseconds to read and
> decode
> > a much as available data from the underlying BIO
> >
> > On my nice old slow system, I was able to receive a 20MB mail from gmail
> > in 110 seconds.
> > Even for yahoo (they use 4096 byte SSL frames) it makes a big
> difference.
> >
> > Thomas
> >
> >
> >
> >
> > Von:K Post <nntp.p...@gmail.com>
> > An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> > Datum:  30.09.2016 19:21
> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> > servers
> >
> >
> >
> > 70 and 71 is fine here (Windows).73 was SUPER fast with SSL messages
> > from gmail, but then we got the idle / delay issues and had to revert to
> > 71.  Haven't had a chance to try 74 (need to wait for after hours)
> >
> >
> > On Fri, Sep 30, 2016 at 1:04 PM, Colin Waring <co...@dolphinict.co.uk>
> > wrote:
> >
> > > 16256 works acceptably but shuts down once or twice a day. 16270 or
> > > 16274_1 gave me problems with delays.
> > >
> > > I suspect the shutting down is a symptom of a different problem as it
> > has
> > > happened for a while.
> > >
> > > On 30 Sep 2016 17:57, Thomas Eckardt <thomas.ecka...@thockar.com>
> wrote:
> > > Hmm ... not OK.
> > >
> > > for my records:
> > >
> > > build 16256 is running fine
> > > builds 16270 and higher make problems
> > >
> > > right?
> > >
> > > Thomas
> > >
> > >
> > >
> > >
> > >
>

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>then I see 9 seconds of idle/damping,

If the complete mail is queued, this time is required to do the post 
checks and conversions (charset conversion, plugins level 2, DKIM) and to 
send the mail data to your MTA.


Thomas





Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  01.10.2016 22:01
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



I've been testing 16275 for a couple hours now.  FAST FAST FAST and stable
so far.   I see no errors in the maillog.

Some notes / questions

1)  11MB attachment (14 MB email after encoding) transfers from google,
total time 25-29 seconds.  It seems that the whole email transfers in 
about
20 seconds, then I see 9 seconds of idle/damping,   I've never before seen
(or noticed I suppose) the idle column grow under normal circumstances.
Did you change the way messages are processed, in terms of doing
scans/analysis post transfer?  This idle counter doesn't matter to me,
speed's great, but I want to insure there's nothing wrong - and I'm 
curious
if something changed on this front.

2) I see no notable difference in speed when I disable TLS for google now.
This is TERRIFIC.  There's certainly processing overhead for SSL, but your
code is so fast now that it's not noticeable.

3) Any point in adding a column to shutdown_list saying what the task at
hand is? AFC?  ClamAV?  Etc?

For reference - we resumed this discussion August 1 (there were previous
attempts at getting this resolved, but they didn't go anywhere).  About 2
months later, 70 or so messages in this thread, a ton of your time,
dedication, and brainpower, we went from a gmail TLS message totaling 14 
MB
taking upwards of 13 minutes and being unreliable now take only 25 seconds
and be rock solid!  I don't know how you figured all of this out, put up
with my persistence, or had the time and brainpower to spare, but you seem
to have conquered this terrible problem!

Tausend dank!  Ich bin Ihnen sehr dankbar für.




On Sat, Oct 1, 2016 at 11:23 AM, Thomas Eckardt 
<thomas.ecka...@thockar.com>
wrote:

> Ken, 2.5.2 build 16275 is in CVS /test
>
> Switching on the 'SSL_read_ahead' flag was a nice idea, but this caused
> some sockets - some times - at any state - to become unreadable.
> I've done extensive test on this - but I was not able to find a
> reproduceable reason for the behavior.
> I removed this code in build 16275. The read ahead is still available 
for
> TLS/SSL and it is used per default by assp. The mechanism is a bit
> different like in build 16273/74.
> Every TLS/SSL socket gets additionally 50 milliseconds to read and 
decode
> a much as available data from the underlying BIO
>
> On my nice old slow system, I was able to receive a 20MB mail from gmail
> in 110 seconds.
> Even for yahoo (they use 4096 byte SSL frames) it makes a big 
difference.
>
> Thomas
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  30.09.2016 19:21
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> 70 and 71 is fine here (Windows).73 was SUPER fast with SSL messages
> from gmail, but then we got the idle / delay issues and had to revert to
> 71.  Haven't had a chance to try 74 (need to wait for after hours)
>
>
> On Fri, Sep 30, 2016 at 1:04 PM, Colin Waring <co...@dolphinict.co.uk>
> wrote:
>
> > 16256 works acceptably but shuts down once or twice a day. 16270 or
> > 16274_1 gave me problems with delays.
> >
> > I suspect the shutting down is a symptom of a different problem as it
> has
> > happened for a while.
> >
> > On 30 Sep 2016 17:57, Thomas Eckardt <thomas.ecka...@thockar.com> 
wrote:
> > Hmm ... not OK.
> >
> > for my records:
> >
> > build 16256 is running fine
> > builds 16270 and higher make problems
> >
> > right?
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:cw <colin.war...@gmail.com>
> > An: ASSP development mailing list 
<assp-test@lists.sourceforge.net>
> > Datum:  30.09.2016 17:19
> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> > servers
> >
> >
> >
> > I've had to roll back now unfortunately as I'm getting email problems
> > again
> > :(
> >
> > On Fri, Sep 30, 2016 at 3:50 PM, cw <colin.war...@gmail.com> wrote:
> >
> > > Mixed results on this. So far no problems with running workers being
> > > logged but the GUI has become incredibly unresponsive. By 
unresponsive
> I
> > > mean I waited a good couple of minutes for the shutdow

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

I've been testing 16275 for a couple hours now.  FAST FAST FAST and stable
so far.   I see no errors in the maillog.

Some notes / questions

1)  11MB attachment (14 MB email after encoding) transfers from google,
total time 25-29 seconds.  It seems that the whole email transfers in about
20 seconds, then I see 9 seconds of idle/damping,   I've never before seen
(or noticed I suppose) the idle column grow under normal circumstances.
Did you change the way messages are processed, in terms of doing
scans/analysis post transfer?  This idle counter doesn't matter to me,
speed's great, but I want to insure there's nothing wrong - and I'm curious
if something changed on this front.

2) I see no notable difference in speed when I disable TLS for google now.
This is TERRIFIC.  There's certainly processing overhead for SSL, but your
code is so fast now that it's not noticeable.

3) Any point in adding a column to shutdown_list saying what the task at
hand is? AFC?  ClamAV?  Etc?

For reference - we resumed this discussion August 1 (there were previous
attempts at getting this resolved, but they didn't go anywhere).  About 2
months later, 70 or so messages in this thread, a ton of your time,
dedication, and brainpower, we went from a gmail TLS message totaling 14 MB
taking upwards of 13 minutes and being unreliable now take only 25 seconds
and be rock solid!  I don't know how you figured all of this out, put up
with my persistence, or had the time and brainpower to spare, but you seem
to have conquered this terrible problem!

Tausend dank!  Ich bin Ihnen sehr dankbar für.




On Sat, Oct 1, 2016 at 11:23 AM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:

> Ken, 2.5.2 build 16275 is in CVS /test
>
> Switching on the 'SSL_read_ahead' flag was a nice idea, but this caused
> some sockets - some times - at any state - to become unreadable.
> I've done extensive test on this - but I was not able to find a
> reproduceable reason for the behavior.
> I removed this code in build 16275. The read ahead is still available for
> TLS/SSL and it is used per default by assp. The mechanism is a bit
> different like in build 16273/74.
> Every TLS/SSL socket gets additionally 50 milliseconds to read and decode
> a much as available data from the underlying BIO
>
> On my nice old slow system, I was able to receive a 20MB mail from gmail
> in 110 seconds.
> Even for yahoo (they use 4096 byte SSL frames) it makes a big difference.
>
> Thomas
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  30.09.2016 19:21
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> 70 and 71 is fine here (Windows).73 was SUPER fast with SSL messages
> from gmail, but then we got the idle / delay issues and had to revert to
> 71.  Haven't had a chance to try 74 (need to wait for after hours)
>
>
> On Fri, Sep 30, 2016 at 1:04 PM, Colin Waring <co...@dolphinict.co.uk>
> wrote:
>
> > 16256 works acceptably but shuts down once or twice a day. 16270 or
> > 16274_1 gave me problems with delays.
> >
> > I suspect the shutting down is a symptom of a different problem as it
> has
> > happened for a while.
> >
> > On 30 Sep 2016 17:57, Thomas Eckardt <thomas.ecka...@thockar.com> wrote:
> > Hmm ... not OK.
> >
> > for my records:
> >
> > build 16256 is running fine
> > builds 16270 and higher make problems
> >
> > right?
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:cw <colin.war...@gmail.com>
> > An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> > Datum:  30.09.2016 17:19
> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> > servers
> >
> >
> >
> > I've had to roll back now unfortunately as I'm getting email problems
> > again
> > :(
> >
> > On Fri, Sep 30, 2016 at 3:50 PM, cw <colin.war...@gmail.com> wrote:
> >
> > > Mixed results on this. So far no problems with running workers being
> > > logged but the GUI has become incredibly unresponsive. By unresponsive
> I
> > > mean I waited a good couple of minutes for the shutdown_list page to
> > load.
> > > The dot on the main page is red yet the workers page is all green.
> > > Scratch that, it has refreshed again and I have a worker stuck:
> > > Worker 3, loop age 252s, action: header (Content-Disposition -attr) :
> :
> > > filename name (stuck)
> > > 30s later and it is healthy again..
> > >
> > > On the server I haven't upgraded the shutdown_list page comes up

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

Ken, 2.5.2 build 16275 is in CVS /test

Switching on the 'SSL_read_ahead' flag was a nice idea, but this caused 
some sockets - some times - at any state - to become unreadable.
I've done extensive test on this - but I was not able to find a 
reproduceable reason for the behavior.
I removed this code in build 16275. The read ahead is still available for 
TLS/SSL and it is used per default by assp. The mechanism is a bit 
different like in build 16273/74.
Every TLS/SSL socket gets additionally 50 milliseconds to read and decode 
a much as available data from the underlying BIO

On my nice old slow system, I was able to receive a 20MB mail from gmail 
in 110 seconds.
Even for yahoo (they use 4096 byte SSL frames) it makes a big difference.

Thomas




Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  30.09.2016 19:21
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



70 and 71 is fine here (Windows).73 was SUPER fast with SSL messages
from gmail, but then we got the idle / delay issues and had to revert to
71.  Haven't had a chance to try 74 (need to wait for after hours)


On Fri, Sep 30, 2016 at 1:04 PM, Colin Waring <co...@dolphinict.co.uk>
wrote:

> 16256 works acceptably but shuts down once or twice a day. 16270 or
> 16274_1 gave me problems with delays.
>
> I suspect the shutting down is a symptom of a different problem as it 
has
> happened for a while.
>
> On 30 Sep 2016 17:57, Thomas Eckardt <thomas.ecka...@thockar.com> wrote:
> Hmm ... not OK.
>
> for my records:
>
> build 16256 is running fine
> builds 16270 and higher make problems
>
> right?
>
> Thomas
>
>
>
>
>
> Von:cw <colin.war...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  30.09.2016 17:19
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> I've had to roll back now unfortunately as I'm getting email problems
> again
> :(
>
> On Fri, Sep 30, 2016 at 3:50 PM, cw <colin.war...@gmail.com> wrote:
>
> > Mixed results on this. So far no problems with running workers being
> > logged but the GUI has become incredibly unresponsive. By unresponsive 
I
> > mean I waited a good couple of minutes for the shutdown_list page to
> load.
> > The dot on the main page is red yet the workers page is all green.
> > Scratch that, it has refreshed again and I have a worker stuck:
> > Worker 3, loop age 252s, action: header (Content-Disposition -attr) : 
:
> > filename name (stuck)
> > 30s later and it is healthy again..
> >
> > On the server I haven't upgraded the shutdown_list page comes up 
within
> > seconds. I'm not sure whether to leave it running or whether this is
> > evidence of the same kind of unresponsiveness that cause me to have to
> roll
> > back earlier this week.
> >
> > On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote:
> >
> >> I wish I'd spotted this before writing out the other message. I'll 
give
> >> it a test now for you.
> >>
> >> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
> >> thomas.ecka...@thockar.com> wrote:
> >>
> >>> Collin, this should no longer happen using the updated 2.5.2 16274_1
> at
> >>> CVS /test
> >>>
> >>> Thomas
> >>>
> >>>
> >>>
> >>> Von:cw <colin.war...@gmail.com>
> >>> An: ASSP development mailing list
> <assp-test@lists.sourceforge.net>
> >>> Datum:  29.09.2016 16:40
> >>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses 
/
> >>> servers
> >>>
> >>>
> >>>
> >>> Hi Thomas,
> >>> I moved up to 16270 following this thread of discussion but then had 
a
> >>> day
> >>> working away. I've come back to huge issues with delays, mails not
> going
> >>> through and many, many of these in the logs:
> >>>
> >>> Info: unable to detect any running worker for a new connection - 
wait
> >>> (max
> >>> 30 seconds)
> >>>
> >>> When I say many, I have over 21,000 lines in today's log file. I 
also
> >>> found
> >>> the GUI unresponsive or not connecting at all and ASSP restarting
> quite
> >>> regularly.
> >>>
> >>> I've dropped back to 16256 and things are instantly better. Do you
> think
> >>> going up to 16273 might improve things over 16270 or am I better
> 

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

70 and 71 is fine here (Windows).73 was SUPER fast with SSL messages
from gmail, but then we got the idle / delay issues and had to revert to
71.  Haven't had a chance to try 74 (need to wait for after hours)


On Fri, Sep 30, 2016 at 1:04 PM, Colin Waring <co...@dolphinict.co.uk>
wrote:

> 16256 works acceptably but shuts down once or twice a day. 16270 or
> 16274_1 gave me problems with delays.
>
> I suspect the shutting down is a symptom of a different problem as it has
> happened for a while.
>
> On 30 Sep 2016 17:57, Thomas Eckardt <thomas.ecka...@thockar.com> wrote:
> Hmm ... not OK.
>
> for my records:
>
> build 16256 is running fine
> builds 16270 and higher make problems
>
> right?
>
> Thomas
>
>
>
>
>
> Von:cw <colin.war...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  30.09.2016 17:19
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> I've had to roll back now unfortunately as I'm getting email problems
> again
> :(
>
> On Fri, Sep 30, 2016 at 3:50 PM, cw <colin.war...@gmail.com> wrote:
>
> > Mixed results on this. So far no problems with running workers being
> > logged but the GUI has become incredibly unresponsive. By unresponsive I
> > mean I waited a good couple of minutes for the shutdown_list page to
> load.
> > The dot on the main page is red yet the workers page is all green.
> > Scratch that, it has refreshed again and I have a worker stuck:
> > Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
> > filename name (stuck)
> > 30s later and it is healthy again..
> >
> > On the server I haven't upgraded the shutdown_list page comes up within
> > seconds. I'm not sure whether to leave it running or whether this is
> > evidence of the same kind of unresponsiveness that cause me to have to
> roll
> > back earlier this week.
> >
> > On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote:
> >
> >> I wish I'd spotted this before writing out the other message. I'll give
> >> it a test now for you.
> >>
> >> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
> >> thomas.ecka...@thockar.com> wrote:
> >>
> >>> Collin, this should no longer happen using the updated 2.5.2 16274_1
> at
> >>> CVS /test
> >>>
> >>> Thomas
> >>>
> >>>
> >>>
> >>> Von:cw <colin.war...@gmail.com>
> >>> An: ASSP development mailing list
> <assp-test@lists.sourceforge.net>
> >>> Datum:  29.09.2016 16:40
> >>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> >>> servers
> >>>
> >>>
> >>>
> >>> Hi Thomas,
> >>> I moved up to 16270 following this thread of discussion but then had a
> >>> day
> >>> working away. I've come back to huge issues with delays, mails not
> going
> >>> through and many, many of these in the logs:
> >>>
> >>> Info: unable to detect any running worker for a new connection - wait
> >>> (max
> >>> 30 seconds)
> >>>
> >>> When I say many, I have over 21,000 lines in today's log file. I also
> >>> found
> >>> the GUI unresponsive or not connecting at all and ASSP restarting
> quite
> >>> regularly.
> >>>
> >>> I've dropped back to 16256 and things are instantly better. Do you
> think
> >>> going up to 16273 might improve things over 16270 or am I better
> holding
> >>> off for now?
> >>> All the best,
> >>> Colin.
> >>>
> >>> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
> >>> <thomas.ecka...@thockar.com>
> >>> wrote:
> >>>
> >>> > I just released 2.5.2 build 16273 at CVS test folder
> >>> >
> >>> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
> >>> >
> >>> > This release should make a very large difference for SSL/TLS mails
> sent
> >>> by
> >>> > hosts that uses small SSL-frame size.
> >>> >
> >>> > Tell me your test results.
> >>> >
> >>> >
> >>> > Thomas
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> > Von:K Post <nntp.p...@gmail.com>
> >>

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Colin Waring]

16256 works acceptably but shuts down once or twice a day. 16270 or 16274_1 
gave me problems with delays.

I suspect the shutting down is a symptom of a different problem as it has 
happened for a while.

On 30 Sep 2016 17:57, Thomas Eckardt <thomas.ecka...@thockar.com> wrote:
Hmm ... not OK.

for my records:

build 16256 is running fine
builds 16270 and higher make problems

right?

Thomas





Von:cw <colin.war...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  30.09.2016 17:19
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses /
servers



I've had to roll back now unfortunately as I'm getting email problems
again
:(

On Fri, Sep 30, 2016 at 3:50 PM, cw <colin.war...@gmail.com> wrote:

> Mixed results on this. So far no problems with running workers being
> logged but the GUI has become incredibly unresponsive. By unresponsive I
> mean I waited a good couple of minutes for the shutdown_list page to
load.
> The dot on the main page is red yet the workers page is all green.
> Scratch that, it has refreshed again and I have a worker stuck:
> Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
> filename name (stuck)
> 30s later and it is healthy again..
>
> On the server I haven't upgraded the shutdown_list page comes up within
> seconds. I'm not sure whether to leave it running or whether this is
> evidence of the same kind of unresponsiveness that cause me to have to
roll
> back earlier this week.
>
> On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote:
>
>> I wish I'd spotted this before writing out the other message. I'll give
>> it a test now for you.
>>
>> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
>> thomas.ecka...@thockar.com> wrote:
>>
>>> Collin, this should no longer happen using the updated 2.5.2 16274_1
at
>>> CVS /test
>>>
>>> Thomas
>>>
>>>
>>>
>>> Von:cw <colin.war...@gmail.com>
>>> An: ASSP development mailing list
<assp-test@lists.sourceforge.net>
>>> Datum:  29.09.2016 16:40
>>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>>> servers
>>>
>>>
>>>
>>> Hi Thomas,
>>> I moved up to 16270 following this thread of discussion but then had a
>>> day
>>> working away. I've come back to huge issues with delays, mails not
going
>>> through and many, many of these in the logs:
>>>
>>> Info: unable to detect any running worker for a new connection - wait
>>> (max
>>> 30 seconds)
>>>
>>> When I say many, I have over 21,000 lines in today's log file. I also
>>> found
>>> the GUI unresponsive or not connecting at all and ASSP restarting
quite
>>> regularly.
>>>
>>> I've dropped back to 16256 and things are instantly better. Do you
think
>>> going up to 16273 might improve things over 16270 or am I better
holding
>>> off for now?
>>> All the best,
>>> Colin.
>>>
>>> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
>>> <thomas.ecka...@thockar.com>
>>> wrote:
>>>
>>> > I just released 2.5.2 build 16273 at CVS test folder
>>> >
>>> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>>> >
>>> > This release should make a very large difference for SSL/TLS mails
sent
>>> by
>>> > hosts that uses small SSL-frame size.
>>> >
>>> > Tell me your test results.
>>> >
>>> >
>>> > Thomas
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Von:K Post <nntp.p...@gmail.com>
>>> > An: ASSP development mailing list
<assp-test@lists.sourceforge.net
>>> >
>>> > Datum:  28.09.2016 19:42
>>> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses
/
>>> > servers
>>> >
>>> >
>>> >
>>> > But I want a postman driving a Ferarri with monster truck tires that
>>> can
>>> > roll over the traffic (and if wishes are being granted, I'd prefer
the
>>> car
>>> > in a deep blue instead of classic red).
>>> >
>>> > We regularly see people attaching large files or a bunch of smaller
>>> ones
>>> > that add up to a big email, I'm talking lots and lots of different
>>> people
>>> > from outside the organization sending to us, and this happens on a
>>> daily
>&

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

Hmm ... not OK.

for my records:

build 16256 is running fine
builds 16270 and higher make problems

right?

Thomas





Von:cw <colin.war...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  30.09.2016 17:19
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



I've had to roll back now unfortunately as I'm getting email problems 
again
:(

On Fri, Sep 30, 2016 at 3:50 PM, cw <colin.war...@gmail.com> wrote:

> Mixed results on this. So far no problems with running workers being
> logged but the GUI has become incredibly unresponsive. By unresponsive I
> mean I waited a good couple of minutes for the shutdown_list page to 
load.
> The dot on the main page is red yet the workers page is all green.
> Scratch that, it has refreshed again and I have a worker stuck:
> Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
> filename name (stuck)
> 30s later and it is healthy again..
>
> On the server I haven't upgraded the shutdown_list page comes up within
> seconds. I'm not sure whether to leave it running or whether this is
> evidence of the same kind of unresponsiveness that cause me to have to 
roll
> back earlier this week.
>
> On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote:
>
>> I wish I'd spotted this before writing out the other message. I'll give
>> it a test now for you.
>>
>> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
>> thomas.ecka...@thockar.com> wrote:
>>
>>> Collin, this should no longer happen using the updated 2.5.2 16274_1 
at
>>> CVS /test
>>>
>>> Thomas
>>>
>>>
>>>
>>> Von:cw <colin.war...@gmail.com>
>>> An: ASSP development mailing list 
<assp-test@lists.sourceforge.net>
>>> Datum:  29.09.2016 16:40
>>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>>> servers
>>>
>>>
>>>
>>> Hi Thomas,
>>> I moved up to 16270 following this thread of discussion but then had a
>>> day
>>> working away. I've come back to huge issues with delays, mails not 
going
>>> through and many, many of these in the logs:
>>>
>>> Info: unable to detect any running worker for a new connection - wait
>>> (max
>>> 30 seconds)
>>>
>>> When I say many, I have over 21,000 lines in today's log file. I also
>>> found
>>> the GUI unresponsive or not connecting at all and ASSP restarting 
quite
>>> regularly.
>>>
>>> I've dropped back to 16256 and things are instantly better. Do you 
think
>>> going up to 16273 might improve things over 16270 or am I better 
holding
>>> off for now?
>>> All the best,
>>> Colin.
>>>
>>> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
>>> <thomas.ecka...@thockar.com>
>>> wrote:
>>>
>>> > I just released 2.5.2 build 16273 at CVS test folder
>>> >
>>> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>>> >
>>> > This release should make a very large difference for SSL/TLS mails 
sent
>>> by
>>> > hosts that uses small SSL-frame size.
>>> >
>>> > Tell me your test results.
>>> >
>>> >
>>> > Thomas
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Von:K Post <nntp.p...@gmail.com>
>>> > An: ASSP development mailing list 
<assp-test@lists.sourceforge.net
>>> >
>>> > Datum:  28.09.2016 19:42
>>> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses 
/
>>> > servers
>>> >
>>> >
>>> >
>>> > But I want a postman driving a Ferarri with monster truck tires that
>>> can
>>> > roll over the traffic (and if wishes are being granted, I'd prefer 
the
>>> car
>>> > in a deep blue instead of classic red).
>>> >
>>> > We regularly see people attaching large files or a bunch of smaller
>>> ones
>>> > that add up to a big email, I'm talking lots and lots of different
>>> people
>>> > from outside the organization sending to us, and this happens on a
>>> daily
>>> > basis.  It's especially popular with photos and huge scans 
multi-page
>>> > 600dpi (which people don't understand can be done at low 
resolution).
>>> > Often it's people sending in scanned official documents for us to
>>

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
filename name (stuck)
30s later and it is healthy again..


This sounds like a bomb regex long time run. Possibly the worker was 
restarted for the green dot.

Thomas




Von:cw <colin.war...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  30.09.2016 17:02
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Mixed results on this. So far no problems with running workers being 
logged
but the GUI has become incredibly unresponsive. By unresponsive I mean I
waited a good couple of minutes for the shutdown_list page to load.
The dot on the main page is red yet the workers page is all green.
Scratch that, it has refreshed again and I have a worker stuck:
Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
filename name (stuck)
30s later and it is healthy again..

On the server I haven't upgraded the shutdown_list page comes up within
seconds. I'm not sure whether to leave it running or whether this is
evidence of the same kind of unresponsiveness that cause me to have to 
roll
back earlier this week.

On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote:

> I wish I'd spotted this before writing out the other message. I'll give 
it
> a test now for you.
>
> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
> thomas.ecka...@thockar.com> wrote:
>
>> Collin, this should no longer happen using the updated 2.5.2 16274_1 at
>> CVS /test
>>
>> Thomas
>>
>>
>>
>> Von:cw <colin.war...@gmail.com>
>> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
>> Datum:  29.09.2016 16:40
>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>> servers
>>
>>
>>
>> Hi Thomas,
>> I moved up to 16270 following this thread of discussion but then had a 
day
>> working away. I've come back to huge issues with delays, mails not 
going
>> through and many, many of these in the logs:
>>
>> Info: unable to detect any running worker for a new connection - wait 
(max
>> 30 seconds)
>>
>> When I say many, I have over 21,000 lines in today's log file. I also
>> found
>> the GUI unresponsive or not connecting at all and ASSP restarting quite
>> regularly.
>>
>> I've dropped back to 16256 and things are instantly better. Do you 
think
>> going up to 16273 might improve things over 16270 or am I better 
holding
>> off for now?
>> All the best,
>> Colin.
>>
>> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
>> <thomas.ecka...@thockar.com>
>> wrote:
>>
>> > I just released 2.5.2 build 16273 at CVS test folder
>> >
>> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>> >
>> > This release should make a very large difference for SSL/TLS mails 
sent
>> by
>> > hosts that uses small SSL-frame size.
>> >
>> > Tell me your test results.
>> >
>> >
>> > Thomas
>> >
>> >
>> >
>> >
>> >
>> > Von:K Post <nntp.p...@gmail.com>
>> > An: ASSP development mailing list 
<assp-test@lists.sourceforge.net>
>> > Datum:  28.09.2016 19:42
>> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses 
/
>> > servers
>> >
>> >
>> >
>> > But I want a postman driving a Ferarri with monster truck tires that 
can
>> > roll over the traffic (and if wishes are being granted, I'd prefer 
the
>> car
>> > in a deep blue instead of classic red).
>> >
>> > We regularly see people attaching large files or a bunch of smaller 
ones
>> > that add up to a big email, I'm talking lots and lots of different
>> people
>> > from outside the organization sending to us, and this happens on a 
daily
>> > basis.  It's especially popular with photos and huge scans multi-page
>> > 600dpi (which people don't understand can be done at low resolution).
>> > Often it's people sending in scanned official documents for us to 
review
>> > an
>> > help them.  They're not our staff, they're the people we help.  They
>> have
>> > a
>> > tendency of not following any instructions, and ignore the fact that 
we
>> > have a web based system for the process.  We can't control it and the
>> > powers that be don't want us lowering the 30 MB threshold across the
>> > board.  Lot of these people use gmail.com addresses and google allows
>> for
>> > up to 25 MB - https

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[cw]

I've had to roll back now unfortunately as I'm getting email problems again
:(

On Fri, Sep 30, 2016 at 3:50 PM, cw <colin.war...@gmail.com> wrote:

> Mixed results on this. So far no problems with running workers being
> logged but the GUI has become incredibly unresponsive. By unresponsive I
> mean I waited a good couple of minutes for the shutdown_list page to load.
> The dot on the main page is red yet the workers page is all green.
> Scratch that, it has refreshed again and I have a worker stuck:
> Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
> filename name (stuck)
> 30s later and it is healthy again..
>
> On the server I haven't upgraded the shutdown_list page comes up within
> seconds. I'm not sure whether to leave it running or whether this is
> evidence of the same kind of unresponsiveness that cause me to have to roll
> back earlier this week.
>
> On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote:
>
>> I wish I'd spotted this before writing out the other message. I'll give
>> it a test now for you.
>>
>> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
>> thomas.ecka...@thockar.com> wrote:
>>
>>> Collin, this should no longer happen using the updated 2.5.2 16274_1 at
>>> CVS /test
>>>
>>> Thomas
>>>
>>>
>>>
>>> Von:cw <colin.war...@gmail.com>
>>> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
>>> Datum:  29.09.2016 16:40
>>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>>> servers
>>>
>>>
>>>
>>> Hi Thomas,
>>> I moved up to 16270 following this thread of discussion but then had a
>>> day
>>> working away. I've come back to huge issues with delays, mails not going
>>> through and many, many of these in the logs:
>>>
>>> Info: unable to detect any running worker for a new connection - wait
>>> (max
>>> 30 seconds)
>>>
>>> When I say many, I have over 21,000 lines in today's log file. I also
>>> found
>>> the GUI unresponsive or not connecting at all and ASSP restarting quite
>>> regularly.
>>>
>>> I've dropped back to 16256 and things are instantly better. Do you think
>>> going up to 16273 might improve things over 16270 or am I better holding
>>> off for now?
>>> All the best,
>>> Colin.
>>>
>>> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
>>> <thomas.ecka...@thockar.com>
>>> wrote:
>>>
>>> > I just released 2.5.2 build 16273 at CVS test folder
>>> >
>>> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>>> >
>>> > This release should make a very large difference for SSL/TLS mails sent
>>> by
>>> > hosts that uses small SSL-frame size.
>>> >
>>> > Tell me your test results.
>>> >
>>> >
>>> > Thomas
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Von:K Post <nntp.p...@gmail.com>
>>> > An: ASSP development mailing list <assp-test@lists.sourceforge.net
>>> >
>>> > Datum:  28.09.2016 19:42
>>> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>>> > servers
>>> >
>>> >
>>> >
>>> > But I want a postman driving a Ferarri with monster truck tires that
>>> can
>>> > roll over the traffic (and if wishes are being granted, I'd prefer the
>>> car
>>> > in a deep blue instead of classic red).
>>> >
>>> > We regularly see people attaching large files or a bunch of smaller
>>> ones
>>> > that add up to a big email, I'm talking lots and lots of different
>>> people
>>> > from outside the organization sending to us, and this happens on a
>>> daily
>>> > basis.  It's especially popular with photos and huge scans multi-page
>>> > 600dpi (which people don't understand can be done at low resolution).
>>> > Often it's people sending in scanned official documents for us to
>>> review
>>> > an
>>> > help them.  They're not our staff, they're the people we help.  They
>>> have
>>> > a
>>> > tendency of not following any instructions, and ignore the fact that we
>>> > have a web based system for the process.  We can't control it and the
>>> > p

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[cw]

Mixed results on this. So far no problems with running workers being logged
but the GUI has become incredibly unresponsive. By unresponsive I mean I
waited a good couple of minutes for the shutdown_list page to load.
The dot on the main page is red yet the workers page is all green.
Scratch that, it has refreshed again and I have a worker stuck:
Worker 3, loop age 252s, action: header (Content-Disposition -attr) : :
filename name (stuck)
30s later and it is healthy again..

On the server I haven't upgraded the shutdown_list page comes up within
seconds. I'm not sure whether to leave it running or whether this is
evidence of the same kind of unresponsiveness that cause me to have to roll
back earlier this week.

On Fri, Sep 30, 2016 at 3:29 PM, cw <colin.war...@gmail.com> wrote:

> I wish I'd spotted this before writing out the other message. I'll give it
> a test now for you.
>
> On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <
> thomas.ecka...@thockar.com> wrote:
>
>> Collin, this should no longer happen using the updated 2.5.2 16274_1 at
>> CVS /test
>>
>> Thomas
>>
>>
>>
>> Von:cw <colin.war...@gmail.com>
>> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
>> Datum:  29.09.2016 16:40
>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>> servers
>>
>>
>>
>> Hi Thomas,
>> I moved up to 16270 following this thread of discussion but then had a day
>> working away. I've come back to huge issues with delays, mails not going
>> through and many, many of these in the logs:
>>
>> Info: unable to detect any running worker for a new connection - wait (max
>> 30 seconds)
>>
>> When I say many, I have over 21,000 lines in today's log file. I also
>> found
>> the GUI unresponsive or not connecting at all and ASSP restarting quite
>> regularly.
>>
>> I've dropped back to 16256 and things are instantly better. Do you think
>> going up to 16273 might improve things over 16270 or am I better holding
>> off for now?
>> All the best,
>> Colin.
>>
>> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
>> <thomas.ecka...@thockar.com>
>> wrote:
>>
>> > I just released 2.5.2 build 16273 at CVS test folder
>> >
>> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>> >
>> > This release should make a very large difference for SSL/TLS mails sent
>> by
>> > hosts that uses small SSL-frame size.
>> >
>> > Tell me your test results.
>> >
>> >
>> > Thomas
>> >
>> >
>> >
>> >
>> >
>> > Von:K Post <nntp.p...@gmail.com>
>> > An: ASSP development mailing list <assp-test@lists.sourceforge.net>
>> > Datum:  28.09.2016 19:42
>> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>> > servers
>> >
>> >
>> >
>> > But I want a postman driving a Ferarri with monster truck tires that can
>> > roll over the traffic (and if wishes are being granted, I'd prefer the
>> car
>> > in a deep blue instead of classic red).
>> >
>> > We regularly see people attaching large files or a bunch of smaller ones
>> > that add up to a big email, I'm talking lots and lots of different
>> people
>> > from outside the organization sending to us, and this happens on a daily
>> > basis.  It's especially popular with photos and huge scans multi-page
>> > 600dpi (which people don't understand can be done at low resolution).
>> > Often it's people sending in scanned official documents for us to review
>> > an
>> > help them.  They're not our staff, they're the people we help.  They
>> have
>> > a
>> > tendency of not following any instructions, and ignore the fact that we
>> > have a web based system for the process.  We can't control it and the
>> > powers that be don't want us lowering the 30 MB threshold across the
>> > board.  Lot of these people use gmail.com addresses and google allows
>> for
>> > up to 25 MB - https://support.google.com/mail/answer/6584
>> >
>> > I think it's really interesting that google seems to use this
>> inefficient
>> > small packet size for SSL, allows for 25MB emails, is a big proponent of
>> > SSL, and at the same time doesn't allow mails to take more than 15
>> minutes
>> > to transfer.  Now that you've made things >much< more efficient on the
>> > ASSP
>> > side, I'm hoping that all will be

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[cw]

I wish I'd spotted this before writing out the other message. I'll give it
a test now for you.

On Fri, Sep 30, 2016 at 2:17 PM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:

> Collin, this should no longer happen using the updated 2.5.2 16274_1 at
> CVS /test
>
> Thomas
>
>
>
> Von:cw <colin.war...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  29.09.2016 16:40
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Hi Thomas,
> I moved up to 16270 following this thread of discussion but then had a day
> working away. I've come back to huge issues with delays, mails not going
> through and many, many of these in the logs:
>
> Info: unable to detect any running worker for a new connection - wait (max
> 30 seconds)
>
> When I say many, I have over 21,000 lines in today's log file. I also
> found
> the GUI unresponsive or not connecting at all and ASSP restarting quite
> regularly.
>
> I've dropped back to 16256 and things are instantly better. Do you think
> going up to 16273 might improve things over 16270 or am I better holding
> off for now?
> All the best,
> Colin.
>
> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt
> <thomas.ecka...@thockar.com>
> wrote:
>
> > I just released 2.5.2 build 16273 at CVS test folder
> >
> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
> >
> > This release should make a very large difference for SSL/TLS mails sent
> by
> > hosts that uses small SSL-frame size.
> >
> > Tell me your test results.
> >
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:K Post <nntp.p...@gmail.com>
> > An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> > Datum:  28.09.2016 19:42
> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> > servers
> >
> >
> >
> > But I want a postman driving a Ferarri with monster truck tires that can
> > roll over the traffic (and if wishes are being granted, I'd prefer the
> car
> > in a deep blue instead of classic red).
> >
> > We regularly see people attaching large files or a bunch of smaller ones
> > that add up to a big email, I'm talking lots and lots of different
> people
> > from outside the organization sending to us, and this happens on a daily
> > basis.  It's especially popular with photos and huge scans multi-page
> > 600dpi (which people don't understand can be done at low resolution).
> > Often it's people sending in scanned official documents for us to review
> > an
> > help them.  They're not our staff, they're the people we help.  They
> have
> > a
> > tendency of not following any instructions, and ignore the fact that we
> > have a web based system for the process.  We can't control it and the
> > powers that be don't want us lowering the 30 MB threshold across the
> > board.  Lot of these people use gmail.com addresses and google allows
> for
> > up to 25 MB - https://support.google.com/mail/answer/6584
> >
> > I think it's really interesting that google seems to use this
> inefficient
> > small packet size for SSL, allows for 25MB emails, is a big proponent of
> > SSL, and at the same time doesn't allow mails to take more than 15
> minutes
> > to transfer.  Now that you've made things >much< more efficient on the
> > ASSP
> > side, I'm hoping that all will be okay.  I just get annoyed by
> > inefficiency.
> >
> >
> > I'm going to tryrunning with npSize of zero, the no queuing size set
> very
> > high and see how that goes.  I want to insure that even the biggest
> emails
> > are scanned for malware before hitting the inbox.
> >
> >
> > I've never said this before, but it seems like google's doing it wrong.
> I
> > think we've exhausted our options here.  If anyone has a Google
> > engineering
> > friend, or a friend of a friend, this might be a good time to have a
> > little
> > chat in an effort to reach someone who can either explain why they're
> use
> > a
> > 1.4 kB frame size for ssl packets, but a normal much bigger size for
> > non-encrypted traffic or to get them to see an error in their ways and
> fix
> > this!
> >
> > Thank you again for all of the discussion and explanation and of course
> > the
> > code changes which have made a huge difference.
> >
> > On Wed, Sep 28, 2016 at 12:59 PM, Thomas Eckardt
> > <thomas.ecka...@thockar.com
> > > wrot

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

Collin, this should no longer happen using the updated 2.5.2 16274_1 at 
CVS /test

Thomas



Von:cw <colin.war...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  29.09.2016 16:40
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Hi Thomas,
I moved up to 16270 following this thread of discussion but then had a day
working away. I've come back to huge issues with delays, mails not going
through and many, many of these in the logs:

Info: unable to detect any running worker for a new connection - wait (max
30 seconds)

When I say many, I have over 21,000 lines in today's log file. I also 
found
the GUI unresponsive or not connecting at all and ASSP restarting quite
regularly.

I've dropped back to 16256 and things are instantly better. Do you think
going up to 16273 might improve things over 16270 or am I better holding
off for now?
All the best,
Colin.

On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt 
<thomas.ecka...@thockar.com>
wrote:

> I just released 2.5.2 build 16273 at CVS test folder
>
> http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>
> This release should make a very large difference for SSL/TLS mails sent 
by
> hosts that uses small SSL-frame size.
>
> Tell me your test results.
>
>
> Thomas
>
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  28.09.2016 19:42
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> But I want a postman driving a Ferarri with monster truck tires that can
> roll over the traffic (and if wishes are being granted, I'd prefer the 
car
> in a deep blue instead of classic red).
>
> We regularly see people attaching large files or a bunch of smaller ones
> that add up to a big email, I'm talking lots and lots of different 
people
> from outside the organization sending to us, and this happens on a daily
> basis.  It's especially popular with photos and huge scans multi-page
> 600dpi (which people don't understand can be done at low resolution).
> Often it's people sending in scanned official documents for us to review
> an
> help them.  They're not our staff, they're the people we help.  They 
have
> a
> tendency of not following any instructions, and ignore the fact that we
> have a web based system for the process.  We can't control it and the
> powers that be don't want us lowering the 30 MB threshold across the
> board.  Lot of these people use gmail.com addresses and google allows 
for
> up to 25 MB - https://support.google.com/mail/answer/6584
>
> I think it's really interesting that google seems to use this 
inefficient
> small packet size for SSL, allows for 25MB emails, is a big proponent of
> SSL, and at the same time doesn't allow mails to take more than 15 
minutes
> to transfer.  Now that you've made things >much< more efficient on the
> ASSP
> side, I'm hoping that all will be okay.  I just get annoyed by
> inefficiency.
>
>
> I'm going to tryrunning with npSize of zero, the no queuing size set 
very
> high and see how that goes.  I want to insure that even the biggest 
emails
> are scanned for malware before hitting the inbox.
>
>
> I've never said this before, but it seems like google's doing it wrong. 
I
> think we've exhausted our options here.  If anyone has a Google
> engineering
> friend, or a friend of a friend, this might be a good time to have a
> little
> chat in an effort to reach someone who can either explain why they're 
use
> a
> 1.4 kB frame size for ssl packets, but a normal much bigger size for
> non-encrypted traffic or to get them to see an error in their ways and 
fix
> this!
>
> Thank you again for all of the discussion and explanation and of course
> the
> code changes which have made a huge difference.
>
> On Wed, Sep 28, 2016 at 12:59 PM, Thomas Eckardt
> <thomas.ecka...@thockar.com
> > wrote:
>
> > >I'm still afraid of running into frequent problems with 25mb
> attachments
> > though.
> >
> > 25MB - I don't know anyone who allows this. Sending a link to a 
download
> > is much smarter.
> > ASSP_AFC has an option to do this for your outgoing mails! -
> > ASSP_AFCWebScript
> >
> > >Can someone else run a debug test with Gmail and TLS to see if you're
> > seeing these tiny packets too?
> >
> > I've done it - same behavior - 1400 byte per SSL frame from gmail.
> >
> >
> > >Would it be an easy modification to show the negotiated cipher in the
> >
> > ASSP does this for years now - simply look in to the received header
> line
> > added by assp

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

updated to 2.5.2 16274_1 at CVS /test - a code correction was required.

Thomas







Von:Thomas Eckardt <thomas.ecka...@thockar.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  30.09.2016 13:32
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



2.5.2 16274 is available at CVS /test - there is no longer any issue on my 

system - but it would be nice, if I would get a positive feedback from 
anyone, before I'll publish it at the root folder of CVS.

Thaoms





Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  30.09.2016 03:00
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Tried again after hours without any better results.  Even NON-SSL
connections are going idle/damping.  Even TINY messages with just text.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 


individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

2.5.2 16274 is available at CVS /test - there is no longer any issue on my 
system - but it would be nice, if I would get a positive feedback from 
anyone, before I'll publish it at the root folder of CVS.

Thaoms





Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  30.09.2016 03:00
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Tried again after hours without any better results.  Even NON-SSL
connections are going idle/damping.  Even TINY messages with just text.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

Saw also some strange behavior. Let's see, if I can it get working much 
better.

Thomas





Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  30.09.2016 00:18
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



very good news, and some not so good news with 16273.

I watched this same 11MB attachment come across using the Shutdown_list
(about 15,000,000 bytes after all said and done).  It transferred in about
40 seconds.  As in OUTSTANDING SPEED.  As in, whatever you did has made a
HUGE HUGE difference.  (looking at the code, nice work.  clever)

HOWEVER, I had to revert as the email never seemed to complete.  *It then
just sat with the idle / damping time increasing indefinitely. * All SSL
connections did this too, fast transfer,then idle forever.  I didn't have
the time to test non-SSL connections.

I tried restarting for safe measure just to be sure.  Same problem.  Once 
I
went back to 16271, normal behavior resumed.

Seems close, but not quite right



On Thu, Sep 29, 2016 at 10:25 AM, cw <colin.war...@gmail.com> wrote:

> Hi Thomas,
> I moved up to 16270 following this thread of discussion but then had a 
day
> working away. I've come back to huge issues with delays, mails not going
> through and many, many of these in the logs:
>
> Info: unable to detect any running worker for a new connection - wait 
(max
> 30 seconds)
>
> When I say many, I have over 21,000 lines in today's log file. I also 
found
> the GUI unresponsive or not connecting at all and ASSP restarting quite
> regularly.
>
> I've dropped back to 16256 and things are instantly better. Do you think
> going up to 16273 might improve things over 16270 or am I better holding
> off for now?
> All the best,
> Colin.
>
> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt <
> thomas.ecka...@thockar.com>
> wrote:
>
> > I just released 2.5.2 build 16273 at CVS test folder
> >
> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
> >
> > This release should make a very large difference for SSL/TLS mails 
sent
> by
> > hosts that uses small SSL-frame size.
> >
> > Tell me your test results.
> >
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:K Post <nntp.p...@gmail.com>
> > An: ASSP development mailing list 
<assp-test@lists.sourceforge.net>
> > Datum:  28.09.2016 19:42
> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> > servers
> >
> >
> >
> > But I want a postman driving a Ferarri with monster truck tires that 
can
> > roll over the traffic (and if wishes are being granted, I'd prefer the
> car
> > in a deep blue instead of classic red).
> >
> > We regularly see people attaching large files or a bunch of smaller 
ones
> > that add up to a big email, I'm talking lots and lots of different 
people
> > from outside the organization sending to us, and this happens on a 
daily
> > basis.  It's especially popular with photos and huge scans multi-page
> > 600dpi (which people don't understand can be done at low resolution).
> > Often it's people sending in scanned official documents for us to 
review
> > an
> > help them.  They're not our staff, they're the people we help.  They 
have
> > a
> > tendency of not following any instructions, and ignore the fact that 
we
> > have a web based system for the process.  We can't control it and the
> > powers that be don't want us lowering the 30 MB threshold across the
> > board.  Lot of these people use gmail.com addresses and google allows
> for
> > up to 25 MB - https://support.google.com/mail/answer/6584
> >
> > I think it's really interesting that google seems to use this 
inefficient
> > small packet size for SSL, allows for 25MB emails, is a big proponent 
of
> > SSL, and at the same time doesn't allow mails to take more than 15
> minutes
> > to transfer.  Now that you've made things >much< more efficient on the
> > ASSP
> > side, I'm hoping that all will be okay.  I just get annoyed by
> > inefficiency.
> >
> >
> > I'm going to tryrunning with npSize of zero, the no queuing size set 
very
> > high and see how that goes.  I want to insure that even the biggest
> emails
> > are scanned for malware before hitting the inbox.
> >
> >
> > I've never said this before, but it seems like google's doing it 
wrong.
> I
> > think we've exhausted our options here.  If anyone has a Google
> > engineering
> > friend, or a friend of a friend, this might be a good time to have a
> >

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

Tried again after hours without any better results.  Even NON-SSL
connections are going idle/damping.  Even TINY messages with just text.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

very good news, and some not so good news with 16273.

I watched this same 11MB attachment come across using the Shutdown_list
(about 15,000,000 bytes after all said and done).  It transferred in about
40 seconds.  As in OUTSTANDING SPEED.  As in, whatever you did has made a
HUGE HUGE difference.  (looking at the code, nice work.  clever)

HOWEVER, I had to revert as the email never seemed to complete.  *It then
just sat with the idle / damping time increasing indefinitely. * All SSL
connections did this too, fast transfer,then idle forever.  I didn't have
the time to test non-SSL connections.

I tried restarting for safe measure just to be sure.  Same problem.  Once I
went back to 16271, normal behavior resumed.

Seems close, but not quite right



On Thu, Sep 29, 2016 at 10:25 AM, cw <colin.war...@gmail.com> wrote:

> Hi Thomas,
> I moved up to 16270 following this thread of discussion but then had a day
> working away. I've come back to huge issues with delays, mails not going
> through and many, many of these in the logs:
>
> Info: unable to detect any running worker for a new connection - wait (max
> 30 seconds)
>
> When I say many, I have over 21,000 lines in today's log file. I also found
> the GUI unresponsive or not connecting at all and ASSP restarting quite
> regularly.
>
> I've dropped back to 16256 and things are instantly better. Do you think
> going up to 16273 might improve things over 16270 or am I better holding
> off for now?
> All the best,
> Colin.
>
> On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt <
> thomas.ecka...@thockar.com>
> wrote:
>
> > I just released 2.5.2 build 16273 at CVS test folder
> >
> > http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
> >
> > This release should make a very large difference for SSL/TLS mails sent
> by
> > hosts that uses small SSL-frame size.
> >
> > Tell me your test results.
> >
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von:    K Post <nntp.p...@gmail.com>
> > An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> > Datum:  28.09.2016 19:42
> > Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> > servers
> >
> >
> >
> > But I want a postman driving a Ferarri with monster truck tires that can
> > roll over the traffic (and if wishes are being granted, I'd prefer the
> car
> > in a deep blue instead of classic red).
> >
> > We regularly see people attaching large files or a bunch of smaller ones
> > that add up to a big email, I'm talking lots and lots of different people
> > from outside the organization sending to us, and this happens on a daily
> > basis.  It's especially popular with photos and huge scans multi-page
> > 600dpi (which people don't understand can be done at low resolution).
> > Often it's people sending in scanned official documents for us to review
> > an
> > help them.  They're not our staff, they're the people we help.  They have
> > a
> > tendency of not following any instructions, and ignore the fact that we
> > have a web based system for the process.  We can't control it and the
> > powers that be don't want us lowering the 30 MB threshold across the
> > board.  Lot of these people use gmail.com addresses and google allows
> for
> > up to 25 MB - https://support.google.com/mail/answer/6584
> >
> > I think it's really interesting that google seems to use this inefficient
> > small packet size for SSL, allows for 25MB emails, is a big proponent of
> > SSL, and at the same time doesn't allow mails to take more than 15
> minutes
> > to transfer.  Now that you've made things >much< more efficient on the
> > ASSP
> > side, I'm hoping that all will be okay.  I just get annoyed by
> > inefficiency.
> >
> >
> > I'm going to tryrunning with npSize of zero, the no queuing size set very
> > high and see how that goes.  I want to insure that even the biggest
> emails
> > are scanned for malware before hitting the inbox.
> >
> >
> > I've never said this before, but it seems like google's doing it wrong.
> I
> > think we've exhausted our options here.  If anyone has a Google
> > engineering
> > friend, or a friend of a friend, this might be a good time to have a
> > little
> > chat in an effort to reach someone who can either explain why they're use
> > a
> > 1.4 kB frame size for ssl packets, but a normal much bigger size for
> > non-encrypted traffic or to get them to see an error in their ways and
> fix
> > this!
> >
> > Thank you again for all of the discussi

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[cw]

Hi Thomas,
I moved up to 16270 following this thread of discussion but then had a day
working away. I've come back to huge issues with delays, mails not going
through and many, many of these in the logs:

Info: unable to detect any running worker for a new connection - wait (max
30 seconds)

When I say many, I have over 21,000 lines in today's log file. I also found
the GUI unresponsive or not connecting at all and ASSP restarting quite
regularly.

I've dropped back to 16256 and things are instantly better. Do you think
going up to 16273 might improve things over 16270 or am I better holding
off for now?
All the best,
Colin.

On Thu, Sep 29, 2016 at 3:15 PM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:

> I just released 2.5.2 build 16273 at CVS test folder
>
> http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/
>
> This release should make a very large difference for SSL/TLS mails sent by
> hosts that uses small SSL-frame size.
>
> Tell me your test results.
>
>
> Thomas
>
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  28.09.2016 19:42
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> But I want a postman driving a Ferarri with monster truck tires that can
> roll over the traffic (and if wishes are being granted, I'd prefer the car
> in a deep blue instead of classic red).
>
> We regularly see people attaching large files or a bunch of smaller ones
> that add up to a big email, I'm talking lots and lots of different people
> from outside the organization sending to us, and this happens on a daily
> basis.  It's especially popular with photos and huge scans multi-page
> 600dpi (which people don't understand can be done at low resolution).
> Often it's people sending in scanned official documents for us to review
> an
> help them.  They're not our staff, they're the people we help.  They have
> a
> tendency of not following any instructions, and ignore the fact that we
> have a web based system for the process.  We can't control it and the
> powers that be don't want us lowering the 30 MB threshold across the
> board.  Lot of these people use gmail.com addresses and google allows for
> up to 25 MB - https://support.google.com/mail/answer/6584
>
> I think it's really interesting that google seems to use this inefficient
> small packet size for SSL, allows for 25MB emails, is a big proponent of
> SSL, and at the same time doesn't allow mails to take more than 15 minutes
> to transfer.  Now that you've made things >much< more efficient on the
> ASSP
> side, I'm hoping that all will be okay.  I just get annoyed by
> inefficiency.
>
>
> I'm going to tryrunning with npSize of zero, the no queuing size set very
> high and see how that goes.  I want to insure that even the biggest emails
> are scanned for malware before hitting the inbox.
>
>
> I've never said this before, but it seems like google's doing it wrong.  I
> think we've exhausted our options here.  If anyone has a Google
> engineering
> friend, or a friend of a friend, this might be a good time to have a
> little
> chat in an effort to reach someone who can either explain why they're use
> a
> 1.4 kB frame size for ssl packets, but a normal much bigger size for
> non-encrypted traffic or to get them to see an error in their ways and fix
> this!
>
> Thank you again for all of the discussion and explanation and of course
> the
> code changes which have made a huge difference.
>
> On Wed, Sep 28, 2016 at 12:59 PM, Thomas Eckardt
> <thomas.ecka...@thockar.com
> > wrote:
>
> > >I'm still afraid of running into frequent problems with 25mb
> attachments
> > though.
> >
> > 25MB - I don't know anyone who allows this. Sending a link to a download
> > is much smarter.
> > ASSP_AFC has an option to do this for your outgoing mails! -
> > ASSP_AFCWebScript
> >
> > >Can someone else run a debug test with Gmail and TLS to see if you're
> > seeing these tiny packets too?
> >
> > I've done it - same behavior - 1400 byte per SSL frame from gmail.
> >
> >
> > >Would it be an easy modification to show the negotiated cipher in the
> >
> > ASSP does this for years now - simply look in to the received header
> line
> > added by assp.
> >
> > -- -- -- Received: from vs10241.internet1.de ([158.181.49.77]
> > helo=vs10241.internet1.de)
> > by xx with SMTPS(TLSv1_2 ECDHE-RSA-AES256-GCM-SHA384)
> (2.5.2);
> > 24 Jul 2016 05:34:12 +0200
> >
> > if you set 'ConnectionLog' at least to verbose, you'll see the
> > 

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

I just released 2.5.2 build 16273 at CVS test folder

http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/

This release should make a very large difference for SSL/TLS mails sent by 
hosts that uses small SSL-frame size.

Tell me your test results.


Thomas





Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  28.09.2016 19:42
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



But I want a postman driving a Ferarri with monster truck tires that can
roll over the traffic (and if wishes are being granted, I'd prefer the car
in a deep blue instead of classic red).

We regularly see people attaching large files or a bunch of smaller ones
that add up to a big email, I'm talking lots and lots of different people
from outside the organization sending to us, and this happens on a daily
basis.  It's especially popular with photos and huge scans multi-page
600dpi (which people don't understand can be done at low resolution).
Often it's people sending in scanned official documents for us to review 
an
help them.  They're not our staff, they're the people we help.  They have 
a
tendency of not following any instructions, and ignore the fact that we
have a web based system for the process.  We can't control it and the
powers that be don't want us lowering the 30 MB threshold across the
board.  Lot of these people use gmail.com addresses and google allows for
up to 25 MB - https://support.google.com/mail/answer/6584

I think it's really interesting that google seems to use this inefficient
small packet size for SSL, allows for 25MB emails, is a big proponent of
SSL, and at the same time doesn't allow mails to take more than 15 minutes
to transfer.  Now that you've made things >much< more efficient on the 
ASSP
side, I'm hoping that all will be okay.  I just get annoyed by 
inefficiency.


I'm going to tryrunning with npSize of zero, the no queuing size set very
high and see how that goes.  I want to insure that even the biggest emails
are scanned for malware before hitting the inbox.


I've never said this before, but it seems like google's doing it wrong.  I
think we've exhausted our options here.  If anyone has a Google 
engineering
friend, or a friend of a friend, this might be a good time to have a 
little
chat in an effort to reach someone who can either explain why they're use 
a
1.4 kB frame size for ssl packets, but a normal much bigger size for
non-encrypted traffic or to get them to see an error in their ways and fix
this!

Thank you again for all of the discussion and explanation and of course 
the
code changes which have made a huge difference.

On Wed, Sep 28, 2016 at 12:59 PM, Thomas Eckardt 
<thomas.ecka...@thockar.com
> wrote:

> >I'm still afraid of running into frequent problems with 25mb 
attachments
> though.
>
> 25MB - I don't know anyone who allows this. Sending a link to a download
> is much smarter.
> ASSP_AFC has an option to do this for your outgoing mails! -
> ASSP_AFCWebScript
>
> >Can someone else run a debug test with Gmail and TLS to see if you're
> seeing these tiny packets too?
>
> I've done it - same behavior - 1400 byte per SSL frame from gmail.
>
>
> >Would it be an easy modification to show the negotiated cipher in the
>
> ASSP does this for years now - simply look in to the received header 
line
> added by assp.
>
> -- -- -- Received: from vs10241.internet1.de ([158.181.49.77]
> helo=vs10241.internet1.de)
> by xx with SMTPS(TLSv1_2 ECDHE-RSA-AES256-GCM-SHA384) 
(2.5.2);
> 24 Jul 2016 05:34:12 +0200
>
> if you set 'ConnectionLog' at least to verbose, you'll see the
> SSL-negotiation results in maillog.txt.
>
> info: started TLS-SSL session for client $cliIP - using
> $Con{$ssls}->{sslversion} , $Con{$ssls}->{sslcipher}
>
> >what other options are there?
>
> Set 'EnableHighPerformance' to very high.
>
> >Are you sure that negotiating a lesser cipher with Gmail wouldn't have
> them switch to sending larger SSL packets?
>
> Yes, I'm 100% sure.
>
> An SSL peer is free to use any SSL frame size between 1 byte and 16kB.
> There is no SSL-negotiation parameter for min or max frame size.
> If gmail.com sends 1.4kB frames, they are free to do it this way. There 
is
> no technical way to force them to send larger frames.
>
> >If neverQueue is set to 12MB, am I correct in saying that we're open to 
a
> targeted exploit, say an .exe in a .zip as long as the email is more 
than
> 12MB?
>
> Yes, you are correct.
>
> >or is that just after the whole message is received?
>
> Yes, after the complete mail is received.
>
> It is like it is - if the postman with his Ferrari is in a traffic jam,
> you'll have to wait longer for your letter or you'll get it the next 
day.

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

But I want a postman driving a Ferarri with monster truck tires that can
roll over the traffic (and if wishes are being granted, I'd prefer the car
in a deep blue instead of classic red).

We regularly see people attaching large files or a bunch of smaller ones
that add up to a big email, I'm talking lots and lots of different people
from outside the organization sending to us, and this happens on a daily
basis.  It's especially popular with photos and huge scans multi-page
600dpi (which people don't understand can be done at low resolution).
Often it's people sending in scanned official documents for us to review an
help them.  They're not our staff, they're the people we help.  They have a
tendency of not following any instructions, and ignore the fact that we
have a web based system for the process.  We can't control it and the
powers that be don't want us lowering the 30 MB threshold across the
board.  Lot of these people use gmail.com addresses and google allows for
up to 25 MB - https://support.google.com/mail/answer/6584

I think it's really interesting that google seems to use this inefficient
small packet size for SSL, allows for 25MB emails, is a big proponent of
SSL, and at the same time doesn't allow mails to take more than 15 minutes
to transfer.  Now that you've made things >much< more efficient on the ASSP
side, I'm hoping that all will be okay.  I just get annoyed by inefficiency.


I'm going to tryrunning with npSize of zero, the no queuing size set very
high and see how that goes.  I want to insure that even the biggest emails
are scanned for malware before hitting the inbox.


I've never said this before, but it seems like google's doing it wrong.  I
think we've exhausted our options here.  If anyone has a Google engineering
friend, or a friend of a friend, this might be a good time to have a little
chat in an effort to reach someone who can either explain why they're use a
1.4 kB frame size for ssl packets, but a normal much bigger size for
non-encrypted traffic or to get them to see an error in their ways and fix
this!

Thank you again for all of the discussion and explanation and of course the
code changes which have made a huge difference.

On Wed, Sep 28, 2016 at 12:59 PM, Thomas Eckardt <thomas.ecka...@thockar.com
> wrote:

> >I'm still afraid of running into frequent problems with 25mb attachments
> though.
>
> 25MB - I don't know anyone who allows this. Sending a link to a download
> is much smarter.
> ASSP_AFC has an option to do this for your outgoing mails! -
> ASSP_AFCWebScript
>
> >Can someone else run a debug test with Gmail and TLS to see if you're
> seeing these tiny packets too?
>
> I've done it - same behavior - 1400 byte per SSL frame from gmail.
>
>
> >Would it be an easy modification to show the negotiated cipher in the
>
> ASSP does this for years now - simply look in to the received header line
> added by assp.
>
> -- -- -- Received: from vs10241.internet1.de ([158.181.49.77]
> helo=vs10241.internet1.de)
> by xx with SMTPS(TLSv1_2 ECDHE-RSA-AES256-GCM-SHA384) (2.5.2);
> 24 Jul 2016 05:34:12 +0200
>
> if you set 'ConnectionLog' at least to verbose, you'll see the
> SSL-negotiation results in maillog.txt.
>
> info: started TLS-SSL session for client $cliIP - using
> $Con{$ssls}->{sslversion} , $Con{$ssls}->{sslcipher}
>
> >what other options are there?
>
> Set 'EnableHighPerformance' to very high.
>
> >Are you sure that negotiating a lesser cipher with Gmail wouldn't have
> them switch to sending larger SSL packets?
>
> Yes, I'm 100% sure.
>
> An SSL peer is free to use any SSL frame size between 1 byte and 16kB.
> There is no SSL-negotiation parameter for min or max frame size.
> If gmail.com sends 1.4kB frames, they are free to do it this way. There is
> no technical way to force them to send larger frames.
>
> >If neverQueue is set to 12MB, am I correct in saying that we're open to a
> targeted exploit, say an .exe in a .zip as long as the email is more than
> 12MB?
>
> Yes, you are correct.
>
> >or is that just after the whole message is received?
>
> Yes, after the complete mail is received.
>
> It is like it is - if the postman with his Ferrari is in a traffic jam,
> you'll have to wait longer for your letter or you'll get it the next day.
>
>
> Thomas
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  28.09.2016 17:03
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Thanks again for the detailed explanation.
>
> 10% faster is terrific, and it's more than 50% faster since before 16270.
> I'm still afraid of running into frequent problems with 25mb attach

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>I'm still afraid of running into frequent problems with 25mb attachments
though.

25MB - I don't know anyone who allows this. Sending a link to a download 
is much smarter.
ASSP_AFC has an option to do this for your outgoing mails! - 
ASSP_AFCWebScript

>Can someone else run a debug test with Gmail and TLS to see if you're
seeing these tiny packets too?

I've done it - same behavior - 1400 byte per SSL frame from gmail.


>Would it be an easy modification to show the negotiated cipher in the

ASSP does this for years now - simply look in to the received header line 
added by assp.

-- -- -- Received: from vs10241.internet1.de ([158.181.49.77] 
helo=vs10241.internet1.de)
by xx with SMTPS(TLSv1_2 ECDHE-RSA-AES256-GCM-SHA384) (2.5.2); 
24 Jul 2016 05:34:12 +0200

if you set 'ConnectionLog' at least to verbose, you'll see the 
SSL-negotiation results in maillog.txt.

info: started TLS-SSL session for client $cliIP - using 
$Con{$ssls}->{sslversion} , $Con{$ssls}->{sslcipher}

>what other options are there?

Set 'EnableHighPerformance' to very high.

>Are you sure that negotiating a lesser cipher with Gmail wouldn't have
them switch to sending larger SSL packets?

Yes, I'm 100% sure.

An SSL peer is free to use any SSL frame size between 1 byte and 16kB. 
There is no SSL-negotiation parameter for min or max frame size.
If gmail.com sends 1.4kB frames, they are free to do it this way. There is 
no technical way to force them to send larger frames.

>If neverQueue is set to 12MB, am I correct in saying that we're open to a
targeted exploit, say an .exe in a .zip as long as the email is more than
12MB?

Yes, you are correct.

>or is that just after the whole message is received?

Yes, after the complete mail is received.

It is like it is - if the postman with his Ferrari is in a traffic jam, 
you'll have to wait longer for your letter or you'll get it the next day.


Thomas



Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  28.09.2016 17:03
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Thanks again for the detailed explanation.

10% faster is terrific, and it's more than 50% faster since before 16270.
I'm still afraid of running into frequent problems with 25mb attachments
though.  Yes, that's too big for email in our opinion, but Gmail allows it
as do most ISP's these days.  It's not terribly unusual for us to have a a
person try to send and email with four or five 4 MB photos - and we can't
stop that trend.  That'll translate into a message somewhere in the range
of 22-25 MB.  I'm not sure if we're fast enough to >consistently< receive
these.  And on top of that, if it's an outside big wig, emailing an inside
big wig, they don't have 15 minutes to waste waiting for the big email to
arrive.  Then they yell at me, which is why I keep asking more of these
questions -- and there's a lot of them here (sorry)


*Are you sure that negotiating a lesser cipher with Gmail wouldn't have
them switch to sending larger SSL packets? * I have a solid high level
understanding of encryption, but once you get down the the packet level,
I'm a bit out of my depth, despite what I've learned from your excellent
explanations.


Can someone else run a debug test with Gmail and TLS to see if you're
seeing these tiny packets too?  I keep going back to *not being able to
believe that Google doing something that was any slower than necessary. * 
They
try to optimize everything and are huge proponents of SSL everywhere.  If
they're sending tons of tiny un-optimized packets, there's got to be a
reason or cause.  I can't imagine that it's a default and deliberate
setting.  I tried calling US support, but this is a network engineering
type of issue - no chance of me getting through to someone there who even
understands the question.

The npSize option and neverQueueSize, only affects things after the whole
message is received right? * It's not like it's doing something
significantly extra at each SSL cylce is it except using up RAM right??*
 My unscientific tests are shower marginally slower speeds when receiving
large email from Google with TLS on and neverQueueSize set crazy high, so
maybe I'm wrong.  Being that this same message through outlook.com only
takes 30 seconds, even with npSize set to 0 (unlimited) and neverQueueSize
= 999,000,000, I'm wondering if the processing power and ram availability
(24 GB) on my assp box combined with low usage (only averaging 5000
messages a day) might be enough to have neverQueueSize really high to
always scan everything.  I'll play around there, but if neverQueueSize is
99MB, will things like ASSP_AFCDetectSpamAttachRe be slow WHILE the 
message
is transferring, or is that just after the whole message is received?

If neverQueue is set to 12MB, am I correct in saying that we're open to a
targeted exploit, say an .exe in a .zip as long as th

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

nstallations are incapable of receiving large TLS emails from Google
regardless of the installation's processing power, RAM, and bandwidth?



On Wed, Sep 28, 2016 at 4:26 AM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:

> >The same email now took only 269 seconds.
>
> This is OK for googles behavior - the 1440 byte SSL frame. Around 300s was
> expected by me for this mail - hmm.. it is 10% better - nice!
>
> Take the following math.
>
> mail size = 15000 kB
> google frame size = 1.44 kB
>
> required assp loop cycles = mail size / frame size = 10.400
>
> mail size = 15000 kB
> outlook.com frame size = 16 kB
>
> required assp loop cycles = mail size / frame size = 940
>
> >I really wonder if I need to tweak my cipher_list.
>
> No!
> Again Ken, the SSL parameters are NOT the problem on your system. Your
> debug log shows a socket read time of max ~ 0.5 milliseconds (typical ~0.3
> ms) for SSL. This read operation includes the time required for the
> decryption of the data. This is very very fast!
> It is simply the amazing count (10.400) of read and process operations
> (cycles) required by assp for such a mail, that causes the overall slow
> mail receive.
>
> >Is ClamAV scanning skipped too?
> Yes.
>
> >Could the plugins be run on the full mail after receipt, regardless of
> size?
>
> Override the config parameters. Keep in mind that 'npSize' may also
> involved in skipping or processing some mail body checks.
>
> >Isn't DKIM checking just a one time thing and not intensive?
>
> The full DKIM check is very intensive. It requires to calculate an RSA/SHA
> hash over the complete mail.
> DCC and Razor are doing something similar.
> ASSP_AFC would make all checks (ClamAV, FileScan, content checks with
> several regular expression, decompression of attachments ) for the
> complete mail. It parses the complete mail at once with Email::MIME. This
> requires a huge amount of memory. Not a big deal on a 64bit OS with 8GB
> RAM and several CPU cores - who can !?
> All these can be done for any mail size, if the system is able to process
> the amount of data fast enough.
> On most havy load systems, the 12.000.000 will be to large and may lead in
> to stucking workers.
>
> ASSP has this set of config parameters - change them to your need - try -
> and if does not work switch back - nothing more easy.
>
> Thomas
>
>
>
> Von:    K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  27.09.2016 21:08
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Our primary internet connection went down again (nothing to do with ASSP)
> which gave me the opportunity to replace 16270 with 16271.  Nothing like
> making lemonade out of lemons...
>
> The same email now took only 269 seconds.  That's about 15x longer than
> with TLS off, but WOW that's way better than it was before.
> I also tried with the blank cipher list, no notable difference
> And with the SSL buffers set to 0 (64 MB), again without a speed
> difference.
>
> SO- you've made a real difference here!!   Is there more optimization to
> be
> made?
>
> The rub is that the exact same message sent through Outlook.com to us,
> took
> exactly 30 seconds, just a 50% overhead when compared to the 19 seconds
> for
> a non-TLS message of the same size, instead of a 1500% overhead for
> encryption when receiving from google.
>
> *Is there some magical debug switch that I could turn on to see what
> encryption Outlook.com is using and compare that to what Google's
> connection to us with?*  I think prohibiting whatever the slow cipher that
> google's using (probably a really strong one) might make the final bit of
> difference.
>
> I'm breathing so much easier now!!  Thank you.
>
>
> On Tue, Sep 27, 2016 at 2:08 PM, K Post <nntp.p...@gmail.com> wrote:
>
> > Despite all the problems we have with personalities and policies  in our
> > organization, the infrastructure is pretty solid.  MTU's are set
> correctly,
> > no fragmentation, no jitter.   There's low latency across the board, and
> > really low bandwidth usage.  If we sent 1000 mails a day, it's a lot.
> >
> > ...and yes, they even pay their bills, though not me very well :)
> >
> > I think it's which SSL algorithm is being used that's at least partially
> > to blame.  I have:
> > SSL_Version: SSLv23:!SSLv3:!SSLv2
> > SSL_Cipher_list: kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:RC4
> > :!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!kECDH:!
> > CAMELLIA128:!IDEA:!SEED

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>The same email now took only 269 seconds.

This is OK for googles behavior - the 1440 byte SSL frame. Around 300s was 
expected by me for this mail - hmm.. it is 10% better - nice!

Take the following math.

mail size = 15000 kB
google frame size = 1.44 kB

required assp loop cycles = mail size / frame size = 10.400

mail size = 15000 kB
outlook.com frame size = 16 kB

required assp loop cycles = mail size / frame size = 940

>I really wonder if I need to tweak my cipher_list.

No!
Again Ken, the SSL parameters are NOT the problem on your system. Your 
debug log shows a socket read time of max ~ 0.5 milliseconds (typical ~0.3 
ms) for SSL. This read operation includes the time required for the 
decryption of the data. This is very very fast!
It is simply the amazing count (10.400) of read and process operations 
(cycles) required by assp for such a mail, that causes the overall slow 
mail receive.

>Is ClamAV scanning skipped too?
Yes.

>Could the plugins be run on the full mail after receipt, regardless of 
size?

Override the config parameters. Keep in mind that 'npSize' may also 
involved in skipping or processing some mail body checks.

>Isn't DKIM checking just a one time thing and not intensive?

The full DKIM check is very intensive. It requires to calculate an RSA/SHA 
hash over the complete mail.
DCC and Razor are doing something similar.
ASSP_AFC would make all checks (ClamAV, FileScan, content checks with 
several regular expression, decompression of attachments ) for the 
complete mail. It parses the complete mail at once with Email::MIME. This 
requires a huge amount of memory. Not a big deal on a 64bit OS with 8GB 
RAM and several CPU cores - who can !?
All these can be done for any mail size, if the system is able to process 
the amount of data fast enough.
On most havy load systems, the 12.000.000 will be to large and may lead in 
to stucking workers. 

ASSP has this set of config parameters - change them to your need - try - 
and if does not work switch back - nothing more easy.

Thomas



Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  27.09.2016 21:08
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Our primary internet connection went down again (nothing to do with ASSP)
which gave me the opportunity to replace 16270 with 16271.  Nothing like
making lemonade out of lemons...

The same email now took only 269 seconds.  That's about 15x longer than
with TLS off, but WOW that's way better than it was before.
I also tried with the blank cipher list, no notable difference
And with the SSL buffers set to 0 (64 MB), again without a speed 
difference.

SO- you've made a real difference here!!   Is there more optimization to 
be
made?

The rub is that the exact same message sent through Outlook.com to us, 
took
exactly 30 seconds, just a 50% overhead when compared to the 19 seconds 
for
a non-TLS message of the same size, instead of a 1500% overhead for
encryption when receiving from google.

*Is there some magical debug switch that I could turn on to see what
encryption Outlook.com is using and compare that to what Google's
connection to us with?*  I think prohibiting whatever the slow cipher that
google's using (probably a really strong one) might make the final bit of
difference.

I'm breathing so much easier now!!  Thank you.


On Tue, Sep 27, 2016 at 2:08 PM, K Post <nntp.p...@gmail.com> wrote:

> Despite all the problems we have with personalities and policies  in our
> organization, the infrastructure is pretty solid.  MTU's are set 
correctly,
> no fragmentation, no jitter.   There's low latency across the board, and
> really low bandwidth usage.  If we sent 1000 mails a day, it's a lot.
>
> ...and yes, they even pay their bills, though not me very well :)
>
> I think it's which SSL algorithm is being used that's at least partially
> to blame.  I have:
> SSL_Version: SSLv23:!SSLv3:!SSLv2
> SSL_Cipher_list: kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:RC4
> :!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!kECDH:!
> CAMELLIA128:!IDEA:!SEED
>
> I tried the default wih SSL_Cipher_List blank before, I don't think 
there
> was a difference (but I've played with so many settings, I really don't
> remember)
>
> And last, on the SSL buffer size.  If set to zero in the gui, on windows
> 2012, it says in green that it's set to 64 MB.  I follow what you're 
saying
> about it readying 4x 16 Kb without a loop cycle.  Is that a good or bad
> thing though?
>
>
>
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any fil

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

The more I look at this, the more I'm guessing Outlook.com is using some
pathetically low cipher and Google's not. One's putting stress on the
system or is just slow, the other isn't.  I'd love to know how to tell what
they're using.   I really wonder if I need to tweak my cipher_list.

ASSP logs the STARTTLS request

info: got STARTTLS request from 209.85.223.182

Could we have it put what cipher's being used there or more useful
encryption info?

Also, with the new test version 16271, and it's default neverqueueSize of
1200 bytes, why would DKIM be skipped?  Isn't DKIM checking just a one
time thing and not intensive?

info: message is too large ( SIZE 15700405 byte > neverQueueSize 1200
byte) to be queued for further internal processing! Skipping DKIM, Plugins
and charset conversion.

I'm also afraid of plugins being skipped - no AFC for large emails.  That
scares me.  Is ClamAV scanning skipped too?  Could the plugins be run on
the full mail after receipt, regardless of size?  I know I can override
this limit in ASSP_Correction, but you've obviously coded this for a
reason.Any way to get a happy medium here (speed but full
functionality)?



On Tue, Sep 27, 2016 at 3:07 PM, K Post  wrote:

> Our primary internet connection went down again (nothing to do with ASSP)
> which gave me the opportunity to replace 16270 with 16271.  Nothing like
> making lemonade out of lemons...
>
> The same email now took only 269 seconds.  That's about 15x longer than
> with TLS off, but WOW that's way better than it was before.
> I also tried with the blank cipher list, no notable difference
> And with the SSL buffers set to 0 (64 MB), again without a speed
> difference.
>
> SO- you've made a real difference here!!   Is there more optimization to
> be made?
>
> The rub is that the exact same message sent through Outlook.com to us,
> took exactly 30 seconds, just a 50% overhead when compared to the 19
> seconds for a non-TLS message of the same size, instead of a 1500% overhead
> for encryption when receiving from google.
>
> *Is there some magical debug switch that I could turn on to see what
> encryption Outlook.com is using and compare that to what Google's
> connection to us with?*  I think prohibiting whatever the slow cipher
> that google's using (probably a really strong one) might make the final bit
> of difference.
>
> I'm breathing so much easier now!!  Thank you.
>
>
> On Tue, Sep 27, 2016 at 2:08 PM, K Post  wrote:
>
>> Despite all the problems we have with personalities and policies  in our
>> organization, the infrastructure is pretty solid.  MTU's are set correctly,
>> no fragmentation, no jitter.   There's low latency across the board, and
>> really low bandwidth usage.  If we sent 1000 mails a day, it's a lot.
>>
>> ...and yes, they even pay their bills, though not me very well :)
>>
>> I think it's which SSL algorithm is being used that's at least partially
>> to blame.  I have:
>> SSL_Version: SSLv23:!SSLv3:!SSLv2
>> SSL_Cipher_list: kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:RC4
>> :!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!kECDH:!C
>> AMELLIA128:!IDEA:!SEED
>>
>> I tried the default wih SSL_Cipher_List blank before, I don't think there
>> was a difference (but I've played with so many settings, I really don't
>> remember)
>>
>> And last, on the SSL buffer size.  If set to zero in the gui, on windows
>> 2012, it says in green that it's set to 64 MB.  I follow what you're saying
>> about it readying 4x 16 Kb without a loop cycle.  Is that a good or bad
>> thing though?
>>
>>
>>
>
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

Our primary internet connection went down again (nothing to do with ASSP)
which gave me the opportunity to replace 16270 with 16271.  Nothing like
making lemonade out of lemons...

The same email now took only 269 seconds.  That's about 15x longer than
with TLS off, but WOW that's way better than it was before.
I also tried with the blank cipher list, no notable difference
And with the SSL buffers set to 0 (64 MB), again without a speed difference.

SO- you've made a real difference here!!   Is there more optimization to be
made?

The rub is that the exact same message sent through Outlook.com to us, took
exactly 30 seconds, just a 50% overhead when compared to the 19 seconds for
a non-TLS message of the same size, instead of a 1500% overhead for
encryption when receiving from google.

*Is there some magical debug switch that I could turn on to see what
encryption Outlook.com is using and compare that to what Google's
connection to us with?*  I think prohibiting whatever the slow cipher that
google's using (probably a really strong one) might make the final bit of
difference.

I'm breathing so much easier now!!  Thank you.


On Tue, Sep 27, 2016 at 2:08 PM, K Post  wrote:

> Despite all the problems we have with personalities and policies  in our
> organization, the infrastructure is pretty solid.  MTU's are set correctly,
> no fragmentation, no jitter.   There's low latency across the board, and
> really low bandwidth usage.  If we sent 1000 mails a day, it's a lot.
>
> ...and yes, they even pay their bills, though not me very well :)
>
> I think it's which SSL algorithm is being used that's at least partially
> to blame.  I have:
> SSL_Version: SSLv23:!SSLv3:!SSLv2
> SSL_Cipher_list: kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:RC4
> :!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!kECDH:!
> CAMELLIA128:!IDEA:!SEED
>
> I tried the default wih SSL_Cipher_List blank before, I don't think there
> was a difference (but I've played with so many settings, I really don't
> remember)
>
> And last, on the SSL buffer size.  If set to zero in the gui, on windows
> 2012, it says in green that it's set to 64 MB.  I follow what you're saying
> about it readying 4x 16 Kb without a loop cycle.  Is that a good or bad
> thing though?
>
>
>
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

Despite all the problems we have with personalities and policies  in our
organization, the infrastructure is pretty solid.  MTU's are set correctly,
no fragmentation, no jitter.   There's low latency across the board, and
really low bandwidth usage.  If we sent 1000 mails a day, it's a lot.

...and yes, they even pay their bills, though not me very well :)

I think it's which SSL algorithm is being used that's at least partially to
blame.  I have:
SSL_Version: SSLv23:!SSLv3:!SSLv2
SSL_Cipher_list: kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:
RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!
kECDH:!CAMELLIA128:!IDEA:!SEED

I tried the default wih SSL_Cipher_List blank before, I don't think there
was a difference (but I've played with so many settings, I really don't
remember)

And last, on the SSL buffer size.  If set to zero in the gui, on windows
2012, it says in green that it's set to 64 MB.  I follow what you're saying
about it readying 4x 16 Kb without a loop cycle.  Is that a good or bad
thing though?
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>You said that the max for all os is 16kB, so it
>seems like ASSP should insure this isn't exceeded.

This is a limit for all SSL connections covered by a RFC. No one can 
exceed it.
If this is ever changes - assp reads from SSL sockets until the SSL buffer 
is empty This is done, because if there are bytes left in the SSL buffer, 
not the server nor the client are able to do a SSL-renegotiation. If this 
limit is ever set to 32kB and anyone is using this, assp will read two 
times 16kB without a loop cycle - or I've change the code.

>But both messages are from Google, so I think they either would or 
wouldn't
>send size, and it wouldn't be dependent on SSL. ( I turn TLS on and off 
for
>Google using NoTLS ip ranges which I get from their SPF)

Who knows? But it is possible.

>We need to start a campaign to have Google send more than 1440 bytes per 
SSL frame. 

good luck :)

It may be a better idea to call the google support of your country and to 
ask - possibly you'll get a helpfull answer after some time.

>Any chance that it's something they have that (seemingly pretty low) 
limit set just for good old me?

- anything in your infrastructure (too low MTU , high paket fragmentation)
- the negotiated SSL parameters
- bad IP reputation
- abused google DNS servers
- unpayed bills :)

stop! now it is getting corny :)

Thomas




Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  27.09.2016 16:39
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Consolidated replies below to a couple of your messages Thomas.


On Tue, Sep 27, 2016 at 8:34 AM, Thomas Eckardt 
<thomas.ecka...@thockar.com>
wrote:

> >an email

>with 11 MB of attachments takes 19 seconds with TLS turned off, and with

>TLS on 662 seconds.


> What if the message SIZE announcement is missing (not sent by google), 
if

TLS is turned off?

You'll get exactly this behavior.

But both messages are from Google, so I think they either would or 
wouldn't
send size, and it wouldn't be dependent on SSL. ( I turn TLS on and off 
for
Google using NoTLS ip ranges which I get from their SPF)

Hurry up! Close all doors and windows - Murphy has left your IT rooms!
>
He'll be back.  That jerk seems to be able to walk through walls.

The default TCP output buffer for a socket on windows differs from version 
to
> version.

w2k3 - 8kB
>
w2K8R2 - 64kB (with some dynamics)
>
w2k12R2 - not sure, but at least 64kB with default dynamics
>
SSL - 16kB encrypted data maximum on all OS
>

I'm running windows 2012r2.  I just noticed that I had TCPBufferSize set 
to
sslrcv = 0, sslsnd = 0.


Under normal conditions any setting here will be not required. But, if you
notice a bad SSL transmission performance in relation to the speed of plan
TCP sockets, it may help to set both SSL buffer size to the size of the
according system TCP buffer.
like: sslrcv = 0, sslsnd = 0

I removed this setting and tested with 16270 (not the latest) and a slight
improvement.  Again, this is just one test though, I don't know if that
really made a difference or if this one email was just faster. Whatever 
the
case, this test was better, but still too slow at 550 seconds.

What I wanted to let you know here is that the GUI at least says that the
SSL buffer size is set to 64kB if you put sslrcv = 0, sslsnd = 0 on my
system (which is consistent with what you said would happen, set to max 
tcp
buffer size for system).  You said that the max for all os is 16kB, so it
seems like ASSP should insure this isn't exceeded.  It might already
internally, but doesn't indicate as such for the green message in the GUI
when changes are applied.  Might just be a display issue vs a functional
one.


We need to start a campaign to have Google send more than 1440 bytes per
SSL frame.  Why would they do that?!?  Any chance that it's something they
have that (seemingly pretty low) limit set just for good old me?   And if
others tend to send a much larger SSL frame, that would explain the speed
disparity between email sources over TLS!


I can't test today's new version right now, but absolutely will ASAP.
Can't disrupt email flow at all during the day, and especially not today 
as
tempers are recovering from a 2 hour long ISP outage earlier.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple 

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

Consolidated replies below to a couple of your messages Thomas.


On Tue, Sep 27, 2016 at 8:34 AM, Thomas Eckardt 
wrote:

> >an email

>with 11 MB of attachments takes 19 seconds with TLS turned off, and with

>TLS on 662 seconds.


> What if the message SIZE announcement is missing (not sent by google), if

TLS is turned off?

You'll get exactly this behavior.

But both messages are from Google, so I think they either would or wouldn't
send size, and it wouldn't be dependent on SSL. ( I turn TLS on and off for
Google using NoTLS ip ranges which I get from their SPF)

Hurry up! Close all doors and windows - Murphy has left your IT rooms!
>
He'll be back.  That jerk seems to be able to walk through walls.

The default TCP output buffer for a socket on windows differs from version to
> version.

w2k3 - 8kB
>
w2K8R2 - 64kB (with some dynamics)
>
w2k12R2 - not sure, but at least 64kB with default dynamics
>
SSL - 16kB encrypted data maximum on all OS
>

I'm running windows 2012r2.  I just noticed that I had TCPBufferSize set to
sslrcv = 0, sslsnd = 0.


Under normal conditions any setting here will be not required. But, if you
notice a bad SSL transmission performance in relation to the speed of plan
TCP sockets, it may help to set both SSL buffer size to the size of the
according system TCP buffer.
like: sslrcv = 0, sslsnd = 0

I removed this setting and tested with 16270 (not the latest) and a slight
improvement.  Again, this is just one test though, I don't know if that
really made a difference or if this one email was just faster. Whatever the
case, this test was better, but still too slow at 550 seconds.

What I wanted to let you know here is that the GUI at least says that the
SSL buffer size is set to 64kB if you put sslrcv = 0, sslsnd = 0 on my
system (which is consistent with what you said would happen, set to max tcp
buffer size for system).  You said that the max for all os is 16kB, so it
seems like ASSP should insure this isn't exceeded.  It might already
internally, but doesn't indicate as such for the green message in the GUI
when changes are applied.  Might just be a display issue vs a functional
one.


We need to start a campaign to have Google send more than 1440 bytes per
SSL frame.  Why would they do that?!?  Any chance that it's something they
have that (seemingly pretty low) limit set just for good old me?   And if
others tend to send a much larger SSL frame, that would explain the speed
disparity between email sources over TLS!


I can't test today's new version right now, but absolutely will ASAP.
Can't disrupt email flow at all during the day, and especially not today as
tempers are recovering from a 2 hour long ISP outage earlier.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

Ken, please undo the changes in 'assp/lib/CorrectASSPcfg.pm' for the test 
with build 16271.

$main::neverQueueSize = 1200;

Thomas



Von:Thomas Eckardt <thomas.ecka...@thockar.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  26.09.2016 10:03
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



First, thank you for the debug file.

There is one big problem. The debug file explains the general behavior of 
the slowing down connection while the data size is growing.
It not explains, why this should only happens at connections from 
gmail.com and only if TLS is used.

looking at the following timeline - the *** lines are from me and are 
showing the count of read-socketcalls within this second


Sep-23-16 21:14:37 [Worker_2]  to: 

testtls@[[ OUR DOMAIN ]].org info: message is too large ( > neverQueueSize 

1200 byte) to be queued for further internal processing! Skipping 
DKMI, Plugins and charset conversion. 
***socketcalls per second (each 1440 byte) 8
...
Sep-23-16 21:22:45 [Worker_2] Sep-23-16 21:14:37 [Worker_2] 
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  24.09.2016 04:05
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



in the debug that I'm about to send you, I'm seeing lines like:

periodically in the file.
Don't know if that's notable or not.

On Fri, Sep 23, 2016 at 9:43 PM, K Post <nntp.p...@gmail.com> wrote:

> I'll get you a sample debug asap.
>
> FYI, I forgot to mention yesterday - I've noticed times (not always) 
when
> watching the SMTP Connections Window with large test gmail emails where 
a
> message will start transferring and after some time (7-8 minutes, maybe 
a
> little less), that google will connect a second time and start 
transferring
> the message again, even though the first one is still being received.
>
> The first message completed after say 12 minutes, and then 3-4 minutes
> after that, the 2nd google connection disconnects and doesn't try again
> (nor should it).
>
> This does NOT happen every time, and I can't find find a reason why it
> would do this on occasion for some but not other big messages.
>
>
> On Fri, Sep 23, 2016 at 7:02 AM, Thomas Eckardt <
> thomas.ecka...@thockar.com> wrote:
>
>> Thank you Ken.
>>
>> Please would you send me the debug log for the large (15MB) TLS mail as
>> ZIP or make it available to me for download anywhere, if it is too 
large
>> for SMTP.
>>
>> Thomas
>>
>>
>
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 


individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>Eagerly awaiting the next version!  Thanks.

pour yourself into work :):)

Tell me if this works better. I have it running in production.

published at CVS/test/assp.pl.gz - version 2.5.2 build 16271

http://assp.cvs.sourceforge.net/viewvc/assp/assp2/test/

Thomas





Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  27.09.2016 14:15
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



My abbreviated notes:

1) What a terrific explanation!!  Thank you Thomas not just for the write
up, but for taking all the time necessary to figure all this out

2) IO::Socket::SSL and Net::SSLeay
Both of mine are the latest versions (newer than the recommended / minimum
that shows in the info gui)

3) A different / additional problem with TLS?  Can we tell what algorithm
Google uses?

So we wasted allot of time looking for a TLS/SSL issue - life is not easy.

***
Back to the 'TLS' relation. Forget it - this issue has totaly nothing to
do with TLS/SSL!
***


What you describe elsewhere makes sense, BUT looking at my tests, an email
with 11 MB of attachments takes 19 seconds with TLS turned off, and with
TLS on 662 seconds.   That's 34 times longer.  I know SSL takes processing
power, but that's a huge overhead no?

*Can we tell from the logs / debug / or some other test what TLS algorithm
google is using*?  Since I only see this horrific slowdown with Google and
TLS, I wonder if I might only allow a (slightly?) lesser algorithm that's
still secure but not quite as intensive.

Note that identical hardware runs Apache on Windows with OpenSSL, Exchange
(SSL), and others without any noticeable slowdown due to the encryption.

You've clearly identified a major different problem with grow, etc, but 
I'm
not so sure that I'm not also experiencing a SSL/TLS problem.



Eagerly awaiting the next version!  Thanks.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

TLS/SSL is much slower than plain communication. The reason is simple - 
the IO buffer size.

The default TCP output buffer for a socket on windows differs from version 
to version.

w2k3 - 8kB
w2K8R2 - 64kB (with some dynamics)
w2k12R2 - not sure, but at least 64kB with default dynamics

SSL - 16kB encrypted data maximum on all OS

If you have a look in to your debug logs, you'll see that google sends 
only 1440 byte per SSL frame, which is one decrypted ethernet frame at the 
default MTU 1500. If you would do a debug on a plain connection, you would 
see 64kB or even more. So it will take at least ~40 times the time of a 
plain frame to receive a SSL junk of 64kB from google.
For my nice old w2k3 the difference is much less.
If for example hotmail uses 16kB SSL frames (~ 11 ethernet frames per SSL 
frame) - it would be 11 times faster than google.

This is caused by the nature of assp. It receives one frame (all available 
data for TLS/SSL connections) for each active connection (in a worker) 
after each other - means one at each cycle of processing.

Enough basic IT stuff - I don't want to bore others.

Thomas 




Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  27.09.2016 14:15
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



My abbreviated notes:

1) What a terrific explanation!!  Thank you Thomas not just for the write
up, but for taking all the time necessary to figure all this out

2) IO::Socket::SSL and Net::SSLeay
Both of mine are the latest versions (newer than the recommended / minimum
that shows in the info gui)

3) A different / additional problem with TLS?  Can we tell what algorithm
Google uses?

So we wasted allot of time looking for a TLS/SSL issue - life is not easy.

***
Back to the 'TLS' relation. Forget it - this issue has totaly nothing to
do with TLS/SSL!
***


What you describe elsewhere makes sense, BUT looking at my tests, an email
with 11 MB of attachments takes 19 seconds with TLS turned off, and with
TLS on 662 seconds.   That's 34 times longer.  I know SSL takes processing
power, but that's a huge overhead no?

*Can we tell from the logs / debug / or some other test what TLS algorithm
google is using*?  Since I only see this horrific slowdown with Google and
TLS, I wonder if I might only allow a (slightly?) lesser algorithm that's
still secure but not quite as intensive.

Note that identical hardware runs Apache on Windows with OpenSSL, Exchange
(SSL), and others without any noticeable slowdown due to the encryption.

You've clearly identified a major different problem with grow, etc, but 
I'm
not so sure that I'm not also experiencing a SSL/TLS problem.



Eagerly awaiting the next version!  Thanks.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

A ... you want to fool me - Ken!
There is nothing changed in the GUI, except the two additionally 25 + 50 
result count options.

Hurry up! Close all doors and windows - Murphy has left your IT rooms! 
:):)


Thomas



Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  27.09.2016 14:19
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



and one last thing, unrelated to the symptoms that I'm seeing, but with
this latest version, the admin GUI seems MUCH faster.  Searching the
maillog is faster, just displaying the initial maillog is much faster.  Is
this just a coincidence?
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>an email
>with 11 MB of attachments takes 19 seconds with TLS turned off, and with
>TLS on 662 seconds.

What if the message SIZE announcement is missing (not sent by google), if 
TLS is turned off?
You'll get exactly this behavior.

Thomas




Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  27.09.2016 14:15
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



My abbreviated notes:

1) What a terrific explanation!!  Thank you Thomas not just for the write
up, but for taking all the time necessary to figure all this out

2) IO::Socket::SSL and Net::SSLeay
Both of mine are the latest versions (newer than the recommended / minimum
that shows in the info gui)

3) A different / additional problem with TLS?  Can we tell what algorithm
Google uses?

So we wasted allot of time looking for a TLS/SSL issue - life is not easy.

***
Back to the 'TLS' relation. Forget it - this issue has totaly nothing to
do with TLS/SSL!
***


What you describe elsewhere makes sense, BUT looking at my tests, an email
with 11 MB of attachments takes 19 seconds with TLS turned off, and with
TLS on 662 seconds.   That's 34 times longer.  I know SSL takes processing
power, but that's a huge overhead no?

*Can we tell from the logs / debug / or some other test what TLS algorithm
google is using*?  Since I only see this horrific slowdown with Google and
TLS, I wonder if I might only allow a (slightly?) lesser algorithm that's
still secure but not quite as intensive.

Note that identical hardware runs Apache on Windows with OpenSSL, Exchange
(SSL), and others without any noticeable slowdown due to the encryption.

You've clearly identified a major different problem with grow, etc, but 
I'm
not so sure that I'm not also experiencing a SSL/TLS problem.



Eagerly awaiting the next version!  Thanks.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

and one last thing, unrelated to the symptoms that I'm seeing, but with
this latest version, the admin GUI seems MUCH faster.  Searching the
maillog is faster, just displaying the initial maillog is much faster.  Is
this just a coincidence?
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

My abbreviated notes:

1) What a terrific explanation!!  Thank you Thomas not just for the write
up, but for taking all the time necessary to figure all this out

2) IO::Socket::SSL and Net::SSLeay
Both of mine are the latest versions (newer than the recommended / minimum
that shows in the info gui)

3) A different / additional problem with TLS?  Can we tell what algorithm
Google uses?

So we wasted allot of time looking for a TLS/SSL issue - life is not easy.

***
Back to the 'TLS' relation. Forget it - this issue has totaly nothing to
do with TLS/SSL!
***


What you describe elsewhere makes sense, BUT looking at my tests, an email
with 11 MB of attachments takes 19 seconds with TLS turned off, and with
TLS on 662 seconds.   That's 34 times longer.  I know SSL takes processing
power, but that's a huge overhead no?

*Can we tell from the logs / debug / or some other test what TLS algorithm
google is using*?  Since I only see this horrific slowdown with Google and
TLS, I wonder if I might only allow a (slightly?) lesser algorithm that's
still secure but not quite as intensive.

Note that identical hardware runs Apache on Windows with OpenSSL, Exchange
(SSL), and others without any noticeable slowdown due to the encryption.

You've clearly identified a major different problem with grow, etc, but I'm
not so sure that I'm not also experiencing a SSL/TLS problem.



Eagerly awaiting the next version!  Thanks.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>line took 620 seconds  That's better than the 772seconds

This is caused by the performance improvement for large mails in this 
version.

>I see a message which I assume is now expected: ( SIZE 15700413 byte > 
neverQueueSize 1200 byte)

This output was also seen in the old versions. Nothing new.

>I saw a X-ASSP-KEEP line in the header too.  Don't know what that means. 
Haven't seen that before.

This should have been there also in older versions. I've improved the code 
there.
While assp receives the message in to the queue, there is no communication 
with the SMTP backend server (MTA).
This can cause SMTP timeouts from this MTA. To prevent the timeout, assp 
sends this 'dummy' header line to the MTA 15 seconds before a timeout 
happens, to keep the connection alive.

>$main::neverQueueSize = 4194304; ... It took 327 seconds.

This looks OK for me. The read operation on your system for TLS requires 
1.5 milliseconds - this is OK. (the rest see below)

>Removing the full message analysis seems like a shame especially since it 
doesn't seem to even stutter if TLS is off.

This was the case all the time if '( SIZE 15700413 byte > neverQueueSize 
1200 byte)' was reached.
Nobody sends a 12MB SPAM. Analyzing 12MB or more for DKIM, Virus, OCR, 
Razor, DCC . is an overkill.

answers:

1 - TLS requires high math operations to decrypt the content.
As long as there is no SSL-rehandshake requested by the sender, the 
running code is the same for all connections.
Only the line

$hasread = $fh->sysread($SMTPbuf, $Con{$fh}->{RCVBUF});

calls the connection specific module (IO::Socket::INET, IO::Socket::INET6 
or IO::Socket::SSL).
What ever time is taken there, we can't controll it. But the new debug 
file shows - there is no TLS time issue.

2 - IO::Socket::SSL and Net::SSLeay

3 - Convert::Scalar->grow works together with the perl on my system

4 - IMHO a not correctly working Convert::Scalar at the used Perl version 
/ OS (maloc and garbage collection)

what happens in Perl:
assume we store some byte (let's use the 1400 byte sent by google).

$buf = 'a' x 1400;

Perl will allocate ~1400 byte of memory for $buf and will write the 1400 
byte in there.
Now we append additionally 1400 byte to $buf (as we do in assp queueing).

$buf = $buf . 'a' x 1400;

Perl will have a look in to the $buf memory area. If there is enought 
space behind the still allocated memory, it will write the new 1400 byte 
behind the existing. This will never be possible in the huge memory area 
of assp with millions of memory changes per second. So what to do? Perl 
will allocate a new area of 2800 byte, copies the existing content to 
there and appends the new 1400 byte. After that, the old memory area is 
marked for the Perl garbage collection to be freed up for any further 
required memory.
This operation for the 1400...2800 byte is very fast because of the short 
content.
But what if we still have 8MB in $buf and we add 1400 byte every some 
milliseconds: alloc->copy->append,alloc->copy->append,alloc..

IMHO this happens on your system and it is done for the mail content to be 
processed, the maillog-buffer and possibly for the crash-analyzer and 
connection-timout-debug.

I know this behavior of Perl. To prevent this alloc->copy->append 
mechanism, assp uses the 'grow' mechanism of the module Convert::Scalar.
This reduces the operation to a simple : append,append,append,ap..
If the sender announces the message size, assp grows the memory area for 
all variables, that needs to grow to the message size.
If assp does not know the message size, MaxBytes are allocated and the 
variables are growed by 1MB, every time the old limit is reached.

There are two option for the issue. 
1. The Convert::Scalar->grow is not working. 
2. Convert::Scalar->grow works, but Perl frees up the allocated unused 
memory at any time.

Both we can't check at assp runtime.

>What is TLS doing that slows things down so much for GOOGLE mails only 
(or at least only google that I've seen be slow)

I think, this (the only) is not right. The symptome is best shown in this 
case.
1 - TLS takes much longer than plain.
2 - google uses high encryption algorythms.
3 - Large mails are sent from google.
4 - grow does not work

(4) is every time the case
all other points may be skipped by other senders : plain, or smaller mail, 
or less encryption - and will lead in to more or less faster processing

The new debug log (just arrived) shows the reason for the slow down of the 
speed.
Everything runs like expected - except the time required between 

[Worker_1] 2_14315_5.020001_UAX#29_UAX#15_WordStem1.27 - required:
>2_14315_UAX#29_UAX#15_WordStem1.27

The perl version is no longer part of the DB version string. Ignore this 
and run a rebuild.



Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  27.09.2016 05:50
Betreff

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>Don't forget that I see the issue from more than just Google.

This is what I expected! This is not related to TLS/SSL it is related to 
the message SIZE announcement.
I looks like there is an undocumented change in the Perl code for version 
> 5.16.3 (possibly >=5.20.0), which prevents or ignores the perl internal 
'grow' call for an empty and never used before variable.

Simply wait for the next release - hope it will fix this issue.

Thomas




Von:Colin Waring <co...@dolphinict.co.uk>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  27.09.2016 12:36
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



I have been running IO::Socket::SSL 2.0.33 though have just updated to 
2.0.38. I don't think this is going to be related as I have seen this 
issue for a long time and will undoubtedly have had previous versions of 
OpenSSL.

Don't forget that I see the issue from more than just Google.

I'm quite pushed for time at the moment. Ken, what did you do specifically 
to grab the necessary debugs? - save me having to stop and think :)

All the best,
Colin Waring.


Colin Waring
Technical Manager
Dolphin ICT Limited
T
+44 (0)151 438 2246 Ext 2003
www.dolphinict.co.uk
co...@dolphinict.co.uk
US15a, Armstrong House, First Avenue, Robin Hood Airport, Doncaster, DN9 
3GA





Dolphin ICT Limited. NOTICE & DISCLAIMER Dolphin ICT Limited, a private 
limited company, with company registration number 6206916, registered in 
the United Kingdom, the registered office of which is at US15a, Armstrong 
House, First Avenue, Robin Hood Airport, Doncaster, DN9 3GA VAT 
registration number GB 918 1896 88. 



-Original Message-
From: K Post [mailto:nntp.p...@gmail.com]
Sent: 27 September 2016 04:53
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

I have IO::Socket::SSL 2.036 installed instead of 2.020.  Could this have 
anything to do with any of this?

On Mon, Sep 26, 2016 at 11:49 PM, K Post <nntp.p...@gmail.com> wrote:

> THANK YOU again for taking all the time on this.  It's nuts that this 
> only seems to happen (to me and others reporting) with TLS on and mail 
> sent through google servers.
>
> I've confirmed the version of Convert::Scalar to be 1.11
>
> I'll get you a debug log privately, but here's what I'm seeing with 
> the latest version:
>
> 11mb attachment, tls on, newest version, but without the 
> $main::neverQueueSize = 4194304; line took 620 seconds.  That's better 
> than the 772seconds that saw before I but still pretty terrible - and 
> of course, that's only one test.
>
> I see a message which I assume is now expected:
> message is too large ( SIZE 15700413 byte > neverQueueSize 1200
> byte) to be queued for further internal processing! Skipping DKIM, 
> Plugins and charset conversion. for that message
>
> I saw a X-ASSP-KEEP line in the header too.  Don't know what that means.
> Haven't seen that before.
>
> Once I added the $main::neverQueueSize = 4194304; line to 
> ASSP_Correct.pm, speed improves for sure.  It took 327 seconds.  Still 
> really slow considering that without TLS it only takes 19 seconds.
> Similar line noting the 4MB size limit Removing the full message 
> analysis seems like a shame especially since it doesn't seem to even 
> stutter if TLS is off.
>
> So more questions for your consideration
> 1) What is TLS doing that slows things down so much for GOOGLE mails 
> only (or at least only google that I've seen be slow)
> 2) What encryption related modules need checking?
> 3) Why would things be fine on your old Windows 2003 rig, but clearly 
> not okay on my (presumably) faster machine
> 4) What is similar between my machine and the others who reported TLS 
> problems with Google.  I know one at least was a Linux rig.
>
>
>
>
>
>
> On Mon, Sep 26, 2016 at 4:02 AM, Thomas Eckardt < 
> thomas.ecka...@thockar.com> wrote:
>
>> First, thank you for the debug file.
>>
>> There is one big problem. The debug file explains the general 
>> behavior of the slowing down connection while the data size is growing.
>> It not explains, why this should only happens at connections from 
>> gmail.com and only if TLS is used.
>>
>> looking at the following timeline - the *** lines are from me and are 
>> showing the count of read-socketcalls within this second
>>
>> 
>> Sep-23-16 21:14:37 [Worker_2] > IO::Socket::INET=GLOB(0x11c1e3bc) (6)<DATA[CR][LF]
>> Sep-23-16 21:14:37 [Worker_2] > Sep-23-16 21:14:38 [Worker_2] > Sep-23-16 21:14:39 [Worker_2] > (each 1440 byte) 164 ...
>> Sep-23-16 21:14:40 [Worker_2] > (each 1440 byte) 167 ...
>> Sep-23-16 21:

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Colin Waring]

I have been running IO::Socket::SSL 2.0.33 though have just updated to 2.0.38. 
I don't think this is going to be related as I have seen this issue for a long 
time and will undoubtedly have had previous versions of OpenSSL.

Don't forget that I see the issue from more than just Google.

I'm quite pushed for time at the moment. Ken, what did you do specifically to 
grab the necessary debugs? - save me having to stop and think :)

All the best,
Colin Waring.


Colin Waring
Technical Manager
Dolphin ICT Limited
T
+44 (0)151 438 2246 Ext 2003
www.dolphinict.co.uk
co...@dolphinict.co.uk
US15a, Armstrong House, First Avenue, Robin Hood Airport, Doncaster, DN9 3GA





Dolphin ICT Limited. NOTICE & DISCLAIMER Dolphin ICT Limited, a private limited 
company, with company registration number 6206916, registered in the United 
Kingdom, the registered office of which is at US15a, Armstrong House, First 
Avenue, Robin Hood Airport, Doncaster, DN9 3GA VAT registration number GB 918 
1896 88. 



-Original Message-
From: K Post [mailto:nntp.p...@gmail.com]
Sent: 27 September 2016 04:53
To: ASSP development mailing list <assp-test@lists.sourceforge.net>
Subject: Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

I have IO::Socket::SSL 2.036 installed instead of 2.020.  Could this have 
anything to do with any of this?

On Mon, Sep 26, 2016 at 11:49 PM, K Post <nntp.p...@gmail.com> wrote:

> THANK YOU again for taking all the time on this.  It's nuts that this 
> only seems to happen (to me and others reporting) with TLS on and mail 
> sent through google servers.
>
> I've confirmed the version of Convert::Scalar to be 1.11
>
> I'll get you a debug log privately, but here's what I'm seeing with 
> the latest version:
>
> 11mb attachment, tls on, newest version, but without the 
> $main::neverQueueSize = 4194304; line took 620 seconds.  That's better 
> than the 772seconds that saw before I but still pretty terrible - and 
> of course, that's only one test.
>
> I see a message which I assume is now expected:
> message is too large ( SIZE 15700413 byte > neverQueueSize 1200
> byte) to be queued for further internal processing! Skipping DKIM, 
> Plugins and charset conversion. for that message
>
> I saw a X-ASSP-KEEP line in the header too.  Don't know what that means.
> Haven't seen that before.
>
> Once I added the $main::neverQueueSize = 4194304; line to 
> ASSP_Correct.pm, speed improves for sure.  It took 327 seconds.  Still 
> really slow considering that without TLS it only takes 19 seconds.
> Similar line noting the 4MB size limit Removing the full message 
> analysis seems like a shame especially since it doesn't seem to even 
> stutter if TLS is off.
>
> So more questions for your consideration
> 1) What is TLS doing that slows things down so much for GOOGLE mails 
> only (or at least only google that I've seen be slow)
> 2) What encryption related modules need checking?
> 3) Why would things be fine on your old Windows 2003 rig, but clearly 
> not okay on my (presumably) faster machine
> 4) What is similar between my machine and the others who reported TLS 
> problems with Google.  I know one at least was a Linux rig.
>
>
>
>
>
>
> On Mon, Sep 26, 2016 at 4:02 AM, Thomas Eckardt < 
> thomas.ecka...@thockar.com> wrote:
>
>> First, thank you for the debug file.
>>
>> There is one big problem. The debug file explains the general 
>> behavior of the slowing down connection while the data size is growing.
>> It not explains, why this should only happens at connections from 
>> gmail.com and only if TLS is used.
>>
>> looking at the following timeline - the *** lines are from me and are 
>> showing the count of read-socketcalls within this second
>>
>> 
>> Sep-23-16 21:14:37 [Worker_2] > IO::Socket::INET=GLOB(0x11c1e3bc) (6)<DATA[CR][LF]
>> Sep-23-16 21:14:37 [Worker_2] > Sep-23-16 21:14:38 [Worker_2] > Sep-23-16 21:14:39 [Worker_2] > (each 1440 byte) 164 ...
>> Sep-23-16 21:14:40 [Worker_2] > (each 1440 byte) 167 ...
>> Sep-23-16 21:14:41 [Worker_2] > (each 1440 byte) 108 ...
>> Sep-23-16 21:14:42 [Worker_2] > (each 1440 byte) 95 ...
>> Sep-23-16 21:14:43 [Worker_2] > (each 1440 byte) 82 ...
>> Sep-23-16 21:14:44 [Worker_2] > (each 1440 byte) 74 ...
>> Sep-23-16 21:15:09 [Worker_2] > (each 1440 byte) 43 ...
>> Sep-23-16 21:15:39 [Worker_2] > (each 1440 byte) 35 ...
>> Sep-23-16 21:16:39 [Worker_2] > (each 1440 byte) 21 ...
>> Sep-23-16 21:18:39 [Worker_2] > (each 1440 byte) 12 ...
>> Sep-23-16 21:22:41 msg79676-04975 209.85.223.177 
>> <nntp.p...@gmail.com>
>> to:
>> testtls@[[ OUR DOMAIN ]].org info: message is too large (

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

ws.
>> At Sep-23-16 21:22:45 the outqueue is empty and the data are sent as they
>> are received (1404 Byte each) with a normal speed.
>>
>> This behavior can't be explained with the usage of TLS, because the code
>> behind is the same for all connections. The read data size for each read
>> operation is always the same 1400 Byte.
>> For me, the behavior can be exactly described with a not working Perl
>> module 'Convert::Scalar' or a code operation, which is done over all the
>> growing data after each read operation.
>> The latter I can exclude. To make this 100% sure, I've made some small
>> changes in the next release (will be published today).
>>
>> It is not possible to check, that 'Convert::Scalar' is working like
>> expected for the 'grow' operation - even the module maintainer is not able
>> to do this in the installation test scripts.
>> At least make sure that version 1.11 is installed.
>>
>> >Sep-23-16 21:14:37 [Worker_2] > maillogbuf=15703707 , outgoing=15703707
>>
>> shows, that the module is installed and called by assp.
>>
>> What are the options to solve the problem?
>>
>> 1. in any case - check that Convert::Scalar version 1.11 is installed
>> 2. Try the new version of assp.pl - I'll publish today. It contains code
>> changes to prevent this behavior, at least to indentify the reason somehow
>> better
>> If the same behavior happens, please provide me the debug file again.
>>
>> 3. if this does not help, add the following code line to the module
>> 'assp/lib/CorrectASSPcfg.pm' in 'sub set { '
>>
>> $main::neverQueueSize = 4194304;
>>
>> This hidden value defines the maximum size of mail data, that should be
>> queued for all full mail checks. If a mail is larger, queueing is switched
>> off - charset conversion, full DKIM checks, DKIM signing and all Plugins
>> for full mail operations (level 2) will be skipped.
>>
>> 4194304 are 4MB - you may try any other value of your choice. Default
>> value is 1200.
>> This solution will also work with the current code.
>>
>>
>> Thomas
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Von:K Post <nntp.p...@gmail.com>
>> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
>> Datum:  24.09.2016 04:05
>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>> servers
>>
>>
>>
>> in the debug that I'm about to send you, I'm seeing lines like:
>> 
>> periodically in the file.
>> Don't know if that's notable or not.
>>
>> On Fri, Sep 23, 2016 at 9:43 PM, K Post <nntp.p...@gmail.com> wrote:
>>
>> > I'll get you a sample debug asap.
>> >
>> > FYI, I forgot to mention yesterday - I've noticed times (not always)
>> when
>> > watching the SMTP Connections Window with large test gmail emails where
>> a
>> > message will start transferring and after some time (7-8 minutes, maybe
>> a
>> > little less), that google will connect a second time and start
>> transferring
>> > the message again, even though the first one is still being received.
>> >
>> > The first message completed after say 12 minutes, and then 3-4 minutes
>> > after that, the 2nd google connection disconnects and doesn't try again
>> > (nor should it).
>> >
>> > This does NOT happen every time, and I can't find find a reason why it
>> > would do this on occasion for some but not other big messages.
>> >
>> >
>> > On Fri, Sep 23, 2016 at 7:02 AM, Thomas Eckardt <
>> > thomas.ecka...@thockar.com> wrote:
>> >
>> >> Thank you Ken.
>> >>
>> >> Please would you send me the debug log for the large (15MB) TLS mail as
>> >> ZIP or make it available to me for download anywhere, if it is too
>> large
>> >> for SMTP.
>> >>
>> >> Thomas
>> >>
>> >>
>> >
>> 
>> --
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>>
>>
>> DISCLAIMER:
>> ***
>> This email and any files transmitted with it may be confidential, legally
>> privileged and protected in law and are intended solely for the use of the
>>
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> ***
>>
>>
>> 
>> --
>>
>> ___
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

First, thank you for the debug file.

There is one big problem. The debug file explains the general behavior of 
the slowing down connection while the data size is growing.
It not explains, why this should only happens at connections from 
gmail.com and only if TLS is used.

looking at the following timeline - the *** lines are from me and are 
showing the count of read-socketcalls within this second


Sep-23-16 21:14:37 [Worker_2]  to: 
testtls@[[ OUR DOMAIN ]].org info: message is too large ( > neverQueueSize 
1200 byte) to be queued for further internal processing! Skipping 
DKMI, Plugins and charset conversion. 
***socketcalls per second (each 1440 byte) 8
...
Sep-23-16 21:22:45 [Worker_2] Sep-23-16 21:14:37 [Worker_2] 
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  24.09.2016 04:05
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



in the debug that I'm about to send you, I'm seeing lines like:

periodically in the file.
Don't know if that's notable or not.

On Fri, Sep 23, 2016 at 9:43 PM, K Post <nntp.p...@gmail.com> wrote:

> I'll get you a sample debug asap.
>
> FYI, I forgot to mention yesterday - I've noticed times (not always) 
when
> watching the SMTP Connections Window with large test gmail emails where 
a
> message will start transferring and after some time (7-8 minutes, maybe 
a
> little less), that google will connect a second time and start 
transferring
> the message again, even though the first one is still being received.
>
> The first message completed after say 12 minutes, and then 3-4 minutes
> after that, the 2nd google connection disconnects and doesn't try again
> (nor should it).
>
> This does NOT happen every time, and I can't find find a reason why it
> would do this on occasion for some but not other big messages.
>
>
> On Fri, Sep 23, 2016 at 7:02 AM, Thomas Eckardt <
> thomas.ecka...@thockar.com> wrote:
>
>> Thank you Ken.
>>
>> Please would you send me the debug log for the large (15MB) TLS mail as
>> ZIP or make it available to me for download anywhere, if it is too 
large
>> for SMTP.
>>
>> Thomas
>>
>>
>
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

in the debug that I'm about to send you, I'm seeing lines like:

periodically in the file.
Don't know if that's notable or not.

On Fri, Sep 23, 2016 at 9:43 PM, K Post  wrote:

> I'll get you a sample debug asap.
>
> FYI, I forgot to mention yesterday - I've noticed times (not always) when
> watching the SMTP Connections Window with large test gmail emails where a
> message will start transferring and after some time (7-8 minutes, maybe a
> little less), that google will connect a second time and start transferring
> the message again, even though the first one is still being received.
>
> The first message completed after say 12 minutes, and then 3-4 minutes
> after that, the 2nd google connection disconnects and doesn't try again
> (nor should it).
>
> This does NOT happen every time, and I can't find find a reason why it
> would do this on occasion for some but not other big messages.
>
>
> On Fri, Sep 23, 2016 at 7:02 AM, Thomas Eckardt <
> thomas.ecka...@thockar.com> wrote:
>
>> Thank you Ken.
>>
>> Please would you send me the debug log for the large (15MB) TLS mail as
>> ZIP or make it available to me for download anywhere, if it is too large
>> for SMTP.
>>
>> Thomas
>>
>>
>
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

I'll get you a sample debug asap.

FYI, I forgot to mention yesterday - I've noticed times (not always) when
watching the SMTP Connections Window with large test gmail emails where a
message will start transferring and after some time (7-8 minutes, maybe a
little less), that google will connect a second time and start transferring
the message again, even though the first one is still being received.

The first message completed after say 12 minutes, and then 3-4 minutes
after that, the 2nd google connection disconnects and doesn't try again
(nor should it).

This does NOT happen every time, and I can't find find a reason why it
would do this on occasion for some but not other big messages.


On Fri, Sep 23, 2016 at 7:02 AM, Thomas Eckardt 
wrote:

> Thank you Ken.
>
> Please would you send me the debug log for the large (15MB) TLS mail as
> ZIP or make it available to me for download anywhere, if it is too large
> for SMTP.
>
> Thomas
>
>
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

Thank you Ken.

Please would you send me the debug log for the large (15MB) TLS mail as 
ZIP or make it available to me for download anywhere, if it is too large 
for SMTP.

Thomas







Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  22.09.2016 18:16
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



It doesn't seem to be linear.  Here's how I tested that theory.

First I did 4 tests from gmail with TLS disabled for the google IP's: 
100KB
file, 1.1MB file, five 1.1MB files in one email, and ten 1.1MB files in 
one
email.
Then I did those same 4 tests with TLS enabled for the google ip's.

100KB NO TLS
received DATA size: 142.81 kByte - sent DATA size: 143.44 kByte
processing time 2 seconds

1.1MB NO TLS
received DATA size: 1.50 MByte - sent DATA size: 1.50 MByte
processing time 6 seconds

5.5MB NO TLS
received DATA size: 7.49 MByte - sent DATA size: 7.49 MByte
processing time 13 seconds

11MB NO TLS
received DATA size: 14.97 MByte - sent DATA size: 14.97 MByte
processing time 19 seconds


Same files attached, now with TLS ON for the google ip addresses

100KB With TLS
received DATA size: 142.87 kByte - sent DATA size: 143.54 kByte
processing time 3 seconds (1 second longer, but still totally acceptable)

1.1MB With TLS
received DATA size: 1.50 MByte - sent DATA size: 1.50 MByte
processing time 27 seconds
about *4.5x *loger than without TLS, only 27 seconds, but that's a pretty
long time for a 1.5mb email

5.5MB TLS
received DATA size: 7.49 MByte - sent DATA size: 7.49 MByte
processing time 318 seconds
about *24x *longer than without TLS
and nearly 1/3 the speed of the 1MB tls version

11.0MB
received DATA size: 14.97 MByte - sent DATA size: 14.97 MByte
processing time 772 seconds
about *40x *longer than without TLS
almost 13 minutes instead of just 19 seconds
about 2.5x the time of the 5.5MB with tls, expected 2x

I can't test larger emails with google, Google will timeout after 15
minutes.

I had debugging on for the gmail address I was sending from and got a huge
debug log as expected.  However, there's nothing useful in there.  I don't
see anything about speed, SSL renegotiation, or anything.


For reference, sending that same 11.0MB email from a test *Outlook.com*
account (whihch uses TLS) gets me:
received DATA size: 14.98 MByte - sent DATA size: 14.98 MByte
processing time 76 seconds   (reasonable in my book for a TLS session)

I also watched other traffic after the tests were done.I happened to
see messages 5MB, 12MB, 17MB all came through quickly from non-Google
sources with TLS on, but other gmail emails with attachments were slow
slow.  I haven't seen any mails be slow over TLS except for google, but
that doesn't mean that there aren't others.

Whatever the case, Gmail is too big of a player in this game to ignore the
problem IMO.


THANKS SO MUCH











On Thu, Sep 22, 2016 at 5:58 AM, Thomas Eckardt 
<thomas.ecka...@thockar.com>
wrote:

> Ken, please check the following.
>
> Investigate a relatve small (eg 100KB), a middle size (1MB) and one mail
> that takes very long.
>
> Is the processing time in a nearly linear relation to the message size?
>
> like:
>
> 100KB - six seconds
> 1MB - one minute
> 2MB - two minutes
> 3MB - three minutes
> 
>
> Or grows the time required for one MB, if the message size grows?
>
> Thomas
>
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  03.08.2016 03:37
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Thanks Thomas, but what OpenSSL should I be using?  I really don't think
> this is the problem, but I might as well eliminate it.  I've got
> activestate's perl 5.20 installed and net::ssleay from the activestate
> ppm.  However,the OpenSSL binaries that I have (I'm talking about the 
FULL
> openssl installation in c:\openssl) not the dll files that net::ssleay
> >might< have, is 1.0.2h from Shiining LIght (
> slproweb.com/products/Win32OpenSSL.html)
>
> ASSP says net::ssleay is OpenSSL 1.0.2g - apparently it hasn't been
> compiled using 1.0.2h yet.  That the readme from net::ssleay talks
> specifically about shining light and that it's best to roll your own
> worries me.
>
> And Bob,
> Thanks for testing this out.  3MB in 25 seconds is about what I'm
> generally
> seeing now that I've tweaked the performance settings of ASSP, but 
without
> TLS, we can receive a 10mb attachment in just a few seconds thanks to a
> fast line.  Curious, if you disable TLS temporarily and send yourself 
that
> same 3mb attachment from gmail, how long does it take?
>
>
>
> On Tue, Aug 2, 2016 at 2:04 PM, Thomas Eckardt
> <thomas.ecka...@th

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

It doesn't seem to be linear.  Here's how I tested that theory.

First I did 4 tests from gmail with TLS disabled for the google IP's: 100KB
file, 1.1MB file, five 1.1MB files in one email, and ten 1.1MB files in one
email.
Then I did those same 4 tests with TLS enabled for the google ip's.

100KB NO TLS
received DATA size: 142.81 kByte - sent DATA size: 143.44 kByte
processing time 2 seconds

1.1MB NO TLS
received DATA size: 1.50 MByte - sent DATA size: 1.50 MByte
processing time 6 seconds

5.5MB NO TLS
received DATA size: 7.49 MByte - sent DATA size: 7.49 MByte
processing time 13 seconds

11MB NO TLS
received DATA size: 14.97 MByte - sent DATA size: 14.97 MByte
processing time 19 seconds


Same files attached, now with TLS ON for the google ip addresses

100KB With TLS
received DATA size: 142.87 kByte - sent DATA size: 143.54 kByte
processing time 3 seconds (1 second longer, but still totally acceptable)

1.1MB With TLS
received DATA size: 1.50 MByte - sent DATA size: 1.50 MByte
processing time 27 seconds
about *4.5x *loger than without TLS, only 27 seconds, but that's a pretty
long time for a 1.5mb email

5.5MB TLS
received DATA size: 7.49 MByte - sent DATA size: 7.49 MByte
processing time 318 seconds
about *24x *longer than without TLS
and nearly 1/3 the speed of the 1MB tls version

11.0MB
received DATA size: 14.97 MByte - sent DATA size: 14.97 MByte
processing time 772 seconds
about *40x *longer than without TLS
almost 13 minutes instead of just 19 seconds
about 2.5x the time of the 5.5MB with tls, expected 2x

I can't test larger emails with google, Google will timeout after 15
minutes.

I had debugging on for the gmail address I was sending from and got a huge
debug log as expected.  However, there's nothing useful in there.  I don't
see anything about speed, SSL renegotiation, or anything.


For reference, sending that same 11.0MB email from a test *Outlook.com*
account (whihch uses TLS) gets me:
received DATA size: 14.98 MByte - sent DATA size: 14.98 MByte
processing time 76 seconds   (reasonable in my book for a TLS session)

I also watched other traffic after the tests were done.I happened to
see messages 5MB, 12MB, 17MB all came through quickly from non-Google
sources with TLS on, but other gmail emails with attachments were slow
slow.  I haven't seen any mails be slow over TLS except for google, but
that doesn't mean that there aren't others.

Whatever the case, Gmail is too big of a player in this game to ignore the
problem IMO.


THANKS SO MUCH











On Thu, Sep 22, 2016 at 5:58 AM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:

> Ken, please check the following.
>
> Investigate a relatve small (eg 100KB), a middle size (1MB) and one mail
> that takes very long.
>
> Is the processing time in a nearly linear relation to the message size?
>
> like:
>
> 100KB - six seconds
> 1MB - one minute
> 2MB - two minutes
> 3MB - three minutes
> 
>
> Or grows the time required for one MB, if the message size grows?
>
> Thomas
>
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  03.08.2016 03:37
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Thanks Thomas, but what OpenSSL should I be using?  I really don't think
> this is the problem, but I might as well eliminate it.  I've got
> activestate's perl 5.20 installed and net::ssleay from the activestate
> ppm.  However,the OpenSSL binaries that I have (I'm talking about the FULL
> openssl installation in c:\openssl) not the dll files that net::ssleay
> >might< have, is 1.0.2h from Shiining LIght (
> slproweb.com/products/Win32OpenSSL.html)
>
> ASSP says net::ssleay is OpenSSL 1.0.2g - apparently it hasn't been
> compiled using 1.0.2h yet.  That the readme from net::ssleay talks
> specifically about shining light and that it's best to roll your own
> worries me.
>
> And Bob,
> Thanks for testing this out.  3MB in 25 seconds is about what I'm
> generally
> seeing now that I've tweaked the performance settings of ASSP, but without
> TLS, we can receive a 10mb attachment in just a few seconds thanks to a
> fast line.  Curious, if you disable TLS temporarily and send yourself that
> same 3mb attachment from gmail, how long does it take?
>
>
>
> On Tue, Aug 2, 2016 at 2:04 PM, Thomas Eckardt
> <thomas.ecka...@thockar.com>
> wrote:
>
> > >Having looked through the Net:SSLEAY readme, there's a bunch that
> > suggests
> > >that it's best to compile your own net:ssleay and OpenSSL on the same
> > >machine with the same settings.
> >
> > This will be the case, if you use the PPM from ActiveState. Perl and all
> > modules are compiled with the same compiler an

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

Ken, please check the following.

Investigate a relatve small (eg 100KB), a middle size (1MB) and one mail 
that takes very long.

Is the processing time in a nearly linear relation to the message size?

like:

100KB - six seconds
1MB - one minute
2MB - two minutes
3MB - three minutes


Or grows the time required for one MB, if the message size grows?

Thomas





Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  03.08.2016 03:37
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Thanks Thomas, but what OpenSSL should I be using?  I really don't think
this is the problem, but I might as well eliminate it.  I've got
activestate's perl 5.20 installed and net::ssleay from the activestate
ppm.  However,the OpenSSL binaries that I have (I'm talking about the FULL
openssl installation in c:\openssl) not the dll files that net::ssleay
>might< have, is 1.0.2h from Shiining LIght (
slproweb.com/products/Win32OpenSSL.html)

ASSP says net::ssleay is OpenSSL 1.0.2g - apparently it hasn't been
compiled using 1.0.2h yet.  That the readme from net::ssleay talks
specifically about shining light and that it's best to roll your own
worries me.

And Bob,
Thanks for testing this out.  3MB in 25 seconds is about what I'm 
generally
seeing now that I've tweaked the performance settings of ASSP, but without
TLS, we can receive a 10mb attachment in just a few seconds thanks to a
fast line.  Curious, if you disable TLS temporarily and send yourself that
same 3mb attachment from gmail, how long does it take?



On Tue, Aug 2, 2016 at 2:04 PM, Thomas Eckardt 
<thomas.ecka...@thockar.com>
wrote:

> >Having looked through the Net:SSLEAY readme, there's a bunch that
> suggests
> >that it's best to compile your own net:ssleay and OpenSSL on the same
> >machine with the same settings.
>
> This will be the case, if you use the PPM from ActiveState. Perl and all
> modules are compiled with the same compiler and header files. 
Net::SSLeay
> is compiled static, means it contains all required openssl code.
>
> >I'd love to find the time to give this a go,
> You'll find something better to do, than to compile this module on
> windows.
>
>
> Thomas
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  02.08.2016 19:42
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Having looked through the Net:SSLEAY readme, there's a bunch that 
suggests
> that it's best to compile your own net:ssleay and OpenSSL on the same
> machine with the same settings. I've not done that, and never have (nor 
do
> I have the skillset to do much more than run a simple make command). I'd
> love to find the time to give this a go, but what do you all think - 
could
> this be it?  Why would gmail.com always be bad, but others not (that 
I've
> seen)?
>
> On Tue, Aug 2, 2016 at 1:22 PM, Thomas Eckardt
> <thomas.ecka...@thockar.com>
> wrote:
>
> > >How do you know the type of encryption that gmail is using?
> >
> > You'll find it in the Received header line written by assp.
> >
> > >I have SSLDebug set to level 3,
> >
> > This helps not much. Most of the SSL-debug output goes to NUL.
> >  But if there were errors in SSL - you would see them in the maillog.
> >
> > >I changed EnableHighPerformace to "very high,"
> > I don't recommend to do this. This cuts the cycle time (poll/select 
wait
> > time) in the workers to a minmum. Even if assp is idle - if this is 
set,
> > it will permanently poll the sockets and will produce unwanted CPU
> > workload. I know 'EnableHighPerformace' sounds magic, but it is more
> > implemented to tweak exceptional environments.
> > How ever, if your host accepts this workload - it is fine.
> >
> > >Anything else I should try tweaking?
> >
> > Don't try to much. Tweak (if) one by one step. Use the
> > 'notes/confighistory.txt' - the old and new values are recoded there.
> >
> > I have an idea about the gmail problem. It may be the case, that they
> > request SSL rehandshakes more or less often depending on the used
> > certificate and/or cipher to raise the security of the connection. 
Such
> a
> > behavior would slow down the SSL speed - BUT, now the bad news, this 
is
> a
> > client request (made my gmail). Perl's Net::SSLeay has no easy way to
> > ignore these requests. The only way would be to pipe all SSL packest
> > through an assp routine (this is possible), which would drop the
> > renegotiation requests. Such a code will slow down ALL SSL tra

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

In the 'assp/debug' folder ???

Thomas




Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  11.08.2016 14:09
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Checking in on this - where am I supposed to see the debug log entries?

On Sat, Aug 6, 2016 at 3:46 PM, K Post <nntp.p...@gmail.com> wrote:

> I put exactly:
> $Con{$fh}->{mailfrom} =~ /\@gmail\.com/ && $Con{$fh}->{SIZE} > 1024000
> into debugCode
>
> I've looks both in the base installation folder and in the logs folder,
> but I don't see any .dbg file.  Any pointers on where else I should 
look?
>
>
> On Thu, Aug 4, 2016 at 9:54 PM, K Post <nntp.p...@gmail.com> wrote:
>
>> Now I'm in a position where the powers that be have requested that TLS 
be
>> disabled because of inbound problems from gmail.  Apparently, gmail 
users
>> who send 25mb+ files have been getting this error more frequently than 
I
>> thought.
>>
>> This is an automatically generated Delivery Status Notification
>>
>> THIS IS A WARNING MESSAGE ONLY.
>>
>> YOU DO NOT NEED TO RESEND YOUR MESSAGE.
>>
>> Delivery to the following recipient has been delayed:
>>
>>  ouru...@ourcharity.org
>>
>> Message will be retried for 1 more day(s)
>>
>> Technical details of temporary failure:
>> Missed upload deadline (899.99s) (state SENT_MESSAGE)
>>
>> One of the major donors got this today, which raised the flag with the
>> directors.  Makes testing really tough
>>
>> I might be able to test this for a little bit after hours this weekend.
>>
>>
>>
>>
>> On Thu, Aug 4, 2016 at 3:32 AM, Thomas Eckardt <
>> thomas.ecka...@thockar.com> wrote:
>>
>>> debug such a connection
>>>
>>> set debugCode to:
>>>
>>> $Con{$fh}->{mailfrom} =~ /\@gmail\.com/ && $Con{$fh}->{SIZE} > 1024000
>>>
>>> 1024000 can be larger
>>>
>>> Thomas
>>>
>>>
>>>
>>>
>>>
>>> Von:K Post <nntp.p...@gmail.com>
>>> An: ASSP development mailing list 
<assp-test@lists.sourceforge.net>
>>> Datum:  03.08.2016 19:08
>>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>>> servers
>>>
>>>
>>>
>>> watching the SMTP Connections GUI, it looks like google starts out 
pretty
>>> fast for the first 2mb or so, but then really slows down.  Might there 
be
>>> something with memory handling on my end?
>>>
>>> after x seconds: total bytes transferred
>>> 10 seconds: 1,400,000 bytes
>>> 30 seconds: 2,600,000 bytes
>>> 55 seconds: 3,800,000 bytes
>>> 90 seconds: 5,300,000 bytes
>>> 160 seconds: 7,500,000 bytes
>>>
>>> Hit 1.4mb in the first 10 seconds, but then slows to a rate of about 
2mb
>>> a
>>> minute, sometimes slower.  Does this help you figure out what might be
>>> going on with gmail?
>>>
>>>
>>>
>>>
>>> On Tue, Aug 2, 2016 at 10:40 PM, K Post <nntp.p...@gmail.com> wrote:
>>>
>>> > activestate just published net::ssleay 1.77 in their repository.
>>> Doesn't
>>> > seem to make any difference in terms of speed.  Capping out at about
>>> 2mb
>>> a
>>> > minute with TLS.
>>> >
>>> > the ssleay.dll that is in c:\perl\site\lib\auto\Net\SSLeay appears 
to
>>> have
>>> > been updated by the ppm.  ASSP in infostats still says:
>>> > OpenSSL 1.0.2h
>>> > OpenSSL-lib 1.0.2g Mar 2016
>>> >
>>> > Is that first line my c:\openssl installation from Shining Light (I
>>> don't
>>> > know anywhere else that 1.0.2h is installed)?
>>> > and OpenSSL-lib is the ssleay.dll that is seen in the
>>> > c:\perl\sit\lib\auto\net\ssleay folder?
>>> >
>>> > Does it matter that there's also a ssleay.dll in c:\openssl that is
>>> surely
>>> > 1.0.2h?
>>> >
>>> > Still, I ask all these questions, but it's only gmail that's giving 
me
>>> a
>>> > headache.  Other senders all seem fine so far, no nearly as fast as
>>> without
>>> > TLS.  For example, I just sent the same 11mb file that google takes
>>> about 7
>>> > minutes to send via Outlook.com and it only took 35 seconds.
>>> >
>>> > thanks again
>>> >

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

Checking in on this - where am I supposed to see the debug log entries?

On Sat, Aug 6, 2016 at 3:46 PM, K Post <nntp.p...@gmail.com> wrote:

> I put exactly:
> $Con{$fh}->{mailfrom} =~ /\@gmail\.com/ && $Con{$fh}->{SIZE} > 1024000
> into debugCode
>
> I've looks both in the base installation folder and in the logs folder,
> but I don't see any .dbg file.  Any pointers on where else I should look?
>
>
> On Thu, Aug 4, 2016 at 9:54 PM, K Post <nntp.p...@gmail.com> wrote:
>
>> Now I'm in a position where the powers that be have requested that TLS be
>> disabled because of inbound problems from gmail.  Apparently, gmail users
>> who send 25mb+ files have been getting this error more frequently than I
>> thought.
>>
>> This is an automatically generated Delivery Status Notification
>>
>> THIS IS A WARNING MESSAGE ONLY.
>>
>> YOU DO NOT NEED TO RESEND YOUR MESSAGE.
>>
>> Delivery to the following recipient has been delayed:
>>
>>  ouru...@ourcharity.org
>>
>> Message will be retried for 1 more day(s)
>>
>> Technical details of temporary failure:
>> Missed upload deadline (899.99s) (state SENT_MESSAGE)
>>
>> One of the major donors got this today, which raised the flag with the
>> directors.  Makes testing really tough
>>
>> I might be able to test this for a little bit after hours this weekend.
>>
>>
>>
>>
>> On Thu, Aug 4, 2016 at 3:32 AM, Thomas Eckardt <
>> thomas.ecka...@thockar.com> wrote:
>>
>>> debug such a connection
>>>
>>> set debugCode to:
>>>
>>> $Con{$fh}->{mailfrom} =~ /\@gmail\.com/ && $Con{$fh}->{SIZE} > 1024000
>>>
>>> 1024000 can be larger
>>>
>>> Thomas
>>>
>>>
>>>
>>>
>>>
>>> Von:K Post <nntp.p...@gmail.com>
>>> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
>>> Datum:  03.08.2016 19:08
>>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>>> servers
>>>
>>>
>>>
>>> watching the SMTP Connections GUI, it looks like google starts out pretty
>>> fast for the first 2mb or so, but then really slows down.  Might there be
>>> something with memory handling on my end?
>>>
>>> after x seconds: total bytes transferred
>>> 10 seconds: 1,400,000 bytes
>>> 30 seconds: 2,600,000 bytes
>>> 55 seconds: 3,800,000 bytes
>>> 90 seconds: 5,300,000 bytes
>>> 160 seconds: 7,500,000 bytes
>>>
>>> Hit 1.4mb in the first 10 seconds, but then slows to a rate of about 2mb
>>> a
>>> minute, sometimes slower.  Does this help you figure out what might be
>>> going on with gmail?
>>>
>>>
>>>
>>>
>>> On Tue, Aug 2, 2016 at 10:40 PM, K Post <nntp.p...@gmail.com> wrote:
>>>
>>> > activestate just published net::ssleay 1.77 in their repository.
>>> Doesn't
>>> > seem to make any difference in terms of speed.  Capping out at about
>>> 2mb
>>> a
>>> > minute with TLS.
>>> >
>>> > the ssleay.dll that is in c:\perl\site\lib\auto\Net\SSLeay appears to
>>> have
>>> > been updated by the ppm.  ASSP in infostats still says:
>>> > OpenSSL 1.0.2h
>>> > OpenSSL-lib 1.0.2g Mar 2016
>>> >
>>> > Is that first line my c:\openssl installation from Shining Light (I
>>> don't
>>> > know anywhere else that 1.0.2h is installed)?
>>> > and OpenSSL-lib is the ssleay.dll that is seen in the
>>> > c:\perl\sit\lib\auto\net\ssleay folder?
>>> >
>>> > Does it matter that there's also a ssleay.dll in c:\openssl that is
>>> surely
>>> > 1.0.2h?
>>> >
>>> > Still, I ask all these questions, but it's only gmail that's giving me
>>> a
>>> > headache.  Other senders all seem fine so far, no nearly as fast as
>>> without
>>> > TLS.  For example, I just sent the same 11mb file that google takes
>>> about 7
>>> > minutes to send via Outlook.com and it only took 35 seconds.
>>> >
>>> > thanks again
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Tue, Aug 2, 2016 at 9:44 PM, K Post <nntp.p...@gmail.com> wrote:
>>> >
>>> >> scratch that Bob.  I'm still closer to 1.5-2mb per minute despite the
>&g

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

I put exactly:
$Con{$fh}->{mailfrom} =~ /\@gmail\.com/ && $Con{$fh}->{SIZE} > 1024000
into debugCode

I've looks both in the base installation folder and in the logs folder, but
I don't see any .dbg file.  Any pointers on where else I should look?


On Thu, Aug 4, 2016 at 9:54 PM, K Post <nntp.p...@gmail.com> wrote:

> Now I'm in a position where the powers that be have requested that TLS be
> disabled because of inbound problems from gmail.  Apparently, gmail users
> who send 25mb+ files have been getting this error more frequently than I
> thought.
>
> This is an automatically generated Delivery Status Notification
>
> THIS IS A WARNING MESSAGE ONLY.
>
> YOU DO NOT NEED TO RESEND YOUR MESSAGE.
>
> Delivery to the following recipient has been delayed:
>
>  ouru...@ourcharity.org
>
> Message will be retried for 1 more day(s)
>
> Technical details of temporary failure:
> Missed upload deadline (899.99s) (state SENT_MESSAGE)
>
> One of the major donors got this today, which raised the flag with the
> directors.  Makes testing really tough
>
> I might be able to test this for a little bit after hours this weekend.
>
>
>
>
> On Thu, Aug 4, 2016 at 3:32 AM, Thomas Eckardt <thomas.ecka...@thockar.com
> > wrote:
>
>> debug such a connection
>>
>> set debugCode to:
>>
>> $Con{$fh}->{mailfrom} =~ /\@gmail\.com/ && $Con{$fh}->{SIZE} > 1024000
>>
>> 1024000 can be larger
>>
>> Thomas
>>
>>
>>
>>
>>
>> Von:K Post <nntp.p...@gmail.com>
>> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
>> Datum:  03.08.2016 19:08
>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>> servers
>>
>>
>>
>> watching the SMTP Connections GUI, it looks like google starts out pretty
>> fast for the first 2mb or so, but then really slows down.  Might there be
>> something with memory handling on my end?
>>
>> after x seconds: total bytes transferred
>> 10 seconds: 1,400,000 bytes
>> 30 seconds: 2,600,000 bytes
>> 55 seconds: 3,800,000 bytes
>> 90 seconds: 5,300,000 bytes
>> 160 seconds: 7,500,000 bytes
>>
>> Hit 1.4mb in the first 10 seconds, but then slows to a rate of about 2mb a
>> minute, sometimes slower.  Does this help you figure out what might be
>> going on with gmail?
>>
>>
>>
>>
>> On Tue, Aug 2, 2016 at 10:40 PM, K Post <nntp.p...@gmail.com> wrote:
>>
>> > activestate just published net::ssleay 1.77 in their repository. Doesn't
>> > seem to make any difference in terms of speed.  Capping out at about 2mb
>> a
>> > minute with TLS.
>> >
>> > the ssleay.dll that is in c:\perl\site\lib\auto\Net\SSLeay appears to
>> have
>> > been updated by the ppm.  ASSP in infostats still says:
>> > OpenSSL 1.0.2h
>> > OpenSSL-lib 1.0.2g Mar 2016
>> >
>> > Is that first line my c:\openssl installation from Shining Light (I
>> don't
>> > know anywhere else that 1.0.2h is installed)?
>> > and OpenSSL-lib is the ssleay.dll that is seen in the
>> > c:\perl\sit\lib\auto\net\ssleay folder?
>> >
>> > Does it matter that there's also a ssleay.dll in c:\openssl that is
>> surely
>> > 1.0.2h?
>> >
>> > Still, I ask all these questions, but it's only gmail that's giving me a
>> > headache.  Other senders all seem fine so far, no nearly as fast as
>> without
>> > TLS.  For example, I just sent the same 11mb file that google takes
>> about 7
>> > minutes to send via Outlook.com and it only took 35 seconds.
>> >
>> > thanks again
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Aug 2, 2016 at 9:44 PM, K Post <nntp.p...@gmail.com> wrote:
>> >
>> >> scratch that Bob.  I'm still closer to 1.5-2mb per minute despite the
>> >> tweaks.
>> >>
>> >> On Tue, Aug 2, 2016 at 9:36 PM, K Post <nntp.p...@gmail.com> wrote:
>> >>
>> >>> Thanks Thomas, but what OpenSSL should I be using?  I really don't
>> think
>> >>> this is the problem, but I might as well eliminate it.  I've got
>> >>> activestate's perl 5.20 installed and net::ssleay from the activestate
>> >>> ppm.  However,the OpenSSL binaries that I have (I'm talking about the
>> FULL
>> >>> openssl installation in c:\openssl) not the dll files that net::ssleay
>> >>> >might< have, is 1.0.2h fr

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

Now I'm in a position where the powers that be have requested that TLS be
disabled because of inbound problems from gmail.  Apparently, gmail users
who send 25mb+ files have been getting this error more frequently than I
thought.

This is an automatically generated Delivery Status Notification

THIS IS A WARNING MESSAGE ONLY.

YOU DO NOT NEED TO RESEND YOUR MESSAGE.

Delivery to the following recipient has been delayed:

 ouru...@ourcharity.org

Message will be retried for 1 more day(s)

Technical details of temporary failure:
Missed upload deadline (899.99s) (state SENT_MESSAGE)

One of the major donors got this today, which raised the flag with the
directors.  Makes testing really tough

I might be able to test this for a little bit after hours this weekend.




On Thu, Aug 4, 2016 at 3:32 AM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:

> debug such a connection
>
> set debugCode to:
>
> $Con{$fh}->{mailfrom} =~ /\@gmail\.com/ && $Con{$fh}->{SIZE} > 1024000
>
> 1024000 can be larger
>
> Thomas
>
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  03.08.2016 19:08
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> watching the SMTP Connections GUI, it looks like google starts out pretty
> fast for the first 2mb or so, but then really slows down.  Might there be
> something with memory handling on my end?
>
> after x seconds: total bytes transferred
> 10 seconds: 1,400,000 bytes
> 30 seconds: 2,600,000 bytes
> 55 seconds: 3,800,000 bytes
> 90 seconds: 5,300,000 bytes
> 160 seconds: 7,500,000 bytes
>
> Hit 1.4mb in the first 10 seconds, but then slows to a rate of about 2mb a
> minute, sometimes slower.  Does this help you figure out what might be
> going on with gmail?
>
>
>
>
> On Tue, Aug 2, 2016 at 10:40 PM, K Post <nntp.p...@gmail.com> wrote:
>
> > activestate just published net::ssleay 1.77 in their repository. Doesn't
> > seem to make any difference in terms of speed.  Capping out at about 2mb
> a
> > minute with TLS.
> >
> > the ssleay.dll that is in c:\perl\site\lib\auto\Net\SSLeay appears to
> have
> > been updated by the ppm.  ASSP in infostats still says:
> > OpenSSL 1.0.2h
> > OpenSSL-lib 1.0.2g Mar 2016
> >
> > Is that first line my c:\openssl installation from Shining Light (I
> don't
> > know anywhere else that 1.0.2h is installed)?
> > and OpenSSL-lib is the ssleay.dll that is seen in the
> > c:\perl\sit\lib\auto\net\ssleay folder?
> >
> > Does it matter that there's also a ssleay.dll in c:\openssl that is
> surely
> > 1.0.2h?
> >
> > Still, I ask all these questions, but it's only gmail that's giving me a
> > headache.  Other senders all seem fine so far, no nearly as fast as
> without
> > TLS.  For example, I just sent the same 11mb file that google takes
> about 7
> > minutes to send via Outlook.com and it only took 35 seconds.
> >
> > thanks again
> >
> >
> >
> >
> >
> > On Tue, Aug 2, 2016 at 9:44 PM, K Post <nntp.p...@gmail.com> wrote:
> >
> >> scratch that Bob.  I'm still closer to 1.5-2mb per minute despite the
> >> tweaks.
> >>
> >> On Tue, Aug 2, 2016 at 9:36 PM, K Post <nntp.p...@gmail.com> wrote:
> >>
> >>> Thanks Thomas, but what OpenSSL should I be using?  I really don't
> think
> >>> this is the problem, but I might as well eliminate it.  I've got
> >>> activestate's perl 5.20 installed and net::ssleay from the activestate
> >>> ppm.  However,the OpenSSL binaries that I have (I'm talking about the
> FULL
> >>> openssl installation in c:\openssl) not the dll files that net::ssleay
> >>> >might< have, is 1.0.2h from Shiining LIght (
> >>> slproweb.com/products/Win32OpenSSL.html)
> >>>
> >>> ASSP says net::ssleay is OpenSSL 1.0.2g - apparently it hasn't been
> >>> compiled using 1.0.2h yet.  That the readme from net::ssleay talks
> >>> specifically about shining light and that it's best to roll your own
> >>> worries me.
> >>>
> >>> And Bob,
> >>> Thanks for testing this out.  3MB in 25 seconds is about what I'm
> >>> generally seeing now that I've tweaked the performance settings of
> ASSP,
> >>> but without TLS, we can receive a 10mb attachment in just a few
> seconds
> >>> thanks to a fast line.  Curious, if you disable TLS temporarily and
> send
> >>> yourself that same 3mb at

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

activestate just published net::ssleay 1.77 in their repository.  Doesn't
seem to make any difference in terms of speed.  Capping out at about 2mb a
minute with TLS.

the ssleay.dll that is in c:\perl\site\lib\auto\Net\SSLeay appears to have
been updated by the ppm.  ASSP in infostats still says:
OpenSSL 1.0.2h
OpenSSL-lib 1.0.2g Mar 2016

Is that first line my c:\openssl installation from Shining Light (I don't
know anywhere else that 1.0.2h is installed)?
and OpenSSL-lib is the ssleay.dll that is seen in the
c:\perl\sit\lib\auto\net\ssleay folder?

Does it matter that there's also a ssleay.dll in c:\openssl that is surely
1.0.2h?

Still, I ask all these questions, but it's only gmail that's giving me a
headache.  Other senders all seem fine so far, no nearly as fast as without
TLS.  For example, I just sent the same 11mb file that google takes about 7
minutes to send via Outlook.com and it only took 35 seconds.

thanks again





On Tue, Aug 2, 2016 at 9:44 PM, K Post <nntp.p...@gmail.com> wrote:

> scratch that Bob.  I'm still closer to 1.5-2mb per minute despite the
> tweaks.
>
> On Tue, Aug 2, 2016 at 9:36 PM, K Post <nntp.p...@gmail.com> wrote:
>
>> Thanks Thomas, but what OpenSSL should I be using?  I really don't think
>> this is the problem, but I might as well eliminate it.  I've got
>> activestate's perl 5.20 installed and net::ssleay from the activestate
>> ppm.  However,the OpenSSL binaries that I have (I'm talking about the FULL
>> openssl installation in c:\openssl) not the dll files that net::ssleay
>> >might< have, is 1.0.2h from Shiining LIght (
>> slproweb.com/products/Win32OpenSSL.html)
>>
>> ASSP says net::ssleay is OpenSSL 1.0.2g - apparently it hasn't been
>> compiled using 1.0.2h yet.  That the readme from net::ssleay talks
>> specifically about shining light and that it's best to roll your own
>> worries me.
>>
>> And Bob,
>> Thanks for testing this out.  3MB in 25 seconds is about what I'm
>> generally seeing now that I've tweaked the performance settings of ASSP,
>> but without TLS, we can receive a 10mb attachment in just a few seconds
>> thanks to a fast line.  Curious, if you disable TLS temporarily and send
>> yourself that same 3mb attachment from gmail, how long does it take?
>>
>>
>>
>> On Tue, Aug 2, 2016 at 2:04 PM, Thomas Eckardt <
>> thomas.ecka...@thockar.com> wrote:
>>
>>> >Having looked through the Net:SSLEAY readme, there's a bunch that
>>> suggests
>>> >that it's best to compile your own net:ssleay and OpenSSL on the same
>>> >machine with the same settings.
>>>
>>> This will be the case, if you use the PPM from ActiveState. Perl and all
>>> modules are compiled with the same compiler and header files. Net::SSLeay
>>> is compiled static, means it contains all required openssl code.
>>>
>>> >I'd love to find the time to give this a go,
>>> You'll find something better to do, than to compile this module on
>>> windows.
>>>
>>>
>>> Thomas
>>>
>>>
>>>
>>>
>>> Von:K Post <nntp.p...@gmail.com>
>>> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
>>> Datum:  02.08.2016 19:42
>>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>>> servers
>>>
>>>
>>>
>>> Having looked through the Net:SSLEAY readme, there's a bunch that
>>> suggests
>>> that it's best to compile your own net:ssleay and OpenSSL on the same
>>> machine with the same settings. I've not done that, and never have (nor
>>> do
>>> I have the skillset to do much more than run a simple make command).  I'd
>>> love to find the time to give this a go, but what do you all think -
>>> could
>>> this be it?  Why would gmail.com always be bad, but others not (that
>>> I've
>>> seen)?
>>>
>>> On Tue, Aug 2, 2016 at 1:22 PM, Thomas Eckardt
>>> <thomas.ecka...@thockar.com>
>>> wrote:
>>>
>>> > >How do you know the type of encryption that gmail is using?
>>> >
>>> > You'll find it in the Received header line written by assp.
>>> >
>>> > >I have SSLDebug set to level 3,
>>> >
>>> > This helps not much. Most of the SSL-debug output goes to NUL.
>>> >  But if there were errors in SSL - you would see them in the maillog.
>>> >
>>> > >I changed EnableHighPerformace to "very high,"
>>> > I don't recommend to do this. This cuts the cyc

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

scratch that Bob.  I'm still closer to 1.5-2mb per minute despite the
tweaks.

On Tue, Aug 2, 2016 at 9:36 PM, K Post <nntp.p...@gmail.com> wrote:

> Thanks Thomas, but what OpenSSL should I be using?  I really don't think
> this is the problem, but I might as well eliminate it.  I've got
> activestate's perl 5.20 installed and net::ssleay from the activestate
> ppm.  However,the OpenSSL binaries that I have (I'm talking about the FULL
> openssl installation in c:\openssl) not the dll files that net::ssleay
> >might< have, is 1.0.2h from Shiining LIght (
> slproweb.com/products/Win32OpenSSL.html)
>
> ASSP says net::ssleay is OpenSSL 1.0.2g - apparently it hasn't been
> compiled using 1.0.2h yet.  That the readme from net::ssleay talks
> specifically about shining light and that it's best to roll your own
> worries me.
>
> And Bob,
> Thanks for testing this out.  3MB in 25 seconds is about what I'm
> generally seeing now that I've tweaked the performance settings of ASSP,
> but without TLS, we can receive a 10mb attachment in just a few seconds
> thanks to a fast line.  Curious, if you disable TLS temporarily and send
> yourself that same 3mb attachment from gmail, how long does it take?
>
>
>
> On Tue, Aug 2, 2016 at 2:04 PM, Thomas Eckardt <thomas.ecka...@thockar.com
> > wrote:
>
>> >Having looked through the Net:SSLEAY readme, there's a bunch that
>> suggests
>> >that it's best to compile your own net:ssleay and OpenSSL on the same
>> >machine with the same settings.
>>
>> This will be the case, if you use the PPM from ActiveState. Perl and all
>> modules are compiled with the same compiler and header files. Net::SSLeay
>> is compiled static, means it contains all required openssl code.
>>
>> >I'd love to find the time to give this a go,
>> You'll find something better to do, than to compile this module on
>> windows.
>>
>>
>> Thomas
>>
>>
>>
>>
>> Von:K Post <nntp.p...@gmail.com>
>> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
>> Datum:  02.08.2016 19:42
>> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
>> servers
>>
>>
>>
>> Having looked through the Net:SSLEAY readme, there's a bunch that suggests
>> that it's best to compile your own net:ssleay and OpenSSL on the same
>> machine with the same settings. I've not done that, and never have (nor do
>> I have the skillset to do much more than run a simple make command).  I'd
>> love to find the time to give this a go, but what do you all think - could
>> this be it?  Why would gmail.com always be bad, but others not (that I've
>> seen)?
>>
>> On Tue, Aug 2, 2016 at 1:22 PM, Thomas Eckardt
>> <thomas.ecka...@thockar.com>
>> wrote:
>>
>> > >How do you know the type of encryption that gmail is using?
>> >
>> > You'll find it in the Received header line written by assp.
>> >
>> > >I have SSLDebug set to level 3,
>> >
>> > This helps not much. Most of the SSL-debug output goes to NUL.
>> >  But if there were errors in SSL - you would see them in the maillog.
>> >
>> > >I changed EnableHighPerformace to "very high,"
>> > I don't recommend to do this. This cuts the cycle time (poll/select wait
>> > time) in the workers to a minmum. Even if assp is idle - if this is set,
>> > it will permanently poll the sockets and will produce unwanted CPU
>> > workload. I know 'EnableHighPerformace' sounds magic, but it is more
>> > implemented to tweak exceptional environments.
>> > How ever, if your host accepts this workload - it is fine.
>> >
>> > >Anything else I should try tweaking?
>> >
>> > Don't try to much. Tweak (if) one by one step. Use the
>> > 'notes/confighistory.txt' - the old and new values are recoded there.
>> >
>> > I have an idea about the gmail problem. It may be the case, that they
>> > request SSL rehandshakes more or less often depending on the used
>> > certificate and/or cipher to raise the security of the connection. Such
>> a
>> > behavior would slow down the SSL speed - BUT, now the bad news, this is
>> a
>> > client request (made my gmail). Perl's Net::SSLeay has no easy way to
>> > ignore these requests. The only way would be to pipe all SSL packest
>> > through an assp routine (this is possible), which would drop the
>> > renegotiation requests. Such a code will slow down ALL SSL traffic
>> > dramaticaly, if written in pu

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

Thanks Thomas, but what OpenSSL should I be using?  I really don't think
this is the problem, but I might as well eliminate it.  I've got
activestate's perl 5.20 installed and net::ssleay from the activestate
ppm.  However,the OpenSSL binaries that I have (I'm talking about the FULL
openssl installation in c:\openssl) not the dll files that net::ssleay
>might< have, is 1.0.2h from Shiining LIght (
slproweb.com/products/Win32OpenSSL.html)

ASSP says net::ssleay is OpenSSL 1.0.2g - apparently it hasn't been
compiled using 1.0.2h yet.  That the readme from net::ssleay talks
specifically about shining light and that it's best to roll your own
worries me.

And Bob,
Thanks for testing this out.  3MB in 25 seconds is about what I'm generally
seeing now that I've tweaked the performance settings of ASSP, but without
TLS, we can receive a 10mb attachment in just a few seconds thanks to a
fast line.  Curious, if you disable TLS temporarily and send yourself that
same 3mb attachment from gmail, how long does it take?



On Tue, Aug 2, 2016 at 2:04 PM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:

> >Having looked through the Net:SSLEAY readme, there's a bunch that
> suggests
> >that it's best to compile your own net:ssleay and OpenSSL on the same
> >machine with the same settings.
>
> This will be the case, if you use the PPM from ActiveState. Perl and all
> modules are compiled with the same compiler and header files. Net::SSLeay
> is compiled static, means it contains all required openssl code.
>
> >I'd love to find the time to give this a go,
> You'll find something better to do, than to compile this module on
> windows.
>
>
> Thomas
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  02.08.2016 19:42
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Having looked through the Net:SSLEAY readme, there's a bunch that suggests
> that it's best to compile your own net:ssleay and OpenSSL on the same
> machine with the same settings. I've not done that, and never have (nor do
> I have the skillset to do much more than run a simple make command).  I'd
> love to find the time to give this a go, but what do you all think - could
> this be it?  Why would gmail.com always be bad, but others not (that I've
> seen)?
>
> On Tue, Aug 2, 2016 at 1:22 PM, Thomas Eckardt
> <thomas.ecka...@thockar.com>
> wrote:
>
> > >How do you know the type of encryption that gmail is using?
> >
> > You'll find it in the Received header line written by assp.
> >
> > >I have SSLDebug set to level 3,
> >
> > This helps not much. Most of the SSL-debug output goes to NUL.
> >  But if there were errors in SSL - you would see them in the maillog.
> >
> > >I changed EnableHighPerformace to "very high,"
> > I don't recommend to do this. This cuts the cycle time (poll/select wait
> > time) in the workers to a minmum. Even if assp is idle - if this is set,
> > it will permanently poll the sockets and will produce unwanted CPU
> > workload. I know 'EnableHighPerformace' sounds magic, but it is more
> > implemented to tweak exceptional environments.
> > How ever, if your host accepts this workload - it is fine.
> >
> > >Anything else I should try tweaking?
> >
> > Don't try to much. Tweak (if) one by one step. Use the
> > 'notes/confighistory.txt' - the old and new values are recoded there.
> >
> > I have an idea about the gmail problem. It may be the case, that they
> > request SSL rehandshakes more or less often depending on the used
> > certificate and/or cipher to raise the security of the connection. Such
> a
> > behavior would slow down the SSL speed - BUT, now the bad news, this is
> a
> > client request (made my gmail). Perl's Net::SSLeay has no easy way to
> > ignore these requests. The only way would be to pipe all SSL packest
> > through an assp routine (this is possible), which would drop the
> > renegotiation requests. Such a code will slow down ALL SSL traffic
> > dramaticaly, if written in pure perl.
> >
> > >We are using a 2048bit certificate.  It's a wildcard (*.ourcharity.org)
> > >cert, but I don't think that has anything to do with it.
> >
> > Who knows? But to exclude this, you may use an innocent selfcert
> > certificate and key - create it with openssl - for a while.
> > BTW. assp will create such certificate and keys, if the 'assp/certs'
> > folder is empty at startup. :):)
> >
> > Thomas
> >
> >
> >
> >
> > Von:K Post <nntp.p...@gmail

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>Having looked through the Net:SSLEAY readme, there's a bunch that 
suggests
>that it's best to compile your own net:ssleay and OpenSSL on the same
>machine with the same settings.

This will be the case, if you use the PPM from ActiveState. Perl and all 
modules are compiled with the same compiler and header files. Net::SSLeay 
is compiled static, means it contains all required openssl code.

>I'd love to find the time to give this a go,
You'll find something better to do, than to compile this module on 
windows.


Thomas 




Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  02.08.2016 19:42
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Having looked through the Net:SSLEAY readme, there's a bunch that suggests
that it's best to compile your own net:ssleay and OpenSSL on the same
machine with the same settings. I've not done that, and never have (nor do
I have the skillset to do much more than run a simple make command).  I'd
love to find the time to give this a go, but what do you all think - could
this be it?  Why would gmail.com always be bad, but others not (that I've
seen)?

On Tue, Aug 2, 2016 at 1:22 PM, Thomas Eckardt 
<thomas.ecka...@thockar.com>
wrote:

> >How do you know the type of encryption that gmail is using?
>
> You'll find it in the Received header line written by assp.
>
> >I have SSLDebug set to level 3,
>
> This helps not much. Most of the SSL-debug output goes to NUL.
>  But if there were errors in SSL - you would see them in the maillog.
>
> >I changed EnableHighPerformace to "very high,"
> I don't recommend to do this. This cuts the cycle time (poll/select wait
> time) in the workers to a minmum. Even if assp is idle - if this is set,
> it will permanently poll the sockets and will produce unwanted CPU
> workload. I know 'EnableHighPerformace' sounds magic, but it is more
> implemented to tweak exceptional environments.
> How ever, if your host accepts this workload - it is fine.
>
> >Anything else I should try tweaking?
>
> Don't try to much. Tweak (if) one by one step. Use the
> 'notes/confighistory.txt' - the old and new values are recoded there.
>
> I have an idea about the gmail problem. It may be the case, that they
> request SSL rehandshakes more or less often depending on the used
> certificate and/or cipher to raise the security of the connection. Such 
a
> behavior would slow down the SSL speed - BUT, now the bad news, this is 
a
> client request (made my gmail). Perl's Net::SSLeay has no easy way to
> ignore these requests. The only way would be to pipe all SSL packest
> through an assp routine (this is possible), which would drop the
> renegotiation requests. Such a code will slow down ALL SSL traffic
> dramaticaly, if written in pure perl.
>
> >We are using a 2048bit certificate.  It's a wildcard (*.ourcharity.org)
> >cert, but I don't think that has anything to do with it.
>
> Who knows? But to exclude this, you may use an innocent selfcert
> certificate and key - create it with openssl - for a while.
> BTW. assp will create such certificate and keys, if the 'assp/certs'
> folder is empty at startup. :):)
>
> Thomas
>
>
>
>
> Von:K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  02.08.2016 18:34
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Thanks for chiming in Thomas with such a detailed response.
>
> First, when Google gives up, it gives a message like:
>
> Technical details of temporary failure:
>
> Missed upload deadline (899.97s) (state SENT_MESSAGE)
>
> So it's 15 minutes that it'll try to send a file for.  At under 2mb a
> minute, anything over about 25megs (considering overhead) will 
ultimately
> fail.  No good since lots of gmail users send us large files.
>
>
> We're on a 100mbit line, both directions, but I'd happily take a 9.1 mb
> attachment sent over TLS taking 2 minutes.  I suspect when i find out 
what
> the problem is that it'll be MUCh faster than that.
>
> We are using a 2048bit certificate.  It's a wildcard (*.ourcharity.org)
> cert, but I don't think that has anything to do with it.
>
> We're using local storage on the Hypver-V host, RAID 10 with 4 7200rpm 
SAS
> drives.  It's not the fasted disk array, but it seems fine.  I can't see
> slow disks impacting TLS like this if non-TLS connections fly.
>
> The hyper-v host is a dual processor, 2.6ghz, 6 core each, 12mb cache.
> I've got a total of 10 cores assigned to the ASSP guest.
>
> I have SSLDebug set to level 3, but I don't see anything in the maillog.
>  How do y

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

Having looked through the Net:SSLEAY readme, there's a bunch that suggests
that it's best to compile your own net:ssleay and OpenSSL on the same
machine with the same settings. I've not done that, and never have (nor do
I have the skillset to do much more than run a simple make command).  I'd
love to find the time to give this a go, but what do you all think - could
this be it?  Why would gmail.com always be bad, but others not (that I've
seen)?

On Tue, Aug 2, 2016 at 1:22 PM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:

> >How do you know the type of encryption that gmail is using?
>
> You'll find it in the Received header line written by assp.
>
> >I have SSLDebug set to level 3,
>
> This helps not much. Most of the SSL-debug output goes to NUL.
>  But if there were errors in SSL - you would see them in the maillog.
>
> >I changed EnableHighPerformace to "very high,"
> I don't recommend to do this. This cuts the cycle time (poll/select wait
> time) in the workers to a minmum. Even if assp is idle - if this is set,
> it will permanently poll the sockets and will produce unwanted CPU
> workload. I know 'EnableHighPerformace' sounds magic, but it is more
> implemented to tweak exceptional environments.
> How ever, if your host accepts this workload - it is fine.
>
> >Anything else I should try tweaking?
>
> Don't try to much. Tweak (if) one by one step. Use the
> 'notes/confighistory.txt' - the old and new values are recoded there.
>
> I have an idea about the gmail problem. It may be the case, that they
> request SSL rehandshakes more or less often depending on the used
> certificate and/or cipher to raise the security of the connection. Such a
> behavior would slow down the SSL speed - BUT, now the bad news, this is a
> client request (made my gmail). Perl's Net::SSLeay has no easy way to
> ignore these requests. The only way would be to pipe all SSL packest
> through an assp routine (this is possible), which would drop the
> renegotiation requests. Such a code will slow down ALL SSL traffic
> dramaticaly, if written in pure perl.
>
> >We are using a 2048bit certificate.  It's a wildcard (*.ourcharity.org)
> >cert, but I don't think that has anything to do with it.
>
> Who knows? But to exclude this, you may use an innocent selfcert
> certificate and key - create it with openssl - for a while.
> BTW. assp will create such certificate and keys, if the 'assp/certs'
> folder is empty at startup. :):)
>
> Thomas
>
>
>
>
> Von:    K Post <nntp.p...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum:  02.08.2016 18:34
> Betreff:Re: [Assp-test] Inbound TLS from gmail.com addresses /
> servers
>
>
>
> Thanks for chiming in Thomas with such a detailed response.
>
> First, when Google gives up, it gives a message like:
>
> Technical details of temporary failure:
>
> Missed upload deadline (899.97s) (state SENT_MESSAGE)
>
> So it's 15 minutes that it'll try to send a file for.  At under 2mb a
> minute, anything over about 25megs (considering overhead) will ultimately
> fail.  No good since lots of gmail users send us large files.
>
>
> We're on a 100mbit line, both directions, but I'd happily take a 9.1 mb
> attachment sent over TLS taking 2 minutes.  I suspect when i find out what
> the problem is that it'll be MUCh faster than that.
>
> We are using a 2048bit certificate.  It's a wildcard (*.ourcharity.org)
> cert, but I don't think that has anything to do with it.
>
> We're using local storage on the Hypver-V host, RAID 10 with 4 7200rpm SAS
> drives.  It's not the fasted disk array, but it seems fine.  I can't see
> slow disks impacting TLS like this if non-TLS connections fly.
>
> The hyper-v host is a dual processor, 2.6ghz, 6 core each, 12mb cache.
> I've got a total of 10 cores assigned to the ASSP guest.
>
> I have SSLDebug set to level 3, but I don't see anything in the maillog.
>  How do you know the type of encryption that gmail is using?  It would be
> nice to compare how gmail is connecting vs outlook.com which seems much
> faster (though not super fast)
>
> I've got SSL_Version set to
> SSLv23:!SSLv3:!SSLv2
>
> and
> SSL_Cipher_List set to
>
> kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!kECDH:!CAMELLIA128:!IDEA:!SEED
>
> my unscientific test of changing the cipher list to the default doesn't
> seem to make a difference.
>
> MinPollTime is 1, I think it always has been.
> I changed EnableHighPerformace to "very high," changed thread cycle time
> to
> 1000, maintenance thread cycle time to 2000, and rebuildthreadcycletime to
> 15.  That def

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>How do you know the type of encryption that gmail is using?

You'll find it in the Received header line written by assp.

>I have SSLDebug set to level 3,

This helps not much. Most of the SSL-debug output goes to NUL.
 But if there were errors in SSL - you would see them in the maillog.

>I changed EnableHighPerformace to "very high,"
I don't recommend to do this. This cuts the cycle time (poll/select wait 
time) in the workers to a minmum. Even if assp is idle - if this is set, 
it will permanently poll the sockets and will produce unwanted CPU 
workload. I know 'EnableHighPerformace' sounds magic, but it is more 
implemented to tweak exceptional environments.
How ever, if your host accepts this workload - it is fine.

>Anything else I should try tweaking? 

Don't try to much. Tweak (if) one by one step. Use the 
'notes/confighistory.txt' - the old and new values are recoded there.

I have an idea about the gmail problem. It may be the case, that they 
request SSL rehandshakes more or less often depending on the used 
certificate and/or cipher to raise the security of the connection. Such a 
behavior would slow down the SSL speed - BUT, now the bad news, this is a 
client request (made my gmail). Perl's Net::SSLeay has no easy way to 
ignore these requests. The only way would be to pipe all SSL packest 
through an assp routine (this is possible), which would drop the 
renegotiation requests. Such a code will slow down ALL SSL traffic 
dramaticaly, if written in pure perl.

>We are using a 2048bit certificate.  It's a wildcard (*.ourcharity.org)
>cert, but I don't think that has anything to do with it. 

Who knows? But to exclude this, you may use an innocent selfcert 
certificate and key - create it with openssl - for a while.
BTW. assp will create such certificate and keys, if the 'assp/certs' 
folder is empty at startup. :):)

Thomas




Von:K Post <nntp.p...@gmail.com>
An: ASSP development mailing list <assp-test@lists.sourceforge.net>
Datum:  02.08.2016 18:34
Betreff:    Re: [Assp-test] Inbound TLS from gmail.com addresses / 
servers



Thanks for chiming in Thomas with such a detailed response.

First, when Google gives up, it gives a message like:

Technical details of temporary failure:

Missed upload deadline (899.97s) (state SENT_MESSAGE)

So it's 15 minutes that it'll try to send a file for.  At under 2mb a
minute, anything over about 25megs (considering overhead) will ultimately
fail.  No good since lots of gmail users send us large files.


We're on a 100mbit line, both directions, but I'd happily take a 9.1 mb
attachment sent over TLS taking 2 minutes.  I suspect when i find out what
the problem is that it'll be MUCh faster than that.

We are using a 2048bit certificate.  It's a wildcard (*.ourcharity.org)
cert, but I don't think that has anything to do with it.

We're using local storage on the Hypver-V host, RAID 10 with 4 7200rpm SAS
drives.  It's not the fasted disk array, but it seems fine.  I can't see
slow disks impacting TLS like this if non-TLS connections fly.

The hyper-v host is a dual processor, 2.6ghz, 6 core each, 12mb cache.
I've got a total of 10 cores assigned to the ASSP guest.

I have SSLDebug set to level 3, but I don't see anything in the maillog.
 How do you know the type of encryption that gmail is using?  It would be
nice to compare how gmail is connecting vs outlook.com which seems much
faster (though not super fast)

I've got SSL_Version set to
SSLv23:!SSLv3:!SSLv2

and
SSL_Cipher_List set to
kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!kECDH:!CAMELLIA128:!IDEA:!SEED

my unscientific test of changing the cipher list to the default doesn't
seem to make a difference.

MinPollTime is 1, I think it always has been.
I changed EnableHighPerformace to "very high," changed thread cycle time 
to
1000, maintenance thread cycle time to 2000, and rebuildthreadcycletime to
15.  That definitely made a difference in the rebuild time, almost halving
it (not that I really care about that though).

Anything else I should try tweaking?  I don't care if there's high CPU
usage, we have reasonable processing power to spare.

Thank you

On Tue, Aug 2, 2016 at 12:02 PM, Thomas Eckardt 
<thomas.ecka...@thockar.com>
wrote:

> I just made simlar tests with my gmail account. I can't reproduce this
> behavior related to gmail.com.
>
> I've sent a 9.1MB attachment in 133 seconds. Gmail used SMTPS(TLSv1_2
> ECDHE-RSA-AES256-GCM-SHA384)- which is commonly used by many
> clients/servers.
> Sender was mail-qt0-f181.google.com ([209.85.216.181]
> helo=mail-qt0-f181.google.com)
> My line speed is 16MB/s inbound and 4MB/s outbound.
>
> I saw many faster SMTPS connections but also many slower - this may 
depend
> on the usage of my ISP connection.
>
> 133 seconds for such a mail is acceptable (I think).
>
> SSLv2/3:!SSLv3:!SSLv2
&g

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

Thanks for chiming in Thomas with such a detailed response.

First, when Google gives up, it gives a message like:

Technical details of temporary failure:

Missed upload deadline (899.97s) (state SENT_MESSAGE)

So it's 15 minutes that it'll try to send a file for.  At under 2mb a
minute, anything over about 25megs (considering overhead) will ultimately
fail.  No good since lots of gmail users send us large files.


We're on a 100mbit line, both directions, but I'd happily take a 9.1 mb
attachment sent over TLS taking 2 minutes.  I suspect when i find out what
the problem is that it'll be MUCh faster than that.

We are using a 2048bit certificate.  It's a wildcard (*.ourcharity.org)
cert, but I don't think that has anything to do with it.

We're using local storage on the Hypver-V host, RAID 10 with 4 7200rpm SAS
drives.  It's not the fasted disk array, but it seems fine.  I can't see
slow disks impacting TLS like this if non-TLS connections fly.

The hyper-v host is a dual processor, 2.6ghz, 6 core each, 12mb cache.
I've got a total of 10 cores assigned to the ASSP guest.

I have SSLDebug set to level 3, but I don't see anything in the maillog.
 How do you know the type of encryption that gmail is using?  It would be
nice to compare how gmail is connecting vs outlook.com which seems much
faster (though not super fast)

I've got SSL_Version set to
SSLv23:!SSLv3:!SSLv2

and
SSL_Cipher_List set to
kEECDH+ECDSA:kEECDH:kEDH:HIGH:+SHA:+RC4:RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!DSS:!PSK:!SRP:!kECDH:!CAMELLIA128:!IDEA:!SEED

my unscientific test of changing the cipher list to the default doesn't
seem to make a difference.

MinPollTime is 1, I think it always has been.
I changed EnableHighPerformace to "very high," changed thread cycle time to
1000, maintenance thread cycle time to 2000, and rebuildthreadcycletime to
15.  That definitely made a difference in the rebuild time, almost halving
it (not that I really care about that though).

Anything else I should try tweaking?  I don't care if there's high CPU
usage, we have reasonable processing power to spare.

Thank you

On Tue, Aug 2, 2016 at 12:02 PM, Thomas Eckardt 
wrote:

> I just made simlar tests with my gmail account. I can't reproduce this
> behavior related to gmail.com.
>
> I've sent a 9.1MB attachment in 133 seconds. Gmail used SMTPS(TLSv1_2
> ECDHE-RSA-AES256-GCM-SHA384)- which is commonly used by many
> clients/servers.
> Sender was mail-qt0-f181.google.com ([209.85.216.181]
> helo=mail-qt0-f181.google.com)
> My line speed is 16MB/s inbound and 4MB/s outbound.
>
> I saw many faster SMTPS connections but also many slower - this may depend
> on the usage of my ISP connection.
>
> 133 seconds for such a mail is acceptable (I think).
>
> SSLv2/3:!SSLv3:!SSLv2
> DEFAULT:!aNULL:!RC4:!MD5
>
> are my SSL settings - not very strong - I know :):)
>
> the privat key used is 2048 Bit long
>
> In front of assp is the ISP-router and a pfsense 2.3.2 with snort 3.2.9.1
> . Snort is configured the very hard way, except the SMTP rules are a bit
> more weak, because I need some spam.
> ASSP is running on a 4 Core 6GB W2K3 enterprise with an absolute uptodate
> ActivePerl 5.16.3 - using all Plugins, features and a replicated MySQL
> 5.6.
> Domain based mail routing (in- and out-bound) is done by hmailserver
> 5.6.4-B2283.
> All components are configured to use SSL/TLS when ever this is possible.
> For testing purposes I use a FreeBSD 10.2 with Perl 5.20 and ASSP - it
> runs the same way stable like the production system.
>
> You see - nothing magic, but maintenained (except the nice old W2K3 - but
> it works like a swiss made watch with an ETA 7750).
>
> I really don't know what I can do to fix up the SSL/TLS problems.
>
> Only to be complete:
> Backend for the mail environment and LDAP stuff is a Domino 9.0.1FP6.
> All the stuff above (and very much more) is running on a single VMWare
> vSphere 5.5 ( 8x 2.66GHz 48GB / x3650M2).
> Backups are done with EMC-Networker + EBR + DataDomain-VE, stored at a
> QNAP 419P+
>
> Thomas
>
>
>
>
> Von:K Post 
> An: ASSP development mailing list 
> Datum:  02.08.2016 00:07
> Betreff:[Assp-test] Inbound TLS from gmail.com addresses / servers
>
>
>
> I originally thought that we had a problem with all TLS inbound email.  As
> it turns out, my conclusion appears to have been wrong.
>
>
>- There are some SLOW servers outside that are just plain slow (nothing
>I can do there),
>
>- TLS seems to work reasonably fast with most inbound mail, though
>significantly slower than without TLS  (5 seconds for an 11mb file
> without
>tls, vs 45 seconds with TLS on)
>
>- GMAIL.com inbound TLS emails are SLOW, no matter what settings I
> tweak
>
>
> With inbound gmail.com message. if I have TLS off, an 11mb attachment is
> delivered through ASSP in under 5 seconds.  With TLS on it takes close to
> 10 minutes, which gets close to 

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Grayhat]

:: On Tue, 2 Aug 2016 18:02:25 +0200
::

 ::
Thomas Eckardt  wrote:

 
> I really don't know what I can do to fix up the SSL/TLS problems. 

Well, Thomas, if the OP agrees, you may make private contacts and
connect to his ASSP box to run some tests, maybe reproducing the issue
while "at the console" may allow you to see what's going on (just an
idea, and maybe a crazy one, but when everything else fails...)

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

>I'm almost certain that the OpenSSL
>installation is not used by ASSP, but I've not been able to get
>confirmation of that here.

There are system calls to the openssl binary in the code of assp.pl.
Depending on the compilation of Net::SSLeay (static or dynamic) the 
openssl libraries (.dll, .so) are required or not.
So openssl is required by assp - it has to be installed.

Thomas



Von:K Post 
An: ASSP development mailing list 
Datum:  02.08.2016 00:07
Betreff:[Assp-test] Inbound TLS from gmail.com addresses / servers



I originally thought that we had a problem with all TLS inbound email.  As
it turns out, my conclusion appears to have been wrong.


   - There are some SLOW servers outside that are just plain slow (nothing
   I can do there),

   - TLS seems to work reasonably fast with most inbound mail, though
   significantly slower than without TLS  (5 seconds for an 11mb file 
without
   tls, vs 45 seconds with TLS on)

   - GMAIL.com inbound TLS emails are SLOW, no matter what settings I 
tweak


With inbound gmail.com message. if I have TLS off, an 11mb attachment is
delivered through ASSP in under 5 seconds.  With TLS on it takes close to
10 minutes, which gets close to gmail's limit.

I've tested with Outlook.com and that same 11mb attachment comes in 
through
ASSP with TLS on in about 45 seconds.

Sending a 30mb attachment from gmail FAILS because it takes too long. 
gmail
will try for I believe 10 minutes to send a message, then it quits and
retries.  After a couple tries, it sends an NDR.

This is a Windows 2012 R2 server, latest ASSP dev, OpenSSL 1.0.2h 
installed
from slproweb.com/products/Win32OpenSSL.html (though I've also tried with
the OpenSSL I downloaded a while back from the ASSP sourceforge site.
 net::ssleay 1.74 (openssl 1.0.2g).  I'm almost certain that the OpenSSL
installation is not used by ASSP, but I've not been able to get
confirmation of that here.

Just updated IO::Socket::SSL to 2.033.
Net::SMTP:SSL 1.02.

CPU usage as reported by assp is 4.78%.  It's not on the fastest machine 
in
the world (it's a hypver-v guest on a decent machine), but it seems speedy
enough.  24gb ram.  We've got similar physical hosts running Exchange as a
guest without any speed issues whatsoever.

Any other info I can provide to help figure this out?

Disabling TLS for any gmail inbound mail isn't a feasible option, plus I
don't know if it really is just google, or just the way that google
connects which others might too...

Thank you all.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[K Post]

Thanks all for your replies.

Collin,

Sorry that you're experiencing the same thing, but I'm happy that we've
found another installation with a completely dissimilar OS and setup that
is affected.

I saw that Net::SSLeay 1.77 is out, but it's not yet available for Windows
- at least not in the ActiveState repository.  Looking at the changelog, I
don't think it'll make any difference.

The big question is if ALL assp installations are seeing this slowness from
gmail.com specifically.  Could it be something that gmail is doing that
ASSP isn't expecting?

Greyhat,
I've had debugging on, but I don't see anything of note.

and FYI, this server is pretty low usage. Generally only one or 2 sessions
at a time.  I did testing during off hours when there was almost no inbound
email.  Even with only 1 session active, if it was a gmail tls it was crazy
slow, but turn off SSL and POW it flies.


On Tue, Aug 2, 2016 at 5:39 AM, Colin Waring  wrote:

> I have to say I've seen this and I posted about it back in January.
>
> https://sourceforge.net/p/assp/mailman/message/34783916/
>
> Back then I saw problems with Gmail, Yahoo Mail and SMTPRoutes. Since then
> I've occasionally fielded calls from different people saying that emails
> aren't coming through and the solution has been to add the IP to noTLSip.
> The problem was much more significant back in January because I was getting
> lots of complaints whereas now it is only occasional.
>
> I'm on a completely different architecture to you.
>
> Ubuntu 14.04.4 LTS, OpenSSL 1.0.1f (latest from apt), Perl v5.18.2,
> Net::SSLeay 1.74, IO::Socket::SSL 2.033, Net::SMTP::SSL 1.03
>
> I've been using cpanm and cpanoutdated to manage module updates, checking
> from within cpan I can see that a number of modules haven't been done that
> way so I'm running upgrade from within CPAN itself to get things up to
> date. One of the updates is Net:SSLeay 1.77 so I'll see what that does.
>
> All the best,
> Colin Waring.
>
>
> Colin Waring
> Technical Manager
> Dolphin ICT Limited
> T
> +44 (0)151 438 2246 Ext 2003
> www.dolphinict.co.uk
> co...@dolphinict.co.uk
> US15a, Armstrong House, First Avenue, Robin Hood Airport, Doncaster, DN9
> 3GA
>
>
>
>
>
> Dolphin ICT Limited. NOTICE & DISCLAIMER Dolphin ICT Limited, a private
> limited company, with company registration number 6206916, registered in
> the United Kingdom, the registered office of which is at US15a, Armstrong
> House, First Avenue, Robin Hood Airport, Doncaster, DN9 3GA VAT
> registration number GB 918 1896 88.
>
>
>
> -Original Message-
> From: K Post [mailto:nntp.p...@gmail.com]
> Sent: 01 August 2016 23:06
> To: ASSP development mailing list 
> Subject: [Assp-test] Inbound TLS from gmail.com addresses / servers
>
> I originally thought that we had a problem with all TLS inbound email.  As
> it turns out, my conclusion appears to have been wrong.
>
>
>- There are some SLOW servers outside that are just plain slow (nothing
>I can do there),
>
>- TLS seems to work reasonably fast with most inbound mail, though
>significantly slower than without TLS  (5 seconds for an 11mb file
> without
>tls, vs 45 seconds with TLS on)
>
>- GMAIL.com inbound TLS emails are SLOW, no matter what settings I tweak
>
>
> With inbound gmail.com message. if I have TLS off, an 11mb attachment is
> delivered through ASSP in under 5 seconds.  With TLS on it takes close to
> 10 minutes, which gets close to gmail's limit.
>
> I've tested with Outlook.com and that same 11mb attachment comes in
> through ASSP with TLS on in about 45 seconds.
>
> Sending a 30mb attachment from gmail FAILS because it takes too long.
> gmail will try for I believe 10 minutes to send a message, then it quits
> and retries.  After a couple tries, it sends an NDR.
>
> This is a Windows 2012 R2 server, latest ASSP dev, OpenSSL 1.0.2h
> installed from slproweb.com/products/Win32OpenSSL.html (though I've also
> tried with the OpenSSL I downloaded a while back from the ASSP sourceforge
> site.
>  net::ssleay 1.74 (openssl 1.0.2g).  I'm almost certain that the OpenSSL
> installation is not used by ASSP, but I've not been able to get
> confirmation of that here.
>
> Just updated IO::Socket::SSL to 2.033.
> Net::SMTP:SSL 1.02.
>
> CPU usage as reported by assp is 4.78%.  It's not on the fastest machine
> in the world (it's a hypver-v guest on a decent machine), but it seems
> speedy enough.  24gb ram.  We've got similar physical hosts running
> Exchange as a guest without any speed issues whatsoever.
>
> Any other info I can provide to help figure this out?
>
> Disabling TLS for any gmail inbound mail isn't a feasible option, plus I
> don't know if it really is just google, or just the way that google
> connects which others might too...
>
> Thank you all.
>
> --
> ___
> Assp-test 

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Thomas Eckardt]

I just made simlar tests with my gmail account. I can't reproduce this 
behavior related to gmail.com.

I've sent a 9.1MB attachment in 133 seconds. Gmail used SMTPS(TLSv1_2 
ECDHE-RSA-AES256-GCM-SHA384)- which is commonly used by many 
clients/servers.
Sender was mail-qt0-f181.google.com ([209.85.216.181] 
helo=mail-qt0-f181.google.com)
My line speed is 16MB/s inbound and 4MB/s outbound.

I saw many faster SMTPS connections but also many slower - this may depend 
on the usage of my ISP connection.

133 seconds for such a mail is acceptable (I think).

SSLv2/3:!SSLv3:!SSLv2
DEFAULT:!aNULL:!RC4:!MD5

are my SSL settings - not very strong - I know :):)

the privat key used is 2048 Bit long

In front of assp is the ISP-router and a pfsense 2.3.2 with snort 3.2.9.1 
. Snort is configured the very hard way, except the SMTP rules are a bit 
more weak, because I need some spam.
ASSP is running on a 4 Core 6GB W2K3 enterprise with an absolute uptodate 
ActivePerl 5.16.3 - using all Plugins, features and a replicated MySQL 
5.6.
Domain based mail routing (in- and out-bound) is done by hmailserver 
5.6.4-B2283.
All components are configured to use SSL/TLS when ever this is possible.
For testing purposes I use a FreeBSD 10.2 with Perl 5.20 and ASSP - it 
runs the same way stable like the production system.

You see - nothing magic, but maintenained (except the nice old W2K3 - but 
it works like a swiss made watch with an ETA 7750). 

I really don't know what I can do to fix up the SSL/TLS problems. 

Only to be complete:
Backend for the mail environment and LDAP stuff is a Domino 9.0.1FP6.
All the stuff above (and very much more) is running on a single VMWare 
vSphere 5.5 ( 8x 2.66GHz 48GB / x3650M2).
Backups are done with EMC-Networker + EBR + DataDomain-VE, stored at a 
QNAP 419P+

Thomas




Von:K Post 
An: ASSP development mailing list 
Datum:  02.08.2016 00:07
Betreff:[Assp-test] Inbound TLS from gmail.com addresses / servers



I originally thought that we had a problem with all TLS inbound email.  As
it turns out, my conclusion appears to have been wrong.


   - There are some SLOW servers outside that are just plain slow (nothing
   I can do there),

   - TLS seems to work reasonably fast with most inbound mail, though
   significantly slower than without TLS  (5 seconds for an 11mb file 
without
   tls, vs 45 seconds with TLS on)

   - GMAIL.com inbound TLS emails are SLOW, no matter what settings I 
tweak


With inbound gmail.com message. if I have TLS off, an 11mb attachment is
delivered through ASSP in under 5 seconds.  With TLS on it takes close to
10 minutes, which gets close to gmail's limit.

I've tested with Outlook.com and that same 11mb attachment comes in 
through
ASSP with TLS on in about 45 seconds.

Sending a 30mb attachment from gmail FAILS because it takes too long. 
gmail
will try for I believe 10 minutes to send a message, then it quits and
retries.  After a couple tries, it sends an NDR.

This is a Windows 2012 R2 server, latest ASSP dev, OpenSSL 1.0.2h 
installed
from slproweb.com/products/Win32OpenSSL.html (though I've also tried with
the OpenSSL I downloaded a while back from the ASSP sourceforge site.
 net::ssleay 1.74 (openssl 1.0.2g).  I'm almost certain that the OpenSSL
installation is not used by ASSP, but I've not been able to get
confirmation of that here.

Just updated IO::Socket::SSL to 2.033.
Net::SMTP:SSL 1.02.

CPU usage as reported by assp is 4.78%.  It's not on the fastest machine 
in
the world (it's a hypver-v guest on a decent machine), but it seems speedy
enough.  24gb ram.  We've got similar physical hosts running Exchange as a
guest without any speed issues whatsoever.

Any other info I can provide to help figure this out?

Disabling TLS for any gmail inbound mail isn't a feasible option, plus I
don't know if it really is just google, or just the way that google
connects which others might too...

Thank you all.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test




DISCLAIMER:
***
This email and any files transmitted with it may be confidential, legally 
privileged and protected in law and are intended solely for the use of the 

individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no 
known virus in this email!
***

--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Robert K Coffman Jr. -Info From Data Corp.]

On 8/2/2016 5:39 AM, Colin Waring wrote:
> I have to say I've seen this and I posted about it back in January.

Colin,

Would you mind sharing your noTLSip list?  I just implemented TLS on 
Friday and I wouldn't mind avoiding this issue.

- Bob Coffman



--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Colin Waring]

I have to say I've seen this and I posted about it back in January.

https://sourceforge.net/p/assp/mailman/message/34783916/

Back then I saw problems with Gmail, Yahoo Mail and SMTPRoutes. Since then I've 
occasionally fielded calls from different people saying that emails aren't 
coming through and the solution has been to add the IP to noTLSip. The problem 
was much more significant back in January because I was getting lots of 
complaints whereas now it is only occasional.

I'm on a completely different architecture to you.

Ubuntu 14.04.4 LTS, OpenSSL 1.0.1f (latest from apt), Perl v5.18.2, Net::SSLeay 
1.74, IO::Socket::SSL 2.033, Net::SMTP::SSL 1.03

I've been using cpanm and cpanoutdated to manage module updates, checking from 
within cpan I can see that a number of modules haven't been done that way so 
I'm running upgrade from within CPAN itself to get things up to date. One of 
the updates is Net:SSLeay 1.77 so I'll see what that does.

All the best,
Colin Waring.


Colin Waring
Technical Manager
Dolphin ICT Limited
T
+44 (0)151 438 2246 Ext 2003
www.dolphinict.co.uk
co...@dolphinict.co.uk
US15a, Armstrong House, First Avenue, Robin Hood Airport, Doncaster, DN9 3GA





Dolphin ICT Limited. NOTICE & DISCLAIMER Dolphin ICT Limited, a private limited 
company, with company registration number 6206916, registered in the United 
Kingdom, the registered office of which is at US15a, Armstrong House, First 
Avenue, Robin Hood Airport, Doncaster, DN9 3GA VAT registration number GB 918 
1896 88. 



-Original Message-
From: K Post [mailto:nntp.p...@gmail.com]
Sent: 01 August 2016 23:06
To: ASSP development mailing list 
Subject: [Assp-test] Inbound TLS from gmail.com addresses / servers

I originally thought that we had a problem with all TLS inbound email.  As it 
turns out, my conclusion appears to have been wrong.


   - There are some SLOW servers outside that are just plain slow (nothing
   I can do there),

   - TLS seems to work reasonably fast with most inbound mail, though
   significantly slower than without TLS  (5 seconds for an 11mb file without
   tls, vs 45 seconds with TLS on)

   - GMAIL.com inbound TLS emails are SLOW, no matter what settings I tweak


With inbound gmail.com message. if I have TLS off, an 11mb attachment is 
delivered through ASSP in under 5 seconds.  With TLS on it takes close to
10 minutes, which gets close to gmail's limit.

I've tested with Outlook.com and that same 11mb attachment comes in through 
ASSP with TLS on in about 45 seconds.

Sending a 30mb attachment from gmail FAILS because it takes too long. gmail 
will try for I believe 10 minutes to send a message, then it quits and retries. 
 After a couple tries, it sends an NDR.

This is a Windows 2012 R2 server, latest ASSP dev, OpenSSL 1.0.2h installed 
from slproweb.com/products/Win32OpenSSL.html (though I've also tried with the 
OpenSSL I downloaded a while back from the ASSP sourceforge site.
 net::ssleay 1.74 (openssl 1.0.2g).  I'm almost certain that the OpenSSL 
installation is not used by ASSP, but I've not been able to get confirmation of 
that here.

Just updated IO::Socket::SSL to 2.033.
Net::SMTP:SSL 1.02.

CPU usage as reported by assp is 4.78%.  It's not on the fastest machine in the 
world (it's a hypver-v guest on a decent machine), but it seems speedy enough.  
24gb ram.  We've got similar physical hosts running Exchange as a guest without 
any speed issues whatsoever.

Any other info I can provide to help figure this out?

Disabling TLS for any gmail inbound mail isn't a feasible option, plus I don't 
know if it really is just google, or just the way that google connects which 
others might too...

Thank you all.
--
___
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test

Re: [Assp-test] Inbound TLS from gmail.com addresses / servers

[Grayhat]

:: On Mon, 1 Aug 2016 18:06:11 -0400
::