Re: svn commit: r295136 - in head: sys/kern sys/netinet sys/sys usr.bin/netstat

[Xin LI]

On Tuesday, February 2, 2016, Alfred Perlstein  wrote:

>
>
> On 2/2/16 11:39 AM, John Baldwin wrote:
>
>> On Tuesday, February 02, 2016 05:57:59 AM Alfred Perlstein wrote:
>>
>>> Author: alfred
>>> Date: Tue Feb  2 05:57:59 2016
>>> New Revision: 295136
>>> URL: https://svnweb.freebsd.org/changeset/base/295136
>>>
>>> Log:
>>>Increase max allowed backlog for listen sockets
>>>from short to int.
>>>   PR: 203922
>>>Submitted by: White Knight 
>>>MFC After: 4 weeks
>>>
>> You do realize that this breaks the ABI of the sysctls used to fetch
>> connection lists (and so will break existing binaries like ucd-snmpd,
>> etc.)
>> and thus can't be MFC'd right?
>>
> OK, I will not MFC it.
>
> Is it worthwhile to extend the xsocket to have padding so that in 11.x and
> beyond we can allow for some changes in the structure?
>
> Another idea I had was to include a version number with the sysctl request
> so that we can send back versioned structures, let me know what you think
> about that.
>
> The first idea will take not more than a few days for me to accomplish,
> the second (versioning the sysctl) probably a few more days than that.


We have similar construction (versioned ioctl) in FreeBSD ZFS but the main
goal is to keep system bootable, not to support all functionalities.  Do we
change the structure often and is it important enough to warrant the
complexity?  I think kmem interface have a simple size check to guard
against world/kernel inconsistency.

I would second John's comment on the necessity of the change though, if one
already have 32K of *backlogged* connections, it's probably not very useful
to allow more coming in.  It sounds like the application itself is
seriously broken, and unless expanding the field have some performance
benefit, I don't think it should stay.

-- 
Xin LI  https://www.delphij.net/
FreeBSD - The Power to Serve! Live free or die
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r295136 - in head: sys/kern sys/netinet sys/sys usr.bin/netstat

[John Baldwin]

On Tuesday, February 02, 2016 05:57:59 AM Alfred Perlstein wrote:
> Author: alfred
> Date: Tue Feb  2 05:57:59 2016
> New Revision: 295136
> URL: https://svnweb.freebsd.org/changeset/base/295136
> 
> Log:
>   Increase max allowed backlog for listen sockets
>   from short to int.
>   
>   PR: 203922
>   Submitted by: White Knight 
>   MFC After: 4 weeks

You do realize that this breaks the ABI of the sysctls used to fetch
connection lists (and so will break existing binaries like ucd-snmpd, etc.)
and thus can't be MFC'd right?

Also, when this patch was brought up on the lists there was the question of
if it is really beneficial to have more than 32k sockets that you haven't
accepted yet.  It's one thing to have lots of concurrent active sockets that
you are servicing, but if your application is so backlogged that there are
32k sockets waiting on accept() it's hard to imagine why having more than
32k sockets waiting on accept() is useful.

-- 
John Baldwin
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r295136 - in head: sys/kern sys/netinet sys/sys usr.bin/netstat

[Alfred Perlstein]

On 2/2/16 12:23 PM, Xin LI wrote:



On Tuesday, February 2, 2016, Alfred Perlstein > wrote:




On 2/2/16 11:39 AM, John Baldwin wrote:

On Tuesday, February 02, 2016 05:57:59 AM Alfred Perlstein wrote:

Author: alfred
Date: Tue Feb  2 05:57:59 2016
New Revision: 295136
URL: https://svnweb.freebsd.org/changeset/base/295136

Log:
   Increase max allowed backlog for listen sockets
   from short to int.
  PR: 203922
   Submitted by: White Knight 
   MFC After: 4 weeks

You do realize that this breaks the ABI of the sysctls used to
fetch
connection lists (and so will break existing binaries like
ucd-snmpd, etc.)
and thus can't be MFC'd right?

OK, I will not MFC it.

Is it worthwhile to extend the xsocket to have padding so that in
11.x and beyond we can allow for some changes in the structure?

Another idea I had was to include a version number with the sysctl
request so that we can send back versioned structures, let me know
what you think about that.

The first idea will take not more than a few days for me to
accomplish, the second (versioning the sysctl) probably a few more
days than that.


We have similar construction (versioned ioctl) in FreeBSD ZFS but the 
main goal is to keep system bootable, not to support all 
functionalities.  Do we change the structure often and is it important 
enough to warrant the complexity?  I think kmem interface have a 
simple size check to guard against world/kernel inconsistency.


I would second John's comment on the necessity of the change though, 
if one already have 32K of *backlogged* connections, it's probably not 
very useful to allow more coming in.  It sounds like the application 
itself is seriously broken, and unless expanding the field have some 
performance benefit, I don't think it should stay.


Imagine a hugely busy image board like 2ch.net, if there is a single 
hiccup, it's very possible to start dropping connections.


I stand by the scalability improvement offered here even though it is an 
edge case.  Linux appears to offer 32 bits of backlog and so should we.


-Alfred
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r295136 - in head: sys/kern sys/netinet sys/sys usr.bin/netstat

[Slawa Olhovchenkov]

On Tue, Feb 02, 2016 at 12:35:47PM -0800, Alfred Perlstein wrote:

> > I would second John's comment on the necessity of the change though, 
> > if one already have 32K of *backlogged* connections, it's probably not 
> > very useful to allow more coming in.  It sounds like the application 
> > itself is seriously broken, and unless expanding the field have some 
> > performance benefit, I don't think it should stay.
> 
> Imagine a hugely busy image board like 2ch.net, if there is a single 
> hiccup, it's very possible to start dropping connections.

In reality start dropping connections in any case: nobody will be
infinity wait of accept (user close browser and go away, etc).

Also, if you have more then 4K backloged connections -- you have
problem, you can't process all connections request and in next second
you will be have 8K, after next second -- 12K and etc.
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r295136 - in head: sys/kern sys/netinet sys/sys usr.bin/netstat

[Alfred Perlstein]

On 2/2/16 1:09 PM, Slawa Olhovchenkov wrote:

On Tue, Feb 02, 2016 at 12:35:47PM -0800, Alfred Perlstein wrote:


I would second John's comment on the necessity of the change though,
if one already have 32K of *backlogged* connections, it's probably not
very useful to allow more coming in.  It sounds like the application
itself is seriously broken, and unless expanding the field have some
performance benefit, I don't think it should stay.

Imagine a hugely busy image board like 2ch.net, if there is a single
hiccup, it's very possible to start dropping connections.

In reality start dropping connections in any case: nobody will be
infinity wait of accept (user close browser and go away, etc).

Also, if you have more then 4K backloged connections -- you have
problem, you can't process all connections request and in next second
you will be have 8K, after next second -- 12K and etc.


Thank you Slawa,

I am pretty familiar with what you are describing which are "cascade 
failures", however in order to understand why such a change makes sense 
I can give you a little early history lesson on a project I developed 
under FreeBSD, and then explain why such a project would probably not 
work with FreeBSD as a platform today (we would have to use Linux or 
custom patches).


Here is that use case:

Back in 1999 I wrote a custom webserver using FreeBSD that was 
processing over 1500 connections per second.


What we were doing was tracking web hits using "hidden gifs".  Now this 
was 1999 with only 100mbit hardware and a pentium 400mhz.  Mind you I 
was doing this with cpu to spare, so having an influx of additional hits 
was OK.


Meaning I could easily deal with backlog.

Now what was important about this case was that EVERY time we served the 
data we were able to monitize it and pay for my salary at the time which 
was working on SMP for FreeBSD and a bunch of other patches.  Any lost 
hits / broken connections would easily cost us money, which in turn 
meant less time on FreeBSD and less time fixing things to scale.


In our case the user would not really know if our "page" didn't load 
because we were just an invisible gif.


So back to the example, let's scale that out to today's numbers.

100mbps -> 10gigE, so that would be 1500 conn/sec -> 150,000 conn/sec.  
so basically at 0.20 of a second of any sort of latency I will be 
overflowing the listen queue and dropping connections.


Now when you still have CPU to spare because connections *are* precious, 
then the model makes sense to slightly over-provision the servers to 
allow for somebacklog to be processed.


So, in today's day and age, it really does make sense to allow for 
buffering more than 32k connections, particularly if the developer knows 
what he is doing.


Does this help explain the reasoning?

thanks!

-Alfred

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r295136 - in head: sys/kern sys/netinet sys/sys usr.bin/netstat

[Adrian Chadd]

On 2 February 2016 at 13:21, Alfred Perlstein  wrote:
>
>
> On 2/2/16 1:09 PM, Slawa Olhovchenkov wrote:
>>
>> On Tue, Feb 02, 2016 at 12:35:47PM -0800, Alfred Perlstein wrote:
>>
 I would second John's comment on the necessity of the change though,
 if one already have 32K of *backlogged* connections, it's probably not
 very useful to allow more coming in.  It sounds like the application
 itself is seriously broken, and unless expanding the field have some
 performance benefit, I don't think it should stay.
>>>
>>> Imagine a hugely busy image board like 2ch.net, if there is a single
>>> hiccup, it's very possible to start dropping connections.
>>
>> In reality start dropping connections in any case: nobody will be
>> infinity wait of accept (user close browser and go away, etc).
>>
>> Also, if you have more then 4K backloged connections -- you have
>> problem, you can't process all connections request and in next second
>> you will be have 8K, after next second -- 12K and etc.
>>
> Thank you Slawa,
>
> I am pretty familiar with what you are describing which are "cascade
> failures", however in order to understand why such a change makes sense I
> can give you a little early history lesson on a project I developed under
> FreeBSD, and then explain why such a project would probably not work with
> FreeBSD as a platform today (we would have to use Linux or custom patches).
>
> Here is that use case:
>
> Back in 1999 I wrote a custom webserver using FreeBSD that was processing
> over 1500 connections per second.
>
> What we were doing was tracking web hits using "hidden gifs".  Now this was
> 1999 with only 100mbit hardware and a pentium 400mhz.  Mind you I was doing
> this with cpu to spare, so having an influx of additional hits was OK.
>
> Meaning I could easily deal with backlog.
>
> Now what was important about this case was that EVERY time we served the
> data we were able to monitize it and pay for my salary at the time which was
> working on SMP for FreeBSD and a bunch of other patches.  Any lost hits /
> broken connections would easily cost us money, which in turn meant less time
> on FreeBSD and less time fixing things to scale.
>
> In our case the user would not really know if our "page" didn't load because
> we were just an invisible gif.
>
> So back to the example, let's scale that out to today's numbers.
>
> 100mbps -> 10gigE, so that would be 1500 conn/sec -> 150,000 conn/sec.  so
> basically at 0.20 of a second of any sort of latency I will be overflowing
> the listen queue and dropping connections.
>
> Now when you still have CPU to spare because connections *are* precious,
> then the model makes sense to slightly over-provision the servers to allow
> for somebacklog to be processed.
>
> So, in today's day and age, it really does make sense to allow for buffering
> more than 32k connections, particularly if the developer knows what he is
> doing.
>
> Does this help explain the reasoning?

Just to add to this: the VM system under ridiculous load (like say,
deciding it can dirty most of your half-terabyte of RAM and get behind
in writing stuff to disk) can cause the system to pause for little
pieces of time. It sucks, but it happens. 0.20 seconds isn't all that
long.

And that's at 150,000 conn/sec. There's TCP locking work that will
hopefully increase that value..


-a
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r295136 - in head: sys/kern sys/netinet sys/sys usr.bin/netstat

[Slawa Olhovchenkov]

On Tue, Feb 02, 2016 at 01:21:52PM -0800, Alfred Perlstein wrote:

> >>> I would second John's comment on the necessity of the change though,
> >>> if one already have 32K of *backlogged* connections, it's probably not
> >>> very useful to allow more coming in.  It sounds like the application
> >>> itself is seriously broken, and unless expanding the field have some
> >>> performance benefit, I don't think it should stay.
> >> Imagine a hugely busy image board like 2ch.net, if there is a single
> >> hiccup, it's very possible to start dropping connections.
> > In reality start dropping connections in any case: nobody will be
> > infinity wait of accept (user close browser and go away, etc).
> >
> > Also, if you have more then 4K backloged connections -- you have
> > problem, you can't process all connections request and in next second
> > you will be have 8K, after next second -- 12K and etc.
> >


> In our case the user would not really know if our "page" didn't load 
> because we were just an invisible gif.
> 
> So back to the example, let's scale that out to today's numbers.
> 
> 100mbps -> 10gigE, so that would be 1500 conn/sec -> 150,000 conn/sec.  
> so basically at 0.20 of a second of any sort of latency I will be 
> overflowing the listen queue and dropping connections.

OK, you talk about very specilal case -- extremaly short connections,
about one data packet. Yes, in this case you got this behaivor.
I think case of 2ch is different.

> Now when you still have CPU to spare because connections *are* precious, 
> then the model makes sense to slightly over-provision the servers to 
> allow for somebacklog to be processed.
> 
> So, in today's day and age, it really does make sense to allow for 
> buffering more than 32k connections, particularly if the developer knows 
> what he is doing.
> 
> Does this help explain the reasoning?

Yes, some special cases may be exist.
___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"

Re: svn commit: r295136 - in head: sys/kern sys/netinet sys/sys usr.bin/netstat

[Alfred Perlstein]

On 2/2/16 11:39 AM, John Baldwin wrote:

On Tuesday, February 02, 2016 05:57:59 AM Alfred Perlstein wrote:

Author: alfred
Date: Tue Feb  2 05:57:59 2016
New Revision: 295136
URL: https://svnweb.freebsd.org/changeset/base/295136

Log:
   Increase max allowed backlog for listen sockets
   from short to int.
   
   PR: 203922

   Submitted by: White Knight 
   MFC After: 4 weeks

You do realize that this breaks the ABI of the sysctls used to fetch
connection lists (and so will break existing binaries like ucd-snmpd, etc.)
and thus can't be MFC'd right?

OK, I will not MFC it.

Is it worthwhile to extend the xsocket to have padding so that in 11.x 
and beyond we can allow for some changes in the structure?


Another idea I had was to include a version number with the sysctl 
request so that we can send back versioned structures, let me know what 
you think about that.


The first idea will take not more than a few days for me to accomplish, 
the second (versioning the sysctl) probably a few more days than that.


thanks,
-Alfred

___
svn-src-head@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/svn-src-head
To unsubscribe, send any mail to "svn-src-head-unsubscr...@freebsd.org"