Re: Clients glomming onto a listener
I have found that fork() on modern machines as a negligible affect on performance and in fact I almost always use inetd instead of writing my own servers, mainly because it is dead reliable, easier to code, and again seems to have negligible affect on performance. One would have to do millions upon millions of connects to notice or care. Having said that, I use AIX mostly, and that performs better under load than Linux on Intel, and even Linux on the IBM p series platform. I would do it cheap and easy and worry about performance after-the-fact. Eric At 04:46 PM 5/10/2011, you wrote: On 10 May 2011, at 4:13 PM, David Schwartz wrote: On 5/10/2011 2:10 AM, John Hollingum wrote: Pretty much immediately after the accept the program forks a handler, but the rogue clients must be glomming onto the main process before the SSL negotiation is complete. Calling 'fork' with an accepted SSL connection has all kinds of known issues. The fundamental problem is that there are many operations that must occur both before and after the 'fork', for different reasons, and obviously can't do both. You could accept just the TCP connection in the main process and do all of the SSL handshake in the forked process (I think IO::Socket::SSL-start_SSL() is what you want for that) --- this would not be a high-performance approach (no SSL session cache, fork overhead) but if it's fast enough it's fast enough. It's possible to use openssl in a non-blocking, event-driven manner but I don't think Perl's SSL modules expose enough of the openssl API to do that. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org Eric S. Eberhard (928) 567-3727 Voice (928) 567-6122 Fax (928) 301-7537 Cell Vertical Integrated Computer Systems, LLC Metropolis Support, LLC For Metropolis support and VICS MBA Supporthttp://www.vicsmba.com Pictures of Snake in Spring http://www.facebook.com/album.php?aid=115547id=1409661701l=1c375e1f49 Pictures of Camp Verde http://www.facebook.com/album.php?aid=12771id=1409661701l=fc0e0a2bcf Pictures of Land Cruiser in Sedona http://www.facebook.com/album.php?aid=50953id=1409661701 Pictures of Flagstaff area near our cabin http://www.facebook.com/album.php?aid=12750id=1409661701 Pictures of Cheryl in a Horse Show http://www.facebook.com/album.php?aid=32484id=1409661701 Pictures of the AZ Desert http://www.facebook.com/album.php?aid=58827id=1409661701 (You can see why we love this state :-) ) __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: Clients glomming onto a listener
On Wed, May 11, 2011 at 08:39:49AM -0700, Eric S. Eberhard wrote: I have found that fork() on modern machines as a negligible affect on performance and in fact I almost always use inetd instead of writing my own servers, mainly because it is dead reliable, easier to code, and again seems to have negligible affect on performance. One would have to do millions upon millions of connects to notice or care. Having said that, I use AIX mostly, and that performs better under load than Linux on Intel, and even Linux on the IBM p series platform. I would do it cheap and easy and worry about performance after-the-fact. Eric Let's not start an OS A is better than OS B discussion here. You can safely fork single-threaded OpenSSL servers right after accept(3), and handle the SSL connection in a child. This makes the memory-resident session cache ineffective, but you can use callbacks to implement an external (Berkeley DB similar or shared memory, ...) session cache. Forking after SSL_accept() is tricky, since your parent process will have partial SSL connections in progress for other clients when a given handshake completes (event-based connection management) or will serialize all handshakes, but as you've observed that's not a good option. So, my suggestion is that a forking server is fine, just use an external session cache. The Postfix SMTP server is an example of this model. There before the TLS handshake, we also have an SMTP STARTTLS handshake, but that does not alter the analysis in any substantive way, just a few more packets to exchange before the TLS connection is ready. Note, Postfix is pre-forking, rather than forking, so there is a pool of processes, that serially accept connections, but this too does not impact the design analysis. - You can use a single process with event-based I/O. - You can use multiple threads in a single process. - You can fork after accept(2) and use an external session cache - You can pre-fork and handle clients serially one per process, with re-use of processes for another client after a client hangs-up. This too requires an external session cache. -- Viktor. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: Clients glomming onto a listener
I was not trying to compare O/S, only point out that my experience is more out of the AIX world than Linux world. I also want to point out again what I was saying ... you don't need to make a server and you don't need to fork() and all kinds of complicated stuff if you write it for inetd. You don't even need to write socket code (stdin/stdout read/write is all you need). The O/S will create the processes and clean them up on disconnects and so forth. Unless you are super performance limited, this is the best way to go because it always works and is always reliable (if inetd fails to function on a Unix O/S then the machine is essentially toast anyway). In addition it is more easily portable if you care about porting to more than one Unix. Using select is not always supported, socket flags not always the same, etc. All a non-issue under inetd. Eric At 08:57 AM 5/11/2011, you wrote: On Wed, May 11, 2011 at 08:39:49AM -0700, Eric S. Eberhard wrote: I have found that fork() on modern machines as a negligible affect on performance and in fact I almost always use inetd instead of writing my own servers, mainly because it is dead reliable, easier to code, and again seems to have negligible affect on performance. One would have to do millions upon millions of connects to notice or care. Having said that, I use AIX mostly, and that performs better under load than Linux on Intel, and even Linux on the IBM p series platform. I would do it cheap and easy and worry about performance after-the-fact. Eric Let's not start an OS A is better than OS B discussion here. You can safely fork single-threaded OpenSSL servers right after accept(3), and handle the SSL connection in a child. This makes the memory-resident session cache ineffective, but you can use callbacks to implement an external (Berkeley DB similar or shared memory, ...) session cache. Forking after SSL_accept() is tricky, since your parent process will have partial SSL connections in progress for other clients when a given handshake completes (event-based connection management) or will serialize all handshakes, but as you've observed that's not a good option. So, my suggestion is that a forking server is fine, just use an external session cache. The Postfix SMTP server is an example of this model. There before the TLS handshake, we also have an SMTP STARTTLS handshake, but that does not alter the analysis in any substantive way, just a few more packets to exchange before the TLS connection is ready. Note, Postfix is pre-forking, rather than forking, so there is a pool of processes, that serially accept connections, but this too does not impact the design analysis. - You can use a single process with event-based I/O. - You can use multiple threads in a single process. - You can fork after accept(2) and use an external session cache - You can pre-fork and handle clients serially one per process, with re-use of processes for another client after a client hangs-up. This too requires an external session cache. -- Viktor. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org Eric S. Eberhard (928) 567-3727 Voice (928) 567-6122 Fax (928) 301-7537 Cell Vertical Integrated Computer Systems, LLC Metropolis Support, LLC For Metropolis support and VICS MBA Supporthttp://www.vicsmba.com Pictures of Snake in Spring http://www.facebook.com/album.php?aid=115547id=1409661701l=1c375e1f49 Pictures of Camp Verde http://www.facebook.com/album.php?aid=12771id=1409661701l=fc0e0a2bcf Pictures of Land Cruiser in Sedona http://www.facebook.com/album.php?aid=50953id=1409661701 Pictures of Flagstaff area near our cabin http://www.facebook.com/album.php?aid=12750id=1409661701 Pictures of Cheryl in a Horse Show http://www.facebook.com/album.php?aid=32484id=1409661701 Pictures of the AZ Desert http://www.facebook.com/album.php?aid=58827id=1409661701 (You can see why we love this state :-) ) __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: Clients glomming onto a listener
Eric, you must be really kidding this time :), servers with this architecture are susceptible to dos and what not..am sure for embedded systems where memory is a big limiting factor the best would be async design, also code becomes easily portable in future. On Wed, May 11, 2011 at 10:39 AM, Eric S. Eberhard fl...@vicsmba.comwrote: I have found that fork() on modern machines as a negligible affect on performance and in fact I almost always use inetd instead of writing my own servers, mainly because it is dead reliable, easier to code, and again seems to have negligible affect on performance. One would have to do millions upon millions of connects to notice or care. Having said that, I use AIX mostly, and that performs better under load than Linux on Intel, and even Linux on the IBM p series platform. I would do it cheap and easy and worry about performance after-the-fact. Eric At 04:46 PM 5/10/2011, you wrote: On 10 May 2011, at 4:13 PM, David Schwartz wrote: On 5/10/2011 2:10 AM, John Hollingum wrote: Pretty much immediately after the accept the program forks a handler, but the rogue clients must be glomming onto the main process before the SSL negotiation is complete. Calling 'fork' with an accepted SSL connection has all kinds of known issues. The fundamental problem is that there are many operations that must occur both before and after the 'fork', for different reasons, and obviously can't do both. You could accept just the TCP connection in the main process and do all of the SSL handshake in the forked process (I think IO::Socket::SSL-start_SSL() is what you want for that) --- this would not be a high-performance approach (no SSL session cache, fork overhead) but if it's fast enough it's fast enough. It's possible to use openssl in a non-blocking, event-driven manner but I don't think Perl's SSL modules expose enough of the openssl API to do that. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org Eric S. Eberhard (928) 567-3727 Voice (928) 567-6122 Fax (928) 301-7537 Cell Vertical Integrated Computer Systems, LLC Metropolis Support, LLC For Metropolis support and VICS MBA Supporthttp://www.vicsmba.com Pictures of Snake in Spring http://www.facebook.com/album.php?aid=115547id=1409661701l=1c375e1f49 Pictures of Camp Verde http://www.facebook.com/album.php?aid=12771id=1409661701l=fc0e0a2bcf Pictures of Land Cruiser in Sedona http://www.facebook.com/album.php?aid=50953id=1409661701 Pictures of Flagstaff area near our cabin http://www.facebook.com/album.php?aid=12750id=1409661701 Pictures of Cheryl in a Horse Show http://www.facebook.com/album.php?aid=32484id=1409661701 Pictures of the AZ Desert http://www.facebook.com/album.php?aid=58827id=1409661701 (You can see why we love this state :-) ) __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: Clients glomming onto a listener
Performance is related to the application. For example, a system that accepts 10 SSL connects per year has different requirements than one that accepts 1000 per second. Obviously there is a middle ground. My point is that theoretical performance differences are very real in the later case, and of no consequence in the first case. Cost of software development and upkeep and system management is much lower using say inetd and not bothering to make a server. I have systems with thousands of SSL connections per minute, holding 500-1000 at a time, going through inetd on a modest AIX box and have zero performance issue. Don't even notice they are there and they take low single digits of CPU usage combined. Depending on the application, usage, hardware, cost of software development, cost of software upkeep, simplicity in system management -- the answer to what is the best way is different. I often find people ignoring that simple concept and developing very complex software to be theoretically faster ... only to end up with complex and buggy code that is hard to manage in an environment where the extra performance was not needed. One has to also consider the cost to develop and manage. So there is no right or wrong answer, I am trying to get the programmer to think ... does he really need, in his case, blistering performance? Can he do it with a simple inetd module (which later could be the core for his own server)? Does he want it up quick and easy with no real management issues? I am only spurring thought, not telling anyone what is right or wrong in their case :-) E At 10:10 AM 5/11/2011, you wrote: Eric, you must be really kidding this time :), servers with this architecture are susceptible to dos and what not..am sure for embedded systems where memory is a big limiting factor the best would be async design, also code becomes easily portable in future. On Wed, May 11, 2011 at 10:39 AM, Eric S. Eberhard mailto:fl...@vicsmba.comfl...@vicsmba.com wrote: I have found that fork() on modern machines as a negligible affect on performance and in fact I almost always use inetd instead of writing my own servers, mainly because it is dead reliable, easier to code, and again seems to have negligible affect on performance. One would have to do millions upon millions of connects to notice or care. Having said that, I use AIX mostly, and that performs better under load than Linux on Intel, and even Linux on the IBM p series platform. I would do it cheap and easy and worry about performance after-the-fact. Eric At 04:46 PM 5/10/2011, you wrote: On 10 May 2011, at 4:13 PM, David Schwartz wrote: On 5/10/2011 2:10 AM, John Hollingum wrote: Pretty much immediately after the accept the program forks a handler, but the rogue clients must be glomming onto the main process before the SSL negotiation is complete. Calling 'fork' with an accepted SSL connection has all kinds of known issues. The fundamental problem is that there are many operations that must occur both before and after the 'fork', for different reasons, and obviously can't do both. You could accept just the TCP connection in the main process and do all of the SSL handshake in the forked process (I think IO::Socket::SSL-start_SSL() is what you want for that) --- this would not be a high-performance approach (no SSL session cache, fork overhead) but if it's fast enough it's fast enough. It's possible to use openssl in a non-blocking, event-driven manner but I don't think Perl's SSL modules expose enough of the openssl API to do that. __ OpenSSL Project http://www.openssl.orghttp://www.openssl.org User Support Mailing List mailto:openssl-users@openssl.orgopenssl-users@openssl.org Automated List Manager mailto:majord...@openssl.orgmajord...@openssl.org Eric S. Eberhard tel:%28928%29%20567-3727(928) 567-3727 Voice tel:%28928%29%20567-6122(928) 567-6122 Fax tel:%28928%29%20301-7537(928) 301-7537 Cell Vertical Integrated Computer Systems, LLC Metropolis Support, LLC For Metropolis support and VICS MBA Supporthttp://www.vicsmba.comhttp://www.vicsmba.com Pictures of Snake in Spring http://www.facebook.com/album.php?aid=115547id=1409661701l=1c375e1f49http://www.facebook.com/album.php?aid=115547id=1409661701l=1c375e1f49 Pictures of Camp Verde http://www.facebook.com/album.php?aid=12771id=1409661701l=fc0e0a2bcfhttp://www.facebook.com/album.php?aid=12771id=1409661701l=fc0e0a2bcf Pictures of Land Cruiser in Sedona http://www.facebook.com/album.php?aid=50953id=1409661701http://www.facebook.com/album.php?aid=50953id=1409661701 Pictures of Flagstaff area near our cabin http://www.facebook.com/album.php?aid=12750id=1409661701http://www.facebook.com/album.php?aid=12750id=1409661701 Pictures of Cheryl in a Horse Show
Re: Clients glomming onto a listener
On 5/10/2011 2:10 AM, John Hollingum wrote: I have a service written in Perl, running on Linux that presents a very simple SSL listener. When this service is hit, it identifies the connecting node from its certificate/peer address and just sends some xml to them containing data from some files in the queue directory that contains their data. All the client does is to open a socket and start reading. This works, but it is susceptible to problems which I believe are caused by clients with bad internet connections (the pathology suggests this). It seems that something unpleasant occurs in the SSL handshake process which causes the socket to hang indefinitely. Nobody else can connect when this has happened. Well, that's what happens if you take a single-threaded listener and tell it complete the SSL handshake. But that's all a distraction. The fact is that the service shouldn't be susceptible to the vagaries of what stupid clients or bad networks do. Right, that's why a single-threaded listener that *ever* waits for a client is a terrible design. Pretty much immediately after the accept the program forks a handler, but the rogue clients must be glomming onto the main process before the SSL negotiation is complete. Calling 'fork' with an accepted SSL connection has all kinds of known issues. The fundamental problem is that there are many operations that must occur both before and after the 'fork', for different reasons, and obviously can't do both. I can't help thinking that I should be able to tell SSL to have some sensible (fairly aggressive) timeouts on connections that fail to complete an SSL handshake. Is this possible? Does it sound like I'm even identifying the problem correctly? It wouldn't help, since it's not SSL that needs to time out but the socket itself. If you want to timeout the socket operations, you have to do it. OpenSSL just reads and writes to and from the BIOs and thus the socket. I'm wondering if I could get more control by using accept in non-blocking mode. Is this worth looking into? That would help if your issue was blocking in 'accept'. Since you are a single-threaded listener that has to handle multiple clients, why are you doing *anything* with blocking sockets calls? That's an obvious red flag. Blocking socket operations only exist to allow sockets to behave like terminals or files for applications that don't know anything specific about sockets. Your application not only is sockets-specific fundamentally but has to do something very unusual that requires non-blocking operations. Timeouts are a possible fix, so long as you only accept connections from trusted clients in the first place. Otherwise, you create a trivial DoS attack. DS __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
Re: Clients glomming onto a listener
On 10 May 2011, at 4:13 PM, David Schwartz wrote: On 5/10/2011 2:10 AM, John Hollingum wrote: Pretty much immediately after the accept the program forks a handler, but the rogue clients must be glomming onto the main process before the SSL negotiation is complete. Calling 'fork' with an accepted SSL connection has all kinds of known issues. The fundamental problem is that there are many operations that must occur both before and after the 'fork', for different reasons, and obviously can't do both. You could accept just the TCP connection in the main process and do all of the SSL handshake in the forked process (I think IO::Socket::SSL-start_SSL() is what you want for that) --- this would not be a high-performance approach (no SSL session cache, fork overhead) but if it's fast enough it's fast enough. It's possible to use openssl in a non-blocking, event-driven manner but I don't think Perl's SSL modules expose enough of the openssl API to do that. __ OpenSSL Project http://www.openssl.org User Support Mailing Listopenssl-users@openssl.org Automated List Manager majord...@openssl.org
