Well let's take this a little deeper. I was just testing two versions, one with gnutls bound libcurl ssl calls in each thread.
It gives me some vector resource nonsense around 450 threads with 768mb of ram. Keep in mind that error is certainly an out of memory situation and nothing to do with my data sets. Further to this it only gets those 450 threads initialized and can not continuously operate them. Now test number two consisted of a nearly identical application however with out major difference. I dropped out the ssl calls in favor of a standard http url to do the exact same workload upon. I was able to built up to 800 threads and maintain them consistently without an issue. This makes me highly suspect of atleast libcurl + gnutls's ssl memory management and that when used in a high thread count environment memory is being sacraficed at some stage. The 8mb default stack is a crash at 350-450 or stability at 800, when the only difference is a well built ssl lib. Tough call, any thoughts? John On Mon, Jun 29, 2009 at 12:54 PM, Phil Blundell <[email protected]> wrote: > On Mon, 2009-06-29 at 12:01 +0300, John D wrote: > > The issue is how much ram curl it is requiring. I literally must > > leave the default stack size within debian at 8mb. > > So with 16gb of ram I am getting 1800-2000 threads. Any more and I > > will encounter a seg fault in relation to thread resources. > > A lower stack and it is curl's chance to seg fault. > > This certainly doesn't sound like the normal curl behaviour. I have a > libcurl-based application running quite happily on a machine which has > only 32MB RAM in total (and no swap). Admittedly this is only with a > single thread, and maybe 10-20 active easy handles at any one time, but > I'm fairly sure curl is not using 8MB of stack space. > > Whereabouts in curl does it segfault if you run it with a smaller stack? > > p. > > >
