Your comment about the Java implementation of TThreadPoolServer got me thinking. Turns out this class will use pretty much any Java thread pool/executor configuration you throw at it, so TThreadPoolServer itself isn't a blocking queue, it just uses a blocking queue by default. Because of that, if I queue up 1000 jobs, 10 of them run and the other 990 fail and typically cause the server to crash.
So, at this point in the code for my server: TThreadPoolServer.Args args = new TThreadPoolServer.Args(serverTransport); args.maxWorkerThreads(10); args.processor(processor); I added: args.executorService(new ScheduledThreadPoolExecutor(10)); Then created the server object: TServer server = new TThreadPoolServer(args); Now Java is smart enough to queue (schedule) the other 990 jobs, so I no longer have this issue. :) (Also setting -Xmx2048m and -XX:-UseGCOverheadLimit helped too.) Thanks for your help! Best, Diane ________________________________________ From: Ben Craig [[email protected]] Sent: Friday, April 05, 2013 4:34 PM To: [email protected] Subject: RE: Question about Multithreaded Thrift Servers Are you running out of ports / file descriptors? I have attempted to write performance tests in the past that stressed connect / disconnect, and I have routinely run into the problem of exhausting ports. This isn't because I failed to call close, but because "closing" a socket doesn't immediately free up the resources in the TCP/IP stack. It typically takes a few minutes for those resources to be freed. From: "Napolitano, Diane" <[email protected]> To: "[email protected]" <[email protected]>, Date: 04/05/2013 03:32 PM Subject: RE: Question about Multithreaded Thrift Servers Hmm, interesting. These requests are actually clients, several thousands of them actually :) , calling methods on the server. We have a cluster setup here, and I have one cluster job that is the server, and several thousand cluster jobs that are each one client, or one request to the server. So, hmm. Thanks, Diane ________________________________________ From: Ben Craig [[email protected]] Sent: Friday, April 05, 2013 4:20 PM To: [email protected] Subject: RE: Question about Multithreaded Thrift Servers You only mention requests, so I'm not sure if these are all going over the same connection or not. One other thing that gets strange is that naming across the different languages isn't consistent. I am most comfortable with the C++ implementation, and so that's the behavior that I described. I vaguely recall that the Java implementation of TThreadPoolServer is more similar to C++'s NonBlockingServer, and I'm not sure what that does. From: "Napolitano, Diane" <[email protected]> To: "[email protected]" <[email protected]>, Date: 04/05/2013 03:02 PM Subject: RE: Question about Multithreaded Thrift Servers Really? So if threads 0 thru 9 are running, and 0-8 finish, the remaining requests in the queue (is there even a queue?) won't make use of 0-8? It's fine if they sit there waiting for 9 to finish, but in this case, even after 9 finishes, additional requests aren't being executed. If > 10 requests are sent to it, the server throws an exception, and only executes the first 10. Also, sometimes it isn't the 10th request that is the slowest, it's some other one (the 2nd or whatever). Sorry, I'm confused because your answer makes it sound like what I'm seeing shouldn't be happening. :/ Thanks, Diane ________________________________________ From: Ben Craig [[email protected]] Sent: Friday, April 05, 2013 3:55 PM To: [email protected] Subject: Re: Question about Multithreaded Thrift Servers The threaded server uses one thread per connection. It does not get a request, and have an available thread process the request. This means that if you send 20 requests, and the 10th request takes a long time to process, the last ten requests will be stalled. From: "Napolitano, Diane" <[email protected]> To: "[email protected]" <[email protected]>, Date: 04/05/2013 02:44 PM Subject: Question about Multithreaded Thrift Servers Hello, I have a Thrift server (just "NameOfServer" here) written in Java which is initialized with the following inside of an inner class called ServerThread: TThreadPoolServer.Args args = new TThreadPoolServer.Args(serverTransport); args.maxWorkerThreads(10); args.processor(processor); TServer server = new TThreadPoolServer(args); server.serve(); Then NameOfServer has two objects: public static NameOfServerHandler handler; public static NameOfServer.Processor processor; And then a ServerThread object is created in my server's main: handler = new NameOfServerHandler(); processor = new NameOfServer.Processor(handler); Runnable r = new ServerThread(processor, portNum); new Thread(r).start(); My question is: am I doing this right? Because when I send more than ten requests to the server, it throws an exception and dies, and all requests beyond the first 10 are stalled. Unless I'm misunderstanding threads, Java, and everything else, aren't these additional requests supposed to be queued and then served when one of the 10 threads becomes available? If you need more code/context than this, definitely let me know. Thanks, Diane
