No prob, It might be worth not using a proxy handler, but rather deriving TServer and creating a TProcessPoolServer
On Tue, Jan 25, 2011 at 12:45 PM, dean forever <[email protected]>wrote: > ok thank you for your so kind reply > a proxy handler is a good idea > I'am so busy for my other works,I didnot make a program now > I will try your way as soon as possible . > Thank you very much > > 2011/1/25 Dvir Volk <[email protected]> > > > if you want to get multiple processes working with one port per machine, > > you > > also have 2 other options: > > 1. there's a forking server in pything that forks a worker child for each > > request. that's a bit resource consuming, forking per each request, so > I'm > > not using it. > > > > 2. you can implement a proxy handler for all thrift requests (using > python > > descriptors), that pushes the request on a multiprocessing queue. > > then you have a pool of subprocess workers that fetches items from the > > pool, > > processes them and returns the response. > > this will also utilize all the CPUs of your machine without the added > > complexity of multiple ports. > > > > On Tue, Jan 25, 2011 at 4:22 AM, dean forever < > [email protected] > > >wrote: > > > > > different ports didnot work for my situation. > > > I putted python server code in different server machines ,and it > > performed > > > a > > > little better. > > > Thank you very much for your kind advice . > > > I will try your suggestion! > > > > > > 2011/1/24 Dvir Volk <[email protected]> > > > > > > > I'm sure it's completely solvable, the only question is why you want > > > python > > > > in the first place. > > > > > > > > Here's a little trick I did that boosted my own performance a lot: > > > > my setup is a PHP API server that handles REST requests, and is a > > thrift > > > > client. It passes the parameters to thrift, serializes the response > > back > > > to > > > > JSON, and sends it to the user. > > > > > > > > The main problem was the python GIL that bounds all threads to a > single > > > CPU > > > > Core > > > > So I did a very simple thing: since my server is mostly stateless, > what > > > it > > > > does is use the python multiprocessing module to spawn 4-6 thread > pool > > > > servers on consecutive ports. > > > > the php client selects a random port, and I'm utilizing the CPU much > > > better > > > > and Increased the RPS by a lot. > > > > > > > > Another thing with python is, that you can optimize CPU heavy parts > of > > > your > > > > program and offload them to C code, either by loading a DLL, > > > > by building a binary module (like the TBinaryProtocolAccelerated), or > > by > > > > using some C precompiler like Cython. > > > > > > > > I wonder how difficult it would be to wrap the whole thrift server as > a > > > > python module and have only the handler written in python. > > > > > > > > On Mon, Jan 24, 2011 at 5:03 PM, dean forever < > > > [email protected] > > > > >wrote: > > > > > > > > > thanks , Dvir! > > > > > finally ,by testing the server, I found the server didnot perform > > > > well.Only > > > > > 10% CPU usage,comparing to 100% CPU usage of Java Tomcat > > > > > I think i need to find a better server program instead of the > thrift > > > > > default > > > > > server. > > > > > > > > > > > > > > > 2011/1/21 Dvir Volk <[email protected]> > > > > > > > > > > > I don't know about the compact transport, but just a python > server > > > tip > > > > - > > > > > > use TBinaryProtocolAccelerated, handles serialization with C code > > > which > > > > > > makes it faster. > > > > > > Also, try not to have more jmeter threads than max threads on the > > > > server. > > > > > > I'm currently testing thrift with jmeter via a PHP proxy rest > > server, > > > > but > > > > > > these tips should apply when testing directly with java, I guess. > > > > > > > > > > > > On Fri, Jan 21, 2011 at 7:23 AM, dean forever < > > > > > [email protected] > > > > > > >wrote: > > > > > > > > > > > > > When I use the default example settings for java client and > > python > > > > > > server( > > > > > > > like tutorial shows) to do Jmeter Tests,lots of errors such as > > > > > > > "TTransportException: Socket already connected" "'Connection > > reset > > > > by > > > > > > > peer" > > > > > > > "Broken pipe" > > > > > > > happened > > > > > > > > > > > > > > when i try TFramedTransport TCompactProtocol in java, it's all > > > right > > > > > > > > > > > > > > in python when I use TFramedTransport ,it shows that no listen > > and > > > > > > accept > > > > > > > method in it.When I added them by > > > > > > > def listen(self): > > > > > > > return self.__trans.listen() > > > > > > > > > > > > > > the python server can run normally > > > > > > > > > > > > > > > > > > > > > but when I start Java Client to request, the python server show > > > lots > > > > of > > > > > > > NameError in TCompactProtocol.py > > > > > > > > > > > > > > Anyone has a better solution to this combine???Please help me > > find > > > a > > > > > > best > > > > > > > solution for java Client and python Server maintaining > > > > > > > high concurrents > > > > > > > > > > > > > > > > > > > > > > > > > > > >
