haha here you again! finally i found my biggest problem is dns!!!
2011/1/25 Dvir Volk <[email protected]>: > No prob, > It might be worth not using a proxy handler, but rather deriving TServer and > creating a TProcessPoolServer > > On Tue, Jan 25, 2011 at 12:45 PM, dean forever > <[email protected]>wrote: > >> ok thank you for your so kind reply >> a proxy handler is a good idea >> I'am so busy for my other works,I didnot make a program now >> I will try your way as soon as possible . >> Thank you very much >> >> 2011/1/25 Dvir Volk <[email protected]> >> >> > if you want to get multiple processes working with one port per machine, >> > you >> > also have 2 other options: >> > 1. there's a forking server in pything that forks a worker child for each >> > request. that's a bit resource consuming, forking per each request, so >> I'm >> > not using it. >> > >> > 2. you can implement a proxy handler for all thrift requests (using >> python >> > descriptors), that pushes the request on a multiprocessing queue. >> > then you have a pool of subprocess workers that fetches items from the >> > pool, >> > processes them and returns the response. >> > this will also utilize all the CPUs of your machine without the added >> > complexity of multiple ports. >> > >> > On Tue, Jan 25, 2011 at 4:22 AM, dean forever < >> [email protected] >> > >wrote: >> > >> > > different ports didnot work for my situation. >> > > I putted python server code in different server machines ,and it >> > performed >> > > a >> > > little better. >> > > Thank you very much for your kind advice . >> > > I will try your suggestion! >> > > >> > > 2011/1/24 Dvir Volk <[email protected]> >> > > >> > > > I'm sure it's completely solvable, the only question is why you want >> > > python >> > > > in the first place. >> > > > >> > > > Here's a little trick I did that boosted my own performance a lot: >> > > > my setup is a PHP API server that handles REST requests, and is a >> > thrift >> > > > client. It passes the parameters to thrift, serializes the response >> > back >> > > to >> > > > JSON, and sends it to the user. >> > > > >> > > > The main problem was the python GIL that bounds all threads to a >> single >> > > CPU >> > > > Core >> > > > So I did a very simple thing: since my server is mostly stateless, >> what >> > > it >> > > > does is use the python multiprocessing module to spawn 4-6 thread >> pool >> > > > servers on consecutive ports. >> > > > the php client selects a random port, and I'm utilizing the CPU much >> > > better >> > > > and Increased the RPS by a lot. >> > > > >> > > > Another thing with python is, that you can optimize CPU heavy parts >> of >> > > your >> > > > program and offload them to C code, either by loading a DLL, >> > > > by building a binary module (like the TBinaryProtocolAccelerated), or >> > by >> > > > using some C precompiler like Cython. >> > > > >> > > > I wonder how difficult it would be to wrap the whole thrift server as >> a >> > > > python module and have only the handler written in python. >> > > > >> > > > On Mon, Jan 24, 2011 at 5:03 PM, dean forever < >> > > [email protected] >> > > > >wrote: >> > > > >> > > > > thanks , Dvir! >> > > > > finally ,by testing the server, I found the server didnot perform >> > > > well.Only >> > > > > 10% CPU usage,comparing to 100% CPU usage of Java Tomcat >> > > > > I think i need to find a better server program instead of the >> thrift >> > > > > default >> > > > > server. >> > > > > >> > > > > >> > > > > 2011/1/21 Dvir Volk <[email protected]> >> > > > > >> > > > > > I don't know about the compact transport, but just a python >> server >> > > tip >> > > > - >> > > > > > use TBinaryProtocolAccelerated, handles serialization with C code >> > > which >> > > > > > makes it faster. >> > > > > > Also, try not to have more jmeter threads than max threads on the >> > > > server. >> > > > > > I'm currently testing thrift with jmeter via a PHP proxy rest >> > server, >> > > > but >> > > > > > these tips should apply when testing directly with java, I guess. >> > > > > > >> > > > > > On Fri, Jan 21, 2011 at 7:23 AM, dean forever < >> > > > > [email protected] >> > > > > > >wrote: >> > > > > > >> > > > > > > When I use the default example settings for java client and >> > python >> > > > > > server( >> > > > > > > like tutorial shows) to do Jmeter Tests,lots of errors such as >> > > > > > > "TTransportException: Socket already connected" "'Connection >> > reset >> > > > by >> > > > > > > peer" >> > > > > > > "Broken pipe" >> > > > > > > happened >> > > > > > > >> > > > > > > when i try TFramedTransport TCompactProtocol in java, it's all >> > > right >> > > > > > > >> > > > > > > in python when I use TFramedTransport ,it shows that no listen >> > and >> > > > > > accept >> > > > > > > method in it.When I added them by >> > > > > > > def listen(self): >> > > > > > > return self.__trans.listen() >> > > > > > > >> > > > > > > the python server can run normally >> > > > > > > >> > > > > > > >> > > > > > > but when I start Java Client to request, the python server show >> > > lots >> > > > of >> > > > > > > NameError in TCompactProtocol.py >> > > > > > > >> > > > > > > Anyone has a better solution to this combine???Please help me >> > find >> > > a >> > > > > > best >> > > > > > > solution for java Client and python Server maintaining >> > > > > > > high concurrents >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
