Hi Can you please put it all in a blog post (ideally with examples), it's a lot of useful info.
I'm willing to help On Fri, Sep 5, 2014 at 10:47 AM, Jolitz Ben - bjolit <ben.jol...@acxiom.com> wrote: > I don’t have specific suggestions on Mongo, but I can share what I’ve > learned in a few months of using PyPy and Tornado. > > You want to make use of CFFI in PyPy to accelerate operations that would > usually be slow in Python, namely encryption and database drivers. But > always test first to see if you really need to go to C. > > /Any/ CPython C-Extensions will torpedo performance. Anything blocking (like > the majority of DB drivers) will similarly destroy Tornado performance. > > Code that is overly dynamic also does lousy on PyPy. If your driver has a > ton of paths or makes idiot use of threading.Lock, expect to have an uphill > struggle in optimization. > > When in doubt, ask yourself if the algorithm is appropriate. > > If you can’t make the Python driver performant and there exists a C-API for > it, then it is trivial to wrap it with CFFI. > > If it doesn’t support nonblocking operations, you can find alternatives. For > example in MySQLdb, others have found you can add the Connection._fd to the > IOLoop and use it to do a send_query, read_query. If you still can’t find an > alternative to a blocking call, you can still mimic nonblocking IO by using > pthreads, a work queue and a callback pthread. It’s not perfect and theres a > lot you can optimize, but it can easily allow you to delegate long running > operations to C. > > Another thing I learned was to avoid generating Python-side c callback > pointers frequently. If C is going to callback into Python with your result > to a unique Request, you’re going to need to tag it appropriately. I prefer > to use dictionaries and the callback attribute handed to me by gen.Task and > pass a unique-enough key to C to callback in with. > > As with everything in optimization, profile your code first. If you’re > losing speed heavily somewhere else, then the above will only serve to > distract you. > > Cheers, > > Ben > > > From: Phyo Arkar <phyo.arkarl...@gmail.com> > Date: Thursday, September 4, 2014 at 1:55 PM > To: Ben Jolitz <ben.jol...@acxiom.com> > Cc: pypy-dev <pypy-dev@python.org> > > Subject: Re: [pypy-dev] Pypy Benchmark of Tornado. > > Thanks alot Ben, > Ok , as PyPy is choice of quora and they also use tornado , i might keep > testing on larget projects. > How about mongodb performance on PyPy , i heard its slower due to no C > Extension (no CFFI) for pypy. > Your suggestion will be very appreciated. > > > On Thu, Sep 4, 2014 at 4:56 AM, Jolitz Ben - bjolit <ben.jol...@acxiom.com> > wrote: >> >> I use Tornado and have found PyPy can yield a 30-50% performance increase >> for a moderately complex project. >> >> Ben >> >> From: Phyo Arkar <phyo.arkarl...@gmail.com> >> Date: Wednesday, September 3, 2014 at 8:21 AM >> To: pypy-dev <pypy-dev@python.org> >> Subject: Re: [pypy-dev] Pypy Benchmark of Tornado. >> >> I expect pypy to be faster in those cases but select io is not cpu >> intensive thing to do so no real benefit using pypy here i guess. >> On Wed, Sep 3, 2014 at 4:43 PM, Phyo Arkar <phyo.arkarl...@gmail.com> >> wrote: >> > >> > It just return json document with a few thousand characters (1053 bytes) >> > $siege -c 400 -t 20s -r 2000 http://localhost:9999/js >> > >> > Python 2.7.7: >> > >> > Lifting the server siege... done. >> > >> > Transactions: 14478 hits >> > Availability: 100.00 % >> > Elapsed time: 19.10 secs >> > Data transferred: 14.54 MB >> > Response time: 0.01 secs >> > Transaction rate: 758.01 trans/sec >> > Throughput: 0.76 MB/sec >> > Concurrency: 8.91 >> > Successful transactions: 14478 >> > Failed transactions: 0 >> > Longest transaction: 1.08 seconds >> > Shortest transaction: 0.00 >> > >> > pypy-2.3.1 stable: >> > >> > Transactions: 15149 hits >> > Availability: 100.00 % >> > Elapsed time: 19.63 secs >> > Data transferred: 15.21 MB >> > Response time: 0.02 secs >> > Transaction rate: 771.73 trans/sec >> > Throughput: 0.77 MB/sec >> > Concurrency: 11.92 >> > Successful transactions: 15149 >> > Failed transactions: 0 >> > Longest transaction: 1.09 seconds >> > Shortest transaction: 0.00 >> > >> > >> > >> > pypy--c-jit-73283-912dd9df99a8-linux64 (latest nightly build) >> > >> > Lifting the server siege... done. >> > >> > Transactions: 14361 hits >> > Availability: 100.00 % >> > Elapsed time: 19.13 secs >> > Data transferred: 14.42 MB >> > Response time: 0.03 secs >> > Transaction rate: 750.71 trans/sec >> > Throughput: 0.75 MB/sec >> > Concurrency: 21.53 >> > Successful transactions: 14361 >> > Failed transactions: 0 >> > Longest transaction: 3.03 seconds >> > Shortest transaction: 0.00 >> > >> > >> > >> > >> > Pypy Nightly have some request Randomly get to 3.0 Seconds , normally >> > those requests (in Cpython) are only ~0.001 to 0.002 sec. >> > >> >> It just return json document with a few thousand characters (1053 bytes) >> $siege -c 400 -t 20s -r 2000 http://localhost:9999/js >> >> Python 2.7.7: >> >> Lifting the server siege... done. >> >> Transactions: 14478 hits >> Availability: 100.00 % >> Elapsed time: 19.10 secs >> Data transferred: 14.54 MB >> Response time: 0.01 secs >> Transaction rate: 758.01 trans/sec >> Throughput: 0.76 MB/sec >> Concurrency: 8.91 >> Successful transactions: 14478 >> Failed transactions: 0 >> Longest transaction: 1.08 seconds >> Shortest transaction: 0.00 >> >> pypy-2.3.1 stable: >> >> Transactions: 15149 hits >> Availability: 100.00 % >> Elapsed time: 19.63 secs >> Data transferred: 15.21 MB >> Response time: 0.02 secs >> Transaction rate: 771.73 trans/sec >> Throughput: 0.77 MB/sec >> Concurrency: 11.92 >> Successful transactions: 15149 >> Failed transactions: 0 >> Longest transaction: 1.09 seconds >> Shortest transaction: 0.00 >> >> >> >> pypy--c-jit-73283-912dd9df99a8-linux64 (latest nightly build) >> >> Lifting the server siege... done. >> >> Transactions: 14361 hits >> Availability: 100.00 % >> Elapsed time: 19.13 secs >> Data transferred: 14.42 MB >> Response time: 0.03 secs >> Transaction rate: 750.71 trans/sec >> Throughput: 0.75 MB/sec >> Concurrency: 21.53 >> Successful transactions: 14361 >> Failed transactions: 0 >> Longest transaction: 3.03 seconds >> Shortest transaction: 0.00 >> >> >> >> >> Pypy Nightly have some request Randomly get to 3.0 Seconds , normally >> those requests (in Cpython) are only ~0.001 to 0.002 sec. >> >> >> *************************************************************************** >> The information contained in this communication is confidential, is >> intended only for the use of the recipient named above, and may be legally >> privileged. >> >> If the reader of this message is not the intended recipient, you are >> hereby notified that any dissemination, distribution or copying of this >> communication is strictly prohibited. >> >> If you have received this communication in error, please resend this >> communication to the sender and delete the original message or any copy >> of it from your computer system. >> >> Thank You. >> >> **************************************************************************** >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev@python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> > > > _______________________________________________ > pypy-dev mailing list > pypy-dev@python.org > https://mail.python.org/mailman/listinfo/pypy-dev > _______________________________________________ pypy-dev mailing list pypy-dev@python.org https://mail.python.org/mailman/listinfo/pypy-dev
