Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-24 Thread Armin Rigo
Hi, On Tue, Aug 23, 2011 at 7:28 PM, Yury Selivanov wrote: > If you read that Armin's email carefully, you notice that he talks about a > low-level primitive called "stacklets", which have some limitations, but are > not intended for a regular use.  Greenlets will be implemented on top of > th

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-23 Thread Yury Selivanov
resumed many times. How does PyPy handle > that?" > > Thanks. > > From: Andy > To: Armin Rigo > Cc: "pypy-dev@python.org" > Sent: Sunday, August 21, 2011 2:31 AM > Subject: Re: [pypy-dev] pypy1.6 slow on string-heavy ops > > Thanks Armin. >

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-22 Thread Andy
From: Andy To: Armin Rigo Cc: "pypy-dev@python.org" Sent: Sunday, August 21, 2011 2:31 AM Subject: Re: [pypy-dev] pypy1.6 slow on string-heavy ops Thanks Armin. I dug around and found an email from you titled "Stacklets". Is that the new solution? If I understand correctly

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-20 Thread Andy
running one? In those cases the greenlet will be blocked and resumed many times. How does PyPy handle that? From: Armin Rigo To: Andy Cc: Jacob Biesinger ; "pypy-dev@python.org" Sent: Saturday, August 20, 2011 5:43 AM Subject: Re: [pypy-dev] pypy1.6

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-20 Thread Armin Rigo
Hi Andy, On Fri, Aug 19, 2011 at 5:48 PM, Andy wrote: > I remember reading in this list that PyPy-JIT would not work with greenlet > because of how greentlet manipulated the C stack and there wasn't any easy > solution. No, I think you are confusing two topics. The existing Stackless PyPy, whi

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-19 Thread Andy
Can you elaborate about this new model? I remember reading in this list that PyPy-JIT would not work with greenlet because of how greentlet manipulated the C stack and there wasn't any easy solution. What has changed? So will PyPy JIT work with gevent which is based on libevent (and soon to be

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-19 Thread Armin Rigo
Hi Jacob, On Fri, Aug 19, 2011 at 2:10 AM, Jacob Biesinger wrote: > A bit OT:  The recent release of ipython added some powerful multiprocessing > features using ZeroMQ.  I've only glanced at pypy's extensive threading > optimizations (e.g., greenlets).  Does pypy jit across thread/process > boun

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-18 Thread Jacob Biesinger
Wow thanks for the quick response. The performance is *much, much* better with the suggested list-join. CPython still beats Pypy, but only by a narrow margin: pypy1.6: 1m33.142s CPython 2.7.1: 1m12.092s Thanks for the advice-- I had forgotten about string immutability and its

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-18 Thread Justin Peel
Yes, I just looked at it. For cases like this where there is effectively only one reference to the string being appended to, it just resizes the string in-place and copies in the string being appended which gives it O(N) performance. It is a hack that is available only because of the reference coun

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-18 Thread Aaron DeVore
Python 2.4 introduced a change that helps improve performance of string concatenation, according to its release notes. I don't know anything beyond that. -Aaron DeVore On Thu, Aug 18, 2011 at 3:31 PM, Justin Peel wrote: > Yes, Vincent's way is the better way to go. To elaborate more on the > pro

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-18 Thread Justin Peel
Yes, Vincent's way is the better way to go. To elaborate more on the problem, string appending is O(N^2) while appending to a list and then joining is an O(N) operation. Why CPython is faster than Pypy at doing the less efficient way is something that I'm not fully sure about, but I believe that it

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-18 Thread Vincent Legoll
Hello, Try this: import sys fasta_file = sys.argv[1] # should be *.fa print 'loading dna from', fasta_file chroms = {} dna = [] for l in open(fasta_file): if l.startswith('>'): # new chromosome if len(dna) > 0: chroms[chrom] = ''.join(dna) chrom = l.strip().repl

Re: [pypy-dev] pypy1.6 slow on string-heavy ops

2011-08-18 Thread Jacob Biesinger
A better breakdown... fasta_file = sys.argv[1] # should be *.fa > print 'loading dna from', fasta_file > chroms = {} > dna = None > for l in open(fasta_file): > if l.startswith('>'): # new chromosome > if dna is not None: > chroms[chrom] = dna > chrom = l.strip().