Re: l = range(int(1E9))
On Sun, 3 May 2015 02:51 pm, Ian Kelly wrote: On Sat, May 2, 2015 at 5:51 PM, Terry Reedy tjre...@udel.edu wrote: On 5/2/2015 5:31 PM, Ian Kelly wrote: Would it have been better if range() had been implemented as xrange() from the beginning? Sure, that would have been great. Except for one small detail: the iterator protocol didn't exist back then. For loops originally used the getitem iterator protocol. xrange objects have a __getitem__ method, but not __iter__ or __next__. As Mark pointed out, they were introduced in 1993. I'm aware of getitem iterators; just didn't realize that xrange used it or was that old. Yep, xrange objects are *not* iterators: py r = xrange(100) py iter(r) is r False py next(r) Traceback (most recent call last): File stdin, line 1, in module TypeError: xrange object is not an iterator -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sun, 3 May 2015 07:40 am, Jon Ribbens wrote: On 2015-05-02, BartC b...@freeuk.com wrote: So do I, I think, if no-one is willing to admit that the original way of implementing range() was a glaring mistake. I think the issue is that nobody else here thinks the original way of iterating was to use range(), for anything other than extremely small ranges. For information, it looks like xrange() was added on 26 Oct 1993, which pre-dates Python 1.0. Ah yes, you are right: my searching failed to find xrange in Python 1.0.1, but it was actually introduced in 1.0.0. This is from the Misc/NEWS file: * New function xrange() creates a range object. Its arguments are the same as those of range(), and when used in a for loop a range objects also behaves identical. The advantage of xrange() over range() is that its representation (if the range contains many elements) is much more compact than that of range(). The disadvantage is that the result cannot be used to initialize a list object or for the Python idiom [RED, GREEN, BLUE] = range(3). On some modern architectures, benchmarks have shown that for i in range(...): ... actually executes *faster* than for i in xrange(...): ..., but on memory starved machines like PCs running DOS range(10) may be just too big to be represented at all... -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sun, 3 May 2015 08:33 am, BartC wrote: OK, so it's just an irritation then, as a workaround has been available for a long time. (For example, if you use xrange, it won't work on 3.x. If you use range, then it might be inefficient on 2.x.) That is trivially easy to deal with. Put this at the top of your module: try: xrange except NameError: xrange = range and then use xrange throughout the module. Or if you prefer: try: range = xrange except NameError: pass and just use range. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sun, 3 May 2015 12:16 pm, Mark Lawrence wrote: I doubt that six will ever make the standard library as 2.7 only has another five years in official support. By that time I suppose we'll to going through the porting pain all over again with the transition from Python 3 to Python 4. Alright, alright, only joking Guido has said that Python 4 will just be an incremental update from 3.x. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sun, 3 May 2015 02:17 am, BartC wrote: But for looping over a simple integer range, then using 'range' to denote the range (and build a list as it used to do), was how it was done. And earlier on people would have been porting coding code to Python at which point a straightforward 'for i=a to b' loop suddenly acquired a substantial overhead it didn't have before! The oldest version of Python that is documented on the python.org site is 1.4, and it had xrange: https://docs.python.org/release/1.4/lib/node26.html The oldest version of Python I have is Python 0.9.1, which has range but not xrange: steve@runes:~/personal/python/Interpreters/python-0.9.1$ ./python0.9.1 range built-in function 'builtin.range' xrange Unhandled exception: undefined name: xrange Stack backtrace (innermost last): File stdin, line 1 but then Python 0.9 was missing a lot of features, such as double quoted strings, and various operators: a Parsing error: file stdin, line 1: a ^ Unhandled exception: run-time error: syntax error 2**3 Parsing error: file stdin, line 1: 2**3 ^ Unhandled exception: run-time error: syntax error It's unfair to consider anything before version 1.0: it is obvious that pre-1.0 Python was still incomplete and a work in progress. If you check the versions here: http://legacy.python.org/download/releases/src/ you will see that xrange was available in 1.1, but (probably) not in 1.0. So xrange has been available since almost the start. By version 1.4, or possibly 1.5, the general advice given as I remember it was that if the number of loops was small, range was faster than xrange, but if it was large, xrange used less memory. On the basis of two software development principles: (1) release early, release often; (2) avoid premature optimization; I don't think there is any problem with range being released before the need for xrange was proven. Python, especially in the early days, was considered more of a scripting language than a general purpose language, and scripting languages often lack a C-style for-loop, using a foreach loop instead. E.g. I believe the canonical way to loop in bash is something like: for $i in `seq start stop` do ... (by memory). If we were creating Python from scratch now, we would never have had an eager range() that returns a list. But back in the early 1990s, that wasn't as obvious as it is in hindsight. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sun, May 3, 2015 at 8:32 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: scripting languages often lack a C-style for-loop, using a foreach loop instead. E.g. I believe the canonical way to loop in bash is something like: for $i in `seq start stop` do ... (by memory). Newer versions of bash have grown an alternative syntax for that common case, but if you need to support older forms, or other shells, then yes, it's something like that. Maybe without the dollar sign. (I can never remember the exact syntax, and I know bash is really finicky about where you put newlines and where you don't, but it looks something like what you had.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sun, 3 May 2015 07:28 am, Tony the Tiger wrote: On Fri, 01 May 2015 14:42:04 +1000, Steven D'Aprano wrote: use l as a variable name, as it looks too much like 1 If you use a better font, they are very different. Besides, a variable name cannot start with a digit (nor can it be a single digit), so it's a given that it's an 'l'. How do you know its a variable? x+1 Is that 1 the constant or l the variable? Yes, use a better font is good advice, if you can -- you don't always have control over the machine you are working on, and the fonts which you have access to -- but even with a better font, they still look two similar for comfort. It's just common sense[1] to avoid easily confusable names whenever possible: advice = 23 advise = 23 makenewhttpfactorywidgetcreator = True makenewhttpfactoryuidgetcreator = False regardless of how awesome your font is. Unless you're spelling out each word letter by letter, sometimes you will misread words. Another one is rn versus m, especially at small text sizes, or if you're silly or unlucky enough to be using Arial. [1] So rare it's a superpower: http://i0.wp.com/lolzombie.com/wp-content/uploads/2010/10/8eqf.jpeg -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sun, May 3, 2015 at 9:15 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: Or if you prefer: try: range = xrange except NameError: pass and just use range. I prefer this idiom, on the basis that code should be written for the more recent version, and have minimal code to support older versions. Then when you drop support for the older versions, all you have to do is delete a bit at the top. The same principle applies to the __future__ import system - you can declare that your Python 2.5 code is able to use the 'with' statement, but you can't (and shouldn't) make your 2.6+ code use 'with' as a name. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sun, May 3, 2015 at 9:16 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Sun, 3 May 2015 12:16 pm, Mark Lawrence wrote: I doubt that six will ever make the standard library as 2.7 only has another five years in official support. By that time I suppose we'll to going through the porting pain all over again with the transition from Python 3 to Python 4. Alright, alright, only joking Guido has said that Python 4 will just be an incremental update from 3.x. There is still, as I understand it, the possibility that Python 4.0 will ditch some things that will have been deprecated for a long time by 3.9. But yes, it'll be no more upheavalous to go from 3.9 to 4.0 than from 3.8 to 3.9. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 03/05/2015 12:30, Chris Angelico wrote: On Sun, May 3, 2015 at 9:16 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: On Sun, 3 May 2015 12:16 pm, Mark Lawrence wrote: I doubt that six will ever make the standard library as 2.7 only has another five years in official support. By that time I suppose we'll to going through the porting pain all over again with the transition from Python 3 to Python 4. Alright, alright, only joking Guido has said that Python 4 will just be an incremental update from 3.x. There is still, as I understand it, the possibility that Python 4.0 will ditch some things that will have been deprecated for a long time by 3.9. But yes, it'll be no more upheavalous to go from 3.9 to 4.0 than from 3.8 to 3.9. ChrisA https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_GET_SIZE -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 30/04/2015 18:20, Ben Finney wrote: Jon Ribbens jon+use...@unequivocal.co.uk writes: If you use xrange() instead of range() then you will get an iterator which will return each of the numbers in turn without any need to create an enormous list of all of them. If you use Python 3 instead of the obsolescent Python 2, the ‘range’ callable has this sensible behaviour by default. When I first looked at Python 20 or so years ago this seemed to be the standard way of writing a for-loop: for i in range(N): I remember being completely astonished at the time that 'range' actually created a list of values from 0 to N-1. Python was already known to be slow yet it deliberately crippled itself by using just about the slowest method imaginable of executing a simple iterative loop? By first creating a list of a million objects! And sometimes you weren't even interested in the values but just wanted to execute something N times so it was a wasted effort. That was eventually fixed with xrange, but why do it like that in the first place? (At the time, I was creating bytecode languages as part of the applications I was writing. An empty for-loop executed just one bytecode per iteration, and it was also the fastest bytecode instruction. An empty for-loop executed three Python bytecodes per iteration last time I looked. It seems that Python used to like making a rod for its own back.) -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 02/05/2015 16:26, BartC wrote: On 30/04/2015 18:20, Ben Finney wrote: Jon Ribbens jon+use...@unequivocal.co.uk writes: If you use xrange() instead of range() then you will get an iterator which will return each of the numbers in turn without any need to create an enormous list of all of them. If you use Python 3 instead of the obsolescent Python 2, the ‘range’ callable has this sensible behaviour by default. When I first looked at Python 20 or so years ago this seemed to be the standard way of writing a for-loop: for i in range(N): I remember being completely astonished at the time that 'range' actually created a list of values from 0 to N-1. Python was already known to be slow yet it deliberately crippled itself by using just about the slowest method imaginable of executing a simple iterative loop? By first creating a list of a million objects! And sometimes you weren't even interested in the values but just wanted to execute something N times so it was a wasted effort. That was eventually fixed with xrange, but why do it like that in the first place? (At the time, I was creating bytecode languages as part of the applications I was writing. An empty for-loop executed just one bytecode per iteration, and it was also the fastest bytecode instruction. An empty for-loop executed three Python bytecodes per iteration last time I looked. It seems that Python used to like making a rod for its own back.) I first started maybe 14 years ago and the standard way of writing a for loop was, and still is:- for item in items: When did this change, or has it always been this way and you were simply using an idiom from other languages? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 02/05/2015 16:40, Mark Lawrence wrote: On 02/05/2015 16:26, BartC wrote: On 30/04/2015 18:20, Ben Finney wrote: Jon Ribbens jon+use...@unequivocal.co.uk writes: If you use xrange() instead of range() then you will get an iterator which will return each of the numbers in turn without any need to create an enormous list of all of them. If you use Python 3 instead of the obsolescent Python 2, the ‘range’ callable has this sensible behaviour by default. When I first looked at Python 20 or so years ago this seemed to be the standard way of writing a for-loop: for i in range(N): I remember being completely astonished at the time that 'range' actually created a list of values from 0 to N-1. I first started maybe 14 years ago and the standard way of writing a for loop was, and still is:- for item in items: When did this change, or has it always been this way and you were simply using an idiom from other languages? Your example is the equivalent of 'forall' in other languages, where you iterate over the values of some collection of data. I agree that most for-loops in Pythonic code probably fall into that category. But for looping over a simple integer range, then using 'range' to denote the range (and build a list as it used to do), was how it was done. And earlier on people would have been porting coding code to Python at which point a straightforward 'for i=a to b' loop suddenly acquired a substantial overhead it didn't have before! -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 02/05/2015 17:17, BartC wrote: On 02/05/2015 16:40, Mark Lawrence wrote: On 02/05/2015 16:26, BartC wrote: On 30/04/2015 18:20, Ben Finney wrote: Jon Ribbens jon+use...@unequivocal.co.uk writes: If you use xrange() instead of range() then you will get an iterator which will return each of the numbers in turn without any need to create an enormous list of all of them. If you use Python 3 instead of the obsolescent Python 2, the ‘range’ callable has this sensible behaviour by default. When I first looked at Python 20 or so years ago this seemed to be the standard way of writing a for-loop: for i in range(N): I remember being completely astonished at the time that 'range' actually created a list of values from 0 to N-1. I first started maybe 14 years ago and the standard way of writing a for loop was, and still is:- for item in items: When did this change, or has it always been this way and you were simply using an idiom from other languages? Your example is the equivalent of 'forall' in other languages, where you iterate over the values of some collection of data. I agree that most for-loops in Pythonic code probably fall into that category. But for looping over a simple integer range, then using 'range' to denote the range (and build a list as it used to do), was how it was done. And earlier on people would have been porting coding code to Python at which point a straightforward 'for i=a to b' loop suddenly acquired a substantial overhead it didn't have before! All you are saying is that they didn't bother reading the docs and learning how to write a *PYTHON* for loop. Failing that don't bother using range, just directly convert your (say) C loop into Python. I really don't see any issue here at all. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Mark Lawrence breamore...@yahoo.co.uk: On 02/05/2015 19:34, BartC wrote: [...] [...] [...] [...] [...] [...] [...] [...] [...] [...] [...] [...] [...] [...] [etc etc etc] I give up. Mark, you do a commendable job admonishing the forum participants against top-posting. Let me bring up another highly recommended practice: trimming, or pruning, your quotes. The proper order is Quote 1 (properly pruned) Your response 1 Quote 2 (properly pruned) Your response 2 URL: http://lipas.uwasa.fi/~ts/http/quote.html Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 02/05/2015 17:39, Mark Lawrence wrote: On 02/05/2015 17:17, BartC wrote: On 02/05/2015 16:40, Mark Lawrence wrote: for item in items: When did this change, or has it always been this way and you were simply using an idiom from other languages? Your example is the equivalent of 'forall' in other languages, where you iterate over the values of some collection of data. I agree that most for-loops in Pythonic code probably fall into that category. But for looping over a simple integer range, then using 'range' to denote the range (and build a list as it used to do), was how it was done. And earlier on people would have been porting coding code to Python at which point a straightforward 'for i=a to b' loop suddenly acquired a substantial overhead it didn't have before! All you are saying is that they didn't bother reading the docs and learning how to write a *PYTHON* for loop. Failing that don't bother using range, just directly convert your (say) C loop into Python. I really don't see any issue here at all. OK, so it's the programmer's fault if as fundamental a concept as a for-loop ranging over integers is implemented inefficiently. He has to transform it into high-level terms, or has to reconstruct it somehow using a while-loop and an incrementing loop index. Now I understand (why Python has been beset for so long with performance problems, if that is a typical attitude!). BTW, why did they introduce the 'xrange' thing; for fun? Or did someone realise there might have been an issue after all? (I tried an empty loop counting to 1 billion in Python 2.x, using 'for i in range'. It ran out of memory. Counting to 100 million instead, it worked, but still used a massive 1.5GB RAM while doing so (and took 6 seconds to count to 100M, not too bad for Python) Outside Python, it might typically take a few seconds to count to 1 billion, and would use virtually no memory. Why would anyone want to loop over all those numbers instead of iterating over actual data? For any number of reasons. Counting how many happy numbers are in that range for one!) -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 02/05/2015 19:34, BartC wrote: On 02/05/2015 17:39, Mark Lawrence wrote: On 02/05/2015 17:17, BartC wrote: On 02/05/2015 16:40, Mark Lawrence wrote: for item in items: When did this change, or has it always been this way and you were simply using an idiom from other languages? Your example is the equivalent of 'forall' in other languages, where you iterate over the values of some collection of data. I agree that most for-loops in Pythonic code probably fall into that category. But for looping over a simple integer range, then using 'range' to denote the range (and build a list as it used to do), was how it was done. And earlier on people would have been porting coding code to Python at which point a straightforward 'for i=a to b' loop suddenly acquired a substantial overhead it didn't have before! All you are saying is that they didn't bother reading the docs and learning how to write a *PYTHON* for loop. Failing that don't bother using range, just directly convert your (say) C loop into Python. I really don't see any issue here at all. OK, so it's the programmer's fault if as fundamental a concept as a for-loop ranging over integers is implemented inefficiently. He has to transform it into high-level terms, or has to reconstruct it somehow using a while-loop and an incrementing loop index. Now I understand (why Python has been beset for so long with performance problems, if that is a typical attitude!). BTW, why did they introduce the 'xrange' thing; for fun? Or did someone realise there might have been an issue after all? (I tried an empty loop counting to 1 billion in Python 2.x, using 'for i in range'. It ran out of memory. Counting to 100 million instead, it worked, but still used a massive 1.5GB RAM while doing so (and took 6 seconds to count to 100M, not too bad for Python) Outside Python, it might typically take a few seconds to count to 1 billion, and would use virtually no memory. Why would anyone want to loop over all those numbers instead of iterating over actual data? For any number of reasons. Counting how many happy numbers are in that range for one!) I give up. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
BartC b...@freeuk.com: (I tried an empty loop counting to 1 billion in Python 2.x, using 'for i in range'. It ran out of memory. Counting to 100 million instead, it worked, but still used a massive 1.5GB RAM while doing so (and took 6 seconds to count to 100M, not too bad for Python) Outside Python, it might typically take a few seconds to count to 1 billion, and would use virtually no memory. Why would anyone want to loop over all those numbers instead of iterating over actual data? For any number of reasons. Counting how many happy numbers are in that range for one!) You're doing it wrong. Here's the Pythonic way to iterate over a billion numbers: def loop(i): if i 10: do_stuff(i) keep_iterating(i + 1) loop(0) Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 02/05/2015 20:15, Mark Lawrence wrote: On 02/05/2015 19:34, BartC wrote: OK, so it's the programmer's fault if as fundamental a concept as a for-loop ranging over integers is implemented inefficiently. He has to transform it into high-level terms, or has to reconstruct it somehow using a while-loop and an incrementing loop index. I give up. So do I, I think, if no-one is willing to admit that the original way of implementing range() was a glaring mistake. -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 5/2/2015 5:40 PM, Jon Ribbens wrote: For information, it looks like xrange() was added on 26 Oct 1993, which pre-dates Python 1.0. 1.0 was released Feb 1991. 3.2 exactly 20 years later. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
I give up. So do I, I think, if no-one is willing to admit that the original way of implementing range() was a glaring mistake. -- Bartc It doesn't matter for small ranges. Anyway it was fixed. -- https://mail.python.org/mailman/listinfo/python-list -- Joel Goldstick http://joelgoldstick.com -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sat, May 2, 2015 at 1:51 PM, BartC b...@freeuk.com wrote: On 02/05/2015 20:15, Mark Lawrence wrote: On 02/05/2015 19:34, BartC wrote: OK, so it's the programmer's fault if as fundamental a concept as a for-loop ranging over integers is implemented inefficiently. He has to transform it into high-level terms, or has to reconstruct it somehow using a while-loop and an incrementing loop index. I give up. So do I, I think, if no-one is willing to admit that the original way of implementing range() was a glaring mistake. range() was and is a *convenience* function. In the real world, the vast majority of for loops are over arrays or other containers, not integers, and those that aren't are usually very small. In non-toy code, using a for loop to count to a billion is highly unusual. So yeah, for a programmer porting code to Python who needed to loop over an array, the correct approach would be to actually loop over the *array* in place of the indices of the array. I don't know why you make this out to be such a big deal; it's a simple conversion. Would it have been better if range() had been implemented as xrange() from the beginning? Sure, that would have been great. Except for one small detail: the iterator protocol didn't exist back then. That wasn't introduced until PEP 234 in Python 2.1, which means that the xrange() function wasn't even *possible* before then. I don't think anybody would claim that Python was perfect when it was first introduced (nor is it perfect now). Like all other software, it has improved over time as a result of iterative refinement. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 02/05/2015 22:40, Jon Ribbens wrote: On 2015-05-02, BartC b...@freeuk.com wrote: So do I, I think, if no-one is willing to admit that the original way of implementing range() was a glaring mistake. I think the issue is that nobody else here thinks the original way of iterating was to use range(), for anything other than extremely small ranges. What /is/ the way to iterate then? I don't have much Python code lying around, but the first couple of files I looked at (not mine), one had this: for i in range(7,-1,-1): for j in range(8): another had: for l in range(1,17): for i in range(1, self.bits[l] + 1): for i in range(256): and so on. Plenty of examples. This was probably converted to Python from elsewhere, but why shouldn't it be able to run code trivially converted from somewhere else, without being unnecessarily slow? (What would be the 'Pythonic' way of writing a Jpeg decoder, as this was, anyway?) I don't think the small size of the range is a mitigating factor, other than it won't run out of memory. Even a small loop can be executed millions of times. For information, it looks like xrange() was added on 26 Oct 1993, which pre-dates Python 1.0. OK, so it's just an irritation then, as a workaround has been available for a long time. (For example, if you use xrange, it won't work on 3.x. If you use range, then it might be inefficient on 2.x.) -- Bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 5/2/2015 11:26 AM, BartC wrote:
When I first looked at Python 20 or so years ago this seemed to be the
standard way of writing a for-loop:
for i in range(N):
As Mark said, the actual syntax was originally
for item in sequence: ...
where sequence technically meant an object with a .__getitem__(x) method
that either returned an item or raised IndexError. In 2.2, 'sequence'
was replaced by the more general concept 'iterable'.
In python, 'for' is and has always been an abbreviation of 'for each' or
'foreach', as used in other languages. C for loops are not 'for each'
loops but are initialized while loops, and are so explained in KR's
book, page 56.
'''
for (initializer; condition; update;) body
is equivalent to
initializer
while (condition) {
body
update;
}
...
Whether to use while or for is largely a matter of taste.
'''
(I substituted 'initializer', 'condition', 'update' for 'expr1',
'expr2', and 'expr3' as more meaningful.) The proper translation of a C
for loop to Python is the translation of the equivalent C while loop.
Translating to a Python 'for range' loop is a bit of a mis-translation.
One should not be surprised if non-equivalent code behaves differently.
I remember being completely astonished at the time that 'range' actually
created a list of values from 0 to N-1.
The purpose and *documented* behavior of range was to produce an
arithmetic sequence. Map, filter, dict.keys, dict.values, dict.items,
many others functions also returned lists. In Python3, the ones I named
above and some others now return 'less spacious' iterables.
Python was already known to be slow yet it deliberately crippled itself
by using just about the slowest method imaginable of executing a simple
iterative loop? By first creating a list of a million objects!
No, you were also free to do the sensible thing when you did not need a
concrete arithmetic sequence (list).
n = 0
while n 100:
pass
n = n+1
And sometimes you weren't even interested in the values but just wanted
to execute something N times so it was a wasted effort.
# even simpler
n = 100
while n:
pass
n -= 1
Using for ... range() is a convenient abbreviation of either pattern.
It works for small values of n, it did not work for large values. When
Python 1.0 was released in Feb 1992, memory for many system was limited
to a megabyte or less. At that time, range(100) was an impossible
object, so no one aware of memory constraints would try to create it.
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 5/2/2015 5:31 PM, Ian Kelly wrote:
Would it have been better if range() had been implemented as xrange()
from the beginning? Sure, that would have been great. Except for one
small detail: the iterator protocol didn't exist back then.
For loops originally used the getitem iterator protocol. xrange objects
have a __getitem__ method, but not __iter__ or __next__. As Mark
pointed out, they were introduced in 1993.
Getitem iterators still work. If iter(ob) is passed object ob without
.__iter__, iter looks for ob.__getitem__ and if present, returns
iterator(ob), where iterator is an internal protocol adapter class. Its
__next__ method calls ob.__getitem__ and converts IndexError to
StopIteration.
In 3.4.3:
class xrange:
def __init__(self, start, stop, step):
if step 1:
raise ValueError('incomplete xrange require positive step')
self.val = start
self.stop = stop
self.step = step
def __getitem__(self, dummy):
val = self.val
self.val = val + self.step
if val self.stop:
return val
else:
raise IndexError('')
it = iter(xrange(1, 10, 3))
print(type(it))
for i in it:
print(i)
#
class 'iterator'
1
4
7
--
Terry Jan Reedy
--
https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sat, May 2, 2015 at 3:28 PM, Tony the Tiger tony@tiger.invalid wrote: On Fri, 01 May 2015 14:42:04 +1000, Steven D'Aprano wrote: use l as a variable name, as it looks too much like 1 If you use a better font, they are very different. Besides, a variable name cannot start with a digit (nor can it be a single digit), so it's a given that it's an 'l'. Of course it can be a single digit. You're assuming that the person reading the code already somehow knows that this is a variable name and not an integer literal. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 2015-05-02, BartC b...@freeuk.com wrote: So do I, I think, if no-one is willing to admit that the original way of implementing range() was a glaring mistake. I think the issue is that nobody else here thinks the original way of iterating was to use range(), for anything other than extremely small ranges. For information, it looks like xrange() was added on 26 Oct 1993, which pre-dates Python 1.0. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 02/05/2015 22:40, Jon Ribbens wrote: On 2015-05-02, BartC b...@freeuk.com wrote: So do I, I think, if no-one is willing to admit that the original way of implementing range() was a glaring mistake. I think the issue is that nobody else here thinks the original way of iterating was to use range(), for anything other than extremely small ranges. For information, it looks like xrange() was added on 26 Oct 1993, which pre-dates Python 1.0. No chance, or so I thought, but to save others from looking it up https://hg.python.org/cpython/annotate/89e1e5d9ccbf/Python/bltinmodule.c -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Sat, May 2, 2015 at 5:51 PM, Terry Reedy tjre...@udel.edu wrote: On 5/2/2015 5:31 PM, Ian Kelly wrote: Would it have been better if range() had been implemented as xrange() from the beginning? Sure, that would have been great. Except for one small detail: the iterator protocol didn't exist back then. For loops originally used the getitem iterator protocol. xrange objects have a __getitem__ method, but not __iter__ or __next__. As Mark pointed out, they were introduced in 1993. I'm aware of getitem iterators; just didn't realize that xrange used it or was that old. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 2015-05-02, Terry Reedy tjre...@udel.edu wrote: On 5/2/2015 5:40 PM, Jon Ribbens wrote: For information, it looks like xrange() was added on 26 Oct 1993, which pre-dates Python 1.0. 1.0 was released Feb 1991. 3.2 exactly 20 years later. No, you are thinking of 0.9, which was indeed February 1991. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 05/02/2015 04:33 PM, BartC wrote: OK, so it's just an irritation then, as a workaround has been available for a long time. (For example, if you use xrange, it won't work on 3.x. If you use range, then it might be inefficient on 2.x.) In both Python 2.7 and 3.3+, you can use the 3rd-party six module to help with forward compatibility: from six.moves import xrange In Python 2.7, it's just the normal xrange (importing is a no op basically), but in Python 3 it points to range. Kind of wish parts of six were in the Python standard library. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 03/05/2015 03:07, Michael Torrie wrote: On 05/02/2015 04:33 PM, BartC wrote: OK, so it's just an irritation then, as a workaround has been available for a long time. (For example, if you use xrange, it won't work on 3.x. If you use range, then it might be inefficient on 2.x.) In both Python 2.7 and 3.3+, you can use the 3rd-party six module to help with forward compatibility: from six.moves import xrange In Python 2.7, it's just the normal xrange (importing is a no op basically), but in Python 3 it points to range. Kind of wish parts of six were in the Python standard library. Links to other useful third party modules are given here https://docs.python.org/3/howto/pyporting.html I doubt that six will ever make the standard library as 2.7 only has another five years in official support. By that time I suppose we'll to going through the porting pain all over again with the transition from Python 3 to Python 4. Alright, alright, only joking :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 5/2/2015 8:01 PM, Jon Ribbens wrote: On 2015-05-02, Terry Reedy tjre...@udel.edu wrote: On 5/2/2015 5:40 PM, Jon Ribbens wrote: For information, it looks like xrange() was added on 26 Oct 1993, which pre-dates Python 1.0. 1.0 was released Feb 1991. 3.2 exactly 20 years later. No, you are thinking of 0.9, which was indeed February 1991. 'scuse me, I meant 1992 -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 2015-05-03, Terry Reedy tjre...@udel.edu wrote: On 5/2/2015 8:01 PM, Jon Ribbens wrote: On 2015-05-02, Terry Reedy tjre...@udel.edu wrote: On 5/2/2015 5:40 PM, Jon Ribbens wrote: For information, it looks like xrange() was added on 26 Oct 1993, which pre-dates Python 1.0. 1.0 was released Feb 1991. 3.2 exactly 20 years later. No, you are thinking of 0.9, which was indeed February 1991. 'scuse me, I meant 1992 No, it was January 1994. http://python-history.blogspot.co.uk/2009/01/brief-timeline-of-python.html -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 2015-05-02, BartC b...@freeuk.com wrote: On 02/05/2015 22:40, Jon Ribbens wrote: I think the issue is that nobody else here thinks the original way of iterating was to use range(), for anything other than extremely small ranges. What /is/ the way to iterate then? Other people have already explained that to you several times. If you want a literal 'for (i = 0; i x; i++)' loop then it's 'for i in xrange(x)' (from Python 1.0 to Python 2.7), but most of the time you simply wouldn't be writing code that works that way. I don't have much Python code lying around, but the first couple of files I looked at (not mine), one had this: for i in range(7,-1,-1): for j in range(8): another had: for l in range(1,17): for i in range(1, self.bits[l] + 1): for i in range(256): and so on. Plenty of examples. ... of extremely small ranges, just as I said. I don't think the small size of the range is a mitigating factor, other than it won't run out of memory. Even a small loop can be executed millions of times. So what? The overhead of creating a list of 17 integers is trivial compared to executing the rest of the code. As the man said: premature optimization is the root of all evil. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 5/1/2015 1:04 AM, Cecil Westerhof wrote: By the way: I also see python3.4 and python3.4m. Any idea where the m stands for? I never heard of that in 18 years of Python, and thought it must be an error, but putting 'python3.4b' into google search return this. https://stackoverflow.com/questions/16675865/difference-between-python3-and-python3m-executibles See first answer. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Friday 1 May 2015 15:25 CEST schreef Michael Torrie: On 04/30/2015 10:19 PM, Cecil Westerhof wrote: I must also confess to being highly impressed, it's a breath of fresh air having an apprentice Pythonista who is looking at doing things the Pythonic way :) When in Rome, do as the Romans do. Besides: there probably is a reason for the Pythonic way. You maybe surprised, then, by how many people that stop by this list wanting to program Java, C# or C++ in Python and get frustrated and upset when the list tries to steer them in a Pythonic direction. Most of them leave upset and never return to the list or Python. I am not really surprised. I know that this is how it often works. :-( But I like to be different. :-D -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Friday 1 May 2015 07:20 CEST schreef Steven D'Aprano: Some programming language virtual machines limit how much memory they will use. The CPython VM isn't one of those, although I understand that both Jython and IronPython are. (I may be wrong -- Jython runs in the JVM, so Jython is. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Cecil Westerhof wrote: By the way: I also see python3.4 and python3.4m. Any idea where the m stands for? I googled for “python3.4m” and found as second result http://stackoverflow.com/questions/16675865/difference-between-python3-and-python3m-executibles In a nutshell: python3.4m was built with configure option “--with-pymalloc” which causes the resulting implementation to use “Pymalloc, a specialized object allocator written by Vladimir Marangozov, […] a feature added to Python 2.1. Pymalloc is intended to be faster than the system malloc() and to have less memory overhead for allocation patterns typical of Python programs. The allocator uses C's malloc() function to get large pools of memory and then fulfills smaller memory requests from these pools.” See also: https://github.com/Homebrew/homebrew/issues/32402 (first result) -- PointedEars Twitter: @PointedEars2 Please do not cc me. / Bitte keine Kopien per E-Mail. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Friday 1 May 2015 13:15 CEST schreef Thomas Lahn: Cecil Westerhof wrote: By the way: I also see python3.4 and python3.4m. Any idea where the m stands for? I googled for “python3.4m” and found as second result Eh, I could/should have done that myself. :-( Nice that you do not burn me. :-D http://stackoverflow.com/questions/16675865/difference-between-python3-and-python3m-executibles In a nutshell: python3.4m was built with configure option “--with-pymalloc” which causes the resulting implementation to use “Pymalloc, a specialized object allocator written by Vladimir Marangozov, […] a feature added to Python 2.1. Pymalloc is intended to be faster than the system malloc() and to have less memory overhead for allocation patterns typical of Python programs. The allocator uses C's malloc() function to get large pools of memory and then fulfills smaller memory requests from these pools.” See also: https://github.com/Homebrew/homebrew/issues/32402 (first result) Something to look in later. Looks interesting, but have a ‘few’ things that are more important. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 04/30/2015 10:19 PM, Cecil Westerhof wrote: I must also confess to being highly impressed, it's a breath of fresh air having an apprentice Pythonista who is looking at doing things the Pythonic way :) When in Rome, do as the Romans do. Besides: there probably is a reason for the Pythonic way. You maybe surprised, then, by how many people that stop by this list wanting to program Java, C# or C++ in Python and get frustrated and upset when the list tries to steer them in a Pythonic direction. Most of them leave upset and never return to the list or Python. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Jon Ribbens jon+use...@unequivocal.co.uk writes: On 2015-04-30, Cecil Westerhof ce...@decebal.nl wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. […] Is there a way to circumvent Python claiming all the memory? You seem to be asking for a way to stop a program doing exactly what it's written to do. I don't know what kind of answer you expect. It's your operating system's job to handle processes. Indeed. In this case, the program is written to gobble up memory, and the operating system kills it. To “circumvent” that behaviour surely reveals the problem: that the operating system isn't handling processes very well. By the way: this is CPython 2.7.8. If you use xrange() instead of range() then you will get an iterator which will return each of the numbers in turn without any need to create an enormous list of all of them. If you use Python 3 instead of the obsolescent Python 2, the ‘range’ callable has this sensible behaviour by default. -- \ “Corporation, n. An ingenious device for obtaining individual | `\ profit without individual responsibility.” —Ambrose Bierce, | _o__) _The Devil's Dictionary_, 1906 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 2015-04-30, Cecil Westerhof ce...@decebal.nl wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? I presume don't do that has already occured to you? You can always use ulimit to limit the memory allowed for the process running Python. -- Grant Edwards grant.b.edwardsYow! Should I get locked at in the PRINCICAL'S gmail.comOFFICE today -- or have a VASECTOMY?? -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 4/30/2015 12:06 PM, Cecil Westerhof wrote: If I execute: l = range(int(1E9) you get a SyntaxError The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? For this specific case, use xrange or 3.x range By the way: this is CPython 2.7.8. -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Cecil Westerhof wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? By the way: this is CPython 2.7.8. On what OS? If I try something similar on Win-8.1 and CPython 2.7.5 (32-bit): python -c for i in range(int(1E9)): pass Traceback (most recent call last): File string, line 1, in module MemoryError --gv -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 04/30/2015 09:06 AM, Cecil Westerhof wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? By the way: this is CPython 2.7.8. Well, that could be considered the problem. In Python3, the range function returns a range object which takes up almost no resources, while in Python2 it produces a list. Both can be iterated over, so they produce the same result in the most common use case (i.e., iteration), but the Python3 version generates the elements only as needed. If you really *wanted* the list (but WHY?) in Python3, do list(range(...)), but then you get what you deserve. :-) Python3: l = range(int(1E9)) l range(0, 10) -- Dr. Gary Herron Department of Computer Science DigiPen Institute of Technology (425) 895-4418 -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Thursday 30 Apr 2015 18:33 CEST schreef Grant Edwards: On 2015-04-30, Cecil Westerhof ce...@decebal.nl wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? I presume don't do that has already occured to you? Well, it is just playing with the possibilities/limits of Python. Better getting the problem now, then when I can not afford to lose time. ;-) You can always use ulimit to limit the memory allowed for the process running Python. That works, yes. Now I get a MemoryError and the other processes are left alone. Now determining what are the best values. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Cecil Westerhof ce...@decebal.nl writes: That works, yes. Now I get a MemoryError and the other processes are left alone. Now determining what are the best values. I would strongly recommend that “best values” includes “run Python version = 3”. One of the many problems you avoid by leaving Python 2 behind is that many functions which used to return entire collections, now return lazy evaluators (such as generators or views) which will not consume memory the way you're describing. Please try to learn Python using only the currently-developed Python 3. -- \ “Listen: we are here on Earth to fart around. Don't let anybody | `\ tell you otherwise.” —_Timequake_, Kurt Vonnegut | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Thursday 30 Apr 2015 18:55 CEST schreef Jon Ribbens: On 2015-04-30, Cecil Westerhof ce...@decebal.nl wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? By the way: this is CPython 2.7.8. It's your operating system's job to handle processes. If you use xrange() instead of range() then you will get an iterator which will return each of the numbers in turn without any need to create an enormous list of all of them. I did it on purpose. Wanted to now the limits. Just need to use ulimit to get a MemoryError. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Thu, 30 Apr 2015 10:05:44 -0700, Gary Herron wrote: On 04/30/2015 09:06 AM, Cecil Westerhof wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? By the way: this is CPython 2.7.8. Well, that could be considered the problem. In Python3, the range function returns a range object which takes up almost no resources, while in Python2 it produces a list. Both can be iterated over, so they produce the same result in the most common use case (i.e., iteration), but the Python3 version generates the elements only as needed. If you really *wanted* the list (but WHY?) in Python3, do list(range(...)), but then you get what you deserve. :-) Python3: l = range(int(1E9)) l range(0, 10) This also leads to a unrelated question, Cecil. Given that you really are just starting to get your Python feet under you, why are you using Python2? Python3 is the standard now, Python2 is really just given legacy support. I'd understand if you were trying to maintain an old codebase with lots of legacy code that was having problematic migrations, but with the opportunity to start fresh? Start fresh. You'll be happier for it. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 2015-04-30, Cecil Westerhof ce...@decebal.nl wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? By the way: this is CPython 2.7.8. It's your operating system's job to handle processes. If you use xrange() instead of range() then you will get an iterator which will return each of the numbers in turn without any need to create an enormous list of all of them. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Thursday 30 Apr 2015 19:12 CEST schreef Rob Gaddi: This also leads to a unrelated question, Cecil. Given that you really are just starting to get your Python feet under you, why are you using Python2? Python3 is the standard now, Python2 is really just given legacy support. I'd understand if you were trying to maintain an old codebase with lots of legacy code that was having problematic migrations, but with the opportunity to start fresh? Start fresh. You'll be happier for it. I try to write code that works with both. Wen looking around in the Netherlands there are more job opportunities with Python 2 as with Python 3. So at the moment I find Python 2 more important, without forgetting Python 3. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 04/30/2015 02:48 PM, alister wrote: On Thu, 30 Apr 2015 20:23:31 +0200, Gisle Vanem wrote: Cecil Westerhof wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? By the way: this is CPython 2.7.8. On what OS? If I try something similar on Win-8.1 and CPython 2.7.5 (32-bit): python -c for i in range(int(1E9)): pass Traceback (most recent call last): File string, line 1, in module MemoryError --gv also MemoryError on Fedora 21 32 bit That's presumably because you end up running out of address space before you run out of swap space. On a 64 bit system the reverse will be true, unless you have a really *really* large swap file ulimit is your friend if you've got a program that wants to gobble up all of swap space. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 30/04/2015 19:50, Cecil Westerhof wrote: Op Thursday 30 Apr 2015 19:12 CEST schreef Rob Gaddi: This also leads to a unrelated question, Cecil. Given that you really are just starting to get your Python feet under you, why are you using Python2? Python3 is the standard now, Python2 is really just given legacy support. I'd understand if you were trying to maintain an old codebase with lots of legacy code that was having problematic migrations, but with the opportunity to start fresh? Start fresh. You'll be happier for it. I try to write code that works with both. Wen looking around in the Netherlands there are more job opportunities with Python 2 as with Python 3. So at the moment I find Python 2 more important, without forgetting Python 3. You might find this useful then in you haven't already seen it https://docs.python.org/3/howto/pyporting.html I must also confess to being highly impressed, it's a breath of fresh air having an apprentice Pythonista who is looking at doing things the Pythonic way :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Thursday 30 Apr 2015 20:59 CEST schreef Dave Angel: On 04/30/2015 02:48 PM, alister wrote: On Thu, 30 Apr 2015 20:23:31 +0200, Gisle Vanem wrote: Cecil Westerhof wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? By the way: this is CPython 2.7.8. On what OS? If I try something similar on Win-8.1 and CPython 2.7.5 (32-bit): python -c for i in range(int(1E9)): pass Traceback (most recent call last): File string, line 1, in module MemoryError --gv also MemoryError on Fedora 21 32 bit That's presumably because you end up running out of address space before you run out of swap space. On a 64 bit system the reverse will be true, unless you have a really *really* large swap file ulimit is your friend if you've got a program that wants to gobble up all of swap space. Yes, my system is openSUSE 64 bit. I really should look into ulimit. The default is: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63768 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 63768 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited That should be fine-tuned. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Grant Edwards schreef op 2015-04-30 18:33: On 2015-04-30, Cecil Westerhof ce...@decebal.nl wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? I presume don't do that has already occured to you? A few weeks ago, I had a bug in a Python script which caused it to consume all memory, thrashed my computer; the system become unresponsive and it was very difficult to kill the process and get everything back in working order. Then I thought (foolishly) that the bug was fixed, so I ran the script again, with the same disastrous results. And then again. Stupid, I know. A way to limit memory usage would have been nice: the script would have been killed before it could grind the whole system to a halt. You can always use ulimit to limit the memory allowed for the process running Python. Sadly in my case the OS is Windows, and as far as I know it doesn't have a ulimit equivalent for limiting memory usage. I guess I could have used 32-bit Python instead of 64-bit Python to limit available memory. -- The saddest aspect of life right now is that science gathers knowledge faster than society gathers wisdom. -- Isaac Asimov Roel Schroeven -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Thu, 30 Apr 2015 20:23:31 +0200, Gisle Vanem wrote: Cecil Westerhof wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. The problem is that after this my swap is completely used, because other processes have swapped to it. This make those programs more slowly. Is there a way to circumvent Python claiming all the memory? By the way: this is CPython 2.7.8. On what OS? If I try something similar on Win-8.1 and CPython 2.7.5 (32-bit): python -c for i in range(int(1E9)): pass Traceback (most recent call last): File string, line 1, in module MemoryError --gv also MemoryError on Fedora 21 32 bit -- I am a traffic light, and Alan Ginzberg kidnapped my laundry in 1927! -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Thursday 30 Apr 2015 19:41 CEST schreef Ben Finney: Cecil Westerhof ce...@decebal.nl writes: That works, yes. Now I get a MemoryError and the other processes are left alone. Now determining what are the best values. I would strongly recommend that “best values” includes “run Python version = 3”. One of the many problems you avoid by leaving Python 2 behind is that many functions which used to return entire collections, now return lazy evaluators (such as generators or views) which will not consume memory the way you're describing. Please try to learn Python using only the currently-developed Python 3. The problem with that is that in my neighbourhood 2 is still mostly exclusively used. I want to learn also 3, but 2 has more priority. The modules I publish on GitHub work with both. :-) -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Thursday 30 Apr 2015 22:53 CEST schreef Mark Lawrence: On 30/04/2015 19:50, Cecil Westerhof wrote: Op Thursday 30 Apr 2015 19:12 CEST schreef Rob Gaddi: This also leads to a unrelated question, Cecil. Given that you really are just starting to get your Python feet under you, why are you using Python2? Python3 is the standard now, Python2 is really just given legacy support. I'd understand if you were trying to maintain an old codebase with lots of legacy code that was having problematic migrations, but with the opportunity to start fresh? Start fresh. You'll be happier for it. I try to write code that works with both. Wen looking around in the Netherlands there are more job opportunities with Python 2 as with Python 3. So at the moment I find Python 2 more important, without forgetting Python 3. You might find this useful then in you haven't already seen it https://docs.python.org/3/howto/pyporting.html I must also confess to being highly impressed, it's a breath of fresh air having an apprentice Pythonista who is looking at doing things the Pythonic way :) When in Rome, do as the Romans do. Besides: there probably is a reason for the Pythonic way. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Fri, 1 May 2015 02:06 am, Cecil Westerhof wrote: If I execute: l = range(int(1E9) Others have already answered your questions about memory. Let me answer the question you didn't ask about style :-) Don't use l as a variable name, as it looks too much like 1. Better to use L, or even better, a meaningful name. Rather than convert a float 1E9 to an int at runtime, better to use an int: range(10**9) With recent versions of CPython, the compiler has a keyhole optimiser which does constant folding. For implementations of Python which don't do constant folding, 10**9 is likely to be faster than int(1E9) -- but even if it isn't, does it matter? It will be very fast one way of the other. The important thing is that 10**9 expresses the intention to use an integer in a more direct fashion than using 1E9. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Friday 1 May 2015 01:12 CEST schreef Ben Finney: Chris Angelico ros...@gmail.com writes: Very easily and simply: Python 3 and Python 2 will always install separately, and the only possible conflicts are over the python command in PATH and which program is associated with .py files. Calling ‘python’ is now ambiguous, and with Python 2 slipping inexorably into the past, increasingly the ‘python’ command is the wrong choice for code that we want to survive in the future. I am seeing a growing call, with which I agree, to recommend explicitly calling ‘python2’ or ‘python3’ as commands. That includes when we type it for direct invocation, or when we set it as the command for automatic execution of a program (e.g. in the “shebang” line of a program). Use the command ‘python2’ or ‘python3’ to be explicit about which Python version you intend to run. Good tip. I used python and python3, but there is also a python2. I learn myself to use that instead of python. By the way: I also see python3.4 and python3.4m. Any idea where the m stands for? -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Friday 1 May 2015 06:42 CEST schreef Steven D'Aprano: On Fri, 1 May 2015 02:06 am, Cecil Westerhof wrote: If I execute: l = range(int(1E9) Others have already answered your questions about memory. Let me answer the question you didn't ask about style :-) That can be very useful. Don't use l as a variable name, as it looks too much like 1. Better to use L, or even better, a meaningful name. Rather than convert a float 1E9 to an int at runtime, better to use an int: range(10**9) With recent versions of CPython, the compiler has a keyhole optimiser which does constant folding. For implementations of Python which don't do constant folding, 10**9 is likely to be faster than int(1E9) -- but even if it isn't, does it matter? It will be very fast one way of the other. The important thing is that 10**9 expresses the intention to use an integer in a more direct fashion than using 1E9. It was just a short hack to show the problem. In real code I would never use l. But I'll remember to use 10**9 instead of 1E9. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 01/05/2015 05:19, Cecil Westerhof wrote: Op Thursday 30 Apr 2015 22:53 CEST schreef Mark Lawrence: On 30/04/2015 19:50, Cecil Westerhof wrote: Op Thursday 30 Apr 2015 19:12 CEST schreef Rob Gaddi: This also leads to a unrelated question, Cecil. Given that you really are just starting to get your Python feet under you, why are you using Python2? Python3 is the standard now, Python2 is really just given legacy support. I'd understand if you were trying to maintain an old codebase with lots of legacy code that was having problematic migrations, but with the opportunity to start fresh? Start fresh. You'll be happier for it. I try to write code that works with both. Wen looking around in the Netherlands there are more job opportunities with Python 2 as with Python 3. So at the moment I find Python 2 more important, without forgetting Python 3. You might find this useful then in you haven't already seen it https://docs.python.org/3/howto/pyporting.html I must also confess to being highly impressed, it's a breath of fresh air having an apprentice Pythonista who is looking at doing things the Pythonic way :) When in Rome, do as the Romans do. Besides: there probably is a reason for the Pythonic way. No probably about it, the threat of the Comfy Chair... -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Fri, 1 May 2015 03:20 am, Ben Finney wrote: Jon Ribbens jon+use...@unequivocal.co.uk writes: On 2015-04-30, Cecil Westerhof ce...@decebal.nl wrote: If I execute: l = range(int(1E9) The python process gobbles up all the memory and is killed. […] Is there a way to circumvent Python claiming all the memory? You seem to be asking for a way to stop a program doing exactly what it's written to do. I don't know what kind of answer you expect. How about a way to stop a program from doing what you didn't intend it to do? As we all know quite well, computers are quite stupid, they do what they are told, not what we want. It's not just to protect against bugs and mistakes. It's also to protect one user from another user, or a single user from hostile code. Just because Cecil tells the computer use up 12GB of RAM in one great big list doesn't mean that the other users of that computer have to acquiesce to that request. It's your operating system's job to handle processes. Indeed. In this case, the program is written to gobble up memory, and the operating system kills it. To “circumvent” that behaviour surely reveals the problem: that the operating system isn't handling processes very well. Indeed indeed :-) Some programming language virtual machines limit how much memory they will use. The CPython VM isn't one of those, although I understand that both Jython and IronPython are. (I may be wrong -- corrections are welcome.) That means you have to use the OS to limit how much memory CPython will use, if you can. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Op Thursday 30 Apr 2015 23:41 CEST schreef Tim Chase: On 2015-04-30 22:18, Cecil Westerhof wrote: Op Thursday 30 Apr 2015 20:59 CEST schreef Dave Angel: ulimit is your friend if you've got a program that wants to gobble up all of swap space. Yes, my system is openSUSE 64 bit. I really should look into ulimit. The default is: [snip] max memory size (kbytes, -m) unlimited Note that AFAIK, ulimit -m doesn't work on Linux http://unix.stackexchange.com/questions/129587/does-ulimit-m-not-work-on-modern-linux Based on some quick testing[1], it doesn't appear to work on OpenBSD or FreeBSD either. Yes, as I already found out you need to use -v. But it would be nice that there was an indication that -m is obsolete. Depending on the available memory you could use something like: ulimit -v $((4 * 1024 * 1024)) I can then do: l = range(int(1E8)) but: l = range(int(1E9)) gives MemoryError. And if I can also do (from another thread): happy_number_list(int(1E8)) if I did not something memory consuming. And if a (Python) process should use significantly less, it could be a good idea to tweak the ulimit call. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Fri, May 1, 2015 at 7:23 AM, ElChino elch...@cnn.cn wrote: Mark Lawrence wrote: You might find this useful then in you haven't already seen it https://docs.python.org/3/howto/pyporting.html The main reason I haven't switched to Python3 (from 2.7.4/MSVC), is fear of a major breakage. How can I be certain that even if I install to different directories, the Python2 and Python3 won't step on each other toes. And what about all the .pyd I have around? Do they need an update? I'm barely familiar with virtualenv BTW; not sure another confusing factor in the equation would be a good idea. Or lessen my fear. Very easily and simply: Python 3 and Python 2 will always install separately, and the only possible conflicts are over the python command in PATH and which program is associated with .py files. You can fix both of them by installing a recent version of Python and using the py.exe launcher: https://www.python.org/dev/peps/pep-0397/ https://docs.python.org/3/using/windows.html#python-launcher-for-windows As to your .pyd files... they'll continue to work with Python 2.7 (and, incidentally, I would suggest upgrading to 2.7.9 or 2.7.10), but you'll need to update them for Python 3. If you got them from a third party, chances are good that there'll be a Py3 build available; if you built them from source, you could try just recompiling the latest source. But if you wrote them yourself, you may well have to make some changes to get them to compile against Python 3. If you have issues, start a new thread; people will be happy to help. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Mark Lawrence wrote: You might find this useful then in you haven't already seen it https://docs.python.org/3/howto/pyporting.html The main reason I haven't switched to Python3 (from 2.7.4/MSVC), is fear of a major breakage. How can I be certain that even if I install to different directories, the Python2 and Python3 won't step on each other toes. And what about all the .pyd I have around? Do they need an update? I'm barely familiar with virtualenv BTW; not sure another confusing factor in the equation would be a good idea. Or lessen my fear. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Chris Angelico wrote: Very easily and simply: Python 3 and Python 2 will always install separately, and the only possible conflicts are over the python command in PATH and which program is associated with .py files. You can fix both of them by installing a recent version of Python and using the py.exe launcher: Thanks for the info. I'll give it a try. -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
Chris Angelico ros...@gmail.com writes: Very easily and simply: Python 3 and Python 2 will always install separately, and the only possible conflicts are over the python command in PATH and which program is associated with .py files. Calling ‘python’ is now ambiguous, and with Python 2 slipping inexorably into the past, increasingly the ‘python’ command is the wrong choice for code that we want to survive in the future. I am seeing a growing call, with which I agree, to recommend explicitly calling ‘python2’ or ‘python3’ as commands. That includes when we type it for direct invocation, or when we set it as the command for automatic execution of a program (e.g. in the “shebang” line of a program). Use the command ‘python2’ or ‘python3’ to be explicit about which Python version you intend to run. -- \ “I don't want to live peacefully with difficult realities, and | `\ I see no virtue in savoring excuses for avoiding a search for | _o__)real answers.” —Paul Z. Myers, 2009-09-12 | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On 2015-04-30 22:18, Cecil Westerhof wrote: Op Thursday 30 Apr 2015 20:59 CEST schreef Dave Angel: ulimit is your friend if you've got a program that wants to gobble up all of swap space. Yes, my system is openSUSE 64 bit. I really should look into ulimit. The default is: [snip] max memory size (kbytes, -m) unlimited Note that AFAIK, ulimit -m doesn't work on Linux http://unix.stackexchange.com/questions/129587/does-ulimit-m-not-work-on-modern-linux Based on some quick testing[1], it doesn't appear to work on OpenBSD or FreeBSD either. -tkc [1] My test was to perform the following: 1) seq 20 data.txt # to generate some random date 2) ed data.txt 3) find the PID of the ed process from another window using ps ax | grep ed 4) make note of the current memory used for that PID using top -p $PID 5) issue ,t$ in ed until the memory-used jumps up in the top window (this copies the entire file to the bottom of the file, doubling the size each time) 6) quit ed without saving (Q) 7) issue ulimit -m, specifying a threshold between the before-the-memory-jump and after-the-memory-jump values 8) repeat steps 2-5 until the memory jumps up in top. Notice that it's possible to do this multiple times and exceed the ulimit -m threshold, something the ulimit should prevent. . -- https://mail.python.org/mailman/listinfo/python-list
Re: l = range(int(1E9))
On Fri, May 1, 2015 at 9:12 AM, Ben Finney ben+pyt...@benfinney.id.au wrote: Chris Angelico ros...@gmail.com writes: Very easily and simply: Python 3 and Python 2 will always install separately, and the only possible conflicts are over the python command in PATH and which program is associated with .py files. Calling ‘python’ is now ambiguous, and with Python 2 slipping inexorably into the past, increasingly the ‘python’ command is the wrong choice for code that we want to survive in the future. I am seeing a growing call, with which I agree, to recommend explicitly calling ‘python2’ or ‘python3’ as commands. Agreed. And on Windows, the py -3 or py -2 options are also available. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
