[Python-Dev] Memory use of PEP 563 vs PEP 649

Hi, There has been some discussion on this mailing list about the memory use of PEP 563 vs PEP 649. It doesn't matter. The memory use of either is small, and any measurements are merely measuring artifacts of the current implementations, and the current on-disk representation of code

[Python-Dev] Memory address vs serial number in reprs

I have problem with the location of hexadecimal memory address in custom reprs. vs The long hexadecimal number makes the repr longer and distracts attention from other useful information. We could get rid of it, but it is useful if we want to distinguish objects of the same type.

Re: [Python-Dev] Memory bitmaps for the Python cyclic garbage collector

[Tim] >> In that case, it's because Python >> _does_ mutate the objects' refcount members under the covers, and so >> the OS ends up making fresh copies of the memory anyway. [Greg Ewing ] > Has anyone ever considered addressing that by moving the > refcounts out of

Re: [Python-Dev] Memory bitmaps for the Python cyclic garbage collector

Tim Peters wrote: In that case, it's because Python _does_ mutate the objects' refcount members under the covers, and so the OS ends up making fresh copies of the memory anyway. Has anyone ever considered addressing that by moving the refcounts out of the objects and keeping them somewhere

Re: [Python-Dev] Memory bitmaps for the Python cyclic garbage collector

[Neil Schemenauer ] > Python objects that participate in cyclic GC (things like lists, dicts, > sets but not strings, ints and floats) have extra memory overhead. I > think it is possible to mostly eliminate this overhead. Also, while > the GC is running, this GC state is

Re: [Python-Dev] Memory bitmaps for the Python cyclic garbage collector

Big +1. I love the idea. str (especially, docstring), dict, and tuples are major memory eater in Python. This may reduce tuple memory usage massively. INADA Naoki On Fri, Sep 8, 2017 at 2:30 AM, Neil Schemenauer wrote: > Python objects that

Re: [Python-Dev] Memory bitmaps for the Python cyclic garbage collector

On Thu, 7 Sep 2017 11:30:12 -0600 Neil Schemenauer wrote: > > * The GC process would work nearly the same as it does now. Rather than > only traversing the linked list, we would also have to crawl over the > GC object arenas, check blocks of memory that have the tracked bit

[Python-Dev] Memory bitmaps for the Python cyclic garbage collector

Python objects that participate in cyclic GC (things like lists, dicts, sets but not strings, ints and floats) have extra memory overhead. I think it is possible to mostly eliminate this overhead. Also, while the GC is running, this GC state is mutated, which destroys copy-on-write

Re: [Python-Dev] Memory

Sorry. Somebody take the keyboard away from me after midnight. I think Google Chrome does have that feature. Definitely Not related to Python dev... Google translate doesn't correctly understand everything. My apologies (again ) for this unrelated post. And, yes, I am the author of the

Re: [Python-Dev] Memory

That would certainly be the most favorable interpretation. On Sun, May 1, 2016 at 10:41 AM, Carl Kleffner wrote: > AFAIK Burkhard A. Meier is the author of the Python GUI Programming > Cookbook (Packtpub). I wonder if someone has hacked is gmail account? > > 2016-05-01

Re: [Python-Dev] Memory

AFAIK Burkhard A. Meier is the author of the Python GUI Programming Cookbook (Packtpub). I wonder if someone has hacked is gmail account? 2016-05-01 18:51 GMT+02:00 Steven D'Aprano : > On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote: > > Well, I put this in

Re: [Python-Dev] Memory

On Sun, May 01, 2016 at 08:08:22AM -0500, Ryan Gonzalez wrote: > Well, I put this in Google Translate...and got this: > > The disk clatters > the Spontie giggles > ~ > hopefully > alliance insures ... > > Not sure if this a useless post or Translate just being weird. Leaning > towards the

Re: [Python-Dev] Memory

Well, I put this in Google Translate...and got this: The disk clatters the Spontie giggles ~ hopefully alliance insures ... Not sure if this a useless post or Translate just being weird. Leaning towards the latter... -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your

[Python-Dev] Memory

~ Die Scheibe klirrt der Spontie kichert ~ hoffentlich Alliiance versichert... ~ https://youtu.be/m3SjCzA71eM ___ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

Re: [Python-Dev] Memory BIO for _ssl

On Mon, Jul 7, 2014 at 1:49 AM, Antoine Pitrou anto...@python.org wrote: Le 05/07/2014 14:04, Geert Jansen a écrit : Since I need this for my Gruvi async framework, I want to volunteer to write a patch. It should be useful as well to Py3K's asyncio and other async frameworks. It would be

Re: [Python-Dev] Memory BIO for _ssl

Hi, Le 05/07/2014 14:04, Geert Jansen a écrit : Since I need this for my Gruvi async framework, I want to volunteer to write a patch. It should be useful as well to Py3K's asyncio and other async frameworks. It would be good to get some feedback before I start on this. Thanks for

[Python-Dev] Memory tests in Unicode

The test_raiseMemError() in test_unicode.py is complicating me the regression tests: tries to use all the memory in my system, which starts to swap as crazy, and almost freezes everything. When the test finishes (always pass ok), I have all the memory flushed so it take a few seconds to go back to

Re: [Python-Dev] Memory tests in Unicode

Facundo Batista wrote: The test_raiseMemError() in test_unicode.py is complicating me the regression tests: tries to use all the memory in my system, which starts to swap as crazy, and almost freezes everything. When the test finishes (always pass ok), I have all the memory flushed so it take

Re: [Python-Dev] Memory tests in Unicode

Hi, Le Saturday 16 August 2008 15:43:28 Facundo Batista, vous avez écrit : The test_raiseMemError() in test_unicode.py is complicating me the regression tests: tries to use all the memory in my system, which starts to swap as crazy, and almost freezes everything. On UNIX, it's possible to

Re: [Python-Dev] Memory tests in Unicode

Facundo Batista facundobatista at gmail.com writes: The test_raiseMemError() in test_unicode.py is complicating me the regression tests: tries to use all the memory in my system, which starts to swap as crazy, and almost freezes everything. When the test finishes (always pass ok), I have all

Re: [Python-Dev] Memory tests in Unicode

2008/8/16 Antoine Pitrou [EMAIL PROTECTED]: If the test does allocate the very large string, it means MemoryError isn't raised, which defeats the purpose of the test. I do *not* want to remove the test. I just want to execute it *only* when I run -u all or -u memory, not everytime I run the

Re: [Python-Dev] Memory tests in Unicode

On Sat, 16 Aug 2008 13:01:33 -0300, Facundo Batista [EMAIL PROTECTED] wrote: 2008/8/16 Antoine Pitrou [EMAIL PROTECTED]: If the test does allocate the very large string, it means MemoryError isn't raised, which defeats the purpose of the test. I do *not* want to remove the test. Antoine

Re: [Python-Dev] Memory tests in Unicode

Facundo Batista facundobatista at gmail.com writes: I do *not* want to remove the test. You misunderstood my remark. If the test takes a lot of time and eats a lot of memory, it's precisely because the MemoryError isn't raised; and the test needs MemoryError to be raised in order to be

Re: [Python-Dev] Memory tests in Unicode

Are you having also this issue? Do you think that it should run not *always*? See bug http://bugs.python.org/issue3556 If no solution is forthcoming quickly, I recommend to remove/disable the test. Regards, Martin ___ Python-Dev mailing list

Re: [Python-Dev] Memory tests in Unicode

2008/8/16 Martin v. Löwis [EMAIL PROTECTED]: See bug http://bugs.python.org/issue3556 If no solution is forthcoming quickly, I recommend to remove/disable the test. The Antoine patch works ok for me, and solves the whole issue. I'm on linux, if somebody can give it a try in another

Re: [Python-Dev] Memory tests in Unicode

Antoine Pitrou wrote: Facundo Batista facundobatista at gmail.com writes: The test_raiseMemError() in test_unicode.py is complicating me the regression tests: tries to use all the memory in my system, which starts to swap as crazy, and almost freezes everything. When the test finishes (always

[Python-Dev] Memory Error while reading large file

Hi I have a problem reading very large text file. When I call the len function to get the total lines in python file.i get memory error . I am reading the list of files in a loop ,2 files are read properly but when the third file is read , It gives an memory error . Sumant Gupta Software

Re: [Python-Dev] Memory Error while reading large file

On Thu, 31 Jul 2008 03:01:42 pm Sumant Gupta wrote: Hi I have a problem reading very large text file. When I call the len function to get the total lines in python file.i get memory error . I am reading the list of files in a loop ,2 files are read properly but when the third file is read ,

Re: [Python-Dev] Memory Error while reading large file

If you have an actual question I'd like to stress this point as well. Any good posting one wants an answer to must include a question, and that question must be explicitly phrased, and terminated with a question mark. (maybe the use of the question mark is more typical in German than in

Re: [Python-Dev] Memory Error while reading large file

Martin v. Löwis wrote: (maybe the use of the question mark is more typical in German than in English; my stomach turns around when I read a question that ends with a full stop) There is no loss in translation here. Proper English requires the use of a question mark just the same as German,

Re: [Python-Dev] Memory Error while reading large file

On Thu, Jul 31, 2008 at 2:38 PM, Scott Dial [EMAIL PROTECTED] wrote: Martin v. Löwis wrote: (maybe the use of the question mark is more typical in German than in English; my stomach turns around when I read a question that ends with a full stop) There is no loss in translation here. Proper

Re: [Python-Dev] Memory Error while reading large file

Martin v. Löwis wrote: (maybe the use of the question mark is more typical in German than in English; my stomach turns around when I read a question that ends with a full stop) No, it's required in English, too. -- Greg ___ Python-Dev mailing

[Python-Dev] Memory benchmarking?

Hi all, is there a good, or standard memory benchmarking system for Python? pybench doesn't return significantly different results when Python 2.6 is compiled with pymalloc and without pymalloc. Thinking on it, I'm not too surprised -- pybench probably benchmarks a lot of stuff -- but some

Re: [Python-Dev] Memory benchmarking?

On 2007-11-29 11:52, Titus Brown wrote: Hi all, is there a good, or standard memory benchmarking system for Python? pybench doesn't return significantly different results when Python 2.6 is compiled with pymalloc and without pymalloc. Thinking on it, I'm not too surprised -- pybench

Re: [Python-Dev] Memory Error the right error for coding cookie promise violation?

On Feb 21, 2006, at 6:53 AM, Bengt Richter wrote: Perhaps a more informative message would be nice. Here's an easy way to trigger it: compile(#-*- coding: ascii -*-\nprint 'ab%c'\n%0x80, '','exec') Traceback (most recent call last): File stdin, line 1, in ? MemoryError Definitely

Re: [Python-Dev] Memory management in the AST parser compiler

Greg Ewing wrote: Neal Norwitz wrote: I'm mostly convinced that using PyObjects would be a good thing. However, making the change isn't free as all the types need to be created and this is likely quite a bit of code. Since they're all so similar, perhaps they could be auto-generated by

Re: [Python-Dev] Memory management in the AST parser compiler

Greg Ewing [EMAIL PROTECTED] writes: Guido van Rossum wrote: To me, that's an argument in favour of always generating a .pyc, even for scripts. I'm not sure I follow the connection. You were saying that if the parser and compiler were slow, it would slow down single-file scripts that

Re: [Python-Dev] Memory management in the AST parser compiler

Nick Coghlan wrote: The ast C structs are already auto-generated by a Python script (asdl_c.py, to be precise). The trick is to make that script generate full PyObjects rather than the simple C structures that it generates now. See the ast-object branch. The second step is to then

Re: [Python-Dev] Memory management in the AST parser compiler

Sure. If they're immutable sharing is fine, but you end up making a copy anyway if you want to make changes, right? Jeremy On 11/30/05, Greg Ewing [EMAIL PROTECTED] wrote: Jeremy Hylton wrote: I still think passing copies is better than sharing live objects between Python and C, Even if

Re: [Python-Dev] Memory management in the AST parser compiler

Neal Norwitz wrote: On 11/28/05, Martin v. Löwis [EMAIL PROTECTED] wrote: Neal Norwitz wrote: Hope this helps explain a bit. Please speak up with how this can be improved. Gotta run. I would rewrite it as [code snipped] For those watching, Greg's and Martin's version were almost the

Re: [Python-Dev] Memory management in the AST parser compiler

On 11/29/05, Nick Coghlan [EMAIL PROTECTED] wrote: When working on the CST-AST parser, there were only a few things I found to be seriously painful about the memory management: 1. Remembering which free_* variant to call for AST nodes 2. Remembering which asdl_seq_*_free variant to

Re: [Python-Dev] Memory management in the AST parser compiler

On 11/18/05, Neil Schemenauer [EMAIL PROTECTED] wrote: Perhaps we should use the memory management technique that the rest of Python uses: reference counting. I don't see why the AST structures couldn't be PyObjects. Me neither. Adding yet another memory allocation scheme to Python's already

Re: [Python-Dev] Memory management in the AST parser compiler

On 11/28/05, Jeremy Hylton [EMAIL PROTECTED] wrote: On 11/28/05, Guido van Rossum [EMAIL PROTECTED] wrote: On 11/18/05, Neil Schemenauer [EMAIL PROTECTED] wrote: Perhaps we should use the memory management technique that the rest of Python uses: reference counting. I don't see why the

Re: [Python-Dev] Memory management in the AST parser compiler

Jeremy Hylton wrote: The reason this thread started was the complaint that reference counting in the compiler is really difficult. Almost every line of code can lead to an error exit. The code becomes quite cluttered when it uses reference counting. Right now, the AST is created with

Re: [Python-Dev] Memory management in the AST parser compiler

[Guido] Then I don't understand why there was discussion of alloca() earlier on -- surely the lifetime of a node should not be limited by the stack frame that allocated it? [Jeremy] Actually this is a pretty good limit, because all these data structures are temporaries used by the

Re: [Python-Dev] Memory management in the AST parser compiler

Here's a somewhat radical idea: Why not write the parser and bytecode compiler in Python? A .pyc could be bootstrapped from it and frozen into the executable. -- Greg Ewing, Computer Science Dept, +--+ University of Canterbury, | A citizen of

Re: [Python-Dev] Memory management in the AST parser compiler

On 11/28/05, Greg Ewing [EMAIL PROTECTED] wrote: Here's a somewhat radical idea: Why not write the parser and bytecode compiler in Python? A .pyc could be bootstrapped from it and frozen into the executable. Is there a specific reason you are leaving out the AST, Greg, or do you count that

Re: [Python-Dev] Memory management in the AST parser compiler

On 11/28/05, Martin v. Löwis [EMAIL PROTECTED] wrote: Neal Norwitz wrote: Hope this helps explain a bit. Please speak up with how this can be improved. Gotta run. I would rewrite it as [code snipped] For those watching, Greg's and Martin's version were almost the same. However, Greg's

Re: [Python-Dev] Memory management in the AST parser compiler

Neal Norwitz wrote: For those watching, Greg's and Martin's version were almost the same. However, Greg's version left in the memory leak, while Martin fixed it by letting the result fall through. Actually, Greg said (correctly) that his version also fixes the leak: he assumed that

Re: [Python-Dev] Memory management in the AST parser compiler

Neil Schemenauer wrote: Fredrik Lundh [EMAIL PROTECTED] wrote: Thomas Lee wrote: Even if it meant we had just one function call - one, safe function call that deallocated all the memory allocated within a function - that we had to put before each and every return, that's better than

Re: [Python-Dev] Memory management in the AST parser compiler

Fredrik Lundh [EMAIL PROTECTED] wrote: Thomas Lee wrote: Even if it meant we had just one function call - one, safe function call that deallocated all the memory allocated within a function - that we had to put before each and every return, that's better than what we have. alloca? Perhaps

Re: [Python-Dev] Memory management in the AST parser compiler

The behavior of libiberty's alloca() replacement might be interesting as well: http://gcc.gnu.org/onlinedocs/libiberty/Functions.html#index-alloca-59 Regards, Michael On 11/18/05, Alex Martelli [EMAIL PROTECTED] wrote: On Nov 17, 2005, at 5:00 PM, Thomas Lee wrote: Portability may also be

[Python-Dev] Memory management in the AST parser compiler

There is a public-domain implementation of alloca at http://www.cs.purdue.edu/homes/apm/courses/BITSC461-fall03/listen-code/listen-1.0-dave/lsl_cpp/alloca.c It would still fail on architectures that don't use a stack frame; other than that, it seems like a reasonable fallback, if alloca is

Re: [Python-Dev] Memory management in the AST parser compiler

On 11/16/05, Fredrik Lundh [EMAIL PROTECTED] wrote: Thomas Lee wrote: Even if it meant we had just one function call - one, safe function call that deallocated all the memory allocated within a function - that we had to put before each and every return, that's better than what we have.

Re: [Python-Dev] Memory management in the AST parser compiler

On 11/16/05, Thomas Lee [EMAIL PROTECTED] wrote: Just messing around with some ideas. I was trying to avoid the ugly macros (note my earlier whinge about a learning curve) but they're the cleanest way I could think of to get around the problem without resorting to a mass deallocation right at

Re: [Python-Dev] Memory management in the AST parser compiler

Portability may also be an issue to take into consideration: http://www.eskimo.com/~scs/C-faq/q7.32.html http://archives.neohapsis.com/archives/postfix/2001-05/1305.html Cheers, Tom Alex Martelli wrote: On Nov 17, 2005, at 12:46 PM, Brett Cannon wrote: ... alloca? (duck) But

Re: [Python-Dev] Memory management in the AST parser compiler

On Nov 17, 2005, at 5:00 PM, Thomas Lee wrote: Portability may also be an issue to take into consideration: Of course -- but so is anno domini... the eskimo.com FAQ is (C) 1995, and the neohapsis.com page just points to the eskimo.com one: http://www.eskimo.com/~scs/C-faq/q7.32.html

Re: [Python-Dev] Memory management in the AST parser compiler

Boy am I wanting RAII from C++ for automatic freeing when scope is left. Maybe we need to come up with a similar thing, like all memory that should be freed once a scope is left must use some special struct that stores references to all created memory locally and then a free call must be

Re: [Python-Dev] Memory management in the AST parser compiler

Niko Matsakis wrote: Boy am I wanting RAII from C++ for automatic freeing when scope is left. Maybe we need to come up with a similar thing, like all memory that should be freed once a scope is left must use some special struct that stores references to all created memory locally and then a free

Re: [Python-Dev] Memory management in the AST parser compiler

By the way, I liked the sound of the arena/pool tree - really good idea. Thomas Lee wrote: Niko Matsakis wrote: Boy am I wanting RAII from C++ for automatic freeing when scope is left. Maybe we need to come up with a similar thing, like all memory that should be freed once a scope is left

Re: [Python-Dev] Memory management in the AST parser compiler

Thomas Lee wrote: As the writer of the crappy code that sparked this conversation, I feel I should say something :) Don't feel bad about it. It turned out the 'helpful' review comments from Neal and I didn't originally work out very well either ;) With the AST compiler being so new, this is

Re: [Python-Dev] Memory management in the AST parser compiler

Just messing around with some ideas. I was trying to avoid the ugly macros (note my earlier whinge about a learning curve) but they're the cleanest way I could think of to get around the problem without resorting to a mass deallocation right at the end of the AST run. Which may not be all that

Re: [Python-Dev] Memory management in the AST parser compiler

2005/11/15, Nick Coghlan [EMAIL PROTECTED]: Specifically, the body of the entire function is written inside a switch statement, with 'break' then used as the equivalent of raise Exception. For example: PyObject* switchAsTry() { switch(0) { default: /* Real

Re: [Python-Dev] Memory management in the AST parser compiler

Marek Baczek Baczyński wrote: 2005/11/15, Nick Coghlan [EMAIL PROTECTED]: It avoids the potential for labelling problems that arises when goto's are used for resource cleanup. It's a far cry from real exception handling, but it's the best solution I've seen within the limits of C. delurk

Re: [Python-Dev] Memory management in the AST parser compiler

As Neal pointed out, it's tricky to write code for the AST parser and compiler without accidentally letting memory leak when the parser or compiler runs into a problem and has to bail out on whatever it was doing. Thomas's patch got to v5 (based on Neal's review comments) with memory

Re: [Python-Dev] Memory management in the AST parser compiler

On 11/15/05, Niko Matsakis [EMAIL PROTECTED] wrote: As Neal pointed out, it's tricky to write code for the AST parser and compiler without accidentally letting memory leak when the parser or compiler runs into a problem and has to bail out on whatever it was doing. Thomas's patch got

Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?

On Mar 4, 2005, at 23:55, Brett C. wrote: Evan uploaded the newest version (@ http://www.python.org/sf/1123430) and he says it is complete. I should point out (very late! It's been a busy couple of weeks) that I fully expect that there may be some issues with the patch that will arise when it

Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?

Brett C. wrote: Assuming a code review says the patch is sane, do we want to go with this garbage collection change? From past discussions I don't remember a consensus on acceptance or rejection, just lots of discussion about ripping out the hacks to allow freeing memory w/o holding the GIL (I

Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?

Evan Jones wrote: Sorry for taking so long to get back to this thread, it has been one of those weeks for me. On Feb 16, 2005, at 2:50, Martin v. Löwis wrote: Evan then understood the feature, and made it possible. This is very true: it was a very useful exercise. I can personally accept

Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?

Sorry for taking so long to get back to this thread, it has been one of those weeks for me. On Feb 16, 2005, at 2:50, Martin v. Löwis wrote: Evan then understood the feature, and made it possible. This is very true: it was a very useful exercise. I can personally accept breaking the code that

Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?

[Evan Jones] After I finally understood what thread-safety guarantees the Python memory allocator needs to provide, I went and did some hard thinking about the code this afternoon. I believe that my modifications provide the same guarantees that the original version did. I do need to declare

Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?

[Tim Peters] As I said before, I don't think we need to support this any more. More, I think we should not -- the support code is excruciatingly subtle, it wasted plenty of your time trying to keep it working, and if we keep it in it's going to continue to waste time over the coming years

Re: [Python-Dev] Memory Allocator Part 2: Did I get it right?

Tim Peters wrote: I'm not certain it is acceptable to make this assumption. Why is it not possible to use the same approach that was previously used (i.e. leak the arenas array)? Do you have something else in mind? I'll talk with Martin about it if he still wants to. Martin, this