Charles-Francois Natali neolo...@free.fr added the comment:
This is definitely a malloc bug.
Test with default malloc on a Debian box:
cf@neobox:~/cpython$ ./python ../issue11849_test.py
*** Python 3.3.0 alpha
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 3778 pts/2
Charles-Francois Natali neolo...@free.fr added the comment:
BTW, after utilize lxml instead of ElementTree, such phenomenon of increasing
memory usage disappeared.
If you looked at the link I posted, you'll see that lxml had some similar
issues and solved it by calling malloc_trim
kaifeng cafe...@gmail.com added the comment:
Found a minor defect of Python 3.2 / 3.3: line 1676 of xml/etree/ElementTree.py
was:
del self.target, self._parser # get rid of circular references
should be:
del self.target, self._target, self.parser, self._parser # get rid of
circular
Charles-Francois Natali neolo...@free.fr added the comment:
kaifeng cafe...@gmail.com added the comment:
I added 'malloc_trim' to the test code and rerun the test with Python 2.5 /
3.2 on CentOS 5.3. The problem still exists.
Well, malloc_trim can fail, but how did you add it ? Did you
kaifeng cafe...@gmail.com added the comment:
I applied your patch to Python 3.2, also I added a function call to
'malloc_trim' via ctypes, as you can see in issue11849_test2.py.
In fact I have a daemon written in Python 2.5, parsing an XML of size 10+ MB
every 5 minutes, after 16+ hours
Charles-Francois Natali neolo...@free.fr added the comment:
The problem is not with Python, but with your libc.
When a program - such as Python - returns memory, it uses the free(3) library
call.
But the libc is free to either return the memory immediately to the kernel
using the relevant
Changes by Charles-Francois Natali neolo...@free.fr:
--
keywords: +patch
Added file: http://bugs.python.org/file21696/gc_trim.diff
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11849
___
Antoine Pitrou pit...@free.fr added the comment:
To sum up, python is returning memory, but your libc is not.
You can force it using malloc_trim, see the attached patch (I'm not at
all suggesting its inclusion, it's just an illustration).
That's an interesting thing, perhaps you want to
kaifeng cafe...@gmail.com added the comment:
I added 'malloc_trim' to the test code and rerun the test with Python 2.5 / 3.2
on CentOS 5.3. The problem still exists.
*** Python 2.5.5 final
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 2567 pts/0S+ 0:00
New submission from Kaifeng Zhu cafe...@gmail.com:
I'm using xml.etree.ElementTree to parse large XML file, while the memory keep
increasing consistently.
You can run attached test script to reproduce it. From 'top' in Linux or 'Task
Manager' in Windows, the memory usage of python is not
Florent Xicluna florent.xicl...@gmail.com added the comment:
Do you experience same issue with current versions of Python? (3.2 or 2.7)
The package was upgraded in latest versions.
--
nosy: +flox
___
Python tracker rep...@bugs.python.org
Kaifeng Zhu cafe...@gmail.com added the comment:
Yes. Just tested with Python 2.7 and 3.2 in Windows 7, the memory usage is
still unexpected high after 'Done' is printed.
--
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue11849
Florent Xicluna florent.xicl...@gmail.com added the comment:
I've tested a small variant of your script, on OSX.
It seems to behave correctly (with 2.5, 2.6, 2.7 and 3.1).
You can force Python to release memory immediately by calling gc.collect().
--
Added file:
Florent Xicluna florent.xicl...@gmail.com added the comment:
this is the output for 2.7.1:
$ python2.7 issue11849_test.py
*** Python 2.7.1 final
--- PID STAT TIME SL RE PAGEIN VSZRSS LIM TSIZ %CPU
%MEM COMMAND
0 2754 S+ 0:00.07 0 0 0 2441472 5372
kaifeng cafe...@gmail.com added the comment:
Python 3.2 On Linux (CentOS 5.3)
*** Python 3.2.0 final
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 15116 pts/0S+ 0:00 1 1316 11055 6452 0.6 python3.2
issue11849_test.py
1 15116 pts/0S+ 0:02
15 matches
Mail list logo
