Re: Python for embedded systems with memory constraints
Hello, Using the best fit for Python will not be a problem, because Python makes allocations of lot of small size blocks.So those split blocks of small sizes are used by Python sometime. And what I observed from my investigation with the memory manager(MM) for Python is , with any MM we cannot eliminate fragmentation and even though Python is memory hungry I cannot allot some 50MB (or more) just for python application because it will add to the embedded system memory cost. So now I only see the solution to clear my memory pool and restart Python without restarting the system (i.e. no power cycle to hardware). I tried to do this when my memory pool is 60% used in these steps: 1) Py_Finalize( ) 2) Reset my Memory pool (i.e. free list links) 3) Then Restart Python by calling Py_Initialize(). But this resulted in Python crash during Py_Initialize(), where I found that the static variables within embedded Python source code are still holding some of the references to my memory pool. So now my question is how do I restart Python (i.e. reinitialize Python) without restarting whole system. Is there a way to reset/re-initilaize those static variables such that it will be possible to re-Initialize Python. Vishnu On 6/10/07, MRAB [EMAIL PROTECTED] wrote: On Jun 9, 1:33 pm, vishnu [EMAIL PROTECTED] wrote: Hi, Thanks Cameron for your suggestions. In fact I am using custom memory sub-allocator where I preallocate a pool of memory during initialization of my application and ensure that Python doesn't make any system mallocs later . With this arrangement, python seems to run out of preallocated memory (of 10MB) after running few simple scripts due to huge external fragmentation. My memory sub-allocator got a good design which uses the best-fit algorithm and coaelescing the adjacent blocks during each free call. If anybody out there used their own memory manager and ran Python without fragmentation , could provide some inputs on this. From what I remember, the best-fit algorithm isn't a good idea because unless the free block was exactly the right size you'd tend to get left with lots of small fragments. (Suppose that the best fit was a free block only 4 bytes bigger than what you want; what can you do with a free block of 4 bytes?) A worst-fit algorithm would leave larger free blocks which are more useful subsequently, but I think that the recommendation was next-fit (ie use the first free block that's big enough, starting from where you found the last one). -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Re: Python for embedded systems with memory constraints
Hi, Thanks Cameron for your suggestions. In fact I am using custom memory sub-allocator where I preallocate a pool of memory during initialization of my application and ensure that Python doesn't make any system mallocs later . With this arrangement, python seems to run out of preallocated memory (of 10MB) after running few simple scripts due to huge external fragmentation. My memory sub-allocator got a good design which uses the best-fit algorithm and coaelescing the adjacent blocks during each free call. If anybody out there used their own memory manager and ran Python without fragmentation , could provide some inputs on this. Thanks in advance. On 6/7/07, Cameron Laird [EMAIL PROTECTED] wrote: In article [EMAIL PROTECTED], vishnu [EMAIL PROTECTED] wrote: Hi there, I am embedding python 2.5 on embedded system running on RTOS where I had strict memory constraints. As python is a huge malloc intensive application, I observed huge memory fragmentation in my system which is leading to out of memory after running few scripts. So I decided to re-initialise the python with out restarting the whole python. I tried to use Py_Finalise() after completion of each script , then call Py_Initialise as is done in below link. http://mail.python.org/pipermail/python-list/2001-November/114253.html Which in every loop it causes a leak of 10K and after some iterations it leaks of 200K etc. After few more runs this crashes. I read some where this leak was solved in 2.5, but with 2.5 also I am having problems And also I found Py_Finalise does not completely cleanup the memory, So how do I re-initialise my memory pool? Does anybody faced this problem earlier and got any solution hack to run the python for a embedded system within own managed memory pool of say 10MB? Any help/ideas are greatly appreciated!. Thanks in advance. Your report is interesting and important--and surprising! I thought Python memory allocation is cleaner than you seem to be observing. I hope one of the core Python maintainers can address this. I haven't worked at this level recently enough to speculate on why it's happen- ing, nor will I soon be in a position to volunteer to research it on my own (although I'd eagerly contract to do so on a modestly paid basis). Depending on your schedule and technology, there are lots of technical fixes that might apply: A. quick-starting Python variations that encourage you to manage memory on a whole-process level; B. use of one of the many Python variants (even PyPy?) that might give you a more favorable memory profile; C. switch to Lua or Tcl as more easily embeddable alternative languages; D. custom memory allocator; ... -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
Python for embedded systems with memory constraints
Hi there, I am embedding python 2.5 on embedded system running on RTOS where I had strict memory constraints. As python is a huge malloc intensive application, I observed huge memory fragmentation in my system which is leading to out of memory after running few scripts. So I decided to re-initialise the python with out restarting the whole python. I tried to use Py_Finalise() after completion of each script , then call Py_Initialise as is done in below link. http://mail.python.org/pipermail/python-list/2001-November/114253.html Which in every loop it causes a leak of 10K and after some iterations it leaks of 200K etc. After few more runs this crashes. I read some where this leak was solved in 2.5, but with 2.5 also I am having problems And also I found Py_Finalise does not completely cleanup the memory, So how do I re-initialise my memory pool? Does anybody faced this problem earlier and got any solution hack to run the python for a embedded system within own managed memory pool of say 10MB? Any help/ideas are greatly appreciated!. Thanks in advance. -- http://mail.python.org/mailman/listinfo/python-list
Extending embedded python of multiple interpreters
Hello All,
I have embedded python 2.5 in to my C application. As we need the
python scripts to run in multi threaded environment I have used
Py_NewInterpreter() and Py_EndInterpreter each time I execute a run
script function.
The code is as follows:
RunScript(char *pScriptName,char *pFuncName,...)
{
PyEval_AcquireLock()
threadState = Py_NewInterpreter();
PyThreadState_Swap(threadState);
/* Import the script module and run the fnc in that script module */
Pyobject *pModule = PyImport_Import(pScriptName);
PyObject *pFunc = PyObject_GetAttrString(pModule, pFuncName);
/* End running the script and calling the script fnc */
Py_EndInterpreter(threadState);
PyEval_ReleaseLock();
}.
And my python initialise looks as :
pythonInit() {
Py_initialise();
PyEval_initThreads();//which enables multi thread support
}
Now I am extending the embedded python to expose my application
library API's to python using SWIG. I written one extension module and
from python i am setting the C structure member values which i
wrapped. I execute this python code using call to RunScript (above
function), it works fine for the first time, but from second time
onwards I get an error - Type Error : in method
structurename_member_set.
Hence my question is the same python code worked for first call, but
from 2nd call it is not working. Could any body help me where I am
making mistake.
Is extension modules does not work properly with multiple
interpreters? Does any body had this behaviour when worked with
extension modules and multiple interpreters creation for each thread.
Thanks in Advance for the help.
Vishnu
--
http://mail.python.org/mailman/listinfo/python-list
Pass pointer from C++ class to Boost Python script ?!!!
Hi,
Did you get an answer to this posting of yours?
What I am trying to do is:
struct Sec
{
...
};
int main(int, char **)
{
Py_Initialize();
...
Sec *s1 = new Sec();
PyObject *args = Py_BuildValue((O), s1); //*** this is where I am
having problem
...
PyObject* res = PyEval_CallObject( func, args);
...
Py_Finalize();
return 0;
}
Can you please tell me how to get convert my pointer to a valid
PyObject*?
Thanks,
Vishnu Mavuram
NOTICE: If received in error, please destroy and notify sender. Sender does
not waive confidentiality or privilege, and use is prohibited.
--
http://mail.python.org/mailman/listinfo/python-list
RE: How to write at a sppecific location
Hi Pooja, Check the fileinput module's input function. ~Vishnu -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Pooja Sent: Tuesday, July 19, 2005 12:29 PM To: python-list@python.org Subject: How to write at a sppecific location Hi All I have one csv file which has some data related to test results. It has following information TestId,Expectedres,Actualres 101,12,13 102,13 103,14 If I want to write ActualRes value in the file , How to do that. I tried using seek but its not working. I am not able to write at a specific location. Please tell me what is the rght way. I have chked all the docs but was not able to find any good solution. -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
RE: Changing a line in a text file
Hi,
Here is the program for replacing 'line2' with 'newline'.
inp_file.txt contains,
line1
line2
line3
line4
EOF
#-- Program starts
#Get the inp file in to a list
inp_file_list = [x for x in open('inp_file.txt')]
#modify the desired line, Note: you should include newline
inp_file_list[1] = 'xxx\n'
#Open the same file
write_to_file = open('inp_file.txt', 'w')
#write the list in to the file
write_to_file.writelines(inp_file_list)
#close the file
write_to_file.close()
#-- Program ends
Hope someone will give better idea.
HTH,
Vishnu
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of
kah
Sent: Monday, April 25, 2005 3:09 PM
To: python-list@python.org
Subject: Changing a line in a text file
How do I change a line in a file??
For example I have the follwing text in my file:
line1
line2
line3
line4
How do I replace 'line2' with 'newline'. Since the write operation in
python will overwrite everything.
Regards,
Kah
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
RE: Begniner Question
if int(choice)==2: else: pass if int(choice)==2: print Here # == There should be some statement present after the # if condition. Likewise place this statement in # all other if conditions else: pass HTH, Vishnu -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Glen Sent: Tuesday, March 22, 2005 12:44 PM To: python-list@python.org Subject: Begniner Question #!/usr/local/bin/python import sys print 1.\tDo Something print 2.\tDo Something print 3.\tDo Something print 4.\tDo Something print 5.\tDo Something print 6.\tExit choice=raw_input(Please Enter Choice: ) if int(choice)==1: print Here else: pass if int(choice)==2: else: pass if int(choice)==3: else: pass if int(choice)==4: else: pass if int(choice)==5: else: pass if int(choice)==6: sys.exit(0) else: pass File ./Code.py, line 20 else: ^ IndentationError: expeted an indented block What am I doing wrong? Thank-you -- http://mail.python.org/mailman/listinfo/python-list -- http://mail.python.org/mailman/listinfo/python-list
RE: file.readlines() - gives me error (bad file descriptor)
logfile = file(r'test.txt','w')
logfile.write('datetime')
logfile.close() # - close the file
logfile = file(r'test.txt','r') # - Open the file in read mode
test=logfile.readlines()
~Vishnu.
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of
[EMAIL PROTECTED]
Sent: Thursday, January 06, 2005 11:53 AM
To: python-list@python.org
Subject: file.readlines() - gives me error (bad file descriptor)
Hey guys,
I can't figure this one out, why is this simple script giving me
problems?
logfile=file(r'test.txt','w')
logfile.write('datetime')
test=logfile.readlines()
When I run it I get the error message:
Traceback (most recent call last):
File C:\Documents and Settings\Gregory\My Documents\Get New Great
Job\testfile.py, line 3, in ?
test=logfile.readlines()
IOError: [Errno 9] Bad file descriptor
I'm running Windows XP, Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC
v.1200 32 bit (Intel)] on win32
Any help would be greatly appricated.
Thanks,
Greg
--
http://mail.python.org/mailman/listinfo/python-list
--
http://mail.python.org/mailman/listinfo/python-list
