On Tue, Aug 7, 2012 at 6:33 AM, Armin Rigo <ar...@tunes.org> wrote: > If you can come up with a more precise scheme, you're welcome. The > issue is to know when it's ok to reserve from that pool and when we > should raise an RPython MemoryError instead. A possible answer would
I'm not convinced this MemoryError is unsolvable yet. Perhaps it can be broken into a couple of steps. When the app requests memory that would leave too little remaining, the memory request needs to be refused and a MemoryError sent instead. This informs the app that it can no longer make memory requests like that and expect them to succeed. The app then has a few options. Leave the exception unhandled and get terminated. Reduce it's memory usage by pruning it's structures. Stop an operation and report the problem, i.e. "Document too large too load". The key point is that the RPython memory system signals the app's memory system so it can respond as needed. If the app tries to handle the MemoryError it will need more memory requests to succeed, for temporaries like Armin points out, to get it's job done. Either the free memory available will be restored by the app to an amount sufficient to avoid further MemoryErrors, or the app basically had it's chance and now RPython needs the remaining memory to successfully describe the apps early demise (print MemoryError and a stack trace showing what the app was doing). So we don't need special pools of memory, new APIs or anything special. We do need to have two out of memory thresholds. The first is enough for RPython to terminate the app and describe the problem. The second, higher threshold, is used to notify the app that it can no longer request memory like that unless it reduces it's usage. Only one MemoryError is sent to the app until either the app frees enough memory or gets terminated. Imaginary code could look something like this: def alloc(size): if free_mem - size < app_reserve + rpython_reserve: if not raised_MemoryError: raised_MemoryError = True raise MemoryError() if free_mem - size < rpython_reserve: print 'MemoryError' print stack_trace() # of app else: raised_MemoryError = False app_reserve and rpython_reserve might be 1K, 10K, 100K, idk. Apps might want a variable they can tweak to further pad the reserve because they know they need extra room to prune their memory systems. My main question is can the RPython memory system do a collection while using only rpython_reserve of memory? Note the MemoryError when RPython terminates the app isn't really a MemoryError. It's not an exception. It can't be caught (it's too late for that). A different name really should be used to avoid confusion. Basically, this design sends one MemoryError to my test code to give it a chance to do something about it. And it will let it keep going for a while. Soon though, it will give up on the test code, terminate it, and report why. No RPython traceback is needed. To me, the key issue is that apps get at least one MemoryError so they can do something about it. The current state, that they may or may not get to do *anything* is kind of scary. -Roger _______________________________________________ pypy-dev mailing list pypy-dev@python.org http://mail.python.org/mailman/listinfo/pypy-dev
