The one thing I didn't include in the post is the output of "vmmap" on
OS X prior to this change.
It showed large amounts of memory both marked as "(freed)" and some
not. That pattern reminded me of internal fragmentation in a malloc
implementation, which is a standard problem to be overcome when
implementing one. If anyone is interested, I could post that as well.
*Joe Khoobyar
*
Chief Technical Officer & Lead Developer
CollectiveX - /Groups that work!/
mobile: 585.245.2902
email: j...@collectivex.com <mailto:j...@collectivex.com>
web: www.collectivex.com
<http://www.collectivex.com/> www.groupsites.com
<http://www.groupsites.com/>
The third-rate mind is only happy when it is thinking with the majority.
The second-rate mind is only happy when it is thinking with the minority.
The first-rate mind is only happy when it is thinking.
— A.A. Milne
Charlie Savage wrote:
Hey Joe,
Revision 783 in subversion applies this change. With this change,
test08.rb holds about 13 MB on a Mac, where it held over 120 MB before.
Ah, interesting. It is amazing how much memory libxml seems to grab
on simple test cases.
We are simply using libxml2's memory management hooks to direct it's
alloc/free functions and related functions to ruby's versions of
those functions. What I originally planned (which is why I'm using
xmlGcMemSetup) was to direct what libxml2 calls "atomic" memory
allocations to the version of the ruby's allocator which is able to
run the garbage collector.
> Basically what this does is have both libxml2 and ruby use the same
> allocator and allow ruby to run it's GC even in response to libxml2
> allocations which keeps memory usage down much more easily. I'm
already
> running this on all of our production servers and it yielded instant
> benefits.
What I gather from this is that somehow libxml is not freeing memory
that it allocates. Except that memory is freed, albeit indirectly,
when Ruby when it runs a garbage collection freeing Ruby objects and
then the underlying libxml objects. And from using valgrind, I
haven't seen any large memory leaks in libxml-ruby for quite a while.
So I'm wondering what is going on. Does libxml just allocate a big
chunk of memory at startup, and then allocates/frees from that memory
over the period that a process runs? And by telling libxml to use
Ruby's allocator, then that issue goes away? If that's the case, is
there some setting to libxml to set the starting memory. Or is there
something else entirely different going on?
And is there any downside to having libxml get memory from Ruby?
Something along the lines that ruby's memory allocation wouldn't be as
efficient for libxml's usage patterns?
I'd just like to understand this a bit better since its such a big
change (not code wise, but running wise). Should we hedge our bets
and make it settable somehow - is that even possible?
Just wondering: does anyone have any feedback on this change or on
this subject? FYI: I've been running it on 32-bit ruby on OS X
Leopard and on several 64-bit UNIX instances for half a week now.
Yeah, I'd have to give it a try on Windows. Theoretically, it sounds
fine, but we'll have to see what really happens.
Anyway, excellent work to figure this out!
Charlie
_______________________________________________
libxml-devel mailing list
libxml-devel@rubyforge.org
http://rubyforge.org/mailman/listinfo/libxml-devel
