[libxml-devel] Memory leak with 0.9.6 and 0.9.7
We are using some code that utilizes libxml and creates XML documents. Using 0.9.5 everything is fine. Using 0.9.6 or 0.9.7, the more we use this code, the higher our memory usage gets. Seems likely that a memory leak was introduced in 0.9.6. We are seeing this same behavior on OS X and Gentoo. ___ libxml-devel mailing list libxml-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/libxml-devel
Re: [libxml-devel] Memory leak with 0.9.6 and 0.9.7
Hi Joseph, We are using some code that utilizes libxml and creates XML documents. Using 0.9.5 everything is fine. Using 0.9.6 or 0.9.7, the more we use this code, the higher our memory usage gets. Seems likely that a memory leak was introduced in 0.9.6. We are seeing this same behavior on OS X and Gentoo. Can you be more specific? Do you know what code is causing the problem? Can can put together a test case? If you can identify the code that is causing the issue, then it should be easy enough to fix. Charlie smime.p7s Description: S/MIME Cryptographic Signature ___ libxml-devel mailing list libxml-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/libxml-devel
Re: [libxml-devel] Namespace issue with validate_schema?
This is likely a PEBKAC issue rather than a bug. I hope I'm submitting to the right list. I'm getting different validation results from libxml in Ruby than I am from a command-line tool (xmllint). I'm using a schema definition from ASTM to validate the output, for what it cost me the schema should be correct. I'm using libxml 0.9.4. I expect a failure, since I haven't added a required node yet (Patient), but I'm trying to reconcile the discrepancy between xmllint and libxml. Yeah, that sure is interesting. My first thought is if libxml-ruby is using the same version of libmxl that xmllint is using? Is this on Windows, Linux, OS X or some other OS? Is this a usage error on my part? Am I misusing libxml or is the xml incorrect? If this is a bug, has it been patched? If not, what information can I provide to help get it fixed? I see the xml you are validating, but can you provide the schema? Mind submitting a bug report at ruby forge? http://rubyforge.org/tracker/?group_id=494 Charlie smime.p7s Description: S/MIME Cryptographic Signature ___ libxml-devel mailing list libxml-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/libxml-devel
Re: [libxml-devel] Memory leak with 0.9.6 and 0.9.7
On Jan 7, 1:08 pm, Charlie Savage wrote: > Hi Joseph, > > > We are using some code that utilizes libxml and creates XML > > documents. Using 0.9.5 everything is fine. Using 0.9.6 or 0.9.7, the > > more we use this code, the higher our memory usage gets. > > > Seems likely that a memory leak was introduced in 0.9.6. We are > > seeing this same behavior on OS X and Gentoo. > > Can you be more specific? Do you know what code is causing the problem? > Can can put together a test case? If you can identify the code that > is causing the issue, then it should be easy enough to fix. > > Charlie > > smime.p7s > 4KViewDownload > I'll see if I can put something together, we know it's in the to_solr_doc from acts_as_solr somewhere. ___ libxml-devel mailing list libxml-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/libxml-devel
Re: [libxml-devel] Memory leak with 0.9.6 and 0.9.7
On Jan 7, 1:38 pm, Joseph Palermo wrote: > On Jan 7, 1:08 pm, Charlie Savage wrote: > > > > > Hi Joseph, > > > > We are using some code that utilizes libxml and creates XML > > > documents. Using 0.9.5 everything is fine. Using 0.9.6 or 0.9.7, the > > > more we use this code, the higher our memory usage gets. > > > > Seems likely that a memory leak was introduced in 0.9.6. We are > > > seeing this same behavior on OS X and Gentoo. > > > Can you be more specific? Do you know what code is causing the problem? > > Can can put together a test case? If you can identify the code that > > is causing the issue, then it should be easy enough to fix. > > > Charlie > > > smime.p7s > > 4KViewDownload > > I'll see if I can put something together, we know it's in the > to_solr_doc from acts_as_solr somewhere. Looks like the problem is in to_s on Node 1.times do node = XML::Node.new 'node' node.to_s end If I run that on 0.9.5 my memory usage is fine, if I run it on 0.9.6 it balloons. ___ libxml-devel mailing list libxml-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/libxml-devel
Re: [libxml-devel] Memory leak with 0.9.6 and 0.9.7
Hi Joseph, I'll see if I can put something together, we know it's in the to_solr_doc from acts_as_solr somewhere. Looks like the problem is in to_s on Node 1.times do node = XML::Node.new 'node' node.to_s end Nice work - that's a great test case. So if I I run this code: 1000.times do node = XML::Node.new 'node' #node.to_s end I see no issues. Adding the .to_s does seem to cause a problem. Are you actually doing so many to_s calls in your code? Anyway, will have to look and see if there is anything obvious in the part of the bindings. Charlie smime.p7s Description: S/MIME Cryptographic Signature ___ libxml-devel mailing list libxml-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/libxml-devel
Re: [libxml-devel] Memory leak with 0.9.6 and 0.9.7
Looks like the problem is in to_s on Node 1.times do node = XML::Node.new 'node' node.to_s end Anyway, will have to look and see if there is anything obvious in the part of the bindings. And fixed in trunk. It was caused by the addition of encoding support in the last couple of releases. The new code forgot to release the internal libxml output buffer. Thanks for the heads-up on this. Charlie smime.p7s Description: S/MIME Cryptographic Signature ___ libxml-devel mailing list libxml-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/libxml-devel
Re: [libxml-devel] Memory leak with 0.9.6 and 0.9.7
On Jan 7, 4:06 pm, Charlie Savage wrote: > Hi Joseph, > > >> I'll see if I can put something together, we know it's in the > >> to_solr_doc from acts_as_solr somewhere. > > > Looks like the problem is in to_s on Node > > > 1.times do > > node = XML::Node.new 'node' > > node.to_s > > end > > Nice work - that's a great test case. > > So if I I run this code: > > 1000.times do >node = XML::Node.new 'node' >#node.to_s > end > > I see no issues. > > Adding the .to_s does seem to cause a problem. Are you actually doing > so many to_s calls in your code? > > Anyway, will have to look and see if there is anything obvious in the > part of the bindings. > > Charlie > > smime.p7s > 4KViewDownload We had a batch job where we had to update ~300,000 records which caused a solr_save for each one which does the to_s. We don't have a lot of extra memory on our hosting provider so it showed up pretty quick. Thanks for being so responsive. ___ libxml-devel mailing list libxml-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/libxml-devel
Re: [libxml-devel] Memory leak with 0.9.6 and 0.9.7
2009/1/7 Charlie Savage : > Hi Joseph, > >>> I'll see if I can put something together, we know it's in the >>> to_solr_doc from acts_as_solr somewhere. >> >> Looks like the problem is in to_s on Node >> >> 1.times do >> node = XML::Node.new 'node' >> node.to_s >> end > > Nice work - that's a great test case. > > So if I I run this code: > > 1000.times do > node = XML::Node.new 'node' > #node.to_s > end > > I see no issues. > > Adding the .to_s does seem to cause a problem. Are you actually doing so > many to_s calls in your code? > > Anyway, will have to look and see if there is anything obvious in the part > of the bindings. Looks like the xmlOutputBufferPtr never gets freed. This might do the trick: Index: ext/libxml/ruby_xml_node.c === --- ext/libxml/ruby_xml_node.c (revision 693) +++ ext/libxml/ruby_xml_node.c (working copy) @@ -570,10 +570,16 @@ xmlNodeDumpOutput(output, xnode->doc, xnode, level, indent, encoding); xmlOutputBufferFlush(output); + VALUE string = Qnil; + if (output->conv) -return rb_str_new2((const char*) output->conv->content); +string = rb_str_new2((const char*) output->conv->content); else -return rb_str_new2((const char*) output->buffer->content); +string = rb_str_new2((const char*) output->buffer->content); + + xmlOutputBufferClose(output); + + return string; } -- Aaron Patterson http://tenderlovemaking.com/ ___ libxml-devel mailing list libxml-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/libxml-devel
Re: [libxml-devel] Memory leak with 0.9.6 and 0.9.7
Hey Aaron, Looks like the xmlOutputBufferPtr never gets freed. Yup, exactly right. I posted a fix to trunk and sent another email to the list, but somehow it never made it through. Index: ext/libxml/ruby_xml_node.c === --- ext/libxml/ruby_xml_node.c (revision 693) +++ ext/libxml/ruby_xml_node.c (working copy) @@ -570,10 +570,16 @@ xmlNodeDumpOutput(output, xnode->doc, xnode, level, indent, encoding); xmlOutputBufferFlush(output); + VALUE string = Qnil; + if (output->conv) -return rb_str_new2((const char*) output->conv->content); +string = rb_str_new2((const char*) output->conv->content); else -return rb_str_new2((const char*) output->buffer->content); +string = rb_str_new2((const char*) output->buffer->content); + + xmlOutputBufferClose(output); + + return string; } Yup, trunk updated to do exactly this. Thanks for the patch. Charlie smime.p7s Description: S/MIME Cryptographic Signature ___ libxml-devel mailing list libxml-devel@rubyforge.org http://rubyforge.org/mailman/listinfo/libxml-devel
