[SDO C++] Leaking libxml memory
One of our PHP users has reported a problem with leaking memory from
libxml2. I started to investigate, but realised that the issue is
independent of PHP, and can be reproduced in a standalone Tuscany
environment. The issue is that memory allocated by libxml2 is not being
freed, so he is seeing a memory leak growing with each invocation.
For example, if we take a small schema:
?xml version=1.0?
xsd:schema xmlns:xsd=http://www.w3.org/2001/XMLSchema;
xsd:element name=courses
xsd:complexType
xsd:sequence
xsd:element name=course minOccurs=0
maxOccurs=unbounded
xsd:complexType
xsd:sequence
xsd:element name=title type=xsd:string/
xsd:element name=description
type=xsd:string/
xsd:element name=credits type=xsd:decimal/
xsd:element name=lastmodified
type=xsd:dateTime minOccurs=1/
/xsd:sequence
xsd:attribute name=cid type=xsd:ID/
/xsd:complexType
/xsd:element
/xsd:sequence
/xsd:complexType
/xsd:element
/xsd:schema
and a trivial function to load it:
void load_schema(char *name)
{
DataFactoryPtr mdg = DataFactory::getDataFactory();
XSDHelperPtr xsh = HelperProvider::getXSDHelper(mdg);
xsh-defineFile(name);
xsh = NULL;
mdg = NULL;
}
(I added the last two lines to try to encourage the reference-counting
pointers to do their work, but they made no difference to the outcome)
then use libxml features to check the memory usage:
int main (int argc, char** argv)
{
xmlInitParser();
load_schema(argv[1]);
xmlCleanupParser();
xmlMemoryDump();
return 0;
}
(note: in order to use these libxml functions, you must recompile with
debug=yes mem_debug=yes)
We see the following output in libxml's .memdump file:
MEMORY ALLOCATED : 108, MAX was 13687
BLOCK NUMBER SIZE TYPE
0 984 3 malloc() in none(0) ID
1 981 4 malloc() in none(0) xsd
2 971 3 malloc() in none(0) ID
3 968 4 malloc() in none(0) xsd
4 875 9 malloc() in none(0) dateTime
5 872 4 malloc() in none(0) xsd
6 861 9 malloc() in none(0) dateTime
7 858 4 malloc() in none(0) xsd
8 770 8 malloc() in none(0) decimal
9 767 4 malloc() in none(0) xsd
10756 8 malloc() in none(0) decimal
11753 4 malloc() in none(0) xsd
12677 7 malloc() in none(0) string
13674 4 malloc() in none(0) xsd
14663 7 malloc() in none(0) string
15660 4 malloc() in none(0) xsd
16584 7 malloc() in none(0) string
17581 4 malloc() in none(0) xsd
18570 7 malloc() in none(0) string
19567 4 malloc() in none(0) xsd
I did this with the M1 release of Tuscany SDO C++ and libxml2 2.6.26.
The good news is that this leak has decreased a lot from an earlier
release he tried previously.
I hope this test seems valid to you. If so, any chance of removing the
remaining leaks? Even better, could this kind of testing be incorporated
in the process?
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: [SDO C++] Leaking libxml memory
Ah yes. I have a fix for this. It is caused by the SDOXMLString::substring
method creating 2 copies of the string and only freeing one of them. this
method is called when parsing the QName string in type=fred:joe so we end
up with an extra fred and joe when this is called.
I will raise a Jira and put the fix in.
Cheers,
On 11/10/06, Caroline Maynard [EMAIL PROTECTED] wrote:
One of our PHP users has reported a problem with leaking memory from
libxml2. I started to investigate, but realised that the issue is
independent of PHP, and can be reproduced in a standalone Tuscany
environment. The issue is that memory allocated by libxml2 is not being
freed, so he is seeing a memory leak growing with each invocation.
For example, if we take a small schema:
?xml version=1.0?
xsd:schema xmlns:xsd=http://www.w3.org/2001/XMLSchema;
xsd:element name=courses
xsd:complexType
xsd:sequence
xsd:element name=course minOccurs=0
maxOccurs=unbounded
xsd:complexType
xsd:sequence
xsd:element name=title type=xsd:string/
xsd:element name=description
type=xsd:string/
xsd:element name=credits
type=xsd:decimal/
xsd:element name=lastmodified
type=xsd:dateTime minOccurs=1/
/xsd:sequence
xsd:attribute name=cid type=xsd:ID/
/xsd:complexType
/xsd:element
/xsd:sequence
/xsd:complexType
/xsd:element
/xsd:schema
and a trivial function to load it:
void load_schema(char *name)
{
DataFactoryPtr mdg = DataFactory::getDataFactory();
XSDHelperPtr xsh = HelperProvider::getXSDHelper(mdg);
xsh-defineFile(name);
xsh = NULL;
mdg = NULL;
}
(I added the last two lines to try to encourage the reference-counting
pointers to do their work, but they made no difference to the outcome)
then use libxml features to check the memory usage:
int main (int argc, char** argv)
{
xmlInitParser();
load_schema(argv[1]);
xmlCleanupParser();
xmlMemoryDump();
return 0;
}
(note: in order to use these libxml functions, you must recompile with
debug=yes mem_debug=yes)
We see the following output in libxml's .memdump file:
MEMORY ALLOCATED : 108, MAX was 13687
BLOCK NUMBER SIZE TYPE
0 984 3 malloc() in none(0) ID
1 981 4 malloc() in none(0) xsd
2 971 3 malloc() in none(0) ID
3 968 4 malloc() in none(0) xsd
4 875 9 malloc() in none(0) dateTime
5 872 4 malloc() in none(0) xsd
6 861 9 malloc() in none(0) dateTime
7 858 4 malloc() in none(0) xsd
8 770 8 malloc() in none(0) decimal
9 767 4 malloc() in none(0) xsd
10756 8 malloc() in none(0) decimal
11753 4 malloc() in none(0) xsd
12677 7 malloc() in none(0) string
13674 4 malloc() in none(0) xsd
14663 7 malloc() in none(0) string
15660 4 malloc() in none(0) xsd
16584 7 malloc() in none(0) string
17581 4 malloc() in none(0) xsd
18570 7 malloc() in none(0) string
19567 4 malloc() in none(0) xsd
I did this with the M1 release of Tuscany SDO C++ and libxml2 2.6.26.
The good news is that this leak has decreased a lot from an earlier
release he tried previously.
I hope this test seems valid to you. If so, any chance of removing the
remaining leaks? Even better, could this kind of testing be incorporated
in the process?
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Pete
Re: [SDO C++] Leaking libxml memory
fixed - http://issues.apache.org/jira/browse/TUSCANY-825
On 11/10/06, Pete Robbins [EMAIL PROTECTED] wrote:
Ah yes. I have a fix for this. It is caused by the SDOXMLString::substring
method creating 2 copies of the string and only freeing one of them. this
method is called when parsing the QName string in type=fred:joe so we end
up with an extra fred and joe when this is called.
I will raise a Jira and put the fix in.
Cheers,
On 11/10/06, Caroline Maynard [EMAIL PROTECTED] wrote:
One of our PHP users has reported a problem with leaking memory from
libxml2. I started to investigate, but realised that the issue is
independent of PHP, and can be reproduced in a standalone Tuscany
environment. The issue is that memory allocated by libxml2 is not being
freed, so he is seeing a memory leak growing with each invocation.
For example, if we take a small schema:
?xml version=1.0?
xsd:schema xmlns:xsd= http://www.w3.org/2001/XMLSchema;
xsd:element name=courses
xsd:complexType
xsd:sequence
xsd:element name=course minOccurs=0
maxOccurs=unbounded
xsd:complexType
xsd:sequence
xsd:element name=title type=xsd:string/
xsd:element name=description
type=xsd:string/
xsd:element name=credits
type=xsd:decimal/
xsd:element name=lastmodified
type=xsd:dateTime minOccurs=1/
/xsd:sequence
xsd:attribute name=cid type=xsd:ID/
/xsd:complexType
/xsd:element
/xsd:sequence
/xsd:complexType
/xsd:element
/xsd:schema
and a trivial function to load it:
void load_schema(char *name)
{
DataFactoryPtr mdg = DataFactory::getDataFactory();
XSDHelperPtr xsh = HelperProvider::getXSDHelper(mdg);
xsh-defineFile(name);
xsh = NULL;
mdg = NULL;
}
(I added the last two lines to try to encourage the reference-counting
pointers to do their work, but they made no difference to the outcome)
then use libxml features to check the memory usage:
int main (int argc, char** argv)
{
xmlInitParser();
load_schema(argv[1]);
xmlCleanupParser();
xmlMemoryDump();
return 0;
}
(note: in order to use these libxml functions, you must recompile with
debug=yes mem_debug=yes)
We see the following output in libxml's .memdump file:
MEMORY ALLOCATED : 108, MAX was 13687
BLOCK NUMBER SIZE TYPE
0 984 3 malloc() in none(0) ID
1 981 4 malloc() in none(0) xsd
2 971 3 malloc() in none(0) ID
3 968 4 malloc() in none(0) xsd
4 875 9 malloc() in none(0) dateTime
5 872 4 malloc() in none(0) xsd
6 861 9 malloc() in none(0) dateTime
7 858 4 malloc() in none(0) xsd
8 770 8 malloc() in none(0) decimal
9 767 4 malloc() in none(0) xsd
10756 8 malloc() in none(0) decimal
11753 4 malloc() in none(0) xsd
12677 7 malloc() in none(0) string
13674 4 malloc() in none(0) xsd
14663 7 malloc() in none(0) string
15660 4 malloc() in none(0) xsd
16584 7 malloc() in none(0) string
17581 4 malloc() in none(0) xsd
18570 7 malloc() in none(0) string
19567 4 malloc() in none(0) xsd
I did this with the M1 release of Tuscany SDO C++ and libxml2 2.6.26.
The good news is that this leak has decreased a lot from an earlier
release he tried previously.
I hope this test seems valid to you. If so, any chance of removing the
remaining leaks? Even better, could this kind of testing be incorporated
in the process?
-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--
Pete
--
Pete
