Memory Leak in GrammarResolver class
------------------------------------
Key: XERCESC-1381
URL: http://issues.apache.org/jira/browse/XERCESC-1381
Project: Xerces-C++
Type: Bug
Components: Validating Parser (Schema) (Xerces 1.5 or up only)
Versions: 2.6.0
Environment: all
Reporter: Christian Will
Attachments: GrammarResolver.cpp.patch
Hi,
the grammar resolver class produces memory leaks under some circumstances. If
we set the features fgXercesUseCachedGrammarInParse,
fgXercesCacheGrammarFromParse and lock the grammar pool all new created
grammars will never be deleted.
Here is my first example. I create a parser, cache the grammar, lock the
grammar pool and parse one instance document. I delete the parser object,
unlock and delete the grammar pool, and terminate the platform utils. After all
objects are deleted we can see that still 3 kbytes are allocated from the
grammar pool memory manager.
I could locate all allocations to [\internal\IGXMLScanner2.cpp(1146)].
=> "fDTDGrammar = new (fGrammarPoolMemoryManager)
DTDGrammar(fGrammarPoolMemoryManager);"
Sample Log 1
------------
init...
parser1 set feature
SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
cache grammar... (T:/testfiles/_FIXML/schema/FIXML-main-4-4.xsd)
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (5)
GrammarPool : 4239962 [ 12829334]bytes, 62711 [117134, 70374]blocks (8)
lock grammarpool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 26456 [ 308510]bytes, 101 [ 2885, 4064]blocks (0)
GrammarPool : 5501456 [ 12829334]bytes, 97049 [151874, 70776]blocks (0)
---------------------------------------------------------------------------
parse with parser1... (T:/testfiles/_FIXML/order100.xml)
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 50846 [ 308510]bytes, 517 [ 3395, 4370]blocks (40)
GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (4)
---------------------------------------------------------------------------
delete parser1...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 5504580 [ 12829334]bytes, 97064 [151889, 70776]blocks (0)
unlock grammarPool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 4243086 [ 12829334]bytes, 62726 [151889,105114]blocks (0)
delete grammarPool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
terminate PlatformUtils...
GrammarPool : 3020 [ 12829334]bytes, 12 [151889,176419]blocks (0)
done.
==-still-allocated-memory-=================================================
pointer: 3df83c0 size: 44bytes sequence: 151875
T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
T:\xerces_cvs\src\xercesc\internal\XMLScanner.cpp(434)
pointer: 3df86d0 size: 32bytes sequence: 151876
T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df87d8 size: 436bytes sequence: 151877
..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df8a70 size: 512bytes sequence: 151878
..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(169)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df8d58 size: 32bytes sequence: 151879
T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df8e60 size: 436bytes sequence: 151880
..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df90f8 size: 512bytes sequence: 151881
..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(173)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df93e0 size: 32bytes sequence: 151882
T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df94e8 size: 436bytes sequence: 151883
..\..\..\..\..\src\xercesc/util/NameIdPool.c(122)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df9780 size: 512bytes sequence: 151884
..\..\..\..\..\src\xercesc/util/NameIdPool.c(136)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(174)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df9a68 size: 24bytes sequence: 151885
T:\xerces_cvs\src\xercesc\util\XMemory.cpp(66)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner.cpp(188)
pointer: 3df9b60 size: 12bytes sequence: 151886
..\..\..\..\..\src\xercesc/util/XMLString.hpp(1667)
T:\xerces_cvs\src\xercesc\validators\DTD\XMLDTDDescriptionImpl.cpp(59)
T:\xerces_cvs\src\xercesc\validators\DTD\DTDGrammar.cpp(177)
T:\xerces_cvs\src\xercesc\internal\IGXMLScanner2.cpp(1146)
==-still-allocated-memory-=================================================
My second example shows the impact a little bit better. I lock the grammar pool
without loading the grammar. I parse the instance document 3 times. And after I
delete all objects and terminate the platform utils I can see that the
associated grammar is still 3 times (11.5MB) in memory.
Sample Log 2
------------
init...
parser1 set feature
SchemaFullChecking,UseCachedGrammarInParse,CacheGrammarFromParse
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
GrammarPool : 1214 [ 1214]bytes, 20 [ 20, 0]blocks (0)
lock grammarpool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 26456 [ 26456]bytes, 101 [ 101, 0]blocks (0)
GrammarPool : 23934 [ 24098]bytes, 597 [ 629, 32]blocks (0)
---------------------------------------------------------------------------
parse with parser1... (T:/testfiles/_FIXML/order100.xml)
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 50862 [ 486594]bytes, 517 [ 3392, 4374]blocks (40)
GrammarPool : 4265794 [ 12855178]bytes, 63302 [117757, 70406]blocks (5)
parse with parser1... (T:/testfiles/_FIXML/order100.xml)
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 52466 [ 497174]bytes, 586 [ 6317, 8535]blocks (65)
GrammarPool : 8120878 [ 16710262]bytes,115564 [224426,140764]blocks (0)
parse with parser1... (T:/testfiles/_FIXML/order100.xml)
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
Parser One : 54586 [ 498778]bytes, 655 [ 9241, 12695]blocks (144)
GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
---------------------------------------------------------------------------
delete parser1...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 11975962 [ 20565346]bytes,167826 [331095,211122]blocks (0)
unlock grammarPool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 11953242 [ 20565346]bytes,167249 [331095,211699]blocks (0)
delete grammarPool...
PlatformUtils: 172842 [ 172842]bytes, 2855 [ 3918, 1162]blocks (0)
GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
terminate PlatformUtils...
GrammarPool : 11565252 [ 20565346]bytes,156786 [331095,222162]blocks (0)
Let me try to explain why this is happening. In GrammarResolver::putGrammar()
can we see that we put our grammar in the table 'fGrammarBucket' if we don't
want to cache our grammar or the grammar pool doesn't want cache our grammar
(because it is locked).
void GrammarResolver::putGrammar(Grammar* const grammarToAdopt)
{
...
if (!fCacheGrammar || !fGrammarPool->cacheGrammar(grammarToAdopt))
{
// either we aren't caching or the grammar pool doesn't want it
// so we need to look after it
fGrammarBucket->put( (void*)
grammarToAdopt->getGrammarDescription()->getGrammarKey(), grammarToAdopt );
if (grammarToAdopt->getGrammarType() == Grammar::SchemaGrammarType)
{
fGrammarsToAddToXSModel->addElement((SchemaGrammar*)
grammarToAdopt);
}
}
}
So far so good. We have all grammars that we don't want to cache or/and can't
be cached in this table. So normaly if it deletes the table fGrammarBucket all
grammars will be deleted. But if the feature fgXercesCacheGrammarFromParse is
set it changes the behavior of the table to not adopte the elements. What we
can see in GrammarResolver::cacheGrammarFromParse().
void GrammarResolver::cacheGrammarFromParse(const bool aValue)
{
reset();
fCacheGrammar = aValue;
fGrammarBucket->setAdoptElements(!fCacheGrammar);
}
I think that is wrong to change the property of our table to not adopt the
elements. Whenever we remove an grammar from our list, for example to cache to
grammar, we call fGrammarBucket->orphanKey(grammarKey), and that means already
that we remove the grammar wihtout deleting it.
I removed this line "fGrammarBucket->setAdoptElements(!fCacheGrammar);" made
some tests and everything looks good.
But I'm not sure if there are no side effects. And that is the reason why I
would like to get a second opinion from somebody. I can also send parts of my
code If you have problems to reproduce the bug.
Regards,
Christian Will
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
If you want more information on JIRA, or have a bug to report see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
