Re: [Wicket-develop] Memory profiling

Mon, 22 Aug 2005 11:24:04 -0700 

but this is perfectly possible
just compare the 2 snapshots
and you are in a view with all the differences.
you can see how many instances of one class where made. How many where cleanup. 

johan


Gili wrote:



    I mean I want to see which new objects were created since my last 
snapshot. The latest JProfiler uses JDK 5 "dynamic instrumentation" so 
it profiles on demand, and is fast.


Gili

Johan Compagner wrote:

What do you mean with which objects?
You can see the type of the object how many there where added

And if you want you can go to the instance count and see who has 
references to them



The last time i checked out JProfiler (now for some time quite happy 
with yourkit) it was always the same
You directly start the profiling (when you start the app) the app is 
immediately very slow.



With yourkit you can just attach the dll in and only trigger the 
profile when ever you want.

And then most profiles you even won't notice.

johan



Gili wrote:




    Have you tried JProfiler recently? I'm just curious what you 
thought of one versus the other.



    I took a quick look at YourKit's feature set. Memory profiling 
wise, it doesn't seem to have more functionality than JProfiler 
(i.e. I can't see *which* objects were added across the two 
snapshots, simply *how many* were added). I think you can achieve 
this indirectly in JProfiler by sorting the live objects by time of 
creation and then look at the most recently created objects; though 
this isn't very clean in my books.


    Anyway, I'll keep on trying.

Gili

Johan Compagner wrote:


www.yourkit.com
Works perfectly the best i can find and i have tried many

Gili wrote:




    Right... I made sure to force garbage collection before taking 
this snapshot. I'm using JProfiler and it doesn't have such a 
great capability of taking two snapshots and comparing them. It 
lets you compare the number of allocated objects between two 
snapshots but not the actual objects that differ, etc.



    Still, I looked at the Tomcat code so I know for sure that it 
does not release its resources. Very annoying that :( What makes 
it worse is that Remy, who is in charge of Tomcat BugZilla, closes 
everyone's bugs as WON'T FIX or INVALID all the time even when 
they make perfect sense. Like for example, Tomcat has a 
cacheMaxSize and cacheTTL setting, but WebappClassLoader ignores 
them. Clearly this is a bug, but he closed it as INVALID without 
an explanation.


Gili

Johan Compagner wrote:


ehmmm

i could read this wrong of course. But this is a allocation 
hotspot view..
So course it is allocating more and more. What i see there is who 
is making Objects in memory
And yes resource loaders are making objects. But they will be 
cleaned up.



The question is what is being held in mem so you should compare 2 
memory snapshots.


One where every thing is warmed up. All images are served once
and then do a refresh of those pages
anohter memory snapshot

and compare those. It shouldn't have grown much...



Gili wrote:




    Upon further investigation I think Tomcat is at fault and 
there is nothing I can do to fix it :( See the attached file...



    I see a constant pattern whereby ClassLoader.getResource() 
keeps on allocating more and more memory over the lifetime of my 
webapp (even after I force garbage collection) and all the calls 
are coming out of Wicket. Now, if I understand this correctly... 
when we use ClassLoader.getResource() it returns a URL but 
Tomcat also caches the bytes behind this resource indefinately 
inside a org.apache.catalina.loader.ResourceEntry which it 
stores inside WebappClassLoader.



    It never flushes these entries which is why they keep on 
building indefinately. So coming back to the original point, I'm 
no longer sure whether resolving the different locales to a 
single PackageResource would make a difference or not. But I 
*do* keep on seeing this allocation grow (to over 10MB after 30 
minutes) so it's worrying me.


Gili

Johan Compagner wrote:


Those resourceStreams of a PackageResource aren't holding the 
complete size of youre stream?
It is just a wrapper around a UrlResourceStream where the real 
resource is closed after streaming it.

So we arenot storing an image in mem. We are just streaming it.


What you keep in mem for youre DynamicImages (in there byte 
buffer) is up to you


johan




Gili wrote:


Hi,


    I'm doing memory profiling on my webapp and it looks to me 
like we allocate a new ResourceStream per locale of the same 
packaged resource. The problem is that if I have a larged 
image exposed through a PackageResource and it is hit from 5 
different locales, we store the exact same image in memory 5 
times.



    Isn't there a way for us to resolve these requests to the 
same image? They are the same file in the underlying 
file-system. Taking this one step further, there should be a 
way for me to signal to Wicket that even my DynamicImages are 
locale-independant. Again, at 500k-1MB per image, that's a lot 
of wasted memory (and I get hits from many different locales 
on my website). Can anyone help me with this optimization?


Thanks,
Gili







-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO

September 19-22, 2005 * San Francisco, CA * Development 
Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * 
Testing & QA
Security * Process Improvement & Measurement * 
http://www.sqe.com/bsce5sf

_______________________________________________
Wicket-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/wicket-develop





------------------------------------------------------------------------ 




    Allocation hot spots

*Session:*     Apache Tomcat 5.x (with tomcat5.exe) on localhost
*Time of export:*     Sunday, August 21, 2005 3:03:54 PM EDT
*JVM time:*     16:17
       *Garbage collector mode: *     Live objects
*Allocations of: *     All classes
*Filtered classes: *     show separately


------------------------------------------------------------------------ 



      Hot spot     Allocated memory     Allocations

    org.apache.velocity.app.VelocityEngine.init      8,260 kB 
(18 %)     123,791
    java.lang.ClassLoader.getResource      5,798 kB (12 %)     
1,380
  12.2% - 5,430 kB - 795 alloc. 
wicket.util.resource.locator.ClassLoaderResourceStreamLocator.locate 
(line: 109)
  12.2% - 5,430 kB - 795 alloc. 
wicket.util.resource.locator.AbstractResourceStreamLocator.locate 
(line: 53)
  12.2% - 5,430 kB - 795 alloc. 
wicket.util.resource.locator.DefaultResourceStreamLocator$1.locate 
(line: 124)
  12.1% - 5,399 kB - 638 alloc. 
wicket.util.resource.locator.ResourceStreamLocator.locate (line: 
180)
  12.1% - 5,399 kB - 638 alloc. 
wicket.markup.html.PackageResource.getResourceStream (line: 337)
  10.2% - 4,566 kB - 602 alloc. 
wicket.protocol.http.WicketServlet.getLastModified(javax.servlet.http.HttpServletRequest) 

  1.5% - 648 kB - 12 alloc. URL: 
/Main/resources/wallpapers/space/Wonderment_by_dilekt_en_US.jpg
  1.1% - 502 kB - 12 alloc. URL: 
/Main/resources/wallpapers/space/Ethereal_Bloom_by_sumopiggy_en_US.jpg 

  1.1% - 482 kB - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/Ashundar_Mystical_Canyon_Final_by_merlin-tbd_en_US.jpg 

  1.0% - 447 kB - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/The_Silver_Star_by_Skybase_en_US.jpg 

  1.0% - 447 kB - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/Outburst_by_dilekt_en_US.jpg
  1.0% - 435 kB - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/Chasm_by_alyn_en_US.jpg
  1.0% - 434 kB - 12 alloc. URL: 
/Main/resources/wallpapers/space/final_by_active-x_en_US.jpg
  0.9% - 383 kB - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/Eternal_Origin_by_j4m3sb0nd_en_US.jpg 

  0.8% - 346 kB - 12 alloc. URL: 
/Main/resources/wallpapers/space/Glacial_Haven_by_dilekt_en_US.jpg
  0.6% - 281 kB - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/Blue_Mist_by_dilekt_en_US.jpg
  0.1% - 30 kB - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/thumbnails/The_Silver_Star_by_Skybase_en_US.jpg 

  0.0% - 18 kB - 12 alloc. URL: 
/Main/resources/shared/css/header.png
  0.0% - 8136 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/thumbnails/Outburst_by_dilekt_en_US.jpg 

  0.0% - 7344 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/space/thumbnails/Wonderment_by_dilekt_en_US.jpg 

  0.0% - 5896 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/space/thumbnails/final_by_active-x_en_US.jpg 

  0.0% - 5648 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/space/thumbnails/Fate_X_by_active-x_en_US.jpg 

  0.0% - 5128 bytes - 12 alloc. URL: 
/Main/resources/shared/css/page.css
  0.0% - 5080 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/css/download_en_US.png
  0.0% - 5040 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/thumbnails/Ashundar_Mystical_Canyon_Final_by_merlin-tbd_en_US.jpg 

  0.0% - 4864 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/css/page.css
  0.0% - 4760 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/thumbnails/Eternal_Origin_by_j4m3sb0nd_en_US.jpg 

  0.0% - 4632 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/space/thumbnails/Glacial_Haven_by_dilekt_en_US.jpg 

  0.0% - 4376 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/space/thumbnails/Sol_Scape_by_dilekt_en_US.jpg 

  0.0% - 4296 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/thumbnails/Blue_Mist_by_dilekt_en_US.jpg 

  0.0% - 4040 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/space/thumbnails/Ethereal_Bloom_by_sumopiggy_en_US.jpg 

  0.0% - 3456 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/sunset/thumbnails/Chasm_by_alyn_en_US.jpg 

  0.0% - 2920 bytes - 14 alloc. URL: 
/Main/resources/base/css/ie5/navigation-shadow-top.png
  0.0% - 1992 bytes - 12 alloc. URL: 
/Main/resources/shared/css/header-right-corner.png
  0.0% - 1944 bytes - 12 alloc. URL: 
/Main/resources/shared/css/image-shadow-bottom-right.png
  0.0% - 1904 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/css/ie5.css
  0.0% - 1880 bytes - 12 alloc. URL: 
/Main/resources/shared/css/page-shadow-bottom.png
  0.0% - 1872 bytes - 12 alloc. URL: 
/Main/resources/shared/css/ie5.css
  0.0% - 1696 bytes - 12 alloc. URL: 
/Main/resources/shared/css/image-shadow-top-right.png
  0.0% - 1656 bytes - 12 alloc. URL: 
/Main/resources/shared/css/image-shadow-bottom-left.png
  0.0% - 1488 bytes - 12 alloc. URL: 
/Main/resources/shared/css/image-shadow-top-left.png
  0.0% - 1352 bytes - 12 alloc. URL: 
/Main/resources/shared/css/css2.css
  0.0% - 1336 bytes - 12 alloc. URL: 
/Main/resources/wallpapers/css/css2.css
  0.0% - 1320 bytes - 12 alloc. URL: 
/Main/resources/shared/css/page-shadow-bottom-right.png
  0.0% - 1208 bytes - 12 alloc. URL: 
/Main/resources/base/css/ie5/navigation-shadow-bottom.png
  0.0% - 1200 bytes - 12 alloc. URL: 
/Main/resources/shared/css/ie5/navigation-gradient-bottom.png
  0.0% - 1192 bytes - 12 alloc. URL: 
/Main/resources/shared/css/page-shadow-bottom-left.png
  0.0% - 1176 bytes - 12 alloc. URL: 
/Main/resources/shared/css/header-right-padding.png
  0.0% - 1160 bytes - 12 alloc. URL: 
/Main/resources/shared/css/image-shadow-bottom.png
  0.0% - 1152 bytes - 12 alloc. URL: 
/Main/resources/shared/css/image-shadow-right.png
  0.0% - 1144 bytes - 12 alloc. URL: 
/Main/resources/shared/css/page-shadow-right.png
  0.0% - 1128 bytes - 12 alloc. URL: 
/Main/resources/shared/css/image-shadow-left.png
  0.0% - 1120 bytes - 12 alloc. URL: 
/Main/resources/shared/css/css2/gradient-bottom.png
  0.0% - 1120 bytes - 12 alloc. URL: 
/Main/resources/shared/css/image-shadow-top.png
  0.0% - 1112 bytes - 12 alloc. URL: 
/Main/resources/shared/css/page-shadow-left.png
  0.0% - 1104 bytes - 12 alloc. URL: 
/Main/resources/base/css/css2/shadow-bottom.png
  1.9% - 832 kB - 36 alloc. 
wicket.protocol.http.WicketServlet.getLastModified(javax.servlet.http.HttpServletRequest) 
(line: 162)
  0.1% - 30 kB - 157 alloc. 
wicket.util.resource.locator.ResourceStreamLocator.locate (line: 
83)
  0.3% - 116 kB - 12 alloc. 
webwork.util.ClassLoaderUtils.getResource (line: 46)
  0.2% - 108 kB - 135 alloc. 
org.ofbiz.core.util.UtilURL.fromResource (line: 49)
  0.2% - 96 kB - 156 alloc. 
com.atlassian.core.util.ClassLoaderUtils.getResource (line: 143)
  0.0% - 17 kB - 12 alloc. 
com.opensymphony.workflow.loader.XMLWorkflowFactory$WorkflowConfig.<init> 
(line: 136)
  0.0% - 8152 bytes - 34 alloc. 
webwork.util.ClassLoaderUtils.getResource (line: 35)
  0.0% - 4272 bytes - 12 alloc. 
com.opensymphony.util.ClassLoaderUtil.getResource (line: 72)
  0.0% - 4224 bytes - 24 alloc. 
com.opensymphony.util.ClassLoaderUtil.getResource (line: 70)
  0.0% - 4024 bytes - 81 alloc. 
wicket.util.resource.locator.ClassLoaderResourceStreamLocator.locate 
(line: 144)
  0.0% - 3648 bytes - 12 alloc. 
com.opensymphony.util.ClassLoaderUtil.getResource (line: 134)
  0.0% - 1704 bytes - 71 alloc. 
wicket.util.resource.locator.ClassLoaderResourceStreamLocator.locate 
(line: 169)
  0.0% - 1272 bytes - 12 alloc. 
com.opensymphony.util.ClassLoaderUtil.getResource (line: 44)
  0.0% - 1224 bytes - 12 alloc. 
com.atlassian.core.util.ClassLoaderUtils.getResource (line: 44)
  0.0% - 928 bytes - 12 alloc. 
com.opensymphony.util.ClassLoaderUtil.getResource (line: 84)
    java.io.ByteArrayOutputStream.toByteArray      4,535 kB (10 
%)     310
    org.apache.velocity.servlet.VelocityServlet.init      4,225 
kB (9 %) 62,618
    org.apache.catalina.startup.Bootstrap.main      2,947 kB (6 
%)     42,434
    java.sql.PreparedStatement.executeQuery      1,652 kB (3 
%)     81

    java.lang.Object.<init>      1,513 kB (3 %)     37,146

    
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run      
1,509 kB (3 %)     7,533

    java.lang.StringBuffer.toString      1,381 kB (3 %)     18,176

    java.lang.reflect.Constructor.newInstance      980 kB (2 
%)     18,263

    java.util.Map.put      923 kB (2 %)     23,728

    org.hibernate.type.BinaryType.deepCopyNotNull      809 kB (1 
%)     18

    java.lang.Class.forName      637 kB (1 %)     10,029
    javax.sql.DataSource.getConnection      606 kB (1 %)     9,113

    java.sql.Connection.prepareStatement      595 kB (1 %)     
3,123
    java.lang.Class.getDeclaredConstructors      591 kB (1 
%)     8,700

    org.apache.velocity.Template.merge      489 kB (1 %)     11,521
    java.util.Properties.load      332 kB (0 %)     7,372
    java.lang.reflect.Method.invoke      332 kB (0 %)     6,275

    
javax.xml.parsers.DocumentBuilder.parse(java.io.InputStream)      
322 kB (0 %)     4,815

    java.lang.ClassLoader.loadClass      319 kB (0 %)     4,685

    
java.awt.GraphicsEnvironment.getAvailableFontFamilyNames      
297 kB (0 %)     9,306
    org.apache.commons.lang.StringUtils.split      262 kB (0 
%)     10

    org.dom4j.io.SAXReader.read      188 kB (0 %)     5,440

    java.util.ResourceBundle.getBundle(java.lang.String)      
181 kB (0 %) 3,592

    java.lang.Integer.<init>      180 kB (0 %)     11,298

    javax.servlet.http.HttpServletRequest.getReader      174 kB 
(0 %)     86

    java.io.OutputStream.write      163 kB (0 %)     6

    java.awt.image.BufferedImage.createGraphics      135 kB (0 
%)     3,362
    
desktopbeautifier.server.database.AbstractCachedImage$$EnhancerByCGLIB$$125e308b.newInstance 
     127 kB (0 %)     3,982

    java.util.HashMap.<init>()      127 kB (0 %)     2,136

    java.util.ResourceBundle.getBundle(java.lang.String, 
java.util.Locale, java.lang.ClassLoader)      125 kB (0 %)     69
    java.sql.PreparedStatement.executeUpdate      122 kB (0 
%)     10

    java.util.ArrayList.<init>()      120 kB (0 %)     3,025

    org.apache.log4j.Category.getInstance      112 kB (0 %)     
1,962
    javax.servlet.http.HttpServletResponse.getWriter      102 kB 
(0 %)     132

    java.lang.Class.newInstance      96 kB (0 %)     2,104
    java.lang.Class.getDeclaredMethod      91 kB (0 %)     1,602

    
javax.xml.parsers.DocumentBuilder.parse(java.lang.String)      
88 kB (0 %)     2,162

    java.util.Iterator.next      79 kB (0 %)     233
    java.beans.Introspector.getBeanInfo      77 kB (0 %)     2,149
    java.lang.Class.getDeclaredMethods      77 kB (0 %)     1,477

    java.lang.ClassLoader.getResourceAsStream      67 kB (0 
%)     176

    java.sql.ResultSet.getString      66 kB (0 %)     1,015

    
org.hibernate.persister.entity.BasicEntityPersister.hydrate      
64 kB (0 %)     2,009

    java.lang.String.getBytes      64 kB (0 %)     2,169

    
javax.xml.parsers.DocumentBuilderFactory.newDocumentBuilder      
57 kB (0 %)     977
    org.apache.commons.logging.LogFactory.getLog      56 kB (0 
%)     1,617
    javax.imageio.ImageIO.createImageInputStream      52 kB (0 
%)     408

    java.util.Hashtable.put      49 kB (0 %)     1,663

    org.hibernate.sql.Alias.toUnquotedAliasStrings      49 kB (0 
%)     3,063

    java.lang.Class.getMethod      48 kB (0 %)     996
    java.util.HashSet.<init>      45 kB (0 %)     1,000
    java.util.HashMap.<init>(int)      39 kB (0 %)     807

    
desktopbeautifier.server.database.Image$$EnhancerByCGLIB$$94713165.newInstance 
     38 kB (0 %)     1,191
    java.security.MessageDigest.getInstance      37 kB (0 %)     
883
    net.jforum.dao.generic.DataAccessDriver.<clinit>      37 kB 
(0 %)     580

    antlr.ANTLRHashString.<init>      35 kB (0 %)     1,488

    org.apache.log4j.xml.DOMConfigurator.configure      34 kB (0 
%)     429
    javax.servlet.jsp.JspFactory.getPageContext      33 kB (0 
%)     38
    net.jforum.ConfigLoader.startCacheEngine      32 kB (0 
%)     528

    java.lang.Float.<init>      31 kB (0 %)     1,990

    javax.activation.MimetypesFileTypeMap.<init>      30 kB (0 
%)     962
    org.hibernate.loader.OuterJoinLoader.initPersisters      30 
kB (0 %)     1,896

    javax.servlet.FilterChain.doFilter      29 kB (0 %)     610
    java.net.URLConnection.getLastModified      26 kB (0 %)     622

    
desktopbeautifier.clientserver.DefaultImageSpecification.<init>      
26 kB (0 %)     856
    
org.apache.velocity.tools.view.servlet.ServletToolboxManager.getInstance 
     26 kB (0 %)     538

    java.io.PrintWriter.<init>      24 kB (0 %)     11
    java.io.File.<init>      24 kB (0 %)     245
    java.util.regex.Pattern.compile      24 kB (0 %)     564

    java.net.URLConnection.getContentLength      23 kB (0 %)     
605
    URL: /download/DesktopBeautifier-Setup.exe      23 kB (0 
%)     103

    java.util.ArrayList.<init>(int)      23 kB (0 %)     592

    java.lang.reflect.Proxy.newProxyInstance      23 kB (0 
%)     512

    org.objectweb.asm.Label.<init>      22 kB (0 %)     469




-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO

September 19-22, 2005 * San Francisco, CA * Development Lifecycle 
Practices
Agile & Plan-Driven Development * Managing Projects & Teams * 
Testing & QA
Security * Process Improvement & Measurement * 
http://www.sqe.com/bsce5sf

_______________________________________________
Wicket-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/wicket-develop






-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO

September 19-22, 2005 * San Francisco, CA * Development Lifecycle 
Practices
Agile & Plan-Driven Development * Managing Projects & Teams * 
Testing & QA
Security * Process Improvement & Measurement * 
http://www.sqe.com/bsce5sf

_______________________________________________
Wicket-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/wicket-develop






-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO

September 19-22, 2005 * San Francisco, CA * Development Lifecycle 
Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing 
& QA
Security * Process Improvement & Measurement * 
http://www.sqe.com/bsce5sf

_______________________________________________
Wicket-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/wicket-develop






-------------------------------------------------------
SF.Net email is Sponsored by the Better Software Conference & EXPO
September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
_______________________________________________
Wicket-develop mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/wicket-develop














    











					Reply via email to