Thanks Ian, will try. I added the -server flag earlier today so I'll see if it helps. I've used the JMX tools (VisualVM) and running GC manually seems to drop back down to a base level (or close).
When tomcat / web app is idle, the heap usage seems to be a saw tooth pattern, which isn't a surprise. The test takes a while to run so have to wait and see if -server helps. I'm using JVM 6. Thanks! On Dec 17, 2012, at 5:16 PM, Ian Boston wrote: > Hi Morrell, > Sounds suspiciously like garbage collection activity. Rather than > attaching a profiler which may impact heap usage, try attaching a JMX > console which should be less impact (eg JConsole) and looking at the > graph of Garbage Collection activity during the slow down. > > If you see it taking up a significant amount of time and CPU, try > changing the type of garbage collector[1]. IIRC the default without > the -server flag is a serial GC. Increasing heap will lengthen the > time between these slowdowns but won't eliminate them. > > Also, you may be able to recreate the slowdown by forcing a full GC > operation from JConsole. Obviously if a full GC operation doesn't > leave the JVM with significantly more free heap, then you need to up > the limit to prevent the JVM spending all its time trying to free > more. > > You didn't say which JVM you are using, the link below is for 6. > Adjust appropriately. > > HTH > Ian > > > 1 > http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#available_collectors > > On 18 December 2012 05:14, Morrell Jacobs <[email protected]> wrote: >> We're running MySQL 5.5.19 for Linux >> >> We had the blobs in the database because of clustering. Since we've moved >> to single stack, I suppose we could pull them out - not a long term >> solution, but we could try and see if it helps. >> >> I'll have our MySQL experts review the settings, the restructuring may be >> accounting for some of the slow down. If we were seeing delays from MySQL, >> I wouldn't expect Tomcat's CPU usage to be pegged though - I would think all >> the threads would be idle, waiting for a response. Does that sound right to >> you? >> >> Thanks >> Morrell >> >> >> On Dec 17, 2012, at 12:48 PM, Ron Wheeler wrote: >> >>> I would play with the JVM memory configuration. >>> It still looks very small. >>> Try doubling it and see if it changes anything. >>> If not, you can look elsewhere and if it does, try making it larger until >>> it stops improving things. >>> >>> What version of MySQL are you running? >>> Have you tuned your MySQL? >>> You might want to look at how MySQL is going to treat these objects and >>> what configuration parameters are recommended when you have large blobs. >>> Have you thought about getting your binary content out of the database. >>> Databases are not very good at 100Mb blobs. >>> Links to 100Mb blobs/files work much better. >>> >>> Is there a compelling reason to put the binary data inside a database? >>> Anytime MySQL has to restructure its indexes or physical storage, you are >>> going to see a pause in the action. >>> >>> Ron >>> >>> On 17/12/2012 12:04 PM, Morrell Jacobs wrote: >>>> Hello all, >>>> >>>> We're currently experience a problem where JackRabbit function will >>>> occasionally slow down; according to our QA, the slow down comes in waves: >>>> good performance for a while (hours) then bad for a while (minutes / >>>> hours). The slow down is on the order of minutes to perform create or >>>> modify an object (creation involves creating a 2-3 levels of nodes, and >>>> setting properties). Normally these operations take 100 - 300 ms, but >>>> during the slow down they can take as much 5 minutes. >>>> >>>> The slow down does not seem to be connected to the load on the server: >>>> we've seen it perform well, when many people are connecting, then slow >>>> down with only one user. I suspect the slow down is related to indexing, >>>> but that's just a guess on my part - I'm looking for something that would >>>> happen at some interval. >>>> >>>> >>>> Environment: >>>> * JackRabbit is built into war running in Tomcat (was 7.0.25, recently >>>> upgraded to 7.0.34) >>>> * Tomcat was running with default configs but recently up'd memory - >>>> -Xms128M -Xmx512M -XX:MaxPermSize=256M >>>> * Repository XML is at end of email >>>> * JackRabbit was originally configured to be clustered, but is now running >>>> as just a single instance. >>>> >>>> >>>> Data: >>>> * Our data is structured similar to a file system, except any object can >>>> be both a file and contain other objects (files); each object has various >>>> metadata (some in properties, some in child nodes), a file and child >>>> objects >>>> * We don't use full text searching and don't want file (binary) contents >>>> indexed; the only way I was able to prevent the search index from >>>> examining the files (some are large 100+ M) was to create my own node >>>> types for files. My node types are identical nt:file and nt:content, >>>> except that they are named prd:file and prd:content; the unrecognized name >>>> seems to prevent indexing from touch them. >>>> >>>> >>>> When I've attempted to use profiling tools, it appears that during the >>>> slow down the CPU is pegged; memory usage is also at or close to the >>>> limit. In depth profiling tools (NetBeans) get overwhelmed during the >>>> slow down. >>>> >>>> We're going thru a variety of tests, but any advice the community can >>>> provide will be greatly appreciated. >>>> >>>> Here's the repository.xml: >>>> <Repository> >>>> <FileSystem class="org.apache.jackrabbit.core.fs.db.DbFileSystem"> >>>> <param name="driver" value="com.mysql.jdbc.Driver"/> >>>> <param name="url" >>>> value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}global_repository?autoReconnect=true&createDatabaseIfNotExist=true"/> >>>> <param name="schema" value="mysql"/> >>>> <param name="schemaObjectPrefix" value="rep_"/> >>>> <param name="user" value="${pm.database.user}"/> >>>> <param name="password" value="${pm.database.pwd}"/> >>>> </FileSystem> >>>> >>>> <!-- >>>> security configuration >>>> --> >>>> <Security appName="Jackrabbit"> >>>> <!-- >>>> security manager: >>>> class: FQN of class implementing the JackrabbitSecurityManager >>>> interface >>>> --> >>>> <SecurityManager >>>> class="org.apache.jackrabbit.core.DefaultSecurityManager" >>>> workspaceName="default"> >>>> <!-- >>>> workspace access: >>>> class: FQN of class implementing the WorkspaceAccessManager >>>> interface >>>> --> >>>> <!-- <WorkspaceAccessManager class="..."/> --> >>>> <!-- <param name="config" value="${rep.home}/security.xml"/> --> >>>> </SecurityManager> >>>> <!-- >>>> access manager: >>>> class: FQN of class implementing the AccessManager interface >>>> --> >>>> <AccessManager >>>> class="org.apache.jackrabbit.core.security.DefaultAccessManager"> >>>> <!-- <param name="config" value="${rep.home}/access.xml"/> --> >>>> </AccessManager> >>>> <LoginModule >>>> class="org.apache.jackrabbit.core.security.authentication.DefaultLoginModule"> >>>> <!-- >>>> anonymous user name ('anonymous' is the default value) >>>> --> >>>> <param name="anonymousId" value="anonymous"/> >>>> <!-- >>>> administrator user id (default value if param is missing is >>>> 'admin') >>>> --> >>>> <param name="adminId" value="admin"/> >>>> </LoginModule> >>>> </Security> >>>> <!-- >>>> location of workspaces root directory and name of default workspace >>>> --> >>>> <Workspaces rootPath="${rep.home}/workspaces" >>>> defaultWorkspace="default"/> >>>> <!-- >>>> workspace configuration template: >>>> used to create the initial workspace if there's no workspace yet >>>> --> >>>> <!-- <Workspace name="default">--> >>>> <Workspace name="default"> >>>> <!-- >>>> virtual file system of the workspace: >>>> class: FQN of class implementing the FileSystem interface >>>> --> >>>> <FileSystem >>>> class="org.apache.jackrabbit.core.fs.local.LocalFileSystem"> >>>> <param name="path" value="${wsp.home}"/> >>>> </FileSystem> >>>> <!-- >>>> persistence manager of the workspace: >>>> class: FQN of class implementing the PersistenceManager >>>> interface >>>> --> >>>> <PersistenceManager >>>> class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager"> >>>> <param name="driver" value="com.mysql.jdbc.Driver"/> >>>> <param name="url" >>>> value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}product?createDatabaseIfNotExist=true"/> >>>> <param name="user" value="${pm.database.user}"/> >>>> <param name="password" value="${pm.database.pwd}"/> >>>> <param name="schemaObjectPrefix" value="Product_"/> >>>> <param name="schema" value="mysql"/> >>>> </PersistenceManager> >>>> <!-- >>>> Search index and the file system it uses. >>>> class: FQN of class implementing the QueryHandler interface >>>> --> >>>> <SearchIndex >>>> class="org.apache.jackrabbit.core.query.lucene.SearchIndex"> >>>> <param name="path" value="${wsp.home}/index"/> >>>> <!--<param name="textFilterClasses" >>>> >>>> value="org.apache.jackrabbit.extractor.PlainTextExtractor,org.apache.jackrabbit.extractor.MsWordTextExtractor,org.apache.jackrabbit.extractor.MsExcelTextExtractor,org.apache.jackrabbit.extractor.MsPowerPointTextExtractor,org.apache.jackrabbit.extractor.PdfTextExtractor,org.apache.jackrabbit.extractor.OpenOfficeTextExtractor,org.apache.jackrabbit.extractor.RTFTextExtractor,org.apache.jackrabbit.extractor.HTMLTextExtractor,org.apache.jackrabbit.extractor.XMLTextExtractor"/> >>>> <param name="extractorPoolSize" value="2"/> >>>> <param name="supportHighlighting" value="true"/> --> >>>> </SearchIndex> >>>> </Workspace> >>>> <!-- >>>> Configures the versioning >>>> --> >>>> <Versioning rootPath="${rep.home}/version"> >>>> <!-- >>>> Configures the filesystem to use for versioning for the >>>> respective >>>> persistence manager >>>> --> >>>> <FileSystem >>>> class="org.apache.jackrabbit.core.fs.local.LocalFileSystem"> >>>> <param name="path" value="${rep.home}/version"/> >>>> </FileSystem> >>>> <!-- >>>> Configures the persistence manager to be used for persisting >>>> version state. >>>> Please note that the current versioning implementation is based >>>> on >>>> a 'normal' persistence manager, but this could change in future >>>> implementations. >>>> --> >>>> <PersistenceManager >>>> class="org.apache.jackrabbit.core.persistence.pool.MySqlPersistenceManager"> >>>> <param name="driver" value="com.mysql.jdbc.Driver"/> >>>> <param name="url" >>>> value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}truedit_versions?createDatabaseIfNotExist=true"/> >>>> <param name="user" value="${pm.database.user}"/> >>>> <param name="password" value="${pm.database.pwd}"/> >>>> <param name="schemaObjectPrefix" value="version_"/> >>>> <param name="schema" value="mysql"/> >>>> </PersistenceManager> >>>> </Versioning> >>>> <!-- >>>> Configures the Data Store for large binary objects. >>>> --> >>>> <DataStore class="org.apache.jackrabbit.core.data.FileDataStore"> >>>> <param name="path" value="${datastore.location.path}"/> >>>> <param name="minRecordLength" value="100"/> >>>> </DataStore> >>>> <!-- >>>> <Cluster id="${cluster.id}" syncDelay="2000"> >>>> <Journal class="org.apache.jackrabbit.core.journal.DatabaseJournal"> >>>> <param name="driver" value="com.mysql.jdbc.Driver" /> >>>> <param name="url" >>>> value="jdbc:mysql://${pm.database.url}:${pm.database.port}/${pm.database.prefix}journal?autoReconnect=true&createDatabaseIfNotExist=true"/> >>>> <param name="schemaObjectPrefix" value="journal_"/> >>>> <param name="databaseType" value="mysql"/> >>>> <param name="user" value="${pm.database.user}"/> >>>> <param name="password" value="${pm.database.pwd}"/> >>>> </Journal> >>>> </Cluster> >>>> --> >>>> >>>> </Repository> >>>> >>>> >>>> >>>> -- >>>> Morrell Jacobs >>>> Chief Software Architect >>>> MEI >>>> 610 Old York Road, Suite 250 >>>> Jenkintown, PA 19046 >>>> Phone: 215-886-5662, ext. 252 >>>> Fax: 215-886-5681 >>>> http://www.maned.com >>>> E-mail: [email protected] >>>> AOL IM: MorrellMEI >>>> >>>> Have you seen Nervous Pixel, MEI's creative services division? >>>> www.nervouspixel.com >>>> >>>> >>>> >>>> >>> >>> >>> -- >>> Ron Wheeler >>> President >>> Artifact Software Inc >>> email: [email protected] >>> skype: ronaldmwheeler >>> phone: 866-970-2435, ext 102 >>> >>
