On 7 Jun 2012, at 23:03, Daniel Mikusa <dmik...@vmware.com> wrote: > ----- Original Message ----- >> Only 52 java threads. It used to fluctuate more (we made some >> changes >> to the app to perform a task in a single thread rather than spawning >> multiple threads, but the crash still occurs) . The number of threads >> is always below 100. >> >> jstack -F 21370 | grep ^Thread | wc -l >> ps -T -p 21370 (This gives me 63) >> >> I don't seem to specify the -Xss option: > > In some applications with a large number of threads (particularly when > running on 64-bit hardware) this setting can cause a problems. The default > value is pretty large (I think it's 1M on 64-bit systems). Since most apps > don't need that large of a value, an easy performance tuning step is to lower > the value of -Xss. > > Since you don't have very many threads, it seems unlikely that this is > causing your problem though. > > That being said, you could try explicitly setting a value for the thread > stack size. Finding the right values takes some testing though. I usually > start with something like 192k and run a few application tests. If I see any > stack overflow exceptions then I increase the value and rerun the tests. > Repeat until there are no stack overflow exceptions. > > > On a different note, what is the specific version of the JVM that you are > running? If it's not the latest, you could always try upgrading to the > latest version.
You need to hook up the VisualVM + Memory Pools plugin. This will show you where the memory is being consumed, if it's by the JVM. p > > >> >> Xms6g -Xmx6g -XX:NewSize=4G -XX:MaxNewSize=4G -XX:SurvivorRatio=6 >> -XX:MaxPermSize=512M -XX:-UseConcMarkSweepGC -XX:+UseStringCache >> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/example/logs >> >> -Jorge >> >> >> >> >> >> >> On Thu, Jun 7, 2012 at 12:07 PM, Daniel Mikusa <dmik...@vmware.com> >> wrote: >>> ----- Original Message ----- >>>> I am using MongoDB through the Java driver allowing up to 100 >>>> connections to the MongoDB server. >>>> I also use DBCP with a max size of 50 JDBC connections. >>>> My webapp uses about 150 JAR files. >>>> There is no native libraries loaded from my webapp as far as I >>>> know. >>>> All the app is pure Java code. (Nevertheless, Tomcat is using the >>>> Tomcat Native Library) >>>> >>>> Is there a way I can monitor the number of file descriptors in use >>>> by >>>> the app? >>>> >>>> I have monitored the number of threads, but I haven't seen >>>> anything >>>> unusual. >>> >>> How many threads have you observed? Total threads, not just >>> threads for the connector. >>> >>> Also, what is the value you are using for thread stack size? -Xss >>> >>> Dan >>> >>> >>> >>> (but it could be that the burst is too fast to get catch by >>>> the monitoring tool) >>>> >>>> -Jorge >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Jun 7, 2012 at 11:44 AM, Christopher Schultz >>>> <ch...@christopherschultz.net> wrote: >>>>> -----BEGIN PGP SIGNED MESSAGE----- >>>>> Hash: SHA1 >>>>> >>>>> Jorge, >>>>> >>>>> On 6/6/12 5:33 PM, Jorge Medina wrote: >>>>>> The web application uses Spring/Postgres/Mongo. >>>>> >>>>> Are you using MongoDB in-process or anything weird like that? Or >>>>> are >>>>> you connecting through some socket-based (or other) API? >>>>> >>>>>> It looks like a memory leak in native code, not java code; so >>>>>> my >>>>>> usual java toolset is not useful. >>>>> >>>>> If what you are observing is accurate (non-heap memory grows, >>>>> heap >>>>> stays reasonable) then it will definitely be more difficult to >>>>> track-down. >>>>> >>>>>> Tomcat runs behind nginx in a EC2 instance. The application >>>>>> uses >>>>>> Sun (now Oracle) JDK 1.6. >>>>>> >>>>>> Any suggestions on what should I look at? >>>>> >>>>> What do your <Connectors> look like? How many JDBC connections >>>>> do >>>>> you >>>>> have in your connection pool (which you are hopefully using!)? >>>>> How >>>>> about the same equivalent for MongoDB? >>>>> >>>>> Does your webapp keep lots of files open? Do you have an >>>>> unusually-large number of JAR files in your webapp? Do you have >>>>> any >>>>> native libraries in use within your webapp? >>>>> >>>>> What are all the non-default system properties that you are >>>>> setting >>>>> at >>>>> JVM launch time (you can easily see this from a 'ps' list)? >>>>> >>>>> Two things that can eat-up native memory fast in a JVM are file >>>>> descriptors and threads, so let's start there. >>>>> >>>>> - -chris >>>>> -----BEGIN PGP SIGNATURE----- >>>>> Version: GnuPG/MacGPG2 v2.0.17 (Darwin) >>>>> Comment: GPGTools - http://gpgtools.org >>>>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ >>>>> >>>>> iEYEARECAAYFAk/Q9ooACgkQ9CaO5/Lv0PDPyQCfVtddxMDOgQbjmMGC3gvnK+Qq >>>>> aZMAnjVu67+9Sm2bdYzAd91ZOrYo3DFI >>>>> =r+vl >>>>> -----END PGP SIGNATURE----- >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
