> Does it happen also on another machine? Yes, but only once (of 5…6 reindexing is running)
> Try latest Oracle JDK to see if it makes any difference Same effect with latest Oracle JVM (JRE) CPU / IO looks ok to me as long as the JVM is running. As soon as the JVM freezes/hangs CPU/IO is idle root@miraculix:~# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 15774 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 9000 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 15774 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Thanks for your input so far! Daniel Am 08.03.2013 um 10:49 schrieb <[email protected]>: > It is really weird that JVM process is frozen and you cannot get thread > dump. I never saw such problem. It looks like HW or OS problem. > Does it happen also on another machine? From your log I also see you use > OpenJDK. Try latest Oracle JDK to see if it makes any difference. > > > > > If you monitor frozen JVM process is there any activity? (CPU/io). Is there > any OS limit on process mem size? Check ulimit settings. (ulimit -a) > > > > > Marek > > > "Hi Marek > > I use jstack -F <pid> to get thread dumps. > - Except once the output of jstack is the only one I get (= the first dump I > provided) > - Once (the jvm was not completely frozen, just the indexing made no further > progress) jstack also made a dump in our log file (= second dump) > > Another result after killing the jvm process with "kill -4 <pid>" (kill -3 > does not work due to a frozen jvm) > http://pastebin.com/qc2bUkbk > > Daniel > > Am 08.03.2013 um 10:16 schrieb <[email protected]>: > >> I checked 2 thread dumps you provided so I can only comment what I saw >> there. I did not see any deadlock. Only >> problem I saw was in main thread where it was waiting for connection. >> >> >> >> >> >> 1. >> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x >> 00007fefd3812000] >> 2. >> java.lang.Thread.State: WAITING (on object monitor) >> 3. >> at java.lang.Object.$$YJP$$wait(Native Method) >> 4. >> at java.lang.Object.wait(Object.java) >> 5. >> at java.lang.Object.wait(Object.java:485) >> 6. >> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject >> (GenericObjectPool.java:1104) >> 7. >> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl. >> GenericObjectPool$Latch) >> 8. >> at org.apache.commons.dbcp.PoolingDataSource.getConnection >> (PoolingDataSource.java:106) >> 9. >> at org.apache.commons.dbcp.BasicDataSource.getConnection >> (BasicDataSource.java:1044) >> 10. >> at org.apache.jackrabbit.core.util.db.ConnectionHelper. >> getConnection(ConnectionHelper.java:439) >> >> >> >> >> >> >> You should be able to get thread dump any time unless JVM process is >> completely frozen IMO. Do you use jstack to get thread dump? >> >> Marek >> >> >> ---------- Původní zpráva ---------- >> Od: Daniel Hobi <[email protected]> >> Datum: 8. 3. 2013 >> Předmět: Re: JVM hangs during startup (indexing) (2) >> >> "Hi Marek >> >> Thanks for your suggestions! >> >> Unfortunately, changing the JVM gc options so that the JVM will no longer >> hang just worked ONCE. >> Therefore I was not able to get a "nice" stack trace anymore :-( >> >> Connection Pool: >> We had set the max connection pool size to 5. As you suggested I set it to > >> 15. Same effect. Even 50 or 100 did not help. >> Interestingly, postgres shows just 3 open connection for jackrabbit. Is > this >> a suspicious behavior or does it simply not need more connections and is >> therefore not spawning new connections? >> >> Enabling debug, setting login and socket timeout on the postgres-jdbc- > driver >> did not help either. >> I even changed the pool implementation from dbcp to c3p0 in jackrabbit- > core- >> 2.6.0 to be "sure" it is not a pool bug. >> Further we are not able to reproduce the hang with: >> - Slower machine, postgres 8.3 >> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far) >> - Faster machine, mysql >> >> This issue is slowly driving me nuts ;-) >> >> Thanks! >> Daniel >> >> >> >> Am 07.03.2013 um 16:24 schrieb [email protected]: >> >>> Also randomly we had problem with not released connections in JBoss >>> connection pool and Postgres. Now long time no occurrence. >>> In any case you can check number/list of open connections in psql . In > our >> >>> case it was >>> 97 open connection (by default 100 open connections in postgres and 3 are > >>> reserved for admin). It happenned even if we had much smaller >>> max connection pool size. As it happened randomly we did not find any >>> solution. We tried to change connection pool size and it does not >>> happen any more. Still no idea what was cause of this. In any case using >>> psql you can find out easily if it is also your problem. >>> >>> Query is for 9.x: SELECT * FROM pg_stat_activity; >>> Marek >>> >>> -- >>> Marek Slama >>> [email protected] >>> >>> >>> >>> ---------- Původní zpráva ---------- >>> Od: [email protected] >>> Datum: 7. 3. 2013 >>> Předmět: Re: JVM hangs during startup (indexing) (2) >>> >>> "Quick look: It hangs in connection pool - connection pool if it cannot >>> return free connection to caller it waits forever by default. >>> How big is your connection pool? (Maximum size? It would be good to patch > >>> connection pool to see who/how many connections is >>> used and why connections are used not release. In our config we use >>> connection max pool size 15 and it is enough.) So first >>> fast fix would be to increase max connection pool size but if there is >>> problem with not released connection it will not help for long. >>> >>> Good news is it is not problem with memory or gc (at least IMO). >>> >>> Marek >>> >>> -- >>> Marek Slama >>> [email protected] >>> >>> >>> >>> ---------- Původní zpráva ---------- >>> Od: Daniel Hobi <[email protected]> >>> Datum: 7. 3. 2013 >>> Předmět: JVM hangs during startup (indexing) (2) >>> >>> "Hi there >>> >>> >>> >>> Some updates on our issue started here: >>> >>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C >> 201 >>> 30115135117.186960%40gmx.net%3E >>> >>> >>> >>> We're still facing the JVM freeze (possibly during minor gcs) even if... >>> >>> ...we updated Jackrabbit from 2.2.13 to 2.6.0. >>> >>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried >>> OpenJDK. >>> >>> ...we increased the JVM heap size (up to 1.5gb). >>> >>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) > available. >>> >>> >>> >>> But there is good news too: >>> >>> - I've just played around with the JVM's garbage collection >>> options. Even if the index process still hangs the JVM will no longer >> block. >>> I was able to get an interesting thread dump: >>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8 >>> >>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the >>> hang occur faster (less than 100 minor gcs) >>> >>> >>> >>> I would be very grateful if someone could have a look at this. We're >> running >>> (slowly but surely) out of ideas :-( >>> >>> >>> >>> Thanks! >>> >>> Daniel""""
