Re: Solrj possible deadlock
[0x9667, 0xb3f1, 0xb3f1) the space 483968K, 14% used [0x9667, 0x9ace04b8, 0x9ace0600, 0xb3f1) compacting perm gen total 23040K, used 22983K [0xb3f1, 0xb559, 0xb7f1) the space 23040K, 99% used [0xb3f1, 0xb5581ff8, 0xb5582000, 0xb559) No shared spaces configured. Cheers. Brett. -- View this message in context: http://www.nabble.com/Solrj-possible-deadlock-tp25530146p25531321.html Sent from the Solr - User mailing list archive at Nabble.com. -- View this message in context: http://www.nabble.com/Solrj-possible-deadlock-tp25530146p25604919.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj possible deadlock
: Well, in the same processes I am using a jdbc connection to get all the : relative paths to the documents I want to index, then I parse the documents : to plain text using tones of open source libraries like POI, PFDBox : etc.(which might account for java2d) then I add them to the index and commit : every 2000 documents. Since nothing in your threaddumps refers to solrj or solr (either as the current method, or in the call stack suggesting that it's code called by Solr(J)), there's really no indication that the problem is even remotely solr related. i suspect that if you commented out all of hte code where you use SolrJ so that you still did all of the parsing and then just wrote the resulting data to /dev/null you would probably still see this behavior -- perhaps one of the other libraries you are using has some semantics you aren't obeying (ie: a parser that must be used single threaded, or an object that must be closed so that it can reset some static state) that is acusing this problem only after some time has elapsed (or on particular permutations of data) -Hoss
Re: Solrj possible deadlock
I had the same problem again yesterday except the process halted after about 20mins this time. pof wrote: Hello, I was running a batch index the other day using the Solrj EmbeddedSolrServer when the process abruptly froze in it's tracks after running for about 4-5 hours and indexing ~400K documents. There were no document locks so it would seem likely that there was some kind of thread deadlock. I was hoping someone might be able to tell me some information about the following thread dump taken at the time: Full thread dump OpenJDK Client VM (1.6.0-b09 mixed mode): DestroyJavaVM prio=10 tid=0x9322a800 nid=0xcef waiting on condition [0x..0x0018a044] java.lang.Thread.State: RUNNABLE Java2D Disposer daemon prio=10 tid=0x0a28cc00 nid=0xf1c in Object.wait() [0x0311d000..0x0311def4] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x97a96840 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) - locked 0x97a96840 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) at sun.java2d.Disposer.run(Disposer.java:143) at java.lang.Thread.run(Thread.java:636) pool-1-thread-1 prio=10 tid=0x93a26c00 nid=0xcf7 waiting on condition [0x08a6a000..0x08a6b074] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x967acfd0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1978) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Low Memory Detector daemon prio=10 tid=0x93a00c00 nid=0xcf5 runnable [0x..0x] java.lang.Thread.State: RUNNABLE CompilerThread0 daemon prio=10 tid=0x09fe9800 nid=0xcf4 waiting on condition [0x..0x096a7af4] java.lang.Thread.State: RUNNABLE Signal Dispatcher daemon prio=10 tid=0x09fe8800 nid=0xcf3 waiting on condition [0x..0x] java.lang.Thread.State: RUNNABLE Finalizer daemon prio=10 tid=0x09fd7000 nid=0xcf2 in Object.wait() [0x005ca000..0x005caef4] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x966e6d40 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) - locked 0x966e6d40 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Reference Handler daemon prio=10 tid=0x09fd2c00 nid=0xcf1 in Object.wait() [0x00579000..0x00579d74] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x966e6dc8 (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked 0x966e6dc8 (a java.lang.ref.Reference$Lock) VM Thread prio=10 tid=0x09fcf800 nid=0xcf0 runnable VM Periodic Task Thread prio=10 tid=0x93a02400 nid=0xcf6 waiting on condition JNI global references: 1072 Heap def new generation total 36288K, used 23695K [0x93f1, 0x9667, 0x9667) eden space 32256K, 73% used [0x93f1, 0x95633f60, 0x95e9) from space 4032K, 0% used [0x95e9, 0x95e9, 0x9628) to space 4032K, 0% used [0x9628, 0x9628, 0x9667) tenured generation total 483968K, used 72129K [0x9667, 0xb3f1, 0xb3f1) the space 483968K, 14% used [0x9667, 0x9ace04b8, 0x9ace0600, 0xb3f1) compacting perm gen total 23040K, used 22983K [0xb3f1, 0xb559, 0xb7f1) the space 23040K, 99% used [0xb3f1, 0xb5581ff8, 0xb5582000, 0xb559) No shared spaces configured. Cheers. Brett. -- View this message in context: http://www.nabble.com/Solrj-possible-deadlock-tp25530146p25531321.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solrj possible deadlock
do you have anything custom going on? The fact that the lock is in java2d seems suspicious... On Sep 23, 2009, at 7:01 PM, pof wrote: I had the same problem again yesterday except the process halted after about 20mins this time. pof wrote: Hello, I was running a batch index the other day using the Solrj EmbeddedSolrServer when the process abruptly froze in it's tracks after running for about 4-5 hours and indexing ~400K documents. There were no document locks so it would seem likely that there was some kind of thread deadlock. I was hoping someone might be able to tell me some information about the following thread dump taken at the time: Full thread dump OpenJDK Client VM (1.6.0-b09 mixed mode): DestroyJavaVM prio=10 tid=0x9322a800 nid=0xcef waiting on condition [0x..0x0018a044] java.lang.Thread.State: RUNNABLE Java2D Disposer daemon prio=10 tid=0x0a28cc00 nid=0xf1c in Object.wait() [0x0311d000..0x0311def4] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x97a96840 (a java.lang.ref.ReferenceQueue $Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: 133) - locked 0x97a96840 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: 149) at sun.java2d.Disposer.run(Disposer.java:143) at java.lang.Thread.run(Thread.java:636) pool-1-thread-1 prio=10 tid=0x93a26c00 nid=0xcf7 waiting on condition [0x08a6a000..0x08a6b074] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x967acfd0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer $ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer $ConditionObject.await(AbstractQueuedSynchronizer.java:1978) at java .util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java: 386) at java .util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java: 1043) at java .util .concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1103) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Low Memory Detector daemon prio=10 tid=0x93a00c00 nid=0xcf5 runnable [0x..0x] java.lang.Thread.State: RUNNABLE CompilerThread0 daemon prio=10 tid=0x09fe9800 nid=0xcf4 waiting on condition [0x..0x096a7af4] java.lang.Thread.State: RUNNABLE Signal Dispatcher daemon prio=10 tid=0x09fe8800 nid=0xcf3 waiting on condition [0x..0x] java.lang.Thread.State: RUNNABLE Finalizer daemon prio=10 tid=0x09fd7000 nid=0xcf2 in Object.wait() [0x005ca000..0x005caef4] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x966e6d40 (a java.lang.ref.ReferenceQueue $Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: 133) - locked 0x966e6d40 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java: 149) at java.lang.ref.Finalizer $FinalizerThread.run(Finalizer.java:177) Reference Handler daemon prio=10 tid=0x09fd2c00 nid=0xcf1 in Object.wait() [0x00579000..0x00579d74] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x966e6dc8 (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked 0x966e6dc8 (a java.lang.ref.Reference$Lock) VM Thread prio=10 tid=0x09fcf800 nid=0xcf0 runnable VM Periodic Task Thread prio=10 tid=0x93a02400 nid=0xcf6 waiting on condition JNI global references: 1072 Heap def new generation total 36288K, used 23695K [0x93f1, 0x9667, 0x9667) eden space 32256K, 73% used [0x93f1, 0x95633f60, 0x95e9) from space 4032K, 0% used [0x95e9, 0x95e9, 0x9628) to space 4032K, 0% used [0x9628, 0x9628, 0x9667) tenured generation total 483968K, used 72129K [0x9667, 0xb3f1, 0xb3f1) the space 483968K, 14% used [0x9667, 0x9ace04b8, 0x9ace0600, 0xb3f1) compacting perm gen total 23040K, used 22983K [0xb3f1, 0xb559, 0xb7f1) the space 23040K, 99% used [0xb3f1, 0xb5581ff8, 0xb5582000, 0xb559) No shared spaces configured. Cheers. Brett. -- View this message in context: http://www.nabble.com/Solrj-possible-deadlock-tp25530146p25531321.html Sent from the Solr - User mailing list archive at Nabble.com.
Solrj possible deadlock
Hello, I was running a batch index the other day using the Solrj EmbeddedSolrServer when the process abruptly froze in it's tracks after running for about 4-5 hours and indexing ~400K documents. There were no document locks so it would seem likely that there was some kind of thread deadlock. I was hoping someone might be able to tell me some information about the following thread dump taken at the time: Full thread dump OpenJDK Client VM (1.6.0-b09 mixed mode): DestroyJavaVM prio=10 tid=0x9322a800 nid=0xcef waiting on condition [0x..0x0018a044] java.lang.Thread.State: RUNNABLE Java2D Disposer daemon prio=10 tid=0x0a28cc00 nid=0xf1c in Object.wait() [0x0311d000..0x0311def4] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x97a96840 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) - locked 0x97a96840 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) at sun.java2d.Disposer.run(Disposer.java:143) at java.lang.Thread.run(Thread.java:636) pool-1-thread-1 prio=10 tid=0x93a26c00 nid=0xcf7 waiting on condition [0x08a6a000..0x08a6b074] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x967acfd0 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1978) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Low Memory Detector daemon prio=10 tid=0x93a00c00 nid=0xcf5 runnable [0x..0x] java.lang.Thread.State: RUNNABLE CompilerThread0 daemon prio=10 tid=0x09fe9800 nid=0xcf4 waiting on condition [0x..0x096a7af4] java.lang.Thread.State: RUNNABLE Signal Dispatcher daemon prio=10 tid=0x09fe8800 nid=0xcf3 waiting on condition [0x..0x] java.lang.Thread.State: RUNNABLE Finalizer daemon prio=10 tid=0x09fd7000 nid=0xcf2 in Object.wait() [0x005ca000..0x005caef4] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x966e6d40 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) - locked 0x966e6d40 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Reference Handler daemon prio=10 tid=0x09fd2c00 nid=0xcf1 in Object.wait() [0x00579000..0x00579d74] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x966e6dc8 (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked 0x966e6dc8 (a java.lang.ref.Reference$Lock) VM Thread prio=10 tid=0x09fcf800 nid=0xcf0 runnable VM Periodic Task Thread prio=10 tid=0x93a02400 nid=0xcf6 waiting on condition JNI global references: 1072 Heap def new generation total 36288K, used 23695K [0x93f1, 0x9667, 0x9667) eden space 32256K, 73% used [0x93f1, 0x95633f60, 0x95e9) from space 4032K, 0% used [0x95e9, 0x95e9, 0x9628) to space 4032K, 0% used [0x9628, 0x9628, 0x9667) tenured generation total 483968K, used 72129K [0x9667, 0xb3f1, 0xb3f1) the space 483968K, 14% used [0x9667, 0x9ace04b8, 0x9ace0600, 0xb3f1) compacting perm gen total 23040K, used 22983K [0xb3f1, 0xb559, 0xb7f1) the space 23040K, 99% used [0xb3f1, 0xb5581ff8, 0xb5582000, 0xb559) No shared spaces configured. Cheers. Brett. -- View this message in context: http://www.nabble.com/Solrj-possible-deadlock-tp25530146p25530146.html Sent from the Solr - User mailing list archive at Nabble.com.
