2010/1/26 Erick Erickson <erickerick...@gmail.com>: > > My indexing script has been running all > > night and has accomplished nothing. I see lots of disk activity > > though, which is weird. > > > One explanation would be that you're memory-starved and > the disk activity you see is thrashing. How much memory > do you allocate to your JVM? A further indication that > this is where you should start looking would be if your > CPU usage is very low at the same time.....
CPU usage was very low. There were lots of free memory. I immediately thought solr caused the disk activity, but I might have been wrong, because the disk activity stopped after a while and the indexing still showed no progress. Does this thread dump reveal anything? It doesn't look like solr is doing much? /Tim <?xml-stylesheet type="text/xsl" href="threaddump.xsl"?> <solr> <core>example</core> <system> <jvm> <version>1.5.0_22-147</version> <name>Java HotSpot(TM) Server VM</name> </jvm> <threadCount> <current>20</current> <peak>23</peak> <daemon>3</daemon> </threadCount> <threadDump> <thread> <id>64</id> <name>pool-8-thread-1</name> <state>WAITING</state> <cpuTime>10430,6650ms</cpuTime> <userTime>8602,0210ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841) </line> <line>at java.util.concurrent.DelayQueue.take(DelayQueue.java:131) </line> <line>at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:533) </line> <line>at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:526) </line> <line>at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470) </line> <line>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) </line> <line>at java.lang.Thread.run(Thread.java:613) </line> </stackTrace> </thread> <thread> <id>60</id> <name>pool-7-thread-1</name> <state>WAITING</state> <cpuTime>132,0080ms</cpuTime> <userTime>13,6950ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841) </line> <line>at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359) </line> <line>at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470) </line> <line>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) </line> <line>at java.lang.Thread.run(Thread.java:613) </line> </stackTrace> </thread> <thread> <id>26</id> <name>DestroyJavaVM</name> <state>RUNNABLE</state> <cpuTime>1453,5030ms</cpuTime> <userTime>1317,7670ms</userTime> <stackTrace/> </thread> <thread> <id>25</id> <name>Timer-2</name> <state>TIMED_WAITING</state> <lock>java.util.taskqu...@6a58c4</lock> <cpuTime>3,2500ms</cpuTime> <userTime>0,8090ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.util.TimerThread.mainLoop(Timer.java:509) </line> <line>at java.util.TimerThread.run(Timer.java:462) </line> </stackTrace> </thread> <thread> <id>24</id> <name>pool-1-thread-1</name> <state>WAITING</state> <cpuTime>41,5590ms</cpuTime> <userTime>39,1740ms</userTime> <stackTrace> <line>at sun.misc.Unsafe.park(Native Method) </line> <line>at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118) </line> <line>at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841) </line> <line>at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359) </line> <line>at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470) </line> <line>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) </line> <line>at java.lang.Thread.run(Thread.java:613) </line> </stackTrace> </thread> <thread> <id>22</id> <name>Timer-1</name> <state>TIMED_WAITING</state> <lock>java.util.taskqu...@e9d100</lock> <cpuTime>96,9640ms</cpuTime> <userTime>74,5150ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.util.TimerThread.mainLoop(Timer.java:509) </line> <line>at java.util.TimerThread.run(Timer.java:462) </line> </stackTrace> </thread> <thread> <id>21</id> <name>btpool0-9 - Acceptor0 SocketConnector @ 0.0.0.0:8983</name> <state>RUNNABLE</state> <inNative/> <cpuTime>26,0110ms</cpuTime> <userTime>23,3400ms</userTime> <stackTrace> <line>at java.net.PlainSocketImpl.socketAccept(Native Method) </line> <line>at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) </line> <line>at java.net.ServerSocket.implAccept(ServerSocket.java:450) </line> <line>at java.net.ServerSocket.accept(ServerSocket.java:421) </line> <line>at org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:97) </line> <line>at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:516) </line> <line>at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) </line> </stackTrace> </thread> <thread> <id>20</id> <name>btpool0-8</name> <state>TIMED_WAITING</state> <lock>org.mortbay.thread.boundedthreadpool$poolthr...@7a17</lock> <cpuTime>734105,5810ms</cpuTime> <userTime>727677,9460ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) </line> </stackTrace> </thread> <thread> <id>19</id> <name>btpool0-7</name> <state>TIMED_WAITING</state> <lock>org.mortbay.thread.boundedthreadpool$poolthr...@7414c8</lock> <cpuTime>798010,4300ms</cpuTime> <userTime>785039,2820ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) </line> </stackTrace> </thread> <thread> <id>18</id> <name>btpool0-6</name> <state>TIMED_WAITING</state> <lock>org.mortbay.thread.boundedthreadpool$poolthr...@e5c339</lock> <cpuTime>719254,0510ms</cpuTime> <userTime>710319,6850ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) </line> </stackTrace> </thread> <thread> <id>17</id> <name>btpool0-5</name> <state>TIMED_WAITING</state> <lock>org.mortbay.thread.boundedthreadpool$poolthr...@d38976</lock> <cpuTime>243756,7410ms</cpuTime> <userTime>240759,1390ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) </line> </stackTrace> </thread> <thread> <id>16</id> <name>btpool0-4</name> <state>TIMED_WAITING</state> <lock>org.mortbay.thread.boundedthreadpool$poolthr...@ad97f5</lock> <cpuTime>501531,8820ms</cpuTime> <userTime>496494,6760ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) </line> </stackTrace> </thread> <thread> <id>15</id> <name>btpool0-3</name> <state>TIMED_WAITING</state> <lock>org.mortbay.thread.boundedthreadpool$poolthr...@8f729c</lock> <cpuTime>506615,3310ms</cpuTime> <userTime>499281,0090ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) </line> </stackTrace> </thread> <thread> <id>14</id> <name>btpool0-2</name> <state>RUNNABLE</state> <cpuTime>463555,4030ms</cpuTime> <userTime>459439,8950ms</userTime> <stackTrace> <line>at sun.management.ThreadImpl.getThreadInfo0(Native Method) </line> <line>at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:142) </line> <line>at org.apache.jsp.admin.threaddump_jsp._jspService(org.apache.jsp.admin.threaddump_jsp:263) </line> <line>at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:80) </line> <line>at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) </line> <line>at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373) </line> <line>at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:464) </line> <line>at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:358) </line> <line>at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) </line> <line>at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487) </line> <line>at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:367) </line> <line>at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) </line> <line>at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) </line> <line>at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) </line> <line>at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) </line> <line>at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:268) </line> <line>at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126) </line> <line>at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:264) </line> <line>at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) </line> <line>at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) </line> <line>at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) </line> <line>at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) </line> <line>at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) </line> <line>at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) </line> <line>at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211) </line> <line>at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) </line> <line>at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) </line> <line>at org.mortbay.jetty.Server.handle(Server.java:285) </line> <line>at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502) </line> <line>at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:821) </line> <line>at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513) </line> <line>at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208) </line> <line>at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378) </line> <line>at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226) </line> <line>at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) </line> </stackTrace> </thread> <thread> <id>13</id> <name>btpool0-1</name> <state>TIMED_WAITING</state> <lock>org.mortbay.thread.boundedthreadpool$poolthr...@3508c0</lock> <cpuTime>474719,7810ms</cpuTime> <userTime>466715,4570ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) </line> </stackTrace> </thread> <thread> <id>12</id> <name>btpool0-0</name> <state>TIMED_WAITING</state> <lock>org.mortbay.thread.boundedthreadpool$poolthr...@dca7d0</lock> <cpuTime>471927,0890ms</cpuTime> <userTime>466073,8550ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) </line> </stackTrace> </thread> <thread> <id>11</id> <name>Timer-0</name> <state>TIMED_WAITING</state> <lock>java.util.taskqu...@5e3f91</lock> <cpuTime>3815,8490ms</cpuTime> <userTime>2435,4160ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.util.TimerThread.mainLoop(Timer.java:509) </line> <line>at java.util.TimerThread.run(Timer.java:462) </line> </stackTrace> </thread> <thread> <id>4</id> <name>Signal Dispatcher</name> <state>RUNNABLE</state> <cpuTime>0,1020ms</cpuTime> <userTime>0,0360ms</userTime> <stackTrace/> </thread> <thread> <id>3</id> <name>Finalizer</name> <state>WAITING</state> <lock>java.lang.ref.referencequeue$l...@9f6c2b</lock> <cpuTime>8,2410ms</cpuTime> <userTime>7,2220ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) </line> <line>at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) </line> <line>at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) </line> </stackTrace> </thread> <thread> <id>2</id> <name>Reference Handler</name> <state>WAITING</state> <lock>java.lang.ref.reference$l...@7692f6</lock> <cpuTime>12,5330ms</cpuTime> <userTime>8,9670ms</userTime> <stackTrace> <line>at java.lang.Object.wait(Native Method) </line> <line>at java.lang.Object.wait(Object.java:474) </line> <line>at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) </line> </stackTrace> </thread> </threadDump> </system> </solr>
