I'll have to defer that one for now..... 2010/1/26 Tim Terlegård <tim.terleg...@gmail.com>
> 2010/1/26 Erick Erickson <erickerick...@gmail.com>: > > > My indexing script has been running all > > > night and has accomplished nothing. I see lots of disk activity > > > though, which is weird. > > > > > > One explanation would be that you're memory-starved and > > the disk activity you see is thrashing. How much memory > > do you allocate to your JVM? A further indication that > > this is where you should start looking would be if your > > CPU usage is very low at the same time..... > > CPU usage was very low. There were lots of free memory. I immediately > thought solr caused the disk activity, but I might have been wrong, > because the disk activity stopped after a while and the indexing still > showed no progress. > > Does this thread dump reveal anything? It doesn't look like solr is doing > much? > > /Tim > > > <?xml-stylesheet type="text/xsl" href="threaddump.xsl"?> > <solr> > <core>example</core> > <system> > <jvm> > <version>1.5.0_22-147</version> > <name>Java HotSpot(TM) Server VM</name> > </jvm> > <threadCount> > <current>20</current> > <peak>23</peak> > <daemon>3</daemon> > </threadCount> > <threadDump> > <thread> > <id>64</id> > <name>pool-8-thread-1</name> > <state>WAITING</state> > <cpuTime>10430,6650ms</cpuTime> > <userTime>8602,0210ms</userTime> > <stackTrace> > <line>at sun.misc.Unsafe.park(Native Method) </line> > <line>at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118) > </line> > <line>at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841) > </line> > <line>at java.util.concurrent.DelayQueue.take(DelayQueue.java:131) </line> > <line>at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:533) > </line> > <line>at > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:526) > </line> > <line>at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470) > </line> > <line>at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) > </line> > <line>at java.lang.Thread.run(Thread.java:613) </line> > </stackTrace> > </thread> > <thread> > <id>60</id> > <name>pool-7-thread-1</name> > <state>WAITING</state> > <cpuTime>132,0080ms</cpuTime> > <userTime>13,6950ms</userTime> > <stackTrace> > <line>at sun.misc.Unsafe.park(Native Method) </line> > <line>at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118) > </line> > <line>at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841) > </line> > <line>at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359) > </line> > <line>at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470) > </line> > <line>at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) > </line> > <line>at java.lang.Thread.run(Thread.java:613) </line> > </stackTrace> > </thread> > <thread> > <id>26</id> > <name>DestroyJavaVM</name> > <state>RUNNABLE</state> > <cpuTime>1453,5030ms</cpuTime> > <userTime>1317,7670ms</userTime> > <stackTrace/> > </thread> > <thread> > <id>25</id> > <name>Timer-2</name> > <state>TIMED_WAITING</state> > <lock>java.util.taskqu...@6a58c4</lock> > <cpuTime>3,2500ms</cpuTime> > <userTime>0,8090ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at java.util.TimerThread.mainLoop(Timer.java:509) </line> > <line>at java.util.TimerThread.run(Timer.java:462) </line> > </stackTrace> > </thread> > <thread> > <id>24</id> > <name>pool-1-thread-1</name> > <state>WAITING</state> > <cpuTime>41,5590ms</cpuTime> > <userTime>39,1740ms</userTime> > <stackTrace> > <line>at sun.misc.Unsafe.park(Native Method) </line> > <line>at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118) > </line> > <line>at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841) > </line> > <line>at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359) > </line> > <line>at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470) > </line> > <line>at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) > </line> > <line>at java.lang.Thread.run(Thread.java:613) </line> > </stackTrace> > </thread> > <thread> > <id>22</id> > <name>Timer-1</name> > <state>TIMED_WAITING</state> > <lock>java.util.taskqu...@e9d100</lock> > <cpuTime>96,9640ms</cpuTime> > <userTime>74,5150ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at java.util.TimerThread.mainLoop(Timer.java:509) </line> > <line>at java.util.TimerThread.run(Timer.java:462) </line> > </stackTrace> > </thread> > <thread> > <id>21</id> > <name>btpool0-9 - Acceptor0 SocketConnector @ 0.0.0.0:8983</name> > <state>RUNNABLE</state> > <inNative/> > <cpuTime>26,0110ms</cpuTime> > <userTime>23,3400ms</userTime> > <stackTrace> > <line>at java.net.PlainSocketImpl.socketAccept(Native Method) </line> > <line>at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) </line> > <line>at java.net.ServerSocket.implAccept(ServerSocket.java:450) </line> > <line>at java.net.ServerSocket.accept(ServerSocket.java:421) </line> > <line>at > org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:97) > </line> > <line>at > org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:516) > </line> > <line>at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) > </line> > </stackTrace> > </thread> > <thread> > <id>20</id> > <name>btpool0-8</name> > <state>TIMED_WAITING</state> > <lock>org.mortbay.thread.boundedthreadpool$poolthr...@7a17</lock> > <cpuTime>734105,5810ms</cpuTime> > <userTime>727677,9460ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) > </line> > </stackTrace> > </thread> > <thread> > <id>19</id> > <name>btpool0-7</name> > <state>TIMED_WAITING</state> > <lock>org.mortbay.thread.boundedthreadpool$poolthr...@7414c8</lock> > <cpuTime>798010,4300ms</cpuTime> > <userTime>785039,2820ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) > </line> > </stackTrace> > </thread> > <thread> > <id>18</id> > <name>btpool0-6</name> > <state>TIMED_WAITING</state> > <lock>org.mortbay.thread.boundedthreadpool$poolthr...@e5c339</lock> > <cpuTime>719254,0510ms</cpuTime> > <userTime>710319,6850ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) > </line> > </stackTrace> > </thread> > <thread> > <id>17</id> > <name>btpool0-5</name> > <state>TIMED_WAITING</state> > <lock>org.mortbay.thread.boundedthreadpool$poolthr...@d38976</lock> > <cpuTime>243756,7410ms</cpuTime> > <userTime>240759,1390ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) > </line> > </stackTrace> > </thread> > <thread> > <id>16</id> > <name>btpool0-4</name> > <state>TIMED_WAITING</state> > <lock>org.mortbay.thread.boundedthreadpool$poolthr...@ad97f5</lock> > <cpuTime>501531,8820ms</cpuTime> > <userTime>496494,6760ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) > </line> > </stackTrace> > </thread> > <thread> > <id>15</id> > <name>btpool0-3</name> > <state>TIMED_WAITING</state> > <lock>org.mortbay.thread.boundedthreadpool$poolthr...@8f729c</lock> > <cpuTime>506615,3310ms</cpuTime> > <userTime>499281,0090ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) > </line> > </stackTrace> > </thread> > <thread> > <id>14</id> > <name>btpool0-2</name> > <state>RUNNABLE</state> > <cpuTime>463555,4030ms</cpuTime> > <userTime>459439,8950ms</userTime> > <stackTrace> > <line>at sun.management.ThreadImpl.getThreadInfo0(Native Method) </line> > <line>at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:142) > </line> > <line>at > org.apache.jsp.admin.threaddump_jsp._jspService(org.apache.jsp.admin.threaddump_jsp:263) > </line> > <line>at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:80) > </line> > <line>at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > </line> > <line>at > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373) > </line> > <line>at > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:464) > </line> > <line>at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:358) > </line> > <line>at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > </line> > <line>at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487) > </line> > <line>at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:367) > </line> > <line>at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > </line> > <line>at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) > </line> > <line>at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) > </line> > <line>at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) > </line> > <line>at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:268) > </line> > <line>at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126) > </line> > <line>at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:264) > </line> > <line>at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) > </line> > <line>at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) > </line> > <line>at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > </line> > <line>at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) > </line> > <line>at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712) > </line> > <line>at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) > </line> > <line>at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211) > </line> > <line>at > org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) > </line> > <line>at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139) > </line> > <line>at org.mortbay.jetty.Server.handle(Server.java:285) </line> > <line>at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502) > </line> > <line>at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:821) > </line> > <line>at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513) > </line> > <line>at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208) > </line> > <line>at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378) > </line> > <line>at > org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226) > </line> > <line>at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442) > </line> > </stackTrace> > </thread> > <thread> > <id>13</id> > <name>btpool0-1</name> > <state>TIMED_WAITING</state> > <lock>org.mortbay.thread.boundedthreadpool$poolthr...@3508c0</lock> > <cpuTime>474719,7810ms</cpuTime> > <userTime>466715,4570ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) > </line> > </stackTrace> > </thread> > <thread> > <id>12</id> > <name>btpool0-0</name> > <state>TIMED_WAITING</state> > <lock>org.mortbay.thread.boundedthreadpool$poolthr...@dca7d0</lock> > <cpuTime>471927,0890ms</cpuTime> > <userTime>466073,8550ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at > org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:482) > </line> > </stackTrace> > </thread> > <thread> > <id>11</id> > <name>Timer-0</name> > <state>TIMED_WAITING</state> > <lock>java.util.taskqu...@5e3f91</lock> > <cpuTime>3815,8490ms</cpuTime> > <userTime>2435,4160ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at java.util.TimerThread.mainLoop(Timer.java:509) </line> > <line>at java.util.TimerThread.run(Timer.java:462) </line> > </stackTrace> > </thread> > <thread> > <id>4</id> > <name>Signal Dispatcher</name> > <state>RUNNABLE</state> > <cpuTime>0,1020ms</cpuTime> > <userTime>0,0360ms</userTime> > <stackTrace/> > </thread> > <thread> > <id>3</id> > <name>Finalizer</name> > <state>WAITING</state> > <lock>java.lang.ref.referencequeue$l...@9f6c2b</lock> > <cpuTime>8,2410ms</cpuTime> > <userTime>7,2220ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) > </line> > <line>at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) > </line> > <line>at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) > </line> > </stackTrace> > </thread> > <thread> > <id>2</id> > <name>Reference Handler</name> > <state>WAITING</state> > <lock>java.lang.ref.reference$l...@7692f6</lock> > <cpuTime>12,5330ms</cpuTime> > <userTime>8,9670ms</userTime> > <stackTrace> > <line>at java.lang.Object.wait(Native Method) </line> > <line>at java.lang.Object.wait(Object.java:474) </line> > <line>at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) > </line> > </stackTrace> > </thread> > </threadDump> > </system> > </solr> >
