> This looks like a problem with the name server lookup. Perhaps your > name server is slow. While running Nutch you could try "talking" to > the name server and see if it responds quickly or if it"s slow. > > You could probably further examine this problem by running Nutch with > `strace", if you are running this on UNIX. > Otis
Hi Otis, I have done some testing and I don't think it is because the domain servers are slow. I can "talk" to the dns servers during the stall and the responses are not slow. I did some additional debugging and found that the crawl stalls when there is 1 or 2 remaining active fetcher threads. Below is the latest SIGQUIT. Also, the tail of strace output (don't know if this is useful). I have found that the remaining thread is in the middle of a fetch. Here are a couple urls that it gets stuck on: http://155.103.6.10/cgi-bin/sic/sicser4?88 http://216.205.119.118/robots.txt I am trying to debug some more, but I would really appreciate some help. Please let me know if any additional information is wanted. Thanks, Ralph --- tail of strace output (i have added some of my own debugging statements) --- 19:47:34 gettimeofday({1100576854, 48010}, NULL) = 0 19:47:34 futex(0xa9a10e5c, FUTEX_WAIT, 4589, {0, 999941000}) = -1 ETIMEDOUT (Connection timed out) 19:47:35 futex(0x805c250, FUTEX_WAKE, 1) = 0 19:47:35 gettimeofday({1100576855, 57496}, NULL) = 0 19:47:35 gettimeofday({1100576855, 57560}, NULL) = 0 19:47:35 write(2, "041115 194735 10 Thread Count:1\n", 32) = 32 19:47:35 gettimeofday({1100576855, 57690}, NULL) = 0 19:47:35 write(2, "041115 194735 10 Thread Size:1\n", 31) = 31 19:47:35 gettimeofday({1100576855, 57773}, NULL) = 0 19:47:35 write(2, "041115 194735 10 I:0 List[i]:Thr"..., 57) = 57 19:47:35 gettimeofday({1100576855, 57853}, NULL) = 0 19:47:35 write(2, "041115 194735 10 NAME:fetcher48\n", 32) = 32 19:47:35 gettimeofday({1100576855, 57933}, NULL) = 0 19:47:35 write(2, "041115 194735 10 Thread[fetcher4"..., 45) = 45 19:47:35 gettimeofday({1100576855, 58014}, NULL) = 0 19:47:35 gettimeofday({1100576855, 58042}, NULL) = 0 19:47:35 gettimeofday({1100576855, 58071}, NULL) = 0 19:47:35 clock_gettime(0, 0xbfffc758) = -1 ENOSYS (Function not implemented) 19:47:35 gettimeofday({1100576855, 58131}, NULL) = 0 19:47:35 futex(0xa9a10e5c, FUTEX_WAIT, 4591, {0, 999940000}) = -1 EINTR (Interrupted system call) 19:47:35 +++ killed by SIGKILL +++ --- SIGQUIT --- Full thread dump Java HotSpot(TM) Client VM (1.5.0-b64 mixed mode, sharing): "fetcher48" prio=1 tid=0x084dc130 nid=0x1e55 runnable [0xaf6c2000..0xaf6c2480] at java.net.Inet4AddressImpl.getHostByAddr(Native Method) at java.net.InetAddress$1.getHostByAddr(InetAddress.java:842) at java.net.InetAddress.getHostFromNameService(InetAddress.java:532) at java.net.InetAddress.getHostName(InetAddress.java:475) at java.net.InetAddress.getHostName(InetAddress.java:447) at java.net.InetSocketAddress.getHostName(InetSocketAddress.java:210) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:341) at java.net.Socket.connect(Socket.java:507) at net.nutch.protocol.http.HttpResponse.<init>(HttpResponse.java:95) at net.nutch.protocol.http.Http.getContent(Http.java:151) at net.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:109) "Low Memory Detector" daemon prio=1 tid=0x080a0bf8 nid=0x1e22 runnable [0x00000000..0x00000000] "CompilerThread0" daemon prio=1 tid=0x0809f748 nid=0x1e21 waiting on condition [0x00000000..0xb14f9fe8] "Signal Dispatcher" daemon prio=1 tid=0x0809e810 nid=0x1e20 waiting on condition [0x00000000..0x00000000] "Finalizer" daemon prio=1 tid=0x0809abf0 nid=0x1e1f in Object.wait() [0xb17fc000..0xb17fc700] at java.lang.Object.wait(Native Method) - waiting on <0x5260bb08> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116) - locked <0x5260bb08> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=1 tid=0x08098d98 nid=0x1e1e in Object.wait() [0xb187d000..0xb187d780] at java.lang.Object.wait(Native Method) - waiting on <0x5260bb88> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:474) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x5260bb88> (a java.lang.ref.Reference$Lock) "main" prio=1 tid=0x0805bea0 nid=0x1e18 waiting on condition [0xbfffc000..0xbfffcc78] at java.lang.Thread.sleep(Native Method) at net.nutch.fetcher.Fetcher.run(Fetcher.java:306) at net.nutch.fetcher.Fetcher.main(Fetcher.java:447) "VM Thread" prio=1 tid=0x080962e8 nid=0x1e1d runnable "VM Periodic Task Thread" prio=1 tid=0x080a2160 nid=0x1e23 waiting on condition ------------------------------------------------------- This SF.Net email is sponsored by: InterSystems CACHE FREE OODBMS DOWNLOAD - A multidimensional database that combines robust object and relational technologies, making it a perfect match for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8 _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
