Could you possibly get a stack dump? kill -QUIT `cat freenet.pid` (the pid of the top java process for the node). It is _possible_ that this is some sort of deadlock.
On Tue, Feb 04, 2003 at 01:51:37PM +0100, Niklas Bergh wrote: > I would like to reinforce my previous speculation. > > One of my machines are running a v551 fred node, current uptime is about > 6 days. The machines processor usage is somewhere around 10 percent or > so but fred is reporting 90-95 percent load and freds network usage is > close to zero. Fred has a constant routing time of 0 ms and 'Active > pooled jobs' is varying between 195 and 210 (so the node is not > completely dead, just almost). The node reports 120 open inbound > connections and 0 open outbound and has a sendqueue of only a couple of > 100 kilobytes (on a normal day the sendqueue should be around 10-20 > megs). My guess is that by tomorrow or so my node will be completely > idle with a whole bunch of open connections. > > My log file shows in the order of 5-10/second of these: > 2003-feb-04 12:19:28 (freenet.ConnectionHandler, QThread-67755): > DataRequest @ e7426ffa8ca356c7 <- 81.97.244.57:18887 > 2003-feb-04 12:19:28 (freenet.ConnectionHandler, QThread-68154): > QueryRejected @ e7426ffa8ca356c7 -> 81.97.244.57:18887 > > And very few (1 every 5 seconds or so) of these: > 2003-feb-04 12:19:28 (freenet.interfaces.PublicInterface, Interface # > tcp/14166): Accepted connection: tcpconnection: 68.36.137.144:1142 > > I guess that's not very good for the network?! > > I am also seeing quite a bit of 'I/O error on socket' messages in the > log, say, about one every 5 seconds or so. Is that normal? > > Looking at the data from my ocm (see below), I am pretty much able to > divide the connections into 3 different categories: > > 1, healthy idle: Connections open that fred currently are neither > sending data to or recieveing data from and none of them seems to be > especially long-lived. Neither do they have a very long 'Idle time', but > it is pretty much always larger than 0. > > 2, healty busy: Connections that I am either reciving data from, or > sending data to or both at the same time, all have 0 'Idle time' and NOT > SO VERY long 'LifeTime'. > > 3, dead: Connections that fred is expecting to either reciving data > from, or send data to or both at the same time, all have 0 'Idle time' > and VERY long 'LifeTime'. > > > The categorization above plus fact that the first and the 8:th > connection (see table below), both category 3, has a sendqueue of 5000 > and 217383 bytes respectively and have had exactly so for a couple of > hours now leads me to the conlusion that: > If fred assumes that it will soon receive data from or send data to a > connection and that that connection somehow dies then this will not be > detected by fred. This will lead to that given enough fred uptime all > connections will become category three. > > At date:Tue Feb 04 12:28:43 GMT+01:00 2003 > Number of open connections: 119 > Number of outbound connections: 0 > Number of inbound connections: 119 > Bytes waiting to be sent: 222383 > Peer addr Send Count Send Size Receiving Messages Idle time Lifetime > Thread Type ID > 129.187.77.195:37175 1 5000 yes 1563 0 428218253 QThread-32403 Inbound > 13c197a > 12.230.157.49:3739 0 0 yes 1 0 374236231 QThread-47873 Inbound 9a912 > 195.64.94.237:4598 0 0 yes 8 0 369743741 QThread-48562 Inbound 3b1a4f > 64.53.216.3:62952 0 0 yes 3107 0 187368120 QThread-61882 Inbound 1d20833 > 216.36.66.253:63518 0 0 yes 74 0 113276883 QThread-67257 Inbound 184d34 > 24.156.225.152:29435 0 0 yes 1 0 99159373 QThread-67657 Inbound 605a1e > 81.56.184.108:21360 0 0 yes 8 0 102771848 QThread-67943 Inbound 1acdc20 > 128.119.74.82:5529 1 217383 yes 2419 0 120988331 QThread-66469 Inbound > 217.231.72.106:63691 0 0 yes 291 0 75686751 QThread-68170 Inbound > 16fde80 > 213.208.127.216:9612 0 0 yes 1 0 61378527 QThread-67899 Inbound 1548f6c > 81.202.48.153:47361 0 0 yes 1 0 23219938 QThread-68397 Inbound 1e3a70b > 213.168.111.28:24519 0 0 yes 5 0 12796500 QThread-68417 Inbound 63aa8c > 217.79.103.2:64978 0 0 yes 2504 0 21038441 QThread-68280 Inbound 1655c93 > 212.94.37.56:1633 0 0 yes 4312 0 46984490 QThread-68082 Inbound d24393 > 217.231.93.106:29932 0 0 no 2 43993 44043 QThread-67887 Inbound 18781ba > 24.80.196.159:37721 0 0 no 2 42281 42281 QThread-68271 Inbound 155dc25 > 68.116.171.200:45803 0 0 no 2 42050 42070 QThread-68279 Inbound 1e81c7e > 63.207.61.194:4831 0 0 no 0 41970 41970 QThread-68043 Inbound 1a6a075 > 12.147.135.154:64849 0 0 no 3 41760 41760 QThread-68093 Inbound 70653f > 213.239.163.118:33102 0 0 no 2 39747 39877 QThread-68299 Inbound 107324d > 132.241.12.15:13861 0 0 no 2 38475 38505 QThread-68110 Inbound 1779213 > 213.93.166.66:59735 0 0 no 0 37133 37133 QThread-68410 Inbound 1172240 > 212.101.16.45:15038 0 0 no 2 36412 36462 QThread-67719 Inbound 1c99c69 > 212.127.187.206:22179 0 0 no 6 36132 74116 QThread-68391 Inbound dc8a70 > 194.236.28.174:4314 0 0 no 12 35992 138529 QThread-68267 Inbound 739cfb > 4.65.15.185:63366 0 0 no 2 35791 35801 QThread-67948 Inbound 13bd964 > 213.93.166.66:59741 0 0 no 1 34950 35311 QThread-67312 Inbound 3d6d49 > 213.93.166.66:59733 0 0 no 3 34950 40078 QThread-68381 Inbound d1f598 > 24.220.163.174:4379 0 0 no 9 34630 88527 QThread-68412 Inbound e10a7f > 80.145.87.114:1957 0 0 no 3 34439 34479 QThread-67877 Inbound 52f695 > 208.42.19.154:26800 0 0 no 0 33338 33338 QThread-67662 Inbound bace1d > 80.181.228.223:4718 0 0 no 3 32747 32787 QThread-67466 Inbound 31e0a7 > 12.251.17.210:2339 0 0 no 4 32677 54879 QThread-68387 Inbound 940a9b > 66.94.78.245:52048 0 0 no 2 30053 46166 QThread-68419 Inbound 1448709 > 128.255.153.66:53823 0 0 no 2 29773 29793 QThread-68404 Inbound 1fd7509 > 64.81.85.237:2920 0 0 no 2 29192 29202 QThread-68258 Inbound 10ecbec > 193.248.18.171:1594 0 0 no 3 28631 28801 QThread-68281 Inbound 6e1f9b > 66.93.82.106:62201 0 0 no 4 28130 30644 QThread-68265 Inbound 1dcff7d > 66.45.116.114:54115 0 0 no 2 25987 26027 QThread-68308 Inbound 970b4 > 65.28.194.229:1778 0 0 no 4 25497 32467 QThread-68031 Inbound 1c65c51 > 81.98.175.202:20443 0 0 no 18 25446 146571 QThread-66374 Inbound 9ab246 > 80.36.151.240:1939 0 0 no 10 25276 76540 QThread-67775 Inbound b26bf7 > 66.76.217.32:1406 0 0 no 2 25226 25266 QThread-67755 Inbound 1d6eea7 > 217.79.103.2:62273 0 0 no 0 25076 25076 QThread-68069 Inbound 1ce2e7a > 129.132.210.65:1578 0 0 no 19 24645 173940 QThread-68398 Inbound 169ec80 > 81.6.6.162:63861 0 0 no 2 24054 24075 QThread-67952 Inbound 14d706a > 129.187.77.195:34493 0 0 no 2 23604 23614 QThread-68323 Inbound 10f586f > 209.142.155.49:11836 0 0 no 2 23063 23524 QThread-68072 Inbound 770a03 > 63.207.61.194:4666 0 0 no 22 22953 153390 QThread-67880 Inbound 15eec93 > 217.79.103.2:62282 0 0 no 0 21901 21901 QThread-66745 Inbound 1325574 > 68.116.70.91:4549 0 0 no 1 21491 21501 QThread-67966 Inbound 1883c84 > 68.116.70.91:4546 0 0 no 3 21491 26488 QThread-68243 Inbound 13d25e0 > 209.142.155.49:23842 0 0 no 1 21090 21130 QThread-67863 Inbound 181ab4d > 209.142.155.49:19501 0 0 yes 26 0 3536936 QThread-68115 Inbound 18877b2 > 217.79.103.2:63458 0 0 yes 17945 0 25573082 QThread-68172 Inbound > 178d466 > 219.110.60.98:4666 0 0 no 3 18697 18777 QThread-68409 Inbound 156ac0 > 68.98.96.3:3657 0 0 no 4 18436 18667 QThread-68252 Inbound 138317 > 80.196.135.164:38006 0 0 no 2 17755 17785 QThread-68407 Inbound 103947d > 213.17.224.83:2662 0 0 no 3 17054 17054 QThread-27934 Inbound 9a345 > 128.214.100.40:56060 0 0 no 2 16744 16754 QThread-68420 Inbound be080e > 66.157.113.225:43851 0 0 no 2 16163 16163 QThread-68167 Inbound 2f36a > 213.46.231.136:43305 0 0 no 34 15983 258401 QThread-68263 Inbound 317261 > 80.131.84.27:40358 0 0 no 96 15282 420755 QThread-67949 Inbound 1de69e8 > 66.94.78.245:52093 0 0 no 0 15172 15172 QThread-68184 Inbound 1bae878 > 219.1.108.43:62816 0 0 no 10 14901 91241 QThread-68266 Inbound 1e75a5c > 131.215.127.44:36541 0 0 no 2 14471 14471 QThread-68154 Inbound 1f4451f > 217.5.33.189:61343 0 0 no 35 13099 184505 QThread-68253 Inbound 94ed > 193.150.225.41:42833 0 0 no 2 12898 12908 QThread-68413 Inbound 1e62563 > 12.237.135.80:53556 0 0 no 2 12348 12398 QThread-68204 Inbound a0878e > 80.192.4.23:50498 0 0 no 4 12248 26788 QThread-67989 Inbound 1239285 > 219.167.63.59:4536 0 0 no 3 12027 12037 QThread-67836 Inbound 688a8f > 64.2.42.86:3745 0 0 no 0 11977 11977 QThread-68026 Inbound 160becd > 216.138.210.150:2422 0 0 no 0 11637 11637 QThread-68306 Inbound 8ec1f5 > 80.126.80.32:2415 0 0 no 14 11216 100865 QThread-67581 Inbound 1f78251 > 66.6.202.96:61438 0 0 no 2 11076 11126 QThread-67591 Inbound 5ff58e > 217.0.84.241:49727 0 0 no 4 10004 11897 QThread-66751 Inbound 1f25464 > 216.222.64.199:4279 0 0 no 1477 8532 1604978 QThread-68295 Inbound > 12957a0 > 68.15.17.90:64982 0 0 no 10 8362 104160 QThread-68278 Inbound 1884a > 62.49.26.143:47480 0 0 no 794 8061 1955021 QThread-67891 Inbound 1f78526 > 68.168.211.82:1461 0 0 no 2 7991 8001 QThread-67862 Inbound 829a3d > 209.232.78.250:3724 0 0 no 2 7130 7150 QThread-66719 Inbound 5c32a6 > 24.201.170.212:1420 0 0 yes 629 0 4855912 QThread-67787 Inbound 1c71d8e > 66.93.82.106:62186 0 0 no 5 6820 27129 QThread-67977 Inbound 1013079 > 62.49.26.143:51309 0 0 no 0 6509 6509 QThread-68283 Inbound 1156ae9 > 213.39.179.58:42948 0 0 no 2 6089 6089 QThread-66742 Inbound 11e20ff > 217.227.157.157:36660 0 0 no 4 5768 34219 QThread-68106 Inbound 1086097 > 4.46.197.159:39031 0 0 no 258 5768 928795 QThread-68160 Inbound 1d0ca28 > 62.123.124.57:46642 0 0 no 2 5748 5758 QThread-68076 Inbound fd8103 > 194.204.62.159:2529 0 0 no 2 5448 5448 QThread-68180 Inbound 1211db4 > 80.192.4.23:50530 0 0 no 2 5147 5147 QThread-68411 Inbound 1252670 > 80.32.164.76:46084 0 0 no 20 5117 151237 QThread-67734 Inbound 1c08b48 > 209.210.55.3:3109 0 0 no 88 5107 398373 QThread-68268 Inbound 17a4e46 > 12.224.240.36:4541 0 0 no 36 4637 242038 QThread-68390 Inbound 11dd7a4 > 218.218.85.115:1378 0 0 no 10 4536 55199 QThread-68282 Inbound 1feadb5 > 216.87.111.151:2149 0 0 no 4 4176 7260 QThread-68293 Inbound 13e5ca3 > 217.126.206.240:2294 0 0 no 2940 4176 3391617 QThread-66741 Inbound > 6e292 > 212.104.155.82:1778 0 0 no 0 4146 4146 QThread-68022 Inbound 1340128 > 194.185.52.169:55702 0 0 no 4 4106 6750 QThread-67890 Inbound 137fe3a > 12.240.220.42:64818 0 0 no 48 4016 66906 QThread-67898 Inbound 1139cb8 > 129.217.168.59:34688 0 0 no 2 4006 4016 QThread-68255 Inbound 95fbc3 > 216.137.76.16:58027 0 0 no 6 3705 17385 QThread-68016 Inbound c3e88a > 12.238.83.186:4455 0 0 no 2 3585 3605 QThread-68143 Inbound 1d7cfe8 > 63.105.26.181:64940 0 0 no 2 3455 3505 QThread-68418 Inbound 1928c5d > 66.31.44.189:3237 0 0 no 2 3124 3134 QThread-67797 Inbound 3d63d1 > 217.227.196.18:1894 0 0 no 2 3094 3104 QThread-67867 Inbound 6438b8 > 138.88.84.46:15926 0 0 no 22 2844 152099 QThread-68275 Inbound 7130a3 > 62.49.26.143:51322 0 0 no 0 2594 2594 QThread-67750 Inbound 1c7de2f > 212.104.155.82:1792 0 0 no 0 2574 2574 QThread-67947 Inbound cc1e1f > 65.94.205.69:1694 0 0 no 23 1813 73175 QThread-68149 Inbound 1e649f5 > 209.239.242.44:65039 0 0 no 2 1802 2554 QThread-68376 Inbound cd9e40 > 12.241.145.72:4292 0 0 no 4 1642 16243 QThread-67847 Inbound 15090ad > 218.129.184.4:64956 0 0 no 4 1041 7771 QThread-67953 Inbound 4a7896 > 217.8.155.184:46746 0 0 no 3 931 941 QThread-68403 Inbound 132e801 > 68.15.17.90:65400 0 0 no 0 881 881 QThread-67884 Inbound 11dd8d7 > 68.58.206.237:4565 0 0 no 95 851 327881 QThread-68272 Inbound ee4497 > 68.97.44.181:4879 0 0 no 12 771 29653 QThread-68259 Inbound 8731c8 > 62.254.35.92:1144 0 0 no 24 441 72374 QThread-68406 Inbound fdd3b5 > 24.25.175.161:46238 0 0 no 12 390 27700 QThread-67492 Inbound 1d54bff > 209.221.197.10:57505 0 0 no 209 200 698655 QThread-68269 Inbound 14ce8e9 > > > And if it is interesting for anyone, my thread status: > Total pooled threads 363 > Available pooled threads 158 > Pooled threads in use 205 > > Class Threads > used > freenet.interfaces.LocalInterface$ConnectionShell 1 > freenet.node.states.data.DataStateInitiator 3 > freenet.Message: NodeAnnouncement 1 > freenet.interfaces.PublicInterface$ConnectionShell 191 > freenet.OpenConnectionManager$ConnectionJob 9 > > > /N > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED]] On Behalf Of Niklas Bergh > Sent: den 26 januari 2003 21:50 > To: [EMAIL PROTECTED] > Subject: Re: [freenet-support] Node suddenly ceases traffic > > > Ed Tomlinson said in the devl-list: > > >Here is a little problem noticed today. There seems to be a case where > > >a thread can hang when waiting for data. This is what I see. > > > >>From the connections applet: > > > >Peer addr Send Count Send Size Receiving Messages Idle time Lifetime > >Thread > Type ID > >64.53.216.3:64638 0 0 yes 1 0 13897805 FThread-41 Inbound bc7f3f > Could this be the very same problem? All my 114 idle inbound connections > looked something like this (not v656 but other versions). /N > ----- Original Message ----- > From: "Niklas Bergh" <[EMAIL PROTECTED]> > To: <[EMAIL PROTECTED]> > Sent: Friday, January 24, 2003 11:33 PM > Subject: Re: [freenet-support] Node suddenly ceases traffic > > > > Well, one of my nodes stopped doing stuff now too... Its log was 2 > gigabytes > > even though it was only set to log at 'minor'... It survived 5 days > > and about 50 gigabytes of transfered data. One curious thing that I > > noted > > though: Routingtime was 0, queued jobs where a whole bunch and load > was at > > 100%, just as usual. The unusual thing was that the node said it had > 114 > > active inbound connections (but the sendqueue was 0 bytes, compared to > a > > more normal day when the sendqueue is a couple of tens of megs). Could > this > > say something new? > > > > /N > > > > > > _______________________________________________ > > support mailing list > > [EMAIL PROTECTED] > > http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support > > > > > _______________________________________________ > support mailing list > [EMAIL PROTECTED] > http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support > > > _______________________________________________ > support mailing list > [EMAIL PROTECTED] > http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support > -- Matthew Toseland [EMAIL PROTECTED][EMAIL PROTECTED] Full time freenet hacker. http://freenetproject.org/ Freenet Distribution Node (temporary) at http://amphibian.dyndns.org:8889/A8dz8aYheps/ ICTHUS.
msg02798/pgp00000.pgp
Description: PGP signature
