Interesting... ideas? ----- Forwarded message from Niklas Bergh <[EMAIL PROTECTED]> -----
From: "Niklas Bergh" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Subject: RE: [freenet-support] Node suddenly ceases traffic Importance: Normal X-BeenThere: [EMAIL PROTECTED] X-Mailman-Version: 2.0.13 Reply-To: [EMAIL PROTECTED] List-Help: <mailto:[EMAIL PROTECTED]?subject=help> List-Post: <mailto:[EMAIL PROTECTED]> List-Subscribe: <http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support>, <mailto:[EMAIL PROTECTED]?subject=subscribe> List-Id: <support.freenetproject.org> List-Unsubscribe: <http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support>, <mailto:[EMAIL PROTECTED]?subject=unsubscribe> List-Archive: <http://hawk.freenetproject.org/pipermail/support/> I would like to reinforce my previous speculation. One of my machines are running a v551 fred node, current uptime is about 6 days. The machines processor usage is somewhere around 10 percent or so but fred is reporting 90-95 percent load and freds network usage is close to zero. Fred has a constant routing time of 0 ms and 'Active pooled jobs' is varying between 195 and 210 (so the node is not completely dead, just almost). The node reports 120 open inbound connections and 0 open outbound and has a sendqueue of only a couple of 100 kilobytes (on a normal day the sendqueue should be around 10-20 megs). My guess is that by tomorrow or so my node will be completely idle with a whole bunch of open connections. My log file shows in the order of 5-10/second of these: 2003-feb-04 12:19:28 (freenet.ConnectionHandler, QThread-67755): DataRequest @ e7426ffa8ca356c7 <- 81.97.244.57:18887 2003-feb-04 12:19:28 (freenet.ConnectionHandler, QThread-68154): QueryRejected @ e7426ffa8ca356c7 -> 81.97.244.57:18887 And very few (1 every 5 seconds or so) of these: 2003-feb-04 12:19:28 (freenet.interfaces.PublicInterface, Interface # tcp/14166): Accepted connection: tcpconnection: 68.36.137.144:1142 I guess that's not very good for the network?! I am also seeing quite a bit of 'I/O error on socket' messages in the log, say, about one every 5 seconds or so. Is that normal? Looking at the data from my ocm (see below), I am pretty much able to divide the connections into 3 different categories: 1, healthy idle: Connections open that fred currently are neither sending data to or recieveing data from and none of them seems to be especially long-lived. Neither do they have a very long 'Idle time', but it is pretty much always larger than 0. 2, healty busy: Connections that I am either reciving data from, or sending data to or both at the same time, all have 0 'Idle time' and NOT SO VERY long 'LifeTime'. 3, dead: Connections that fred is expecting to either reciving data from, or send data to or both at the same time, all have 0 'Idle time' and VERY long 'LifeTime'. The categorization above plus fact that the first and the 8:th connection (see table below), both category 3, has a sendqueue of 5000 and 217383 bytes respectively and have had exactly so for a couple of hours now leads me to the conlusion that: If fred assumes that it will soon receive data from or send data to a connection and that that connection somehow dies then this will not be detected by fred. This will lead to that given enough fred uptime all connections will become category three. At date:Tue Feb 04 12:28:43 GMT+01:00 2003 Number of open connections: 119 Number of outbound connections: 0 Number of inbound connections: 119 Bytes waiting to be sent: 222383 Peer addr Send Count Send Size Receiving Messages Idle time Lifetime Thread Type ID 129.187.77.195:37175 1 5000 yes 1563 0 428218253 QThread-32403 Inbound 13c197a 12.230.157.49:3739 0 0 yes 1 0 374236231 QThread-47873 Inbound 9a912 195.64.94.237:4598 0 0 yes 8 0 369743741 QThread-48562 Inbound 3b1a4f 64.53.216.3:62952 0 0 yes 3107 0 187368120 QThread-61882 Inbound 1d20833 216.36.66.253:63518 0 0 yes 74 0 113276883 QThread-67257 Inbound 184d34 24.156.225.152:29435 0 0 yes 1 0 99159373 QThread-67657 Inbound 605a1e 81.56.184.108:21360 0 0 yes 8 0 102771848 QThread-67943 Inbound 1acdc20 128.119.74.82:5529 1 217383 yes 2419 0 120988331 QThread-66469 Inbound 217.231.72.106:63691 0 0 yes 291 0 75686751 QThread-68170 Inbound 16fde80 213.208.127.216:9612 0 0 yes 1 0 61378527 QThread-67899 Inbound 1548f6c 81.202.48.153:47361 0 0 yes 1 0 23219938 QThread-68397 Inbound 1e3a70b 213.168.111.28:24519 0 0 yes 5 0 12796500 QThread-68417 Inbound 63aa8c 217.79.103.2:64978 0 0 yes 2504 0 21038441 QThread-68280 Inbound 1655c93 212.94.37.56:1633 0 0 yes 4312 0 46984490 QThread-68082 Inbound d24393 217.231.93.106:29932 0 0 no 2 43993 44043 QThread-67887 Inbound 18781ba 24.80.196.159:37721 0 0 no 2 42281 42281 QThread-68271 Inbound 155dc25 68.116.171.200:45803 0 0 no 2 42050 42070 QThread-68279 Inbound 1e81c7e 63.207.61.194:4831 0 0 no 0 41970 41970 QThread-68043 Inbound 1a6a075 12.147.135.154:64849 0 0 no 3 41760 41760 QThread-68093 Inbound 70653f 213.239.163.118:33102 0 0 no 2 39747 39877 QThread-68299 Inbound 107324d 132.241.12.15:13861 0 0 no 2 38475 38505 QThread-68110 Inbound 1779213 213.93.166.66:59735 0 0 no 0 37133 37133 QThread-68410 Inbound 1172240 212.101.16.45:15038 0 0 no 2 36412 36462 QThread-67719 Inbound 1c99c69 212.127.187.206:22179 0 0 no 6 36132 74116 QThread-68391 Inbound dc8a70 194.236.28.174:4314 0 0 no 12 35992 138529 QThread-68267 Inbound 739cfb 4.65.15.185:63366 0 0 no 2 35791 35801 QThread-67948 Inbound 13bd964 213.93.166.66:59741 0 0 no 1 34950 35311 QThread-67312 Inbound 3d6d49 213.93.166.66:59733 0 0 no 3 34950 40078 QThread-68381 Inbound d1f598 24.220.163.174:4379 0 0 no 9 34630 88527 QThread-68412 Inbound e10a7f 80.145.87.114:1957 0 0 no 3 34439 34479 QThread-67877 Inbound 52f695 208.42.19.154:26800 0 0 no 0 33338 33338 QThread-67662 Inbound bace1d 80.181.228.223:4718 0 0 no 3 32747 32787 QThread-67466 Inbound 31e0a7 12.251.17.210:2339 0 0 no 4 32677 54879 QThread-68387 Inbound 940a9b 66.94.78.245:52048 0 0 no 2 30053 46166 QThread-68419 Inbound 1448709 128.255.153.66:53823 0 0 no 2 29773 29793 QThread-68404 Inbound 1fd7509 64.81.85.237:2920 0 0 no 2 29192 29202 QThread-68258 Inbound 10ecbec 193.248.18.171:1594 0 0 no 3 28631 28801 QThread-68281 Inbound 6e1f9b 66.93.82.106:62201 0 0 no 4 28130 30644 QThread-68265 Inbound 1dcff7d 66.45.116.114:54115 0 0 no 2 25987 26027 QThread-68308 Inbound 970b4 65.28.194.229:1778 0 0 no 4 25497 32467 QThread-68031 Inbound 1c65c51 81.98.175.202:20443 0 0 no 18 25446 146571 QThread-66374 Inbound 9ab246 80.36.151.240:1939 0 0 no 10 25276 76540 QThread-67775 Inbound b26bf7 66.76.217.32:1406 0 0 no 2 25226 25266 QThread-67755 Inbound 1d6eea7 217.79.103.2:62273 0 0 no 0 25076 25076 QThread-68069 Inbound 1ce2e7a 129.132.210.65:1578 0 0 no 19 24645 173940 QThread-68398 Inbound 169ec80 81.6.6.162:63861 0 0 no 2 24054 24075 QThread-67952 Inbound 14d706a 129.187.77.195:34493 0 0 no 2 23604 23614 QThread-68323 Inbound 10f586f 209.142.155.49:11836 0 0 no 2 23063 23524 QThread-68072 Inbound 770a03 63.207.61.194:4666 0 0 no 22 22953 153390 QThread-67880 Inbound 15eec93 217.79.103.2:62282 0 0 no 0 21901 21901 QThread-66745 Inbound 1325574 68.116.70.91:4549 0 0 no 1 21491 21501 QThread-67966 Inbound 1883c84 68.116.70.91:4546 0 0 no 3 21491 26488 QThread-68243 Inbound 13d25e0 209.142.155.49:23842 0 0 no 1 21090 21130 QThread-67863 Inbound 181ab4d 209.142.155.49:19501 0 0 yes 26 0 3536936 QThread-68115 Inbound 18877b2 217.79.103.2:63458 0 0 yes 17945 0 25573082 QThread-68172 Inbound 178d466 219.110.60.98:4666 0 0 no 3 18697 18777 QThread-68409 Inbound 156ac0 68.98.96.3:3657 0 0 no 4 18436 18667 QThread-68252 Inbound 138317 80.196.135.164:38006 0 0 no 2 17755 17785 QThread-68407 Inbound 103947d 213.17.224.83:2662 0 0 no 3 17054 17054 QThread-27934 Inbound 9a345 128.214.100.40:56060 0 0 no 2 16744 16754 QThread-68420 Inbound be080e 66.157.113.225:43851 0 0 no 2 16163 16163 QThread-68167 Inbound 2f36a 213.46.231.136:43305 0 0 no 34 15983 258401 QThread-68263 Inbound 317261 80.131.84.27:40358 0 0 no 96 15282 420755 QThread-67949 Inbound 1de69e8 66.94.78.245:52093 0 0 no 0 15172 15172 QThread-68184 Inbound 1bae878 219.1.108.43:62816 0 0 no 10 14901 91241 QThread-68266 Inbound 1e75a5c 131.215.127.44:36541 0 0 no 2 14471 14471 QThread-68154 Inbound 1f4451f 217.5.33.189:61343 0 0 no 35 13099 184505 QThread-68253 Inbound 94ed 193.150.225.41:42833 0 0 no 2 12898 12908 QThread-68413 Inbound 1e62563 12.237.135.80:53556 0 0 no 2 12348 12398 QThread-68204 Inbound a0878e 80.192.4.23:50498 0 0 no 4 12248 26788 QThread-67989 Inbound 1239285 219.167.63.59:4536 0 0 no 3 12027 12037 QThread-67836 Inbound 688a8f 64.2.42.86:3745 0 0 no 0 11977 11977 QThread-68026 Inbound 160becd 216.138.210.150:2422 0 0 no 0 11637 11637 QThread-68306 Inbound 8ec1f5 80.126.80.32:2415 0 0 no 14 11216 100865 QThread-67581 Inbound 1f78251 66.6.202.96:61438 0 0 no 2 11076 11126 QThread-67591 Inbound 5ff58e 217.0.84.241:49727 0 0 no 4 10004 11897 QThread-66751 Inbound 1f25464 216.222.64.199:4279 0 0 no 1477 8532 1604978 QThread-68295 Inbound 12957a0 68.15.17.90:64982 0 0 no 10 8362 104160 QThread-68278 Inbound 1884a 62.49.26.143:47480 0 0 no 794 8061 1955021 QThread-67891 Inbound 1f78526 68.168.211.82:1461 0 0 no 2 7991 8001 QThread-67862 Inbound 829a3d 209.232.78.250:3724 0 0 no 2 7130 7150 QThread-66719 Inbound 5c32a6 24.201.170.212:1420 0 0 yes 629 0 4855912 QThread-67787 Inbound 1c71d8e 66.93.82.106:62186 0 0 no 5 6820 27129 QThread-67977 Inbound 1013079 62.49.26.143:51309 0 0 no 0 6509 6509 QThread-68283 Inbound 1156ae9 213.39.179.58:42948 0 0 no 2 6089 6089 QThread-66742 Inbound 11e20ff 217.227.157.157:36660 0 0 no 4 5768 34219 QThread-68106 Inbound 1086097 4.46.197.159:39031 0 0 no 258 5768 928795 QThread-68160 Inbound 1d0ca28 62.123.124.57:46642 0 0 no 2 5748 5758 QThread-68076 Inbound fd8103 194.204.62.159:2529 0 0 no 2 5448 5448 QThread-68180 Inbound 1211db4 80.192.4.23:50530 0 0 no 2 5147 5147 QThread-68411 Inbound 1252670 80.32.164.76:46084 0 0 no 20 5117 151237 QThread-67734 Inbound 1c08b48 209.210.55.3:3109 0 0 no 88 5107 398373 QThread-68268 Inbound 17a4e46 12.224.240.36:4541 0 0 no 36 4637 242038 QThread-68390 Inbound 11dd7a4 218.218.85.115:1378 0 0 no 10 4536 55199 QThread-68282 Inbound 1feadb5 216.87.111.151:2149 0 0 no 4 4176 7260 QThread-68293 Inbound 13e5ca3 217.126.206.240:2294 0 0 no 2940 4176 3391617 QThread-66741 Inbound 6e292 212.104.155.82:1778 0 0 no 0 4146 4146 QThread-68022 Inbound 1340128 194.185.52.169:55702 0 0 no 4 4106 6750 QThread-67890 Inbound 137fe3a 12.240.220.42:64818 0 0 no 48 4016 66906 QThread-67898 Inbound 1139cb8 129.217.168.59:34688 0 0 no 2 4006 4016 QThread-68255 Inbound 95fbc3 216.137.76.16:58027 0 0 no 6 3705 17385 QThread-68016 Inbound c3e88a 12.238.83.186:4455 0 0 no 2 3585 3605 QThread-68143 Inbound 1d7cfe8 63.105.26.181:64940 0 0 no 2 3455 3505 QThread-68418 Inbound 1928c5d 66.31.44.189:3237 0 0 no 2 3124 3134 QThread-67797 Inbound 3d63d1 217.227.196.18:1894 0 0 no 2 3094 3104 QThread-67867 Inbound 6438b8 138.88.84.46:15926 0 0 no 22 2844 152099 QThread-68275 Inbound 7130a3 62.49.26.143:51322 0 0 no 0 2594 2594 QThread-67750 Inbound 1c7de2f 212.104.155.82:1792 0 0 no 0 2574 2574 QThread-67947 Inbound cc1e1f 65.94.205.69:1694 0 0 no 23 1813 73175 QThread-68149 Inbound 1e649f5 209.239.242.44:65039 0 0 no 2 1802 2554 QThread-68376 Inbound cd9e40 12.241.145.72:4292 0 0 no 4 1642 16243 QThread-67847 Inbound 15090ad 218.129.184.4:64956 0 0 no 4 1041 7771 QThread-67953 Inbound 4a7896 217.8.155.184:46746 0 0 no 3 931 941 QThread-68403 Inbound 132e801 68.15.17.90:65400 0 0 no 0 881 881 QThread-67884 Inbound 11dd8d7 68.58.206.237:4565 0 0 no 95 851 327881 QThread-68272 Inbound ee4497 68.97.44.181:4879 0 0 no 12 771 29653 QThread-68259 Inbound 8731c8 62.254.35.92:1144 0 0 no 24 441 72374 QThread-68406 Inbound fdd3b5 24.25.175.161:46238 0 0 no 12 390 27700 QThread-67492 Inbound 1d54bff 209.221.197.10:57505 0 0 no 209 200 698655 QThread-68269 Inbound 14ce8e9 And if it is interesting for anyone, my thread status: Total pooled threads 363 Available pooled threads 158 Pooled threads in use 205 Class Threads used freenet.interfaces.LocalInterface$ConnectionShell 1 freenet.node.states.data.DataStateInitiator 3 freenet.Message: NodeAnnouncement 1 freenet.interfaces.PublicInterface$ConnectionShell 191 freenet.OpenConnectionManager$ConnectionJob 9 /N -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Niklas Bergh Sent: den 26 januari 2003 21:50 To: [EMAIL PROTECTED] Subject: Re: [freenet-support] Node suddenly ceases traffic Ed Tomlinson said in the devl-list: >Here is a little problem noticed today. There seems to be a case where >a thread can hang when waiting for data. This is what I see. > >>From the connections applet: > >Peer addr Send Count Send Size Receiving Messages Idle time Lifetime >Thread Type ID >64.53.216.3:64638 0 0 yes 1 0 13897805 FThread-41 Inbound bc7f3f Could this be the very same problem? All my 114 idle inbound connections looked something like this (not v656 but other versions). /N ----- Original Message ----- From: "Niklas Bergh" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Friday, January 24, 2003 11:33 PM Subject: Re: [freenet-support] Node suddenly ceases traffic > Well, one of my nodes stopped doing stuff now too... Its log was 2 gigabytes > even though it was only set to log at 'minor'... It survived 5 days > and about 50 gigabytes of transfered data. One curious thing that I > noted > though: Routingtime was 0, queued jobs where a whole bunch and load was at > 100%, just as usual. The unusual thing was that the node said it had 114 > active inbound connections (but the sendqueue was 0 bytes, compared to a > more normal day when the sendqueue is a couple of tens of megs). Could this > say something new? > > /N > > > _______________________________________________ > support mailing list > [EMAIL PROTECTED] > http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support > _______________________________________________ support mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support _______________________________________________ support mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support ----- End forwarded message ----- -- Matthew Toseland [EMAIL PROTECTED][EMAIL PROTECTED] Full time freenet hacker. http://freenetproject.org/ Freenet Distribution Node (temporary) at http://amphibian.dyndns.org:8889/A8dz8aYheps/ ICTHUS.
msg06565/pgp00000.pgp
Description: PGP signature
