Here is another clue: on a regular basis (it has happened at least 5 times in the last two months), I find my node in the morning with the max amount of connections open, all inbound. In that state, it does nothing useful. If I try to access Freesites, the browser will send the request and wait forever for a reply. All I have to do is stop and start my node and everything is fine again for a few days...
Yves
On Tuesday, February 4, 2003, at 05:55 PM, Matthew Toseland wrote:
Interesting... ideas?
----- Forwarded message from Niklas Bergh <[EMAIL PROTECTED]> -----
From: "Niklas Bergh" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Subject: RE: [freenet-support] Node suddenly ceases traffic
Importance: Normal
X-BeenThere: [EMAIL PROTECTED]
X-Mailman-Version: 2.0.13
Reply-To: [EMAIL PROTECTED]
List-Help: <mailto:[EMAIL PROTECTED]?subject=help>
List-Post: <mailto:[EMAIL PROTECTED]>
List-Subscribe: <http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support>,
<mailto:[EMAIL PROTECTED]?subject=subscribe>
List-Id: <support.freenetproject.org>
List-Unsubscribe: <http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support>,
<mailto:[EMAIL PROTECTED]?subject=unsubscribe>
List-Archive: <http://hawk.freenetproject.org/pipermail/support/>
I would like to reinforce my previous speculation.
One of my machines are running a v551 fred node, current uptime is about
6 days. The machines processor usage is somewhere around 10 percent or
so but fred is reporting 90-95 percent load and freds network usage is
close to zero. Fred has a constant routing time of 0 ms and 'Active
pooled jobs' is varying between 195 and 210 (so the node is not
completely dead, just almost). The node reports 120 open inbound
connections and 0 open outbound and has a sendqueue of only a couple of
100 kilobytes (on a normal day the sendqueue should be around 10-20
megs). My guess is that by tomorrow or so my node will be completely
idle with a whole bunch of open connections.
My log file shows in the order of 5-10/second of these:
2003-feb-04 12:19:28 (freenet.ConnectionHandler, QThread-67755):
DataRequest @ e7426ffa8ca356c7 <- 81.97.244.57:18887
2003-feb-04 12:19:28 (freenet.ConnectionHandler, QThread-68154):
QueryRejected @ e7426ffa8ca356c7 -> 81.97.244.57:18887
And very few (1 every 5 seconds or so) of these:
2003-feb-04 12:19:28 (freenet.interfaces.PublicInterface, Interface #
tcp/14166): Accepted connection: tcpconnection: 68.36.137.144:1142
I guess that's not very good for the network?!
I am also seeing quite a bit of 'I/O error on socket' messages in the
log, say, about one every 5 seconds or so. Is that normal?
Looking at the data from my ocm (see below), I am pretty much able to
divide the connections into 3 different categories:
1, healthy idle: Connections open that fred currently are neither
sending data to or recieveing data from and none of them seems to be
especially long-lived. Neither do they have a very long 'Idle time', but
it is pretty much always larger than 0.
2, healty busy: Connections that I am either reciving data from, or
sending data to or both at the same time, all have 0 'Idle time' and NOT
SO VERY long 'LifeTime'.
3, dead: Connections that fred is expecting to either reciving data
from, or send data to or both at the same time, all have 0 'Idle time'
and VERY long 'LifeTime'.
The categorization above plus fact that the first and the 8:th
connection (see table below), both category 3, has a sendqueue of 5000
and 217383 bytes respectively and have had exactly so for a couple of
hours now leads me to the conlusion that:
If fred assumes that it will soon receive data from or send data to a
connection and that that connection somehow dies then this will not be
detected by fred. This will lead to that given enough fred uptime all
connections will become category three.
At date:Tue Feb 04 12:28:43 GMT+01:00 2003
Number of open connections: 119
Number of outbound connections: 0
Number of inbound connections: 119
Bytes waiting to be sent: 222383
Peer addr Send Count Send Size Receiving Messages Idle time Lifetime
Thread Type ID
129.187.77.195:37175 1 5000 yes 1563 0 428218253 QThread-32403 Inbound
13c197a
12.230.157.49:3739 0 0 yes 1 0 374236231 QThread-47873 Inbound 9a912
195.64.94.237:4598 0 0 yes 8 0 369743741 QThread-48562 Inbound 3b1a4f
64.53.216.3:62952 0 0 yes 3107 0 187368120 QThread-61882 Inbound 1d20833
216.36.66.253:63518 0 0 yes 74 0 113276883 QThread-67257 Inbound 184d34
24.156.225.152:29435 0 0 yes 1 0 99159373 QThread-67657 Inbound 605a1e
81.56.184.108:21360 0 0 yes 8 0 102771848 QThread-67943 Inbound 1acdc20
128.119.74.82:5529 1 217383 yes 2419 0 120988331 QThread-66469 Inbound
217.231.72.106:63691 0 0 yes 291 0 75686751 QThread-68170 Inbound
16fde80
213.208.127.216:9612 0 0 yes 1 0 61378527 QThread-67899 Inbound 1548f6c
81.202.48.153:47361 0 0 yes 1 0 23219938 QThread-68397 Inbound 1e3a70b
213.168.111.28:24519 0 0 yes 5 0 12796500 QThread-68417 Inbound 63aa8c
217.79.103.2:64978 0 0 yes 2504 0 21038441 QThread-68280 Inbound 1655c93
212.94.37.56:1633 0 0 yes 4312 0 46984490 QThread-68082 Inbound d24393
217.231.93.106:29932 0 0 no 2 43993 44043 QThread-67887 Inbound 18781ba
24.80.196.159:37721 0 0 no 2 42281 42281 QThread-68271 Inbound 155dc25
68.116.171.200:45803 0 0 no 2 42050 42070 QThread-68279 Inbound 1e81c7e
63.207.61.194:4831 0 0 no 0 41970 41970 QThread-68043 Inbound 1a6a075
12.147.135.154:64849 0 0 no 3 41760 41760 QThread-68093 Inbound 70653f
213.239.163.118:33102 0 0 no 2 39747 39877 QThread-68299 Inbound 107324d
132.241.12.15:13861 0 0 no 2 38475 38505 QThread-68110 Inbound 1779213
213.93.166.66:59735 0 0 no 0 37133 37133 QThread-68410 Inbound 1172240
212.101.16.45:15038 0 0 no 2 36412 36462 QThread-67719 Inbound 1c99c69
212.127.187.206:22179 0 0 no 6 36132 74116 QThread-68391 Inbound dc8a70
194.236.28.174:4314 0 0 no 12 35992 138529 QThread-68267 Inbound 739cfb
4.65.15.185:63366 0 0 no 2 35791 35801 QThread-67948 Inbound 13bd964
213.93.166.66:59741 0 0 no 1 34950 35311 QThread-67312 Inbound 3d6d49
213.93.166.66:59733 0 0 no 3 34950 40078 QThread-68381 Inbound d1f598
24.220.163.174:4379 0 0 no 9 34630 88527 QThread-68412 Inbound e10a7f
80.145.87.114:1957 0 0 no 3 34439 34479 QThread-67877 Inbound 52f695
208.42.19.154:26800 0 0 no 0 33338 33338 QThread-67662 Inbound bace1d
80.181.228.223:4718 0 0 no 3 32747 32787 QThread-67466 Inbound 31e0a7
12.251.17.210:2339 0 0 no 4 32677 54879 QThread-68387 Inbound 940a9b
66.94.78.245:52048 0 0 no 2 30053 46166 QThread-68419 Inbound 1448709
128.255.153.66:53823 0 0 no 2 29773 29793 QThread-68404 Inbound 1fd7509
64.81.85.237:2920 0 0 no 2 29192 29202 QThread-68258 Inbound 10ecbec
193.248.18.171:1594 0 0 no 3 28631 28801 QThread-68281 Inbound 6e1f9b
66.93.82.106:62201 0 0 no 4 28130 30644 QThread-68265 Inbound 1dcff7d
66.45.116.114:54115 0 0 no 2 25987 26027 QThread-68308 Inbound 970b4
65.28.194.229:1778 0 0 no 4 25497 32467 QThread-68031 Inbound 1c65c51
81.98.175.202:20443 0 0 no 18 25446 146571 QThread-66374 Inbound 9ab246
80.36.151.240:1939 0 0 no 10 25276 76540 QThread-67775 Inbound b26bf7
66.76.217.32:1406 0 0 no 2 25226 25266 QThread-67755 Inbound 1d6eea7
217.79.103.2:62273 0 0 no 0 25076 25076 QThread-68069 Inbound 1ce2e7a
129.132.210.65:1578 0 0 no 19 24645 173940 QThread-68398 Inbound 169ec80
81.6.6.162:63861 0 0 no 2 24054 24075 QThread-67952 Inbound 14d706a
129.187.77.195:34493 0 0 no 2 23604 23614 QThread-68323 Inbound 10f586f
209.142.155.49:11836 0 0 no 2 23063 23524 QThread-68072 Inbound 770a03
63.207.61.194:4666 0 0 no 22 22953 153390 QThread-67880 Inbound 15eec93
217.79.103.2:62282 0 0 no 0 21901 21901 QThread-66745 Inbound 1325574
68.116.70.91:4549 0 0 no 1 21491 21501 QThread-67966 Inbound 1883c84
68.116.70.91:4546 0 0 no 3 21491 26488 QThread-68243 Inbound 13d25e0
209.142.155.49:23842 0 0 no 1 21090 21130 QThread-67863 Inbound 181ab4d
209.142.155.49:19501 0 0 yes 26 0 3536936 QThread-68115 Inbound 18877b2
217.79.103.2:63458 0 0 yes 17945 0 25573082 QThread-68172 Inbound
178d466
219.110.60.98:4666 0 0 no 3 18697 18777 QThread-68409 Inbound 156ac0
68.98.96.3:3657 0 0 no 4 18436 18667 QThread-68252 Inbound 138317
80.196.135.164:38006 0 0 no 2 17755 17785 QThread-68407 Inbound 103947d
213.17.224.83:2662 0 0 no 3 17054 17054 QThread-27934 Inbound 9a345
128.214.100.40:56060 0 0 no 2 16744 16754 QThread-68420 Inbound be080e
66.157.113.225:43851 0 0 no 2 16163 16163 QThread-68167 Inbound 2f36a
213.46.231.136:43305 0 0 no 34 15983 258401 QThread-68263 Inbound 317261
80.131.84.27:40358 0 0 no 96 15282 420755 QThread-67949 Inbound 1de69e8
66.94.78.245:52093 0 0 no 0 15172 15172 QThread-68184 Inbound 1bae878
219.1.108.43:62816 0 0 no 10 14901 91241 QThread-68266 Inbound 1e75a5c
131.215.127.44:36541 0 0 no 2 14471 14471 QThread-68154 Inbound 1f4451f
217.5.33.189:61343 0 0 no 35 13099 184505 QThread-68253 Inbound 94ed
193.150.225.41:42833 0 0 no 2 12898 12908 QThread-68413 Inbound 1e62563
12.237.135.80:53556 0 0 no 2 12348 12398 QThread-68204 Inbound a0878e
80.192.4.23:50498 0 0 no 4 12248 26788 QThread-67989 Inbound 1239285
219.167.63.59:4536 0 0 no 3 12027 12037 QThread-67836 Inbound 688a8f
64.2.42.86:3745 0 0 no 0 11977 11977 QThread-68026 Inbound 160becd
216.138.210.150:2422 0 0 no 0 11637 11637 QThread-68306 Inbound 8ec1f5
80.126.80.32:2415 0 0 no 14 11216 100865 QThread-67581 Inbound 1f78251
66.6.202.96:61438 0 0 no 2 11076 11126 QThread-67591 Inbound 5ff58e
217.0.84.241:49727 0 0 no 4 10004 11897 QThread-66751 Inbound 1f25464
216.222.64.199:4279 0 0 no 1477 8532 1604978 QThread-68295 Inbound
12957a0
68.15.17.90:64982 0 0 no 10 8362 104160 QThread-68278 Inbound 1884a
62.49.26.143:47480 0 0 no 794 8061 1955021 QThread-67891 Inbound 1f78526
68.168.211.82:1461 0 0 no 2 7991 8001 QThread-67862 Inbound 829a3d
209.232.78.250:3724 0 0 no 2 7130 7150 QThread-66719 Inbound 5c32a6
24.201.170.212:1420 0 0 yes 629 0 4855912 QThread-67787 Inbound 1c71d8e
66.93.82.106:62186 0 0 no 5 6820 27129 QThread-67977 Inbound 1013079
62.49.26.143:51309 0 0 no 0 6509 6509 QThread-68283 Inbound 1156ae9
213.39.179.58:42948 0 0 no 2 6089 6089 QThread-66742 Inbound 11e20ff
217.227.157.157:36660 0 0 no 4 5768 34219 QThread-68106 Inbound 1086097
4.46.197.159:39031 0 0 no 258 5768 928795 QThread-68160 Inbound 1d0ca28
62.123.124.57:46642 0 0 no 2 5748 5758 QThread-68076 Inbound fd8103
194.204.62.159:2529 0 0 no 2 5448 5448 QThread-68180 Inbound 1211db4
80.192.4.23:50530 0 0 no 2 5147 5147 QThread-68411 Inbound 1252670
80.32.164.76:46084 0 0 no 20 5117 151237 QThread-67734 Inbound 1c08b48
209.210.55.3:3109 0 0 no 88 5107 398373 QThread-68268 Inbound 17a4e46
12.224.240.36:4541 0 0 no 36 4637 242038 QThread-68390 Inbound 11dd7a4
218.218.85.115:1378 0 0 no 10 4536 55199 QThread-68282 Inbound 1feadb5
216.87.111.151:2149 0 0 no 4 4176 7260 QThread-68293 Inbound 13e5ca3
217.126.206.240:2294 0 0 no 2940 4176 3391617 QThread-66741 Inbound
6e292
212.104.155.82:1778 0 0 no 0 4146 4146 QThread-68022 Inbound 1340128
194.185.52.169:55702 0 0 no 4 4106 6750 QThread-67890 Inbound 137fe3a
12.240.220.42:64818 0 0 no 48 4016 66906 QThread-67898 Inbound 1139cb8
129.217.168.59:34688 0 0 no 2 4006 4016 QThread-68255 Inbound 95fbc3
216.137.76.16:58027 0 0 no 6 3705 17385 QThread-68016 Inbound c3e88a
12.238.83.186:4455 0 0 no 2 3585 3605 QThread-68143 Inbound 1d7cfe8
63.105.26.181:64940 0 0 no 2 3455 3505 QThread-68418 Inbound 1928c5d
66.31.44.189:3237 0 0 no 2 3124 3134 QThread-67797 Inbound 3d63d1
217.227.196.18:1894 0 0 no 2 3094 3104 QThread-67867 Inbound 6438b8
138.88.84.46:15926 0 0 no 22 2844 152099 QThread-68275 Inbound 7130a3
62.49.26.143:51322 0 0 no 0 2594 2594 QThread-67750 Inbound 1c7de2f
212.104.155.82:1792 0 0 no 0 2574 2574 QThread-67947 Inbound cc1e1f
65.94.205.69:1694 0 0 no 23 1813 73175 QThread-68149 Inbound 1e649f5
209.239.242.44:65039 0 0 no 2 1802 2554 QThread-68376 Inbound cd9e40
12.241.145.72:4292 0 0 no 4 1642 16243 QThread-67847 Inbound 15090ad
218.129.184.4:64956 0 0 no 4 1041 7771 QThread-67953 Inbound 4a7896
217.8.155.184:46746 0 0 no 3 931 941 QThread-68403 Inbound 132e801
68.15.17.90:65400 0 0 no 0 881 881 QThread-67884 Inbound 11dd8d7
68.58.206.237:4565 0 0 no 95 851 327881 QThread-68272 Inbound ee4497
68.97.44.181:4879 0 0 no 12 771 29653 QThread-68259 Inbound 8731c8
62.254.35.92:1144 0 0 no 24 441 72374 QThread-68406 Inbound fdd3b5
24.25.175.161:46238 0 0 no 12 390 27700 QThread-67492 Inbound 1d54bff
209.221.197.10:57505 0 0 no 209 200 698655 QThread-68269 Inbound 14ce8e9
And if it is interesting for anyone, my thread status:
Total pooled threads 363
Available pooled threads 158
Pooled threads in use 205
Class Threads
used
freenet.interfaces.LocalInterface$ConnectionShell 1
freenet.node.states.data.DataStateInitiator 3
freenet.Message: NodeAnnouncement 1
freenet.interfaces.PublicInterface$ConnectionShell 191
freenet.OpenConnectionManager$ConnectionJob 9
/N
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED]] On Behalf Of Niklas Bergh
Sent: den 26 januari 2003 21:50
To: [EMAIL PROTECTED]
Subject: Re: [freenet-support] Node suddenly ceases traffic
Ed Tomlinson said in the devl-list:
Here is a little problem noticed today. There seems to be a case where
a thread can hang when waiting for data. This is what I see.From the connections applet:Peer addr Send Count Send Size Receiving Messages Idle time Lifetime ThreadType IDCould this be the very same problem? All my 114 idle inbound connections64.53.216.3:64638 0 0 yes 1 0 13897805 FThread-41 Inbound bc7f3f
looked something like this (not v656 but other versions). /N
----- Original Message -----
From: "Niklas Bergh" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Friday, January 24, 2003 11:33 PM
Subject: Re: [freenet-support] Node suddenly ceases traffic
Well, one of my nodes stopped doing stuff now too... Its log was 2gigabyteseven though it was only set to log at 'minor'... It survived 5 days and about 50 gigabytes of transfered data. One curious thing that I noted though: Routingtime was 0, queued jobs where a whole bunch and loadwas at100%, just as usual. The unusual thing was that the node said it had114active inbound connections (but the sendqueue was 0 bytes, compared toamore normal day when the sendqueue is a couple of tens of megs). Couldthissay something new? /N _______________________________________________ support mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support
_______________________________________________
support mailing list
[EMAIL PROTECTED]
http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support
_______________________________________________
support mailing list
[EMAIL PROTECTED]
http://hawk.freenetproject.org/cgi-bin/mailman/listinfo/support
----- End forwarded message -----
--
Matthew Toseland
[EMAIL PROTECTED][EMAIL PROTECTED]
Full time freenet hacker.
http://freenetproject.org/
Freenet Distribution Node (temporary) at http://amphibian.dyndns.org:8889/A8dz8aYheps/
ICTHUS.
<mime-attachment>
_______________________________________________ devl mailing list [EMAIL PROTECTED] http://hawk.freenetproject.org:8080/cgi-bin/mailman/listinfo/devl
