[freenet-support] Re: 5029 overloads
Toad wrote: Reading between the lines... check your log file, have you had many OutOfMemoryErrors? While gathering the data to answer your question, I found something strange. Get this: * /servlet/nodeinfo/performance/general reports an uptime of 1 day 16 hours 24 minutes * My log file started at 12:30pm, Oct 30th * It is currently 11:30pm, Oct 31st * That's a time span of only one day and 11 hours! hmmm * Also, I notice, strangely, that upstream bw is now reaching as high as 130%. In answer to your question, here is my first OutOfMemory error. Note the date/time: Oct 31, 2003 9:19:08 PM (freenet.thread.YThreadFactory$YThread, YThread-5710, ERROR): Unhandled exception java.lang.OutOfMemoryError in job freenet.Message: QueryRejected @[EMAIL PROTECTED] for tcp/connection: 368124.92.223.200:21214,[EMAIL PROTECTED],[EMAIL PROTECTED], sending null:-1 for [EMAIL PROTECTED] (DSA(bb9b 353f 42ba 692f 9661 f977 321d b395 fa51 af23),tcp/24.92.223.200:21214, sessions=1, presentations=1, ID=DSA(bb9b 353f 42ba 692f 9661 f977 321d b395 fa51 af23)): outbound attempts=20:20/40 @ 938bd22ed9a2279e: htl=15, reason=Node overloaded @ 1067663947477 java.lang.OutOfMemoryError There's a ton of OutOfMemory errors between the first and last. The last one is this: Oct 31, 2003 9:57:57 PM (freenet.thread.YThreadFactory$YThread, YThread-5751, ERROR): Unhandled exception java.lang.ArrayIndexOutOfBoundsException: 1 = 1 in job freenet.Message: DataRequest @[EMAIL PROTECTED] for tcp/connection: 172766.45.116.114:47351,[EMAIL PROTECTED],[EMAIL PROTECTED], sending null:46254 for [EMAIL PROTECTED] (DSA(ce25 bb0c 5942 8b7e 8019 6cec 64e0 f41e 0c0b 0856),tcp/groovy.danky.com:47351, sessions=1, presentations=1, ID=DSA(ce25 bb0c 5942 8b7e 8019 6cec 64e0 f41e 0c0b 0856)): outbound attempts=20:18/38 @ 6fd7ef27e588bcc5 @ 1067666277759 java.lang.ArrayIndexOutOfBoundsException: 1 = 1 at java.util.Vector.elementAt(Unknown Source) at freenet.support.Heap.remove(Heap.java:122) at freenet.support.Heap.access$000(Heap.java:11) at freenet.support.Heap$Element.remove(Heap.java:59) at freenet.node.StandardMessageHandler$Ticket.received(StandardMessageHandler.java:162) at freenet.node.StandardMessageHandler$Ticket.access$100(StandardMessageHandler.java:124) at freenet.node.StandardMessageHandler.handle(StandardMessageHandler.java:72) at freenet.Ticker$Event.run(Ticker.java:323) at freenet.thread.YThreadFactory$YThread.run(YThreadFactory.java:195) I am using YThreadFactory (obviously) and maximumThreads=300. Also, I'm allocating 128MB to java memory. I'll keep it running until the outgoing bandwidth gets really low. Recommendations? Or does this indicate a bug needing to be fixed? -Martin On Wed, Oct 29, 2003 at 10:29:49PM -0800, Martin Stone Davis wrote: I'm using default values for everything. Here are some relevant lines from my freenet.ini: # Select which implementation of ThreadFactory to use. Q: QThreadFactory (default). F: FastThreadFactory. Y: YetAnotherThreadFactory. %threadFactory=Q # Should we use thread management? If this number is defined and non-zero, this specifies the max number of threads in the pool. If this is overrun, connections will be rejected and events won't execute on time. %maximumThreads=120 But I'm confused. Why does General Information say maximumThreads=40, while freenet.ini says it's 120 by default? And does this mean the real default value (40) is being set way too low? -Martin Brandon Low wrote: What is your thread limit? From the looks of that it is incredibly low... Also, are you using YThreads or QThreads... what you might want to do is increase your thread limit to something closer to 200 or 300 and change to YThreads. big snip ___ Support mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support
Re: [freenet-support] Re: 5029 overloads
I am using YThreadFactory (obviously) and maximumThreads=300. Also, I'm allocating 128MB to java memory. I'll keep it running until the outgoing bandwidth gets really low. Recommendations? Or does this indicate a bug needing to be fixed? -Martin Given the amount of requests your that your node receive 128M probably isn't enough Try for a while with 192M or so. Well, Muxing and SIPv2 will both lead to decreased memory usage. Muxing will affect memory useage directly due to decreased amount of connections while SIPv2 will lead to decreased CPU which probably will cause some memory savings due to decreased enqueueing of stuff. /N ___ Support mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support
Re: [freenet-support] Re: 5029 overloads
On Fri, Oct 31, 2003 at 11:45:53PM -0800, Martin Stone Davis wrote: I am using YThreadFactory (obviously) and maximumThreads=300. Also, I'm allocating 128MB to java memory. I'll keep it running until the outgoing bandwidth gets really low. maximumThreads=300 = you need to give it more memory. Recommend -Xmx256m, at least. Recommendations? Or does this indicate a bug needing to be fixed? -Martin -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support
Re: [freenet-support] Re: 5029 overloads
On Sat, Nov 01, 2003 at 08:35:07AM -0500, Evan Berggren Daniel wrote: On Sat, 1 Nov 2003, Toad wrote: maximumThreads=300 = you need to give it more memory. Recommend -Xmx256m, at least. Would it make sense to have freenet put a cap on the threads based on available memory, similar to how it caps the routing table size based on max connections? Perhaps, but how to calculate it? Evan Daniel -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support
Re: [freenet-support] Re: 5029 overloads
On Sat, 1 Nov 2003, Toad wrote: Would it make sense to have freenet put a cap on the threads based on available memory, similar to how it caps the routing table size based on max connections? Perhaps, but how to calculate it? I'd go with something relatively simple. How many threads seems to work with 128MB available? 256MB? do a linear interpolation. It seems reasonable to assume there is a base + per-thread memory usage. Another option would be to query the JVM for available memory at regular intervals, and use that as part of the load calculation. If memory gets too tight, start decreasing the max threads parameter. Of course, I don't know much about the freenet code and how easy that would be. My experience with other code is that OOMs are remarkably hard to recover from, and so need to be avoided in advance. Evan Daniel ___ Support mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support
Re: [freenet-support] Re: 5029 overloads
Reading between the lines... check your log file, have you had many OutOfMemoryErrors? On Wed, Oct 29, 2003 at 10:29:49PM -0800, Martin Stone Davis wrote: I'm using default values for everything. Here are some relevant lines from my freenet.ini: # Select which implementation of ThreadFactory to use. Q: QThreadFactory (default). F: FastThreadFactory. Y: YetAnotherThreadFactory. %threadFactory=Q # Should we use thread management? If this number is defined and non-zero, this specifies the max number of threads in the pool. If this is overrun, connections will be rejected and events won't execute on time. %maximumThreads=120 But I'm confused. Why does General Information say maximumThreads=40, while freenet.ini says it's 120 by default? And does this mean the real default value (40) is being set way too low? -Martin Brandon Low wrote: What is your thread limit? From the looks of that it is incredibly low... Also, are you using YThreads or QThreads... what you might want to do is increase your thread limit to something closer to 200 or 300 and change to YThreads. --B On Wed, 10/29/03 at 22:06:53 -0800, Martin Stone Davis wrote: I've been running 5029 for over a day. I noticed recently that outgoing bandwidth decreased significantly. Is this problem already known about? From General Information: Current routingTime 207ms Curent messageSendTimeRequest 496ms Pooled threads running jobs 47 (117.5%) Pooled threads which are idle 45 Current upstream bandwidth usage2962 bytes/second (24.1%) Reason for refusing connections:activeThreads(47) = maximumThreads (40) Reason for QueryRejecting requests: Estimated load (100%) overloadLow (85%) It's normal for the node to sometimes reject connections or requests for a limited period. If you're seeing rejections continuously the node is overloaded or something is wrong (i.e. a bug). Current estimated load 100% [Rejecting incoming connections and requests!] Reason for load:Load due to thread limit = 117.5% Load due to routingTime = 73.1% = 85% + 15% * (207.953 - 1000.000) / 1000.000 = overloadLow (85%) Load due to messageSendTimeRequest = 82.5% = 85% + 15% * (496.871 - 1000.000) / 3000.000 = overloadLow (85%) Load due to output bandwidth limiting = 0% because outputBytes(177714) = limit (589824.009 ) = outLimitCutoff (0.8) * outputBandwidthLimit (12288) * 60 From Environment: Thread Pool Total pooled threads75 Available pooled threads34 Pooled threads in use 41 Pooled Thread Consumers Class Threads used Checkpoint: Connection opener 1 freenet.Message: Accepted 1 freenet.Message: DataNotFound 1 freenet.Message: DataReply 1 freenet.Message: DataRequest1 freenet.Message: QueryRejected 2 freenet.Message: QueryRestarted 1 freenet.interfaces.LocalNIOInterface$ConnectionShell12 freenet.node.states.data.DataStateInitiator 14 freenet.node.states.data.TrailerWriteCallbackMessage:true:true 5 freenet.node.states.request.RequestInitiator2 -Martin ___ Support mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support ___ Support mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support
