Re: [freenet-support] High disk IO
On Saturday 14 May 2011 02:53:25 Dennis Nezic wrote: > On Wed, 11 May 2011 17:48:36 +0100, Matthew Toseland wrote: > > On Wednesday 11 May 2011 02:31:51 Dennis Nezic wrote: > > > On Fri, 6 May 2011 15:55:59 -0400, Dennis Nezic wrote: > > > > On Fri, 6 May 2011 18:57:13 +0100, Matthew Toseland wrote: > > > > > On Thursday 05 May 2011 03:26:45 Dennis Nezic wrote: > > > > > > For the past several builds, I'm not sure exactly when the > > > > > > issue started, before build01364 at least, my node > > > > > > experiences quite high disk IO, which bogs down my computer. > > > > > > I think once I left it alone, and it resolved itself after a > > > > > > few hours, but usually I have to restart Freenet to make my > > > > > > server more useable. > > > > > > > > > > > > My "vmstat 5" reports bi (bytes-in, that is: reading) between > > > > > > 4000-1. (Under 100 after I stop Freenet.) > > > > > > > > > > > > My graphs[1] show it to be suspiciously periodic, although > > > > > > that might just be a coincidence. > > > > > > > > > > > > Is there some kind of background (periodic) database/datastore > > > > > > task that might be causing this? My wrapper.log shows no > > > > > > output during these events, nor does fproxy say that anything > > > > > > is going on. > > > > > > > > > > Active downloads/uploads? > > > > > > > > Yep ... a few active downloads -- about 30MBs each. (One got > > > > completed much earlier in the day -- but nothing unusual was > > > > happening during the problem peaks.) It happened again last night > > > > [2][3], suspiciously almost exactly 24 hours after the last time. > > > > I didn't stop my node this time, and it "fixed" itself (at least > > > > the plateau) after over 2 hours, although disk IO was still quite > > > > high throughout the night [2] and early morning. > > > > > > > > > > > > [1] > > > > > > http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io.jpg > > > > [2] That's what she said. > > > > [3] > > > > http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io-2.jpg > > > > > > It definitely seems daily/periodic... it seems to occur pretty much > > > every single day at 8pm. It just happened an hour ago, for example, > > > and resulted in "load average: 7.09, 6.12, 5.38" which seriously > > > bogged down my system, so I had to kill/restart my node :\. Ideas? > > > > Maybe you could get a stack dump during that period? > > My stack dumps didn't show anything glaringly wrong. (No (or just once > a) BLOCKED threads.) However, I think the problem has to do with FMS -- > shutting down FMS stopped the high-IO. (The reason why I originally > suspected Freenet was involved, however, was because stopping my node > also seemed to stop the IO a few seconds afterwards.) I'm still not > sure whether it's purely an FMS bug (perhaps having to do with my > 150MB fms.db3 file), or some kind of FMS-Freenet interaction. I will > try to investigate further. You're looking for threads that are persistently RUNNABLE across multiple dumps. An interaction with FMS is quite possible... signature.asc Description: This is a digitally signed message part. ___ Support mailing list [email protected] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[email protected]?subject=unsubscribe
Re: [freenet-support] High disk IO
On Wed, 11 May 2011 17:48:36 +0100, Matthew Toseland wrote: > On Wednesday 11 May 2011 02:31:51 Dennis Nezic wrote: > > On Fri, 6 May 2011 15:55:59 -0400, Dennis Nezic wrote: > > > On Fri, 6 May 2011 18:57:13 +0100, Matthew Toseland wrote: > > > > On Thursday 05 May 2011 03:26:45 Dennis Nezic wrote: > > > > > For the past several builds, I'm not sure exactly when the > > > > > issue started, before build01364 at least, my node > > > > > experiences quite high disk IO, which bogs down my computer. > > > > > I think once I left it alone, and it resolved itself after a > > > > > few hours, but usually I have to restart Freenet to make my > > > > > server more useable. > > > > > > > > > > My "vmstat 5" reports bi (bytes-in, that is: reading) between > > > > > 4000-1. (Under 100 after I stop Freenet.) > > > > > > > > > > My graphs[1] show it to be suspiciously periodic, although > > > > > that might just be a coincidence. > > > > > > > > > > Is there some kind of background (periodic) database/datastore > > > > > task that might be causing this? My wrapper.log shows no > > > > > output during these events, nor does fproxy say that anything > > > > > is going on. > > > > > > > > Active downloads/uploads? > > > > > > Yep ... a few active downloads -- about 30MBs each. (One got > > > completed much earlier in the day -- but nothing unusual was > > > happening during the problem peaks.) It happened again last night > > > [2][3], suspiciously almost exactly 24 hours after the last time. > > > I didn't stop my node this time, and it "fixed" itself (at least > > > the plateau) after over 2 hours, although disk IO was still quite > > > high throughout the night [2] and early morning. > > > > > > > > > > [1] > > > > > http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io.jpg > > > [2] That's what she said. > > > [3] > > > http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io-2.jpg > > > > It definitely seems daily/periodic... it seems to occur pretty much > > every single day at 8pm. It just happened an hour ago, for example, > > and resulted in "load average: 7.09, 6.12, 5.38" which seriously > > bogged down my system, so I had to kill/restart my node :\. Ideas? > > Maybe you could get a stack dump during that period? My stack dumps didn't show anything glaringly wrong. (No (or just once a) BLOCKED threads.) However, I think the problem has to do with FMS -- shutting down FMS stopped the high-IO. (The reason why I originally suspected Freenet was involved, however, was because stopping my node also seemed to stop the IO a few seconds afterwards.) I'm still not sure whether it's purely an FMS bug (perhaps having to do with my 150MB fms.db3 file), or some kind of FMS-Freenet interaction. I will try to investigate further. ___ Support mailing list [email protected] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[email protected]?subject=unsubscribe
Re: [freenet-support] High disk IO
On Wednesday 11 May 2011 02:31:51 Dennis Nezic wrote: > On Fri, 6 May 2011 15:55:59 -0400, Dennis Nezic wrote: > > On Fri, 6 May 2011 18:57:13 +0100, Matthew Toseland wrote: > > > On Thursday 05 May 2011 03:26:45 Dennis Nezic wrote: > > > > For the past several builds, I'm not sure exactly when the issue > > > > started, before build01364 at least, my node experiences quite > > > > high disk IO, which bogs down my computer. I think once I left it > > > > alone, and it resolved itself after a few hours, but usually I > > > > have to restart Freenet to make my server more useable. > > > > > > > > My "vmstat 5" reports bi (bytes-in, that is: reading) between > > > > 4000-1. (Under 100 after I stop Freenet.) > > > > > > > > My graphs[1] show it to be suspiciously periodic, although that > > > > might just be a coincidence. > > > > > > > > Is there some kind of background (periodic) database/datastore > > > > task that might be causing this? My wrapper.log shows no output > > > > during these events, nor does fproxy say that anything is going > > > > on. > > > > > > Active downloads/uploads? > > > > Yep ... a few active downloads -- about 30MBs each. (One got completed > > much earlier in the day -- but nothing unusual was happening during > > the problem peaks.) It happened again last night [2][3], suspiciously > > almost exactly 24 hours after the last time. I didn't stop my node > > this time, and it "fixed" itself (at least the plateau) after over 2 > > hours, although disk IO was still quite high throughout the night [2] > > and early morning. > > > > > > > > [1] > > > > http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io.jpg > > [2] That's what she said. > > [3] > > http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io-2.jpg > > It definitely seems daily/periodic... it seems to occur pretty much > every single day at 8pm. It just happened an hour ago, for example, and > resulted in "load average: 7.09, 6.12, 5.38" which seriously bogged > down my system, so I had to kill/restart my node :\. Ideas? Maybe you could get a stack dump during that period? signature.asc Description: This is a digitally signed message part. ___ Support mailing list [email protected] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[email protected]?subject=unsubscribe
Re: [freenet-support] High disk IO
On Fri, 6 May 2011 15:55:59 -0400, Dennis Nezic wrote: > On Fri, 6 May 2011 18:57:13 +0100, Matthew Toseland wrote: > > On Thursday 05 May 2011 03:26:45 Dennis Nezic wrote: > > > For the past several builds, I'm not sure exactly when the issue > > > started, before build01364 at least, my node experiences quite > > > high disk IO, which bogs down my computer. I think once I left it > > > alone, and it resolved itself after a few hours, but usually I > > > have to restart Freenet to make my server more useable. > > > > > > My "vmstat 5" reports bi (bytes-in, that is: reading) between > > > 4000-1. (Under 100 after I stop Freenet.) > > > > > > My graphs[1] show it to be suspiciously periodic, although that > > > might just be a coincidence. > > > > > > Is there some kind of background (periodic) database/datastore > > > task that might be causing this? My wrapper.log shows no output > > > during these events, nor does fproxy say that anything is going > > > on. > > > > Active downloads/uploads? > > Yep ... a few active downloads -- about 30MBs each. (One got completed > much earlier in the day -- but nothing unusual was happening during > the problem peaks.) It happened again last night [2][3], suspiciously > almost exactly 24 hours after the last time. I didn't stop my node > this time, and it "fixed" itself (at least the plateau) after over 2 > hours, although disk IO was still quite high throughout the night [2] > and early morning. > > > > > > [1] > > > http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io.jpg > [2] That's what she said. > [3] > http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io-2.jpg It definitely seems daily/periodic... it seems to occur pretty much every single day at 8pm. It just happened an hour ago, for example, and resulted in "load average: 7.09, 6.12, 5.38" which seriously bogged down my system, so I had to kill/restart my node :\. Ideas? ___ Support mailing list [email protected] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[email protected]?subject=unsubscribe
Re: [freenet-support] High disk IO
On Fri, 6 May 2011 18:57:13 +0100, Matthew Toseland wrote: > On Thursday 05 May 2011 03:26:45 Dennis Nezic wrote: > > For the past several builds, I'm not sure exactly when the issue > > started, before build01364 at least, my node experiences quite high > > disk IO, which bogs down my computer. I think once I left it alone, > > and it resolved itself after a few hours, but usually I have to > > restart Freenet to make my server more useable. > > > > My "vmstat 5" reports bi (bytes-in, that is: reading) between > > 4000-1. (Under 100 after I stop Freenet.) > > > > My graphs[1] show it to be suspiciously periodic, although that > > might just be a coincidence. > > > > Is there some kind of background (periodic) database/datastore task > > that might be causing this? My wrapper.log shows no output during > > these events, nor does fproxy say that anything is going on. > > Active downloads/uploads? Yep ... a few active downloads -- about 30MBs each. (One got completed much earlier in the day -- but nothing unusual was happening during the problem peaks.) It happened again last night [2][3], suspiciously almost exactly 24 hours after the last time. I didn't stop my node this time, and it "fixed" itself (at least the plateau) after over 2 hours, although disk IO was still quite high throughout the night [2] and early morning. > > > > [1] > > http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io.jpg [2] That's what she said. [3] http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io-2.jpg ___ Support mailing list [email protected] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[email protected]?subject=unsubscribe
Re: [freenet-support] High disk IO
On Thursday 05 May 2011 03:26:45 Dennis Nezic wrote: > For the past several builds, I'm not sure exactly when the issue > started, before build01364 at least, my node experiences quite high > disk IO, which bogs down my computer. I think once I left it alone, and > it resolved itself after a few hours, but usually I have to restart > Freenet to make my server more useable. > > My "vmstat 5" reports bi (bytes-in, that is: reading) between > 4000-1. (Under 100 after I stop Freenet.) > > My graphs[1] show it to be suspiciously periodic, although that might > just be a coincidence. > > Is there some kind of background (periodic) database/datastore task that > might be causing this? My wrapper.log shows no output during these > events, nor does fproxy say that anything is going on. Active downloads/uploads? > > > [1] > http://dennisn.dyndns.org/guest/pubstuff/freenet-periodic-high-io.jpg signature.asc Description: This is a digitally signed message part. ___ Support mailing list [email protected] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[email protected]?subject=unsubscribe
