Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Monday 04 May 2009 21:21:50 Victor Denisov wrote: Hello, Since my node autoupdated to 1208, my system's performance degraded so much as to render it completely unusable while Freenet is running. I can't perform simplest tasks, such as surfing or typing a document, as switching between tabs in Opera can take 10+ seconds and delay between typing a letter and it showing up in Word can be several seconds as well. Anything more stressing (such as running a game or developing an application) is simply not possible. From what I can see, the reason for this is that Freenet makes hundreds of small disk i/o ops per second, basically blocking the OS from accessing the hard drive for swapping and such. The above is definitely affected by the queue size. First, I tried adding ~ 100 random files from Thaw when the node first updated, but hadn't had the patience to wait for the request to complete (I think I waited at least 20 minutes, perhaps more - with the system being nearly paralyzed by the constant HDD thrashing). With just 3 or 4 files being put in the queue the system starts to stutter noticeably, provided that files start downloading and not hang at 0%. The only time when I can run Freenet as a real background app is when I don't have any files in the queue and FMS isn't running (which seems more or less pointless to me :-(). My system is set up as follows: AMD Athlon 64 X2 6000+ 4 Gb RAM 2x250 Gb 7200 rpm SATA2 HDD (mirror via mobo's built-in nVidia 570 RAID controller) Windows XP x64 Java 1.6.0_07 64-bit Of course, all the latest updates/patches/drivers, etc. Freenet uses 5 Gb datastore on an unencrypted partition. Interesting thing I noticed was that Freenet significantly underutilized the memory I provide it with. From 320 Mb heap memory available, I hadn't seen it allocate more than ~ 80 Mb. Any thoughts on why this could be happening? I hadn't seen anyone complain about the performance of 1208/09 yet, so it is probably something with my machine :-(, but it beats me what it could be :-(. This is the downside of db4o. If it is a widespread problem, we're gonna have to revert it. Which means throwing away more than 6 months work largely funded by Google's $18K. My database queue is usually pretty empty, even with queued downloads, but I have 8G and fast mirrored disks... On Tuesday 05 May 2009 03:22:06 Juiceman wrote: I see it too. I have my node installed on it's own separate disk yet it stalls my quad-core system. It's as if the HDD controller chip is so swamped with disk ops that the OS and other apps are starved for resources. I'm seeing disk queues exceeding 500 vs the 40 my workgroup server hits. I also see Freenet using 2650 handles. That twice the next highest app which is my antivirus and 10x the average app. Heck, even the core Windows processes don't use more than 300-600 usually. Any recommendations for improving performance? Is BDB datastore any better (more stable or less disk IO) than salted-hash? I could try going back to that... 2650 handles is strange, on unix we are generally limited to 1024 and generally we don't exceed that. Both of your problems may be caused by flaky hardware, but frankly we do need to run on flaky real world hardware. :| No, changing the datastore won't help. On Tuesday 05 May 2009 10:30:56 Victor Denisov wrote: I see it too. I have my node installed on it's own separate disk yet it stalls my quad-core system. It's as if the HDD controller chip is so swamped with disk ops that the OS and other apps are starved for resources. I'm seeing disk queues exceeding 500 vs the 40 my workgroup server hits. Same here. Enormous disk queues. I've also compared i/o counts with i/o bytes read/written - that's how I know that i/o operations are small. In the statistics screen, I routinely see 100+ outstanding database jobs. It can't be good. This just confirms that disk I/O is the problem ... and almost certainly caused by db4o as it goes away if nothing is queued. :( Opinions of other folk? Does this happen to you? Are there lots of complaints of similar problems on FMS? Is the recent increase in the performance of automated tests the result of the network shrinking due to lots of people uninstalling? signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
[freenet-support] Please answer a quick survey on Freenet
What OS do you use for Freenet? What is your current datastore size set to? What is your output bandwidth limit set to? What actual bandwidth usage do you typically get? This will help us to make decisions about new performance features ... signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
What OS do you use for Freenet? ---winxp What is your current datastore size set to? ---10G What is your output bandwidth limit set to? ---100Mbps What actual bandwidth usage do you typically get? ---10Mbps This will help us to make decisions about new performance features ... ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe = = = = = = = = = = = = = = = = = = = = support@freenetproject.org ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
On Wed, May 6, 2009 at 14:54, Matthew Toseland t...@amphibian.dyndns.org wrote: What OS do you use for Freenet? WinXP What is your current datastore size set to? 100 GB What is your output bandwidth limit set to? 32 kb/s What actual bandwidth usage do you typically get? 34 kb/s ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 What OS do you use for Freenet? Windows XP x64 What is your current datastore size set to? 5 Gb What is your output bandwidth limit set to? 100 Kb/s What actual bandwidth usage do you typically get? Until 1208, around 75-85 Kb/s, with 1208+ anywhere from 10 to 50 Kb/s, depending on how much disk thrashing is going on. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAYwn1O5++4rTuI0RAosNAJ9zYaUKLu8MTM1dPoUzqyQ9QVxjvACghTut OkNaGnQLsDYcm9QPZd1TOao= =86xr -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
Am Mittwoch, 6. Mai 2009 14:54:16 schrieb Matthew Toseland: What OS do you use for Freenet? Gentoo, amd64 What is your current datastore size set to? 240G (dedicated disk) What is your output bandwidth limit set to? 70KB What actual bandwidth usage do you typically get? ~45KB This will help us to make decisions about new performance features ... MfG saces ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matthew Toseland wrote: This is the downside of db4o. If it is a widespread problem, we're gonna have to revert it. Which means throwing away more than 6 months work largely funded by Google's $18K. I think that using a database is a good idea (although I personally would've opted for a relational database such as Derby). So I'd prefer to try and understand and fix the issue rather than hiding from it :-). My database queue is usually pretty empty, even with queued downloads, but I have 8G and fast mirrored disks... The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb cache, SATA2, no NCQ - though definitely not the slowest out there. I see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll probably have to test the same from inside Java to make absolutely sure that it's not some weird JVM issue on my platform, though. 2650 handles is strange, on unix we are generally limited to 1024 and generally we don't exceed that. Both of your problems may be caused by flaky hardware, but frankly we do need to run on flaky real world hardware. :| I don't have Freenet running right now, will check it later. But I2P is using 2670 handles right now, and Azureus uses 1450 - so 2600 for Freenet is definitely nothing out of the ordinary on Windows. Oh, and the highest handle user on my machine is MySQL, which uses ~69000 handles and works absolutely fine :-). Same here. Enormous disk queues. I've also compared i/o counts with i/o bytes read/written - that's how I know that i/o operations are small. In the statistics screen, I routinely see 100+ outstanding database jobs. It can't be good. This just confirms that disk I/O is the problem ... and almost certainly caused by db4o as it goes away if nothing is queued. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAZQf1O5++4rTuI0RAr5KAKCPuXmaqThbq0g8jVxdGwj7fNGZ/wCgsBBw YybivVLzs7FlJHvqXbvDJwA= =SWCw -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 What OS do you use for Freenet? Ubuntu GNU/Linux 9.04 What is your current datastore size set to? 20 GiB What is your output bandwidth limit set to? 220 KiB/sec What actual bandwidth usage do you typically get? 50 KiB/sec This will help us to make decisions about new performance features ... Yippie! - Volodya - -- http://freedom.libsyn.com/ Echo of Freedom, Radical Podcast http://eng.anarchopedia.org/ Anarchopedia, A Free Knowledge Portal http://www.freedomporn.org/ Freedom Porn, anarchist and activist smut None of us are free until all of us are free.~ Mihail Bakunin -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkoBlwAACgkQuWy2EFICg+0COACgpDuqysrASaEkxT6HwW9o6i+E 3uAAoMZUtCBQMMQa5jqN+qtC+mrHwdrw =yqs9 -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: Matthew Toseland wrote: This is the downside of db4o. If it is a widespread problem, we're gonna have to revert it. Which means throwing away more than 6 months work largely funded by Google's $18K. I think that using a database is a good idea (although I personally would've opted for a relational database such as Derby). So I'd prefer to try and understand and fix the issue rather than hiding from it :-). My database queue is usually pretty empty, even with queued downloads, but I have 8G and fast mirrored disks... The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. Well, a major objective of the db4o rewrite was precisely this. And I don't see that this is a problem - the OS will use the rest to cache the node.db4o file so that only writes need to go to disk. My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb cache, SATA2, no NCQ - though definitely not the slowest out there. I see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll probably have to test the same from inside Java to make absolutely sure that it's not some weird JVM issue on my platform, though. I have seagate 1TB disks mirrored, may explain the difference. 2650 handles is strange, on unix we are generally limited to 1024 and generally we don't exceed that. Both of your problems may be caused by flaky hardware, but frankly we do need to run on flaky real world hardware. :| I don't have Freenet running right now, will check it later. But I2P is using 2670 handles right now, and Azureus uses 1450 - so 2600 for Freenet is definitely nothing out of the ordinary on Windows. Oh, and the highest handle user on my machine is MySQL, which uses ~69000 handles and works absolutely fine :-). Same here. Enormous disk queues. I've also compared i/o counts with i/o bytes read/written - that's how I know that i/o operations are small. In the statistics screen, I routinely see 100+ outstanding database jobs. It can't be good. This just confirms that disk I/O is the problem ... and almost certainly caused by db4o as it goes away if nothing is queued. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. No, because it's an I/O problem, not a CPU/memory problem! Regards, Victor Denisov. signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: Matthew Toseland wrote: This is the downside of db4o. If it is a widespread problem, we're gonna have to revert it. Which means throwing away more than 6 months work largely funded by Google's $18K. I think that using a database is a good idea (although I personally would've opted for a relational database such as Derby). So I'd prefer to try and understand and fix the issue rather than hiding from it :-). My database queue is usually pretty empty, even with queued downloads, but I have 8G and fast mirrored disks... The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb cache, SATA2, no NCQ - though definitely not the slowest out there. I see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll probably have to test the same from inside Java to make absolutely sure that it's not some weird JVM issue on my platform, though. 2650 handles is strange, on unix we are generally limited to 1024 and generally we don't exceed that. Both of your problems may be caused by flaky hardware, but frankly we do need to run on flaky real world hardware. :| I don't have Freenet running right now, will check it later. But I2P is using 2670 handles right now, and Azureus uses 1450 - so 2600 for Freenet is definitely nothing out of the ordinary on Windows. Oh, and the highest handle user on my machine is MySQL, which uses ~69000 handles and works absolutely fine :-). Same here. Enormous disk queues. I've also compared i/o counts with i/o bytes read/written - that's how I know that i/o operations are small. In the statistics screen, I routinely see 100+ outstanding database jobs. It can't be good. This just confirms that disk I/O is the problem ... and almost certainly caused by db4o as it goes away if nothing is queued. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Do you have uploads queued as well as downloads? Generally uploads cost a bit more than downloads do with db4o... signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
Matthew Toseland schrieb: What OS do you use for Freenet? Ubuntu 8.04 server What is your current datastore size set to? 30GB What is your output bandwidth limit set to? 300KiB/s What actual bandwidth usage do you typically get? 1.5MiB/s This will help us to make decisions about new performance features ... ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote: Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Maybe turning off logging helps? Does Juiceman also have logging enabled? This appears to only affect Microsoft users? ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
On Wed, 6 May 2009 13:54:16 +0100, Matthew Toseland wrote: What OS do you use for Freenet? Gentoo Linux, amd64 What is your current datastore size set to? 8GiB What is your output bandwidth limit set to? 9KB/s What actual bandwidth usage do you typically get? 7KB/s (average, my ISP throttles) ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
What OS do you use for Freenet? Ubuntu GNU Linux 9.04 AMD64 What is your current datastore size set to? 16 GB What is your output bandwidth limit set to? 900 KB/s What actual bandwidth usage do you typically get? 65KB/s when no QOS, and between 0 and 25 KB/s when use the port 53 (when my ISP blocks other UDP ports). This will help us to make decisions about new performance features ... On Wed, May 6, 2009 at 4:30 PM, Dennis Nezic denn...@dennisn.dyndns.orgwrote: On Wed, 6 May 2009 13:54:16 +0100, Matthew Toseland wrote: What OS do you use for Freenet? Gentoo Linux, amd64 What is your current datastore size set to? 8GiB What is your output bandwidth limit set to? 9KB/s What actual bandwidth usage do you typically get? 7KB/s (average, my ISP throttles) ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
[freenet-support] CPU usage Re: Is it my system, or had builds 1208-1209 have severe performance issues?
On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: Matthew Toseland wrote: This is the downside of db4o. If it is a widespread problem, we're gonna have to revert it. Which means throwing away more than 6 months work largely funded by Google's $18K. I think that using a database is a good idea (although I personally would've opted for a relational database such as Derby). So I'd prefer to try and understand and fix the issue rather than hiding from it :-). My database queue is usually pretty empty, even with queued downloads, but I have 8G and fast mirrored disks... The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb cache, SATA2, no NCQ - though definitely not the slowest out there. I see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll probably have to test the same from inside Java to make absolutely sure that it's not some weird JVM issue on my platform, though. 2650 handles is strange, on unix we are generally limited to 1024 and generally we don't exceed that. Both of your problems may be caused by flaky hardware, but frankly we do need to run on flaky real world hardware. :| I don't have Freenet running right now, will check it later. But I2P is using 2670 handles right now, and Azureus uses 1450 - so 2600 for Freenet is definitely nothing out of the ordinary on Windows. Oh, and the highest handle user on my machine is MySQL, which uses ~69000 handles and works absolutely fine :-). Same here. Enormous disk queues. I've also compared i/o counts with i/o bytes read/written - that's how I know that i/o operations are small. In the statistics screen, I routinely see 100+ outstanding database jobs. It can't be good. This just confirms that disk I/O is the problem ... and almost certainly caused by db4o as it goes away if nothing is queued. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Regards, Victor Denisov. One other thing, for both you and Juiceman: How's the CPU usage? Given how much RAM you have I would expect node.db4o to be cached in memory (how big is it?). But doing a read through the OS to the OS disk cache may cost a lot of CPU (context switch etc) ... Is there a lot of CPU usage for the freenet process? To the point that it might be the cause of the poor overall system performance? And how much CPU usage is system? signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wednesday 06 May 2009 15:26:12 Dennis Nezic wrote: On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote: Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Maybe turning off logging helps? Does Juiceman also have logging enabled? This appears to only affect Microsoft users? You have a node on a linux system with queued downloads and no performance issues? signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
On Wed, May 6, 2009 at 8:54 AM, Matthew Toseland t...@amphibian.dyndns.org wrote: What OS do you use for Freenet? WinXP_SP3 What is your current datastore size set to? 20G What is your output bandwidth limit set to? 60KB What actual bandwidth usage do you typically get? 42 avg prior to db4o Additional info: 512MB ram allocated to Freenet uptime 24x7 prior to db4o This will help us to make decisions about new performance features ... ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] CPU usage Re: Is it my system, or had builds 1208-1209 have severe performance issues?
On Wed, May 6, 2009 at 11:37 AM, Matthew Toseland t...@amphibian.dyndns.org wrote: On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: Matthew Toseland wrote: This is the downside of db4o. If it is a widespread problem, we're gonna have to revert it. Which means throwing away more than 6 months work largely funded by Google's $18K. I think that using a database is a good idea (although I personally would've opted for a relational database such as Derby). So I'd prefer to try and understand and fix the issue rather than hiding from it :-). My database queue is usually pretty empty, even with queued downloads, but I have 8G and fast mirrored disks... The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb cache, SATA2, no NCQ - though definitely not the slowest out there. I see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll probably have to test the same from inside Java to make absolutely sure that it's not some weird JVM issue on my platform, though. 2650 handles is strange, on unix we are generally limited to 1024 and generally we don't exceed that. Both of your problems may be caused by flaky hardware, but frankly we do need to run on flaky real world hardware. :| I don't have Freenet running right now, will check it later. But I2P is using 2670 handles right now, and Azureus uses 1450 - so 2600 for Freenet is definitely nothing out of the ordinary on Windows. Oh, and the highest handle user on my machine is MySQL, which uses ~69000 handles and works absolutely fine :-). Same here. Enormous disk queues. I've also compared i/o counts with i/o bytes read/written - that's how I know that i/o operations are small. In the statistics screen, I routinely see 100+ outstanding database jobs. It can't be good. This just confirms that disk I/O is the problem ... and almost certainly caused by db4o as it goes away if nothing is queued. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Regards, Victor Denisov. One other thing, for both you and Juiceman: How's the CPU usage? Given how much RAM you have I would expect node.db4o to be cached in memory (how big is it?). But doing a read through the OS to the OS disk cache may cost a lot of CPU (context switch etc) ... Is there a lot of CPU usage for the freenet process? To the point that it might be the cause of the poor overall system performance? And how much CPU usage is system? Freenet CPU usage fluctuates between 2 and 27% of a quad core system. The rest of the machine rarely uses more than 15% unless I am gaming, then it still only hits 50%. CPU usage is quite acceptable for now. I have 3GB of RAM, 512 allocated to Freenet. -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
On Wednesday 06 May 2009 13:54:16 Matthew Toseland wrote: What OS do you use for Freenet? What is your current datastore size set to? What is your output bandwidth limit set to? What actual bandwidth usage do you typically get? This will help us to make decisions about new performance features ... Additional useful info, if you don't mind parting with it: Network, friends and physical security levels. signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matthew Toseland a écrit : What OS do you use for Freenet? Debian Lenny What is your current datastore size set to? 50Go What is your output bandwidth limit set to? 100 mb/s What actual bandwidth usage do you typically get? 50ko/s -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQIcBAEBAgAGBQJKAcT+AAoJEELek5QwRDhpvnUQAJ9e4XsX+VSup/LT9wG4r76G xwLXafIN0AsuSZr5tW74aE8gRvCoIlYjVDErD1ZPQoPPhU6IV1xTn6F83kRHTF1c zXzkQQrn+FXylWANzZudm8MQjZvZZWfIowzqOhowPYsMRxOSIivwbUN249zMlq5j 0wtVcSj/A30prYokFnZz3isEMVfGY0OuIqj6Fn4Pk6R8HlgwhlVdv9y6e9Ii8Lln 1sCtFTpTq6xnMs0FTuZlEo1wXvj0CZsdAOkW51kY+Oqi+EApt3JuR3aTw+89cEo9 3LV+kBNBE+2mOPzd0WDcTj8zdlz8iaMS8+ttIo6p7UtBeOVW5T0Kwgh2sp0OTuRh lCABHNsTeXJ18SWC6FRdaKsvVJK5YHNmg+x+ek+V9tv1T8ribekCKiZvlKjlHayO jCdGFB9I71lrXhM9oBOg4vMYP+s9fQ9RM3A/aq+n611iNuiuYmkusoCYFX/uj+q0 qIguvQxlQTaTWObvgFKe5N/Z3zj1+G6oiVfHJw8r1+CieWLt7J3Am24YOTD1g4et Ls1pOLvTUIgfYou+nDw5Hq1j3AOwaN3mfBKyyz22cLhA9FY4xkc4LKW1W0ShWB1D ESb515ZoQpULZ8Dqj8fSP6K7zWvZK0gM5OEg3L1r92u0/26W4UyErPABuDDUDTEw WeaPEqmI8YecpofbIw0W =cRRe -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. Well, a major objective of the db4o rewrite was precisely this. And I don't see that this is a problem - the OS will use the rest to cache the node.db4o file so that only writes need to go to disk. Yes, this I understand. I was one of those complaining of Freenet using too much RAM :-(. But, IMO, using as much memory as possible (out of the dedicated pool) could be important for performance. For example, by increasing buffer sizes in db4o we can possibly make flushes more organized, reducing disk writes substantially. I wonder if there are ways to tune db4o performance without rewriting the code, are there any handles to turn in the db4o config? I have seagate 1TB disks mirrored, may explain the difference. Unlikely, IMO. 1 Tb drives should have better throughput, but Freenet is definitely limited by seek times, which should be only marginally better. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. No, because it's an I/O problem, not a CPU/memory problem! I was thinking more about allocation/invocation counts, available in both profilers. Perhaps some method/query is being called unexpectedly often, or a certain object is being persisted too often, etc. Also, it could be that a certain method blocks too often and for too much time in Windows, leading to poor I/O performance. But it's a long shot, this I agree with. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAdMB1O5++4rTuI0RAi/tAKC8SIIIeUVlQzYtntg22Uxjywp59ACdG5zQ ICyGAM6ubeh6I8JFIDwkjLs= =9uoG -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Do you have uploads queued as well as downloads? Generally uploads cost a bit more than downloads do with db4o... No, only downloads. Total queued size varied between 25 Mb and 350 Mb in my tests (but actual total file size was often more than reported by Freenet, as some keys stayed at 0% for the duration of the test). Also, to clarify things, no background applications of notice (such as other P2P apps or distributed computing clients) were running during the test. I regularly run Azureus, eMule and I2P, but they all were stopped for the entire duration Freenet was running, as were MySQL and MS SQL Server instances I work on. I also tried disabling my antivirus/personal firewall (Agnitum Outpost Security Suite), but it didn't result in a noticeable improvement in performance. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAdRl1O5++4rTuI0RAh5KAKDCoFmtLlfKGp0/2GZ4SQv9+/NmSQCfX2YW oIuQHcSPsl7/smqkL1b9Ykk= =7RhK -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Maybe turning off logging helps? Does Juiceman also have logging enabled? This appears to only affect Microsoft users? On ERROR, Freenet writes about 2-5 Kb per 10 minutes, which is really nothing. On NORMAL, it writes up to 5 Mb per 10 minutes, or ~ 8 Kb/s (which probably translates into ~10 to 20 writes per second, considering internal buffering and OS write-behind caching). Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAdhA1O5++4rTuI0RAm0tAJ4p5tTvVrYmhgRdPxXsTDM0wElxqgCbBYoD xGvyyHMG94ZlzrOrNI+fWaU= =E+sh -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] CPU usage Re: Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 One other thing, for both you and Juiceman: How's the CPU usage? Given how much RAM you have I would expect node.db4o to be cached in memory (how big is it?). But doing a read through the OS to the OS disk cache may cost a lot of CPU (context switch etc) ... Is there a lot of CPU usage for the freenet process? To the point that it might be the cause of the poor overall system performance? And how much CPU usage is system? node.db4o is ~ 25 Mb right now, with, IIRC, ~40 downloads queued, but not many actually progressing. CPU usage for the Freenet process is relatively low (I'd say on order of 10-15%). I'll try and see how much kernel time Freenet uses (will have to learn how to check this), but kernel CPU load (something which is easily checked out from Task Manager) is about half the total CPU load when Freenet is running. Note that firewall contributes to this number, as its driver runs in kernel space, obviously. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAdp51O5++4rTuI0RAjMjAKCl/V875N7OabYqP6h8/e3CkTKawwCgtqZX XoaOWVG2QeKaF/4q3U8N2pk= =lIfc -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Additional useful info, if you don't mind parting with it: Network, friends and physical security levels. All on NORMAL here. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAdtL1O5++4rTuI0RAuk7AKCe4zdHlH64Pxr6bvbUudQ/REtJowCg4ajH zO5o2MtuUn2mQqzsKd9kst4= =sL3K -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
On 5/6/09, Matthew Toseland t...@amphibian.dyndns.org wrote: What OS do you use for Freenet? FreeBSD 7.2-PRERELEASE What is your current datastore size set to? 300 GiB What is your output bandwidth limit set to? 100 KiB/s What actual bandwidth usage do you typically get? ~50 KiB/s This will help us to make decisions about new performance features ... ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
Matthew Toseland a écrit : What OS do you use for Freenet? Debian Squeeze What is your current datastore size set to? 100GiB What is your output bandwidth limit set to? 40KiB What actual bandwidth usage do you typically get? 40KiB ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
2009/5/6 Matthew Toseland t...@amphibian.dyndns.org: What OS do you use for Freenet? Gentoo hardened What is your current datastore size set to? 30 GB What is your output bandwidth limit set to? 15KB/s What actual bandwidth usage do you typically get? 13.7KB/s in the last 6 days ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Please answer a quick survey on Freenet
On Wed, May 6, 2009 at 8:54 AM, Matthew Toseland t...@amphibian.dyndns.org wrote: What OS do you use for Freenet? Debian Lenny What is your current datastore size set to? 100GiB What is your output bandwidth limit set to? 30KiB/s What actual bandwidth usage do you typically get? 28KiB/s This will help us to make decisions about new performance features ... ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wed, 6 May 2009 16:38:10 +0100, Matthew Toseland wrote: On Wednesday 06 May 2009 15:26:12 Dennis Nezic wrote: On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote: Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Maybe turning off logging helps? Does Juiceman also have logging enabled? This appears to only affect Microsoft users? You have a node on a linux system with queued downloads and no performance issues? Correct. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] CPU usage Re: Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, May 6, 2009 at 12:50 PM, Juiceman wrote: On Wed, May 6, 2009 at 11:37 AM, Matthew Toseland wrote: On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: Matthew Toseland wrote: This is the downside of db4o. If it is a widespread problem, we're gonna have to revert it. Which means throwing away more than 6 months work largely funded by Google's $18K. I think that using a database is a good idea (although I personally would've opted for a relational database such as Derby). So I'd prefer to try and understand and fix the issue rather than hiding from it :-). My database queue is usually pretty empty, even with queued downloads, but I have 8G and fast mirrored disks... The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb cache, SATA2, no NCQ - though definitely not the slowest out there. I see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll probably have to test the same from inside Java to make absolutely sure that it's not some weird JVM issue on my platform, though. 2650 handles is strange, on unix we are generally limited to 1024 and generally we don't exceed that. Both of your problems may be caused by flaky hardware, but frankly we do need to run on flaky real world hardware. :| I don't have Freenet running right now, will check it later. But I2P is using 2670 handles right now, and Azureus uses 1450 - so 2600 for Freenet is definitely nothing out of the ordinary on Windows. Oh, and the highest handle user on my machine is MySQL, which uses ~69000 handles and works absolutely fine :-). Same here. Enormous disk queues. I've also compared i/o counts with i/o bytes read/written - that's how I know that i/o operations are small. In the statistics screen, I routinely see 100+ outstanding database jobs. It can't be good. This just confirms that disk I/O is the problem ... and almost certainly caused by db4o as it goes away if nothing is queued. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Regards, Victor Denisov. One other thing, for both you and Juiceman: How's the CPU usage? Given how much RAM you have I would expect node.db4o to be cached in memory (how big is it?). But doing a read through the OS to the OS disk cache may cost a lot of CPU (context switch etc) ... Is there a lot of CPU usage for the freenet process? To the point that it might be the cause of the poor overall system performance? And how much CPU usage is system? Freenet CPU usage fluctuates between 2 and 27% of a quad core system. The rest of the machine rarely uses more than 15% unless I am gaming, then it still only hits 50%. CPU usage is quite acceptable for now. I have 3GB of RAM, 512 allocated to Freenet. Node.db4o was 375 MB. No uploads, 1 GB of queued downloads. How often is this file written to? Anyway to queue writes in a RAM buffer and write to disk periodically? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Use GnuPG with Firefox : http://getfiregpg.org (Version: 0.7.5) iEYEARECAAYFAkoCFKUACgkQ4esu1mlKOs/ocgCfdm8v9JstR1RrHMg3SM1/NnUK kvkAnj/fg5e0JCFwsJpPL+y+sEtC2/4V =4EOX -END PGP SIGNATURE- ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wednesday 06 May 2009 19:12:17 Victor Denisov wrote: The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. Well, a major objective of the db4o rewrite was precisely this. And I don't see that this is a problem - the OS will use the rest to cache the node.db4o file so that only writes need to go to disk. Yes, this I understand. I was one of those complaining of Freenet using too much RAM :-(. But, IMO, using as much memory as possible (out of the dedicated pool) could be important for performance. For example, by increasing buffer sizes in db4o we can possibly make flushes more organized, reducing disk writes substantially. No, I don't think we can, db4o is read-committed so avoiding commit() for a long period doesn't work afaik. I wonder if there are ways to tune db4o performance without rewriting the code, are there any handles to turn in the db4o config? It may be possible to increase the cache size, but this is purely for avoiding calling into the OS disk cache. I have seagate 1TB disks mirrored, may explain the difference. Unlikely, IMO. 1 Tb drives should have better throughput, but Freenet is definitely limited by seek times, which should be only marginally better. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. No, because it's an I/O problem, not a CPU/memory problem! I was thinking more about allocation/invocation counts, available in both profilers. Perhaps some method/query is being called unexpectedly often, or a certain object is being persisted too often, etc. Also, it could be that a certain method blocks too often and for too much time in Windows, leading to poor I/O performance. But it's a long shot, this I agree with. Regards, Victor Denisov. signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wednesday 06 May 2009 19:18:13 Victor Denisov wrote: Do you have uploads queued as well as downloads? Generally uploads cost a bit more than downloads do with db4o... No, only downloads. Total queued size varied between 25 Mb and 350 Mb in my tests (but actual total file size was often more than reported by Freenet, as some keys stayed at 0% for the duration of the test). Also, to clarify things, no background applications of notice (such as other P2P apps or distributed computing clients) were running during the test. I regularly run Azureus, eMule and I2P, but they all were stopped for the entire duration Freenet was running, as were MySQL and MS SQL Server instances I work on. I also tried disabling my antivirus/personal firewall (Agnitum Outpost Security Suite), but it didn't result in a noticeable improvement in performance. Okay. And you have plenty of RAM. How big is the node.db4o file? I'm assuming it fits very comfortably in RAM, so what we are talking about here are *writes*. Also, the node behaves like this (constant heavy disk i/o making using the system very problematic) for a long time, hours on end? Or just for spurts now and then? Please could you get me some debug information? Set the log level details to freenet.support.PrioritizedSerialExecutor:MINOR Let the node run for an hour or so. Send me your statistics page and the contents of your last log (maybe narrow it down by grep'ing for PrioritizedSerialExecutor, hopefully there shouldn't be any keys or anything on the output). Regards, Victor Denisov. signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] CPU usage Re: Is it my system , or had builds 1208-1209 have severe performa nce issues?
On Wednesday 06 May 2009 23:52:22 Juiceman wrote: On Wed, May 6, 2009 at 12:50 PM, Juiceman wrote: On Wed, May 6, 2009 at 11:37 AM, Matthew Toseland wrote: On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: Matthew Toseland wrote: This is the downside of db4o. If it is a widespread problem, we're gonna have to revert it. Which means throwing away more than 6 months work largely funded by Google's $18K. I think that using a database is a good idea (although I personally would've opted for a relational database such as Derby). So I'd prefer to try and understand and fix the issue rather than hiding from it :-). My database queue is usually pretty empty, even with queued downloads, but I have 8G and fast mirrored disks... The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb cache, SATA2, no NCQ - though definitely not the slowest out there. I see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll probably have to test the same from inside Java to make absolutely sure that it's not some weird JVM issue on my platform, though. 2650 handles is strange, on unix we are generally limited to 1024 and generally we don't exceed that. Both of your problems may be caused by flaky hardware, but frankly we do need to run on flaky real world hardware. :| I don't have Freenet running right now, will check it later. But I2P is using 2670 handles right now, and Azureus uses 1450 - so 2600 for Freenet is definitely nothing out of the ordinary on Windows. Oh, and the highest handle user on my machine is MySQL, which uses ~69000 handles and works absolutely fine :-). Same here. Enormous disk queues. I've also compared i/o counts with i/o bytes read/written - that's how I know that i/o operations are small. In the statistics screen, I routinely see 100+ outstanding database jobs. It can't be good. This just confirms that disk I/O is the problem ... and almost certainly caused by db4o as it goes away if nothing is queued. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Regards, Victor Denisov. One other thing, for both you and Juiceman: How's the CPU usage? Given how much RAM you have I would expect node.db4o to be cached in memory (how big is it?). But doing a read through the OS to the OS disk cache may cost a lot of CPU (context switch etc) ... Is there a lot of CPU usage for the freenet process? To the point that it might be the cause of the poor overall system performance? And how much CPU usage is system? Freenet CPU usage fluctuates between 2 and 27% of a quad core system. The rest of the machine rarely uses more than 15% unless I am gaming, then it still only hits 50%. CPU usage is quite acceptable for now. I have 3GB of RAM, 512 allocated to Freenet. Node.db4o was 375 MB. No uploads, 1 GB of queued downloads. How often is this file written to? Anyway to queue writes in a RAM buffer and write to disk periodically? I don't think so, at least not easily i.e. not without a custom IoAdapter able to buffer many commits separately. What I don't understand is what all these writes are *for*. If it's just downloads, most of the time it should just be selecting a SplitFileFetcherSubSegment, fetching all the blocks in it (without accessing the database), updating them all at once when they've failed, and then selecting a new segment - roughly every 2 minutes. However, I guess if most of the fetches succeed, that produces a lot more traffic. We have to write the block to disk when we fetch it, look up who owns it (because many fetchers can have a claim on one block), probably copy it, tell the SFFS and SFFSS about it, write the update to the SFFS, and then when we've got all the blocks for a segment do a load more work. signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] CPU usage Re: Is it my system, or had builds 1208-1209 have severe performance issues?
On Thu, 7 May 2009 00:23:37 +0100, Matthew Toseland wrote: On Wednesday 06 May 2009 23:52:22 Juiceman wrote: On Wed, May 6, 2009 at 12:50 PM, Juiceman wrote: On Wed, May 6, 2009 at 11:37 AM, Matthew Toseland wrote: On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: Matthew Toseland wrote: This is the downside of db4o. If it is a widespread problem, we're gonna have to revert it. Which means throwing away more than 6 months work largely funded by Google's $18K. I think that using a database is a good idea (although I personally would've opted for a relational database such as Derby). So I'd prefer to try and understand and fix the issue rather than hiding from it :-). My database queue is usually pretty empty, even with queued downloads, but I have 8G and fast mirrored disks... The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb cache, SATA2, no NCQ - though definitely not the slowest out there. I see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll probably have to test the same from inside Java to make absolutely sure that it's not some weird JVM issue on my platform, though. 2650 handles is strange, on unix we are generally limited to 1024 and generally we don't exceed that. Both of your problems may be caused by flaky hardware, but frankly we do need to run on flaky real world hardware. :| I don't have Freenet running right now, will check it later. But I2P is using 2670 handles right now, and Azureus uses 1450 - so 2600 for Freenet is definitely nothing out of the ordinary on Windows. Oh, and the highest handle user on my machine is MySQL, which uses ~69000 handles and works absolutely fine :-). Same here. Enormous disk queues. I've also compared i/o counts with i/o bytes read/written - that's how I know that i/o operations are small. In the statistics screen, I routinely see 100+ outstanding database jobs. It can't be good. This just confirms that disk I/O is the problem ... and almost certainly caused by db4o as it goes away if nothing is queued. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Regards, Victor Denisov. One other thing, for both you and Juiceman: How's the CPU usage? Given how much RAM you have I would expect node.db4o to be cached in memory (how big is it?). But doing a read through the OS to the OS disk cache may cost a lot of CPU (context switch etc) ... Is there a lot of CPU usage for the freenet process? To the point that it might be the cause of the poor overall system performance? And how much CPU usage is system? Freenet CPU usage fluctuates between 2 and 27% of a quad core system. The rest of the machine rarely uses more than 15% unless I am gaming, then it still only hits 50%. CPU usage is quite acceptable for now. I have 3GB of RAM, 512 allocated to Freenet. Node.db4o was 375 MB. No uploads, 1 GB of queued downloads. How often is this file written to? Anyway to queue writes in a RAM buffer and write to disk periodically? I don't think so, at least not easily i.e. not without a custom IoAdapter able to buffer many commits separately. What I don't understand is what all these writes are *for*. If it's just downloads, most of the time it should just be selecting a SplitFileFetcherSubSegment, fetching all the blocks in it (without accessing the database), updating them all at once when they've failed, and then selecting a new segment - roughly every 2 minutes. However, I guess if most of the fetches succeed, that produces a lot more traffic. We have to write the block to disk when we fetch it, look up who owns it (because many fetchers can have a claim on one block), probably copy it, tell the SFFS and SFFSS about it, write the update to the SFFS, and then when we've got all the blocks for a segment do a load more work. My 34MiB node.db40 is written to every couple of seconds. Every-second writes are common. Sometimes the filesize increases -- often times it stays the same -- although every time it changes (according to md5sum). Maybe for larger
Re: [freenet-support] Please answer a quick survey on Freenet
On Wednesday 06 May 2009 05:54:16 am Matthew Toseland wrote: What OS do you use for Freenet? Debian Squeeze (was Lenny until a few weeks ago) What is your current datastore size set to? 8GB (fits on a double-layer DVD for regular backups. I just don't trust hard drives.) What is your output bandwidth limit set to? 64kb/s What actual bandwidth usage do you typically get? ~40kb/s This will help us to make decisions about new performance features ... -- Sincerely, Jack Mudge jakyk...@theanythingbox.com ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
[freenet-support] Please answer a quick survey on Freenet
What OS do you use for Freenet? What is your current datastore size set to? What is your output bandwidth limit set to? What actual bandwidth usage do you typically get? This will help us to make decisions about new performance features ... -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/support/attachments/20090506/aeb2423d/attachment.pgp>
[freenet-support] Please answer a quick survey on Freenet
What OS do you use for Freenet? ---winxp What is your current datastore size set to? ---10G What is your output bandwidth limit set to? ---100Mbps What actual bandwidth usage do you typically get? ---10Mbps This will help us to make decisions about new performance features ... ___ Support mailing list Support at freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-request at freenetproject.org?subject=unsubscribe = = = = = = = = = = = = = = = = = = = = support at freenetproject.org
[freenet-support] Please answer a quick survey on Freenet
On Wed, May 6, 2009 at 14:54, Matthew Toseland wrote: > What OS do you use for Freenet? WinXP > > What is your current datastore size set to? 100 GB > > What is your output bandwidth limit set to? 32 kb/s > > What actual bandwidth usage do you typically get? 34 kb/s
[freenet-support] Please answer a quick survey on Freenet
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > What OS do you use for Freenet? Windows XP x64 > What is your current datastore size set to? 5 Gb > What is your output bandwidth limit set to? 100 Kb/s > What actual bandwidth usage do you typically get? Until 1208, around 75-85 Kb/s, with 1208+ anywhere from 10 to 50 Kb/s, depending on how much disk thrashing is going on. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAYwn1O5++4rTuI0RAosNAJ9zYaUKLu8MTM1dPoUzqyQ9QVxjvACghTut OkNaGnQLsDYcm9QPZd1TOao= =86xr -END PGP SIGNATURE-
[freenet-support] Please answer a quick survey on Freenet
Am Mittwoch, 6. Mai 2009 14:54:16 schrieb Matthew Toseland: > What OS do you use for Freenet? Gentoo, amd64 > What is your current datastore size set to? 240G (dedicated disk) > What is your output bandwidth limit set to? 70KB > What actual bandwidth usage do you typically get? ~45KB > This will help us to make decisions about new performance features ... MfG saces
[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matthew Toseland wrote: > This is the downside of db4o. If it is a widespread problem, we're gonna have > to revert it. Which means throwing away more than 6 months work largely > funded by Google's $18K. I think that using a database is a good idea (although I personally would've opted for a relational database such as Derby). So I'd prefer to try and understand and fix the issue rather than hiding from it :-). > My database queue is usually pretty empty, even with queued downloads, but I > have 8G and fast mirrored disks... The problem's that Freenet *doesn't* even use the amount of memory I provide it with (I'm yet to see it use more than 120 megs out of 320 I allow for the heap). I'd be willing to dedicate as much memory as required if only it'd help. My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb cache, SATA2, no NCQ - though definitely not the slowest out there. I see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll probably have to test the same from inside Java to make absolutely sure that it's not some weird JVM issue on my platform, though. > 2650 handles is strange, on unix we are generally limited to 1024 and > generally we don't exceed that. Both of your problems may be caused by flaky > hardware, but frankly we do need to run on flaky real world hardware. :| I don't have Freenet running right now, will check it later. But I2P is using 2670 handles right now, and Azureus uses 1450 - so 2600 for Freenet is definitely nothing out of the ordinary on Windows. Oh, and the highest handle user on my machine is MySQL, which uses ~69000 handles and works absolutely fine :-). >> Same here. Enormous disk queues. I've also compared i/o counts with i/o >> bytes read/written - that's how I know that i/o operations are small. In >> the statistics screen, I routinely see 100+ outstanding database jobs. >> It can't be good. > > This just confirms that disk I/O is the problem ... and almost certainly > caused by db4o as it goes away if nothing is queued. My thinking exactly. Would providing you with a snapshot of CPU/memory performance under YourKit Profiler (I have academic licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK distributive) on my machine help? Any logging I can turn on to help? BTW, I have logging set to ERROR for now, as with NORMAL level it logs ~2Mb per minute, adding noticeably to overall disk contention. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAZQf1O5++4rTuI0RAr5KAKCPuXmaqThbq0g8jVxdGwj7fNGZ/wCgsBBw YybivVLzs7FlJHvqXbvDJwA= =SWCw -END PGP SIGNATURE-
[freenet-support] Please answer a quick survey on Freenet
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > What OS do you use for Freenet? Ubuntu GNU/Linux 9.04 > What is your current datastore size set to? 20 GiB > What is your output bandwidth limit set to? 220 KiB/sec > What actual bandwidth usage do you typically get? 50 KiB/sec > This will help us to make decisions about new performance features ... Yippie! - Volodya - -- http://freedom.libsyn.com/ Echo of Freedom, Radical Podcast http://eng.anarchopedia.org/ Anarchopedia, A Free Knowledge Portal http://www.freedomporn.org/ Freedom Porn, anarchist and activist smut "None of us are free until all of us are free."~ Mihail Bakunin -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkoBlwAACgkQuWy2EFICg+0COACgpDuqysrASaEkxT6HwW9o6i+E 3uAAoMZUtCBQMMQa5jqN+qtC+mrHwdrw =yqs9 -END PGP SIGNATURE-
[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: > Matthew Toseland wrote: > > This is the downside of db4o. If it is a widespread problem, we're gonna have > > to revert it. Which means throwing away more than 6 months work largely > > funded by Google's $18K. > > I think that using a database is a good idea (although I personally > would've opted for a relational database such as Derby). So I'd prefer > to try and understand and fix the issue rather than hiding from it :-). > > > My database queue is usually pretty empty, even with queued downloads, but I > > have 8G and fast mirrored disks... > > The problem's that Freenet *doesn't* even use the amount of memory I > provide it with (I'm yet to see it use more than 120 megs out of 320 I > allow for the heap). I'd be willing to dedicate as much memory as > required if only it'd help. Well, a major objective of the db4o rewrite was precisely this. And I don't see that this is a problem - the OS will use the rest to cache the node.db4o file so that only writes need to go to disk. > > My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb > cache, SATA2, no NCQ - though definitely not the slowest out there. I > see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files > and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll > probably have to test the same from inside Java to make absolutely sure > that it's not some weird JVM issue on my platform, though. I have seagate 1TB disks mirrored, may explain the difference. > > > 2650 handles is strange, on unix we are generally limited to 1024 and > > generally we don't exceed that. Both of your problems may be caused by flaky > > hardware, but frankly we do need to run on flaky real world hardware. :| > > I don't have Freenet running right now, will check it later. But I2P is > using 2670 handles right now, and Azureus uses 1450 - so 2600 for > Freenet is definitely nothing out of the ordinary on Windows. Oh, and > the highest handle user on my machine is MySQL, which uses ~69000 > handles and works absolutely fine :-). > > >> Same here. Enormous disk queues. I've also compared i/o counts with i/o > >> bytes read/written - that's how I know that i/o operations are small. In > >> the statistics screen, I routinely see 100+ outstanding database jobs. > >> It can't be good. > > > > This just confirms that disk I/O is the problem ... and almost certainly > > caused by db4o as it goes away if nothing is queued. > > My thinking exactly. Would providing you with a snapshot of CPU/memory > performance under YourKit Profiler (I have academic licenses for both > 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK > distributive) on my machine help? Any logging I can turn on to help? > BTW, I have logging set to ERROR for now, as with NORMAL level it logs > ~2Mb per minute, adding noticeably to overall disk contention. No, because it's an I/O problem, not a CPU/memory problem! > > Regards, > Victor Denisov. -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/support/attachments/20090506/9e515922/attachment.pgp>
[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: > Matthew Toseland wrote: > > This is the downside of db4o. If it is a widespread problem, we're gonna have > > to revert it. Which means throwing away more than 6 months work largely > > funded by Google's $18K. > > I think that using a database is a good idea (although I personally > would've opted for a relational database such as Derby). So I'd prefer > to try and understand and fix the issue rather than hiding from it :-). > > > My database queue is usually pretty empty, even with queued downloads, but I > > have 8G and fast mirrored disks... > > The problem's that Freenet *doesn't* even use the amount of memory I > provide it with (I'm yet to see it use more than 120 megs out of 320 I > allow for the heap). I'd be willing to dedicate as much memory as > required if only it'd help. > > My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb > cache, SATA2, no NCQ - though definitely not the slowest out there. I > see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files > and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll > probably have to test the same from inside Java to make absolutely sure > that it's not some weird JVM issue on my platform, though. > > > 2650 handles is strange, on unix we are generally limited to 1024 and > > generally we don't exceed that. Both of your problems may be caused by flaky > > hardware, but frankly we do need to run on flaky real world hardware. :| > > I don't have Freenet running right now, will check it later. But I2P is > using 2670 handles right now, and Azureus uses 1450 - so 2600 for > Freenet is definitely nothing out of the ordinary on Windows. Oh, and > the highest handle user on my machine is MySQL, which uses ~69000 > handles and works absolutely fine :-). > > >> Same here. Enormous disk queues. I've also compared i/o counts with i/o > >> bytes read/written - that's how I know that i/o operations are small. In > >> the statistics screen, I routinely see 100+ outstanding database jobs. > >> It can't be good. > > > > This just confirms that disk I/O is the problem ... and almost certainly > > caused by db4o as it goes away if nothing is queued. > > My thinking exactly. Would providing you with a snapshot of CPU/memory > performance under YourKit Profiler (I have academic licenses for both > 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK > distributive) on my machine help? Any logging I can turn on to help? > BTW, I have logging set to ERROR for now, as with NORMAL level it logs > ~2Mb per minute, adding noticeably to overall disk contention. Do you have uploads queued as well as downloads? Generally uploads cost a bit more than downloads do with db4o... -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/support/attachments/20090506/15cad097/attachment.pgp>
[freenet-support] Please answer a quick survey on Freenet
Matthew Toseland schrieb: > What OS do you use for Freenet? Ubuntu 8.04 server > > What is your current datastore size set to? 30GB > > What is your output bandwidth limit set to? 300KiB/s > > What actual bandwidth usage do you typically get? 1.5MiB/s > > This will help us to make decisions about new performance features ... > > > > > ___ > Support mailing list > Support at freenetproject.org > http://news.gmane.org/gmane.network.freenet.support > Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support > Or mailto:support-request at freenetproject.org?subject=unsubscribe
[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote: > Any logging I can turn on to help? BTW, I have logging set to ERROR > for now, as with NORMAL level it logs ~2Mb per minute, adding > noticeably to overall disk contention. Maybe turning off logging helps? Does Juiceman also have logging enabled? This appears to only affect Microsoft users?
[freenet-support] Please answer a quick survey on Freenet
On Wed, 6 May 2009 13:54:16 +0100, Matthew Toseland wrote: > What OS do you use for Freenet? Gentoo Linux, amd64 > What is your current datastore size set to? 8GiB > What is your output bandwidth limit set to? 9KB/s > What actual bandwidth usage do you typically get? 7KB/s (average, my ISP throttles)
[freenet-support] Please answer a quick survey on Freenet
What OS do you use for Freenet? Ubuntu GNU Linux 9.04 AMD64 What is your current datastore size set to? 16 GB What is your output bandwidth limit set to? 900 KB/s What actual bandwidth usage do you typically get? 65KB/s when no QOS, and between 0 and 25 KB/s when use the port 53 (when my ISP blocks other UDP ports). This will help us to make decisions about new performance features ... On Wed, May 6, 2009 at 4:30 PM, Dennis Nezic wrote: > On Wed, 6 May 2009 13:54:16 +0100, Matthew Toseland wrote: > > > What OS do you use for Freenet? > Gentoo Linux, amd64 > > > What is your current datastore size set to? > 8GiB > > > What is your output bandwidth limit set to? > 9KB/s > > > What actual bandwidth usage do you typically get? > 7KB/s (average, my ISP throttles) > ___ > Support mailing list > Support at freenetproject.org > http://news.gmane.org/gmane.network.freenet.support > Unsubscribe at > http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support > Or mailto:support-request at freenetproject.org?subject=unsubscribe > -- next part -- An HTML attachment was scrubbed... URL: <https://emu.freenetproject.org/pipermail/support/attachments/20090506/df19c887/attachment.html>
[freenet-support] CPU usage Re: Is it my system, or had builds 1208-1209 have severe performance issues?
On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: > Matthew Toseland wrote: > > This is the downside of db4o. If it is a widespread problem, we're gonna have > > to revert it. Which means throwing away more than 6 months work largely > > funded by Google's $18K. > > I think that using a database is a good idea (although I personally > would've opted for a relational database such as Derby). So I'd prefer > to try and understand and fix the issue rather than hiding from it :-). > > > My database queue is usually pretty empty, even with queued downloads, but I > > have 8G and fast mirrored disks... > > The problem's that Freenet *doesn't* even use the amount of memory I > provide it with (I'm yet to see it use more than 120 megs out of 320 I > allow for the heap). I'd be willing to dedicate as much memory as > required if only it'd help. > > My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb > cache, SATA2, no NCQ - though definitely not the slowest out there. I > see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files > and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll > probably have to test the same from inside Java to make absolutely sure > that it's not some weird JVM issue on my platform, though. > > > 2650 handles is strange, on unix we are generally limited to 1024 and > > generally we don't exceed that. Both of your problems may be caused by flaky > > hardware, but frankly we do need to run on flaky real world hardware. :| > > I don't have Freenet running right now, will check it later. But I2P is > using 2670 handles right now, and Azureus uses 1450 - so 2600 for > Freenet is definitely nothing out of the ordinary on Windows. Oh, and > the highest handle user on my machine is MySQL, which uses ~69000 > handles and works absolutely fine :-). > > >> Same here. Enormous disk queues. I've also compared i/o counts with i/o > >> bytes read/written - that's how I know that i/o operations are small. In > >> the statistics screen, I routinely see 100+ outstanding database jobs. > >> It can't be good. > > > > This just confirms that disk I/O is the problem ... and almost certainly > > caused by db4o as it goes away if nothing is queued. > > My thinking exactly. Would providing you with a snapshot of CPU/memory > performance under YourKit Profiler (I have academic licenses for both > 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK > distributive) on my machine help? Any logging I can turn on to help? > BTW, I have logging set to ERROR for now, as with NORMAL level it logs > ~2Mb per minute, adding noticeably to overall disk contention. > > Regards, > Victor Denisov. One other thing, for both you and Juiceman: How's the CPU usage? Given how much RAM you have I would expect node.db4o to be cached in memory (how big is it?). But doing a read through the OS to the OS disk cache may cost a lot of CPU (context switch etc) ... Is there a lot of CPU usage for the freenet process? To the point that it might be the cause of the poor overall system performance? And how much CPU usage is system? -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/support/attachments/20090506/f8c8b6fd/attachment.pgp>
[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wednesday 06 May 2009 15:26:12 Dennis Nezic wrote: > On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote: > > Any logging I can turn on to help? BTW, I have logging set to ERROR > > for now, as with NORMAL level it logs ~2Mb per minute, adding > > noticeably to overall disk contention. > > Maybe turning off logging helps? Does Juiceman also have logging > enabled? This appears to only affect Microsoft users? You have a node on a linux system with queued downloads and no performance issues? -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/support/attachments/20090506/87f4102e/attachment.pgp>
[freenet-support] Please answer a quick survey on Freenet
On Wed, May 6, 2009 at 8:54 AM, Matthew Toseland wrote: > What OS do you use for Freenet? WinXP_SP3 > What is your current datastore size set to? 20G > What is your output bandwidth limit set to? 60KB > What actual bandwidth usage do you typically get? 42 avg prior to db4o Additional info: 512MB ram allocated to Freenet uptime 24x7 prior to db4o > This will help us to make decisions about new performance features ... > > ___ > Support mailing list > Support at freenetproject.org > http://news.gmane.org/gmane.network.freenet.support > Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support > Or mailto:support-request at freenetproject.org?subject=unsubscribe > -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin
[freenet-support] CPU usage Re: Is it my system, or had builds 1208-1209 have severe performance issues?
On Wed, May 6, 2009 at 11:37 AM, Matthew Toseland wrote: > On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: >> Matthew Toseland wrote: >> > This is the downside of db4o. If it is a widespread problem, we're gonna > have >> > to revert it. Which means throwing away more than 6 months work largely >> > funded by Google's $18K. >> >> I think that using a database is a good idea (although I personally >> would've opted for a relational database such as Derby). So I'd prefer >> to try and understand and fix the issue rather than hiding from it :-). >> >> > My database queue is usually pretty empty, even with queued downloads, but > I >> > have 8G and fast mirrored disks... >> >> The problem's that Freenet *doesn't* even use the amount of memory I >> provide it with (I'm yet to see it use more than 120 megs out of 320 I >> allow for the heap). I'd be willing to dedicate as much memory as >> required if only it'd help. >> >> My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb >> cache, SATA2, no NCQ - though definitely not the slowest out there. I >> see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files >> and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll >> probably have to test the same from inside Java to make absolutely sure >> that it's not some weird JVM issue on my platform, though. >> >> > 2650 handles is strange, on unix we are generally limited to 1024 and >> > generally we don't exceed that. Both of your problems may be caused by > flaky >> > hardware, but frankly we do need to run on flaky real world hardware. :| >> >> I don't have Freenet running right now, will check it later. But I2P is >> using 2670 handles right now, and Azureus uses 1450 - so 2600 for >> Freenet is definitely nothing out of the ordinary on Windows. Oh, and >> the highest handle user on my machine is MySQL, which uses ~69000 >> handles and works absolutely fine :-). >> >> >> Same here. Enormous disk queues. I've also compared i/o counts with i/o >> >> bytes read/written - that's how I know that i/o operations are small. In >> >> the statistics screen, I routinely see 100+ outstanding database jobs. >> >> It can't be good. >> > >> > This just confirms that disk I/O is the problem ... and almost certainly >> > caused by db4o as it goes away if nothing is queued. >> >> My thinking exactly. Would providing you with a snapshot of CPU/memory >> performance under YourKit Profiler (I have academic licenses for both >> 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK >> distributive) on my machine help? Any logging I can turn on to help? >> BTW, I have logging set to ERROR for now, as with NORMAL level it logs >> ~2Mb per minute, adding noticeably to overall disk contention. >> >> Regards, >> Victor Denisov. > > One other thing, for both you and Juiceman: > How's the CPU usage? Given how much RAM you have I would expect node.db4o to > be cached in memory (how big is it?). But doing a read through the OS to the > OS disk cache may cost a lot of CPU (context switch etc) ... Is there a lot > of CPU usage for the freenet process? To the point that it might be the cause > of the poor overall system performance? And how much CPU usage is system? > Freenet CPU usage fluctuates between 2 and 27% of a quad core system. The rest of the machine rarely uses more than 15% unless I am gaming, then it still only hits 50%. CPU usage is quite acceptable for now. I have 3GB of RAM, 512 allocated to Freenet. -- I may disagree with what you have to say, but I shall defend, to the death, your right to say it. - Voltaire Those who would give up Liberty, to purchase temporary Safety, deserve neither Liberty nor Safety. - Ben Franklin
[freenet-support] Please answer a quick survey on Freenet
On Wednesday 06 May 2009 13:54:16 Matthew Toseland wrote: > What OS do you use for Freenet? > > What is your current datastore size set to? > > What is your output bandwidth limit set to? > > What actual bandwidth usage do you typically get? > > This will help us to make decisions about new performance features ... > Additional useful info, if you don't mind parting with it: Network, friends and physical security levels. -- next part -- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 835 bytes Desc: This is a digitally signed message part. URL: <https://emu.freenetproject.org/pipermail/support/attachments/20090506/612874cf/attachment.pgp>
[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 >> The problem's that Freenet *doesn't* even use the amount of memory I >> provide it with (I'm yet to see it use more than 120 megs out of 320 I >> allow for the heap). I'd be willing to dedicate as much memory as >> required if only it'd help. > > Well, a major objective of the db4o rewrite was precisely this. And I don't > see that this is a problem - the OS will use the rest to cache the node.db4o > file so that only writes need to go to disk. Yes, this I understand. I was one of those complaining of Freenet using too much RAM :-(. But, IMO, using as much memory as possible (out of the dedicated pool) could be important for performance. For example, by increasing buffer sizes in db4o we can possibly make flushes more "organized", reducing disk writes substantially. I wonder if there are ways to tune db4o performance without rewriting the code, are there any handles to turn in the db4o config? > I have seagate 1TB disks mirrored, may explain the difference. Unlikely, IMO. 1 Tb drives should have better throughput, but Freenet is definitely limited by seek times, which should be only marginally better. >> My thinking exactly. Would providing you with a snapshot of CPU/memory >> performance under YourKit Profiler (I have academic licenses for both >> 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK >> distributive) on my machine help? Any logging I can turn on to help? >> BTW, I have logging set to ERROR for now, as with NORMAL level it logs >> ~2Mb per minute, adding noticeably to overall disk contention. > > No, because it's an I/O problem, not a CPU/memory problem! I was thinking more about allocation/invocation counts, available in both profilers. Perhaps some method/query is being called unexpectedly often, or a certain object is being persisted too often, etc. Also, it could be that a certain method blocks too often and for too much time in Windows, leading to poor I/O performance. But it's a long shot, this I agree with. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAdMB1O5++4rTuI0RAi/tAKC8SIIIeUVlQzYtntg22Uxjywp59ACdG5zQ ICyGAM6ubeh6I8JFIDwkjLs= =9uoG -END PGP SIGNATURE-
[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > Do you have uploads queued as well as downloads? Generally uploads cost a bit > more than downloads do with db4o... No, only downloads. Total queued size varied between 25 Mb and 350 Mb in my tests (but actual total file size was often more than reported by Freenet, as some keys stayed at 0% for the duration of the test). Also, to clarify things, no background applications of notice (such as other P2P apps or distributed computing clients) were running during the test. I regularly run Azureus, eMule and I2P, but they all were stopped for the entire duration Freenet was running, as were MySQL and MS SQL Server instances I work on. I also tried disabling my antivirus/personal firewall (Agnitum Outpost Security Suite), but it didn't result in a noticeable improvement in performance. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAdRl1O5++4rTuI0RAh5KAKDCoFmtLlfKGp0/2GZ4SQv9+/NmSQCfX2YW oIuQHcSPsl7/smqkL1b9Ykk= =7RhK -END PGP SIGNATURE-
[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > Maybe turning off logging helps? Does Juiceman also have logging > enabled? This appears to only affect Microsoft users? On ERROR, Freenet writes about 2-5 Kb per 10 minutes, which is really nothing. On NORMAL, it writes up to 5 Mb per 10 minutes, or ~ 8 Kb/s (which probably translates into ~10 to 20 writes per second, considering internal buffering and OS write-behind caching). Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAdhA1O5++4rTuI0RAm0tAJ4p5tTvVrYmhgRdPxXsTDM0wElxqgCbBYoD xGvyyHMG94ZlzrOrNI+fWaU= =E+sh -END PGP SIGNATURE-
[freenet-support] Please answer a quick survey on Freenet
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 > Additional useful info, if you don't mind parting with it: > Network, friends and physical security levels. All on NORMAL here. Regards, Victor Denisov. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKAdtL1O5++4rTuI0RAuk7AKCe4zdHlH64Pxr6bvbUudQ/REtJowCg4ajH zO5o2MtuUn2mQqzsKd9kst4= =sL3K -END PGP SIGNATURE-
[freenet-support] Please answer a quick survey on Freenet
On 5/6/09, Matthew Toseland wrote: > What OS do you use for Freenet? > FreeBSD 7.2-PRERELEASE > What is your current datastore size set to? > 300 GiB > What is your output bandwidth limit set to? > 100 KiB/s > What actual bandwidth usage do you typically get? > ~50 KiB/s > This will help us to make decisions about new performance features ... >
[freenet-support] Please answer a quick survey on Freenet
Matthew Toseland a ?crit : > What OS do you use for Freenet? > Debian Squeeze > What is your current datastore size set to? > 100GiB > What is your output bandwidth limit set to? > 40KiB > What actual bandwidth usage do you typically get? > 40KiB
[freenet-support] Please answer a quick survey on Freenet
On Wed, May 6, 2009 at 8:54 AM, Matthew Toseland wrote: > What OS do you use for Freenet? Debian Lenny > What is your current datastore size set to? 100GiB > What is your output bandwidth limit set to? 30KiB/s > What actual bandwidth usage do you typically get? 28KiB/s > This will help us to make decisions about new performance features ... > > ___ > Support mailing list > Support at freenetproject.org > http://news.gmane.org/gmane.network.freenet.support > Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support > Or mailto:support-request at freenetproject.org?subject=unsubscribe >
[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?
On Wed, 6 May 2009 16:38:10 +0100, Matthew Toseland wrote: > On Wednesday 06 May 2009 15:26:12 Dennis Nezic wrote: > > On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote: > > > Any logging I can turn on to help? BTW, I have logging set to > > > ERROR for now, as with NORMAL level it logs ~2Mb per minute, > > > adding noticeably to overall disk contention. > > > > Maybe turning off logging helps? Does Juiceman also have logging > > enabled? This appears to only affect Microsoft users? > > You have a node on a linux system with queued downloads and no > performance issues? Correct.
[freenet-support] CPU usage Re: Is it my system, or had builds 1208-1209 have severe performance issues?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, May 6, 2009 at 12:50 PM, Juiceman wrote: > On Wed, May 6, 2009 at 11:37 AM, Matthew Toseland > wrote: >> On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: >>> Matthew Toseland wrote: >>> > This is the downside of db4o. If it is a widespread problem, we're gonna >> have >>> > to revert it. Which means throwing away more than 6 months work largely >>> > funded by Google's $18K. >>> >>> I think that using a database is a good idea (although I personally >>> would've opted for a relational database such as Derby). So I'd prefer >>> to try and understand and fix the issue rather than hiding from it :-). >>> >>> > My database queue is usually pretty empty, even with queued downloads, but >> I >>> > have 8G and fast mirrored disks... >>> >>> The problem's that Freenet *doesn't* even use the amount of memory I >>> provide it with (I'm yet to see it use more than 120 megs out of 320 I >>> allow for the heap). I'd be willing to dedicate as much memory as >>> required if only it'd help. >>> >>> My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb >>> cache, SATA2, no NCQ - though definitely not the slowest out there. I >>> see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files >>> and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll >>> probably have to test the same from inside Java to make absolutely sure >>> that it's not some weird JVM issue on my platform, though. >>> >>> > 2650 handles is strange, on unix we are generally limited to 1024 and >>> > generally we don't exceed that. Both of your problems may be caused by >> flaky >>> > hardware, but frankly we do need to run on flaky real world hardware. :| >>> >>> I don't have Freenet running right now, will check it later. But I2P is >>> using 2670 handles right now, and Azureus uses 1450 - so 2600 for >>> Freenet is definitely nothing out of the ordinary on Windows. Oh, and >>> the highest handle user on my machine is MySQL, which uses ~69000 >>> handles and works absolutely fine :-). >>> >>> >> Same here. Enormous disk queues. I've also compared i/o counts with i/o >>> >> bytes read/written - that's how I know that i/o operations are small. In >>> >> the statistics screen, I routinely see 100+ outstanding database jobs. >>> >> It can't be good. >>> > >>> > This just confirms that disk I/O is the problem ... and almost certainly >>> > caused by db4o as it goes away if nothing is queued. >>> >>> My thinking exactly. Would providing you with a snapshot of CPU/memory >>> performance under YourKit Profiler (I have academic licenses for both >>> 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK >>> distributive) on my machine help? Any logging I can turn on to help? >>> BTW, I have logging set to ERROR for now, as with NORMAL level it logs >>> ~2Mb per minute, adding noticeably to overall disk contention. >>> >>> Regards, >>> Victor Denisov. >> >> One other thing, for both you and Juiceman: >> How's the CPU usage? Given how much RAM you have I would expect node.db4o to >> be cached in memory (how big is it?). But doing a read through the OS to the >> OS disk cache may cost a lot of CPU (context switch etc) ... Is there a lot >> of CPU usage for the freenet process? To the point that it might be the cause >> of the poor overall system performance? And how much CPU usage is system? >> > > Freenet CPU usage fluctuates between 2 and 27% of a quad core system. > The rest of the machine rarely uses more than 15% unless I am gaming, > then it still only hits 50%. CPU usage is quite acceptable for now. > I have 3GB of RAM, 512 allocated to Freenet. Node.db4o was 375 MB. No uploads, 1 GB of queued downloads. How often is this file written to? Anyway to queue writes in a RAM buffer and write to disk periodically? -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Use GnuPG with Firefox : http://getfiregpg.org (Version: 0.7.5) iEYEARECAAYFAkoCFKUACgkQ4esu1mlKOs/ocgCfdm8v9JstR1RrHMg3SM1/NnUK kvkAnj/fg5e0JCFwsJpPL+y+sEtC2/4V =4EOX -END PGP SIGNATURE-
[freenet-support] CPU usage Re: Is it my system, or had builds 1208-1209 have severe performance issues?
On Thu, 7 May 2009 00:23:37 +0100, Matthew Toseland wrote: > On Wednesday 06 May 2009 23:52:22 Juiceman wrote: > > > > On Wed, May 6, 2009 at 12:50 PM, Juiceman wrote: > > > On Wed, May 6, 2009 at 11:37 AM, Matthew Toseland > > > wrote: > > >> On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote: > > >>> Matthew Toseland wrote: > > >>> > This is the downside of db4o. If it is a widespread problem, > > >>> > we're > gonna > > >> have > > >>> > to revert it. Which means throwing away more than 6 months > > >>> > work > largely > > >>> > funded by Google's $18K. > > >>> > > >>> I think that using a database is a good idea (although I > > >>> personally would've opted for a relational database such as > > >>> Derby). So I'd prefer to try and understand and fix the issue > > >>> rather than hiding from it :-). > > >>> > > >>> > My database queue is usually pretty empty, even with queued > > >>> > downloads, > but > > >> I > > >>> > have 8G and fast mirrored disks... > > >>> > > >>> The problem's that Freenet *doesn't* even use the amount of > > >>> memory I provide it with (I'm yet to see it use more than 120 > > >>> megs out of 320 I allow for the heap). I'd be willing to > > >>> dedicate as much memory as required if only it'd help. > > >>> > > >>> My hard drives are nothing special - 250Gb 7200 RPM Seagate > > >>> ones, 16 Mb cache, SATA2, no NCQ - though definitely not the > > >>> slowest out there. I see ~35 Mb/s read speed and ~28 Mb/s write > > >>> speed for medium-sized files and ~5 Mb/s to 8 Mb/s for small > > >>> files in the tests I'd done. I'll probably have to test the > > >>> same from inside Java to make absolutely sure that it's not > > >>> some weird JVM issue on my platform, though. > > >>> > > >>> > 2650 handles is strange, on unix we are generally limited to > > >>> > 1024 and generally we don't exceed that. Both of your > > >>> > problems may be caused by > > >> flaky > > >>> > hardware, but frankly we do need to run on flaky real world > hardware. :| > > >>> > > >>> I don't have Freenet running right now, will check it later. > > >>> But I2P is using 2670 handles right now, and Azureus uses 1450 > > >>> - so 2600 for Freenet is definitely nothing out of the ordinary > > >>> on Windows. Oh, and the highest handle user on my machine is > > >>> MySQL, which uses ~69000 handles and works absolutely fine :-). > > >>> > > >>> >> Same here. Enormous disk queues. I've also compared i/o > > >>> >> counts with > i/o > > >>> >> bytes read/written - that's how I know that i/o operations > > >>> >> are small. > In > > >>> >> the statistics screen, I routinely see 100+ outstanding > > >>> >> database > jobs. > > >>> >> It can't be good. > > >>> > > > >>> > This just confirms that disk I/O is the problem ... and > > >>> > almost > certainly > > >>> > caused by db4o as it goes away if nothing is queued. > > >>> > > >>> My thinking exactly. Would providing you with a snapshot of > > >>> CPU/memory performance under YourKit Profiler (I have academic > > >>> licenses for both 7.5 and 8.0, IIRC) or VisualVM (which is now > > >>> a part of the JDK distributive) on my machine help? Any logging > > >>> I can turn on to help? BTW, I have logging set to ERROR for > > >>> now, as with NORMAL level it logs ~2Mb per minute, adding > > >>> noticeably to overall disk contention. > > >>> > > >>> Regards, > > >>> Victor Denisov. > > >> > > >> One other thing, for both you and Juiceman: > > >> How's the CPU usage? Given how much RAM you have I would expect > > >> node.db4o > to > > >> be cached in memory (how big is it?). But doing a read through > > >> the OS to > the > > >> OS disk cache may cost a lot of CPU (context switch etc) ... Is > > >> there a > lot > > >> of CPU usage for the freenet process? To the point that it might > > >> be the > cause > > >> of the poor overall system performance? And how much CPU usage > > >> is system? > > >> > > > > > > Freenet CPU usage fluctuates between 2 and 27% of a quad core > > > system. The rest of the machine rarely uses more than 15% unless > > > I am gaming, then it still only hits 50%. CPU usage is quite > > > acceptable for now. I have 3GB of RAM, 512 allocated to Freenet. > > > > Node.db4o was 375 MB. No uploads, 1 GB of queued downloads. > > > > How often is this file written to? Anyway to queue writes in a RAM > > buffer and write to disk periodically? > > I don't think so, at least not easily i.e. not without a custom > IoAdapter able to buffer many commits separately. What I don't > understand is what all these writes are *for*. If it's just > downloads, most of the time it should just be selecting a > SplitFileFetcherSubSegment, fetching all the blocks in it (without > accessing the database), updating them all at once when they've > failed, and then selecting a new segment - roughly every 2 minutes. > > However, I guess if most of the fetches succeed, that produces a lot > more traffic. We have to write the block to disk when we fetch it, >
[freenet-support] Please answer a quick survey on Freenet
On Wednesday 06 May 2009 05:54:16 am Matthew Toseland wrote: > What OS do you use for Freenet? Debian Squeeze (was Lenny until a few weeks ago) > > What is your current datastore size set to? 8GB (fits on a double-layer DVD for regular backups. I just don't trust hard drives.) > > What is your output bandwidth limit set to? 64kb/s > > What actual bandwidth usage do you typically get? ~40kb/s > > This will help us to make decisions about new performance features ... -- Sincerely, Jack Mudge jakykong at theanythingbox.com
