Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Saturday 06 June 2009 22:56:03 Victor Denisov wrote:
  Do you (anyone, everyone, especially on windows with low end
  hardware) get good performance with queued downloads on 1214 now? Can
  I close the bug concerning this thread? (#3075)
 
 I've ran 1215 for a few hours now under various loads, and can say that
 on my machine the node works much better now, unless I really stress it
 (like queuing 100+ downloads).

What happens if you queue 100 downloads? I guess the problem is that when it 
reaches a certain point (e.g. fetching the metadata on a large file) it has to 
do a lot of work resulting in disk thrashing for a long period when you have 
loads of them ... but it should settle eventually?
 
 Regards,
 Victor Denisov.


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

 Do you (anyone, everyone, especially on windows with low end
 hardware) get good performance with queued downloads on 1214 now? Can
 I close the bug concerning this thread? (#3075)

I've ran 1215 for a few hours now under various loads, and can say that
on my machine the node works much better now, unless I really stress it
(like queuing 100+ downloads).

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKKuXzx7AVSvyjsUARAgs2AKDFiHxWk8sHMgt7QcJ6bdqHUFFjGgCcCqYk
5GDeuqmAcm7WdxzrqN5GyYc=
=g+cn
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Monday 04 May 2009 21:21:50 Victor Denisov wrote:
 Hello,
 
 Since my node autoupdated to 1208, my system's performance degraded so
 much as to render it completely unusable while Freenet is running. I
 can't perform simplest tasks, such as surfing or typing a document, as
 switching between tabs in Opera can take 10+ seconds and delay between
 typing a letter and it showing up in Word can be several seconds as
 well. Anything more stressing (such as running a game or developing an
 application) is simply not possible.
 
 From what I can see, the reason for this is that Freenet makes hundreds
 of small disk i/o ops per second, basically blocking the OS from
 accessing the hard drive for swapping and such.
 
 The above is definitely affected by the queue size. First, I tried
 adding ~ 100 random files from Thaw when the node first updated, but
 hadn't had the patience to wait for the request to complete (I think I
 waited at least 20 minutes, perhaps more - with the system being nearly
 paralyzed by the constant HDD thrashing). With just 3 or 4 files being
 put in the queue the system starts to stutter noticeably, provided that
 files start downloading and not hang at 0%. The only time when I can run
 Freenet as a real background app is when I don't have any files in the
 queue and FMS isn't running (which seems more or less pointless to me :-().
 
 My system is set up as follows:
 
 AMD Athlon 64 X2 6000+
 4 Gb RAM
 2x250 Gb 7200 rpm SATA2 HDD (mirror via mobo's built-in nVidia 570 RAID
 controller)
 Windows XP x64
 Java 1.6.0_07 64-bit
 Of course, all the latest updates/patches/drivers, etc.
 Freenet uses 5 Gb datastore on an unencrypted partition.
 
 Interesting thing I noticed was that Freenet significantly underutilized
 the memory I provide it with. From 320 Mb heap memory available, I
 hadn't seen it allocate more than ~ 80 Mb.
 
 Any thoughts on why this could be happening? I hadn't seen anyone
 complain about the performance of 1208/09 yet, so it is probably
 something with my machine :-(, but it beats me what it could be :-(.

Do you (anyone, everyone, especially on windows with low end hardware) get good 
performance with queued downloads on 1214 now? Can I close the bug concerning 
this thread? (#3075)
 
 Regards,
 Victor Denisov.


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Juiceman]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1



On Fri, Jun 5, 2009 at 3:55 PM, Matthew Toseland wrote:
 On Monday 04 May 2009 21:21:50 Victor Denisov wrote:
 Hello,

 Since my node autoupdated to 1208, my system's performance degraded so
 much as to render it completely unusable while Freenet is running. I
 can't perform simplest tasks, such as surfing or typing a document, as
 switching between tabs in Opera can take 10+ seconds and delay between
 typing a letter and it showing up in Word can be several seconds as
 well. Anything more stressing (such as running a game or developing an
 application) is simply not possible.

 From what I can see, the reason for this is that Freenet makes hundreds
 of small disk i/o ops per second, basically blocking the OS from
 accessing the hard drive for swapping and such.

 The above is definitely affected by the queue size. First, I tried
 adding ~ 100 random files from Thaw when the node first updated, but
 hadn't had the patience to wait for the request to complete (I think I
 waited at least 20 minutes, perhaps more - with the system being nearly
 paralyzed by the constant HDD thrashing). With just 3 or 4 files being
 put in the queue the system starts to stutter noticeably, provided that
 files start downloading and not hang at 0%. The only time when I can run
 Freenet as a real background app is when I don't have any files in the
 queue and FMS isn't running (which seems more or less pointless to me :-().

 My system is set up as follows:

 AMD Athlon 64 X2 6000+
 4 Gb RAM
 2x250 Gb 7200 rpm SATA2 HDD (mirror via mobo's built-in nVidia 570 RAID
 controller)
 Windows XP x64
 Java 1.6.0_07 64-bit
 Of course, all the latest updates/patches/drivers, etc.
 Freenet uses 5 Gb datastore on an unencrypted partition.

 Interesting thing I noticed was that Freenet significantly underutilized
 the memory I provide it with. From 320 Mb heap memory available, I
 hadn't seen it allocate more than ~ 80 Mb.

 Any thoughts on why this could be happening? I hadn't seen anyone
 complain about the performance of 1208/09 yet, so it is probably
 something with my machine :-(, but it beats me what it could be :-(.

 Do you (anyone, everyone, especially on windows with low end hardware) get 
 good performance with queued downloads on 1214 now? Can I close the bug 
 concerning this thread? (#3075)

I don't have low-end hardware, but my Windows node gets excellent
performance now, thank you! =)

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Use GnuPG with Firefox : http://getfiregpg.org (Version: 0.7.5)

iEYEARECAAYFAkopjd4ACgkQ4esu1mlKOs9jggCffyjjlNw/MBZOlDKKGeD+45uE
rSMAnROPiAZTiJFQx5Z8RxcG1YQtOIca
=ZQ0I
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208 -1209 have severe performance issues?

[Matthew Toseland]

On Sunday 17 May 2009 22:24:57 Juiceman wrote:
 On Fri, May 15, 2009 at 2:37 PM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  On Wednesday 13 May 2009 18:29:47 Evan Daniel wrote:
  On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
  t...@amphibian.dyndns.org wrote:
   On Friday 08 May 2009 17:40:58 Juiceman wrote:
Weird.  node.db4o was an insane 375 MB.  I deleted it and and added 
a
bunch of downloads.  Now it is less than 10 MB.  That definitely
helped some with the disk thrashing.
   
I think I found the main problem, and I'm embarrassed to say
apparantly I had xmlspider plugin running and writing GB+ files to 
the
same disk the node resides on.  I turned this off and the disk 
usage
became manageable.
   
I also upgraded my HDD from an older 2 MB cache model to one with 
16
MB and now Freenet is zipping along nicely.
   
I did see some errors in the log so I am sending it to Toad for
  review.
   
P.S. I would recommend not installing the xmlspider by default on
   installs.
   
Victor - might this be your issue as well?
   
ROFL. So that just leaves victor...
  
   Is it normal that node.db4o never shrinks?  I have completed all the
   downloads I had running and removed them from the page, yet node.db4o
   doesn't get smaller.  I have rebooted the node also.  This IMHO is bad
   because it will eventually kill performance with disk access...
  
   Yes, the only way to ensure it shrinks is to defrag it. This is on the
  todo
   list, but it does not seem urgent to me. Is it really a huge, 
monstrous,
   evil, all-consuming problem more urgent than the 500 other things we 
have
  to
   deal with?
 
  I see two issues.  First, my node.db4o has broken 100MiB.  That's not
  a problem, but eventually it would be.  I can deal with this by
  emptying my download / upload queues, deleting it, and re-adding any
  keys, but that's annoying.  It's not urgent, but an option to defrag
  at startup would be nice if it doesn't take too much of your time.
 
  Second issue is a minor security thing.  I'm probably less paranoid
  than most Freenet users, but I would like to know that after I
  download a file, the traces left behind by doing so are well defined.
  That would include the file itself and the fact that its blocks are in
  my cache.  I'd rather not also have that info in the node.db4o file
  (is it encrypted?).  Again, not urgent, but worth dealing with
  eventually.  The truly paranoid will have motion detectors that
  unmount their encrypted filesystems and start scrubbing RAM before the
  Bad Guys (TM) can sit down at the keyboard, right?
 
  Evan Daniel
 
  On Thursday 14 May 2009 01:54:02 Dennis Nezic wrote:
 
  Or have the node automatically delete it when the queues are empty?
 
  Automatically deleting node.db4o when there is nothing queued might work. 
The
  main problem is that we would then not be able to put things other than
  queued requests into it. Meaning if we want to persist e.g. stats, passive
  requests etc, we will need a separate database.
 
 Is that much work?  Have a filequeue.db4o and a nodeinfo.db4o  Then we
 can safely delete the filequeue when there are no pending persistent
 requests?

True, it might be worth it.
 
  We don't encrypt node.db4o at present. We should have the option of 
encrypting
  it for those who don't want to encrypt the whole drive, but then we would
  need a way to ask the user for the password on startup, or put it in some
  easily shreddable file (shredding files doesn't work with modern
  filesystems).
 
  But for the really paranoid, db4o is a bit of a PITA. There is no way we 
can
  guarantee that no traces of old requests are present, because db4o doesn't
  have garbage collection. All we can say is we've tested it and debugged 
the
  leaks found by the tests. But it is certainly possible for bugs introduced
  since then, or not found, to cause leakage of objects.

If we encrypt the persistent client layer, and only start it up with a 
password, then we will still want the rest of the node to work when we don't 
have the password. So probably we should have two databases...


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Juiceman]

On Fri, May 15, 2009 at 2:37 PM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 On Wednesday 13 May 2009 18:29:47 Evan Daniel wrote:
 On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  On Friday 08 May 2009 17:40:58 Juiceman wrote:
   Weird.  node.db4o was an insane 375 MB.  I deleted it and and added a
   bunch of downloads.  Now it is less than 10 MB.  That definitely
   helped some with the disk thrashing.
  
   I think I found the main problem, and I'm embarrassed to say
   apparantly I had xmlspider plugin running and writing GB+ files to the
   same disk the node resides on.  I turned this off and the disk usage
   became manageable.
  
   I also upgraded my HDD from an older 2 MB cache model to one with 16
   MB and now Freenet is zipping along nicely.
  
   I did see some errors in the log so I am sending it to Toad for
 review.
  
   P.S. I would recommend not installing the xmlspider by default on
  installs.
  
   Victor - might this be your issue as well?
  
   ROFL. So that just leaves victor...
 
  Is it normal that node.db4o never shrinks?  I have completed all the
  downloads I had running and removed them from the page, yet node.db4o
  doesn't get smaller.  I have rebooted the node also.  This IMHO is bad
  because it will eventually kill performance with disk access...
 
  Yes, the only way to ensure it shrinks is to defrag it. This is on the
 todo
  list, but it does not seem urgent to me. Is it really a huge, monstrous,
  evil, all-consuming problem more urgent than the 500 other things we have
 to
  deal with?

 I see two issues.  First, my node.db4o has broken 100MiB.  That's not
 a problem, but eventually it would be.  I can deal with this by
 emptying my download / upload queues, deleting it, and re-adding any
 keys, but that's annoying.  It's not urgent, but an option to defrag
 at startup would be nice if it doesn't take too much of your time.

 Second issue is a minor security thing.  I'm probably less paranoid
 than most Freenet users, but I would like to know that after I
 download a file, the traces left behind by doing so are well defined.
 That would include the file itself and the fact that its blocks are in
 my cache.  I'd rather not also have that info in the node.db4o file
 (is it encrypted?).  Again, not urgent, but worth dealing with
 eventually.  The truly paranoid will have motion detectors that
 unmount their encrypted filesystems and start scrubbing RAM before the
 Bad Guys (TM) can sit down at the keyboard, right?

 Evan Daniel

 On Thursday 14 May 2009 01:54:02 Dennis Nezic wrote:

 Or have the node automatically delete it when the queues are empty?

 Automatically deleting node.db4o when there is nothing queued might work. The
 main problem is that we would then not be able to put things other than
 queued requests into it. Meaning if we want to persist e.g. stats, passive
 requests etc, we will need a separate database.

Is that much work?  Have a filequeue.db4o and a nodeinfo.db4o  Then we
can safely delete the filequeue when there are no pending persistent
requests?

 We don't encrypt node.db4o at present. We should have the option of encrypting
 it for those who don't want to encrypt the whole drive, but then we would
 need a way to ask the user for the password on startup, or put it in some
 easily shreddable file (shredding files doesn't work with modern
 filesystems).

 But for the really paranoid, db4o is a bit of a PITA. There is no way we can
 guarantee that no traces of old requests are present, because db4o doesn't
 have garbage collection. All we can say is we've tested it and debugged the
 leaks found by the tests. But it is certainly possible for bugs introduced
 since then, or not found, to cause leakage of objects.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 13 May 2009 18:29:47 Evan Daniel wrote:
> On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
>  wrote:
> > On Friday 08 May 2009 17:40:58 Juiceman wrote:
> >> >> Weird. ?node.db4o was an insane 375 MB. ?I deleted it and and added a
> >> >> bunch of downloads. ?Now it is less than 10 MB. ?That definitely
> >> >> helped some with the disk thrashing.
> >> >>
> >> >> I think I found the main problem, and I'm embarrassed to say
> >> >> apparantly I had xmlspider plugin running and writing GB+ files to the
> >> >> same disk the node resides on. ?I turned this off and the disk usage
> >> >> became manageable.
> >> >>
> >> >> I also upgraded my HDD from an older 2 MB cache model to one with 16
> >> >> MB and now Freenet is zipping along nicely.
> >> >>
> >> >> I did see some errors in the log so I am sending it to Toad for 
review.
> >> >>
> >> >> P.S. I would recommend not installing the xmlspider by default on
> > installs.
> >> >>
> >> >> Victor - might this be your issue as well?
> >> >
> >> > ROFL. So that just leaves victor...
> >>
> >> Is it normal that node.db4o never shrinks? ?I have completed all the
> >> downloads I had running and removed them from the page, yet node.db4o
> >> doesn't get smaller. ?I have rebooted the node also. ?This IMHO is bad
> >> because it will eventually kill performance with disk access...
> >
> > Yes, the only way to ensure it shrinks is to defrag it. This is on the 
todo
> > list, but it does not seem urgent to me. Is it really a huge, monstrous,
> > evil, all-consuming problem more urgent than the 500 other things we have 
to
> > deal with?
> 
> I see two issues.  First, my node.db4o has broken 100MiB.  That's not
> a problem, but eventually it would be.  I can deal with this by
> emptying my download / upload queues, deleting it, and re-adding any
> keys, but that's annoying.  It's not urgent, but an option to defrag
> at startup would be nice if it doesn't take too much of your time.
> 
How much have you had in your queue so far?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 13 May 2009 18:29:47 Evan Daniel wrote:
> On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
>  wrote:
> > On Friday 08 May 2009 17:40:58 Juiceman wrote:
> >> >> Weird. ?node.db4o was an insane 375 MB. ?I deleted it and and added a
> >> >> bunch of downloads. ?Now it is less than 10 MB. ?That definitely
> >> >> helped some with the disk thrashing.
> >> >>
> >> >> I think I found the main problem, and I'm embarrassed to say
> >> >> apparantly I had xmlspider plugin running and writing GB+ files to the
> >> >> same disk the node resides on. ?I turned this off and the disk usage
> >> >> became manageable.
> >> >>
> >> >> I also upgraded my HDD from an older 2 MB cache model to one with 16
> >> >> MB and now Freenet is zipping along nicely.
> >> >>
> >> >> I did see some errors in the log so I am sending it to Toad for 
review.
> >> >>
> >> >> P.S. I would recommend not installing the xmlspider by default on
> > installs.
> >> >>
> >> >> Victor - might this be your issue as well?
> >> >
> >> > ROFL. So that just leaves victor...
> >>
> >> Is it normal that node.db4o never shrinks? ?I have completed all the
> >> downloads I had running and removed them from the page, yet node.db4o
> >> doesn't get smaller. ?I have rebooted the node also. ?This IMHO is bad
> >> because it will eventually kill performance with disk access...
> >
> > Yes, the only way to ensure it shrinks is to defrag it. This is on the 
todo
> > list, but it does not seem urgent to me. Is it really a huge, monstrous,
> > evil, all-consuming problem more urgent than the 500 other things we have 
to
> > deal with?
> 
> I see two issues.  First, my node.db4o has broken 100MiB.  That's not
> a problem, but eventually it would be.  I can deal with this by
> emptying my download / upload queues, deleting it, and re-adding any
> keys, but that's annoying.  It's not urgent, but an option to defrag
> at startup would be nice if it doesn't take too much of your time.
> 
> Second issue is a minor security thing.  I'm probably less paranoid
> than most Freenet users, but I would like to know that after I
> download a file, the traces left behind by doing so are well defined.
> That would include the file itself and the fact that its blocks are in
> my cache.  I'd rather not also have that info in the node.db4o file
> (is it encrypted?).  Again, not urgent, but worth dealing with
> eventually.  The truly paranoid will have motion detectors that
> unmount their encrypted filesystems and start scrubbing RAM before the
> Bad Guys (TM) can sit down at the keyboard, right?
> 
> Evan Daniel

On Thursday 14 May 2009 01:54:02 Dennis Nezic wrote:
> 
> Or have the node automatically delete it when the queues are empty?

Automatically deleting node.db4o when there is nothing queued might work. The 
main problem is that we would then not be able to put things other than 
queued requests into it. Meaning if we want to persist e.g. stats, passive 
requests etc, we will need a separate database.

We don't encrypt node.db4o at present. We should have the option of encrypting 
it for those who don't want to encrypt the whole drive, but then we would 
need a way to ask the user for the password on startup, or put it in some 
easily shreddable file (shredding files doesn't work with modern 
filesystems).

But for the really paranoid, db4o is a bit of a PITA. There is no way we can 
guarantee that no traces of old requests are present, because db4o doesn't 
have garbage collection. All we can say is we've tested it and debugged the 
leaks found by the tests. But it is certainly possible for bugs introduced 
since then, or not found, to cause leakage of objects.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Evan Daniel]

On Fri, May 15, 2009 at 2:38 PM, Matthew Toseland
 wrote:
> On Wednesday 13 May 2009 18:29:47 Evan Daniel wrote:
>> On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
>>  wrote:
>> > On Friday 08 May 2009 17:40:58 Juiceman wrote:
>> >> >> Weird. ?node.db4o was an insane 375 MB. ?I deleted it and and added a
>> >> >> bunch of downloads. ?Now it is less than 10 MB. ?That definitely
>> >> >> helped some with the disk thrashing.
>> >> >>
>> >> >> I think I found the main problem, and I'm embarrassed to say
>> >> >> apparantly I had xmlspider plugin running and writing GB+ files to the
>> >> >> same disk the node resides on. ?I turned this off and the disk usage
>> >> >> became manageable.
>> >> >>
>> >> >> I also upgraded my HDD from an older 2 MB cache model to one with 16
>> >> >> MB and now Freenet is zipping along nicely.
>> >> >>
>> >> >> I did see some errors in the log so I am sending it to Toad for
> review.
>> >> >>
>> >> >> P.S. I would recommend not installing the xmlspider by default on
>> > installs.
>> >> >>
>> >> >> Victor - might this be your issue as well?
>> >> >
>> >> > ROFL. So that just leaves victor...
>> >>
>> >> Is it normal that node.db4o never shrinks? ?I have completed all the
>> >> downloads I had running and removed them from the page, yet node.db4o
>> >> doesn't get smaller. ?I have rebooted the node also. ?This IMHO is bad
>> >> because it will eventually kill performance with disk access...
>> >
>> > Yes, the only way to ensure it shrinks is to defrag it. This is on the
> todo
>> > list, but it does not seem urgent to me. Is it really a huge, monstrous,
>> > evil, all-consuming problem more urgent than the 500 other things we have
> to
>> > deal with?
>>
>> I see two issues. ?First, my node.db4o has broken 100MiB. ?That's not
>> a problem, but eventually it would be. ?I can deal with this by
>> emptying my download / upload queues, deleting it, and re-adding any
>> keys, but that's annoying. ?It's not urgent, but an option to defrag
>> at startup would be nice if it doesn't take too much of your time.
>>
> How much have you had in your queue so far?
>

About 3GiB, maybe a little less.

Evan Daniel

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 13 May 2009 18:29:47 Evan Daniel wrote:
 On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  On Friday 08 May 2009 17:40:58 Juiceman wrote:
   Weird.  node.db4o was an insane 375 MB.  I deleted it and and added a
   bunch of downloads.  Now it is less than 10 MB.  That definitely
   helped some with the disk thrashing.
  
   I think I found the main problem, and I'm embarrassed to say
   apparantly I had xmlspider plugin running and writing GB+ files to the
   same disk the node resides on.  I turned this off and the disk usage
   became manageable.
  
   I also upgraded my HDD from an older 2 MB cache model to one with 16
   MB and now Freenet is zipping along nicely.
  
   I did see some errors in the log so I am sending it to Toad for 
review.
  
   P.S. I would recommend not installing the xmlspider by default on
  installs.
  
   Victor - might this be your issue as well?
  
   ROFL. So that just leaves victor...
 
  Is it normal that node.db4o never shrinks?  I have completed all the
  downloads I had running and removed them from the page, yet node.db4o
  doesn't get smaller.  I have rebooted the node also.  This IMHO is bad
  because it will eventually kill performance with disk access...
 
  Yes, the only way to ensure it shrinks is to defrag it. This is on the 
todo
  list, but it does not seem urgent to me. Is it really a huge, monstrous,
  evil, all-consuming problem more urgent than the 500 other things we have 
to
  deal with?
 
 I see two issues.  First, my node.db4o has broken 100MiB.  That's not
 a problem, but eventually it would be.  I can deal with this by
 emptying my download / upload queues, deleting it, and re-adding any
 keys, but that's annoying.  It's not urgent, but an option to defrag
 at startup would be nice if it doesn't take too much of your time.
 
 Second issue is a minor security thing.  I'm probably less paranoid
 than most Freenet users, but I would like to know that after I
 download a file, the traces left behind by doing so are well defined.
 That would include the file itself and the fact that its blocks are in
 my cache.  I'd rather not also have that info in the node.db4o file
 (is it encrypted?).  Again, not urgent, but worth dealing with
 eventually.  The truly paranoid will have motion detectors that
 unmount their encrypted filesystems and start scrubbing RAM before the
 Bad Guys (TM) can sit down at the keyboard, right?
 
 Evan Daniel

On Thursday 14 May 2009 01:54:02 Dennis Nezic wrote:
 
 Or have the node automatically delete it when the queues are empty?

Automatically deleting node.db4o when there is nothing queued might work. The 
main problem is that we would then not be able to put things other than 
queued requests into it. Meaning if we want to persist e.g. stats, passive 
requests etc, we will need a separate database.

We don't encrypt node.db4o at present. We should have the option of encrypting 
it for those who don't want to encrypt the whole drive, but then we would 
need a way to ask the user for the password on startup, or put it in some 
easily shreddable file (shredding files doesn't work with modern 
filesystems).

But for the really paranoid, db4o is a bit of a PITA. There is no way we can 
guarantee that no traces of old requests are present, because db4o doesn't 
have garbage collection. All we can say is we've tested it and debugged the 
leaks found by the tests. But it is certainly possible for bugs introduced 
since then, or not found, to cause leakage of objects.


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 13 May 2009 18:29:47 Evan Daniel wrote:
 On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  On Friday 08 May 2009 17:40:58 Juiceman wrote:
   Weird.  node.db4o was an insane 375 MB.  I deleted it and and added a
   bunch of downloads.  Now it is less than 10 MB.  That definitely
   helped some with the disk thrashing.
  
   I think I found the main problem, and I'm embarrassed to say
   apparantly I had xmlspider plugin running and writing GB+ files to the
   same disk the node resides on.  I turned this off and the disk usage
   became manageable.
  
   I also upgraded my HDD from an older 2 MB cache model to one with 16
   MB and now Freenet is zipping along nicely.
  
   I did see some errors in the log so I am sending it to Toad for 
review.
  
   P.S. I would recommend not installing the xmlspider by default on
  installs.
  
   Victor - might this be your issue as well?
  
   ROFL. So that just leaves victor...
 
  Is it normal that node.db4o never shrinks?  I have completed all the
  downloads I had running and removed them from the page, yet node.db4o
  doesn't get smaller.  I have rebooted the node also.  This IMHO is bad
  because it will eventually kill performance with disk access...
 
  Yes, the only way to ensure it shrinks is to defrag it. This is on the 
todo
  list, but it does not seem urgent to me. Is it really a huge, monstrous,
  evil, all-consuming problem more urgent than the 500 other things we have 
to
  deal with?
 
 I see two issues.  First, my node.db4o has broken 100MiB.  That's not
 a problem, but eventually it would be.  I can deal with this by
 emptying my download / upload queues, deleting it, and re-adding any
 keys, but that's annoying.  It's not urgent, but an option to defrag
 at startup would be nice if it doesn't take too much of your time.
 
How much have you had in your queue so far?


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Evan Daniel]

On Fri, May 15, 2009 at 2:38 PM, Matthew Toseland
t...@amphibian.dyndns.org wrote:
 On Wednesday 13 May 2009 18:29:47 Evan Daniel wrote:
 On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  On Friday 08 May 2009 17:40:58 Juiceman wrote:
   Weird.  node.db4o was an insane 375 MB.  I deleted it and and added a
   bunch of downloads.  Now it is less than 10 MB.  That definitely
   helped some with the disk thrashing.
  
   I think I found the main problem, and I'm embarrassed to say
   apparantly I had xmlspider plugin running and writing GB+ files to the
   same disk the node resides on.  I turned this off and the disk usage
   became manageable.
  
   I also upgraded my HDD from an older 2 MB cache model to one with 16
   MB and now Freenet is zipping along nicely.
  
   I did see some errors in the log so I am sending it to Toad for
 review.
  
   P.S. I would recommend not installing the xmlspider by default on
  installs.
  
   Victor - might this be your issue as well?
  
   ROFL. So that just leaves victor...
 
  Is it normal that node.db4o never shrinks?  I have completed all the
  downloads I had running and removed them from the page, yet node.db4o
  doesn't get smaller.  I have rebooted the node also.  This IMHO is bad
  because it will eventually kill performance with disk access...
 
  Yes, the only way to ensure it shrinks is to defrag it. This is on the
 todo
  list, but it does not seem urgent to me. Is it really a huge, monstrous,
  evil, all-consuming problem more urgent than the 500 other things we have
 to
  deal with?

 I see two issues.  First, my node.db4o has broken 100MiB.  That's not
 a problem, but eventually it would be.  I can deal with this by
 emptying my download / upload queues, deleting it, and re-adding any
 keys, but that's annoying.  It's not urgent, but an option to defrag
 at startup would be nice if it doesn't take too much of your time.

 How much have you had in your queue so far?


About 3GiB, maybe a little less.

Evan Daniel
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Dennis Nezic]

> On Wed, 13 May 2009 13:29:47 -0400,
> Evan Daniel  wrote:
>
> On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
>  wrote:
> > On Friday 08 May 2009 17:40:58 Juiceman wrote:
> >> >> Weird. ?node.db4o was an insane 375 MB. ?I deleted it and and
> >> >> added a bunch of downloads. ?Now it is less than 10 MB. ?That
> >> >> definitely helped some with the disk thrashing.
> >> >>
> >> >> I think I found the main problem, and I'm embarrassed to say
> >> >> apparantly I had xmlspider plugin running and writing GB+ files
> >> >> to the same disk the node resides on. ?I turned this off and
> >> >> the disk usage became manageable.
> >> >>
> >> >> I also upgraded my HDD from an older 2 MB cache model to one
> >> >> with 16 MB and now Freenet is zipping along nicely.
> >> >>
> >> >> I did see some errors in the log so I am sending it to Toad for
> >> >> review.
> >> >>
> >> >> P.S. I would recommend not installing the xmlspider by default
> >> >> on
> > installs.
> >> >>
> >> >> Victor - might this be your issue as well?
> >> >
> >> > ROFL. So that just leaves victor...
> >>
> >> Is it normal that node.db4o never shrinks? ?I have completed all
> >> the downloads I had running and removed them from the page, yet
> >> node.db4o doesn't get smaller. ?I have rebooted the node also.
> >> ?This IMHO is bad because it will eventually kill performance with
> >> disk access...
> >
> > Yes, the only way to ensure it shrinks is to defrag it. This is on
> > the todo list, but it does not seem urgent to me. Is it really a
> > huge, monstrous, evil, all-consuming problem more urgent than the
> > 500 other things we have to deal with?
> 
> I see two issues.  First, my node.db4o has broken 100MiB.  That's not
> a problem, but eventually it would be.  I can deal with this by
> emptying my download / upload queues, deleting it, and re-adding any
> keys, but that's annoying.  It's not urgent, but an option to defrag
> at startup would be nice if it doesn't take too much of your time.

Or have the node automatically delete it when the queues are empty?

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Evan Daniel]

On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
 wrote:
> On Friday 08 May 2009 17:40:58 Juiceman wrote:
>> >> Weird. ?node.db4o was an insane 375 MB. ?I deleted it and and added a
>> >> bunch of downloads. ?Now it is less than 10 MB. ?That definitely
>> >> helped some with the disk thrashing.
>> >>
>> >> I think I found the main problem, and I'm embarrassed to say
>> >> apparantly I had xmlspider plugin running and writing GB+ files to the
>> >> same disk the node resides on. ?I turned this off and the disk usage
>> >> became manageable.
>> >>
>> >> I also upgraded my HDD from an older 2 MB cache model to one with 16
>> >> MB and now Freenet is zipping along nicely.
>> >>
>> >> I did see some errors in the log so I am sending it to Toad for review.
>> >>
>> >> P.S. I would recommend not installing the xmlspider by default on
> installs.
>> >>
>> >> Victor - might this be your issue as well?
>> >
>> > ROFL. So that just leaves victor...
>>
>> Is it normal that node.db4o never shrinks? ?I have completed all the
>> downloads I had running and removed them from the page, yet node.db4o
>> doesn't get smaller. ?I have rebooted the node also. ?This IMHO is bad
>> because it will eventually kill performance with disk access...
>
> Yes, the only way to ensure it shrinks is to defrag it. This is on the todo
> list, but it does not seem urgent to me. Is it really a huge, monstrous,
> evil, all-consuming problem more urgent than the 500 other things we have to
> deal with?

I see two issues.  First, my node.db4o has broken 100MiB.  That's not
a problem, but eventually it would be.  I can deal with this by
emptying my download / upload queues, deleting it, and re-adding any
keys, but that's annoying.  It's not urgent, but an option to defrag
at startup would be nice if it doesn't take too much of your time.

Second issue is a minor security thing.  I'm probably less paranoid
than most Freenet users, but I would like to know that after I
download a file, the traces left behind by doing so are well defined.
That would include the file itself and the fact that its blocks are in
my cache.  I'd rather not also have that info in the node.db4o file
(is it encrypted?).  Again, not urgent, but worth dealing with
eventually.  The truly paranoid will have motion detectors that
unmount their encrypted filesystems and start scrubbing RAM before the
Bad Guys (TM) can sit down at the keyboard, right?

Evan Daniel

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Friday 08 May 2009 17:40:58 Juiceman wrote:
> >> Weird. ?node.db4o was an insane 375 MB. ?I deleted it and and added a
> >> bunch of downloads. ?Now it is less than 10 MB. ?That definitely
> >> helped some with the disk thrashing.
> >>
> >> I think I found the main problem, and I'm embarrassed to say
> >> apparantly I had xmlspider plugin running and writing GB+ files to the
> >> same disk the node resides on. ?I turned this off and the disk usage
> >> became manageable.
> >>
> >> I also upgraded my HDD from an older 2 MB cache model to one with 16
> >> MB and now Freenet is zipping along nicely.
> >>
> >> I did see some errors in the log so I am sending it to Toad for review.
> >>
> >> P.S. I would recommend not installing the xmlspider by default on 
installs.
> >>
> >> Victor - might this be your issue as well?
> >
> > ROFL. So that just leaves victor...
> 
> Is it normal that node.db4o never shrinks?  I have completed all the
> downloads I had running and removed them from the page, yet node.db4o
> doesn't get smaller.  I have rebooted the node also.  This IMHO is bad
> because it will eventually kill performance with disk access...

Yes, the only way to ensure it shrinks is to defrag it. This is on the todo 
list, but it does not seem urgent to me. Is it really a huge, monstrous, 
evil, all-consuming problem more urgent than the 500 other things we have to 
deal with?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

Re: [freenet-support] Is it my system, or had builds 1208 -1209 have severe performance issues?

[Matthew Toseland]

On Friday 08 May 2009 17:40:58 Juiceman wrote:
  Weird.  node.db4o was an insane 375 MB.  I deleted it and and added a
  bunch of downloads.  Now it is less than 10 MB.  That definitely
  helped some with the disk thrashing.
 
  I think I found the main problem, and I'm embarrassed to say
  apparantly I had xmlspider plugin running and writing GB+ files to the
  same disk the node resides on.  I turned this off and the disk usage
  became manageable.
 
  I also upgraded my HDD from an older 2 MB cache model to one with 16
  MB and now Freenet is zipping along nicely.
 
  I did see some errors in the log so I am sending it to Toad for review.
 
  P.S. I would recommend not installing the xmlspider by default on 
installs.
 
  Victor - might this be your issue as well?
 
  ROFL. So that just leaves victor...
 
 Is it normal that node.db4o never shrinks?  I have completed all the
 downloads I had running and removed them from the page, yet node.db4o
 doesn't get smaller.  I have rebooted the node also.  This IMHO is bad
 because it will eventually kill performance with disk access...

Yes, the only way to ensure it shrinks is to defrag it. This is on the todo 
list, but it does not seem urgent to me. Is it really a huge, monstrous, 
evil, all-consuming problem more urgent than the 500 other things we have to 
deal with?


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Juiceman]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1



On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland  wrote:
 On Friday 08 May 2009 17:40:58 Juiceman wrote:
  Weird.  node.db4o was an insane 375 MB.  I deleted it and and added a
  bunch of downloads.  Now it is less than 10 MB.  That definitely
  helped some with the disk thrashing.
 
  I think I found the main problem, and I'm embarrassed to say
  apparantly I had xmlspider plugin running and writing GB+ files to the
  same disk the node resides on.  I turned this off and the disk usage
  became manageable.
 
  I also upgraded my HDD from an older 2 MB cache model to one with 16
  MB and now Freenet is zipping along nicely.
 
  I did see some errors in the log so I am sending it to Toad for review.
 
  P.S. I would recommend not installing the xmlspider by default on
 installs.
 
  Victor - might this be your issue as well?
 
  ROFL. So that just leaves victor...

 Is it normal that node.db4o never shrinks?  I have completed all the
 downloads I had running and removed them from the page, yet node.db4o
 doesn't get smaller.  I have rebooted the node also.  This IMHO is bad
 because it will eventually kill performance with disk access...

 Yes, the only way to ensure it shrinks is to defrag it. This is on the todo
 list, but it does not seem urgent to me. Is it really a huge, monstrous,
 evil, all-consuming problem more urgent than the 500 other things we have to
 deal with?

 ___
 Support mailing list
 Support@freenetproject.org
 http://news.gmane.org/gmane.network.freenet.support
 Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
 Or mailto:support-requ...@freenetproject.org?subject=unsubscribe


No it's not urgent.


-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Use GnuPG with Firefox : http://getfiregpg.org (Version: 0.7.5)

iEYEARECAAYFAkoK9B4ACgkQ4esu1mlKOs+4oACfSPF+0SKk3u7VuLmUElOBvUpL
BWgAoJw4Xdl54vX+57MVfdCSYE6nKRLY
=bpAj
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Dennis Nezic]

 On Wed, 13 May 2009 13:29:47 -0400,
 Evan Daniel eva...@gmail.com wrote:

 On Wed, May 13, 2009 at 7:33 AM, Matthew Toseland
 t...@amphibian.dyndns.org wrote:
  On Friday 08 May 2009 17:40:58 Juiceman wrote:
   Weird.  node.db4o was an insane 375 MB.  I deleted it and and
   added a bunch of downloads.  Now it is less than 10 MB.  That
   definitely helped some with the disk thrashing.
  
   I think I found the main problem, and I'm embarrassed to say
   apparantly I had xmlspider plugin running and writing GB+ files
   to the same disk the node resides on.  I turned this off and
   the disk usage became manageable.
  
   I also upgraded my HDD from an older 2 MB cache model to one
   with 16 MB and now Freenet is zipping along nicely.
  
   I did see some errors in the log so I am sending it to Toad for
   review.
  
   P.S. I would recommend not installing the xmlspider by default
   on
  installs.
  
   Victor - might this be your issue as well?
  
   ROFL. So that just leaves victor...
 
  Is it normal that node.db4o never shrinks?  I have completed all
  the downloads I had running and removed them from the page, yet
  node.db4o doesn't get smaller.  I have rebooted the node also.
   This IMHO is bad because it will eventually kill performance with
  disk access...
 
  Yes, the only way to ensure it shrinks is to defrag it. This is on
  the todo list, but it does not seem urgent to me. Is it really a
  huge, monstrous, evil, all-consuming problem more urgent than the
  500 other things we have to deal with?
 
 I see two issues.  First, my node.db4o has broken 100MiB.  That's not
 a problem, but eventually it would be.  I can deal with this by
 emptying my download / upload queues, deleting it, and re-adding any
 keys, but that's annoying.  It's not urgent, but an option to defrag
 at startup would be nice if it doesn't take too much of your time.

Or have the node automatically delete it when the queues are empty?
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

>> Victor - might this be your issue as well?
> 
> ROFL. So that just leaves victor...

Sorry, was away on a long weekend :-(. I'll fire up the node first thing
tomorrow with requested logging and will report back.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKCLKx1O5++4rTuI0RAuPCAJ9O7iV6LPZGbNcw0tidlfBrUwkmBQCgkYP4
6dGKs9FBDOjDh3OIVwodZ3k=
=sGqU
-END PGP SIGNATURE-

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

 Victor - might this be your issue as well?
 
 ROFL. So that just leaves victor...

Sorry, was away on a long weekend :-(. I'll fire up the node first thing
tomorrow with requested logging and will report back.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKCLKx1O5++4rTuI0RAuPCAJ9O7iV6LPZGbNcw0tidlfBrUwkmBQCgkYP4
6dGKs9FBDOjDh3OIVwodZ3k=
=sGqU
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Friday 08 May 2009 06:01:06 Juiceman wrote:
> 
> On Wed, May 6, 2009 at 7:12 PM, Matthew Toseland  wrote:
> > On Wednesday 06 May 2009 19:18:13 Victor Denisov wrote:
> >> > Do you have uploads queued as well as downloads? Generally uploads cost 
a
> > bit
> >> > more than downloads do with db4o...
> >>
> >> No, only downloads. Total queued size varied between 25 Mb and 350 Mb in
> >> my tests (but actual total file size was often more than reported by
> >> Freenet, as some keys stayed at 0% for the duration of the test).
> >>
> >> Also, to clarify things, no background applications of notice (such as
> >> other P2P apps or distributed computing clients) were running during the
> >> test. I regularly run Azureus, eMule and I2P, but they all were stopped
> >> for the entire duration Freenet was running, as were MySQL and MS SQL
> >> Server instances I work on. I also tried disabling my antivirus/personal
> >> firewall (Agnitum Outpost Security Suite), but it didn't result in a
> >> noticeable improvement in performance.
> >
> > Okay. And you have plenty of RAM. How big is the node.db4o file? I'm 
assuming
> > it fits very comfortably in RAM, so what we are talking about here are
> > *writes*.
> >
> > Also, the node behaves like this (constant heavy disk i/o making using the
> > system very problematic) for a long time, hours on end? Or just for spurts
> > now and then?
> >
> > Please could you get me some debug information?
> >
> > Set the log level details to 
freenet.support.PrioritizedSerialExecutor:MINOR
> >
> > Let the node run for an hour or so. Send me your statistics page and the
> > contents of your last log (maybe narrow it down by grep'ing for
> > PrioritizedSerialExecutor, hopefully there shouldn't be any keys or 
anything
> > on the output).
> 
> Weird.  node.db4o was an insane 375 MB.  I deleted it and and added a
> bunch of downloads.  Now it is less than 10 MB.  That definitely
> helped some with the disk thrashing.
> 
> I think I found the main problem, and I'm embarrassed to say
> apparantly I had xmlspider plugin running and writing GB+ files to the
> same disk the node resides on.  I turned this off and the disk usage
> became manageable.
> 
> I also upgraded my HDD from an older 2 MB cache model to one with 16
> MB and now Freenet is zipping along nicely.
> 
> I did see some errors in the log so I am sending it to Toad for review.
> 
> P.S. I would recommend not installing the xmlspider by default on installs.
> 
> Victor - might this be your issue as well?

ROFL. So that just leaves victor...
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Juiceman]

>> Weird. ?node.db4o was an insane 375 MB. ?I deleted it and and added a
>> bunch of downloads. ?Now it is less than 10 MB. ?That definitely
>> helped some with the disk thrashing.
>>
>> I think I found the main problem, and I'm embarrassed to say
>> apparantly I had xmlspider plugin running and writing GB+ files to the
>> same disk the node resides on. ?I turned this off and the disk usage
>> became manageable.
>>
>> I also upgraded my HDD from an older 2 MB cache model to one with 16
>> MB and now Freenet is zipping along nicely.
>>
>> I did see some errors in the log so I am sending it to Toad for review.
>>
>> P.S. I would recommend not installing the xmlspider by default on installs.
>>
>> Victor - might this be your issue as well?
>
> ROFL. So that just leaves victor...

Is it normal that node.db4o never shrinks?  I have completed all the
downloads I had running and removed them from the page, yet node.db4o
doesn't get smaller.  I have rebooted the node also.  This IMHO is bad
because it will eventually kill performance with disk access...

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Juiceman]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1



On Wed, May 6, 2009 at 7:12 PM, Matthew Toseland  wrote:
> On Wednesday 06 May 2009 19:18:13 Victor Denisov wrote:
>> > Do you have uploads queued as well as downloads? Generally uploads cost a
> bit
>> > more than downloads do with db4o...
>>
>> No, only downloads. Total queued size varied between 25 Mb and 350 Mb in
>> my tests (but actual total file size was often more than reported by
>> Freenet, as some keys stayed at 0% for the duration of the test).
>>
>> Also, to clarify things, no background applications of notice (such as
>> other P2P apps or distributed computing clients) were running during the
>> test. I regularly run Azureus, eMule and I2P, but they all were stopped
>> for the entire duration Freenet was running, as were MySQL and MS SQL
>> Server instances I work on. I also tried disabling my antivirus/personal
>> firewall (Agnitum Outpost Security Suite), but it didn't result in a
>> noticeable improvement in performance.
>
> Okay. And you have plenty of RAM. How big is the node.db4o file? I'm assuming
> it fits very comfortably in RAM, so what we are talking about here are
> *writes*.
>
> Also, the node behaves like this (constant heavy disk i/o making using the
> system very problematic) for a long time, hours on end? Or just for spurts
> now and then?
>
> Please could you get me some debug information?
>
> Set the log level details to freenet.support.PrioritizedSerialExecutor:MINOR
>
> Let the node run for an hour or so. Send me your statistics page and the
> contents of your last log (maybe narrow it down by grep'ing for
> PrioritizedSerialExecutor, hopefully there shouldn't be any keys or anything
> on the output).

Weird.  node.db4o was an insane 375 MB.  I deleted it and and added a
bunch of downloads.  Now it is less than 10 MB.  That definitely
helped some with the disk thrashing.

I think I found the main problem, and I'm embarrassed to say
apparantly I had xmlspider plugin running and writing GB+ files to the
same disk the node resides on.  I turned this off and the disk usage
became manageable.

I also upgraded my HDD from an older 2 MB cache model to one with 16
MB and now Freenet is zipping along nicely.

I did see some errors in the log so I am sending it to Toad for review.

P.S. I would recommend not installing the xmlspider by default on installs.

Victor - might this be your issue as well?

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Use GnuPG with Firefox : http://getfiregpg.org (Version: 0.7.5)

iEYEARECAAYFAkoDvJMACgkQ4esu1mlKOs9TxQCeJmhz1SP4Qu2Y0RgK3V2Kyl/p
ALoAnikDQ/ki8CAAy0+rYnqKHrIBWiqQ
=9hlX
-END PGP SIGNATURE-

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Friday 08 May 2009 06:01:06 Juiceman wrote:
 
 On Wed, May 6, 2009 at 7:12 PM, Matthew Toseland  wrote:
  On Wednesday 06 May 2009 19:18:13 Victor Denisov wrote:
   Do you have uploads queued as well as downloads? Generally uploads cost 
a
  bit
   more than downloads do with db4o...
 
  No, only downloads. Total queued size varied between 25 Mb and 350 Mb in
  my tests (but actual total file size was often more than reported by
  Freenet, as some keys stayed at 0% for the duration of the test).
 
  Also, to clarify things, no background applications of notice (such as
  other P2P apps or distributed computing clients) were running during the
  test. I regularly run Azureus, eMule and I2P, but they all were stopped
  for the entire duration Freenet was running, as were MySQL and MS SQL
  Server instances I work on. I also tried disabling my antivirus/personal
  firewall (Agnitum Outpost Security Suite), but it didn't result in a
  noticeable improvement in performance.
 
  Okay. And you have plenty of RAM. How big is the node.db4o file? I'm 
assuming
  it fits very comfortably in RAM, so what we are talking about here are
  *writes*.
 
  Also, the node behaves like this (constant heavy disk i/o making using the
  system very problematic) for a long time, hours on end? Or just for spurts
  now and then?
 
  Please could you get me some debug information?
 
  Set the log level details to 
freenet.support.PrioritizedSerialExecutor:MINOR
 
  Let the node run for an hour or so. Send me your statistics page and the
  contents of your last log (maybe narrow it down by grep'ing for
  PrioritizedSerialExecutor, hopefully there shouldn't be any keys or 
anything
  on the output).
 
 Weird.  node.db4o was an insane 375 MB.  I deleted it and and added a
 bunch of downloads.  Now it is less than 10 MB.  That definitely
 helped some with the disk thrashing.
 
 I think I found the main problem, and I'm embarrassed to say
 apparantly I had xmlspider plugin running and writing GB+ files to the
 same disk the node resides on.  I turned this off and the disk usage
 became manageable.
 
 I also upgraded my HDD from an older 2 MB cache model to one with 16
 MB and now Freenet is zipping along nicely.
 
 I did see some errors in the log so I am sending it to Toad for review.
 
 P.S. I would recommend not installing the xmlspider by default on installs.
 
 Victor - might this be your issue as well?

ROFL. So that just leaves victor...


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Juiceman]

 Weird.  node.db4o was an insane 375 MB.  I deleted it and and added a
 bunch of downloads.  Now it is less than 10 MB.  That definitely
 helped some with the disk thrashing.

 I think I found the main problem, and I'm embarrassed to say
 apparantly I had xmlspider plugin running and writing GB+ files to the
 same disk the node resides on.  I turned this off and the disk usage
 became manageable.

 I also upgraded my HDD from an older 2 MB cache model to one with 16
 MB and now Freenet is zipping along nicely.

 I did see some errors in the log so I am sending it to Toad for review.

 P.S. I would recommend not installing the xmlspider by default on installs.

 Victor - might this be your issue as well?

 ROFL. So that just leaves victor...

Is it normal that node.db4o never shrinks?  I have completed all the
downloads I had running and removed them from the page, yet node.db4o
doesn't get smaller.  I have rebooted the node also.  This IMHO is bad
because it will eventually kill performance with disk access...
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 06 May 2009 19:18:13 Victor Denisov wrote:
> > Do you have uploads queued as well as downloads? Generally uploads cost a 
bit
> > more than downloads do with db4o...
> 
> No, only downloads. Total queued size varied between 25 Mb and 350 Mb in
> my tests (but actual total file size was often more than reported by
> Freenet, as some keys stayed at 0% for the duration of the test).
> 
> Also, to clarify things, no background applications of notice (such as
> other P2P apps or distributed computing clients) were running during the
> test. I regularly run Azureus, eMule and I2P, but they all were stopped
> for the entire duration Freenet was running, as were MySQL and MS SQL
> Server instances I work on. I also tried disabling my antivirus/personal
> firewall (Agnitum Outpost Security Suite), but it didn't result in a
> noticeable improvement in performance.

Okay. And you have plenty of RAM. How big is the node.db4o file? I'm assuming 
it fits very comfortably in RAM, so what we are talking about here are 
*writes*.

Also, the node behaves like this (constant heavy disk i/o making using the 
system very problematic) for a long time, hours on end? Or just for spurts 
now and then?

Please could you get me some debug information?

Set the log level details to freenet.support.PrioritizedSerialExecutor:MINOR

Let the node run for an hour or so. Send me your statistics page and the 
contents of your last log (maybe narrow it down by grep'ing for 
PrioritizedSerialExecutor, hopefully there shouldn't be any keys or anything 
on the output).
> 
> Regards,
> Victor Denisov.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 06 May 2009 19:12:17 Victor Denisov wrote:
> >> The problem's that Freenet *doesn't* even use the amount of memory I
> >> provide it with (I'm yet to see it use more than 120 megs out of 320 I
> >> allow for the heap). I'd be willing to dedicate as much memory as
> >> required if only it'd help.
> >
> > Well, a major objective of the db4o rewrite was precisely this. And I 
don't
> > see that this is a problem - the OS will use the rest to cache the 
node.db4o
> > file so that only writes need to go to disk.
> 
> Yes, this I understand. I was one of those complaining of Freenet using
> too much RAM :-(. But, IMO, using as much memory as possible (out of the
> dedicated pool) could be important for performance. For example, by
> increasing buffer sizes in db4o we can possibly make flushes more
> "organized", reducing disk writes substantially. 

No, I don't think we can, db4o is read-committed so avoiding commit() for a 
long period doesn't work afaik.

> I wonder if there are 
> ways to tune db4o performance without rewriting the code, are there any
> handles to turn in the db4o config?

It may be possible to increase the cache size, but this is purely for avoiding 
calling into the OS disk cache.
> 
> > I have seagate 1TB disks mirrored, may explain the difference.
> 
> Unlikely, IMO. 1 Tb drives should have better throughput, but Freenet is
> definitely limited by seek times, which should be only marginally better.
> 
> >> My thinking exactly. Would providing you with a snapshot of CPU/memory
> >> performance under YourKit Profiler (I have academic licenses for both
> >> 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK
> >> distributive) on my machine help? Any logging I can turn on to help?
> >> BTW, I have logging set to ERROR for now, as with NORMAL level it logs
> >> ~2Mb per minute, adding noticeably to overall disk contention.
> >
> > No, because it's an I/O problem, not a CPU/memory problem!
> 
> I was thinking more about allocation/invocation counts, available in
> both profilers. Perhaps some method/query is being called unexpectedly
> often, or a certain object is being persisted too often, etc. Also, it
> could be that a certain method blocks too often and for too much time in
> Windows, leading to poor I/O performance. But it's a long shot, this I
> agree with.
> 
> Regards,
> Victor Denisov.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

> Maybe turning off logging helps? Does Juiceman also have logging
> enabled? This appears to only affect Microsoft users?

On ERROR, Freenet writes about 2-5 Kb per 10 minutes, which is really
nothing. On NORMAL, it writes up to 5 Mb per 10 minutes, or ~ 8 Kb/s
(which probably translates into ~10 to 20 writes per second, considering
internal buffering and OS write-behind caching).

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKAdhA1O5++4rTuI0RAm0tAJ4p5tTvVrYmhgRdPxXsTDM0wElxqgCbBYoD
xGvyyHMG94ZlzrOrNI+fWaU=
=E+sh
-END PGP SIGNATURE-

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

> Do you have uploads queued as well as downloads? Generally uploads cost a bit 
> more than downloads do with db4o...

No, only downloads. Total queued size varied between 25 Mb and 350 Mb in
my tests (but actual total file size was often more than reported by
Freenet, as some keys stayed at 0% for the duration of the test).

Also, to clarify things, no background applications of notice (such as
other P2P apps or distributed computing clients) were running during the
test. I regularly run Azureus, eMule and I2P, but they all were stopped
for the entire duration Freenet was running, as were MySQL and MS SQL
Server instances I work on. I also tried disabling my antivirus/personal
firewall (Agnitum Outpost Security Suite), but it didn't result in a
noticeable improvement in performance.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKAdRl1O5++4rTuI0RAh5KAKDCoFmtLlfKGp0/2GZ4SQv9+/NmSQCfX2YW
oIuQHcSPsl7/smqkL1b9Ykk=
=7RhK
-END PGP SIGNATURE-

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

>> The problem's that Freenet *doesn't* even use the amount of memory I
>> provide it with (I'm yet to see it use more than 120 megs out of 320 I
>> allow for the heap). I'd be willing to dedicate as much memory as
>> required if only it'd help.
> 
> Well, a major objective of the db4o rewrite was precisely this. And I don't 
> see that this is a problem - the OS will use the rest to cache the node.db4o 
> file so that only writes need to go to disk.

Yes, this I understand. I was one of those complaining of Freenet using
too much RAM :-(. But, IMO, using as much memory as possible (out of the
dedicated pool) could be important for performance. For example, by
increasing buffer sizes in db4o we can possibly make flushes more
"organized", reducing disk writes substantially. I wonder if there are
ways to tune db4o performance without rewriting the code, are there any
handles to turn in the db4o config?

> I have seagate 1TB disks mirrored, may explain the difference.

Unlikely, IMO. 1 Tb drives should have better throughput, but Freenet is
definitely limited by seek times, which should be only marginally better.

>> My thinking exactly. Would providing you with a snapshot of CPU/memory
>> performance under YourKit Profiler (I have academic licenses for both
>> 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK
>> distributive) on my machine help? Any logging I can turn on to help?
>> BTW, I have logging set to ERROR for now, as with NORMAL level it logs
>> ~2Mb per minute, adding noticeably to overall disk contention.
> 
> No, because it's an I/O problem, not a CPU/memory problem!

I was thinking more about allocation/invocation counts, available in
both profilers. Perhaps some method/query is being called unexpectedly
often, or a certain object is being persisted too often, etc. Also, it
could be that a certain method blocks too often and for too much time in
Windows, leading to poor I/O performance. But it's a long shot, this I
agree with.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKAdMB1O5++4rTuI0RAi/tAKC8SIIIeUVlQzYtntg22Uxjywp59ACdG5zQ
ICyGAM6ubeh6I8JFIDwkjLs=
=9uoG
-END PGP SIGNATURE-

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Dennis Nezic]

On Wed, 6 May 2009 16:38:10 +0100, Matthew Toseland wrote:
> On Wednesday 06 May 2009 15:26:12 Dennis Nezic wrote:
> > On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote:
> > > Any logging I can turn on to help? BTW, I have logging set to
> > > ERROR for now, as with NORMAL level it logs ~2Mb per minute,
> > > adding noticeably to overall disk contention.
> > 
> > Maybe turning off logging helps? Does Juiceman also have logging
> > enabled? This appears to only affect Microsoft users?
> 
> You have a node on a linux system with queued downloads and no
> performance issues?

Correct.

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Matthew Toseland wrote:
> This is the downside of db4o. If it is a widespread problem, we're gonna have 
> to revert it. Which means throwing away more than 6 months work largely 
> funded by Google's $18K.

I think that using a database is a good idea (although I personally
would've opted for a relational database such as Derby). So I'd prefer
to try and understand and fix the issue rather than hiding from it :-).

> My database queue is usually pretty empty, even with queued downloads, but I 
> have 8G and fast mirrored disks...

The problem's that Freenet *doesn't* even use the amount of memory I
provide it with (I'm yet to see it use more than 120 megs out of 320 I
allow for the heap). I'd be willing to dedicate as much memory as
required if only it'd help.

My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb
cache, SATA2, no NCQ - though definitely not the slowest out there. I
see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files
and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll
probably have to test the same from inside Java to make absolutely sure
that it's not some weird JVM issue on my platform, though.

> 2650 handles is strange, on unix we are generally limited to 1024 and 
> generally we don't exceed that. Both of your problems may be caused by flaky 
> hardware, but frankly we do need to run on flaky real world hardware. :|

I don't have Freenet running right now, will check it later. But I2P is
using 2670 handles right now, and Azureus uses 1450 - so 2600 for
Freenet is definitely nothing out of the ordinary on Windows. Oh, and
the highest handle user on my machine is MySQL, which uses ~69000
handles and works absolutely fine :-).

>> Same here. Enormous disk queues. I've also compared i/o counts with i/o
>> bytes read/written - that's how I know that i/o operations are small. In
>> the statistics screen, I routinely see 100+ outstanding database jobs.
>> It can't be good.
> 
> This just confirms that disk I/O is the problem ... and almost certainly 
> caused by db4o as it goes away if nothing is queued.

My thinking exactly. Would providing you with a snapshot of CPU/memory
performance under YourKit Profiler (I have academic licenses for both
7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK
distributive) on my machine help? Any logging I can turn on to help?
BTW, I have logging set to ERROR for now, as with NORMAL level it logs
~2Mb per minute, adding noticeably to overall disk contention.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKAZQf1O5++4rTuI0RAr5KAKCPuXmaqThbq0g8jVxdGwj7fNGZ/wCgsBBw
YybivVLzs7FlJHvqXbvDJwA=
=SWCw
-END PGP SIGNATURE-

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 06 May 2009 15:26:12 Dennis Nezic wrote:
> On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote:
> > Any logging I can turn on to help? BTW, I have logging set to ERROR
> > for now, as with NORMAL level it logs ~2Mb per minute, adding
> > noticeably to overall disk contention.
> 
> Maybe turning off logging helps? Does Juiceman also have logging
> enabled? This appears to only affect Microsoft users?

You have a node on a linux system with queued downloads and no performance 
issues?
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote:
> Matthew Toseland wrote:
> > This is the downside of db4o. If it is a widespread problem, we're gonna 
have
> > to revert it. Which means throwing away more than 6 months work largely
> > funded by Google's $18K.
> 
> I think that using a database is a good idea (although I personally
> would've opted for a relational database such as Derby). So I'd prefer
> to try and understand and fix the issue rather than hiding from it :-).
> 
> > My database queue is usually pretty empty, even with queued downloads, but 
I
> > have 8G and fast mirrored disks...
> 
> The problem's that Freenet *doesn't* even use the amount of memory I
> provide it with (I'm yet to see it use more than 120 megs out of 320 I
> allow for the heap). I'd be willing to dedicate as much memory as
> required if only it'd help.
> 
> My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb
> cache, SATA2, no NCQ - though definitely not the slowest out there. I
> see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files
> and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll
> probably have to test the same from inside Java to make absolutely sure
> that it's not some weird JVM issue on my platform, though.
> 
> > 2650 handles is strange, on unix we are generally limited to 1024 and
> > generally we don't exceed that. Both of your problems may be caused by 
flaky
> > hardware, but frankly we do need to run on flaky real world hardware. :|
> 
> I don't have Freenet running right now, will check it later. But I2P is
> using 2670 handles right now, and Azureus uses 1450 - so 2600 for
> Freenet is definitely nothing out of the ordinary on Windows. Oh, and
> the highest handle user on my machine is MySQL, which uses ~69000
> handles and works absolutely fine :-).
> 
> >> Same here. Enormous disk queues. I've also compared i/o counts with i/o
> >> bytes read/written - that's how I know that i/o operations are small. In
> >> the statistics screen, I routinely see 100+ outstanding database jobs.
> >> It can't be good.
> >
> > This just confirms that disk I/O is the problem ... and almost certainly
> > caused by db4o as it goes away if nothing is queued.
> 
> My thinking exactly. Would providing you with a snapshot of CPU/memory
> performance under YourKit Profiler (I have academic licenses for both
> 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK
> distributive) on my machine help? Any logging I can turn on to help?
> BTW, I have logging set to ERROR for now, as with NORMAL level it logs
> ~2Mb per minute, adding noticeably to overall disk contention.

Do you have uploads queued as well as downloads? Generally uploads cost a bit 
more than downloads do with db4o...
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote:
> Matthew Toseland wrote:
> > This is the downside of db4o. If it is a widespread problem, we're gonna 
have
> > to revert it. Which means throwing away more than 6 months work largely
> > funded by Google's $18K.
> 
> I think that using a database is a good idea (although I personally
> would've opted for a relational database such as Derby). So I'd prefer
> to try and understand and fix the issue rather than hiding from it :-).
> 
> > My database queue is usually pretty empty, even with queued downloads, but 
I
> > have 8G and fast mirrored disks...
> 
> The problem's that Freenet *doesn't* even use the amount of memory I
> provide it with (I'm yet to see it use more than 120 megs out of 320 I
> allow for the heap). I'd be willing to dedicate as much memory as
> required if only it'd help.

Well, a major objective of the db4o rewrite was precisely this. And I don't 
see that this is a problem - the OS will use the rest to cache the node.db4o 
file so that only writes need to go to disk.
> 
> My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb
> cache, SATA2, no NCQ - though definitely not the slowest out there. I
> see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files
> and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll
> probably have to test the same from inside Java to make absolutely sure
> that it's not some weird JVM issue on my platform, though.

I have seagate 1TB disks mirrored, may explain the difference.
> 
> > 2650 handles is strange, on unix we are generally limited to 1024 and
> > generally we don't exceed that. Both of your problems may be caused by 
flaky
> > hardware, but frankly we do need to run on flaky real world hardware. :|
> 
> I don't have Freenet running right now, will check it later. But I2P is
> using 2670 handles right now, and Azureus uses 1450 - so 2600 for
> Freenet is definitely nothing out of the ordinary on Windows. Oh, and
> the highest handle user on my machine is MySQL, which uses ~69000
> handles and works absolutely fine :-).
> 
> >> Same here. Enormous disk queues. I've also compared i/o counts with i/o
> >> bytes read/written - that's how I know that i/o operations are small. In
> >> the statistics screen, I routinely see 100+ outstanding database jobs.
> >> It can't be good.
> >
> > This just confirms that disk I/O is the problem ... and almost certainly
> > caused by db4o as it goes away if nothing is queued.
> 
> My thinking exactly. Would providing you with a snapshot of CPU/memory
> performance under YourKit Profiler (I have academic licenses for both
> 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK
> distributive) on my machine help? Any logging I can turn on to help?
> BTW, I have logging set to ERROR for now, as with NORMAL level it logs
> ~2Mb per minute, adding noticeably to overall disk contention.

No, because it's an I/O problem, not a CPU/memory problem!
> 
> Regards,
> Victor Denisov.
-- next part --
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: This is a digitally signed message part.
URL: 

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Dennis Nezic]

On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote:
> Any logging I can turn on to help? BTW, I have logging set to ERROR
> for now, as with NORMAL level it logs ~2Mb per minute, adding
> noticeably to overall disk contention.

Maybe turning off logging helps? Does Juiceman also have logging
enabled? This appears to only affect Microsoft users?

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Monday 04 May 2009 21:21:50 Victor Denisov wrote:
 Hello,
 
 Since my node autoupdated to 1208, my system's performance degraded so
 much as to render it completely unusable while Freenet is running. I
 can't perform simplest tasks, such as surfing or typing a document, as
 switching between tabs in Opera can take 10+ seconds and delay between
 typing a letter and it showing up in Word can be several seconds as
 well. Anything more stressing (such as running a game or developing an
 application) is simply not possible.
 
 From what I can see, the reason for this is that Freenet makes hundreds
 of small disk i/o ops per second, basically blocking the OS from
 accessing the hard drive for swapping and such.
 
 The above is definitely affected by the queue size. First, I tried
 adding ~ 100 random files from Thaw when the node first updated, but
 hadn't had the patience to wait for the request to complete (I think I
 waited at least 20 minutes, perhaps more - with the system being nearly
 paralyzed by the constant HDD thrashing). With just 3 or 4 files being
 put in the queue the system starts to stutter noticeably, provided that
 files start downloading and not hang at 0%. The only time when I can run
 Freenet as a real background app is when I don't have any files in the
 queue and FMS isn't running (which seems more or less pointless to me :-().
 
 My system is set up as follows:
 
 AMD Athlon 64 X2 6000+
 4 Gb RAM
 2x250 Gb 7200 rpm SATA2 HDD (mirror via mobo's built-in nVidia 570 RAID
 controller)
 Windows XP x64
 Java 1.6.0_07 64-bit
 Of course, all the latest updates/patches/drivers, etc.
 Freenet uses 5 Gb datastore on an unencrypted partition.
 
 Interesting thing I noticed was that Freenet significantly underutilized
 the memory I provide it with. From 320 Mb heap memory available, I
 hadn't seen it allocate more than ~ 80 Mb.
 
 Any thoughts on why this could be happening? I hadn't seen anyone
 complain about the performance of 1208/09 yet, so it is probably
 something with my machine :-(, but it beats me what it could be :-(.

This is the downside of db4o. If it is a widespread problem, we're gonna have 
to revert it. Which means throwing away more than 6 months work largely 
funded by Google's $18K.

My database queue is usually pretty empty, even with queued downloads, but I 
have 8G and fast mirrored disks...

On Tuesday 05 May 2009 03:22:06 Juiceman wrote:
 
 I see it too.  I have my node installed on it's own separate disk yet
 it stalls my quad-core system.  It's as if the HDD controller chip is
 so swamped with disk ops that the OS and other apps are starved for
 resources.  I'm seeing disk queues exceeding 500 vs the 40 my
 workgroup server hits.
 I also see Freenet using 2650 handles.  That twice the next highest
 app which is my antivirus and 10x the average app.  Heck, even the
 core Windows processes don't use more than 300-600 usually.
 
 Any recommendations for improving performance?  Is BDB datastore any
 better (more stable or less disk IO) than salted-hash?  I could try
 going back to that...

2650 handles is strange, on unix we are generally limited to 1024 and 
generally we don't exceed that. Both of your problems may be caused by flaky 
hardware, but frankly we do need to run on flaky real world hardware. :|

No, changing the datastore won't help.

On Tuesday 05 May 2009 10:30:56 Victor Denisov wrote:
  I see it too.  I have my node installed on it's own separate disk yet
  it stalls my quad-core system.  It's as if the HDD controller chip is
  so swamped with disk ops that the OS and other apps are starved for
  resources.  I'm seeing disk queues exceeding 500 vs the 40 my
  workgroup server hits.
 
 Same here. Enormous disk queues. I've also compared i/o counts with i/o
 bytes read/written - that's how I know that i/o operations are small. In
 the statistics screen, I routinely see 100+ outstanding database jobs.
 It can't be good.

This just confirms that disk I/O is the problem ... and almost certainly 
caused by db4o as it goes away if nothing is queued.

:(

Opinions of other folk? Does this happen to you? Are there lots of complaints 
of similar problems on FMS?

Is the recent increase in the performance of automated tests the result of the 
network shrinking due to lots of people uninstalling?


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Matthew Toseland wrote:
 This is the downside of db4o. If it is a widespread problem, we're gonna have 
 to revert it. Which means throwing away more than 6 months work largely 
 funded by Google's $18K.

I think that using a database is a good idea (although I personally
would've opted for a relational database such as Derby). So I'd prefer
to try and understand and fix the issue rather than hiding from it :-).

 My database queue is usually pretty empty, even with queued downloads, but I 
 have 8G and fast mirrored disks...

The problem's that Freenet *doesn't* even use the amount of memory I
provide it with (I'm yet to see it use more than 120 megs out of 320 I
allow for the heap). I'd be willing to dedicate as much memory as
required if only it'd help.

My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb
cache, SATA2, no NCQ - though definitely not the slowest out there. I
see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files
and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll
probably have to test the same from inside Java to make absolutely sure
that it's not some weird JVM issue on my platform, though.

 2650 handles is strange, on unix we are generally limited to 1024 and 
 generally we don't exceed that. Both of your problems may be caused by flaky 
 hardware, but frankly we do need to run on flaky real world hardware. :|

I don't have Freenet running right now, will check it later. But I2P is
using 2670 handles right now, and Azureus uses 1450 - so 2600 for
Freenet is definitely nothing out of the ordinary on Windows. Oh, and
the highest handle user on my machine is MySQL, which uses ~69000
handles and works absolutely fine :-).

 Same here. Enormous disk queues. I've also compared i/o counts with i/o
 bytes read/written - that's how I know that i/o operations are small. In
 the statistics screen, I routinely see 100+ outstanding database jobs.
 It can't be good.
 
 This just confirms that disk I/O is the problem ... and almost certainly 
 caused by db4o as it goes away if nothing is queued.

My thinking exactly. Would providing you with a snapshot of CPU/memory
performance under YourKit Profiler (I have academic licenses for both
7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK
distributive) on my machine help? Any logging I can turn on to help?
BTW, I have logging set to ERROR for now, as with NORMAL level it logs
~2Mb per minute, adding noticeably to overall disk contention.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKAZQf1O5++4rTuI0RAr5KAKCPuXmaqThbq0g8jVxdGwj7fNGZ/wCgsBBw
YybivVLzs7FlJHvqXbvDJwA=
=SWCw
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote:
 Matthew Toseland wrote:
  This is the downside of db4o. If it is a widespread problem, we're gonna 
have
  to revert it. Which means throwing away more than 6 months work largely
  funded by Google's $18K.
 
 I think that using a database is a good idea (although I personally
 would've opted for a relational database such as Derby). So I'd prefer
 to try and understand and fix the issue rather than hiding from it :-).
 
  My database queue is usually pretty empty, even with queued downloads, but 
I
  have 8G and fast mirrored disks...
 
 The problem's that Freenet *doesn't* even use the amount of memory I
 provide it with (I'm yet to see it use more than 120 megs out of 320 I
 allow for the heap). I'd be willing to dedicate as much memory as
 required if only it'd help.

Well, a major objective of the db4o rewrite was precisely this. And I don't 
see that this is a problem - the OS will use the rest to cache the node.db4o 
file so that only writes need to go to disk.
 
 My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb
 cache, SATA2, no NCQ - though definitely not the slowest out there. I
 see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files
 and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll
 probably have to test the same from inside Java to make absolutely sure
 that it's not some weird JVM issue on my platform, though.

I have seagate 1TB disks mirrored, may explain the difference.
 
  2650 handles is strange, on unix we are generally limited to 1024 and
  generally we don't exceed that. Both of your problems may be caused by 
flaky
  hardware, but frankly we do need to run on flaky real world hardware. :|
 
 I don't have Freenet running right now, will check it later. But I2P is
 using 2670 handles right now, and Azureus uses 1450 - so 2600 for
 Freenet is definitely nothing out of the ordinary on Windows. Oh, and
 the highest handle user on my machine is MySQL, which uses ~69000
 handles and works absolutely fine :-).
 
  Same here. Enormous disk queues. I've also compared i/o counts with i/o
  bytes read/written - that's how I know that i/o operations are small. In
  the statistics screen, I routinely see 100+ outstanding database jobs.
  It can't be good.
 
  This just confirms that disk I/O is the problem ... and almost certainly
  caused by db4o as it goes away if nothing is queued.
 
 My thinking exactly. Would providing you with a snapshot of CPU/memory
 performance under YourKit Profiler (I have academic licenses for both
 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK
 distributive) on my machine help? Any logging I can turn on to help?
 BTW, I have logging set to ERROR for now, as with NORMAL level it logs
 ~2Mb per minute, adding noticeably to overall disk contention.

No, because it's an I/O problem, not a CPU/memory problem!
 
 Regards,
 Victor Denisov.


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 06 May 2009 14:43:59 Victor Denisov wrote:
 Matthew Toseland wrote:
  This is the downside of db4o. If it is a widespread problem, we're gonna 
have
  to revert it. Which means throwing away more than 6 months work largely
  funded by Google's $18K.
 
 I think that using a database is a good idea (although I personally
 would've opted for a relational database such as Derby). So I'd prefer
 to try and understand and fix the issue rather than hiding from it :-).
 
  My database queue is usually pretty empty, even with queued downloads, but 
I
  have 8G and fast mirrored disks...
 
 The problem's that Freenet *doesn't* even use the amount of memory I
 provide it with (I'm yet to see it use more than 120 megs out of 320 I
 allow for the heap). I'd be willing to dedicate as much memory as
 required if only it'd help.
 
 My hard drives are nothing special - 250Gb 7200 RPM Seagate ones, 16 Mb
 cache, SATA2, no NCQ - though definitely not the slowest out there. I
 see ~35 Mb/s read speed and ~28 Mb/s write speed for medium-sized files
 and ~5 Mb/s to 8 Mb/s for small files in the tests I'd done. I'll
 probably have to test the same from inside Java to make absolutely sure
 that it's not some weird JVM issue on my platform, though.
 
  2650 handles is strange, on unix we are generally limited to 1024 and
  generally we don't exceed that. Both of your problems may be caused by 
flaky
  hardware, but frankly we do need to run on flaky real world hardware. :|
 
 I don't have Freenet running right now, will check it later. But I2P is
 using 2670 handles right now, and Azureus uses 1450 - so 2600 for
 Freenet is definitely nothing out of the ordinary on Windows. Oh, and
 the highest handle user on my machine is MySQL, which uses ~69000
 handles and works absolutely fine :-).
 
  Same here. Enormous disk queues. I've also compared i/o counts with i/o
  bytes read/written - that's how I know that i/o operations are small. In
  the statistics screen, I routinely see 100+ outstanding database jobs.
  It can't be good.
 
  This just confirms that disk I/O is the problem ... and almost certainly
  caused by db4o as it goes away if nothing is queued.
 
 My thinking exactly. Would providing you with a snapshot of CPU/memory
 performance under YourKit Profiler (I have academic licenses for both
 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK
 distributive) on my machine help? Any logging I can turn on to help?
 BTW, I have logging set to ERROR for now, as with NORMAL level it logs
 ~2Mb per minute, adding noticeably to overall disk contention.

Do you have uploads queued as well as downloads? Generally uploads cost a bit 
more than downloads do with db4o...


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Dennis Nezic]

On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote:
 Any logging I can turn on to help? BTW, I have logging set to ERROR
 for now, as with NORMAL level it logs ~2Mb per minute, adding
 noticeably to overall disk contention.

Maybe turning off logging helps? Does Juiceman also have logging
enabled? This appears to only affect Microsoft users?
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 06 May 2009 15:26:12 Dennis Nezic wrote:
 On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote:
  Any logging I can turn on to help? BTW, I have logging set to ERROR
  for now, as with NORMAL level it logs ~2Mb per minute, adding
  noticeably to overall disk contention.
 
 Maybe turning off logging helps? Does Juiceman also have logging
 enabled? This appears to only affect Microsoft users?

You have a node on a linux system with queued downloads and no performance 
issues?


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

 The problem's that Freenet *doesn't* even use the amount of memory I
 provide it with (I'm yet to see it use more than 120 megs out of 320 I
 allow for the heap). I'd be willing to dedicate as much memory as
 required if only it'd help.
 
 Well, a major objective of the db4o rewrite was precisely this. And I don't 
 see that this is a problem - the OS will use the rest to cache the node.db4o 
 file so that only writes need to go to disk.

Yes, this I understand. I was one of those complaining of Freenet using
too much RAM :-(. But, IMO, using as much memory as possible (out of the
dedicated pool) could be important for performance. For example, by
increasing buffer sizes in db4o we can possibly make flushes more
organized, reducing disk writes substantially. I wonder if there are
ways to tune db4o performance without rewriting the code, are there any
handles to turn in the db4o config?

 I have seagate 1TB disks mirrored, may explain the difference.

Unlikely, IMO. 1 Tb drives should have better throughput, but Freenet is
definitely limited by seek times, which should be only marginally better.

 My thinking exactly. Would providing you with a snapshot of CPU/memory
 performance under YourKit Profiler (I have academic licenses for both
 7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK
 distributive) on my machine help? Any logging I can turn on to help?
 BTW, I have logging set to ERROR for now, as with NORMAL level it logs
 ~2Mb per minute, adding noticeably to overall disk contention.
 
 No, because it's an I/O problem, not a CPU/memory problem!

I was thinking more about allocation/invocation counts, available in
both profilers. Perhaps some method/query is being called unexpectedly
often, or a certain object is being persisted too often, etc. Also, it
could be that a certain method blocks too often and for too much time in
Windows, leading to poor I/O performance. But it's a long shot, this I
agree with.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKAdMB1O5++4rTuI0RAi/tAKC8SIIIeUVlQzYtntg22Uxjywp59ACdG5zQ
ICyGAM6ubeh6I8JFIDwkjLs=
=9uoG
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

 Do you have uploads queued as well as downloads? Generally uploads cost a bit 
 more than downloads do with db4o...

No, only downloads. Total queued size varied between 25 Mb and 350 Mb in
my tests (but actual total file size was often more than reported by
Freenet, as some keys stayed at 0% for the duration of the test).

Also, to clarify things, no background applications of notice (such as
other P2P apps or distributed computing clients) were running during the
test. I regularly run Azureus, eMule and I2P, but they all were stopped
for the entire duration Freenet was running, as were MySQL and MS SQL
Server instances I work on. I also tried disabling my antivirus/personal
firewall (Agnitum Outpost Security Suite), but it didn't result in a
noticeable improvement in performance.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKAdRl1O5++4rTuI0RAh5KAKDCoFmtLlfKGp0/2GZ4SQv9+/NmSQCfX2YW
oIuQHcSPsl7/smqkL1b9Ykk=
=7RhK
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

 Maybe turning off logging helps? Does Juiceman also have logging
 enabled? This appears to only affect Microsoft users?

On ERROR, Freenet writes about 2-5 Kb per 10 minutes, which is really
nothing. On NORMAL, it writes up to 5 Mb per 10 minutes, or ~ 8 Kb/s
(which probably translates into ~10 to 20 writes per second, considering
internal buffering and OS write-behind caching).

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKAdhA1O5++4rTuI0RAm0tAJ4p5tTvVrYmhgRdPxXsTDM0wElxqgCbBYoD
xGvyyHMG94ZlzrOrNI+fWaU=
=E+sh
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Dennis Nezic]

On Wed, 6 May 2009 16:38:10 +0100, Matthew Toseland wrote:
 On Wednesday 06 May 2009 15:26:12 Dennis Nezic wrote:
  On Wed, 06 May 2009 17:43:59 +0400, Victor Denisov wrote:
   Any logging I can turn on to help? BTW, I have logging set to
   ERROR for now, as with NORMAL level it logs ~2Mb per minute,
   adding noticeably to overall disk contention.
  
  Maybe turning off logging helps? Does Juiceman also have logging
  enabled? This appears to only affect Microsoft users?
 
 You have a node on a linux system with queued downloads and no
 performance issues?

Correct.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 06 May 2009 19:12:17 Victor Denisov wrote:
  The problem's that Freenet *doesn't* even use the amount of memory I
  provide it with (I'm yet to see it use more than 120 megs out of 320 I
  allow for the heap). I'd be willing to dedicate as much memory as
  required if only it'd help.
 
  Well, a major objective of the db4o rewrite was precisely this. And I 
don't
  see that this is a problem - the OS will use the rest to cache the 
node.db4o
  file so that only writes need to go to disk.
 
 Yes, this I understand. I was one of those complaining of Freenet using
 too much RAM :-(. But, IMO, using as much memory as possible (out of the
 dedicated pool) could be important for performance. For example, by
 increasing buffer sizes in db4o we can possibly make flushes more
 organized, reducing disk writes substantially. 

No, I don't think we can, db4o is read-committed so avoiding commit() for a 
long period doesn't work afaik.

 I wonder if there are 
 ways to tune db4o performance without rewriting the code, are there any
 handles to turn in the db4o config?

It may be possible to increase the cache size, but this is purely for avoiding 
calling into the OS disk cache.
 
  I have seagate 1TB disks mirrored, may explain the difference.
 
 Unlikely, IMO. 1 Tb drives should have better throughput, but Freenet is
 definitely limited by seek times, which should be only marginally better.
 
  My thinking exactly. Would providing you with a snapshot of CPU/memory
  performance under YourKit Profiler (I have academic licenses for both
  7.5 and 8.0, IIRC) or VisualVM (which is now a part of the JDK
  distributive) on my machine help? Any logging I can turn on to help?
  BTW, I have logging set to ERROR for now, as with NORMAL level it logs
  ~2Mb per minute, adding noticeably to overall disk contention.
 
  No, because it's an I/O problem, not a CPU/memory problem!
 
 I was thinking more about allocation/invocation counts, available in
 both profilers. Perhaps some method/query is being called unexpectedly
 often, or a certain object is being persisted too often, etc. Also, it
 could be that a certain method blocks too often and for too much time in
 Windows, leading to poor I/O performance. But it's a long shot, this I
 agree with.
 
 Regards,
 Victor Denisov.


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Matthew Toseland]

On Wednesday 06 May 2009 19:18:13 Victor Denisov wrote:
  Do you have uploads queued as well as downloads? Generally uploads cost a 
bit
  more than downloads do with db4o...
 
 No, only downloads. Total queued size varied between 25 Mb and 350 Mb in
 my tests (but actual total file size was often more than reported by
 Freenet, as some keys stayed at 0% for the duration of the test).
 
 Also, to clarify things, no background applications of notice (such as
 other P2P apps or distributed computing clients) were running during the
 test. I regularly run Azureus, eMule and I2P, but they all were stopped
 for the entire duration Freenet was running, as were MySQL and MS SQL
 Server instances I work on. I also tried disabling my antivirus/personal
 firewall (Agnitum Outpost Security Suite), but it didn't result in a
 noticeable improvement in performance.

Okay. And you have plenty of RAM. How big is the node.db4o file? I'm assuming 
it fits very comfortably in RAM, so what we are talking about here are 
*writes*.

Also, the node behaves like this (constant heavy disk i/o making using the 
system very problematic) for a long time, hours on end? Or just for spurts 
now and then?

Please could you get me some debug information?

Set the log level details to freenet.support.PrioritizedSerialExecutor:MINOR

Let the node run for an hour or so. Send me your statistics page and the 
contents of your last log (maybe narrow it down by grep'ing for 
PrioritizedSerialExecutor, hopefully there shouldn't be any keys or anything 
on the output).
 
 Regards,
 Victor Denisov.


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

> I see it too.  I have my node installed on it's own separate disk yet
> it stalls my quad-core system.  It's as if the HDD controller chip is
> so swamped with disk ops that the OS and other apps are starved for
> resources.  I'm seeing disk queues exceeding 500 vs the 40 my
> workgroup server hits.

Same here. Enormous disk queues. I've also compared i/o counts with i/o
bytes read/written - that's how I know that i/o operations are small. In
the statistics screen, I routinely see 100+ outstanding database jobs.
It can't be good.

> I also see Freenet using 2650 handles.  That twice the next highest
> app which is my antivirus and 10x the average app.  Heck, even the
> core Windows processes don't use more than 300-600 usually.

I don't think 2500 handles is something unusual. I'm too lazy to check
what Freenet is holding, but my guess would be that most of them are
file handles for various files opened by both Freenet and JVM itself.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKAAdQ1O5++4rTuI0RAkaIAKC1tbckZbjaXuSzsAXrQuKiRaDWngCg3Igh
dlHcS68QC/kx2AGDY5L4q/0=
=cpb4
-END PGP SIGNATURE-

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hello,

Since my node autoupdated to 1208, my system's performance degraded so
much as to render it completely unusable while Freenet is running. I
can't perform simplest tasks, such as surfing or typing a document, as
switching between tabs in Opera can take 10+ seconds and delay between
typing a letter and it showing up in Word can be several seconds as
well. Anything more stressing (such as running a game or developing an
application) is simply not possible.

- From what I can see, the reason for this is that Freenet makes hundreds
of small disk i/o ops per second, basically blocking the OS from
accessing the hard drive for swapping and such.

The above is definitely affected by the queue size. First, I tried
adding ~ 100 random files from Thaw when the node first updated, but
hadn't had the patience to wait for the request to complete (I think I
waited at least 20 minutes, perhaps more - with the system being nearly
paralyzed by the constant HDD thrashing). With just 3 or 4 files being
put in the queue the system starts to stutter noticeably, provided that
files start downloading and not hang at 0%. The only time when I can run
Freenet as a real background app is when I don't have any files in the
queue and FMS isn't running (which seems more or less pointless to me :-().

My system is set up as follows:

AMD Athlon 64 X2 6000+
4 Gb RAM
2x250 Gb 7200 rpm SATA2 HDD (mirror via mobo's built-in nVidia 570 RAID
controller)
Windows XP x64
Java 1.6.0_07 64-bit
Of course, all the latest updates/patches/drivers, etc.
Freenet uses 5 Gb datastore on an unencrypted partition.

Interesting thing I noticed was that Freenet significantly underutilized
the memory I provide it with. From 320 Mb heap memory available, I
hadn't seen it allocate more than ~ 80 Mb.

Any thoughts on why this could be happening? I hadn't seen anyone
complain about the performance of 1208/09 yet, so it is probably
something with my machine :-(, but it beats me what it could be :-(.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJ/05e1O5++4rTuI0RAmPbAJ9AMm6/Jz9j9r9RFNtmuI2V1CV4JgCghLy/
CNBUL964LvIr+khl9fPvqwI=
=+5BE
-END PGP SIGNATURE-

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

 I see it too.  I have my node installed on it's own separate disk yet
 it stalls my quad-core system.  It's as if the HDD controller chip is
 so swamped with disk ops that the OS and other apps are starved for
 resources.  I'm seeing disk queues exceeding 500 vs the 40 my
 workgroup server hits.

Same here. Enormous disk queues. I've also compared i/o counts with i/o
bytes read/written - that's how I know that i/o operations are small. In
the statistics screen, I routinely see 100+ outstanding database jobs.
It can't be good.

 I also see Freenet using 2650 handles.  That twice the next highest
 app which is my antivirus and 10x the average app.  Heck, even the
 core Windows processes don't use more than 300-600 usually.

I don't think 2500 handles is something unusual. I'm too lazy to check
what Freenet is holding, but my guess would be that most of them are
file handles for various files opened by both Freenet and JVM itself.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFKAAdQ1O5++4rTuI0RAkaIAKC1tbckZbjaXuSzsAXrQuKiRaDWngCg3Igh
dlHcS68QC/kx2AGDY5L4q/0=
=cpb4
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Victor Denisov]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hello,

Since my node autoupdated to 1208, my system's performance degraded so
much as to render it completely unusable while Freenet is running. I
can't perform simplest tasks, such as surfing or typing a document, as
switching between tabs in Opera can take 10+ seconds and delay between
typing a letter and it showing up in Word can be several seconds as
well. Anything more stressing (such as running a game or developing an
application) is simply not possible.

- From what I can see, the reason for this is that Freenet makes hundreds
of small disk i/o ops per second, basically blocking the OS from
accessing the hard drive for swapping and such.

The above is definitely affected by the queue size. First, I tried
adding ~ 100 random files from Thaw when the node first updated, but
hadn't had the patience to wait for the request to complete (I think I
waited at least 20 minutes, perhaps more - with the system being nearly
paralyzed by the constant HDD thrashing). With just 3 or 4 files being
put in the queue the system starts to stutter noticeably, provided that
files start downloading and not hang at 0%. The only time when I can run
Freenet as a real background app is when I don't have any files in the
queue and FMS isn't running (which seems more or less pointless to me :-().

My system is set up as follows:

AMD Athlon 64 X2 6000+
4 Gb RAM
2x250 Gb 7200 rpm SATA2 HDD (mirror via mobo's built-in nVidia 570 RAID
controller)
Windows XP x64
Java 1.6.0_07 64-bit
Of course, all the latest updates/patches/drivers, etc.
Freenet uses 5 Gb datastore on an unencrypted partition.

Interesting thing I noticed was that Freenet significantly underutilized
the memory I provide it with. From 320 Mb heap memory available, I
hadn't seen it allocate more than ~ 80 Mb.

Any thoughts on why this could be happening? I hadn't seen anyone
complain about the performance of 1208/09 yet, so it is probably
something with my machine :-(, but it beats me what it could be :-(.

Regards,
Victor Denisov.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.5 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJ/05e1O5++4rTuI0RAmPbAJ9AMm6/Jz9j9r9RFNtmuI2V1CV4JgCghLy/
CNBUL964LvIr+khl9fPvqwI=
=+5BE
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] Is it my system, or had builds 1208-1209 have severe performance issues?

[Juiceman]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1



On Mon, May 4, 2009 at 4:21 PM, Victor Denisov  wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Hello,

 Since my node autoupdated to 1208, my system's performance degraded so
 much as to render it completely unusable while Freenet is running. I
 can't perform simplest tasks, such as surfing or typing a document, as
 switching between tabs in Opera can take 10+ seconds and delay between
 typing a letter and it showing up in Word can be several seconds as
 well. Anything more stressing (such as running a game or developing an
 application) is simply not possible.

 - From what I can see, the reason for this is that Freenet makes hundreds
 of small disk i/o ops per second, basically blocking the OS from
 accessing the hard drive for swapping and such.

 The above is definitely affected by the queue size. First, I tried
 adding ~ 100 random files from Thaw when the node first updated, but
 hadn't had the patience to wait for the request to complete (I think I
 waited at least 20 minutes, perhaps more - with the system being nearly
 paralyzed by the constant HDD thrashing). With just 3 or 4 files being
 put in the queue the system starts to stutter noticeably, provided that
 files start downloading and not hang at 0%. The only time when I can run
 Freenet as a real background app is when I don't have any files in the
 queue and FMS isn't running (which seems more or less pointless to me :-().

 My system is set up as follows:

 AMD Athlon 64 X2 6000+
 4 Gb RAM
 2x250 Gb 7200 rpm SATA2 HDD (mirror via mobo's built-in nVidia 570 RAID
 controller)
 Windows XP x64
 Java 1.6.0_07 64-bit
 Of course, all the latest updates/patches/drivers, etc.
 Freenet uses 5 Gb datastore on an unencrypted partition.

 Interesting thing I noticed was that Freenet significantly underutilized
 the memory I provide it with. From 320 Mb heap memory available, I
 hadn't seen it allocate more than ~ 80 Mb.

 Any thoughts on why this could be happening? I hadn't seen anyone
 complain about the performance of 1208/09 yet, so it is probably
 something with my machine :-(, but it beats me what it could be :-(.

 Regards,
 Victor Denisov.
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1.4.5 (MingW32)
 Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

 iD8DBQFJ/05e1O5++4rTuI0RAmPbAJ9AMm6/Jz9j9r9RFNtmuI2V1CV4JgCghLy/
 CNBUL964LvIr+khl9fPvqwI=
 =+5BE
 -END PGP SIGNATURE-
 ___
 Support mailing list
 Support@freenetproject.org
 http://news.gmane.org/gmane.network.freenet.support
 Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
 Or mailto:support-requ...@freenetproject.org?subject=unsubscribe


I see it too.  I have my node installed on it's own separate disk yet
it stalls my quad-core system.  It's as if the HDD controller chip is
so swamped with disk ops that the OS and other apps are starved for
resources.  I'm seeing disk queues exceeding 500 vs the 40 my
workgroup server hits.
I also see Freenet using 2650 handles.  That twice the next highest
app which is my antivirus and 10x the average app.  Heck, even the
core Windows processes don't use more than 300-600 usually.

Any recommendations for improving performance?  Is BDB datastore any
better (more stable or less disk IO) than salted-hash?  I could try
going back to that...

- --
I may disagree with what you have to say, but I shall defend, to the
death, your right to say it. - Voltaire
Those who would give up Liberty, to purchase temporary Safety, deserve
neither Liberty nor Safety. - Ben Franklin


-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Use GnuPG with Firefox : http://getfiregpg.org (Version: 0.7.5)

iEYEARECAAYFAkn/ossACgkQ4esu1mlKOs9OXwCgjCi0mhIFVesE+9RZkBh2pmrd
XuIAn2FfFyf22zKyJjr+86qltyZRX42/
=iNxc
-END PGP SIGNATURE-
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe