Re: [freenet-support] changing permanent temp folder

2010-04-02 Thread Matthew Toseland
On Tuesday 09 March 2010 13:20:08 Daniel Stork wrote:
 
 Hi, I'm having problems with files stuck at 100% and having to wait hours and 
 days for them to decode.
 It seems that despite having all the parts freenet just sits on it for some 
 reason. Is there some reason for this?

I have created this bug on the bug tracker for your problem. *Please* could you 
register a user account and Monitor Issue on this bug:
https://bugs.freenetproject.org/view.php?id=4031

Yes. Most likely it is not in fact stuck at 100% of the file but at 100% of the 
top layer of the file. Big files get divided into multiple layers, with each 
layer telling us how to fetch the next. The last layer contains the actual data 
and the last but one layer contains all CHKs for the actual data. For a big 
file, the last layer is HUGE, and so are the data structures corresponding to 
it. Because we use a database (which allows us to fetch big files on small 
amounts of memory), and because our data structures are relatively inefficient, 
processing the last but one layer for a big file can take a *long* time 
involving a vast amount of disk access. If this is the problem, the best way to 
solve it is to change the data structures; we will do so when we make the next 
set of major changes (probably combined with LDPC codes and using the same 
decryption key for all blocks of a splitfile). Another possible answer is that 
it is decompressing, but I wouldn't expect disruptive levels of disk I/O if 
this is the problem.

One way to find out is to get a thread dump from the Statistics page while the 
node is busy doing these things. This is written to wrapper.log, so please get 
one while it is hammering away at 100%, and then send us it.

 Would it be possible that you guys fix this? cuz it would be great not to 
 have to wait days to decode a file that's already here.

As a great many national governments have discovered, anything involving 
databases is hard. :|

It will get fixed, but not immediately. However, that thread dump would be 
helpful.
 
 The node.db4o file also becomes enourmous for some reason (over half Gb) 
 after a few days.

You can defrag it, there are two config options related to this, one is 
something like defrag on restart and the other is whether to reset the first 
option after defragging. Then restart the node. This should dramatically reduce 
the size of node.db4o, for a short while. But make sure it doesn't run out of 
disk space during the defrag.

 It seems to be that this is what causes the files to take so long to decode, 
 and it is also making the a high-end computer totally unresponsive.

Disks suck. Especially domestic grade disks, but all disks suck. And 
unfortunately handling large amounts of data with limited RAM involves disks 
(most cheap SSDs suck too, although there are some very good recent ones). But 
I agree it could be *vastly* more efficient. It's simply a question of priority 
- for how many users is this a serious problem, more serious than e.g. the fact 
that, for example, the file never completes in the first place?
 
 In the meantime, until this hopefully gets fixed, could you tell me if 
 there's a way to change the location of the persistent temp folder,
 I would like to put the node.db4o on a ram disk, but the persistent temp is 
 too large for this, so I would have to separate the two,
 and could not find settings in freenet.ini,

There is a config option for moving the persistent temp dir in advanced mode. 
I'm not sure if it's settable on the fly. You can however shut the node down 
and put:
node.persistentTempDir=somewhere
 
 Thanks a lot, and thanks for all your work on freenet, it's awsome.

Thanks!


signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] changing permanent temp folder

2010-03-09 Thread Evan Daniel
On Tue, Mar 9, 2010 at 8:20 AM, Daniel Stork stork...@yahoo.com wrote:


 Hi, I'm having problems with files stuck at 100% and having to wait hours
 and days for them to decode.
 It seems that despite having all the parts freenet just sits on it for some
 reason. Is there some reason for this?
 Would it be possible that you guys fix this? cuz it would be great not to
 have to wait days to decode a file that's already here.

 The node.db4o file also becomes enourmous for some reason (over half Gb)
 after a few days.
 It seems to be that this is what causes the files to take so long to decode,
 and it is also making the a high-end computer totally unresponsive.

 In the meantime, until this hopefully gets fixed, could you tell me if
 there's a way to change the location of the persistent temp folder,
 I would like to put the node.db4o on a ram disk, but the persistent temp is
 too large for this, so I would have to separate the two,
 and could not find settings in freenet.ini,

 Thanks a lot, and thanks for all your work on freenet, it's awsome.

Hopefully this will get fixed at some point.

Could you please give us a copy of your stats page, in advanced mode?

Short term, you can remove some or all of the keys from your download
queue, and then put them back slowly, making sure the queue never gets
overly large.  (That is, save a list of things to download somewhere,
download a few, put in new ones as old ones finish.)

You could move the node.db4o elsewhere, and then add a symlink to the
new location.

Evan Daniel
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe