On Tuesday 09 March 2010 13:20:08 Daniel Stork wrote:
>
> Hi, I'm having problems with files stuck at 100% and having to wait hours and
> days for them to decode.
> It seems that despite having all the parts freenet just sits on it for some
> reason. Is there some reason for this?
I have created this bug on the bug tracker for your problem. *Please* could you
register a user account and Monitor Issue on this bug:
https://bugs.freenetproject.org/view.php?id=4031
Yes. Most likely it is not in fact stuck at 100% of the file but at 100% of the
top layer of the file. Big files get divided into multiple layers, with each
layer telling us how to fetch the next. The last layer contains the actual data
and the last but one layer contains all CHKs for the actual data. For a big
file, the last layer is HUGE, and so are the data structures corresponding to
it. Because we use a database (which allows us to fetch big files on small
amounts of memory), and because our data structures are relatively inefficient,
processing the last but one layer for a big file can take a *long* time
involving a vast amount of disk access. If this is the problem, the best way to
solve it is to change the data structures; we will do so when we make the next
set of major changes (probably combined with LDPC codes and using the same
decryption key for all blocks of a splitfile). Another possible answer is that
it is decompressing, but I wouldn't expect disruptive levels of disk I/O if
this is the problem.
One way to find out is to get a thread dump from the Statistics page while the
node is busy doing these things. This is written to wrapper.log, so please get
one while it is hammering away at 100%, and then send us it.
> Would it be possible that you guys fix this? cuz it would be great not to
> have to wait days to decode a file that's already here.
As a great many national governments have discovered, anything involving
databases is hard. :|
It will get fixed, but not immediately. However, that thread dump would be
helpful.
>
> The node.db4o file also becomes enourmous for some reason (over half Gb)
> after a few days.
You can defrag it, there are two config options related to this, one is
something like "defrag on restart" and the other is whether to reset the first
option after defragging. Then restart the node. This should dramatically reduce
the size of node.db4o, for a short while. But make sure it doesn't run out of
disk space during the defrag.
> It seems to be that this is what causes the files to take so long to decode,
> and it is also making the a high-end computer totally unresponsive.
Disks suck. Especially domestic grade disks, but all disks suck. And
unfortunately handling large amounts of data with limited RAM involves disks
(most cheap SSDs suck too, although there are some very good recent ones). But
I agree it could be *vastly* more efficient. It's simply a question of priority
- for how many users is this a serious problem, more serious than e.g. the fact
that, for example, the file never completes in the first place?
>
> In the meantime, until this hopefully gets fixed, could you tell me if
> there's a way to change the location of the persistent temp folder,
> I would like to put the node.db4o on a ram disk, but the persistent temp is
> too large for this, so I would have to separate the two,
> and could not find settings in freenet.ini,
There is a config option for moving the persistent temp dir in advanced mode.
I'm not sure if it's settable on the fly. You can however shut the node down
and put:
node.persistentTempDir=
>
> Thanks a lot, and thanks for all your work on freenet, it's awsome.
Thanks!
signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
