Re: [freenet-support] Datastore resize
Matthew Toseland: On Friday 05 Oct 2012 15:00:13 Jep wrote: Matthew Toseland: On Thursday 04 Oct 2012 21:12:36 Jep wrote: Datastore(PUBKEY-cache) maintenance in progress: xxx/xxx. We have changed the datastore so that it will cause less load on your computer (particularly disk accesses), but we have to rebuild the indexes first.. This is coming up every time I have to reboot. Who or what is 'we', is this planned, and if so, why haven't I anywhere seen it announced that this is supposed to happen? Because it was implemented months ago? And is it intended we do this each and every bootup? Also months ago I haven't seen anything on it. No, clearly not. It's supposed to switch to the slot filters and then live happily ever after. Hmmm ... Apparently we use the same message when migrating to slot filters from bloom filters as we do when rebuilding because of an unclean shutdown. I have fixed this in the source, it will be included in the next build; we will use a different message if the .slotfilter file doesn't exist. In your datastore/ folder you should have the following .slotfilter files, as well as other stuff: CHK-cache.slotfilter CHK-clientcache.slotfilter CHK-store.slotfilter PUBKEY-cache.slotfilter PUBKEY-clientcache.slotfilter PUBKEY-store.slotfilter SSK-cache.slotfilter SSK-clientcache.slotfilter SSK-store.slotfilter Do all these files exist? If so then it probably isn't migrating - it's probably rebuilding because of an unclean shutdown. In other words, Freenet has been killed by the wrapper because of taking too long to shut down, or you had a power cut, or pushed the power button or something. wrapper.log may reveal whether Freenet is regularly having problems shutting down? Those files are present allright, do have system freezes so that's the problem. Thanks for the response! ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Datastore resize
On Friday 05 Oct 2012 15:00:13 Jep wrote: Matthew Toseland: On Thursday 04 Oct 2012 21:12:36 Jep wrote: Datastore(PUBKEY-cache) maintenance in progress: xxx/xxx. We have changed the datastore so that it will cause less load on your computer (particularly disk accesses), but we have to rebuild the indexes first.. This is coming up every time I have to reboot. Who or what is 'we', is this planned, and if so, why haven't I anywhere seen it announced that this is supposed to happen? Because it was implemented months ago? And is it intended we do this each and every bootup? Also months ago I haven't seen anything on it. No, clearly not. It's supposed to switch to the slot filters and then live happily ever after. Hmmm ... Apparently we use the same message when migrating to slot filters from bloom filters as we do when rebuilding because of an unclean shutdown. I have fixed this in the source, it will be included in the next build; we will use a different message if the .slotfilter file doesn't exist. In your datastore/ folder you should have the following .slotfilter files, as well as other stuff: CHK-cache.slotfilter CHK-clientcache.slotfilter CHK-store.slotfilter PUBKEY-cache.slotfilter PUBKEY-clientcache.slotfilter PUBKEY-store.slotfilter SSK-cache.slotfilter SSK-clientcache.slotfilter SSK-store.slotfilter Do all these files exist? If so then it probably isn't migrating - it's probably rebuilding because of an unclean shutdown. In other words, Freenet has been killed by the wrapper because of taking too long to shut down, or you had a power cut, or pushed the power button or something. wrapper.log may reveal whether Freenet is regularly having problems shutting down? signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Datastore resize
On Friday 28 Sep 2012 21:54:45 Pascal wrote: With the 1st stage of the rebuild taking 3 weeks, and 1416 becoming mandatory in 2 weeks, if I had just waited my node would be unusable for a week. I noticed an option to do the resize at startup instead that was supposed to be faster, so I set that and upgraded. I let it run for a bit and then did the math, it was still going to take 3 weeks. CPU and HDD were both practically idle. I don't know what a resize does, but it sure seems like there is a bug in there somewhere. It sleeps 100ms after processing a batch of 128 blocks, to try to avoid severe performance effects on the rest of the system. Resize on startup turns this off, so *should* be slightly faster. I have no idea why it was so slow for you but I imagine it has to do with hard disk seeks being slow - the number of bytes transferred could well be relatively low even though the hard disk is seeking constantly. -Pascal On 9/28/2012 4:10 AM, Matthew Toseland wrote: On Tuesday 25 Sep 2012 02:38:07 Pascal wrote: Changed my datastore size from 750g to 850g just over 5 hours ago. It now shows Datastore(CHK-store) resize in progress: 108040/10717960. At this rate it will take about 3 weeks just to finish CHK-store. Does it them take another 3 weeks to do CHK-cache? (They were the same size beforehand). Datastore resizing is slow. Especially if it's already mostly full. :( ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Datastore resize
On Tuesday 25 Sep 2012 02:38:07 Pascal wrote: Changed my datastore size from 750g to 850g just over 5 hours ago. It now shows Datastore(CHK-store) resize in progress: 108040/10717960. At this rate it will take about 3 weeks just to finish CHK-store. Does it them take another 3 weeks to do CHK-cache? (They were the same size beforehand). Datastore resizing is slow. Especially if it's already mostly full. :( signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Datastore resize
On Fri, 28 Sep 2012 10:10:11 +0100, Matthew Toseland wrote: On Tuesday 25 Sep 2012 02:38:07 Pascal wrote: Changed my datastore size from 750g to 850g just over 5 hours ago. It now shows Datastore(CHK-store) resize in progress: 108040/10717960. At this rate it will take about 3 weeks just to finish CHK-store. Does it them take another 3 weeks to do CHK-cache? (They were the same size beforehand). Datastore resizing is slow. Especially if it's already mostly full. :( 3-weeks slow? ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Datastore resize
With the 1st stage of the rebuild taking 3 weeks, and 1416 becoming mandatory in 2 weeks, if I had just waited my node would be unusable for a week. I noticed an option to do the resize at startup instead that was supposed to be faster, so I set that and upgraded. I let it run for a bit and then did the math, it was still going to take 3 weeks. CPU and HDD were both practically idle. I don't know what a resize does, but it sure seems like there is a bug in there somewhere. -Pascal On 9/28/2012 4:10 AM, Matthew Toseland wrote: On Tuesday 25 Sep 2012 02:38:07 Pascal wrote: Changed my datastore size from 750g to 850g just over 5 hours ago. It now shows Datastore(CHK-store) resize in progress: 108040/10717960. At this rate it will take about 3 weeks just to finish CHK-store. Does it them take another 3 weeks to do CHK-cache? (They were the same size beforehand). Datastore resizing is slow. Especially if it's already mostly full. :( ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
[freenet-support] Datastore resize
Changed my datastore size from 750g to 850g just over 5 hours ago. It now shows Datastore(CHK-store) resize in progress: 108040/10717960. At this rate it will take about 3 weeks just to finish CHK-store. Does it them take another 3 weeks to do CHK-cache? (They were the same size beforehand). -Pascal ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Datastore resize in loop-no problem any more, a question still
On Tuesday 03 August 2010 14:06:12 Jep wrote: Jep schreef: My box seems to be big on FN loops! Increased the data store of Freenet from 40 to 50 gB and this 'Datastore(CHK-cache) resize in progress: 159982/721390 ' keeps on starting anew from zero, once the high number is reached. A restart of FN didn't help. What now? Restart winz? xp sp3, 2 gig memory, newest java. Hmm. This I caused myself, it's OK now. The resize process stops for some time, and then starts again with other datastore parts, that made me think it started over, so I tried a FN restart. Why does it take so long, now I'm writing anyway? It was chewing at it all night. Making a new file using the cli command 'fsutil file createnew' takes seconds only, no matter the size. Because it has to move the data around, which takes a lot of seeking, because of how the datastore works. signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] Datastore resize in loop-no problem any more, a question still
Jep schreef: My box seems to be big on FN loops! Increased the data store of Freenet from 40 to 50 gB and this 'Datastore(CHK-cache) resize in progress: 159982/721390 ' keeps on starting anew from zero, once the high number is reached. A restart of FN didn't help. What now? Restart winz? xp sp3, 2 gig memory, newest java. Hmm. This I caused myself, it's OK now. The resize process stops for some time, and then starts again with other datastore parts, that made me think it started over, so I tried a FN restart. Why does it take so long, now I'm writing anyway? It was chewing at it all night. Making a new file using the cli command 'fsutil file createnew' takes seconds only, no matter the size. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
[freenet-support] Datastore resize in loop
My box seems to be big on FN loops! Increased the data store of Freenet from 40 to 50 gB and this 'Datastore(CHK-cache) resize in progress: 159982/721390 ' keeps on starting anew from zero, once the high number is reached. A restart of FN didn't help. What now? Restart winz? xp sp3, 2 gig memory, newest java. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
