Re: [freenet-support] data store 500GB?
On 04/06/2015 11:29 AM, xor wrote: On Sunday, March 29, 2015 05:54:43 PM Dennis New wrote: On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote: IIRC, bloom filters contribute to a memory usage of 1/3000th of the data store size, which would be around 1 GB of RAM for a 3TB data store This calculation/check should be done by the software when the data store size is set. ACK. As a quick fix, I would like to suggest we add the 1/3000th information to the l10n string at least. Can someone do the l10n fix for now, or at least file a bug? The localization for the memory limit already includes: Freenet will need more memory if you have more queued requests, or use chat forums and other plugins, and to a lesser extent if you have a larger datastore or a fast connection. I can file a bug if nobody wants to deal with it, just tell me :) I've filed #6506 with thoughts on how to make automatic adjustments. [0] [0] https://bugs.freenetproject.org/view.php?id=6506 signature.asc Description: OpenPGP digital signature ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store 500GB?
On Apr 6, 2015, at 11:29 AM, xor x...@freenetproject.org wrote: On Sunday, March 29, 2015 05:54:43 PM Dennis New wrote: On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote: IIRC, bloom filters contribute to a memory usage of 1/3000th of the data store size, which would be around 1 GB of RAM for a 3TB data store This calculation/check should be done by the software when the data store size is set. ACK. As a quick fix, I would like to suggest we add the 1/3000th information to the l10n string at least. Can someone do the l10n fix for now, or at least file a bug? I can file a bug if nobody wants to deal with it, just tell me :) It’s a bit over my head but I’ll help test if needed. Thanks for looking into it! Eric Chadbourne http://Nonprofit-CRM.org/ ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store 500GB?
On Sunday, March 29, 2015 05:54:43 PM Dennis New wrote: On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote: IIRC, bloom filters contribute to a memory usage of 1/3000th of the data store size, which would be around 1 GB of RAM for a 3TB data store This calculation/check should be done by the software when the data store size is set. ACK. As a quick fix, I would like to suggest we add the 1/3000th information to the l10n string at least. Can someone do the l10n fix for now, or at least file a bug? I can file a bug if nobody wants to deal with it, just tell me :) Thanks signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store 500GB?
On 28-03-15 20:36, Steve Dougherty wrote: On 03/28/2015 12:07 PM, Eric Chadbourne wrote: Hi All, Is it possible to use more than a 500GB data store? On a fresh installation the largest selectable value for the datastore is 500GB. I would like to use 3TB if possible. When I attempt to change the value via the web page I get an out of memory error. If I change the value in freenet.ini the application won’t start. I may not be setting the value correctly. Internal error: please report java.lang.OutOfMemoryError: Java heap space What is Maximum memory usage under Wrapper on the Configuration Core settings page set to? For a store that big it seems like it'll have to be higher. I think this correctly identifies the problem at hand: IIRC, bloom filters contribute to a memory usage of 1/3000th of the data store size, which would be around 1 GB of RAM for a 3TB data store — not counting memory consumption of other parts of Freenet, Java memory management overhead, etc. Your problem can probably be solved by raising the memory limit.* Apart from the limits imposed by your storage and RAM capacities, there is no limit to datastore size (apart from some 2^63 bytes limits imposed by internal representation, but that should never be an issue in the foreseeable future). Keep in mind that other settings may need additional tweaking for optimal performance when such a large datastore is used. Thank you for running Freenet! -- Bert * You can change the memory limit through the web interface as Steve pointed out, or change the `wrapper.java.maxmemory' option in wrapper.conf (after stopping Freenet): the value is the memory limit in megabytes. signature.asc Description: OpenPGP digital signature ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store 500GB?
Hi Bert and Steve, Excellent and informative answers. I’ll increase max memory, physical RAM, and poke around other settings. Thanks! — Eric Chadbourne http://Nonprofit-CRM.org/ On Mar 29, 2015, at 9:50 AM, Bert Massop bert.mas...@gmail.com wrote: On 28-03-15 20:36, Steve Dougherty wrote: On 03/28/2015 12:07 PM, Eric Chadbourne wrote: Hi All, Is it possible to use more than a 500GB data store? On a fresh installation the largest selectable value for the datastore is 500GB. I would like to use 3TB if possible. When I attempt to change the value via the web page I get an out of memory error. If I change the value in freenet.ini the application won’t start. I may not be setting the value correctly. Internal error: please report java.lang.OutOfMemoryError: Java heap space What is Maximum memory usage under Wrapper on the Configuration Core settings page set to? For a store that big it seems like it'll have to be higher. I think this correctly identifies the problem at hand: IIRC, bloom filters contribute to a memory usage of 1/3000th of the data store size, which would be around 1 GB of RAM for a 3TB data store — not counting memory consumption of other parts of Freenet, Java memory management overhead, etc. Your problem can probably be solved by raising the memory limit.* Apart from the limits imposed by your storage and RAM capacities, there is no limit to datastore size (apart from some 2^63 bytes limits imposed by internal representation, but that should never be an issue in the foreseeable future). Keep in mind that other settings may need additional tweaking for optimal performance when such a large datastore is used. Thank you for running Freenet! -- Bert * You can change the memory limit through the web interface as Steve pointed out, or change the `wrapper.java.maxmemory' option in wrapper.conf (after stopping Freenet): the value is the memory limit in megabytes. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store 500GB?
Gesendet: Sonntag, 29. März 2015 um 15:50 Uhr Von: Bert Massop bert.mas...@gmail.com An: support@freenetproject.org Betreff: Re: [freenet-support] data store 500GB? On 28-03-15 20:36, Steve Dougherty wrote: On 03/28/2015 12:07 PM, Eric Chadbourne wrote: Hi All, Is it possible to use more than a 500GB data store? On a fresh installation the largest selectable value for the datastore is 500GB. I would like to use 3TB if possible. When I attempt to change the value via the web page I get an out of memory error. If I change the value in freenet.ini the application won’t start. I may not be setting the value correctly. Internal error: please report java.lang.OutOfMemoryError: Java heap space What is Maximum memory usage under Wrapper on the Configuration Core settings page set to? For a store that big it seems like it'll have to be higher. I think this correctly identifies the problem at hand: IIRC, bloom filters contribute to a memory usage of 1/3000th of the data store size, which would be around 1 GB of RAM for a 3TB data store — not counting memory consumption of other parts of Freenet, Java memory management overhead, etc. Your problem can probably be solved by raising the memory limit.* Apart from the limits imposed by your storage and RAM capacities, there is no limit to datastore size (apart from some 2^63 bytes limits imposed by internal representation, but that should never be an issue in the foreseeable future). Keep in mind that other settings may need additional tweaking for optimal performance when such a large datastore is used. Thank you for running Freenet! -- Bert I always got these out memory errors when I tried to run a 3TiB node under Windows 7 64-bit, regardsless of the said settings. When I switched to Linux (Kubuntu 14.04) it worked immediately. Could it also depend on the java version? I'm using OpenJDK under Linux now. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store 500GB?
On 29-03-15 19:35, Wolfram Goetz wrote: I always got these out memory errors when I tried to run a 3TiB node under Windows 7 64-bit, regardsless of the said settings. When I switched to Linux (Kubuntu 14.04) it worked immediately. Could it also depend on the java version? I'm using OpenJDK under Linux now. I'm not sure it is related to the Java version at hand. What I suspect to have been the culprit in your situation, is the architecture of the Java installation you employed on your Windows machine. 32-bit applications on Windows can only use 2 GiB of virtual memory (or 3 GiB with the large address awareness flag set in the executable image), and Java is quite liberal in its virtual memory usage… Oracle recommends using a 32-bit Java under most conditions, even on a 64-bit Windows installation. On 64-bit Linux, you'll probably have had a 64-bit Java by default, on Windows, it might have been a 32-bit one. If you can reproduce this on a 64-bit JVM (you can check the version and architecture with the command `java -version'), please file a bug report. -- Bert signature.asc Description: OpenPGP digital signature ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store 500GB?
On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote: IIRC, bloom filters contribute to a memory usage of 1/3000th of the data store size, which would be around 1 GB of RAM for a 3TB data store This calculation/check should be done by the software when the data store size is set. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store 500GB?
On 03/28/2015 12:07 PM, Eric Chadbourne wrote: Hi All, Is it possible to use more than a 500GB data store? On a fresh installation the largest selectable value for the datastore is 500GB. I would like to use 3TB if possible. When I attempt to change the value via the web page I get an out of memory error. If I change the value in freenet.ini the application won’t start. I may not be setting the value correctly. Internal error: please report java.lang.OutOfMemoryError: Java heap space What is Maximum memory usage under Wrapper on the Configuration Core settings page set to? For a store that big it seems like it'll have to be higher. signature.asc Description: OpenPGP digital signature ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
[freenet-support] data store 500GB?
Hi All, Is it possible to use more than a 500GB data store? On a fresh installation the largest selectable value for the datastore is 500GB. I would like to use 3TB if possible. When I attempt to change the value via the web page I get an out of memory error. If I change the value in freenet.ini the application won’t start. I may not be setting the value correctly. Internal error: please report java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3284) at freenet.store.saltedhash.ResizablePersistentIntBuffer.resize(ResizablePersistentIntBuffer.java:268) at freenet.store.saltedhash.SaltedHashFreenetStore.setMaxKeys(SaltedHashFreenetStore.java:1947) at freenet.store.CachingFreenetStore.setMaxKeys(CachingFreenetStore.java:246) at freenet.store.StoreCallback.setMaxKeys(StoreCallback.java:62) at freenet.node.Node$32.set(Node.java:2017) at freenet.node.Node$32.set(Node.java:1995) at freenet.config.Option.set(Option.java:73) at freenet.config.Option.setValue(Option.java:62) at freenet.clients.http.ConfigToadlet.handleMethodPOST(ConfigToadlet.java:329) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at freenet.clients.http.ToadletContextImpl.callToadletMethod(ToadletContextImpl.java:735) at freenet.clients.http.ToadletContextImpl.handle(ToadletContextImpl.java:648) at freenet.clients.http.SimpleToadletServer$SocketHandler.run(SimpleToadletServer.java:1031) at freenet.support.PooledExecutor$MyThread.innerRun(PooledExecutor.java:248) at freenet.support.PooledExecutor$MyThread.realRun(PooledExecutor.java:188) at freenet.support.io.NativeThread.run(NativeThread.java:129) Thanks for any tips, — Eric Chadbourne http://Nonprofit-CRM.org/ ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
[freenet-support] Data store wiped - 11 Gb gone.
On checking the environment I see my data store was wiped. Down from 11+ Gb when I lasted check to less than 700 Mb. I haven't looked at the logs because I don't know if anyone cares, and I wouldn't know what to look for. But I'd be happy to do so if anyone has an interest in this and wants to tell me what to look for. My guess would be the data store was completely wiped to zero, which means it's probably been refilling for about a day. The even that corresponds with that approximate timeframe would be my LAN losing its internet connection for 30 minutes, and being restored with a new IP address. I'm sad to lose my 11 Gb which I've been nursing along since July. Is this to be expected? Or is it a problem, but there are bigger problems to worry about? With thanks for any reply. ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] DATA STORE
Hey there! I have a 6+ gigabyte datastore(yeah, yeah, hold the praise. Thank you, thank you...) What I can't figure is this... When I stop Freenet for whatever reason, and then restart it, it looks to me like it should load some of my previously acquired data and images etc. quickly, getting them from my own datastore. Instead though, here we go again, tapping fingers on desk, going for coffee, taking a nap, going out for a movie, taking a vacation when, upon return, VOILA, I'm back to where I was before restarting Freenet. How can I force my system to look through my own datastore first OR how can I force my system to put my stuff in my datastore and then go there to retrieve it before going out on a world cruise for more? Anonymity issues? What? The stuff I have is all encrypted anyway. Please help me to force a local search/retrieve first. IS THERE HELP? P.S. The recent Linux/Windoze duels, though off the subject, are quite entertaining. I'm rooting for the underpenguin. -- miguel [EMAIL PROTECTED] ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] DATA STORE
On Wed, Jun 16, 2004 at 03:55:08AM -0700, miguel wrote: Hey there! I have a 6+ gigabyte datastore(yeah, yeah, hold the praise. Thank you, thank you...) My 3 nodes (usually 1-2 of them are running) have DSs of: storeSize=0.9G (second unstable node used for testing) storeSize=19G (stable node) storeSize=18G (unstable node) But then I get paid for this... :) I know that Iakin has 100GB+ stores on some of his nodes. What I can't figure is this... When I stop Freenet for whatever reason, and then restart it, it looks to me like it should load some of my previously acquired data and images etc. quickly, getting them from my own datastore. Instead though, here we go again, tapping fingers on desk, going for coffee, taking a nap, going out for a movie, taking a vacation when, upon return, VOILA, I'm back to where I was before restarting Freenet. What are you complaining about here? Is data being lost from the store or are you just annoyed that it takes so long to restart the node? How can I force my system to look through my own datastore first It does. It's just that not everything that goes through your node is cached. This is because your store is full, so we try to enhance routing by probabilistic caching. At some point we may implement a local-only cache to supplant the datastore; this would disappear on restart for security reasons. OR how can I force my system to put my stuff in my datastore and then go there to retrieve it before going out on a world cruise for more? Anonymity issues? What? The stuff I have is all encrypted anyway. Please help me to force a local search/retrieve first. Anonymity is more complex than it's all encrypted anyway. We cannot always cache locally requested data, while probabilistically caching everything else, because then an attacker could tell what you've been browsing. We could use a strict LRU for all data, however the simulations suggest that probabilistic caching should improve routing performance by reducing the routing dilution that happens as a result of excessive caching. We DID try turning off pcaching for a while and it didn't help performance. IS THERE HELP? -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] DATA STORE
Toad wrote: On Wed, Jun 16, 2004 at 03:55:08AM -0700, miguel wrote: What I can't figure is this... When I stop Freenet for whatever reason, and then restart it, it looks to me like it should load some of my previously acquired data and images etc. quickly, getting them from my own datastore. Instead though, here we go again, tapping fingers on desk, going for coffee, taking a nap, going out for a movie, taking a vacation when, upon return, VOILA, I'm back to where I was before restarting Freenet. What are you complaining about here? Is data being lost from the store or are you just annoyed that it takes so long to restart the node? Anytime I've had to restart my node, it appears that the data store is just plain gone. I can go from having my resources at 60+% full one minutes, restart, and I'm at 0. That and it then seems to need to be active again for 10+ hours before clicking on anything is practical. Unfortunately I'm only able to run a transient node at the moment, something I'm working to remedy, and doing so seems almost worthless. Unfortunately, from what I'm understanding of the rest of your response, it doesn't sound as if there's anything to be done about this, other then not restart. Is that a correct interpretation ? Mike S. ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] DATA STORE
On Wed, Jun 16, 2004 at 10:07:02AM -0400, Michael R. Stork wrote: Toad wrote: On Wed, Jun 16, 2004 at 03:55:08AM -0700, miguel wrote: What I can't figure is this... When I stop Freenet for whatever reason, and then restart it, it looks to me like it should load some of my previously acquired data and images etc. quickly, getting them from my own datastore. Instead though, here we go again, tapping fingers on desk, going for coffee, taking a nap, going out for a movie, taking a vacation when, upon return, VOILA, I'm back to where I was before restarting Freenet. What are you complaining about here? Is data being lost from the store or are you just annoyed that it takes so long to restart the node? Anytime I've had to restart my node, it appears that the data store is just plain gone. I can go from having my resources at 60+% full one minutes, restart, and I'm at 0. That and it then seems to need to be active again for 10+ hours before clicking on anything is practical. Unfortunately I'm only able to run a transient node at the moment, something I'm working to remedy, and doing so seems almost worthless. Hmm. That rather indicates a bug. Do you have storeIndex enabled? How much of the data in question is in store/temp/ ? Unfortunately, from what I'm understanding of the rest of your response, it doesn't sound as if there's anything to be done about this, other then not restart. Is that a correct interpretation ? Mike S. ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED] -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
[freenet-support] data store
is there a problem if my nodes data store reaches 100% ? Robert Greenage [EMAIL PROTECTED] Why Wait? Move to EarthLink. ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
Re: [freenet-support] data store
On Fri, Jun 04, 2004 at 06:06:46AM -0500, Robert Greenage wrote: is there a problem if my nodes data store reaches 100% ? No. -- Matthew J Toseland - [EMAIL PROTECTED] Freenet Project Official Codemonkey - http://freenetproject.org/ ICTHUS - Nothing is impossible. Our Boss says so. signature.asc Description: Digital signature ___ Support mailing list [EMAIL PROTECTED] http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://dodo.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:[EMAIL PROTECTED]
