Re: [freenet-support] data store > 500GB?
On 04/06/2015 11:29 AM, xor wrote: > On Sunday, March 29, 2015 05:54:43 PM Dennis New wrote: >> On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote: >>> IIRC, bloom filters contribute to a memory usage of 1/3000th of the >>> data store size, which would be around 1 GB of RAM for a 3TB data >>> store >> >> This calculation/check should be done by the software when the data >> store size is set. > > ACK. > As a quick fix, I would like to suggest we add the 1/3000th information to > the > l10n string at least. > Can someone do the l10n fix for now, or at least file a bug? The localization for the memory limit already includes: "Freenet will need more memory if you have more queued requests, or use chat forums and other plugins, and to a lesser extent if you have a larger datastore or a fast connection." > I can file a bug if nobody wants to deal with it, just tell me :) I've filed #6506 with thoughts on how to make automatic adjustments. [0] [0] https://bugs.freenetproject.org/view.php?id=6506 signature.asc Description: OpenPGP digital signature ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store > 500GB?
> On Apr 6, 2015, at 11:29 AM, xor wrote: > > On Sunday, March 29, 2015 05:54:43 PM Dennis New wrote: >> On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote: >>> IIRC, bloom filters contribute to a memory usage of 1/3000th of the >>> data store size, which would be around 1 GB of RAM for a 3TB data >>> store >> >> This calculation/check should be done by the software when the data >> store size is set. > > ACK. > As a quick fix, I would like to suggest we add the 1/3000th information to > the > l10n string at least. > Can someone do the l10n fix for now, or at least file a bug? > > I can file a bug if nobody wants to deal with it, just tell me :) It’s a bit over my head but I’ll help test if needed. Thanks for looking into it! Eric Chadbourne http://Nonprofit-CRM.org/ ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store > 500GB?
On Sunday, March 29, 2015 05:54:43 PM Dennis New wrote: > On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote: > > IIRC, bloom filters contribute to a memory usage of 1/3000th of the > > data store size, which would be around 1 GB of RAM for a 3TB data > > store > > This calculation/check should be done by the software when the data > store size is set. ACK. As a quick fix, I would like to suggest we add the 1/3000th information to the l10n string at least. Can someone do the l10n fix for now, or at least file a bug? I can file a bug if nobody wants to deal with it, just tell me :) Thanks signature.asc Description: This is a digitally signed message part. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store > 500GB?
On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote: > IIRC, bloom filters contribute to a memory usage of 1/3000th of the > data store size, which would be around 1 GB of RAM for a 3TB data > store This calculation/check should be done by the software when the data store size is set. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store > 500GB?
On 29-03-15 19:35, Wolfram Goetz wrote: > > I always got these out memory errors when I tried to run a 3TiB node under > Windows 7 64-bit, regardsless of the said settings. When I switched to Linux > (Kubuntu 14.04) it worked immediately. Could it also depend on the java > version? I'm using OpenJDK under Linux now. I'm not sure it is related to the Java version at hand. What I suspect to have been the culprit in your situation, is the architecture of the Java installation you employed on your Windows machine. 32-bit applications on Windows can only use 2 GiB of virtual memory (or 3 GiB with the large address awareness flag set in the executable image), and Java is quite liberal in its virtual memory usage… Oracle recommends using a 32-bit Java under most conditions, even on a 64-bit Windows installation. On 64-bit Linux, you'll probably have had a 64-bit Java by default, on Windows, it might have been a 32-bit one. If you can reproduce this on a 64-bit JVM (you can check the version and architecture with the command `java -version'), please file a bug report. -- Bert signature.asc Description: OpenPGP digital signature ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store > 500GB?
> Gesendet: Sonntag, 29. März 2015 um 15:50 Uhr > Von: "Bert Massop" > An: support@freenetproject.org > Betreff: Re: [freenet-support] data store > 500GB? > > On 28-03-15 20:36, Steve Dougherty wrote: > > On 03/28/2015 12:07 PM, Eric Chadbourne wrote: > >> Hi All, > >> > >> Is it possible to use more than a 500GB data store? > >> > >> On a fresh installation the largest selectable value for the > >> datastore is 500GB. I would like to use 3TB if possible. When I > >> attempt to change the value via the web page I get an out of memory > >> error. If I change the value in freenet.ini the application won’t > >> start. I may not be setting the value correctly. > >> > >> Internal error: please report > >> > >> java.lang.OutOfMemoryError: Java heap space > > > > What is "Maximum memory usage" under "Wrapper" on the Configuration > > > Core settings page set to? For a store that big it seems like it'll have > > to be higher. > > I think this correctly identifies the problem at hand: IIRC, bloom > filters contribute to a memory usage of 1/3000th of the data store size, > which would be around 1 GB of RAM for a 3TB data store — not counting > memory consumption of other parts of Freenet, Java memory management > overhead, etc. > > Your problem can probably be solved by raising the memory limit.* Apart > from the limits imposed by your storage and RAM capacities, there is no > limit to datastore size (apart from some 2^63 bytes limits imposed by > internal representation, but that should never be an issue in the > foreseeable future). > > Keep in mind that other settings may need additional tweaking for > optimal performance when such a large datastore is used. > > Thank you for running Freenet! > > -- Bert I always got these out memory errors when I tried to run a 3TiB node under Windows 7 64-bit, regardsless of the said settings. When I switched to Linux (Kubuntu 14.04) it worked immediately. Could it also depend on the java version? I'm using OpenJDK under Linux now. ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store > 500GB?
Hi Bert and Steve, Excellent and informative answers. I’ll increase max memory, physical RAM, and poke around other settings. Thanks! — Eric Chadbourne http://Nonprofit-CRM.org/ > On Mar 29, 2015, at 9:50 AM, Bert Massop wrote: > > On 28-03-15 20:36, Steve Dougherty wrote: >> On 03/28/2015 12:07 PM, Eric Chadbourne wrote: >>> Hi All, >>> >>> Is it possible to use more than a 500GB data store? >>> >>> On a fresh installation the largest selectable value for the >>> datastore is 500GB. I would like to use 3TB if possible. When I >>> attempt to change the value via the web page I get an out of memory >>> error. If I change the value in freenet.ini the application won’t >>> start. I may not be setting the value correctly. >>> >>> Internal error: please report >>> >>> java.lang.OutOfMemoryError: Java heap space >> >> What is "Maximum memory usage" under "Wrapper" on the Configuration > >> Core settings page set to? For a store that big it seems like it'll have >> to be higher. > > I think this correctly identifies the problem at hand: IIRC, bloom > filters contribute to a memory usage of 1/3000th of the data store size, > which would be around 1 GB of RAM for a 3TB data store — not counting > memory consumption of other parts of Freenet, Java memory management > overhead, etc. > > Your problem can probably be solved by raising the memory limit.* Apart > from the limits imposed by your storage and RAM capacities, there is no > limit to datastore size (apart from some 2^63 bytes limits imposed by > internal representation, but that should never be an issue in the > foreseeable future). > > Keep in mind that other settings may need additional tweaking for > optimal performance when such a large datastore is used. > > Thank you for running Freenet! > > -- Bert > > > * You can change the memory limit through the web interface as Steve > pointed out, or change the `wrapper.java.maxmemory' option in > wrapper.conf (after stopping Freenet): the value is the memory limit in > megabytes. > > ___ > Support mailing list > Support@freenetproject.org > http://news.gmane.org/gmane.network.freenet.support > Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support > Or mailto:support-requ...@freenetproject.org?subject=unsubscribe ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store > 500GB?
On 28-03-15 20:36, Steve Dougherty wrote: > On 03/28/2015 12:07 PM, Eric Chadbourne wrote: >> Hi All, >> >> Is it possible to use more than a 500GB data store? >> >> On a fresh installation the largest selectable value for the >> datastore is 500GB. I would like to use 3TB if possible. When I >> attempt to change the value via the web page I get an out of memory >> error. If I change the value in freenet.ini the application won’t >> start. I may not be setting the value correctly. >> >> Internal error: please report >> >> java.lang.OutOfMemoryError: Java heap space > > What is "Maximum memory usage" under "Wrapper" on the Configuration > > Core settings page set to? For a store that big it seems like it'll have > to be higher. I think this correctly identifies the problem at hand: IIRC, bloom filters contribute to a memory usage of 1/3000th of the data store size, which would be around 1 GB of RAM for a 3TB data store — not counting memory consumption of other parts of Freenet, Java memory management overhead, etc. Your problem can probably be solved by raising the memory limit.* Apart from the limits imposed by your storage and RAM capacities, there is no limit to datastore size (apart from some 2^63 bytes limits imposed by internal representation, but that should never be an issue in the foreseeable future). Keep in mind that other settings may need additional tweaking for optimal performance when such a large datastore is used. Thank you for running Freenet! -- Bert * You can change the memory limit through the web interface as Steve pointed out, or change the `wrapper.java.maxmemory' option in wrapper.conf (after stopping Freenet): the value is the memory limit in megabytes. signature.asc Description: OpenPGP digital signature ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
Re: [freenet-support] data store > 500GB?
On 03/28/2015 12:07 PM, Eric Chadbourne wrote: > Hi All, > > Is it possible to use more than a 500GB data store? > > On a fresh installation the largest selectable value for the > datastore is 500GB. I would like to use 3TB if possible. When I > attempt to change the value via the web page I get an out of memory > error. If I change the value in freenet.ini the application won’t > start. I may not be setting the value correctly. > > Internal error: please report > > java.lang.OutOfMemoryError: Java heap space What is "Maximum memory usage" under "Wrapper" on the Configuration > Core settings page set to? For a store that big it seems like it'll have to be higher. signature.asc Description: OpenPGP digital signature ___ Support mailing list Support@freenetproject.org http://news.gmane.org/gmane.network.freenet.support Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
