Re: [freenet-support] data store 500GB?

2015-04-08 Thread Steve Dougherty
On 04/06/2015 11:29 AM, xor wrote:
 On Sunday, March 29, 2015 05:54:43 PM Dennis New wrote:
 On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote:
 IIRC, bloom filters contribute to a memory usage of 1/3000th of the
 data store size, which would be around 1 GB of RAM for a 3TB data
 store

 This calculation/check should be done by the software when the data
 store size is set.
 
 ACK.
 As a quick fix, I would like to suggest we add the 1/3000th information to 
 the 
 l10n string at least.
 Can someone do the l10n fix for now, or at least file a bug?

The localization for the memory limit already includes:

Freenet will need more memory if you have more queued requests, or use
chat forums and other plugins, and to a lesser extent if you have a
larger datastore or a fast connection.

 I can file a bug if nobody wants to deal with it, just tell me :)

I've filed #6506 with thoughts on how to make automatic adjustments. [0]

[0] https://bugs.freenetproject.org/view.php?id=6506




signature.asc
Description: OpenPGP digital signature
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] data store 500GB?

2015-04-08 Thread Eric Chadbourne

 On Apr 6, 2015, at 11:29 AM, xor x...@freenetproject.org wrote:
 
 On Sunday, March 29, 2015 05:54:43 PM Dennis New wrote:
 On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote:
 IIRC, bloom filters contribute to a memory usage of 1/3000th of the
 data store size, which would be around 1 GB of RAM for a 3TB data
 store
 
 This calculation/check should be done by the software when the data
 store size is set.
 
 ACK.
 As a quick fix, I would like to suggest we add the 1/3000th information to 
 the 
 l10n string at least.
 Can someone do the l10n fix for now, or at least file a bug?
 
 I can file a bug if nobody wants to deal with it, just tell me :)


It’s a bit over my head but I’ll help test if needed.

Thanks for looking into it!

Eric Chadbourne
http://Nonprofit-CRM.org/
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] data store 500GB?

2015-04-06 Thread xor
On Sunday, March 29, 2015 05:54:43 PM Dennis New wrote:
 On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote:
  IIRC, bloom filters contribute to a memory usage of 1/3000th of the
  data store size, which would be around 1 GB of RAM for a 3TB data
  store
 
 This calculation/check should be done by the software when the data
 store size is set.

ACK.
As a quick fix, I would like to suggest we add the 1/3000th information to the 
l10n string at least.
Can someone do the l10n fix for now, or at least file a bug?

I can file a bug if nobody wants to deal with it, just tell me :)

Thanks

signature.asc
Description: This is a digitally signed message part.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] data store 500GB?

2015-03-29 Thread Bert Massop
On 28-03-15 20:36, Steve Dougherty wrote:
 On 03/28/2015 12:07 PM, Eric Chadbourne wrote:
 Hi All,

 Is it possible to use more than a 500GB data store?

 On a fresh installation the largest selectable value for the 
 datastore is 500GB.  I would like to use 3TB if possible.  When I 
 attempt to change the value via the web page I get an out of memory 
 error.  If I change the value in freenet.ini the application won’t 
 start.  I may not be setting the value correctly.

 Internal error: please report

 java.lang.OutOfMemoryError: Java heap space
 
 What is Maximum memory usage under Wrapper on the Configuration 
 Core settings page set to? For a store that big it seems like it'll have
 to be higher.

I think this correctly identifies the problem at hand: IIRC, bloom
filters contribute to a memory usage of 1/3000th of the data store size,
which would be around 1 GB of RAM for a 3TB data store — not counting
memory consumption of other parts of Freenet, Java memory management
overhead, etc.

Your problem can probably be solved by raising the memory limit.* Apart
from the limits imposed by your storage and RAM capacities, there is no
limit to datastore size (apart from some 2^63 bytes limits imposed by
internal representation, but that should never be an issue in the
foreseeable future).

Keep in mind that other settings may need additional tweaking for
optimal performance when such a large datastore is used.

Thank you for running Freenet!

-- Bert


* You can change the memory limit through the web interface as Steve
pointed out, or change the `wrapper.java.maxmemory' option in
wrapper.conf (after stopping Freenet): the value is the memory limit in
megabytes.



signature.asc
Description: OpenPGP digital signature
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] data store 500GB?

2015-03-29 Thread Eric Chadbourne
Hi Bert and Steve,

Excellent and informative answers.  I’ll increase max memory, physical RAM, and 
poke around other settings.

Thanks!

—
Eric Chadbourne
http://Nonprofit-CRM.org/




 On Mar 29, 2015, at 9:50 AM, Bert Massop bert.mas...@gmail.com wrote:
 
 On 28-03-15 20:36, Steve Dougherty wrote:
 On 03/28/2015 12:07 PM, Eric Chadbourne wrote:
 Hi All,
 
 Is it possible to use more than a 500GB data store?
 
 On a fresh installation the largest selectable value for the 
 datastore is 500GB.  I would like to use 3TB if possible.  When I 
 attempt to change the value via the web page I get an out of memory 
 error.  If I change the value in freenet.ini the application won’t 
 start.  I may not be setting the value correctly.
 
 Internal error: please report
 
 java.lang.OutOfMemoryError: Java heap space
 
 What is Maximum memory usage under Wrapper on the Configuration 
 Core settings page set to? For a store that big it seems like it'll have
 to be higher.
 
 I think this correctly identifies the problem at hand: IIRC, bloom
 filters contribute to a memory usage of 1/3000th of the data store size,
 which would be around 1 GB of RAM for a 3TB data store — not counting
 memory consumption of other parts of Freenet, Java memory management
 overhead, etc.
 
 Your problem can probably be solved by raising the memory limit.* Apart
 from the limits imposed by your storage and RAM capacities, there is no
 limit to datastore size (apart from some 2^63 bytes limits imposed by
 internal representation, but that should never be an issue in the
 foreseeable future).
 
 Keep in mind that other settings may need additional tweaking for
 optimal performance when such a large datastore is used.
 
 Thank you for running Freenet!
 
 -- Bert
 
 
 * You can change the memory limit through the web interface as Steve
 pointed out, or change the `wrapper.java.maxmemory' option in
 wrapper.conf (after stopping Freenet): the value is the memory limit in
 megabytes.
 
 ___
 Support mailing list
 Support@freenetproject.org
 http://news.gmane.org/gmane.network.freenet.support
 Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
 Or mailto:support-requ...@freenetproject.org?subject=unsubscribe
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] data store 500GB?

2015-03-29 Thread Wolfram Goetz


 Gesendet: Sonntag, 29. März 2015 um 15:50 Uhr
 Von: Bert Massop bert.mas...@gmail.com
 An: support@freenetproject.org
 Betreff: Re: [freenet-support] data store  500GB?

 On 28-03-15 20:36, Steve Dougherty wrote:
  On 03/28/2015 12:07 PM, Eric Chadbourne wrote:
  Hi All,
 
  Is it possible to use more than a 500GB data store?
 
  On a fresh installation the largest selectable value for the 
  datastore is 500GB.  I would like to use 3TB if possible.  When I 
  attempt to change the value via the web page I get an out of memory 
  error.  If I change the value in freenet.ini the application won’t 
  start.  I may not be setting the value correctly.
 
  Internal error: please report
 
  java.lang.OutOfMemoryError: Java heap space
  
  What is Maximum memory usage under Wrapper on the Configuration 
  Core settings page set to? For a store that big it seems like it'll have
  to be higher.
 
 I think this correctly identifies the problem at hand: IIRC, bloom
 filters contribute to a memory usage of 1/3000th of the data store size,
 which would be around 1 GB of RAM for a 3TB data store — not counting
 memory consumption of other parts of Freenet, Java memory management
 overhead, etc.
 
 Your problem can probably be solved by raising the memory limit.* Apart
 from the limits imposed by your storage and RAM capacities, there is no
 limit to datastore size (apart from some 2^63 bytes limits imposed by
 internal representation, but that should never be an issue in the
 foreseeable future).
 
 Keep in mind that other settings may need additional tweaking for
 optimal performance when such a large datastore is used.
 
 Thank you for running Freenet!
 
 -- Bert

I always got these out memory errors when I tried to run a 3TiB node under 
Windows 7 64-bit, regardsless of the said settings. When I switched to Linux 
(Kubuntu 14.04) it worked immediately. Could it also depend on the java 
version? I'm using OpenJDK under Linux now.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] data store 500GB?

2015-03-29 Thread Bert Massop
On 29-03-15 19:35, Wolfram Goetz wrote:
 
 I always got these out memory errors when I tried to run a 3TiB node under 
 Windows 7 64-bit, regardsless of the said settings. When I switched to Linux 
 (Kubuntu 14.04) it worked immediately. Could it also depend on the java 
 version? I'm using OpenJDK under Linux now.

I'm not sure it is related to the Java version at hand. What I suspect
to have been the culprit in your situation, is the architecture of the
Java installation you employed on your Windows machine.

32-bit applications on Windows can only use 2 GiB of virtual memory (or
3 GiB with the large address awareness flag set in the executable
image), and Java is quite liberal in its virtual memory usage…

Oracle recommends using a 32-bit Java under most conditions, even on a
64-bit Windows installation. On 64-bit Linux, you'll probably have had a
64-bit Java by default, on Windows, it might have been a 32-bit one.

If you can reproduce this on a 64-bit JVM (you can check the version and
architecture with the command `java -version'), please file a bug report.

-- Bert



signature.asc
Description: OpenPGP digital signature
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] data store 500GB?

2015-03-29 Thread Dennis New
On Sun, 29 Mar 2015 15:50:34 +0200, Bert Massop wrote:
 IIRC, bloom filters contribute to a memory usage of 1/3000th of the
 data store size, which would be around 1 GB of RAM for a 3TB data
 store

This calculation/check should be done by the software when the data
store size is set.
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

Re: [freenet-support] data store 500GB?

2015-03-28 Thread Steve Dougherty
On 03/28/2015 12:07 PM, Eric Chadbourne wrote:
 Hi All,
 
 Is it possible to use more than a 500GB data store?
 
 On a fresh installation the largest selectable value for the 
 datastore is 500GB.  I would like to use 3TB if possible.  When I 
 attempt to change the value via the web page I get an out of memory 
 error.  If I change the value in freenet.ini the application won’t 
 start.  I may not be setting the value correctly.
 
 Internal error: please report
 
 java.lang.OutOfMemoryError: Java heap space

What is Maximum memory usage under Wrapper on the Configuration 
Core settings page set to? For a store that big it seems like it'll have
to be higher.



signature.asc
Description: OpenPGP digital signature
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe

[freenet-support] data store 500GB?

2015-03-28 Thread Eric Chadbourne
Hi All,

Is it possible to use more than a 500GB data store?

On a fresh installation the largest selectable value for the datastore is 
500GB.  I would like to use 3TB if possible.  When I attempt to change the 
value via the web page I get an out of memory error.  If I change the value in 
freenet.ini the application won’t start.  I may not be setting the value 
correctly.

Internal error: please report

java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3284)
at 
freenet.store.saltedhash.ResizablePersistentIntBuffer.resize(ResizablePersistentIntBuffer.java:268)
at 
freenet.store.saltedhash.SaltedHashFreenetStore.setMaxKeys(SaltedHashFreenetStore.java:1947)
at freenet.store.CachingFreenetStore.setMaxKeys(CachingFreenetStore.java:246)
at freenet.store.StoreCallback.setMaxKeys(StoreCallback.java:62)
at freenet.node.Node$32.set(Node.java:2017)
at freenet.node.Node$32.set(Node.java:1995)
at freenet.config.Option.set(Option.java:73)
at freenet.config.Option.setValue(Option.java:62)
at freenet.clients.http.ConfigToadlet.handleMethodPOST(ConfigToadlet.java:329)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
freenet.clients.http.ToadletContextImpl.callToadletMethod(ToadletContextImpl.java:735)
at freenet.clients.http.ToadletContextImpl.handle(ToadletContextImpl.java:648)
at 
freenet.clients.http.SimpleToadletServer$SocketHandler.run(SimpleToadletServer.java:1031)
at freenet.support.PooledExecutor$MyThread.innerRun(PooledExecutor.java:248)
at freenet.support.PooledExecutor$MyThread.realRun(PooledExecutor.java:188)
at freenet.support.io.NativeThread.run(NativeThread.java:129)

Thanks for any tips,

—
Eric Chadbourne
http://Nonprofit-CRM.org/
___
Support mailing list
Support@freenetproject.org
http://news.gmane.org/gmane.network.freenet.support
Unsubscribe at http://emu.freenetproject.org/cgi-bin/mailman/listinfo/support
Or mailto:support-requ...@freenetproject.org?subject=unsubscribe