Re: java heap size

[Ted Dunning]

Of course, with only 32 - 200 MB of heap, almost anything will collect
pretty quickly!

If you are only storing a little bit of data (coordination, basically) then
a small heap might work fine.  Turn on gc logging and read what you get.  If
that is ok even with high load, then you should be fine.

One application where I saw a need for memory was when we stored user
sessions in ZK.

On Tue, Mar 16, 2010 at 10:46 AM, Patrick Hunt  wrote:

> Add on top of that the poor performance of the GC in 1.6 jvms and more
> memory is critical/necessity.
>

Re: java heap size

[Patrick Hunt]

We have some ZK clusters that service 1 clients, using ZK for group 
membership, leader election, sharding, etc... Additionally we have 
clusters that service multi-tennant instances (more than one service 
using a ZK cluster). Add on top of that the poor performance of the GC 
in 1.6 jvms and more memory is critical/necessity.


Patrick

Lei Zhang wrote:

Thanks for your reply Mahadev -

I wonder if you can elaborate on how you get to your observation of most
applications that use ZooKeeper needing JVM heap size > 64M. Our platform
uses ZooKeeper to manager cluster of hundreds of nodes with -Xmx32m. The
only time I observed OutOfMemory issue is when my application goes into
endless loop injecting messaging into a ZooKeeper queue.

Allowing ZooKeeper server to take up 3G out of 4G physical memory seems
excessive, imprudent.

Re: java heap size

[Lei Zhang]

Thanks for your reply Mahadev -

I wonder if you can elaborate on how you get to your observation of most
applications that use ZooKeeper needing JVM heap size > 64M. Our platform
uses ZooKeeper to manager cluster of hundreds of nodes with -Xmx32m. The
only time I observed OutOfMemory issue is when my application goes into
endless loop injecting messaging into a ZooKeeper queue.

Allowing ZooKeeper server to take up 3G out of 4G physical memory seems
excessive, imprudent.

Re: java heap size

[Mahadev Konar]

I am not sure I understand it.

Lei, the default JVM size actually would not fit the JVM heap size of most
of the applications that use ZooKeeper. Its always good to understand the
requirements that are needed to run a system. I am not sure how its
confusing. Is it the language or how to set it?

Thanks
mahadev


On 3/15/10 8:04 PM, "Lei Zhang"  wrote:

> Sorry I was too terse - I meant to say the default JVM heap size setting
> probably fits the need of most applications. Singling out this setting in
> Zookeeper Admin Guide is confusing rather than helping.

Re: java heap size

[Lei Zhang]

Sorry I was too terse - I meant to say the default JVM heap size setting
probably fits the need of most applications. Singling out this setting in
Zookeeper Admin Guide is confusing rather than helping.

Re: java heap size

[Lei Zhang]

Well JVM's default maximum heap size is 64M according to
this
.
I think this is a doc bug.

Re: java heap size

[Ted Dunning]

Your understanding is correct.  But if you set a heap size nearly as big as
your physical memory (or larger) then java may allocate that heap which will
cause swapping.

So swapping is definitely done by the OS, but it is the applications like
Java that can cause the OS to do it.

On Mon, Mar 15, 2010 at 5:05 PM, Lei Zhang  wrote:

> My understanding is size of heap directly affects how many objects you can
> keep in memory, how soon you get "OutOfMemory" exception. Can somebody
> explain how this affects swapping - which I believe is controlled by OS,
> not
> JVM?
>

java heap size

[Lei Zhang]

Hi:
I am puzzled by the Zookeeper Admin Guide's advice on choosing Java heap
size:

"Set the Java heap size. This is very important to avoid swapping, which
will seriously degrade ZooKeeper performance. To determine the correct
value, use load tests, and make sure you are well below the usage limit that
would cause you to swap. Be conservative - use a maximum heap size of 3GB
for a 4GB machine."

My understanding is size of heap directly affects how many objects you can
keep in memory, how soon you get "OutOfMemory" exception. Can somebody
explain how this affects swapping - which I believe is controlled by OS, not
JVM?

Thanks,
Lei