Re: [Resin-interest] implicit stack memory size?

2009-01-09 Thread Jay Ballinger
Thanks much, guys, for the replies.

We've arrived at the same conclusion that Scott suggests - a GB for the 
heap and the other GB for the product (roughly) of the threads and the 
stack size in our windows 32-bit environment.

Now it's an exercise in testing the actual -Xss size that we want to run 
with - which is proving to be a bit elusive. Understanding that local 
primitives and heap-object addresses are stored on the stack, I can't 
envision a need for a stack size anywhere bigger than 1MB (for our app, 
YMMV). Even 1MB is probably way over doing it.



There's a few of us here at work who have been lurkers on this list for 
a long time - or at least a couple of years anyway. We've always found 
this list valuable and appreciate everyone's involvement. Thanks again.


+ jay



Scott Ferguson wrote:
 On Jan 8, 2009, at 5:23 PM, Rob Lockstone wrote:
 
 On Jan 8, 2009, at 13:52, Jay Ballinger wrote:

 Hi everyone,

 We're running Resin 3.0.18 Pro on a Windows 32-bit system and we're
 looking to understand/tune our memory settings a bit.

 According to http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
 the stack size is set to 2048k unless set otherwise. Is this true?
 In using JConsole we can't see where any explicit -Xss setting is
 passed and we can't readily determine what the current stack size is
 set to.
 
 Try -verbose when starting.  Since the -Xss is a command line option,  
 it should be reported.  Also, JConsole should be able to see the  
 command line options on one of the system beans.
 
 I think Caucho tends to err on the side of caution in this regard, and
 is also (understandably) more *nix-central than Windows. So take that
 2048K as being appropriate for most situations with a grain of salt
 because situations can vary wildly.
 
 Basically.  There's no way for us to know if your application needs  
 more stack or more threads.  So I assumed 1G heap, leaving 1G for  
 other stuff, and then figured the default should support 1000 threads,  
 giving 1m as a default, which should be enough stack unless you have  
 massively recursive applications, and enough threads for most apps.   
 You know your own requirements best, of course.
 
 I believe Linux is the most finicky about the stack allocation, so it  
 was the OS we needed to solve (and it was the OS which exposed the  
 issue first.)
 
 I'm really looking forward for the 64-bit transition to complete so we  
 can use as much VM as possible.
 
 -- Scott
 
 

 So, here's a short list of questions. And thanks for your help.

 - If JConsole doesn't show -Xss, is it really there for resin?
 Whether it's explicitly set or not, there IS a stack size, which would
 end up being the default stack size of the operating system if the JVM
 itself doesn't set it (I do not think it does, but could be wrong and
 don't feel like looking it up now).

 - How was 2048k is an appropriate value for most situations.
 determined?
 See above.

 - Can resin/jvm/jconsole report the current stack memory setting?
 I don't know if there's a way to query a running JVM for this
 information. It's best if you explicitly set the stack size using the
 JVM option -Xss (but see below).

 (general sun jvm on windows questions)
 - What is the default stack memory size?
 That's too general of a question. It depends on the OS, and (perhaps)
 the version of the OS.

 - When a thread is created, does it grab all of the stack memory, or
 can the stack memory start small and grow to the default or -Xss  
 size?
 Again, might depend on the operating system. Contrary to what Knut
 said, and to the best of my knowledge and in the testing that I did
 (it's been a while), all of the memory for the stack is pre-allocated
 (at least from the perspective of the JVM itself). But it's easy
 enough to write a program to check this.

 Thread stacks, at least on Windows, are more complicated than just a
 single memory allocation (down at the Windows API level) because
 you've got both reserved and committed memory allocations, and one
 always has to be = the other (I forget which), and there's a 64K
 granularity in sizing, and you're dealing with physical and virtual
 memory, and this is all set in the linker and therefore configurable,
 but can also be set programmatically on a per-thread basis. In other
 words, it's quite involved. See this page as an example of how complex
 this kind of memory management is when you go into how the underlying
 OS is actually managing all of this 
 http://msdn.microsoft.com/en-us/library/ms686774(VS.85).aspx
 .
 Remember, thread allocation/destruction as well as context switching
 and just about everything else is managed by the operating system, not
 Java (unless you're running green threads, which I don't think is
 even around anymore?). This is why the memory used by the thread's
 stack is not part of the Java heap. On a 32-bit machine, the maximum
 addressable space for an individual java program is 2G. So, for
 example, if you run java 

[Resin-interest] implicit stack memory size?

2009-01-08 Thread Jay Ballinger
Hi everyone,

We're running Resin 3.0.18 Pro on a Windows 32-bit system and we're 
looking to understand/tune our memory settings a bit.

According to http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp 
the stack size is set to 2048k unless set otherwise. Is this true? In 
using JConsole we can't see where any explicit -Xss setting is passed 
and we can't readily determine what the current stack size is set to.

So, here's a short list of questions. And thanks for your help.

- If JConsole doesn't show -Xss, is it really there for resin?
- How was 2048k is an appropriate value for most situations. determined?
- Can resin/jvm/jconsole report the current stack memory setting?

(general sun jvm on windows questions)
- What is the default stack memory size?
- When a thread is created, does it grab all of the stack memory, or can 
the stack memory start small and grow to the default or -Xss size?


Thanks, everyone.

+ jay



___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest


Re: [Resin-interest] implicit stack memory size?

2009-01-08 Thread Jay Ballinger
Knut,

Thanks for the info.

Just a check on the math - if Win32 has a 2GB max addressable space, 
that includes heap and stack and misc. overhead, right?

If we set resin heap to 1GB and allow each thread to use the 'default' 
of 2048k, and we see the number of threads running at about 700, then we 
have the potential of 2.4GB total memory usage [ 1024MB + (700*2MB) ]. 
This exceeds the 2GB limit. However, if, as you suggest, each thread 
only uses a fraction of the 2MB available to each thread, we could see 
the total memory somewhere between 1GB and 2GB.

So, my question is if we use -Xss1024k, each thread is really using 
between 0 and 1024k depending on what that thread really needs?


+ jay


Knut Forkalsrud wrote:
 On Thu, Jan 8, 2009 at 1:52 PM, Jay Ballinger jay.ballin...@gmail.com wrote:
 - When a thread is created, does it grab all of the stack memory, or can
 the stack memory start small and grow to the default or -Xss size?
 
 If I'm not mistaken the memory is allocated, but it is virtual memory,
 so unless it is actually used there is no penalty.
 
 The only problem with allocating too much stack space is that you can
 run out of address space on a 32 bit JVM.
 
 With typically 2GB max addressable for the app and 10 MBytes per stack
 you are limited to 200 threads.
 
 -Knut
 
 
 ___
 resin-interest mailing list
 resin-interest@caucho.com
 http://maillist.caucho.com/mailman/listinfo/resin-interest


___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest


Re: [Resin-interest] implicit stack memory size?

2009-01-08 Thread Rob Lockstone
On Jan 8, 2009, at 13:52, Jay Ballinger wrote:

 Hi everyone,

 We're running Resin 3.0.18 Pro on a Windows 32-bit system and we're  
 looking to understand/tune our memory settings a bit.

 According to http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp 
  the stack size is set to 2048k unless set otherwise. Is this true?  
 In using JConsole we can't see where any explicit -Xss setting is  
 passed and we can't readily determine what the current stack size is  
 set to.

I think Caucho tends to err on the side of caution in this regard, and  
is also (understandably) more *nix-central than Windows. So take that  
2048K as being appropriate for most situations with a grain of salt  
because situations can vary wildly.

 So, here's a short list of questions. And thanks for your help.

 - If JConsole doesn't show -Xss, is it really there for resin?

Whether it's explicitly set or not, there IS a stack size, which would  
end up being the default stack size of the operating system if the JVM  
itself doesn't set it (I do not think it does, but could be wrong and  
don't feel like looking it up now).

 - How was 2048k is an appropriate value for most situations.  
 determined?

See above.

 - Can resin/jvm/jconsole report the current stack memory setting?

I don't know if there's a way to query a running JVM for this  
information. It's best if you explicitly set the stack size using the  
JVM option -Xss (but see below).

 (general sun jvm on windows questions)
 - What is the default stack memory size?

That's too general of a question. It depends on the OS, and (perhaps)  
the version of the OS.

 - When a thread is created, does it grab all of the stack memory, or  
 can the stack memory start small and grow to the default or -Xss size?

Again, might depend on the operating system. Contrary to what Knut  
said, and to the best of my knowledge and in the testing that I did  
(it's been a while), all of the memory for the stack is pre-allocated  
(at least from the perspective of the JVM itself). But it's easy  
enough to write a program to check this.

Thread stacks, at least on Windows, are more complicated than just a  
single memory allocation (down at the Windows API level) because  
you've got both reserved and committed memory allocations, and one  
always has to be = the other (I forget which), and there's a 64K  
granularity in sizing, and you're dealing with physical and virtual  
memory, and this is all set in the linker and therefore configurable,  
but can also be set programmatically on a per-thread basis. In other  
words, it's quite involved. See this page as an example of how complex  
this kind of memory management is when you go into how the underlying  
OS is actually managing all of this 
http://msdn.microsoft.com/en-us/library/ms686774(VS.85).aspx 
 .

Remember, thread allocation/destruction as well as context switching  
and just about everything else is managed by the operating system, not  
Java (unless you're running green threads, which I don't think is  
even around anymore?). This is why the memory used by the thread's  
stack is not part of the Java heap. On a 32-bit machine, the maximum  
addressable space for an individual java program is 2G. So, for  
example, if you run java with these parameters on a 32-bit machine:

java -server -Xms1500M -Xmx1500M -Xss1M

then you'll leave enough room to run ~548 threads (2048M - 1500M =  
548M / 1M stack = 548 threads). Of course, that assumes that none of  
the 548M of non-java-heap memory is being used for anything else,  
which is a false assumption, perhaps hugely false  depending on what  
your java program is doing, e.g. graphics operations or the like.

Something I discovered while optimizing our Resin on Windows  
environments is that the version of Windows and Java we use (64-bit  
Windows 2003 Server with Java 1.5) will allow a minimum stack size of  
128K, and if you try to specify a stack size of 256K or 512K, Windows  
(or maybe the JVM, I don't know which one for sure) automatically  
jumps you up to 1M. So you have a choice between 128K, 1M, 2M, ??? (I  
didn't test above 2M).

Fwiw, we've got dozens of Windows servers running Resin Pro 3.0.x  
(upgrading to 3.1.x in the not-too-distant future), and we run all of  
them with a 128K stack size. In general, unless you're running code  
that's highly recursive, you don't need very large stack sizes in Java  
programming.

Rob



___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest


Re: [Resin-interest] implicit stack memory size?

2009-01-08 Thread Serge Knystautas
Rob Lockstone wrote:
 (general sun jvm on windows questions)
 - What is the default stack memory size?
 
 That's too general of a question. It depends on the OS, and (perhaps)  
 the version of the OS.

I did this research a while ago, but seems relevant:

   Default in Sun JDK 1.2 on unix was 128kb
   Default in Sun JDK 1.3 on unix was 256kb
   Default in Sun JDK 1.4 on windows is 512kb
   Default in IBM JDK 1.5 is 256kb on all platforms 
(http://tinyurl.com/gbfd4)
   Default in JRockit JDK 1.5 is 64k (win) or 128kb (linux) 
(http://tinyurl.com/2hevdw)
   The old volano chat scalability reports used 32kb

-- 
Serge Knystautas
Lokitech  software . strategy . design  http://www.lokitech.com
p. 301.656.5501
e. ser...@lokitech.com


___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest


Re: [Resin-interest] implicit stack memory size?

2009-01-08 Thread Rob Lockstone
As you can see, it's all over the map, which is probably why: (1)  
Caucho sets it explicitly (and to a very conservative value); (2) it's  
always best to set the stack size yourself so you know what you're  
getting; and (3) test it yourself with the value(s) you'd like to use  
in the environments you want to use it to make sure it works (see my  
previous comment about only 128K and 1M working when I did my testing).

Rob

On Jan 8, 2009, at 17:28, Serge Knystautas wrote:

 Rob Lockstone wrote:
 (general sun jvm on windows questions)
 - What is the default stack memory size?

 That's too general of a question. It depends on the OS, and (perhaps)
 the version of the OS.

 I did this research a while ago, but seems relevant:

   Default in Sun JDK 1.2 on unix was 128kb
   Default in Sun JDK 1.3 on unix was 256kb
   Default in Sun JDK 1.4 on windows is 512kb
   Default in IBM JDK 1.5 is 256kb on all platforms
 (http://tinyurl.com/gbfd4)
   Default in JRockit JDK 1.5 is 64k (win) or 128kb (linux)
 (http://tinyurl.com/2hevdw)
   The old volano chat scalability reports used 32kb

 -- 
 Serge Knystautas
 Lokitech  software . strategy . design  http://www.lokitech.com
 p. 301.656.5501
 e. ser...@lokitech.com




___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest


Re: [Resin-interest] implicit stack memory size?

2009-01-08 Thread Scott Ferguson

On Jan 8, 2009, at 5:23 PM, Rob Lockstone wrote:

 On Jan 8, 2009, at 13:52, Jay Ballinger wrote:

 Hi everyone,

 We're running Resin 3.0.18 Pro on a Windows 32-bit system and we're
 looking to understand/tune our memory settings a bit.

 According to http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
 the stack size is set to 2048k unless set otherwise. Is this true?
 In using JConsole we can't see where any explicit -Xss setting is
 passed and we can't readily determine what the current stack size is
 set to.

Try -verbose when starting.  Since the -Xss is a command line option,  
it should be reported.  Also, JConsole should be able to see the  
command line options on one of the system beans.

 I think Caucho tends to err on the side of caution in this regard, and
 is also (understandably) more *nix-central than Windows. So take that
 2048K as being appropriate for most situations with a grain of salt
 because situations can vary wildly.

Basically.  There's no way for us to know if your application needs  
more stack or more threads.  So I assumed 1G heap, leaving 1G for  
other stuff, and then figured the default should support 1000 threads,  
giving 1m as a default, which should be enough stack unless you have  
massively recursive applications, and enough threads for most apps.   
You know your own requirements best, of course.

I believe Linux is the most finicky about the stack allocation, so it  
was the OS we needed to solve (and it was the OS which exposed the  
issue first.)

I'm really looking forward for the 64-bit transition to complete so we  
can use as much VM as possible.

-- Scott




 So, here's a short list of questions. And thanks for your help.

 - If JConsole doesn't show -Xss, is it really there for resin?

 Whether it's explicitly set or not, there IS a stack size, which would
 end up being the default stack size of the operating system if the JVM
 itself doesn't set it (I do not think it does, but could be wrong and
 don't feel like looking it up now).

 - How was 2048k is an appropriate value for most situations.
 determined?

 See above.

 - Can resin/jvm/jconsole report the current stack memory setting?

 I don't know if there's a way to query a running JVM for this
 information. It's best if you explicitly set the stack size using the
 JVM option -Xss (but see below).

 (general sun jvm on windows questions)
 - What is the default stack memory size?

 That's too general of a question. It depends on the OS, and (perhaps)
 the version of the OS.

 - When a thread is created, does it grab all of the stack memory, or
 can the stack memory start small and grow to the default or -Xss  
 size?

 Again, might depend on the operating system. Contrary to what Knut
 said, and to the best of my knowledge and in the testing that I did
 (it's been a while), all of the memory for the stack is pre-allocated
 (at least from the perspective of the JVM itself). But it's easy
 enough to write a program to check this.

 Thread stacks, at least on Windows, are more complicated than just a
 single memory allocation (down at the Windows API level) because
 you've got both reserved and committed memory allocations, and one
 always has to be = the other (I forget which), and there's a 64K
 granularity in sizing, and you're dealing with physical and virtual
 memory, and this is all set in the linker and therefore configurable,
 but can also be set programmatically on a per-thread basis. In other
 words, it's quite involved. See this page as an example of how complex
 this kind of memory management is when you go into how the underlying
 OS is actually managing all of this 
 http://msdn.microsoft.com/en-us/library/ms686774(VS.85).aspx
 .

 Remember, thread allocation/destruction as well as context switching
 and just about everything else is managed by the operating system, not
 Java (unless you're running green threads, which I don't think is
 even around anymore?). This is why the memory used by the thread's
 stack is not part of the Java heap. On a 32-bit machine, the maximum
 addressable space for an individual java program is 2G. So, for
 example, if you run java with these parameters on a 32-bit machine:

 java -server -Xms1500M -Xmx1500M -Xss1M

 then you'll leave enough room to run ~548 threads (2048M - 1500M =
 548M / 1M stack = 548 threads). Of course, that assumes that none of
 the 548M of non-java-heap memory is being used for anything else,
 which is a false assumption, perhaps hugely false  depending on what
 your java program is doing, e.g. graphics operations or the like.

 Something I discovered while optimizing our Resin on Windows
 environments is that the version of Windows and Java we use (64-bit
 Windows 2003 Server with Java 1.5) will allow a minimum stack size of
 128K, and if you try to specify a stack size of 256K or 512K, Windows
 (or maybe the JVM, I don't know which one for sure) automatically
 jumps you up to 1M. So you have a choice between 128K, 1M, 2M, ???