Thanks for the replies. This post is moderately long as it includes
responses
to most of the replies.
Before I reply to replies, I will quickly describe some add'l data
that I obtained from further tests. I was lead to a line of
investigation by the description at the following URL:
http://www.coderanch.com/t/419739/Linux-UNIX/JVM-Memory-Linux
I executed a simpler test in Java only, one that prints a simple
"Sleeping 30sec" message then sleeps for 30sec. The 30sec delay is so
that I can record the memory allocated by JVM.
* on a Windows XP laptop a 2.2GHz dual-core CPU and 2GB RAM
* on several Linux v2.6 systems w/ 8 cores of ~2.6GHz and 16GB RAM
* on several Linux v2.4 systems w/ the same configs as those w/
the v2.6 kernels
All tests were executed w/ a v6 JVM.
NOTE - Linux version numbers refer to the kernel's version.
In short, only the linux v2.6 kernels showed the exorbitant
memory-utilization that I reported previously. On both Windows XP and
Linux v2.4, the memory allocated was on the order of MBs, not
HUNDREDS of MBs.
NOTE - in practice, I consider the 10, 11 and 12MBs that I observed to
be on the order of MBs, not TENs of MBs.
One would guess then that the overcommit feature of the Linux v2.6
kernel is causing this impression of exorbitant memory-utilization.
The question remains then whether the overcommit affects the
scalability of the JVM on Linux v2.6 systems. IOW, for a simple
program that causes the JVM to "allocate" hundreds of megs of RAM on a
multi-gigabyte system, will the number of JVM instances be severely
limited? For example, if the JVM allocates 400MB on a 16GB system, is
the total number of JVM instances limited to ~40 (16000/400 or to the
~1300 (16000/12) that I would otherwise expect. I can explore that
indepedently and share the results here later if anyone is interested.
---> Richard Newman wrote:
rlehr> Simply stated, given that it stated clearly in this group that
Clojure
rlehr> runs, on average, slower than the JVM, is the JVM's high
memory-
rlehr> utilization a non-issue for the Clojure community as well?
>
> In short: yes.
>
First, thanks for the info, Richard.
Interesting. I am now left to wonder if your disposition is typical
of most JVM
developers and deployers. I'll have to continue asking my inquiry.
> Sure, under load that will increase, but it's still negligible in the
> context of available memory (only a few percent). Additionally, in my
> experience the memory "profile" of Java server applications is better
> than our C++ applications, which typically experience some bloating or
> memory leak issues.
Is that a positive for Java? Or a negative for the programmers that
wrote your C++ apps?
> I'd take steady-state 300MB memory consumption any day over a
> startup at 50MB ending up at 1.4GB after a couple of months' uptime
> (a situation I've seen with C++ server applications).
In general, I agree. But I would need to take into account few other
variables before I made that decision for a specific application.
Consider, for an app that starts at 50MB yet has allocated 1.4GB of
"leaked memory" after a couple months, that application could easily
restart itself very quickly; and that's a low-cost operation. I know
it seems like a hack. But
> So, in short: if you're on a low-resource machine, running graphical
> Java clients: yes, the JVM's memory consumption might concern you,
> as will its startup time.
Errrm - so Java's value is limited to large systems ? That statement
does not seem to take into account all of the small systems, including
mobile devices, that use Java. What am I missing here?
> You're probably baffled because your definition of "unacceptable" is
> very different to most using Java SE/EE. The JVM is monolithic
> (something Project Jigsaw is trying to fix, precisely to reduce
> memory usage in constrained scenarios), ...
Good point. However, my definition of "acceptable" is reasonable in
the most general sense. I am familiar w/ cost-to-benefit analysis of
the choices that designers make. For any benefit that a system
provides, there is a limit on the cost that one is willing to pay.
When a comparable C++ progie allocates 40k, I do not consider an
initial allocation of 400MB to be acceptable for any system. It's
wasteful enough to begin reconsidering whether the benefits of the
system in question are worth the cost.
IOW, when I see a increase of 4 orders of magnitude of attribute A in
exchange for 1 order of magnitude of improvement in attribute B, my
alarms are triggered. I do my value calculations, start asking
questions and looking for something to kill. In this case, the JVM
allocating 400MB instead of 40k; I expected something on the order of
MBs, not HUNDREDs of MBs.
NOTE that the above comment is a general statement and does not
reflect my observations of the v6 JVM on Windows XP and Linux v2.4.x.
> ...but it doesn't bother most people running Java services on
> 2008/2009-era servers.
If my initial observations were true for all cases then I suspect that
your comment would reflect poorly on your group of "most people".
Sorry. :(
---> Chris Dean wrote:
> For my part, I'm relatively happy with the jvm's performance. I
> agree that the memory is usage is (in general) higher that I would
> like, but the performance of the system as a whole has been good.
Thanks.
> Finding a system with the best cpu-performance, memory use,
> libraries, community, etc is challenging and I for one am happy with
> the tradeoffs that Clojure has made.
Telling it like it is. :) The tricky business of cost-to-benefit...
---> Jonah Benton wrote
> ... there are few useful conclusions that one can draw from a JVM's
> RSS and VIRT.
True. Except that I can compare these metrics to values obtained for
corresponding programs in other languages and on similar OSes, i.e.,
linux vs linux, Windows vs Windows. A metric may not be accurate.
But if it is applied consistently then it can be used for comparisons.
> It's unfortunate, but in general, Java app memory behavior can't be
> reliably reported on by standard unix tools.
See preceding comment.
> Further, for non-trivial applications, GC parameters often need to
> be custom-tuned to avoid pathological behavior.
Right.
> In its defense, the knobs the JVM makes available are much more
> granular than knobs one typically has available at configuration or
> run time in C++.
Or any other language or "platform", for that matter. That is an
impressive feature of the JVM.
> That said, in my experience, GC configuration is a big problem for
> typical Java users, and it contributes to ongoing concerns about
> Java. Every few weeks it seems another colleague reports service
> outages or frequent restarts with an app, usually a web app. These
> problems are usually fixable with a few changes to GC
> configuration. But few Java developers have familiarized themselves
> with the JVM's behavior at that level, and few operations folks,
> even those who have skills in troubleshooting application problems,
> are familiar in particular with JVM tuning.
Interesting observation - noted.
> In any event, try running your JVM with
>
> -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails
> -XX:+PrintHeapAtGC -Xloggc:/tmp/gc.out
>
> ...
OK - thanks. I'll try those.
---> Boris Mizhen wrote:
> Try also visualvm (comes with jdk 1.6 )
> https://visualvm.dev.java.net/
OK - thanks.
---> kyle smith wrote:
> > I am dealing w/ some performance constraints in the current
> > implementation of my application.
>
> More information, please. What in general is your app? What are the
> specs of your machine? What kind of data does your app deal with?
>
> > So, although Clojure offers some
> > features that would solve some of my problems, I am concerned about
> > Clojure's and the JVM's performance issues. I read the "when
> > performance matters" thread and, as usual, was struck by the usual
> > omission of memory-utilization from the discussion of the JVM's
> > performance characteristics. This has always puzzled me about the
> > Java world and the C++ vs Java debate in general.
>
> The reason it's ignored is that it usually only matters in micro-
> benchmarks. (and because ram is dirt cheap)
I do not consider a net change of 4 orders of magnitude for a
brain-dead simple progie to be a "micro-benchmark".
And cheap hardware is a cheap excuse for waste. Costs need to be
balanced. Systems have finite max-RAM capacities. So more RAM
eventually equates to more systems and higher TCO due to greater
expense on HW, data center space, cooling, electricity, etc., and
man-hours for maintenance.
Regardless, as I noted above, I observed those originally reported
exorbitant numbers only on linux v2.6 kernels.
> Probably >99% of people spout this off without any hard numbers.
> You obviously have numbers, but you should be comparing apps to apps,
> not benchmarks to benchmarks.
I am and a very simple app, too. Just start the JVM w/ a simple
println and sleep.
> For example, depending on the problem domain, it may be very possible
> to 'cheat'.
If I have to "cheat" too much then I eventually lose the value that
the JVM purportedly avails to us. So that is no panacea.
---> Jarkko Oranen wrote:
> I doubt the VIRT size matters at all. From what I know, it
> represents the address space that is available to the process...
No.
> You can try tuning the java VM and decrease its heap size and other
> things; see if it makes a difference.
Funny thing about that. I tried to limit the heap-size w/ the "-Xmx"
and "-Xms" CLI parameters. They exacerbated the situation, i.e., the
JVM allocated MORE memory. Go figure.
-robert
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
-~----------~----~----~----~------~----~------~--~---
