26 jan 2010 kl. 16.55 skrev Emmanuel Lcharny:
Dan Lawesson a écrit :
Hi,
Until yesterday I was under the impression that heap buffers are
the way to go and direct buffers are more of a relic of at most
historical value. This was a belief that was at least partly
supported by the Apache MINA project documentation of the upcoming
version 2.0. Yesterday my view on heap buffers was completely
reversed when I witnessed a perfectly working legacy system (Java
1.4.2 something) barf and die due to lack of direct memory even
though we did not explicitly use any direct heap buffers at all.
Post mortem heap dumps show that there had been a creation of 100
direct buffers per second and none of them were cleaned since heap
memory was fine and no full GC sweep was triggered. The heap
contained a chain of 200k cleaners after 30 minutes of uptime. The
reason I have not witnessed this before is probably that usually
these systems also use a lot of heap memory and therefore trigger
full GC which triggers the cleaners.
After a while we realized that in case you use heap buffers, you
get a temporary direct buffer. Clearly, at least in this old JVM
these temporary direct buffers are not even pooled (that was the
WTF of the day!).
This lead me to DIRMINA-391 http://issues.apache.org/jira/browse/DIRMINA-391
. Trustin Lee advocates heap buffers, even though the issue
presents a strong case against them. Does this mean that this
behavior of using non-pooled temporary direct buffers is a thing of
the past, i.e., more modern JVMs are smarter than that? This is the
only explanation I can find to why MINA 2.0 uses heap buffers per
default and also has dropped the pooling feature of older MINA
versions.
The reason I write this mail is that both https://issues.apache.org/jira/browse/DIRMINA-576
and the linked Sun issue seem to contradict this. They both make a
strong case for always using pooled direct buffers. What is the
community consensus on heap buffers?
Concensus, I don't know. My strong opinion is that direct buffers
are to be used when dealing with low level IO, and heap buffers must
be used for everything else.
The fact that you can kill your OS because you eat all of it's
memory seems to me a big no-no for DirectBuffer. If I use Heap
buffers and if I suck up all my JVM memory, too bad for me, my code
and my salary, but at least, it impacts only my application's JVM.
In your case, the Direct buffers are used to exchange data with a
socket in a IOread, done by the JVM : makes perfect sense. The
question is why aren't they cleaned ?
I have to dig back into those issues I guess...
Thank you very much for your reply. It is now clear that I had not
researched the issue thoroughly enough before asking the question. The
JVM does pool the direct buffers, very much to the contrary of my ill-
informed statement above.
I was lead to think that there was no pooling because there was
approximately 100 new allocations per second in the system I analyzed.
For some reason it did not occur to me that the clever and efficient
way to implement the pooling is to do it in thread local pools, thus
eliminating any need for synchronization. Comments on the relevant Sun
bug issues and DIRMINA-576 seemed to further reinforce my suspicion
that something was actually wrong in the JVM handling of the buffers.
My mistake was to not understand that the pooling is thread local and
that I focused my attention on the protocol handling low level module
of the system, failing to realize that a business logic part of the
system called write operations on the protocol module in new (as in
"new Thread()") threads due to a peculiarity of the implementation of
a timer callback.
Now I believe that I can safely return to my previous belief that heap
buffers are to be preferred in essentially all situations (in line
with the statements of Trustin Lee in DIRMINA-391) as long I make sure
that the total number of threads involved in read() and write() are
limited (by for example pooling of threads). For the problem at hand
it was not feasible to redesign the thread model for the business
logic module, so we settled for explicit (non-thread local) pooling of
direct buffers used for the write operation.
