Thanks for all the advices.
Can we conclude that:
1. The problem is related to a series of problems, as Ivan summarized. It is
about the pattern to release native resources in java.
2. It is related to gc, if we do not intend to design another mechanism to
track object reference. Only when the object cannot be referenced, have we
the right to release the native resource of it. Actually the problem to
release native resources can occur in any object besides direct byte buffer
and zip related objects.
But now we have some design problems:
1. When to trigger gc?
It might be too late when native memory allocation fails. It will cost a
period of time for VM to complete gc and invoke phantom reference, not to
mention other resource releases written in finalize method, as Gregory
mentions.
2. The decision will be made on which level?
System wide, per process, or per VM?
I think it is the easiest to monitor memory resource on system wide
level: Normally os will report its free physical memory and total physical
memory, which we can rely on to deduce a tension in memory allocation.
Per process is a little difficult:how much memory it used is regarded
to exist memory shortage?
Per VM is the most difficult: besides the problem from per process, we
can hardly discriminate the usage of each vm in one process. The normal
c-library just tells me the size we allocate in malloc by no message of the
memory it releases in free.
3. Is it wise to allocate memory in a managed heap? There is some difference
between Xiao Feng and Jimmy's advice:
Xiao Feng considers a way to allocate direct byte buffer on java heap.
It will be great if it works: Then we will have no problem in releasing
native resources.
Jimmy proposes we can give each VM a memory booking. It also works, I
believe. But is it a little too onerous for us to implement a malloc/free
function ourselves?
Here are my own opinion:
Although system wide memory monitor can be influnced by other process, I
prefer it. Base on per process or per vm memory allocation, we can get just
the information about virtual memory allocation. Hardly can we deduce that
memory shortage occurs only from the virtual memory one process or vm uses :
by page exchanging, os is able to provide more physical memory for one
process or another.Native memory allocation failure is the direct result
from shortage of system physical memory : os cannot commit enough physial
memory for the required virtual pages.
Since I am not an expert in the area, excuse me if I miss something.:)
On 2/3/07, LvJimmy，Jing <[EMAIL PROTECTED]> wrote:
2007/2/2, Xiao-Feng Li <[EMAIL PROTECTED]>:
> Thank you, Jimmy and Leo, for the clear explanations.
>
> I think temporarily you can put the System.gc() there to workaround
> this issue. Yes, this is only a workaround since we cannot rely on it
> without explicit guarantee in its semantics.
>
> To really solve this problem, we might need a careful design. Your
> proposed approach looks like a possible solution. Have you thought to
> solve the problem by using GC to manage the raw memory blocks? Without
> deep understanding of the problem, I have some wild idea below. Please
> see if it is reasonable or stupid. I will read the spec of ByteBuffer
> to get more understanding.
>
> Since what the ByeBuffer needs are starting address and capacity, it
> doesn't really care if this piece of memory is in Java heap or in
> native runtime memory ( I assume.) We probably can provide some
> special kind of virtual Java object that serves as raw memory block to
> nio. In this works, we need not monitor the native runtime memory
> usage.
>
Ah, this solution sounds interesting to me, yes, if we malloc memory
in heap then we no longer need to monitor the native memory. But we
may need a kit of methods offer this heap-memory service:
1) a function of malloc_in_heap()
2) a function of free_heap_block()
3) heap monitor should record this malloc/free action
4) ...
However I do think this may be a good solution, and it still depends
on how easy can it be implemented. Note not only direct ByteBuffer
require this native memory usage.
> This approach need certain contract between Java classes and GC about
> the special kind of Java object. Probably we can write a layer of Java
> class wrapper for raw memory allocation, which hides the contract from
> other common classes.
>
As we have a portlib, this can be implemented in such hymem_malloc or so.
> At the same time, I am thinking of your proposed approach, and will
> get back later.
>
Thanks, waiting for you the GC guru's appoach towards heap memory
allocation :)
> Thanks,
> xiaofeng
>
> On 2/2/07, Leo Li <[EMAIL PROTECTED]> wrote:
> > Hi, Xiao-Feng:
> > Excuse me if I am confusing you.
> > The direct byte buffer holds a block of native memory space used
in nio
> > operation. The native byte buffer will be freed through the byte
buffer is
> > gc collected and added to ReferenceQueue monitored by MemorySpy series
> > classes.
> > But the gc will not be triggered if the java heap is still empty.
Here
> > is an example: I allocate a large block of native memory for a byte
buffer
> > while the byte buffer itself is quite small. Thus quickly native heap
is
> > depleted while VM still does not think gc is needed since java heap
has a
> > lot of free memory.
> >
> >
> > import java.nio.*;
> > public class Test {
> > public static void main(String[] args) throws Exception {
> > for(int i = 0;i<1000;i++)
> > {
> > ByteBuffer byteBuffer = ByteBuffer.allocateDirect
(10240000);
> > //System.gc();
> > }
> > }
> > }
> > RI runs well but Harmony will soon throw OutOfMemory exception.
> > But if remove the comment before System.gc(), Harmony will become ok.
> > So my point is that we need a mechanism to notify VM to start gc if we
have
> > no more native memory because sometimes problem will be solved if gc
is
> > fired. Of cause we cannot avoid a user always malloc space but never
free
> > them.:)
> >
> > Good luck!
> >
> > On 2/2/07, Xiao-Feng Li <[EMAIL PROTECTED]> wrote:
> > >
> > > Leo, I can't fully understand your problem and proposal. I am
working
> > > in DRLVM GC component, I want to understand how GC can tackle the
> > > issue you meet.
> > >
> > > You said the native memory is allocated (probably with malloc) for
> > > byte buffer. I wonder how the process goes. I assume it's in native
> > > code. Then you use malloc to allocate a block of memory, and put
some
> > > Java object (byte buffer) into it? Is this what you do? I wonder how
> > > do you manipulate Java object placement in native code; and if the
> > > native memory block is out of Java heap, do you expect GC to manage
> > > it?
> > >
> > > You said when GC reclaims those byte buffer object, the native
memory
> > > block can be freed in native code. How do you know if an object is
> > > reclaimed by GC or not?
> > >
> > > Thanks,
> > > xiaofeng
> > >
> > > On 2/2/07, Leo Li < [EMAIL PROTECTED]> wrote:
> > > > Hi, all:
> > > > After applying patch3073, I added the support for direct byte
buffer
> > > in
> > > > nio module as spec requires. But now here exists a problem: The
native
> > > > memory allocated to the byte buffer can be released when gc
releases the
> > > > byte buffer object, however, if the native heap is full while java
heap
> > > > still has space, gc will not be triggered.
> > > > It seems that RI will start gc before native memory heap is
depleted
> > > and
> > > > thus prevents out-of-memory error.
> > > >
> > > > Then our work focuses on:
> > > > 1. When gc is required.
> > > > 2. Trigger a gc.
> > > > The first one requires that we get support from operating
system,
> > > since
> > > > the memory allocated in the native code, for example by malloc, is
out
> > > of
> > > > the control of java VM.( I have ever thought of counting the used
memory
> > > in
> > > > hymemory_allocate, but the plan fails since hymemory_free will not
> > > report
> > > > how much space is released.)
> > > > The second one needs the help from VM. System.gc() is not so
reliable
> > > as
> > > > spec says, so it is necessary to have a internal channel to notify
VM to
> > > > start gc.
> > > >
> > > > One solution, I think, is to let a monitor thread in VM to
check
> > > whether
> > > > OS physical memory is low. For example, the
> > > QueryMemoryResourceNotification
> > > > of win32 API is a candidate. Although the interrupt model is more
> > > effective:
> > > > win32 SDK provides a CreateMemoryResourceNotification to get a
handler
> > > on
> > > > which the monitor thread can wait, maybe on other platforms, OS
does not
> > > > supply such a convenience. So the monitor thread in the VM might
have to
> > >
> > > > check the OS resource once for a while and if necessary the
monitor
> > > thread
> > > > will call a GC.
> > > >
> > > > My suggestion is first to add some function to monitor memory
in
> > > portlib,
> > > > since it is highly related to platforms and in portlib there has
been
> > > > some useful tools.(Thanks Mark to point out that.) On the other
hand, we
> > > can
> > > > negotiate a channel to trigger gc in VM. Actually I am not an
expert on
> > > VM
> > > > since I am not sure whether there has been some monitor thread and
the
> > > load
> > > > on performance if such a monitor thread is added to the VM...
> > > >
> > > >
> > > > --
> > > > Leo Li
> > > > China Software Development Lab, IBM
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Leo Li
> > China Software Development Lab, IBM
> >
> >
>
--
Best Regards!
Jimmy, Jing Lv
China Software Development Lab, IBM
--
Leo Li
China Software Development Lab, IBM
