Peter, The DGC Threads are in the server JVM. They are nearly all for the exported Futures.
Thanks, Bryan > -----Original Message----- > From: Peter Jones [mailto:p...@roundroom.net] > Sent: Friday, January 13, 2012 12:04 AM > To: u...@river.apache.org > Cc: dev@river.apache.org > Subject: Re: DGC threads issue > > Bryan, > > DGC threads should not be per exported object. Generally > speaking, they tend to be per endpoint (at which there are > one or more remote objects exported). Are you using any sort > of custom endpoint implementation? Problems like this can > occur when an endpoint implementation doesn't implement > Object.equals and hashCode appropriately, so the expected > batching of threads (and communication) per endpoint does not occur. > > It might help to see, from a thread dump, exactly which DGC > threads are causing this problem. And they are in the server > JVM (with all the exported remote objects), not the remote > callers' JVM(s)? > > -- Peter > > > On Jan 12, 2012, at 3:45 PM, Tom Hobbs wrote: > > > Hi Bryan, > > > > Sorry that no one got back to you about this. I'm afraid > that I don't > > know the answer to your question, I've copied the dev list > into this > > email in case someone who monitors that list (but not this one) has > > any ideas. > > > > Best regards, > > > > Tom > > > > On Thu, Jan 12, 2012 at 2:29 PM, Bryan Thompson > <br...@systap.com> wrote: > >> Just to follow up on this thread myself. I modified the > pattern to return a "thick" future rather than a proxy for > the future. This caused the RMI call to wait on the server > until the future was done and then sent back the outcome. > This "fixed" the DGC memory/thread leak by reducing the > number of exported proxies drammatically. > >> > >> In terms of best practices, is distributed DGC simply not > useful for exported objects with short life spans? Can it > only be used with proxies for relatively long lived services? > >> > >> Thanks, > >> Bryan > >> > >>> -----Original Message----- > >>> From: Bryan Thompson > >>> Sent: Tuesday, January 03, 2012 12:06 PM > >>> To: u...@river.apache.org > >>> Subject: DGC threads issue > >>> > >>> Hello, > >>> > >>> Background: > >>> > >>> I am seeing what would appear to be one DGC thread allocated per > >>> exported object. This is using River 2.2 and Sun JDK 1.6.0_17. > >>> Relevant configuration parameters are below. > >>> > >>> I am observing problems with the DGC threads not being > retired on a > >>> timely basis. The exported objects are proxies for Futures which > >>> are being executed on the service. The code pattern is such that > >>> the proxied Future goes out of lexical scope quite > quickly. E.g., > >>> rmiCallReturningProxyForFuture().get(). > >>> > >>> Under a modest load, a large number of such Futures are exported > >>> which results in a large number of long lived DGC threads. This > >>> turns into a problem for the JVM due to the stack allocation per > >>> thread. Presumably this is not good for other reasons as well > >>> (e.g., scheduling). > >>> > >>> I have tried to override the leaseValue and checkInterval > defaults > >>> per the configuration options below. I suspect that the lease > >>> interval is somehow not being obeyed, which is presumably > a problem > >>> on my end. However, I can verify that the configuration > values are > >>> in fact showing up in > >>> System.getProperties() for at least some of the JVMs > involved (the > >>> one which drives the workload and the one that I am > monitoring with > >>> the large number of DGC lease threads). > >>> > >>> Some questions: > >>> > >>> Is this one-thread-per-exported proxy the expected > behavior when DGC > >>> is requested for the exported object? > >>> > >>> The DGC lease checker threads appear to expire ~14 - 15 minutes > >>> after I terminate the process which was originating the RMI > >>> requests. This is close the sum of the default > leaseValue (10m) and > >>> checkInterval (5m) parameters, but maybe there is some > other timeout > >>> which is controlling this? If this is the sum of those > parameters, > >>> why would the DGC lease threads live until the sum of > those values? > >>> I thought that the lease would expire after the leaseValue (10m > >>> default). > >>> > >>> Can the issue I am observing be caused by a low heap > pressure on the > >>> JVM to which the RMI proxies were exported? If it fails > to GC those > >>> proxies, even though they are reachable, could that cause DGC to > >>> continue to retain those proxies on the JVM which exported them? > >>> > >>> Is there any way to configure DGC to use a thread pool or to have > >>> the leases managed by a single thread? > >>> > >>> Is it possible that there is an interaction with the > useNIO option? > >>> > >>> Relevant options that I am using include: > >>> > >>> -Dcom.sun.jini.jeri.tcp.useNIO=true > >>> -Djava.rmi.dgc.leaseValue=30000 > >>> -Dsun.rmi.dgc.checkInterval=15000 > >>> -Dsun.rmi.transport.tcp.connectionPool=true > >>> > >>> Thanks in advance, > >>> Bryan > >
