On 12/1/06, Daniel Daley <[EMAIL PROTECTED]> wrote:
Hmm well up until last night I thought this was the case. Normally once we reached the 4600 range the server would be reporting 130+ connected clients when there was no way that could be true. Last night when I checked the total connections had reached over 5000 but the connected users was closer to 30. I wonder if it might not just be chance and has to do with how many connections hang on the server.
Ok, so please continue with the experiment if convenient and more results are appreciated. Thanks. --Dan--
On Nov 29, 2006, at 6:13 PM, Steven Gong wrote: 4600 is an accurate figure? So this can be reproduced easily right? Next thing I'd like to make clear is: does it matter that these connections all come from the same IP or different IPs? On 11/30/06, Daniel Daley <[EMAIL PROTECTED]> wrote: > > Unfortunately it's not really any specific code. We have users that use > the FLVPlayback component as well as just random skins they've created > themselves. Without a doubt though, it happens after around 4600 connections > have been made to the server (I count by using the client ID generated by > red5). These are all VOD connections as well, no live streams. > > Thanks, > > --Dan- > > On Nov 28, 2006, at 6:08 PM, Steven Gong wrote: > > Hi Dan, > Do you have any test client that could reproduce this quickly? That may > help us a lot for figuring out what is wrong. Thanks. :-) > > On 11/29/06, Daniel Daley <[EMAIL PROTECTED]> wrote: > > > > Now that the new FLVReader code is in place I am able to handle many > > more concurrent users but continue having one strange issue. After > > about 4600 connections have occurred the heap suddenly grows to the > > point that a full garbage collect hardly drops it (as seen in the log > > below). > > > > A short bit after this occurs the server will not longer stream but > > will still accept connections. My only thought as to what it might be > > is that I've had trouble with the job scheduling service hanging > > suddenly after a period of running. I had my own scheduled jobs > > during that time so I think it accelerated the process but I know > > that the stop command is scheduled and perhaps if the scheduler is > > locking it would cause this issue? I wouldn't think it would happen > > so quickly though if that was the case, as everything runs perfectly > > beforehand with the heap being garbage collected down to around 60 or > > 70MB. Any ideas? > > > > Thanks, > > > > --Dan-- > > > > 2006-11-28 18:45: 39.973964500 [INFO] 16657086 pool-1-thread-6: > > ( streamSubscriberClose ) 2006-11-28 18:45:39 Client: 4518 > > streamSubscriberClose > > 2006-11-28 18:45:39.985626500 [INFO] 16657098 pool-1-thread-1: > > ( streamPlaylistVODItemSeek ) 2006-11-28 18:45:39 Client: 4549 > > streamPlaylistVODItemSeek > > 2006-11-28 18:45:40.234276500 [Full GC 499584K->486582K(511808K), > > 3.9512268 secs] > > 2006-11-28 18:45:44.386985500 [Full GC 499584K->490826K(511808K), > > 4.0028856 secs] > > 2006-11-28 18:45: 48.544464500 [Full GC 499584K->487926K(511808K), > > 5.3713444 secs] > > 2006-11-28 18:45: 54.095456500 [Full GC 499583K->493405K(511808K), > > 4.0693999 secs] > > 2006-11-28 18:45:58.261174500 [Full GC[Unloading class > > sun.reflect.GeneratedConstructorAccessor4] > > 2006-11-28 18:46:02.380410500 499583K->495486K(511808K), 4.1192612 > > secs] > > 2006-11-28 18:46:02.454677500 [Full GC 499583K->496817K(511808K), > > 4.0788536 secs] > > 2006-11-28 18:46: 06.576321500 [Full GC[Unloading class > > sun.reflect.GeneratedMethodAccessor24] > > 2006-11-28 18:46: 08.452441500 [Unloading class > > sun.reflect.GeneratedMethodAccessor2] > > 2006-11-28 18:46:08.452444500 [Unloading class > > sun.reflect.GeneratedMethodAccessor15] > > 2006-11-28 18:46:08.452485500 [Unloading class > > sun.reflect.GeneratedMethodAccessor14 ] > > 2006-11-28 18:46:08.452739500 [Unloading class > > sun.reflect.GeneratedMethodAccessor12 ] > > 2006-11-28 18:46:08.452801500 [Unloading class > > sun.reflect.GeneratedMethodAccessor20] > > 2006-11-28 18:46:08.452805500 [Unloading class > > sun.reflect.GeneratedMethodAccessor3] > > 2006-11-28 18:46:08.452820500 [Unloading class > > sun.reflect.GeneratedMethodAccessor6] > > 2006-11-28 18:46:08.452836500 [Unloading class > > sun.reflect.GeneratedMethodAccessor5 ] > > 2006-11-28 18:46:08.452850500 [Unloading class > > sun.reflect.GeneratedMethodAccessor22 ] > > 2006-11-28 18:46:08.452890500 [Unloading class > > sun.reflect.GeneratedMethodAccessor11] > > 2006-11-28 18:46:08.453125500 [Unloading class > > sun.reflect.GeneratedMethodAccessor25] > > 2006-11-28 18:46:08.453197500 [Unloading class > > sun.reflect.GeneratedMethodAccessor17] > > 2006-11-28 18:46:08.453232500 [Unloading class > > sun.reflect.GeneratedMethodAccessor13 ] > > 2006-11-28 18:46:08.453314500 [Unloading class > > sun.reflect.GeneratedMethodAccessor4 ] > > 2006-11-28 18:46:12.096960500 499573K->495985K(511808K), 5.5206938 > > secs] > > 2006-11-28 18:46:12.154356500 [Full GC 499584K->498842K(511808K), > > 4.2219567 secs] > > 2006-11-28 18:46:16.387928500 [Full GC 499584K->499102K(511808K), > > 4.1133314 secs] > > 2006-11-28 18:46:20.524919500 [Full GC[Unloading class > > sun.reflect.GeneratedMethodAccessor1] > > 2006-11-28 18:46: 24.684359500 499584K->499222K(511808K), 4.1594666 > > secs] > > 2006-11-28 18:46: 24.706426500 [Full GC 499584K->497661K(511808K), > > 5.6004981 secs] > > 2006-11-28 18:46:30.357607500 [Full GC 499583K->499583K(511808K), > > 4.1456096 secs] > > 2006-11-28 18:46:34.503350500 [Full GC 499583K->498283K(511808K), > > 5.5813694 secs] > > 2006-11-28 18:46:40.125999500 [Full GC 499579K->499579K(511808K), > > 4.1222203 secs] > > 2006-11-28 18:46: 44.248342500 [Full GC 499579K->498764K(511808K), > > 5.5525750 secs] > > 2006-11-28 18:46: 49.836342500 [Full GC 499583K->499583K(511808K), > > 4.2713130 secs] > > 2006-11-28 18:46:54.107789500 [Full GC 499583K->499000K(511808K), > > 5.4783996 secs] > > 2006-11-28 18:46:59.616781500 [Full GC 499571K->499571K(511808K), > > 4.3601245 secs] > > 2006-11-28 18:47:03.977038500 [Full GC 499571K->499231K(511808K), > > 5.6121831 secs] > > 2006-11-28 18:47: 09.598690500 [Full GC 499584K->499584K(511808K), > > 4.1334214 secs] > > 2006-11-28 18:47: 13.732243500 [Full GC 499584K->499344K(511808K), > > 5.4238636 secs] > > 2006-11-28 18:47:19.162038500 [Full GC 499583K->499583K(511808K), > > 4.1761207 secs] > > 2006-11-28 18:47:23.338292500 [Full GC 499583K->499435K(511808K), > > 4.0381495 secs] > > 2006-11-28 18:47:27.397388500 [Full GC 499578K->499578K(511808K), > > 4.1922121 secs] > > 2006-11-28 18:47: 31.589735500 [Full GC 499578K->499504K(511808K), > > 5.5022513 secs] > > 2006-11-28 18:47: 37.095978500 [Full GC 499583K->499583K(511808K), > > 4.1805690 secs] > > 2006-11-28 18:47:41.276684500 [Full GC 499583K->499503K(511808K), > > 4.0814699 secs] > > 2006-11-28 18:47:45.363748500 [Full GC 499583K->499583K(511808K), > > 4.1720694 secs] > > 2006-11-28 18:47:49.535943500 [Full GC 499583K->499544K(511808K), > > 4.1699713 secs] > > 2006-11-28 18:47: 53.755950500 [Full GC 499583K->499583K(511808K), > > 4.1593166 secs] > > 2006-11-28 18:47: 57.917455500 [Full GC 499583K->499552K(511808K), > > 5.5514382 secs] > > 2006-11-28 18:48:03.472669500 [Full GC 499584K->499584K(511808K), > > 4.1725040 secs] > > 2006-11-28 18:48:07.659044500 [Full GC 499584K->499584K(511808K), > > 4.1750055 secs] > > 2006-11-28 18:48:11.847801500 [Full GC 499584K->499584K(511808K), > > 4.1131328 secs] > > 2006-11-28 18:48: 15.979586500 [Full GC 499584K->496903K(511808K), > > 5.6763932 secs] > > 2006-11-28 18:48: 21.657594500 Exception in thread "Callback > > Thread" [ERROR] 16818770 pool-1-thread-7: > > ( org.red5.server.service.ServiceInvoker.error ) Error executing > > call: Service: null Method: deleteStream Num Params: 10: 1 > > 2006-11-28 18:48:21.658716500 java.lang.OutOfMemoryError: Java heap > > space[ERROR] 16818779 pool-1-thread-7: > > ( org.red5.server.service.ServiceInvoker.error ) Service invocation > > error > > 2006-11-28 18:48:21.667577500 > > java.lang.reflect.InvocationTargetException > > 2006-11-28 18:48:21.667580500 at > > sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) > > 2006-11-28 18:48: 21.667582500 at > > sun.reflect.DelegatingMethodAccessorImpl.invoke > > (DelegatingMethodAccessorImpl.java:25) > > 2006-11-28 18:48:21.667585500 at java.lang.reflect.Method.invoke > > (Method.java:585) > > 2006-11-28 18:48: 21.667731500 at > > org.red5.server.service.ServiceInvoker.invoke (ServiceInvoker.java > > :161) > > 2006-11-28 18:48:21.667734500 at > > org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:135) > > 2006-11-28 18:48: 21.667736500 at > > org.red5.server.net.rtmp.RTMPHandler.onInvoke (RTMPHandler.java:276) > > 2006-11-28 18:48:21.667739500 at > > org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived > > (BaseRTMPHandler.java:114) > > 2006-11-28 18:48: 21.667741500 at > > org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived > > (RTMPMinaIoHandler.java:78) > > 2006-11-28 18:48:21.667752500 at > > org.apache.mina.common.support.AbstractIoFilterChain$2.messageReceived > > (AbstractIoFilterChain.java:189) > > 2006-11-28 18:48:21.667756500 at > > > > org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRece > > ived(AbstractIoFilterChain.java:502) > > 2006-11-28 18:48:21.667759500 at > > org.apache.mina.common.support.AbstractIoFilterChain.access$1000 > > (AbstractIoFilterChain.java:52) > > 2006-11-28 18:48:21.667775500 at > > org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl > > $1.messageReceived( AbstractIoFilterChain.java:777) > > 2006-11-28 18:48:21.667778500 at > > org.red5.io.filter.ExecutorFilter.processEvent(ExecutorFilter.java > > :231) > > 2006-11-28 18:48:21.667781500 at org.red5.io.filter.ExecutorFilter > > $ProcessEventsRunnable.run( ExecutorFilter.java:279) > > 2006-11-28 18:48: 21.667784500 at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask > > (ThreadPoolExecutor.java:650) > > 2006-11-28 18:48:21.667792500 at > > java.util.concurrent.ThreadPoolExecutor$Worker.run > > (ThreadPoolExecutor.java :675) > > 2006-11-28 18:48:21.667795500 at java.lang.Thread.run(Thread.java > > :595) > > 2006-11-28 18:48:21.667797500 Caused by: java.lang.OutOfMemoryError: > > Java heap space > > 2006-11-28 18:48: 21.668241500 > > 2006-11-28 18:48: 21.669689500 Exception in thread "btpool0-1" > > java.lang.OutOfMemoryError: Java heap space > > 2006-11-28 18:48:21.671628500 Exception in thread > > "FlowControlService" java.lang.OutOfMemoryError: Java heap space > > > > > > _______________________________________________ > > Red5 mailing list > > [email protected] > > http://osflash.org/mailman/listinfo/red5_osflash.org > > > > > > -- > I cannot tell why this heart languishes in silence. It is for small > needs it never asks, or knows or remembers. -- Tagore > > Best Regards > Steven Gong_______________________________________________ > Red5 mailing list > [email protected] > http://osflash.org/mailman/listinfo/red5_osflash.org > > > > _______________________________________________ > Red5 mailing list > [email protected] > http://osflash.org/mailman/listinfo/red5_osflash.org > > > -- I cannot tell why this heart languishes in silence. It is for small needs it never asks, or knows or remembers. -- Tagore Best Regards Steven Gong_______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org _______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org
-- I cannot tell why this heart languishes in silence. It is for small needs it never asks, or knows or remembers. -- Tagore Best Regards Steven Gong
_______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org
