On 12/1/06, Daniel Daley <[EMAIL PROTECTED]> wrote:
I will continue to research what I can. I'm hoping over the next couple of days to find more. Are you thinking you may have some code out to combat the hanging connections any time soon? Maybe I should wait to try that if it will be soon rather than testing a possible known issue.
We are currently investigating it and your test will help us a lot in finding out the root cause. :-) Thanks for all your great work!
--Dan-- On Nov 30, 2006, at 5:42 PM, Steven Gong wrote: On 12/1/06, Daniel Daley <[EMAIL PROTECTED]> wrote: > > Hmm well up until last night I thought this was the case. Normally once > we reached the 4600 range the server would be reporting 130+ connected > clients when there was no way that could be true. Last night when I checked > the total connections had reached over 5000 but the connected users was > closer to 30. I wonder if it might not just be chance and has to do with how > many connections hang on the server. > Ok, so please continue with the experiment if convenient and more results are appreciated. Thanks. --Dan-- > > On Nov 29, 2006, at 6:13 PM, Steven Gong wrote: > > 4600 is an accurate figure? So this can be reproduced easily right? > > Next thing I'd like to make clear is: does it matter that these > connections all come from the same IP or different IPs? > > On 11/30/06, Daniel Daley <[EMAIL PROTECTED]> wrote: > > > > Unfortunately it's not really any specific code. We have users that > > use the FLVPlayback component as well as just random skins they've created > > themselves. Without a doubt though, it happens after around 4600 connections > > have been made to the server (I count by using the client ID generated by > > red5). These are all VOD connections as well, no live streams. > > > > Thanks, > > > > --Dan- > > > > On Nov 28, 2006, at 6:08 PM, Steven Gong wrote: > > > > Hi Dan, > > Do you have any test client that could reproduce this quickly? That > > may help us a lot for figuring out what is wrong. Thanks. :-) > > > > On 11/29/06, Daniel Daley <[EMAIL PROTECTED] > wrote: > > > > > > Now that the new FLVReader code is in place I am able to handle many > > > > > > more concurrent users but continue having one strange issue. After > > > about 4600 connections have occurred the heap suddenly grows to the > > > point that a full garbage collect hardly drops it (as seen in the > > > log > > > below). > > > > > > A short bit after this occurs the server will not longer stream but > > > will still accept connections. My only thought as to what it might > > > be > > > is that I've had trouble with the job scheduling service hanging > > > suddenly after a period of running. I had my own scheduled jobs > > > during that time so I think it accelerated the process but I know > > > that the stop command is scheduled and perhaps if the scheduler is > > > locking it would cause this issue? I wouldn't think it would happen > > > so quickly though if that was the case, as everything runs perfectly > > > > > > beforehand with the heap being garbage collected down to around 60 > > > or > > > 70MB. Any ideas? > > > > > > Thanks, > > > > > > --Dan-- > > > > > > 2006-11-28 18:45: 39.973964500 [INFO] 16657086 pool-1-thread-6: > > > ( streamSubscriberClose ) 2006-11-28 18:45:39 Client: 4518 > > > streamSubscriberClose > > > 2006-11-28 18:45:39.985626500 [INFO] 16657098 pool-1-thread-1: > > > ( streamPlaylistVODItemSeek ) 2006-11-28 18:45:39 Client: 4549 > > > streamPlaylistVODItemSeek > > > 2006-11-28 18:45:40.234276500 [Full GC 499584K->486582K(511808K), > > > 3.9512268 secs] > > > 2006-11-28 18:45:44.386985500 [Full GC 499584K->490826K(511808K), > > > 4.0028856 secs] > > > 2006-11-28 18:45: 48.544464500 [Full GC 499584K->487926K(511808K), > > > 5.3713444 secs] > > > 2006-11-28 18:45: 54.095456500 [Full GC 499583K->493405K(511808K), > > > 4.0693999 secs] > > > 2006-11-28 18:45:58.261174500 [Full GC[Unloading class > > > sun.reflect.GeneratedConstructorAccessor4] > > > 2006-11-28 18:46:02.380410500 499583K->495486K(511808K), 4.1192612 > > > secs] > > > 2006-11-28 18:46:02.454677500 [Full GC 499583K->496817K(511808K), > > > 4.0788536 secs] > > > 2006-11-28 18:46: 06.576321500 [Full GC[Unloading class > > > sun.reflect.GeneratedMethodAccessor24] > > > 2006-11-28 18:46: 08.452441500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor2] > > > 2006-11-28 18:46:08.452444500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor15] > > > 2006-11-28 18:46:08.452485500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor14 ] > > > 2006-11-28 18:46:08.452739500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor12 ] > > > 2006-11-28 18:46:08.452801500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor20] > > > 2006-11-28 18:46:08.452805500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor3] > > > 2006-11-28 18:46:08.452820500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor6] > > > 2006-11-28 18:46:08.452836500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor5 ] > > > 2006-11-28 18:46:08.452850500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor22 ] > > > 2006-11-28 18:46:08.452890500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor11] > > > 2006-11-28 18:46:08.453125500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor25] > > > 2006-11-28 18:46:08.453197500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor17] > > > 2006-11-28 18:46:08.453232500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor13 ] > > > 2006-11-28 18:46:08.453314500 [Unloading class > > > sun.reflect.GeneratedMethodAccessor4 ] > > > 2006-11-28 18:46:12.096960500 499573K->495985K(511808K), 5.5206938 > > > secs] > > > 2006-11-28 18:46:12.154356500 [Full GC 499584K->498842K(511808K), > > > 4.2219567 secs] > > > 2006-11-28 18:46:16.387928500 [Full GC 499584K->499102K(511808K), > > > 4.1133314 secs] > > > 2006-11-28 18:46:20.524919500 [Full GC[Unloading class > > > sun.reflect.GeneratedMethodAccessor1] > > > 2006-11-28 18:46: 24.684359500 499584K->499222K(511808K), 4.1594666 > > > secs] > > > 2006-11-28 18:46: 24.706426500 [Full GC 499584K->497661K(511808K), > > > 5.6004981 secs] > > > 2006-11-28 18:46:30.357607500 [Full GC 499583K->499583K(511808K), > > > 4.1456096 secs] > > > 2006-11-28 18:46:34.503350500 [Full GC 499583K->498283K(511808K), > > > 5.5813694 secs] > > > 2006-11-28 18:46:40.125999500 [Full GC 499579K->499579K(511808K), > > > 4.1222203 secs] > > > 2006-11-28 18:46: 44.248342500 [Full GC 499579K->498764K(511808K), > > > 5.5525750 secs] > > > 2006-11-28 18:46: 49.836342500 [Full GC 499583K->499583K(511808K), > > > 4.2713130 secs] > > > 2006-11-28 18:46:54.107789500 [Full GC 499583K->499000K(511808K), > > > 5.4783996 secs] > > > 2006-11-28 18:46:59.616781500 [Full GC 499571K->499571K(511808K), > > > 4.3601245 secs] > > > 2006-11-28 18:47:03.977038500 [Full GC 499571K->499231K(511808K), > > > 5.6121831 secs] > > > 2006-11-28 18:47: 09.598690500 [Full GC 499584K->499584K(511808K), > > > 4.1334214 secs] > > > 2006-11-28 18:47: 13.732243500 [Full GC 499584K->499344K(511808K), > > > 5.4238636 secs] > > > 2006-11-28 18:47:19.162038500 [Full GC 499583K->499583K(511808K), > > > 4.1761207 secs] > > > 2006-11-28 18:47:23.338292500 [Full GC 499583K->499435K(511808K), > > > 4.0381495 secs] > > > 2006-11-28 18:47:27.397388500 [Full GC 499578K->499578K(511808K), > > > 4.1922121 secs] > > > 2006-11-28 18:47: 31.589735500 [Full GC 499578K->499504K(511808K), > > > 5.5022513 secs] > > > 2006-11-28 18:47: 37.095978500 [Full GC 499583K->499583K(511808K), > > > 4.1805690 secs] > > > 2006-11-28 18:47:41.276684500 [Full GC 499583K->499503K(511808K), > > > 4.0814699 secs] > > > 2006-11-28 18:47:45.363748500 [Full GC 499583K->499583K(511808K), > > > 4.1720694 secs] > > > 2006-11-28 18:47:49.535943500 [Full GC 499583K->499544K(511808K), > > > 4.1699713 secs] > > > 2006-11-28 18:47: 53.755950500 [Full GC 499583K->499583K(511808K), > > > 4.1593166 secs] > > > 2006-11-28 18:47: 57.917455500 [Full GC 499583K->499552K(511808K), > > > 5.5514382 secs] > > > 2006-11-28 18:48:03.472669500 [Full GC 499584K->499584K(511808K), > > > 4.1725040 secs] > > > 2006-11-28 18:48:07.659044500 [Full GC 499584K->499584K(511808K), > > > 4.1750055 secs] > > > 2006-11-28 18:48:11.847801500 [Full GC 499584K->499584K(511808K), > > > 4.1131328 secs] > > > 2006-11-28 18:48: 15.979586500 [Full GC 499584K->496903K(511808K), > > > 5.6763932 secs] > > > 2006-11-28 18:48: 21.657594500 Exception in thread "Callback > > > Thread" [ERROR] 16818770 pool-1-thread-7: > > > ( org.red5.server.service.ServiceInvoker.error ) Error executing > > > call: Service: null Method: deleteStream Num Params: 10: 1 > > > 2006-11-28 18:48:21.658716500 java.lang.OutOfMemoryError: Java heap > > > space[ERROR] 16818779 pool-1-thread-7: > > > ( org.red5.server.service.ServiceInvoker.error ) Service invocation > > > error > > > 2006-11-28 18:48:21.667577500 > > > java.lang.reflect.InvocationTargetException > > > 2006-11-28 18:48:21.667580500 at > > > sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source) > > > 2006-11-28 18:48: 21.667582500 at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke > > > (DelegatingMethodAccessorImpl.java:25) > > > 2006-11-28 18:48:21.667585500 at java.lang.reflect.Method.invoke > > > (Method.java:585) > > > 2006-11-28 18:48: 21.667731500 at > > > org.red5.server.service.ServiceInvoker.invoke (ServiceInvoker.java > > > :161) > > > 2006-11-28 18:48:21.667734500 at > > > org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java > > > :135) > > > 2006-11-28 18:48: 21.667736500 at > > > org.red5.server.net.rtmp.RTMPHandler.onInvoke (RTMPHandler.java:276) > > > 2006-11-28 18:48:21.667739500 at > > > org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived > > > (BaseRTMPHandler.java:114) > > > 2006-11-28 18:48: 21.667741500 at > > > org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived > > > (RTMPMinaIoHandler.java:78) > > > 2006-11-28 18:48:21.667752500 at > > > > > > org.apache.mina.common.support.AbstractIoFilterChain$2.messageReceived > > > (AbstractIoFilterChain.java:189) > > > 2006-11-28 18:48:21.667756500 at > > > > > > org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageRece > > > ived(AbstractIoFilterChain.java:502) > > > 2006-11-28 18:48:21.667759500 at > > > org.apache.mina.common.support.AbstractIoFilterChain.access$1000 > > > (AbstractIoFilterChain.java:52) > > > 2006-11-28 18:48:21.667775500 at > > > org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl > > > $1.messageReceived( AbstractIoFilterChain.java:777) > > > 2006-11-28 18:48:21.667778500 at > > > org.red5.io.filter.ExecutorFilter.processEvent(ExecutorFilter.java > > > :231) > > > 2006-11-28 18:48:21.667781500 at org.red5.io.filter.ExecutorFilter > > > $ProcessEventsRunnable.run( ExecutorFilter.java:279) > > > 2006-11-28 18:48: 21.667784500 at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask > > > (ThreadPoolExecutor.java:650) > > > 2006-11-28 18:48:21.667792500 at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run > > > (ThreadPoolExecutor.java :675) > > > 2006-11-28 18:48:21.667795500 at java.lang.Thread.run(Thread.java > > > :595) > > > 2006-11-28 18:48:21.667797500 Caused by: java.lang.OutOfMemoryError: > > > Java heap space > > > 2006-11-28 18:48: 21.668241500 > > > 2006-11-28 18:48: 21.669689500 Exception in thread "btpool0-1" > > > java.lang.OutOfMemoryError: Java heap space > > > 2006-11-28 18:48:21.671628500 Exception in thread > > > "FlowControlService" java.lang.OutOfMemoryError: Java heap space > > > > > > > > > _______________________________________________ > > > Red5 mailing list > > > [email protected] > > > http://osflash.org/mailman/listinfo/red5_osflash.org > > > > > > > > > > > -- > > I cannot tell why this heart languishes in silence. It is for small > > needs it never asks, or knows or remembers. -- Tagore > > > > Best Regards > > Steven Gong_______________________________________________ > > Red5 mailing list > > [email protected] > > http://osflash.org/mailman/listinfo/red5_osflash.org > > > > > > > > _______________________________________________ > > Red5 mailing list > > [email protected] > > http://osflash.org/mailman/listinfo/red5_osflash.org > > > > > > > > > -- > I cannot tell why this heart languishes in silence. It is for small > needs it never asks, or knows or remembers. -- Tagore > > Best Regards > Steven Gong _______________________________________________ > Red5 mailing list > [email protected] > http://osflash.org/mailman/listinfo/red5_osflash.org > > > > _______________________________________________ > Red5 mailing list > [email protected] > http://osflash.org/mailman/listinfo/red5_osflash.org > > > -- I cannot tell why this heart languishes in silence. It is for small needs it never asks, or knows or remembers. -- Tagore Best Regards Steven Gong_______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org _______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org
-- I cannot tell why this heart languishes in silence. It is for small needs it never asks, or knows or remembers. -- Tagore Best Regards Steven Gong
_______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org
