Server performance: Generally, the server had 1.4 M of RAM free. Also, the CPU load was usually in single digits. When there were problems with the stream, the CPU went up. When the biggest failure occurred, CPU was at 90+%. When the bad sync was happening, the CPU was at 30%.
Overall bandwidth never exceeded 6 Mbps, so there was only trivial demand on our Internet connection and network. (Internet connection is 1GBit and server is on a 100 Mbit switch.) Next time we do this, I think we'll set up a chat instead of using the email list :-) Thanks again to everyone. Interalab wrote: > Here are the interesting entries from the log: > > This was thrown when the number of live streams was up to about 43 and > then I switched from the live stream to a video on demand. All clients > were dropped and many 'out of memory errors' where thrown until we > restarted the server. > > [ERROR] 4273322 SocketAcceptorIoProcessor-0.0:( > InMemoryPullPullPipe.error ) exception when pulling message from provider > java.lang.OutOfMemoryError: Java heap space > at java.lang.Integer.toString(Integer.java:306) > at java.lang.Integer.toString(Integer.java:116) > at java.lang.String.valueOf(String.java:2932) > at > org.red5.io.FileKeyFrameMetaCache.saveKeyFrameMeta(FileKeyFrameMetaCache.java:154) > at org.red5.io.flv.impl.FLVReader.analyzeKeyFrames(FLVReader.java:686) > at org.red5.io.flv.impl.FLVReader.postInitialize(FLVReader.java:334) > at org.red5.io.flv.impl.FLVReader.<init>(FLVReader.java:147) > at org.red5.io.flv.impl.FLV.getReader(FLV.java:157) > at > org.red5.server.stream.provider.FileProvider.init(FileProvider.java:215) > at > org.red5.server.stream.provider.FileProvider.pullMessage(FileProvider.java:118) > at > org.red5.server.messaging.InMemoryPullPullPipe.pullMessage(InMemoryPullPullPipe.java:75) > at > org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.pullAndPush(PlaylistSubscriberStream.java:1061) > at > org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.play(PlaylistSubscriberStream.java:871) > at > org.red5.server.stream.PlaylistSubscriberStream.play(PlaylistSubscriberStream.java:180) > at org.red5.server.stream.StreamService.play(StreamService.java:219) > at org.red5.server.stream.StreamService.play(StreamService.java:249) > at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:181) > at org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:170) > at org.red5.server.net.rtmp.RTMPHandler.onInvoke(RTMPHandler.java:348) > at > org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:126) > at > org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived(RTMPMinaIoHandler.java:117) > at > org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:700) > at > org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:361) > at > org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53) > at > org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:785) > at > org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimpleProtocolDecoderOutput.java:62) > at > org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:187) > at > org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:361) > at > org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53) > [INFO] 4273327 SocketAcceptorIoProcessor-0.0:( > PlaylistSubscriberStream.info ) Scheduled stop in: 1121572707 > > > > This was thrown just before the out of memory errors. > > [ERROR] 68909 SocketAcceptorIoProcessor-0.3:( > FileKeyFrameMetaCache.error ) Could not parse XML file. > org.xml.sax.SAXParseException: Premature end of file. > at org.apache.xerces.parsers.DOMParser.parse(Unknown Source) > at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source) > at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180) > at > org.red5.io.FileKeyFrameMetaCache.loadKeyFrameMeta(FileKeyFrameMetaCache.java:75) > at org.red5.io.flv.impl.FLVReader.analyzeKeyFrames(FLVReader.java:583) > at org.red5.io.flv.impl.FLVReader.postInitialize(FLVReader.java:334) > at org.red5.io.flv.impl.FLVReader.<init>(FLVReader.java:147) > at org.red5.io.flv.impl.FLV.getReader(FLV.java:157) > at > org.red5.server.stream.provider.FileProvider.init(FileProvider.java:215) > at > org.red5.server.stream.provider.FileProvider.pullMessage(FileProvider.java:118) > at > org.red5.server.messaging.InMemoryPullPullPipe.pullMessage(InMemoryPullPullPipe.java:75) > at > org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.pullAndPush(PlaylistSubscriberStream.java:1061) > at > org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.play(PlaylistSubscriberStream.java:871) > at > org.red5.server.stream.PlaylistSubscriberStream.play(PlaylistSubscriberStream.java:180) > at org.red5.server.stream.StreamService.play(StreamService.java:219) > at org.red5.server.stream.StreamService.play(StreamService.java:249) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.red5.server.service.ServiceInvoker.invoke(ServiceInvoker.java:181) > at org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java:170) > at org.red5.server.net.rtmp.RTMPHandler.onInvoke(RTMPHandler.java:348) > at > org.red5.server.net.rtmp.BaseRTMPHandler.messageReceived(BaseRTMPHandler.java:126) > at > org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived(RTMPMinaIoHandler.java:117) > at > org.apache.mina.common.support.AbstractIoFilterChain$TailFilter.messageReceived(AbstractIoFilterChain.java:700) > at > org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:361) > at > org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53) > at > org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:785) > at > org.apache.mina.filter.codec.support.SimpleProtocolDecoderOutput.flush(SimpleProtocolDecoderOutput.java:62) > at > org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:187) > at > org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:361) > at > org.apache.mina.common.support.AbstractIoFilterChain.access$1100(AbstractIoFilterChain.java:53) > at > org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl$1.messageReceived(AbstractIoFilterChain.java:785) > at > org.apache.mina.common.support.AbstractIoFilterChain$HeadFilter.messageReceived(AbstractIoFilterChain.java:616) > at > org.apache.mina.common.support.AbstractIoFilterChain.callNextMessageReceived(AbstractIoFilterChain.java:361) > at > org.apache.mina.common.support.AbstractIoFilterChain.fireMessageReceived(AbstractIoFilterChain.java:352) > at > org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:288) > at > org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:248) > at > org.apache.mina.transport.socket.nio.SocketIoProcessor.access$500(SocketIoProcessor.java:49) > at > org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:540) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:43) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) > at java.lang.Thread.run(Thread.java:619) > [INFO] 82057 SocketAcceptorIoProcessor-0.0:( > PlaylistSubscriberStream.info ) Scheduled stop in: 914 > [INFO] 82991 DefaultQuartzScheduler_Worker-10:( > PlaylistSubscriberStream.info ) in stop > [INFO] 82993 DefaultQuartzScheduler_Worker-10:( > PlaylistSubscriberStream.info ) Stop > > > > > A large number of these messages were in the log when the live stream > wouldn't play right. The audio was way out of sync with the video. > Only affected the live stream, not the on demand. > > > [WARN] 2993687 SocketAcceptorIoProcessor-0.3:( ConnectionConsumer.warn ) > Skipping message with negative timestamp. > [WARN] 2993688 SocketAcceptorIoProcessor-0.3:( ConnectionConsumer.warn ) > Skipping message with negative timestamp. > > > _______________________________________________ > Red5 mailing list > [email protected] > http://osflash.org/mailman/listinfo/red5_osflash.org > _______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org
