Hi peoples, I think I found a nasty issue (or I'm doing something wrong ;)). I use Red5 as a streaming video server like used on http://www.geefme.nl/
The problem is that when serving several different and big flv files (300MB+) and having more than 20 clients (some connecting, seeking or just watching) Red5 stops responding for a couple of seconds and than resumes. First I tested on a single core, single processor P4 3.0GHz. At 15~20+ clients this event occurs frequently (freezing for a couple of seconds) and the server has a CPU usage of 100%. Than I tested on a dual core, single processor Core2duo 2GHz. At the same amount of clients the same problems occurs but the CPU usage hits 25%... 20 clients is approximate 20~25Mbps of traffic and when the application 'freezes', the traffic drops to 100Kbps and the CPU maxes out (single proc = 100% on the Core2duo system: 25%). For you're information; I run Debian Linux first with JRE 1.5 and later with JRE 1.6 (tested with both JRE's). Both machines have 4GB memory and the FLV's are served from a fileserver using NFS connected by a 1Gbps Ethernet connection. When I checked the logs I found out all IO uses the same thread. Like this: [DEBUG] 2007-02-05 23:02:40,621 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) giveMovie; found 7 [DEBUG] 2007-02-05 23:02:40,621 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) 512 -- .flv [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) found: 512 [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) 1024 -- .flv [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) found: 1024 [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) 33 -- .flv [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) found: 33 [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) 56 -- .flv [DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) found: 56 [DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) 256 -- .flv [DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) found: 256 [DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) 128 -- .flv [DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) found: 128 [DEBUG] 2007-02-05 23:02:40,624 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) 64 -- .flv [DEBUG] 2007-02-05 23:02:40,624 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) found: 64 [DEBUG] 2007-02-05 23:02:40,760 SocketAcceptorIoProcessor-0.0:( Application.debug ) Success, APPconnect from /62.45.13.69:1151 [ERROR] 2007-02-05 23:02:40,764 SocketAcceptorIoProcessor-0.0:( FLVReader.error ) Check for KeyFramesCache [DEBUG] 2007-02-05 23:02:40,808 accessCheckerThread:( accessChecker.debug ) accessChecker run [DEBUG] 2007-02-05 23:02:40,816 accessCheckerThread:( ipAccessChecker.debug ) ipAccessChecker run [DEBUG] 2007-02-05 23:02:40,992 Thread-25:( messageFetcher.debug ) Fetching message [ERROR] 2007-02-05 23:02:42,046 SocketAcceptorIoProcessor-0.0:( FLVReader.error ) Loaded KeyFramesCache for file /data/fcs/red5/_webapps/1euro50/streams/shar [DEBUG] 2007-02-05 23:02:42,047 SocketAcceptorIoProcessor-0.0:( flvPlayer.debug ) APPdisconnect. 81.68.67.83 [DEBUG] 2007-02-05 23:02:43,577 SocketAcceptorIoProcessor-0.0:( Application.debug ) giveLastMessage [DEBUG] 2007-02-05 23:02:43,605 SocketAcceptorIoProcessor-0.0:( Application.debug ) Success, APPconnect from /82.192.92.180:55136 [DEBUG] 2007-02-05 23:02:43,624 SocketAcceptorIoProcessor-0.0:( Application.debug ) giveLastMessage [DEBUG] 2007-02-05 23:02:43,625 SocketAcceptorIoProcessor-0.0:( Application.debug ) Success, APPconnect from /82.74.68.183:3902 [DEBUG] 2007-02-05 23:02:43,627 SocketAcceptorIoProcessor-0.0:( Application.debug ) Success, APPconnect from /212.187.62.65:3287 [DEBUG] 2007-02-05 23:02:43,628 SocketAcceptorIoProcessor-0.0:( Application.debug ) Success, APPconnect from /84.28.208.229:2588 [DEBUG] 2007-02-05 23:02:43,636 SocketAcceptorIoProcessor-0.0:( Application.debug ) Success, APPconnect from /84.26.48.12:1542 [DEBUG] 2007-02-05 23:02:43,649 SocketAcceptorIoProcessor-0.0:( Application.debug ) Success, APPconnect from /80.126.13.14:49991 [DEBUG] 2007-02-05 23:02:43,651 SocketAcceptorIoProcessor-0.0:( Application.debug ) giveLastMessage The first lines (flvPlayer.debug) are from my main application handler. The 'error' line (Check for KeyFramesCache) is patched by me in the Red5 FLVReader class, as far as i know, the same call serves the file. Please notice all debug calls come from the same thread "SocketAcceptorIoProcessor-0.0", even the "Application.debug" lines with "giveLastMessage" and the "APPconnect" messages, they're from a totally different application running on the same server. I think the 'freezing' problem is caused by the seek feature. Can it "hold" a thread for a while? It consumes a lot of CPU time (which is obvious though) but on a multiprocessor machine, the application runs on only 1 processor..... So all other calls/buffering-routines stall when a seek is performed, and seeking in a 300MB+ FLV file could take a while I think :) Can it also be a threading issue? I already played with the configuration files, changed the max threads variables, even tried switching from Jetty to TomCat, but it makes no difference. In debugmode I see 2 SocketAcceptor processes but in both (production and develop) environments all debug messages come from "SocketAcceptorIoProcessor-0.0"; for all active applications; for all connected clients. How can I take advantage of a multiprocessor system? Am I doing something wrong? Kind regards, Martin Schipper _______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org
