Hi peoples,
I think I found a nasty issue (or I'm doing something wrong ;)).
I use Red5 as a streaming video server like used on http://
www.geefme.nl/
The problem is that when serving several different and big flv files
(300MB+) and having more than 20 clients (some connecting, seeking or
just watching) Red5 stops responding for a couple of seconds and than
resumes.
First I tested on a single core, single processor P4 3.0GHz. At 15~20+
clients this event occurs frequently (freezing for a couple of
seconds)
and the server has a CPU usage of 100%.
Than I tested on a dual core, single processor Core2duo 2GHz. At the
same amount of clients the same problems occurs but the CPU usage hits
25%...
20 clients is approximate 20~25Mbps of traffic and when the
application
'freezes', the traffic drops to 100Kbps and the CPU maxes out (single
proc = 100% on the Core2duo system: 25%).
For you're information; I run Debian Linux first with JRE 1.5 and
later
with JRE 1.6 (tested with both JRE's). Both machines have 4GB
memory and
the FLV's are served from a fileserver using NFS connected by a 1Gbps
Ethernet connection.
When I checked the logs I found out all IO uses the same thread.
Like this:
[DEBUG] 2007-02-05 23:02:40,621 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) giveMovie; found 7
[DEBUG] 2007-02-05 23:02:40,621 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) 512 -- .flv
[DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) found: 512
[DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) 1024 -- .flv
[DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) found: 1024
[DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) 33 -- .flv
[DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) found: 33
[DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) 56 -- .flv
[DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) found: 56
[DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) 256 -- .flv
[DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) found: 256
[DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) 128 -- .flv
[DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) found: 128
[DEBUG] 2007-02-05 23:02:40,624 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) 64 -- .flv
[DEBUG] 2007-02-05 23:02:40,624 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) found: 64
[DEBUG] 2007-02-05 23:02:40,760 SocketAcceptorIoProcessor-0.0:(
Application.debug ) Success, APPconnect from /62.45.13.69:1151
[ERROR] 2007-02-05 23:02:40,764 SocketAcceptorIoProcessor-0.0:(
FLVReader.error ) Check for KeyFramesCache
[DEBUG] 2007-02-05 23:02:40,808 accessCheckerThread:(
accessChecker.debug ) accessChecker run
[DEBUG] 2007-02-05 23:02:40,816 accessCheckerThread:(
ipAccessChecker.debug ) ipAccessChecker run
[DEBUG] 2007-02-05 23:02:40,992 Thread-25:( messageFetcher.debug )
Fetching message
[ERROR] 2007-02-05 23:02:42,046 SocketAcceptorIoProcessor-0.0:(
FLVReader.error ) Loaded KeyFramesCache for file
/data/fcs/red5/_webapps/1euro50/streams/shar
[DEBUG] 2007-02-05 23:02:42,047 SocketAcceptorIoProcessor-0.0:(
flvPlayer.debug ) APPdisconnect. 81.68.67.83
[DEBUG] 2007-02-05 23:02:43,577 SocketAcceptorIoProcessor-0.0:(
Application.debug ) giveLastMessage
[DEBUG] 2007-02-05 23:02:43,605 SocketAcceptorIoProcessor-0.0:(
Application.debug ) Success, APPconnect from /82.192.92.180:55136
[DEBUG] 2007-02-05 23:02:43,624 SocketAcceptorIoProcessor-0.0:(
Application.debug ) giveLastMessage
[DEBUG] 2007-02-05 23:02:43,625 SocketAcceptorIoProcessor-0.0:(
Application.debug ) Success, APPconnect from /82.74.68.183:3902
[DEBUG] 2007-02-05 23:02:43,627 SocketAcceptorIoProcessor-0.0:(
Application.debug ) Success, APPconnect from /212.187.62.65:3287
[DEBUG] 2007-02-05 23:02:43,628 SocketAcceptorIoProcessor-0.0:(
Application.debug ) Success, APPconnect from /84.28.208.229:2588
[DEBUG] 2007-02-05 23:02:43,636 SocketAcceptorIoProcessor-0.0:(
Application.debug ) Success, APPconnect from /84.26.48.12:1542
[DEBUG] 2007-02-05 23:02:43,649 SocketAcceptorIoProcessor-0.0:(
Application.debug ) Success, APPconnect from /80.126.13.14:49991
[DEBUG] 2007-02-05 23:02:43,651 SocketAcceptorIoProcessor-0.0:(
Application.debug ) giveLastMessage
The first lines (flvPlayer.debug) are from my main application
handler.
The 'error' line (Check for KeyFramesCache) is patched by me in the
Red5
FLVReader class, as far as i know, the same call serves the file.
Please
notice all debug calls come from the same thread
"SocketAcceptorIoProcessor-0.0", even the "Application.debug" lines
with
"giveLastMessage" and the "APPconnect" messages, they're from a
totally
different application running on the same server.
I think the 'freezing' problem is caused by the seek feature. Can it
"hold" a thread for a while? It consumes a lot of CPU time (which is
obvious though) but on a multiprocessor machine, the application
runs on
only 1 processor..... So all other calls/buffering-routines stall
when a
seek is performed, and seeking in a 300MB+ FLV file could take a
while I
think :)
Can it also be a threading issue?
I already played with the configuration files, changed the max threads
variables, even tried switching from Jetty to TomCat, but it makes no
difference.
In debugmode I see 2 SocketAcceptor processes but in both (production
and develop) environments all debug messages come from
"SocketAcceptorIoProcessor-0.0"; for all active applications; for all
connected clients.
How can I take advantage of a multiprocessor system?
Am I doing something wrong?
Kind regards,
Martin Schipper
_______________________________________________
Red5 mailing list
[email protected]
http://osflash.org/mailman/listinfo/red5_osflash.org
