Hello, Hmm, interesting ?..
"top" says an even load of 15-19% when the problem occurs Cpu0 : 19.6% us, 14.3% sy, 0.0% ni, 65.4% id, 0.0% wa, 0.3% hi, 0.3% si Cpu1 : 17.2% us, 15.9% sy, 0.0% ni, 66.9% id, 0.0% wa, 0.0% hi, 0.0% si Cpu2 : 17.2% us, 15.9% sy, 0.0% ni, 66.9% id, 0.0% wa, 0.0% hi, 0.0% si Cpu3 : 16.9% us, 16.6% sy, 0.0% ni, 66.6% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 2074848k total, 2022916k used, 51932k free, 78664k buffers Swap: 2096472k total, 439104k used, 1657368k free, 934176k cached It seems that something happens every 19 secs. I get this using "vmstat 1" . Not easy to see if you don't use a fixed font, but every 19. line the cpu colum shows a higher value, and that is where the stream freezes. $ vmstat 1 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 439104 65724 78632 928228 0 0 1 0 1 0 1 1 98 0 0 0 439104 65916 78632 928228 0 0 0 136 1604 4221 1 1 98 0 4 0 439104 65916 78632 928228 0 0 0 0 1518 3216 14 13 72 0 0 0 439104 66172 78632 928228 0 0 0 0 1530 2448 33 30 37 0 0 0 439104 66324 78632 928228 0 0 0 0 1582 3893 1 1 98 0 0 0 439104 66580 78632 928228 0 0 0 0 1410 3185 0 1 99 0 0 0 439104 66580 78632 928228 0 0 0 800 1432 3420 1 1 98 0 0 0 439104 66900 78632 928228 0 0 0 0 1494 3758 1 1 98 0 0 0 439104 66900 78632 928228 0 0 0 0 1594 4170 1 1 98 0 0 0 439104 67316 78632 928228 0 0 0 0 1546 3907 1 1 98 0 0 0 439104 67316 78632 928228 0 0 0 0 1587 4249 1 1 97 0 0 0 439104 67636 78632 928228 0 0 0 0 1576 4000 1 1 98 0 0 0 439104 67628 78632 928228 0 0 0 124 1584 4124 1 1 98 0 0 0 439104 67692 78632 928228 0 0 0 0 1532 4062 1 1 98 0 0 0 439104 67564 78632 928228 0 0 0 0 1649 4412 2 1 96 0 0 0 439104 67564 78632 928228 0 0 0 0 1436 4042 1 1 98 0 4 0 439104 67564 78632 928228 0 0 0 0 1488 3915 1 1 98 0 0 0 439104 67564 78632 928228 0 0 0 108 1444 3862 1 1 99 0 0 0 439104 67500 78632 928228 0 0 0 0 1617 4221 3 1 96 0 0 0 439104 67500 78632 928228 0 0 0 0 1478 4138 2 1 97 0 4 0 439104 67180 78632 928228 0 0 0 0 1488 2910 19 18 63 0 1 0 439104 66988 78632 928228 0 0 0 16 1370 2127 33 26 41 0 0 0 439104 66660 78632 928228 0 0 0 192 1480 3449 1 1 98 0 0 0 439104 66084 78632 928228 0 0 0 0 1440 3261 1 0 99 0 0 0 439104 65364 78632 928228 0 0 0 0 1551 3552 0 1 99 0 0 0 439104 65172 78632 928228 0 0 0 0 1518 3750 1 1 98 0 0 0 439104 64660 78632 928228 0 0 0 328 1602 3986 1 1 98 0 0 0 439104 63636 78632 928228 0 0 0 0 1553 4016 1 1 98 0 0 0 439104 62796 78632 928228 0 0 0 136 1705 4250 1 1 98 0 0 0 439104 62748 78632 928228 0 0 0 0 1490 3835 1 1 98 0 0 0 439104 62748 78632 928228 0 0 0 0 1563 4054 1 1 98 0 0 0 439104 62748 78632 928488 0 0 0 0 1527 3882 1 1 98 0 0 0 439104 62724 78632 928488 0 0 0 0 1746 4396 1 1 98 0 0 0 439104 62732 78632 928488 0 0 0 216 1482 3963 1 1 98 0 0 0 439104 62604 78632 928488 0 0 0 0 1558 4116 2 1 97 0 0 0 439104 62284 78632 928488 0 0 0 0 1574 3987 1 1 98 0 0 0 439104 62284 78632 928488 0 0 0 4 1649 4134 1 1 98 0 0 0 439104 62292 78632 928488 0 0 0 0 1547 3916 1 1 98 0 4 0 439104 62292 78632 928488 0 0 0 96 1485 3049 18 18 64 0 0 0 439104 62292 78632 928488 0 0 0 0 1519 3672 6 5 88 0 0 0 439104 62292 78632 928488 0 0 0 0 1477 3418 1 1 99 0 1 0 439104 62292 78632 928488 0 0 0 0 1512 3694 1 1 98 0 0 0 439104 62292 78632 928488 0 0 0 0 1481 3669 1 1 98 0 0 0 439104 62292 78632 928488 0 0 0 136 1527 3839 0 1 98 0 0 0 439104 62292 78632 928488 0 0 0 0 1561 4037 1 1 98 0 2 0 439104 62308 78632 928488 0 0 0 0 1492 3739 1 1 98 0 0 0 439104 62300 78632 928488 0 0 0 0 1708 4366 1 1 98 0 0 0 439104 62300 78632 928488 0 0 0 0 1457 3664 1 0 99 0 0 0 439104 62300 78632 928488 0 0 0 200 1534 4074 1 1 98 0 0 0 439104 62300 78632 928488 0 0 0 0 1575 4100 2 1 97 0 0 0 439104 62292 78632 928488 0 0 0 0 1513 3885 1 1 98 0 0 0 439104 62284 78632 928488 0 0 0 0 1610 4078 1 1 98 0 0 0 439104 62412 78632 928488 0 0 0 0 1582 4078 1 1 98 0 0 0 439104 62412 78632 928488 0 0 0 252 1535 4156 2 1 98 0 0 0 439104 62668 78632 928488 0 0 0 0 1567 4088 1 1 98 0 1 0 439104 62668 78632 928488 0 0 0 0 1511 3951 1 1 99 0 4 0 439104 62988 78632 928488 0 0 0 0 1444 3024 18 17 65 0 0 0 439104 63076 78632 928488 0 0 0 0 1379 2399 31 28 41 0 0 0 439104 63076 78632 928488 0 0 0 0 1398 3212 1 1 99 0 1 0 439104 63076 78632 928488 0 0 0 180 1417 3218 1 1 98 0 0 0 439104 63076 78632 928488 0 0 0 0 1513 3777 2 0 98 0 0 0 439104 63268 78632 928488 0 0 0 16 1517 3770 1 1 98 0 0 0 439104 63252 78632 928488 0 0 0 0 1539 4031 1 1 99 0 0 0 439104 62732 78632 928488 0 0 0 296 1566 4155 1 2 97 0 0 0 439104 63052 78632 928488 0 0 0 0 1504 3947 1 1 98 0 1 0 439104 63372 78632 928488 0 0 0 0 1539 4031 1 1 98 0 0 0 439104 63372 78632 928488 0 0 0 660 1533 4080 1 1 98 0 0 0 439104 63692 78632 928488 0 0 0 0 1514 3811 1 1 98 0 1 0 439104 63692 78632 928488 0 0 0 208 1587 4357 1 1 97 0 1 0 439104 64012 78632 928488 0 0 0 0 1714 4355 2 1 98 0 0 0 439104 64012 78632 928488 0 0 0 0 1595 4263 1 1 98 0 0 0 439104 64012 78632 928488 0 0 0 0 1535 4118 1 1 98 0 0 0 439104 64012 78632 928488 0 0 0 0 1605 4145 1 1 97 0 0 0 439104 64084 78632 928488 0 0 0 304 1610 4054 1 1 98 0 4 0 439104 64084 78632 928488 0 0 0 0 1408 3470 8 6 86 0 0 0 439104 64084 78632 928488 0 0 0 0 1320 1711 42 39 20 0 0 0 439104 64084 78632 928488 0 0 0 0 1431 3203 1 1 99 0 0 0 439104 64156 78632 928488 0 0 0 0 1459 3320 0 1 99 0 Martin Schipper wrote: > Hi Klaus, > > That sounds familiar indeed; If you monitor your CPU usage, is one CPU > doing 100% and the other almost 0%? > > Grtz, > > Martin > > Klaus wrote: > >> I think I experince the same. >> >> I'm running a live webcam chat. It's all published lived and not recorded. >> >> At first everything is working fine, but after a while (hours?) , the >> stream freezes. Also for a few seconds and then resumes. Even if red5 is >> only taking up less than 1mbit of bandwidth. >> >> It also takes up more and more memory. appserver-33 ? >> >> After I restart red5, it works fine again for a while. So at the moment >> I restart red5 when no one is online :p >> >> I'm running red5 0.6 rc1 on redhat enterprise 4, 2gb ram, 2 x 3.2ghz >> cpu's , java1_5 >> >> Regard, >> Klaus >> >> >> >> >> Martin Schipper wrote: >> >> >>> Ey Thijs, >>> >>> I use the trunk version (1681) today I'll try updating, I saw the >>> current Rev is 1700. >>> I'm also wondering if others have experience with a heavily loaded >>> Red5 on a multiprocessor machine, does it use all processors? >>> >>> Tnx, >>> >>> Martin >>> >>> Thijs Triemstra | Collab wrote: >>> >>> >>>> Hoi Martin, >>>> >>>> what version of red5 are you using? There a bug with large flv's >>>> which is fixed, just wondering if you're already using that version >>>> or not.. >>>> check http://jira.red5.org/browse/SN-2 and http://jira.red5.org/ >>>> browse/APPSERVER-8 for details. >>>> >>>> grt, >>>> >>>> Thijs >>>> >>>> >>>> Op 6-feb-2007, om 13:38 heeft Martin Schipper het volgende geschreven: >>>> >>>> >>>> >>>> >>>>> Hi peoples, >>>>> >>>>> I think I found a nasty issue (or I'm doing something wrong ;)). >>>>> I use Red5 as a streaming video server like used on http:// >>>>> www.geefme.nl/ >>>>> >>>>> The problem is that when serving several different and big flv files >>>>> (300MB+) and having more than 20 clients (some connecting, seeking or >>>>> just watching) Red5 stops responding for a couple of seconds and than >>>>> resumes. >>>>> >>>>> First I tested on a single core, single processor P4 3.0GHz. At 15~20+ >>>>> clients this event occurs frequently (freezing for a couple of >>>>> seconds) >>>>> and the server has a CPU usage of 100%. >>>>> Than I tested on a dual core, single processor Core2duo 2GHz. At the >>>>> same amount of clients the same problems occurs but the CPU usage hits >>>>> 25%... >>>>> 20 clients is approximate 20~25Mbps of traffic and when the >>>>> application >>>>> 'freezes', the traffic drops to 100Kbps and the CPU maxes out (single >>>>> proc = 100% on the Core2duo system: 25%). >>>>> >>>>> For you're information; I run Debian Linux first with JRE 1.5 and >>>>> later >>>>> with JRE 1.6 (tested with both JRE's). Both machines have 4GB >>>>> memory and >>>>> the FLV's are served from a fileserver using NFS connected by a 1Gbps >>>>> Ethernet connection. >>>>> >>>>> When I checked the logs I found out all IO uses the same thread. >>>>> Like this: >>>>> >>>>> [DEBUG] 2007-02-05 23:02:40,621 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) giveMovie; found 7 >>>>> [DEBUG] 2007-02-05 23:02:40,621 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) 512 -- .flv >>>>> [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) found: 512 >>>>> [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) 1024 -- .flv >>>>> [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) found: 1024 >>>>> [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) 33 -- .flv >>>>> [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) found: 33 >>>>> [DEBUG] 2007-02-05 23:02:40,622 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) 56 -- .flv >>>>> [DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) found: 56 >>>>> [DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) 256 -- .flv >>>>> [DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) found: 256 >>>>> [DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) 128 -- .flv >>>>> [DEBUG] 2007-02-05 23:02:40,623 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) found: 128 >>>>> [DEBUG] 2007-02-05 23:02:40,624 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) 64 -- .flv >>>>> [DEBUG] 2007-02-05 23:02:40,624 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) found: 64 >>>>> [DEBUG] 2007-02-05 23:02:40,760 SocketAcceptorIoProcessor-0.0:( >>>>> Application.debug ) Success, APPconnect from /62.45.13.69:1151 >>>>> [ERROR] 2007-02-05 23:02:40,764 SocketAcceptorIoProcessor-0.0:( >>>>> FLVReader.error ) Check for KeyFramesCache >>>>> [DEBUG] 2007-02-05 23:02:40,808 accessCheckerThread:( >>>>> accessChecker.debug ) accessChecker run >>>>> [DEBUG] 2007-02-05 23:02:40,816 accessCheckerThread:( >>>>> ipAccessChecker.debug ) ipAccessChecker run >>>>> [DEBUG] 2007-02-05 23:02:40,992 Thread-25:( messageFetcher.debug ) >>>>> Fetching message >>>>> [ERROR] 2007-02-05 23:02:42,046 SocketAcceptorIoProcessor-0.0:( >>>>> FLVReader.error ) Loaded KeyFramesCache for file >>>>> /data/fcs/red5/_webapps/1euro50/streams/shar >>>>> [DEBUG] 2007-02-05 23:02:42,047 SocketAcceptorIoProcessor-0.0:( >>>>> flvPlayer.debug ) APPdisconnect. 81.68.67.83 >>>>> [DEBUG] 2007-02-05 23:02:43,577 SocketAcceptorIoProcessor-0.0:( >>>>> Application.debug ) giveLastMessage >>>>> [DEBUG] 2007-02-05 23:02:43,605 SocketAcceptorIoProcessor-0.0:( >>>>> Application.debug ) Success, APPconnect from /82.192.92.180:55136 >>>>> [DEBUG] 2007-02-05 23:02:43,624 SocketAcceptorIoProcessor-0.0:( >>>>> Application.debug ) giveLastMessage >>>>> [DEBUG] 2007-02-05 23:02:43,625 SocketAcceptorIoProcessor-0.0:( >>>>> Application.debug ) Success, APPconnect from /82.74.68.183:3902 >>>>> [DEBUG] 2007-02-05 23:02:43,627 SocketAcceptorIoProcessor-0.0:( >>>>> Application.debug ) Success, APPconnect from /212.187.62.65:3287 >>>>> [DEBUG] 2007-02-05 23:02:43,628 SocketAcceptorIoProcessor-0.0:( >>>>> Application.debug ) Success, APPconnect from /84.28.208.229:2588 >>>>> [DEBUG] 2007-02-05 23:02:43,636 SocketAcceptorIoProcessor-0.0:( >>>>> Application.debug ) Success, APPconnect from /84.26.48.12:1542 >>>>> [DEBUG] 2007-02-05 23:02:43,649 SocketAcceptorIoProcessor-0.0:( >>>>> Application.debug ) Success, APPconnect from /80.126.13.14:49991 >>>>> [DEBUG] 2007-02-05 23:02:43,651 SocketAcceptorIoProcessor-0.0:( >>>>> Application.debug ) giveLastMessage >>>>> >>>>> The first lines (flvPlayer.debug) are from my main application >>>>> handler. >>>>> The 'error' line (Check for KeyFramesCache) is patched by me in the >>>>> Red5 >>>>> FLVReader class, as far as i know, the same call serves the file. >>>>> Please >>>>> notice all debug calls come from the same thread >>>>> "SocketAcceptorIoProcessor-0.0", even the "Application.debug" lines >>>>> with >>>>> "giveLastMessage" and the "APPconnect" messages, they're from a >>>>> totally >>>>> different application running on the same server. >>>>> >>>>> I think the 'freezing' problem is caused by the seek feature. Can it >>>>> "hold" a thread for a while? It consumes a lot of CPU time (which is >>>>> obvious though) but on a multiprocessor machine, the application >>>>> runs on >>>>> only 1 processor..... So all other calls/buffering-routines stall >>>>> when a >>>>> seek is performed, and seeking in a 300MB+ FLV file could take a >>>>> while I >>>>> think :) >>>>> >>>>> Can it also be a threading issue? >>>>> I already played with the configuration files, changed the max threads >>>>> variables, even tried switching from Jetty to TomCat, but it makes no >>>>> difference. >>>>> In debugmode I see 2 SocketAcceptor processes but in both (production >>>>> and develop) environments all debug messages come from >>>>> "SocketAcceptorIoProcessor-0.0"; for all active applications; for all >>>>> connected clients. >>>>> >>>>> How can I take advantage of a multiprocessor system? >>>>> Am I doing something wrong? >>>>> >>>>> Kind regards, >>>>> >>>>> Martin Schipper >>>>> >>>>> >>>>> _______________________________________________ >>>>> Red5 mailing list >>>>> [email protected] >>>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Red5 mailing list >>>> [email protected] >>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>> >>>> >>>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> Red5 mailing list >>> [email protected] >>> http://osflash.org/mailman/listinfo/red5_osflash.org >>> >>> >>> >> _______________________________________________ >> Red5 mailing list >> [email protected] >> http://osflash.org/mailman/listinfo/red5_osflash.org >> >> > > > _______________________________________________ > Red5 mailing list > [email protected] > http://osflash.org/mailman/listinfo/red5_osflash.org > > _______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org
