Some interesting info from the logs, here is at the point of rebooting, would the logs be causing memory leaks ?
[INFO] 7299657 pool-3-thread-16:( VODApplication.appJoin ) Client joined app 1038 at time 1177748584250 [INFO] 7844 pool-3-thread-9:( VODApplication.appJoin ) Client joined app 0 at time 1177748620265 Dan Rossi wrote: > Looks like even when load balanced it blocks connections after time. Its > all the exceptions im sure of it, no out of memory errors which is > bizarre until i try and reboot the service which it logs all the jvm > dumps because it hanged, ive kept the threadcount to 4 as setting it to > the max it can do causes issues which was 2700. > > This one is fairly bizarre as it wasnt an outofmemory error as i usually > get, we load files from a network drive using a UNC name like so > file:////computername/sharename > > jvm 2 | 2007/04/28 00:45:32 | java.io.IOException: The specified > network name is no longer available > INFO | jvm 2 | 2007/04/28 00:45:32 | at > sun.nio.ch.FileDispatcher.read0(Native Method) > INFO | jvm 2 | 2007/04/28 00:45:32 | at > sun.nio.ch.FileDispatcher.read(Unknown Source) > INFO | jvm 2 | 2007/04/28 00:45:32 | at > sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source) > INFO | jvm 2 | 2007/04/28 00:45:32 | at > sun.nio.ch.IOUtil.read(Unknown Source) > INFO | jvm 2 | 2007/04/28 00:45:32 | at > sun.nio.ch.FileChannelImpl.read(Unknown Source) > INFO | jvm 2 | 2007/04/28 00:45:32 | at > org.red5.io.flv.impl.FLVReader.fillBuffer(FLVReader.java:319) > INFO | jvm 2 | 2007/04/28 00:45:32 | at > org.red5.io.flv.impl.FLVReader.fillBuffer(FLVReader.java:268) > INFO | jvm 2 | 2007/04/28 00:45:32 | at > org.red5.io.flv.impl.FLVReader.readTag(FLVReader.java:545) > > Dan Rossi wrote: > >> Seemed to kind of stablise for a while, however im looking at the >> performance monitor and the page faults column is high, is this an issue >> ? It reached about 1900 threads and then rebooted from the outofmemory >> errors. I set the threadcount to 2700 as that is aparantly the max the >> JVM can do. >> >> Dan Rossi wrote: >> >> >>> I tried adding these into the wrapper config to no avail, causes to use >>> more virtual memory , and memory gets quite large >>> >>> wrapper.java.additional.1=-Xms1024m >>> wrapper.java.additional.2=-Xmx1024m >>> #wrapper.java.additional.3=-XX:NewSize=512m >>> #wrapper.java.additional.4=-XX:MaxNewSize=512m >>> #wrapper.java.additional.5=-XX:SurvivorRatio=8 >>> >>> Dan Rossi wrote: >>> >>> >>> >>>> Also is there any reason why we need so much memory resources, our >>>> traffic has always been the same, and windows media only uses up about >>>> 500MB. will we need to get 4GB now ? >>>> >>>> Dan Rossi wrote: >>>> >>>> >>>> >>>> >>>>> It seems after messing with the memory limits, red5 rejects connections >>>>> but doesnt throw exceptions or out of memory errors, and cant connect >>>>> until it reboots. Is there any recommendations of tweaking a windows >>>>> server with 2GB of ram, ie tweaking the memory and cpu allocation and >>>>> pagefile size etc. what other java options should i be adding ? How can >>>>> i monitor what red5 is doing internally ? >>>>> >>>>> >>>>> Dan Rossi wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Nothing special, its being caused every 2 hours. >>>>>> >>>>>> Main server is just running red5, the 2 slave machines are also running >>>>>> windows media and IIS which were also getting the errors however the >>>>>> server has 2GB of ram and the resource monitor only shows 700MB being >>>>>> used etc. Its only started happening since updates not long ago. >>>>>> >>>>>> Lenny Sorey wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> Dan, >>>>>>> >>>>>>> Forgive me if you have answered these questions previously. >>>>>>> >>>>>>> What are you running in RED5 which causes the out of memory problem. >>>>>>> Video Streaming, Live Broadcast a combination of both? >>>>>>> >>>>>>> What all do you have actually running on these servers. >>>>>>> >>>>>>> RED5 (jetty), IIS, SQL Server, etc? >>>>>>> >>>>>>> How many users are connected when this error occurs? >>>>>>> >>>>>>> Can you tell in the scheme of things when and where the out of memory >>>>>>> problems are occurring? >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Lenny >>>>>>> >>>>>>> How long does RED5 until you receive your Out of Memory error message >>>>>>> in the logs. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 4/26/07, *Dan Rossi* <[EMAIL PROTECTED] >>>>>>> <mailto:[EMAIL PROTECTED]>> wrote: >>>>>>> >>>>>>> Hi we are constantly getting these errors on our windows 2003 >>>>>>> servers >>>>>>> after the latest svn updates. Im trying to work out how to tweak the >>>>>>> resources on our windows servers to fix these kind of errors >>>>>>> causing the >>>>>>> java wrapper service to reboot. Im trying to work out what are the >>>>>>> optimal memory and cpu allocation settings to use and java options >>>>>>> for >>>>>>> resources, we have 2G of ram, and dual P4 3G on all machines, even >>>>>>> after >>>>>>> load balancing its still doing this. Has only just started >>>>>>> happening. >>>>>>> >>>>>>> # java.lang.OutOfMemoryError: requested 32756 bytes for >>>>>>> ChunkPool::allocate. Out of swap space? >>>>>>> >>>>>>> Here is the advanced log >>>>>>> >>>>>>> 0x48127800 JavaThread "pool-2-thread-584" [_thread_blocked, id=8832] >>>>>>> 0x4741e400 JavaThread "pool-2-thread-583" [_thread_blocked, id=2692] >>>>>>> 0x51bc4400 JavaThread "pool-2-thread-582" [_thread_blocked, id=3668] >>>>>>> 0x5325f400 JavaThread "pool-2-thread-581" [_thread_blocked, id=8752] >>>>>>> 0x47ba8800 JavaThread "pool-2-thread-580" [_thread_blocked, id=9936] >>>>>>> 0x50fac400 JavaThread "pool-2-thread-579" [_thread_blocked, >>>>>>> id=10100] >>>>>>> 0x47ec1c00 JavaThread "pool-2-thread-578" [_thread_blocked, id=7584] >>>>>>> >>>>>>> >>>>>>> VM Arguments: >>>>>>> jvm_args: -Xms512m -Xmx1000m -Djava.library.path=../lib >>>>>>> -Dwrapper.key=z6ob3QwdVVUuY_JG -Dwrapper.port=32000 >>>>>>> -Dwrapper.use_system_time=TRUE -Dwrapper.version=3.1.2 >>>>>>> -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE >>>>>>> -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 >>>>>>> java_command: org.tanukisoftware.wrapper.WrapperSimpleApp >>>>>>> org.red5.server.Standalone >>>>>>> Launcher Type: SUN_STANDARD >>>>>>> >>>>>>> --------------- S Y S T E M --------------- >>>>>>> >>>>>>> OS: Windows Server 2003 family Build 3790 Service Pack 1 >>>>>>> >>>>>>> CPU:total 2 family 15, cmov, cx8, fxsr, mmx, sse, sse2, ht >>>>>>> >>>>>>> Memory: 4k page, physical 2030356k(997988k free), swap >>>>>>> 3359568k(2099096k >>>>>>> free) >>>>>>> >>>>>>> vm_info: Java HotSpot(TM) Client VM (1.6.0_01-b06) for windows-x86, >>>>>>> built on Mar 14 2007 00:24:02 by "java_re" with unknown MS VC++:1310 >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Red5 mailing list >>>>>>> [email protected] <mailto:[email protected]> >>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>>>>> <http://osflash.org/mailman/listinfo/red5_osflash.org> >>>>>>> >>>>>>> >>>>>>> ------------------------------------------------------------------------ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Red5 mailing list >>>>>>> [email protected] >>>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> Red5 mailing list >>>>>> [email protected] >>>>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> Red5 mailing list >>>>> [email protected] >>>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Red5 mailing list >>>> [email protected] >>>> http://osflash.org/mailman/listinfo/red5_osflash.org >>>> >>>> >>>> >>>> >>>> >>> _______________________________________________ >>> Red5 mailing list >>> [email protected] >>> http://osflash.org/mailman/listinfo/red5_osflash.org >>> >>> >>> >>> >> _______________________________________________ >> Red5 mailing list >> [email protected] >> http://osflash.org/mailman/listinfo/red5_osflash.org >> >> >> > > > _______________________________________________ > Red5 mailing list > [email protected] > http://osflash.org/mailman/listinfo/red5_osflash.org > > _______________________________________________ Red5 mailing list [email protected] http://osflash.org/mailman/listinfo/red5_osflash.org
