Re: [Red5] About the Fix of FLV and FLVReader in regards with OutOfMemoryError when playing large FLVs

Tue, 21 Nov 2006 18:04:19 -0800 

Created as APPSERVER-8 although I must warn you the code is horid heh.

Thanks,

--Dan--

On Nov 21, 2006, at 6:31 PM, Steven Gong wrote:


Dan,
Please submit a ticket on JIRA and attach your version of FLVReader on it. I will take a look at it later. Thanks. :-) 

On 11/22/06, Daniel Daley <[EMAIL PROTECTED]> wrote:
I hacked together a quick version of FLVReader that eliminates the memory mapping and pulls straight from the FileInputStream channel and it so far has been able to handle much much more. The load on the server is terribly high of course. Is this something that could be implemented in the server as an option? It would probably be better load wise if I created a buffer of a fixed size (couple hundred K or something) and it would just keep that buffer filled with data from the file as it needed it. I'll give that a shot if I get a chance but having something like this in the actual Red5 distribution would be wonderful, as loading the whole file to memory puts way too small of a cap on how much we can serve. 

Thanks,

--Dan--


On Nov 21, 2006, at 1:19 PM, Daniel Daley wrote:
Looking at the code I guess it does load the file into memory completely. 

channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size());
Wouldn't it be more efficient/scalable to have it seek around the FileInputStream and read the appropriate bytes for header data etc, and then have a configurable sized ByteBuffer for the reading of the main file data? 

Thanks,

--Dan--


On Nov 21, 2006, at 12:09 PM, Daniel Daley wrote:
What I think I'm running into here is the 2 gig per process memory limit of 32 bit processors. Is there any way to reduce the amount of memory required per connection? Right now it is still acting as if it loads the entire file into memory as our files are on average 20 megs and we can get up to around 100 connections before the crash. Our server has 4 gigs of ram in it so we have plenty left to use. I don't understand the java threading system very well, but shouldn't the threading that red5 does create a separate process with it's own limits per connection? Or do all connections share the single master red5 process? 

Thanks,

--Dan--

On Nov 21, 2006, at 11:23 AM, Daniel Daley wrote:
I spoke too soon. Once the connections starting picking up today we went back to the previous errors. This one was at 100 connected viewers:
2006-11-21 13:17:59.572534500 [ERROR] 194490 pool-1-thread-4: ( org.red5.io.flv.impl.FLVReader.error ) FLVReader :: FLVReader :: 
>
2006-11-21 13:17:59.572539500
2006-11-21 13:17: 59.572541500 java.io.IOException: Cannot allocate memory 2006-11-21 13:17:59.572543500 at sun.nio.ch.FileChannelImpl.map0(Native Method) 2006-11-21 13:17:59.572545500 at sun.nio.ch.FileChannelImpl.map (FileChannelImpl.java:742) 2006-11-21 13:17:59.572548500 at org.red5.io.flv.impl.FLVReader.<init>(FLVReader.java:106) 2006-11-21 13:17:59.572550500 at org.red5.io.flv.impl.FLV.getReader(FLV.java :172) 2006-11-21 13:17:59.572733500 at org.red5.server.stream.provider.FileProvider.init (FileProvider.java:179) 2006-11-21 13:17:59.572737500 at org.red5.server.stream.provider.FileProvider.pullMessage (FileProvider.java:87) 2006-11-21 13:17:59.572739500 at org.red5.server.messaging.InMemoryPullPullPipe.pullMessage (InMemoryPullPullPipe.java:72) 2006-11-21 13:17:59.572742500 at org.red5.server.stream.PlaylistSubscriberStream $PlayEngine.pullAndPush (PlaylistSubscriberStr 
eam.java:869)
2006-11-21 13:17:59.572789500 at org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.play (PlaylistSubscriberStream.jav 
a:703)
2006-11-21 13:17: 59.572792500 at org.red5.server.stream.PlaylistSubscriberStream.play (PlaylistSubscriberStream.java:127) 2006-11-21 13:17:59.572795500 at org.red5.server.stream.StreamService.play(StreamService.java:175) 2006-11-21 13:17:59.572797500 at org.red5.server.stream.StreamService.play(StreamService.java:183) 2006-11-21 13:17:59.572821500 at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source) 2006-11-21 13:17:59.572823500 at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:25) 2006-11-21 13:17:59.572826500 at java.lang.reflect.Method.invoke(Method.java:585) 2006-11-21 13:17:59.572828500 at org.red5.server.service.ServiceInvoker.invoke (ServiceInvoker.java:161) 2006-11-21 13:17:59.572830500 at org.red5.server.net.rtmp.RTMPHandler.invokeCall(RTMPHandler.java: 287) 2006-11-21 13:17:59.572840500 at org.red5.server.net.rtmp.RTMPHandler.onInvoke(RTMPHandler.java:469) 2006-11-21 13:17:59.572842500 at org.red5.server.net.rtmp.RTMPHandler.messageReceived (RTMPHandler.java :147) 2006-11-21 13:17:59.572844500 at org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived (RTMPMinaIoHandler.java:78) 2006-11-21 13:17:59.572847500 at org.apache.mina.common.support.AbstractIoFilterChain $2.messageReceived (AbstractIoFilterChain 
.java:189)
2006-11-21 13:17:59.572856500 at org.apache.mina.common.support.AbstractIoFilterChain.callNextMessag eReceived(AbstractIoFilte 
rChain.java:502)
2006-11-21 13:17: 59.572858500 at org.apache.mina.common.support.AbstractIoFilterChain.access$1000 (AbstractIoFilterChain.java: 
52)
2006-11-21 13:17:59.572861500 at org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl $1.messageReceived (AbstractIoF 
ilterChain.java:777)
2006-11-21 13:17:59.572876500 at org.red5.io.filter.ExecutorFilter.processEvent (ExecutorFilter.java:231) 2006-11-21 13:17:59.572879500 at org.red5.io.filter.ExecutorFilter$ProcessEventsRunnable.run (ExecutorFilter.java:279) 2006-11-21 13:17:59.572882500 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask (ThreadPoolExecutor.java:650) 2006-11-21 13:17:59.572885500 at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:675) 2006-11-21 13:17:59.572952500 at java.lang.Thread.run (Thread.java:595) 


On Nov 20, 2006, at 5:58 PM, Steven Gong wrote:




On 11/21/06, Daniel Daley <[EMAIL PROTECTED] > wrote:
Looks like it actually just crashes when it reaches whatever amount of ram the process is limited to. Being 32 bit if I leave all limits 
off once red5 allocates 2 gigs it crashes. Oddly though since this
patch the ram just seems to grow endlessly until it dies rather than fluctuating like it did before. Could there be something that is not 
releasing memory after each connection?
It's quite strange that your result is different from that of Bill's.
Could you please try to disable the buffer by setting the type as "none"? And use NoCacheImpl please. 

Thanks,

--Dan--


On Nov 20, 2006, at 3:01 PM, Daniel Daley wrote:
> Looks like mine is crashing much more frequently now with this patch > but with the same errors each time. What's odd is the server does not > in the least seem to be out of memory when it happens. Here's the 
> output in my log:
>
> 2006-11-20 16:58:39.560730500 [ERROR] 437909 pool-1-thread-4:
> ( org.red5.io.flv.impl.FLVReader.error ) FLVReader :: FLVReader ::> 
> 2006-11-20 16:58:39.560735500
> 2006-11-20 16:58:39.560737500 java.io.IOException: Cannot allocate 
> memory
> 2006-11-20 16:58:39.560739500 at sun.nio.ch.FileChannelImpl.map0 
> (Native Method)
> 2006-11-20 16:58:39.560741500 at sun.nio.ch.FileChannelImpl.map 
> (FileChannelImpl.java:742)
> 2006-11-20 16:58:39.560744500   at
> org.red5.io.flv.impl.FLVReader.<init>(FLVReader.java:106)
> 2006-11-20 16:58:39.560746500 at org.red5.io.flv.impl.FLV.getReader 
> (FLV.java:172)
> 2006-11-20 16:58:39.560917500   at
> org.red5.server.stream.provider.FileProvider.init (FileProvider.java: 
> 179)
> 2006-11-20 16:58: 39.560920500   at
> org.red5.server.stream.provider.FileProvider.pullMessage
> (FileProvider.java:87)
> 2006-11-20 16:58:39.560922500   at
> org.red5.server.messaging.InMemoryPullPullPipe.pullMessage
> (InMemoryPullPullPipe.java:72)
> 2006-11-20 16:58:39.560925500   at
> org.red5.server.stream.PlaylistSubscriberStream $PlayEngine.pullAndPush 
> (PlaylistSubscriberStream.java:869)
> 2006-11-20 16:58: 39.560937500   at
> org.red5.server.stream.PlaylistSubscriberStream$PlayEngine.play
> (PlaylistSubscriberStream.java:703)
> 2006-11-20 16:58:39.560940500   at
> org.red5.server.stream.PlaylistSubscriberStream.play
> (PlaylistSubscriberStream.java:127)
> 2006-11-20 16:58:39.560943500   at
> org.red5.server.stream.StreamService.play(StreamService.java: 175) 
> 2006-11-20 16:58:39.560945500   at
> org.red5.server.stream.StreamService.play (StreamService.java: 183) 
> 2006-11-20 16:58:39.560960500   at
> sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
> 2006-11-20 16:58:39.560962500   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke
> (DelegatingMethodAccessorImpl.java:25)
> 2006-11-20 16:58:39.560965500 at java.lang.reflect.Method.invoke 
> (Method.java:585)
> 2006-11-20 16:58:39.560967500   at
> org.red5.server.service.ServiceInvoker.invoke (ServiceInvoker.java:161) 
> 2006-11-20 16:58:39.560969500   at
> org.red5.server.net.rtmp.RTMPHandler.invokeCall (RTMPHandler.java:287) 
> 2006-11-20 16:58:39.560978500   at
> org.red5.server.net.rtmp.RTMPHandler.onInvoke (RTMPHandler.java:469) 
> 2006-11-20 16:58:39.560980500   at
> org.red5.server.net.rtmp.RTMPHandler.messageReceived (RTMPHandler.java: 
> 147)
> 2006-11-20 16:58:39.560982500   at
> org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived
> (RTMPMinaIoHandler.java:78)
> 2006-11-20 16:58:39.560985500   at
> org.apache.mina.common.support.AbstractIoFilterChain $2.messageReceived 
> (AbstractIoFilterChain.java:189)
> 2006-11-20 16:58: 39.560993500   at
> org.apache.mina.common.support.AbstractIoFilterChain.callNextMessa geRe 
> ce
> ived(AbstractIoFilterChain.java:502)
> 2006-11-20 16:58:39.560996500   at
> org.apache.mina.common.support.AbstractIoFilterChain.access$1000
> (AbstractIoFilterChain.java:52)
> 2006-11-20 16:58:39.560998500   at
> org.apache.mina.common.support.AbstractIoFilterChain$EntryImpl
> $1.messageReceived(AbstractIoFilterChain.java:777)
> 2006-11-20 16:58: 39.561012500   at
> org.red5.io.filter.ExecutorFilter.processEvent (ExecutorFilter.java: 
> 231)
> 2006-11-20 16:58:39.561015500 at org.red5.io.filter.ExecutorFilter 
> $ProcessEventsRunnable.run(ExecutorFilter.java :279)
> 2006-11-20 16:58:39.561017500   at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask
> (ThreadPoolExecutor.java:650)
> 2006-11-20 16:58:39.561019500   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run
> (ThreadPoolExecutor.java:675)
> 2006-11-20 16:58:39.561028500 at java.lang.Thread.run (Thread.java: 
> 595)
>
>
> On Nov 18, 2006, at 2:48 AM, Steven Gong wrote:
>
>> Hi Paul,
>> I have just committed some codes to fix the OutOfMemoryError. The 
>> related revision numbers are 1555, 1556 and 1557.
>>
>> The OutOfMemoryError is mainly due to the buffer in FLVReader so I >> added a new static oroperty 'maxBufferSize' to constrain the buffer >> size and also added a way to disable the buffer if none of 'auto', 
>> 'direct' and 'heap' buffer type is configured. The property can
>> also be configured in red5-common.xml.
>>
>> Another improvement is to let ICacheStore create ICacheable
>> instances so that in the case of NoCacheImpl, no instances will be 
>> created thus we won't create byte array in this case.
>>
>> Could you please have a review? Thanks. :-)
>>
>> As the OutOfMemoryError bug gives rise to many complaints of our >> users, I also cc this mail to red5 list for notification purpose. 
>>
>> --
>> I cannot tell why this heart languishes in silence. It is for small 
>> needs it never asks, or knows or remembers.  -- Tagore
>>
>> Best Regards
>> Steven Gong
>> _______________________________________________
>> Red5 mailing list
>> [email protected]
>> http://osflash.org/mailman/listinfo/red5_osflash.org
>
>
> _______________________________________________
> Red5 mailing list
> [email protected]
> http://osflash.org/mailman/listinfo/red5_osflash.org


_______________________________________________
Red5 mailing list
[email protected]
http://osflash.org/mailman/listinfo/red5_osflash.org



--
I cannot tell why this heart languishes in silence. It is for small needs it never asks, or knows or remembers. -- Tagore 

Best Regards
Steven Gong
_______________________________________________
Red5 mailing list
[email protected]
http://osflash.org/mailman/listinfo/red5_osflash.org


_______________________________________________
Red5 mailing list
[email protected]
http://osflash.org/mailman/listinfo/red5_osflash.org


_______________________________________________
Red5 mailing list
[email protected]
http://osflash.org/mailman/listinfo/red5_osflash.org


_______________________________________________
Red5 mailing list
[email protected]
http://osflash.org/mailman/listinfo/red5_osflash.org



_______________________________________________
Red5 mailing list
[email protected]
http://osflash.org/mailman/listinfo/red5_osflash.org





--
I cannot tell why this heart languishes in silence. It is for small needs it never asks, or knows or remembers. -- Tagore 

Best Regards
Steven Gong
_______________________________________________
Red5 mailing list
[email protected]
http://osflash.org/mailman/listinfo/red5_osflash.org



_______________________________________________
Red5 mailing list
[email protected]
http://osflash.org/mailman/listinfo/red5_osflash.org

Reply via email to