Re: [FFmpeg-devel] 32bit transcoding app running out of memory
Hi Ronald, actually the problem was occurring transcoding quicktime files with ProRes settings and not with h264. Changing the number of threads would not solve the problem. I noticed that memory usage would keep growing as long i reached a certain point at which i would get a message like "Delay between the first packet and last packet in the muxing queue is 10004900 > 1000: forcing output". When transcoding 8k files i would not get to this point as at 4GB the app would crash. The solution at the moment is to set the output format context's "max_interleave_delta" at a value lower than 1000 after the output context is allocated and before avformat_write_header(). Doing this i am forcing the encoder to output frames, memory usage is kept steady, and i managed to transcode 8k files with no errors. But is this a correct workaround? Or should i set some flag too? I tried also to set the output context flag to AVFMT_FLAG_SHORTEST but i am not getting the same result. Thank you, bye, Simone. ________ From: Simone Donadini Sent: Friday, March 15, 2019 11:42 AM To: FFmpeg development discussions and patches Subject: RE: [FFmpeg-devel] 32bit transcoding app running out of memory Hi Ronald, yes, we are using our own codec wrapped inside FFmpeg. Thank you for your suggestion, i will try with limiting the number of threads launched by FFmpeg. Simone. From: ffmpeg-devel [ffmpeg-devel-boun...@ffmpeg.org] on behalf of Ronald S. Bultje [rsbul...@gmail.com] Sent: Friday, March 15, 2019 11:24 AM To: FFmpeg development discussions and patches Subject: Re: [FFmpeg-devel] 32bit transcoding app running out of memory Hi, On Thu, Mar 14, 2019 at 7:52 AM Simone Donadini < simone.donad...@avolites.com> wrote: > > 2019-03-14 11:28 GMT+01:00, Simone Donadini < > simone.donad...@avolites.com>: > > > While transcoding video files with larger resolution (8K) the app, > > > which is 32bit, will run out of memory (>4GB). > > > It does not look like a memory leak, as memory usage will grow as long > as > > > transcoding a certain amount of frames and then it will stay steady. It > > > looks like the amount of memory usage depends on the resolution of the > > > > > frame, but even with a 8K frame using >4GB is not expected. > > > > Why do you think so? > > With our codec one encoded frame 7680x4320 should be ~100MB. > Am I right in interpreting this as that you're not actually using an existing FFMpeg encoder, but rather your own codec's encoder wrapped into FFmpeg by yourself? > h264 8K @60fps I think several dozen frame allocations in the decoder (b/c of frame threading) can happen, depending on the machine. 8*4*1.5*2=96MB/frame (assuming 10bits 8K, you didn't mention), so for 3 dozen (assuming n_threads=20-30), we're talking 3.6GB. To limit, use ffmpeg -thread_type none -i .. or ffmpeg -threads $n -i .. where $n is a smaller number (e.g. 2 or 4) than the default (which is system-dependent). Ronald ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] 32bit transcoding app running out of memory
Hi Ronald, yes, we are using our own codec wrapped inside FFmpeg. Thank you for your suggestion, i will try with limiting the number of threads launched by FFmpeg. Simone. From: ffmpeg-devel [ffmpeg-devel-boun...@ffmpeg.org] on behalf of Ronald S. Bultje [rsbul...@gmail.com] Sent: Friday, March 15, 2019 11:24 AM To: FFmpeg development discussions and patches Subject: Re: [FFmpeg-devel] 32bit transcoding app running out of memory Hi, On Thu, Mar 14, 2019 at 7:52 AM Simone Donadini < simone.donad...@avolites.com> wrote: > > 2019-03-14 11:28 GMT+01:00, Simone Donadini < > simone.donad...@avolites.com>: > > > While transcoding video files with larger resolution (8K) the app, > > > which is 32bit, will run out of memory (>4GB). > > > It does not look like a memory leak, as memory usage will grow as long > as > > > transcoding a certain amount of frames and then it will stay steady. It > > > looks like the amount of memory usage depends on the resolution of the > > > > > frame, but even with a 8K frame using >4GB is not expected. > > > > Why do you think so? > > With our codec one encoded frame 7680x4320 should be ~100MB. > Am I right in interpreting this as that you're not actually using an existing FFMpeg encoder, but rather your own codec's encoder wrapped into FFmpeg by yourself? > h264 8K @60fps I think several dozen frame allocations in the decoder (b/c of frame threading) can happen, depending on the machine. 8*4*1.5*2=96MB/frame (assuming 10bits 8K, you didn't mention), so for 3 dozen (assuming n_threads=20-30), we're talking 3.6GB. To limit, use ffmpeg -thread_type none -i .. or ffmpeg -threads $n -i .. where $n is a smaller number (e.g. 2 or 4) than the default (which is system-dependent). Ronald ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] 32bit transcoding app running out of memory
Hi Carl, Thanks for your reply. The main issue we are seeing is the huge memory footprint, directly allocated by FFMPEG. I'm no expert in FFMPEG, but it looks like it is doing this to encode multiple frames at the same time. Is this correct and expected behaviour? If this is correct, is there a way to limit the memory usage so that it doesn't go above 4GB. Would we be able to set a maximum amount of transcodeable frames at one time? As previously stated, the app is built as a 32 bit windows application. Using the large address aware flag to allow for a maximum of 4GB of memory on a 64bit OS. Do you have any advice on how to reduce the memory footprint of FFMPEG? Kind Regards, Simone Donadini From: ffmpeg-devel [ffmpeg-devel-boun...@ffmpeg.org] on behalf of Carl Eugen Hoyos [ceffm...@gmail.com] Sent: Thursday, March 14, 2019 11:36 PM To: FFmpeg development discussions and patches Subject: Re: [FFmpeg-devel] 32bit transcoding app running out of memory 2019-03-14 12:51 GMT+01:00, Simone Donadini : >> 2019-03-14 11:28 GMT+01:00, Simone Donadini >> : >> > we are developing a transcoder using ffmpeg based on the >> > transcoding example >> > https://www.ffmpeg.org/doxygen/4.1/transcoding_8c-example.html >> > and we are encountering some problem with memory usage >> > transcoding files with large resolutions. >> > When transcoding 4K files, the memory usage will go up to >> > 3.4GB but then it will stay steady for the rest of the conversion, >> > and clean every thing at the end. >> >> massif (or an equivalent application) can tell you where the memory >> is allocated, some decoders absolutely need these allocations, apart >> from that my guess is that (a part of) the allocation does not happen >> in FFmpeg. > > ok, i will try with it. But i am not seeing any memory leak with smaller > files, so i am not sure why there would be any when using a greater > resolution. massif does not detect memory leaks, so I don't understand your comment. >> > While transcoding video files with larger resolution (8K) the app, >> > which is 32bit, will run out of memory (>4GB). >> > It does not look like a memory leak, as memory usage will grow >> > as long as transcoding a certain amount of frames and then it >> > will stay steady. It looks like the amount of memory usage >> > depends on the resolution of the frame, I forgot: Of course it does. >>> but even with a 8K frame using >4GB is not expected. >> >> Why do you think so? > > With our codec one encoded frame 7680x4320 should be ~100MB. No, memory requirements can be >3G (You did not give enough information about input video and encoder settings, "same properties" are not valid settings). Carl Eugen ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] 32bit transcoding app running out of memory
> 2019-03-14 11:28 GMT+01:00, Simone Donadini : > > Hi there, > > we are developing a transcoder using ffmpeg based on the transcoding example > > https://www.ffmpeg.org/doxygen/4.1/transcoding_8c-example.html and we are > > encountering some problem with memory usage transcoding files with large > > resolutions. > > When transcoding 4K files, the memory usage will go up to 3.4GB but then it > > will stay steady for the rest of the conversion, and clean every thing at > > the end. > > massif (or an equivalent application) can tell you where the memory > is allocated, some decoders absolutely need these allocations, apart > from that my guess is that (a part of) the allocation does not happen > in FFmpeg. ok, i will try with it. But i am not seeing any memory leak with smaller files, so i am not sure why there would be any when using a greater resolution. > > > While transcoding video files with larger resolution (8K) the app, > > which is 32bit, will run out of memory (>4GB). > > It does not look like a memory leak, as memory usage will grow as long as > > transcoding a certain amount of frames and then it will stay steady. It > > looks like the amount of memory usage depends on the resolution of the > > > frame, but even with a 8K frame using >4GB is not expected. > > Why do you think so? > With our codec one encoded frame 7680x4320 should be ~100MB. > > Is this a known behavior? > > Is there a way to limit the number of frames loaded on memory > > or the amount of memory while transcoding? > > Yes. > (You did not give enough information about input video and > encoder settings.) At the moment i am testing with a file h264 8K @60fps, and the encoder is using the same properties (framerate - size) of the source file. I am using no filters. > > > Or building a 64bit version would fix this issue? > > Most likely. > > Carl Eugen > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
[FFmpeg-devel] 32bit transcoding app running out of memory
Hi there, we are developing a transcoder using ffmpeg based on the transcoding example https://www.ffmpeg.org/doxygen/4.1/transcoding_8c-example.html and we are encountering some problem with memory usage transcoding files with large resolutions. When transcoding 4K files, the memory usage will go up to 3.4GB but then it will stay steady for the rest of the conversion, and clean every thing at the end. While transcoding video files with larger resolution (8K) the app, which is 32bit, will run out of memory (>4GB). It does not look like a memory leak, as memory usage will grow as long as transcoding a certain amount of frames and then it will stay steady. It looks like the amount of memory usage depends on the resolution of the frame, but even with a 8K frame using >4GB is not expected. Is this a known behavior? Is there a way to limit the number of frames loaded on memory or the amount of memory while transcoding? Or building a 64bit version would fix this issue? Any help would really be appreciated. Thank you, Simone. ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
