Re: [FFmpeg-devel] [FFmpeg-user] fMP4 generation speed

[Ronak]

> On Jul 25, 2018, at 11:42 PM, Liu Steven  wrote:
> 
> 
> 
>> 在 2018年7月26日，上午11:30，Ronak Patel  写道：
>> 
>> 
>> 
>>> On Jul 25, 2018, at 11:25 PM, Ronak Patel 
>>>  wrote:
>>> 
>>> 
>>> 
 On Jul 25, 2018, at 11:12 PM, Liu Steven  wrote:
 
 
 
> 在 2018年7月26日，上午7:04，Ronak Patel  写道：
> 
> Hi,
> 
> We’d like to start working on this real soon and want to make sure we 
> properly fix this. We understand this code is used by more than just fMP4 
> assets and even live streaming use cases.
> 
> Can anyone please help shed some light on all use cases we should make 
> sure are working after we make our fixes?
> 
> We’re afraid that the number of use cases here are unknown and we’ll be 
> stuck fixing this for an unknown time.
> 
> Can anyone please shed some light on this?
 
 you can try -f segment or -f hls or -f dash to do waht you want try, if 
 all of them cannot suuport your feature, you can submmit patch here.
 or you can submmit a new hls muxer patch here if you cannot fix all the 
 parameters for the old hls muxer.
 
>> 
>> Ok I’m going to try to fix mpegts, live streams, fmp4 and the other options 
>> that you already have.
>> 
>> Are there any unit tests or other tests we can run to ensure everything is 
>> still working?
>> 
>> If not we will have to make sure it works manually.
> You can add fate for the current hls.
> 
> BTW, you can try hls_list_size 0 to make sure it can make the feature which 
> you need.


I tried this, but it has no effect. So, here's what I'm thinking to solve this:

1. The HLS Manifest Header should be written in the hls_init method, not in the 
hls_window one.
2. Hls_window should only append a new segment definition to the end of the 
manifest file.
3. We can write out HLS_X_ENDLIST if we've reached the end of the file, or if 
the option says to attach it.

Would there be any reason why this algorithm wouldn't work to solve all 
possible use cases? We wouldn't have to keep regenerating the entire file just 
to append a new segment to the end of it.

Ronak


___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [FFmpeg-user] fMP4 generation speed

[Liu Steven]

> 在 2018年7月26日，上午11:30，Ronak Patel  写道：
> 
> 
> 
>> On Jul 25, 2018, at 11:25 PM, Ronak Patel 
>>  wrote:
>> 
>> 
>> 
>>> On Jul 25, 2018, at 11:12 PM, Liu Steven  wrote:
>>> 
>>> 
>>> 
 在 2018年7月26日，上午7:04，Ronak Patel  写道：
 
 Hi,
 
 We’d like to start working on this real soon and want to make sure we 
 properly fix this. We understand this code is used by more than just fMP4 
 assets and even live streaming use cases.
 
 Can anyone please help shed some light on all use cases we should make 
 sure are working after we make our fixes?
 
 We’re afraid that the number of use cases here are unknown and we’ll be 
 stuck fixing this for an unknown time.
 
 Can anyone please shed some light on this?
>>> 
>>> you can try -f segment or -f hls or -f dash to do waht you want try, if all 
>>> of them cannot suuport your feature, you can submmit patch here.
>>> or you can submmit a new hls muxer patch here if you cannot fix all the 
>>> parameters for the old hls muxer.
>>> 
> 
> Ok I’m going to try to fix mpegts, live streams, fmp4 and the other options 
> that you already have.
> 
> Are there any unit tests or other tests we can run to ensure everything is 
> still working?
> 
> If not we will have to make sure it works manually.
You can add fate for the current hls.

BTW, you can try hls_list_size 0 to make sure it can make the feature which you 
need.
> 
> 
 
 Thanks
 
 Ronak
 
> On Jul 13, 2018, at 8:05 AM, Ronak Patel 
>  wrote:
> 
> I was wondering if anyone on the development list was familiar with this 
> code and could provide pointers on how to fix up the code due to the 
> below problem.
> 
> 
>> On Jul 12, 2018, at 7:27 PM, Ronak Patel 
>>  wrote:
>> 
>> 
>> 
>>> On Jul 12, 2018, at 6:21 PM, Ronak Patel 
>>>  wrote:
>>> 
>>> Hey Carl,
>>> 
>>> So I dug into this more today and I have root caused what's exactly 
>>> happening here. 
>>> 
>>> The problematic code is this: 
>>> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368 
>>> 
>>> This is where the filename is set and the next line actually opens the 
>>> file. 
>>> 
>>> The logic for this hls_window method is the following:
>>> 
>>> 1. Make a new temporary file.
>>> 2. Write out a new HLS manifest header.
>>> 3. Loop through all available segments and write out all of the entries 
>>> for them.
>>> 4. Close the temporary file when finished.
>>> 5. Rename the temporary file to the target file name.
>>> 6. Rinse and repeat for every single fragment.
>>> 
>>> do you mean append fragment info to the m3u8 list, don’t refresh it?
>> 
>> Yup that’s what I mean. The fact that ffmpeg is refreshing the entire 
>> manifest hundreds of thousands of times is the reason for this massive slow 
>> down.
>> 
>> We tried changing the code a little to only write the manifest once once at 
>> end, and we can now fragment a 60 hour audio file in less than one minute. 
>> Less than one minute compared to 4 days is a BIG time savings.
>> 
>> 
>>> 
>>> Therefore, if you can imagine a 153 hour audio file, we write out a 
>>> totally new HLS manifest 550800 times (153 * 60 * 60 assuming a 1s 
>>> fragment duration) that gets progressively larger as each fragment is 
>>> generated.
>>> 
>>> This is a classic O(N^2) algorithm implementation, instead of:
>>> 
>>> 1. Creating the destination file up front & write the manifest header.
>>> 2. Append the new segment to the file.
>>> 3. If this is the last segment, write out EXT-X-ENDLIST.
>>> 
>>> There's no looping involved, nor the need to make temporary files.
>>> 
>>> FYI that I've noticed the same sort of pattern being applied to MPEG 
>>> DASH: 
>>> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786 
>>> 
>>> 
>>> To implement something like this, looks like we'd have to significantly 
>>> re-engineer the code. Do you have any pointers on how to go about doing 
>>> this? Or, would you be able to help do this?
>>> 
>> 
>> I was thinking about this and I think it must have been implemented this 
>> way because of live hls streams? Not writing to the main manifest 
>> directly and keeping that stream open does release any file system locks 
>> that there would be otherwise. 
>> 
>> Is this the reason why this was done this way at all?
>> 
>> 
>>> Thanks for all your help,
>>> 
>>> Ronak
>>> 
 On Jun 27, 2018, at 2:04 PM, Carl Zwanzig  wrote:
 
 Hi,
 
 I haven't traced it out completely, but take a look at the flag 
 HLS_TEMP_FILE in 

Re: [FFmpeg-devel] [FFmpeg-user] fMP4 generation speed

[Ronak Patel]

> On Jul 25, 2018, at 11:25 PM, Ronak Patel  
> wrote:
> 
> 
> 
>> On Jul 25, 2018, at 11:12 PM, Liu Steven  wrote:
>> 
>> 
>> 
>>> 在 2018年7月26日，上午7:04，Ronak Patel  写道：
>>> 
>>> Hi,
>>> 
>>> We’d like to start working on this real soon and want to make sure we 
>>> properly fix this. We understand this code is used by more than just fMP4 
>>> assets and even live streaming use cases.
>>> 
>>> Can anyone please help shed some light on all use cases we should make sure 
>>> are working after we make our fixes?
>>> 
>>> We’re afraid that the number of use cases here are unknown and we’ll be 
>>> stuck fixing this for an unknown time.
>>> 
>>> Can anyone please shed some light on this?
>> 
>> you can try -f segment or -f hls or -f dash to do waht you want try, if all 
>> of them cannot suuport your feature, you can submmit patch here.
>> or you can submmit a new hls muxer patch here if you cannot fix all the 
>> parameters for the old hls muxer.
>> 

Ok I’m going to try to fix mpegts, live streams, fmp4 and the other options 
that you already have.

Are there any unit tests or other tests we can run to ensure everything is 
still working?

If not we will have to make sure it works manually.


>>> 
>>> Thanks
>>> 
>>> Ronak
>>> 
 On Jul 13, 2018, at 8:05 AM, Ronak Patel 
  wrote:
 
 I was wondering if anyone on the development list was familiar with this 
 code and could provide pointers on how to fix up the code due to the below 
 problem.
 
 
> On Jul 12, 2018, at 7:27 PM, Ronak Patel 
>  wrote:
> 
> 
> 
>> On Jul 12, 2018, at 6:21 PM, Ronak Patel 
>>  wrote:
>> 
>> Hey Carl,
>> 
>> So I dug into this more today and I have root caused what's exactly 
>> happening here. 
>> 
>> The problematic code is this: 
>> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368 
>> 
>> This is where the filename is set and the next line actually opens the 
>> file. 
>> 
>> The logic for this hls_window method is the following:
>> 
>> 1. Make a new temporary file.
>> 2. Write out a new HLS manifest header.
>> 3. Loop through all available segments and write out all of the entries 
>> for them.
>> 4. Close the temporary file when finished.
>> 5. Rename the temporary file to the target file name.
>> 6. Rinse and repeat for every single fragment.
>> 
>> do you mean append fragment info to the m3u8 list, don’t refresh it?
> 
> Yup that’s what I mean. The fact that ffmpeg is refreshing the entire 
> manifest hundreds of thousands of times is the reason for this massive slow 
> down.
> 
> We tried changing the code a little to only write the manifest once once at 
> end, and we can now fragment a 60 hour audio file in less than one minute. 
> Less than one minute compared to 4 days is a BIG time savings.
> 
> 
>> 
>> Therefore, if you can imagine a 153 hour audio file, we write out a 
>> totally new HLS manifest 550800 times (153 * 60 * 60 assuming a 1s 
>> fragment duration) that gets progressively larger as each fragment is 
>> generated.
>> 
>> This is a classic O(N^2) algorithm implementation, instead of:
>> 
>> 1. Creating the destination file up front & write the manifest header.
>> 2. Append the new segment to the file.
>> 3. If this is the last segment, write out EXT-X-ENDLIST.
>> 
>> There's no looping involved, nor the need to make temporary files.
>> 
>> FYI that I've noticed the same sort of pattern being applied to MPEG 
>> DASH: 
>> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786 
>> 
>> 
>> To implement something like this, looks like we'd have to significantly 
>> re-engineer the code. Do you have any pointers on how to go about doing 
>> this? Or, would you be able to help do this?
>> 
> 
> I was thinking about this and I think it must have been implemented this 
> way because of live hls streams? Not writing to the main manifest 
> directly and keeping that stream open does release any file system locks 
> that there would be otherwise. 
> 
> Is this the reason why this was done this way at all?
> 
> 
>> Thanks for all your help,
>> 
>> Ronak
>> 
>>> On Jun 27, 2018, at 2:04 PM, Carl Zwanzig  wrote:
>>> 
>>> Hi,
>>> 
>>> I haven't traced it out completely, but take a look at the flag 
>>> HLS_TEMP_FILE in libavformat/hlsenc.c.
>>> 
>>> Later,
>>> 
>>> z!
>> ___
>> 
>> Thanks
>> 
>> Steven Liu
>>> 
>> 
>> 
>> 
>> ___
>> ffmpeg-devel mailing list
>> ffmpeg-devel@ffmpeg.org
>> 

Re: [FFmpeg-devel] [FFmpeg-user] fMP4 generation speed

[Ronak Patel]

> On Jul 25, 2018, at 11:12 PM, Liu Steven  wrote:
> 
> 
> 
>> 在 2018年7月26日，上午7:04，Ronak Patel  写道：
>> 
>> Hi,
>> 
>> We’d like to start working on this real soon and want to make sure we 
>> properly fix this. We understand this code is used by more than just fMP4 
>> assets and even live streaming use cases.
>> 
>> Can anyone please help shed some light on all use cases we should make sure 
>> are working after we make our fixes?
>> 
>> We’re afraid that the number of use cases here are unknown and we’ll be 
>> stuck fixing this for an unknown time.
>> 
>> Can anyone please shed some light on this?
> 
> you can try -f segment or -f hls or -f dash to do waht you want try, if all 
> of them cannot suuport your feature, you can submmit patch here.
> or you can submmit a new hls muxer patch here if you cannot fix all the 
> parameters for the old hls muxer.
> 
>> 
>> Thanks
>> 
>> Ronak
>> 
>>> On Jul 13, 2018, at 8:05 AM, Ronak Patel 
>>>  wrote:
>>> 
>>> I was wondering if anyone on the development list was familiar with this 
>>> code and could provide pointers on how to fix up the code due to the below 
>>> problem.
>>> 
>>> 
 On Jul 12, 2018, at 7:27 PM, Ronak Patel 
  wrote:
 
 
 
> On Jul 12, 2018, at 6:21 PM, Ronak Patel 
>  wrote:
> 
> Hey Carl,
> 
> So I dug into this more today and I have root caused what's exactly 
> happening here. 
> 
> The problematic code is this: 
> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368 
> 
> This is where the filename is set and the next line actually opens the 
> file. 
> 
> The logic for this hls_window method is the following:
> 
> 1. Make a new temporary file.
> 2. Write out a new HLS manifest header.
> 3. Loop through all available segments and write out all of the entries 
> for them.
> 4. Close the temporary file when finished.
> 5. Rename the temporary file to the target file name.
> 6. Rinse and repeat for every single fragment.
> 
> do you mean append fragment info to the m3u8 list, don’t refresh it?

Yup that’s what I mean. The fact that ffmpeg is refreshing the entire manifest 
hundreds of thousands of times is the reason for this massive slow down.

We tried changing the code a little to only write the manifest once once at 
end, and we can now fragment a 60 hour audio file in less than one minute. Less 
than one minute compared to 4 days is a BIG time savings.


> 
> Therefore, if you can imagine a 153 hour audio file, we write out a 
> totally new HLS manifest 550800 times (153 * 60 * 60 assuming a 1s 
> fragment duration) that gets progressively larger as each fragment is 
> generated.
> 
> This is a classic O(N^2) algorithm implementation, instead of:
> 
> 1. Creating the destination file up front & write the manifest header.
> 2. Append the new segment to the file.
> 3. If this is the last segment, write out EXT-X-ENDLIST.
> 
> There's no looping involved, nor the need to make temporary files.
> 
> FYI that I've noticed the same sort of pattern being applied to MPEG 
> DASH: 
> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786 
> 
> 
> To implement something like this, looks like we'd have to significantly 
> re-engineer the code. Do you have any pointers on how to go about doing 
> this? Or, would you be able to help do this?
> 
 
 I was thinking about this and I think it must have been implemented this 
 way because of live hls streams? Not writing to the main manifest directly 
 and keeping that stream open does release any file system locks that there 
 would be otherwise. 
 
 Is this the reason why this was done this way at all?
 
 
> Thanks for all your help,
> 
> Ronak
> 
>> On Jun 27, 2018, at 2:04 PM, Carl Zwanzig  wrote:
>> 
>> Hi,
>> 
>> I haven't traced it out completely, but take a look at the flag 
>> HLS_TEMP_FILE in libavformat/hlsenc.c.
>> 
>> Later,
>> 
>> z!
> ___
> 
> Thanks
> 
> Steven Liu
>> 
> 
> 
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [FFmpeg-user] fMP4 generation speed

[Liu Steven]

> 在 2018年7月26日，上午7:04，Ronak Patel  写道：
> 
> Hi,
> 
> We’d like to start working on this real soon and want to make sure we 
> properly fix this. We understand this code is used by more than just fMP4 
> assets and even live streaming use cases.
> 
> Can anyone please help shed some light on all use cases we should make sure 
> are working after we make our fixes?
> 
> We’re afraid that the number of use cases here are unknown and we’ll be stuck 
> fixing this for an unknown time.
> 
> Can anyone please shed some light on this?

you can try -f segment or -f hls or -f dash to do waht you want try, if all of 
them cannot suuport your feature, you can submmit patch here.
or you can submmit a new hls muxer patch here if you cannot fix all the 
parameters for the old hls muxer.

> 
> Thanks
> 
> Ronak
> 
>> On Jul 13, 2018, at 8:05 AM, Ronak Patel  
>> wrote:
>> 
>> I was wondering if anyone on the development list was familiar with this 
>> code and could provide pointers on how to fix up the code due to the below 
>> problem.
>> 
>> 
>>> On Jul 12, 2018, at 7:27 PM, Ronak Patel 
>>>  wrote:
>>> 
>>> 
>>> 
 On Jul 12, 2018, at 6:21 PM, Ronak Patel 
  wrote:
 
 Hey Carl,
 
 So I dug into this more today and I have root caused what's exactly 
 happening here. 
 
 The problematic code is this: 
 https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368 
 
 This is where the filename is set and the next line actually opens the 
 file. 
 
 The logic for this hls_window method is the following:
 
 1. Make a new temporary file.
 2. Write out a new HLS manifest header.
 3. Loop through all available segments and write out all of the entries 
 for them.
 4. Close the temporary file when finished.
 5. Rename the temporary file to the target file name.
 6. Rinse and repeat for every single fragment.

do you mean append fragment info to the m3u8 list, don’t refresh it?
 
 Therefore, if you can imagine a 153 hour audio file, we write out a 
 totally new HLS manifest 550800 times (153 * 60 * 60 assuming a 1s 
 fragment duration) that gets progressively larger as each fragment is 
 generated.
 
 This is a classic O(N^2) algorithm implementation, instead of:
 
 1. Creating the destination file up front & write the manifest header.
 2. Append the new segment to the file.
 3. If this is the last segment, write out EXT-X-ENDLIST.
 
 There's no looping involved, nor the need to make temporary files.
 
 FYI that I've noticed the same sort of pattern being applied to MPEG DASH: 
 https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786 
 
 
 To implement something like this, looks like we'd have to significantly 
 re-engineer the code. Do you have any pointers on how to go about doing 
 this? Or, would you be able to help do this?
 
>>> 
>>> I was thinking about this and I think it must have been implemented this 
>>> way because of live hls streams? Not writing to the main manifest directly 
>>> and keeping that stream open does release any file system locks that there 
>>> would be otherwise. 
>>> 
>>> Is this the reason why this was done this way at all?
>>> 
>>> 
 Thanks for all your help,
 
 Ronak
 
> On Jun 27, 2018, at 2:04 PM, Carl Zwanzig  wrote:
> 
> Hi,
> 
> I haven't traced it out completely, but take a look at the flag 
> HLS_TEMP_FILE in libavformat/hlsenc.c.
> 
> Later,
> 
> z!
 ___

Thanks

Steven Liu
> 



___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [FFmpeg-user] fMP4 generation speed

[Ronak Patel]

Hi,

We’d like to start working on this real soon and want to make sure we properly 
fix this. We understand this code is used by more than just fMP4 assets and 
even live streaming use cases.

Can anyone please help shed some light on all use cases we should make sure are 
working after we make our fixes?

We’re afraid that the number of use cases here are unknown and we’ll be stuck 
fixing this for an unknown time.

Can anyone please shed some light on this?

Thanks

Ronak

> On Jul 13, 2018, at 8:05 AM, Ronak Patel  
> wrote:
> 
> I was wondering if anyone on the development list was familiar with this code 
> and could provide pointers on how to fix up the code due to the below problem.
> 
> 
>> On Jul 12, 2018, at 7:27 PM, Ronak Patel  
>> wrote:
>> 
>> 
>> 
>>> On Jul 12, 2018, at 6:21 PM, Ronak Patel 
>>>  wrote:
>>> 
>>> Hey Carl,
>>> 
>>> So I dug into this more today and I have root caused what's exactly 
>>> happening here. 
>>> 
>>> The problematic code is this: 
>>> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368 
>>> 
>>> This is where the filename is set and the next line actually opens the 
>>> file. 
>>> 
>>> The logic for this hls_window method is the following:
>>> 
>>> 1. Make a new temporary file.
>>> 2. Write out a new HLS manifest header.
>>> 3. Loop through all available segments and write out all of the entries for 
>>> them.
>>> 4. Close the temporary file when finished.
>>> 5. Rename the temporary file to the target file name.
>>> 6. Rinse and repeat for every single fragment.
>>> 
>>> Therefore, if you can imagine a 153 hour audio file, we write out a totally 
>>> new HLS manifest 550800 times (153 * 60 * 60 assuming a 1s fragment 
>>> duration) that gets progressively larger as each fragment is generated.
>>> 
>>> This is a classic O(N^2) algorithm implementation, instead of:
>>> 
>>> 1. Creating the destination file up front & write the manifest header.
>>> 2. Append the new segment to the file.
>>> 3. If this is the last segment, write out EXT-X-ENDLIST.
>>> 
>>> There's no looping involved, nor the need to make temporary files.
>>> 
>>> FYI that I've noticed the same sort of pattern being applied to MPEG DASH: 
>>> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786 
>>> 
>>> 
>>> To implement something like this, looks like we'd have to significantly 
>>> re-engineer the code. Do you have any pointers on how to go about doing 
>>> this? Or, would you be able to help do this?
>>> 
>> 
>> I was thinking about this and I think it must have been implemented this way 
>> because of live hls streams? Not writing to the main manifest directly and 
>> keeping that stream open does release any file system locks that there would 
>> be otherwise. 
>> 
>> Is this the reason why this was done this way at all?
>> 
>> 
>>> Thanks for all your help,
>>> 
>>> Ronak
>>> 
 On Jun 27, 2018, at 2:04 PM, Carl Zwanzig  wrote:
 
 Hi,
 
 I haven't traced it out completely, but take a look at the flag 
 HLS_TEMP_FILE in libavformat/hlsenc.c.
 
 Later,
 
 z!
>>> ___
>>> ffmpeg-user mailing list
>>> ffmpeg-u...@ffmpeg.org
>>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>>> 
>>> To unsubscribe, visit link above, or email
>>> ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
>> 
>> ___
>> ffmpeg-user mailing list
>> ffmpeg-u...@ffmpeg.org
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>> 
>> To unsubscribe, visit link above, or email
>> ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
> 
> ___
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] [FFmpeg-user] fMP4 generation speed

[Ronak Patel]

I was wondering if anyone on the development list was familiar with this code 
and could provide pointers on how to fix up the code due to the below problem.


> On Jul 12, 2018, at 7:27 PM, Ronak Patel  
> wrote:
> 
> 
> 
>> On Jul 12, 2018, at 6:21 PM, Ronak Patel  
>> wrote:
>> 
>> Hey Carl,
>> 
>> So I dug into this more today and I have root caused what's exactly 
>> happening here. 
>> 
>> The problematic code is this: 
>> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/hlsenc.c#L1368 
>> 
>> This is where the filename is set and the next line actually opens the file. 
>> 
>> The logic for this hls_window method is the following:
>> 
>> 1. Make a new temporary file.
>> 2. Write out a new HLS manifest header.
>> 3. Loop through all available segments and write out all of the entries for 
>> them.
>> 4. Close the temporary file when finished.
>> 5. Rename the temporary file to the target file name.
>> 6. Rinse and repeat for every single fragment.
>> 
>> Therefore, if you can imagine a 153 hour audio file, we write out a totally 
>> new HLS manifest 550800 times (153 * 60 * 60 assuming a 1s fragment 
>> duration) that gets progressively larger as each fragment is generated.
>> 
>> This is a classic O(N^2) algorithm implementation, instead of:
>> 
>> 1. Creating the destination file up front & write the manifest header.
>> 2. Append the new segment to the file.
>> 3. If this is the last segment, write out EXT-X-ENDLIST.
>> 
>> There's no looping involved, nor the need to make temporary files.
>> 
>> FYI that I've noticed the same sort of pattern being applied to MPEG DASH: 
>> https://github.com/FFmpeg/FFmpeg/blob/master/libavformat/dashenc.c#L786 
>> 
>> 
>> To implement something like this, looks like we'd have to significantly 
>> re-engineer the code. Do you have any pointers on how to go about doing 
>> this? Or, would you be able to help do this?
>> 
> 
> I was thinking about this and I think it must have been implemented this way 
> because of live hls streams? Not writing to the main manifest directly and 
> keeping that stream open does release any file system locks that there would 
> be otherwise. 
> 
> Is this the reason why this was done this way at all?
> 
> 
>> Thanks for all your help,
>> 
>> Ronak
>> 
>>> On Jun 27, 2018, at 2:04 PM, Carl Zwanzig  wrote:
>>> 
>>> Hi,
>>> 
>>> I haven't traced it out completely, but take a look at the flag 
>>> HLS_TEMP_FILE in libavformat/hlsenc.c.
>>> 
>>> Later,
>>> 
>>> z!
>> ___
>> ffmpeg-user mailing list
>> ffmpeg-u...@ffmpeg.org
>> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
>> 
>> To unsubscribe, visit link above, or email
>> ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
> 
> ___
> ffmpeg-user mailing list
> ffmpeg-u...@ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-user
> 
> To unsubscribe, visit link above, or email
> ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel