Re: [FFmpeg-devel] Stream selection algorithm crossing program boundaries

[Gyan Doshi]

Hi Devin,

On 5/18/2018 9:17 PM, Devin Heitmueller wrote:

Indeed when I dug into the code it became clear that there is added complexity when multiple input files are involved, if for no other reason than it’s not immediately clear which input stream comes from which file and hence it’s difficult to know which AVFormatContext to be looking at for a given stream. 


Since the proposed change will have to be in open_output_file() in 
ffmpeg_opt.c, you can check InputStream->file_index or do you mean 
something else?



I suppose a question worth asking is whether a user would really be passing in 
multiple files where the streams are already arranged into programs and expect 
the audio to come from a different input file by default?


Based on the thousands of users I've engaged with, at SE sites, most 
users have little to middling familiarity with ffmpeg, have cribbed and 
perhaps adjusted a command from a similar operation discussed elsewhere, 
and would be unaware of whether a file sports Programs or otherwise.


if the user had clear and specific needs, they would supply -map and 
this conversation is moot. We are talking about those users relying 
either on ffmpeg "magic" or traditional behaviour they have encountered 
in past use. If multiple inputs are specified, that's a good clue that 
all inputs are intended for some use, and to me, of higher salience than 
presence of Programs, because the former is an express declaration by 
the user to ffmpeg of the resources required. The latter may be an 
incidental property of the input of which the user may or may not be aware.


Regards,
Gyan
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] Stream selection algorithm crossing program boundaries

[Devin Heitmueller]

Hello Gyan,

Thanks for your comments.  See inline:

> On May 18, 2018, at 10:24 AM, Gyan Doshi  wrote:
> If nb_input_files is 1, coupling the stream selection seems to me an 
> improvement. But in the scenario of multiple inputs and no -map options, 
> users usually intend to want a stream from all the sources, so in that case, 
> using one of the streams as a 'key' for the other streams will break 
> expectations. Unless the other inputs are meant for filtering, like a 
> watermark, or background music.

Indeed when I dug into the code it became clear that there is added complexity 
when multiple input files are involved, if for no other reason than it’s not 
immediately clear which input stream comes from which file and hence it’s 
difficult to know which AVFormatContext to be looking at for a given stream.  I 
need the AVFormatContext to look at the program list, as well as potentially 
reusing av_find_best_stream() if possible.

I suppose a question worth asking is whether a user would really be passing in 
multiple files where the streams are already arranged into programs and expect 
the audio to come from a different input file by default?  My instinct would be 
that even if he/she passed in multiple files, the default behavior should 
probably be to pick the audio associated with the video as defined by the 
program map (and render no audio if none is in the corresponding program).  Of 
course the user can always use -map to do something different.

I’m tempted to make the heuristic something like the following pseudocode:

If (nb_programs > 0) {
  // Streams arranged into programs, so find first program
  // Pick first video from that program
  // Call av_find_best_stream() to find best audio for that video
} else {
  // Use existing heuristic
}

With the above approach, we’re taking for granted that if the source input has 
been arranged into programs, then the author had some preferences in mind with 
regards to which streams to use by default, and thus by default we should 
respect those decisions rather than trying to guess which streams to play (even 
if he/she happened to specify multiple input files).  If the content isn’t 
arranged into programs then there’s no deterministic way to know which to play 
by default, so fall back to the current algorithm.  And of course if that’s not 
what the user wants then he/she can always use -map.

If the above is unacceptable, I’m not against making the new heuristic only if 
both the file contains one or more programs and there is only a single input 
file.  In fact the code for that is actually easier to implement (since I only 
have to ever look at the AVFormatContext for the first input and not have to 
try to correlate streams back to their corresponding input files).  However I’m 
not sure if the behavior should really vary based on whether multiple input 
files are specified.


> Your query has come at an inopportune time, since I intended to rewrite the 
> Stream Selection chapter this weekend, since there are some quirks and 
> nuances not covered or made plain, at present. But if you intend to proceed, 
> of course, I'll wait.

If we can come to a consensus on what the behavior should be (and believe it 
will be accepted upstream), then I can take a crack at the actual 
implementation.

Devin

___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

Re: [FFmpeg-devel] Stream selection algorithm crossing program boundaries

[Gyan Doshi]

On 5/18/2018 6:50 PM, Devin Heitmueller wrote:


Is this something anyone is actively looking into?  If not, I'll dig into it and see 
about respecting program membership in ffmpeg's selection algorithm.  My larger concern 
is that with MPEG-TS streams, the commonly accepted heuristic is to play the first 
audio/video streams in the first PMT entry (this is what VLC does for example), rather 
than trying to pick the "best quality" streams, and it's not clear whether such 
a change in the selection algorithm would be accepted upstream (since it would make 
selection of audio/video in MPEG-TS streams behave differently than other formats).

Thoughts?


If nb_input_files is 1, coupling the stream selection seems to me an 
improvement. But in the scenario of multiple inputs and no -map options, 
users usually intend to want a stream from all the sources, so in that 
case, using one of the streams as a 'key' for the other streams will 
break expectations. Unless the other inputs are meant for filtering, 
like a watermark, or background music.



Your query has come at an inopportune time, since I intended to rewrite 
the Stream Selection chapter this weekend, since there are some quirks 
and nuances not covered or made plain, at present. But if you intend to 
proceed, of course, I'll wait.


Regards,
Gyan
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

[FFmpeg-devel] Stream selection algorithm crossing program boundaries

[Devin Heitmueller]

Hello all,

By default, ffmpeg will pick the "best" audio and video streams if the user 
doesn’t explicitly indicate such using the “-map” argument.  However it 
completely ignores which programs streams are a part of, which can lead to 
cases where the video is picked from one program and the audio is from a 
completely different program.

Is this something anyone is actively looking into?  If not, I'll dig into it 
and see about respecting program membership in ffmpeg's selection algorithm.  
My larger concern is that with MPEG-TS streams, the commonly accepted heuristic 
is to play the first audio/video streams in the first PMT entry (this is what 
VLC does for example), rather than trying to pick the "best quality" streams, 
and it's not clear whether such a change in the selection algorithm would be 
accepted upstream (since it would make selection of audio/video in MPEG-TS 
streams behave differently than other formats).

Thoughts?

Devin

---
Devin Heitmueller - LTN Global Communications
dheitmuel...@ltnglobal.com




___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel