On Tue, Mar 10, 2015 at 8:29 PM, Randell Jesup <[email protected]> wrote:
> What there isn't currently is a vision for how video processing will > occur, and how we'll deal with some of the cases mentioned by roc. It can > be done very painfully once we land Canvas->MS input, by doing MS -> > video_element -> Canvas (and process) -> MS. This isn't great, and doesn't > help with things like synchronization, etc. (And maybe Canvas will be part > of the final solution space, but we don't know yet.) > > I'd love to know where we want to go for with video processing and > synchronization. That may be a orthogonal issue; if so, great. I'm not > sure it's *all* orthogonal, though. > Under the FoxEye project there's a proposal for Worker-based video processing. It doesn't require any kind of MSG blocking though. > In this context, input streams are, in real life, very unlikely to be >> unable to deliver data, and not delivering data is certainly not the >> common case, so it should not be something that is expensive to compute >> b(or computed at all). I think it can be handled just outside the graph >> without much problems and less complexity: >> > > (Audio) Sources will be one of: realtime (mic, RTCPeerConnection, > WebAudio), or non-realtime (streaming, perhaps some dynamically-generated > data). Note that realtime sources *shouldn't* underrun, but they can. > Also note that mic sources may involve a time domain boundary and thus > resampling (and in some odd cases, transitioning data between > differently-clocked subgraphs could cause "realtime" sources to face > underrun, if they aren't resampled. > > - Microphone are going to come in sync with audio in the rather near >> future (once we do full-duplex audio streams), and the graph is driven >> by the audio stream, so it cannot under-run. >> > > This assumes that we have one output and that all sources are synced to > that output, or that streams never cross output time domain boundaries. > I'm not sure this will be the case moving forward, especially as we start > to enable output selection and multiple outputs. > I hope we can avoid using blocking to handle this. We can time-stretch audio instead. > > Additionally, blocking is (part of) what makes the MSG complicated to >> read and reason about: for example, getting the current time for a >> stream is a non-straightforward operation where it should be a >> subtraction. >> >> Pragmatically, I think we should look into removing "blocking" from the >> MSG, to move toward a piece of code that does one thing fast, easier to >> read. >> > > I admit I'd love to remove blocking, or simplify it. I do want to make > sure we don't turn around and make things that much or more harder for > ourselves elsewhere or in the future, or force a bunch of boilerplate code > to exist in lots of places where it can be gotten wrong. (see some of > Derf's comments). Perhaps some standardized helper class would simplify > the job of supporting this in sources/sinks. > > Can we flesh out a bit more of exactly what removing it would result in? > Especially for the cases Benjamin/etc brought up? How much would multiple > outputs (and multiple inputs) and streams that bridge output (clock) > domains complicate things? > Can we save MSE videos without reencoding? I would hope so, in which case MSG is probably not involved. Looking again at my original message, use-case class #1 can be handled by labeling dead-time video/audio as such. Can someone go on record as saying classes #2 and #3 aren't worth worrying about? Rob -- oIo otoeololo oyooouo otohoaoto oaonoyooonoeo owohooo oioso oaonogoroyo owoiotoho oao oboroootohoeoro oooro osoiosotoeoro owoiololo oboeo osouobojoeocoto otooo ojouodogomoeonoto.o oAogoaoiono,o oaonoyooonoeo owohooo osoaoyoso otooo oao oboroootohoeoro oooro osoiosotoeoro,o o‘oRoaocoao,o’o oioso oaonosowoeoroaoboloeo otooo otohoeo ocooouoroto.o oAonodo oaonoyooonoeo owohooo osoaoyoso,o o‘oYooouo ofooooolo!o’o owoiololo oboeo oiono odoaonogoeoro ooofo otohoeo ofoioroeo ooofo ohoeololo. _______________________________________________ dev-media mailing list [email protected] https://lists.mozilla.org/listinfo/dev-media
