Re: Can JS Algorithm object finally solve html5 codecs gridlock and move us forward?
Hi Ladislav
On Fri, Jan 18, 2013 at 11:51 PM, neuralll neura...@gmail.com wrote:
Hi guys
I am playing with web audio api and mozilla audio api but I got frustrated
about so much html5 potential wasted due to infinite codecs deadlock support
in browsers holding progress back for so long.
This even resulted to people writing codecs purely in JS.
Unfortunately performance is obviously slow and limiting to browsers allowing
raw data from js
And slow mainly because of huffman and imdct.
But looking at block diagrams of pretty much any widespread codec like theora
ogg mpg aac mp4
it striked me.
Since they pretty much share the same building blocs. and which themself are
not that much patent encumbered.
So why not just introduce new Algorithm object in JS
and just register huffman idct imdct filterbank in it. for browsers its easy
task they just make visible already heavily hw optimized pretty much 4 or so
functions that are already in them
I've been trying to address the performance bottlenecks for signal
processing on the web platform a lot lately. Imho, you're on the right
track, approaching this from the angle of adding basic building blocks
rather than complete solutions.
One significant addition for ES6 is Number#clz(), which is currently a
rather significant bottleneck in our codecs.
Aside from that, at Audio WG [1] we're working on a DSP API [2] that
in best case scenarios will give you very close to (or sometimes even
better than) native performance for vector processing. Currently the
DSP API includes the DSP interface (add, sub, mul, div, ramp, etc),
the FFT interface and a Filter interface that lets you specify
arbitrary coefficients, optimizing the algorithms under the hood based
on whether you have a biquad filter or long convolution or whatever.
It might be worth adding DCT/MDCT to the DSP API, but actually I
wonder if at least DCT can just be made a special case of FFT without
being very wasteful. This needs some thinking. Anyway, would be great
to get some implementer interest for the DSP API. Current
implementations include a JS polyfill, my node C++ module
implementation ('dsp' on npm) and a partial SpiderMonkey prototype
implementation by Jens Nockert.
Huffman isn't directly related to signals so that probably doesn't
belong in the DSP API. I'm not convinced it really makes sense to have
a native implementation for it anyway... However, wasn't there some
Archive API in the works or something that might have something
related?
David mentioned mentioned parallelization, but at least for audio that
doesn't make much sense, at least unless you're trying to decode all
of a few hours long track into memory as fast as possible, which isn't
a likely scenario. That said, we're moving aurora.js to do the
decoding off the main thread to be less vulnerable to audio interrupts
and blocking the main thread as little as possible, but so far it has
increased our overall CPU usage (as anticipated), because with the
current Audio APIs, we have to transfer the decoded audio to the main
thread for playback. Transferables help significantly, but that
problem has to be solved at the Audio API level.
Cheers,
Jussi
[1] http://www.w3.org/2011/audio/
[2] http://people.opera.com/mage/dspapi/
___
es-discuss mailing list
es-discuss@mozilla.org
https://mail.mozilla.org/listinfo/es-discuss
Can JS Algorithm object finally solve html5 codecs gridlock and move us forward?
Hi guys I am playing with web audio api and mozilla audio api but I got frustrated about so much html5 potential wasted due to infinite codecs deadlock support in browsers holding progress back for so long. This even resulted to people writing codecs purely in JS. Unfortunately performance is obviously slow and limiting to browsers allowing raw data from js And slow mainly because of huffman and imdct. But looking at block diagrams of pretty much any widespread codec like theora ogg mpg aac mp4 it striked me. Since they pretty much share the same building blocs. and which themself are not that much patent encumbered. So why not just introduce new Algorithm object in JS and just register huffman idct imdct filterbank in it. for browsers its easy task they just make visible already heavily hw optimized pretty much 4 or so functions that are already in them And then suddenly you can have theora ogg flac acc or pretty much any media decoded in js in all browsers but with speeds approaching those on native hw. just look at incredible work of talented guys at official.fm and their js codecs even for flac Another advantage of hw accelerated thing like huffman in js is that it sits in pretty much every block of data around us. compressed files like gz bz2 cab jar zip rar 7z images pdf jpg png tiff and pretty much any music or movie format. This can create interesting new usage possibilities for JS and heavily complement new things like web-sockets faster transport or webgl custom textures / compressed meshes or fileapi custom format decoders / renderers in js So even thou its just 4 functions (that are already implemented just need to be registered in JS) and the impact on html5 and web future landscape can be interesting. All that is needed is agreement on content of such Algorithm object it can surely promote current or even new open unencumbered codecs in all browsers. I bet small but powerful teams like opera guys will be more than happy and way better than risking another gif like threat from current patent submarine formats. So what you guys think about such new Algorithm object and registration of those existing 4 funcs in new JS Ecmascript6 ? Ladislav Nevery Sent from my iPad ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
Re: Can JS Algorithm object finally solve html5 codecs gridlock and move us forward?
Hi Ladislav, Le 18/01/2013 22:51, neuralll a écrit : Hi guys I am playing with web audio api and mozilla audio api but I got frustrated about so much html5 potential wasted due to infinite codecs deadlock support in browsers holding progress back for so long. This even resulted to people writing codecs purely in JS. Unfortunately performance is obviously slow and limiting to browsers allowing raw data from js How slow? I'm not familiar with the performance story here and I wonder how slow it is, for which sound quality on a modern browser with normal hardware, etc. Do transferables WebWorkers make the story better? (I don't know how much these algorithms work in parallel) What is the slowness due to in current browsers? Is there a known bottleneck in current browsers implementations? And slow mainly because of huffman and imdct. But looking at block diagrams of pretty much any widespread codec like theora ogg mpg aac mp4 it striked me. Since they pretty much share the same building blocs. and which themself are not that much patent encumbered. So why not just introduce new Algorithm object in JS and just register huffman idct imdct filterbank in it. for browsers its easy task they just make visible already heavily hw optimized pretty much 4 or so functions that are already in them And then suddenly you can have theora ogg flac acc or pretty much any media decoded in js in all browsers but with speeds approaching those on native hw. What is the expected performance gains? And I expect numbers as an answer, hopefully backed by some benchmarks. just look at incredible work of talented guys at official.fm and their js codecs even for flac So... that's a JS codec? I think if they made it work, it's basically an argument in favor of *not* adding these algorithms to the language. Reading a bit more about it [1]. Quotes: With [these JS libraries], it is possible to play MP3, Apple Lossless, FLAC, and AAC even in browsers without native support. With JSMad we showed that decoding audio in JavaScript is possible, courtesy of these APIs and the hard work of browser creators and spec authors. I'm kind of puzzled. They made it work to a level they find acceptable with current browser capabilities. What else do you need? David [1] http://labs.official.fm/codecs/ ___ es-discuss mailing list es-discuss@mozilla.org https://mail.mozilla.org/listinfo/es-discuss
