HTML imports in Firefox
I was a little surprised to read Mozilla don't plan on shipping HTML imports and will re-evaluate after modules are supported: https://hacks.mozilla.org/2014/12/mozilla-and-web-components/ Why would modules affect the decision to ship HTML imports? Imports get you: - the ability to import style and markup, not just script - a tree-like dependency structure with automatic deduplication - parallel loading/parsing - ability to start fetching sub-resources before script has parsed and run - tidy markup (sub-resources go in their import documents, not all dumped in to head) I don't see how modules has any impact on this. In a web environment where components will commonly use style, templates and the like, imports seem like the obvious choice over modules. The webcomponents.org polyfill for imports has a couple of caveats, so it doesn't look like it's totally equivalent and portable with browsers with native support, like Chrome which has shipped it already. I'm keen to see Mozilla ship this feature too. Ashley
Re: HTML imports in Firefox
On 15 December 2014 at 19:09, Boris Zbarsky bzbar...@mit.edu wrote: But more to the point, we're not shipping imports because we've gotten feedback from a number of people that imports are not solving the problems they actually need solved. We'd prefer to not ship imports and then need to ship yet another HTML import system that solves those problems. Well, imports work better for us than Javascript modules, for the reasons I gave. I hadn't given any feedback because everything looked great with HTML imports and I was simply waiting for it to arrive in browsers. Maybe the process biases feedback towards the negative? I guess you never hear the chorus of cool, can't wait! from everyone looking forwards to it? On 15 December 2014 at 19:09, Boris Zbarsky bzbar...@mit.edu wrote: On 12/15/14, 1:10 PM, Ashley Gullen wrote: Why would modules affect the decision to ship HTML imports? Because the interaction of the various import systems with each other needs to be specified, for one thing. But more to the point, we're not shipping imports because we've gotten feedback from a number of people that imports are not solving the problems they actually need solved. We'd prefer to not ship imports and then need to ship yet another HTML import system that solves those problems. The webcomponents.org http://webcomponents.org polyfill for imports has a couple of caveats, so it doesn't look like it's totally equivalent and portable with browsers with native support, like Chrome which has shipped it already. Chrome has shipped a lot of things in this space already. Feel free to mail me privately for my feelings on the matter. chrome shipping something is not sufficient reason to ship something we're pretty sure is deficient, I'm afraid. -Boris
Re: do not deprecate synchronous XMLHttpRequest
I am on the side that synchronous AJAX should definitely be deprecated, except in web workers where sync stuff is OK. Especially on the modern web, there are two really good alternatives: - write your code in a web worker where synchronous calls don't hang the browser - write async code which doesn't hang the browser With modern tools like Promises and the new Fetch API, I can't think of any reason to write a synchronous AJAX request on the main thread, when an async one could have been written instead with probably little extra effort. Alas, existing codebases rely on it, so it cannot be removed easily. But I can't see why anyone would argue that it's a good design principle to make possibly seconds-long synchronous calls on the UI thread. On 9 February 2015 at 19:33, George Calvert george.calv...@loudthink.com wrote: I third Michaela and Gregg. It is the app and site developers' job to decide whether the user should wait on the server — not the standard's and, 99.9% of the time, not the browser's either. I agree a well-designed site avoids synchronous calls. BUT — there still are plenty of real-world cases where the best choice is having the user wait: Like when subsequent options depend on the server's reply. Or more nuanced, app/content-specific cases where rewinding after an earlier transaction fails is detrimental to the overall UX or simply impractical to code. Let's focus our energies elsewhere — dispensing with browser warnings that tell me what I already know and with deprecating features that are well-entrenched and, on occasion, incredibly useful. Thanks, George Calvert
Showing dialog from HTML import
I filed crbug.com/458799 for Chrome recently since showing a dialog defined in a HTML import did not work as I thought it should. I can't compare to any other browsers since right now AFAIK Chrome is the only browser shipping both HTML imports and dialog. There seemed to be some confusion from the browser developers over what the spec says ought to happen. It sounds like the current spec says that showing a dialog marks the node document of the dialog as being blocked. I would imagine a dialog in a HTML import has the invisible HTML import document as the node document of the dialog. Therefore it never appears on-screen (in the main document) when you call showModal(), but still throws if you try to open it twice. It can be worked around with JS: the dialog element can be detached from its current document and appended to the main document, and then showModal() works as expected. I think the spec may need to be altered. I think it's an obvious use case that a web component may want to show a dialog, and naturally that dialog will be defined in one of the web component's HTML imports. As it stands though the dialog will never appear unless the JS hack is used. I can also foresee that a large codebase making use of lots of dialogs ends up with loads of dialogs appended to the main document, instead of leaving them in the components where they came from. Not sure how this would be specified though - perhaps the dialog should be attached to the main document in showModal(), then re-inserted where it came from in close()? Sounds like a bit of a hack in itself... Ashley
Re: Showing dialog from HTML import
Filed: https://www.w3.org/Bugs/Public/show_bug.cgi?id=28051 Ashley On 18 February 2015 at 15:29, Dimitri Glazkov dglaz...@google.com wrote: Can you file a spec bug? https://www.w3.org/Bugs/Public/enter_bug.cgi?comment=blocked=20683short_desc=%5Bimports%5D%3A%20product=WebAppsWGimport=import%20Model :DG On Wed, Feb 18, 2015 at 5:18 AM, Ashley Gullen ash...@scirra.com wrote: I filed crbug.com/458799 for Chrome recently since showing a dialog defined in a HTML import did not work as I thought it should. I can't compare to any other browsers since right now AFAIK Chrome is the only browser shipping both HTML imports and dialog. There seemed to be some confusion from the browser developers over what the spec says ought to happen. It sounds like the current spec says that showing a dialog marks the node document of the dialog as being blocked. I would imagine a dialog in a HTML import has the invisible HTML import document as the node document of the dialog. Therefore it never appears on-screen (in the main document) when you call showModal(), but still throws if you try to open it twice. It can be worked around with JS: the dialog element can be detached from its current document and appended to the main document, and then showModal() works as expected. I think the spec may need to be altered. I think it's an obvious use case that a web component may want to show a dialog, and naturally that dialog will be defined in one of the web component's HTML imports. As it stands though the dialog will never appear unless the JS hack is used. I can also foresee that a large codebase making use of lots of dialogs ends up with loads of dialogs appended to the main document, instead of leaving them in the components where they came from. Not sure how this would be specified though - perhaps the dialog should be attached to the main document in showModal(), then re-inserted where it came from in close()? Sounds like a bit of a hack in itself... Ashley
Standardising canvas-driven background images
Forgive me if I've missed past discussion on this feature but I need it so I'm wondering what the status of it is. (Ref: https://www.webkit.org/blog/176/css-canvas-drawing/ and http://updates.html5rocks.com/2012/12/Canvas-driven-background-images, also known as -webkit-canvas() or -moz-element()) The use case I have for it is this: we are building a large web app that could end up dealing with thousands of dynamically generated icons since it deals with large user-generated projects. The most efficient way to deal with this many small images is to basically sprite sheet them on to a canvas 2d context. For example a 512x512 canvas would have room for a grid of 256 different 32x32 icons. (These are drawn scaled down from user-generated content, so they are not known at the time the app loads and so a normal image cannot be used.) To display an icon, a 32x32 div sets its background image to the canvas at an offset, like a normal CSS sprite sheet but with a canvas. -webkit-canvas solves this, but I immediately ran in to bugs (in Chrome updating the canvas does not always redraw the background image), and as far as I can tell it has an uncertain future so I'm wary of depending on it. The workarounds are: - toDataURL() - synchronous so will jank the main thread, data URL inflation (+30% size), general insanity of dumping a huge string in to CSS properties - toBlob() - asynchronous which raises complexity problems (needs a way of firing events to all dependent icons to update them; updating them requires DOM/style changes; needs to handle awkward cases like the canvas changing while toBlob() is processing; needs to be carefully scheduled to avoid thrashing toBlob() if changes being made regularly e.g. as network requests complete). I also assume this uses more memory, since it effectively requires creating a separate image the same size which is stored in addition to the canvas. In comparison being able to put a canvas in a background images solves this elegantly: there is no need to convert the canvas or update the DOM as it changes, and it seems the memory overhead would be lower. It also opens up other use cases such as animated backgrounds. I see there may be security concerns around -moz-element() since it can use any DOM content. This does not appear to be necessary or even useful (what use cases is arbitrary DOM content for?). If video is desirable, then video can already be rendered to canvases, so -webkit-canvas still covers that. Therefore I would like to propose standardising this feature based off the -webkit-canvas() implementation. Ashley Gullen Scirra.com
Re: Privileged context features and JavaScript
I think the existence of the API functions should indicate the browser has the capability, and then the API returns an error if it's not allowed to be used in the current context. I think this would improve the quality of messages seen by users, since for example removing the Geolocation API entirely could result in a web page message like sorry, your browser does not support Geolocation, try updating your browser - which is incorrect, it really should say something like geolocation permission denied. On 17 April 2015 at 08:38, Elliott Sprehn espr...@chromium.org wrote: It's preferable not to do that for us because you can then create a static heap snapshot at compile time and memcpy to start JS contexts faster. On Apr 17, 2015 12:03 AM, Boris Zbarsky bzbar...@mit.edu wrote: On 4/17/15 2:52 AM, Boris Zbarsky wrote: If that preference is toggled, we in fact remove the API entirely, so that 'geolocation' in navigator tests false. Oh, I meant to mention: this is more web-compatible than having the API entrypoints throw, because it can be object-detected. Of course we could have made the API entrypoints just always reject the request instead, I guess; removing the API altogether was somewhat simpler to do. -Boris
Re: PSA: publishing new WD of Gamepad on April 14
Why doesn't the Gamepad API fire events for button pushes or axis movements? For example when pressing a mouse button or moving the mouse the browser fires mousedown and mousemove. The Gamepad API however requires you to passively poll the state regularly (probably in rAF) and look for changes yourself. Why does it not fire events like gamepadbuttondown or gamepadaxischange? This would have a few advantages: - it would be consistent with the way all other input events are handled on the web platform - it is easier to program for. As it stands since there is nothing like a gamepadbuttondown event, so if you want one, you have to implement it yourself by polling the state, keeping the previous polled state, and comparing the differences looking for a previously up but currently down state and then run your handler. - browsers have a couple of important features that can only work in a user gesture, such as opening a popup window, copying to the clipboard, or - critically for games! - starting audio (or video) playback on mobile. Since the Gamepad API does not fire events, this does not integrate nicely with the existing user gesture model, and therefore currently no browser allows these features to be triggered by gamepad input. Considering the use case of a gamepad controlling a browser game on a tablet, it's pretty embarrassing that you can't play audio without resorting to some other kind of input, like regularly leaning forwards to touch the screen. This could involve significant changes to the spec, but I think it's necessary. It looks a bit like a first draft that never got reconsidered. Ashley Gullen Scirra.com On 9 April 2015 at 12:52, Arthur Barstow art.bars...@gmail.com wrote: Hi All, A new Working Draft publication of Gamepad is planned for April 14 using the following version as the basis: https://w3c.github.io/gamepad/publish/gamepad.html If anyone has any major concerns about this, please reply right away. A few notes about this spec: * This spec is now using Github https://github.com/w3c/gamepad and the ED is https://w3c.github.io/gamepad/gamepad.html. PRs are welcome and encouraged. * The permissions of the spec's now obsolete Hg repository will be set to read-only. * After Ted copies the open Bugzilla bugs to Github, the spec's Bugzilla component will be marked `Historical` and set to read-only. -Thanks, ArtB
Re: Clipboard API: remove dangerous formats from mandatory data types
The browser could copy a terminal command to wipe the disk, and the user could run it. Or copy a URL to a web page that has a known security exploit, and then the user pastes it in to the address bar. Maybe we shouldn't allow copying anything at all? Actually I think this is just security by paranoia. If pasting corrupt images in to another app is enough of a problem to warrant concern, that app should be fixed, rather than hobble the browser. On 9 Jun 2015 21:23, Daniel Cheng dch...@google.com wrote: On Tue, Jun 9, 2015 at 12:27 PM Paul Libbrecht p...@hoplahup.net wrote: Daniel, this does not make sense to me. All these image parsers exploits can be triggered by an img tag, or? Similarly for XML using an XHR or some particular XML formats (RSS, SVG, XHTML, ...) in markup. Sure. That's why Chrome only decodes images in sandboxed processes like the renderer. There's absolutely no difference in the mistrust we should have between content brought by an HTML page and content brought by a JavaScript, or? Hence we should just not accept the reason of knowing of broken parsers to be a reason to change the standards! The difference is that copy image to clipboard never writes a GIF/JPG/PNG to the clipboard. The trusted browser process grabs the raw decoded bitmap from the renderer, validates the size information and some other stuff, and then writes it to the clipboard as a bitmap of the appropriate format. On the other hand, supporting this from JS means the untrusted renderer would get to control and place arbitrary GIF/JPG/PNG into the clipboard. It's not really feasible to do any corresponding validation that the data isn't bad. If, as a president, you need to decide to change the roads because a particular car was built massively and needs a particularirty of your roads, you would also find it nonsense, or? You're making me feel like France which did this for a particular type of trains which required to change all platforms because their ordered trains were already built too wide It's unfortunate, but I doubt any native apps attempt to securely decode images. Previously, it didn't matter, since the clipboard content came from other native apps, and if you can't trust other native apps, you're kind of hosed anyway. However, going from web content - native crosses a security boundary, which means these sort of issues need to be taken into consideration. Paul On 9/06/15 21:15, Daniel Cheng wrote: I'm not against considering more formats to be dangerous. =) In particular: JS: I'm not support what context we'd ever want to support this, since we go out of our way to try prevent XSS in HTML pastes. XML: I wouldn't mind getting rid of this. XML parsers seem to have RCE bugs on a semi-regular basis. Daniel On Tue, Jun 9, 2015 at 12:01 PM Olli Pettay o...@pettay.fi wrote: On 06/09/2015 09:39 PM, Daniel Cheng wrote: Currently, the Clipboard API [1] mandates support for a number of formats. Unfortunately, we do not believe it is possible to safely support writing a number of formats to the clipboard: - image/png - image/jpg, image/jpeg - image/gif If these types are supported, malicious web content can trivially write a malformed GIF/JPG/PNG to the clipboard and trigger code execution when pasting in a program with a vulnerable image decoder. This provides a trivial way to bypass the sandbox that web content is usually in. Given this, I'd like to propose that we remove the above formats from the list of mandatory data types, and avoid adding support for any more complex formats. Daniel [1] http://www.w3.org/TR/clipboard-apis/#mandatory-data-types-1 Why would text/html, application/xhtml+xml, image/svg+xml, application/xml, text/xml, application/javascript be any safer if the program which the data is pasted to has vulnerable html/xml/js parsing? -Olli
Re: Async Image - ImageData conversion
I was advised to mark this as an Editor's draft, and have done so. I also
added a new proposed ImageData.fromImage method, with rationale included in
the document. It's at the same URL:
https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
On 19 June 2015 at 13:43, Ashley Gullen ash...@scirra.com wrote:
I've not done this before, so I've no idea if this is the right/useful
approach, but I drafted a spec for it here:
https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
Let me know if you have any feedback on this.
On 18 June 2015 at 22:27, Travis Leithead travis.leith...@microsoft.com
wrote:
Cool. Want to write it up as a small extension spec? We have a new
Incubator Community Group coming up soon that would be an ideal place to
drop a small spec into and let implementers experiment with it and get
broader community feedback on the idea.
*From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
Gullen
*Sent:* Wednesday, June 17, 2015 2:06 PM
*To:* Travis Leithead
*Cc:* public-webapps@w3.org
*Subject:* Re: Async Image - ImageData conversion
That seems like a good start. I suppose there should be a
putImageDataAsync counterpart too? Then we can do:
Blob - Image via: load blob URL
Blob - ImageData via: load blob URL - Canvas drawImage
- getImageDataAsync
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - getImageDataAsync
ImageData - Blob via: Canvas putImageDataAsync - Canvas toBlob
ImageData - Image via: Canvas putImageDataAsync - Canvas toBlob - load
blob URL
I think the potential problems with this are:
- putImageDataAsync is the logical counterpart to getImageDataAsync, but
what happens if you make other synchronous draw calls while
putImageDataAsync is processing? What actually ends up in the canvas and
how is this defined?
- some of the conversion steps seem pretty long winded, in particular
Blob - ImageData and ImageData - Image. If implementors think they can
optimise these to be as efficient as direct conversion that's fine, but is
that actually doable with such a roundabout API?
If ImageData has:
PromiseBlob toBlob();
PromiseImageData fromBlob(blob);
then we can use the following conversions without new canvas functions:
Blob - Image via: load blob URL
Blob - ImageData via: ImageData.fromBlob
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - Canvas toBlob -
ImageData.fromBlob
ImageData - Blob via: ImageData.toBlob
ImageData - Image via: ImageData.toBlob - load blob URL
That looks like a much more reasonable set of conversions to me, and is
all async.
I was thinking about toImage and fromImage, but it seems to be a bigger
step to add a new way to make images, and there's the question of what the
src for the image returned by toImage should be.
On 17 June 2015 at 18:59, Travis Leithead travis.leith...@microsoft.com
wrote:
I think solving at least the first-order problem of extracting data
from the Canvas async is do-able.
Something like:
PromiseImageData getImageDataAsync(x,y,w,h);
seems sensible to add J
*From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
Gullen
*Sent:* Wednesday, June 17, 2015 10:00 AM
*To:* public-webapps@w3.org
*Subject:* Async Image - ImageData conversion
I was wondering if there is anything on the standards track to
asynchronously get an ImageData object from an Image?
We have a web app dealing with large sprite sheets (up to 2048x2048), and
at some point we need to get an ImageData for this. Currently the only way
I know of doing this is via a canvas 2d context like so:
function ImageToImageData(img)
{
let w = img.width;
let h = img.height;
let canvas = document.createElement(canvas);
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext(2d);
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};
This whole function is synchronous. With large images, in Chrome the
getImageData call can jank for well over 100ms on a high-end desktop
machine. Maybe this can be faster but I really doubt this whole function is
likely to be done in 16ms on low-end devices, so I believe there ought to
be an async alternative. As far as I am aware there is none.
Ideally we could have an async ImageData.fromImage(img) method or
similar. It would also be useful if we could decode directly from Blob to
ImageData without having to go via an Image as well (perhaps
ImageData.fromBlob(blob) ?).
More broadly, it would be good to have a comprehensive set of async
conversion functions between Blob, Image, and ImageData. Blob - Image can
already be done by setting the image src to a blob URL, but basically
everything else requires going synchronously through a 2D
Re: Async Image - ImageData conversion
I see that OffscreenCanvas also specifies a close() method for ImageBitmap. Perhaps browsers could use copy-on-write with the toImageData() method? Then you can efficiently either transfer or copy depending on if you close the ImageBitmap you transferred to an ImageData. Transfer: createImageBitmap() - imageBitmap.toImageData() - imageBitmap.close() No copy made, contents transferred to the imageData createImageBitmap() - imageBitmap.toImageData() - access resulting imageData.data Copy made on first access On 24 June 2015 at 21:54, Kenneth Russell k...@google.com wrote: On Wed, Jun 24, 2015 at 1:28 PM, Ashley Gullen ash...@scirra.com wrote: Sorry for the confusion. Yes, the latest URL is: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? (I updated the URL to say that instead) The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. However I'm not sure what the point of that would be, since it seems to be how drawing HTMLImageElements to a canvas already works. I see a couple of options: - allow lazy decoding in ImageBitmap, and therefore converting to ImageData is an explicit request to decompress - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. Therefore, converting to ImageData indicates the intent to modify this content, therefore a copy is warranted. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. Perhaps this could be done by specifying a transferToImageData method. This proposal makes ImageBitmap neuterable, and there's intent to implement it in order to allow rendering to canvas contexts from worker threads: https://wiki.whatwg.org/wiki/OffscreenCanvas . -Ken - add some kind of load method on ImageBitmap. It may store its contents in compressed form, but calling load causes it to be decompressed (or loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn with undue latency after the load call. Therefore for the use case of low-latency rendering load can always be called immediately after creation, but for conversion purposes not calling load avoids duplicating memory. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. In this case making a copy for the ImageData is also warranted. I'm a web developer proposing this because it would be useful for my purposes, I've no idea which of these would be favoured by implementors or the spec process. I'm happy to get involved in spec work though so please let me know if you have any feedback/suggestions on anything I'm doing here. Ashley On 24 June 2015 at 19:12, Boris Zbarsky bzbar...@mit.edu wrote: On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Ashley, We at Mozilla were just discussing this proposal, and we have a concern. With this proposal, going from an img to an ImageData requires conversion of an intermediate ImageBitmap. Unfortunately, an ImageBitmap is specified to be paintable without undue latency, which we interpret to mean it needs to have decoded image data, and the ImageData will end up needing to copy said data in practice (because it needs an editable copy). That creates two copies of the decoded image data in memory, which seems fairly undesirable on memory-constrained devices. -Boris
Re: Subject=Re: Async Image - ImageData conversion
Good point - it makes sense to have ImageBitmap as the hub of all conversion. I've drafted a new spec which instead proposes ImageBitmap.toBlob() and ImageBitmap.toImageData(): https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html This should cover all conversion cases asynchronously. On 23 June 2015 at 20:34, Justin Novosad ju...@google.com wrote: Based on feedback received from web developers, new APIs that move image data around should also strive to eliminate intermediate copies of the image data to avoid memory bloat. I think your proposal achieves that, but I think memory footprint reduction could be a stated objective of the proposal. For example, the current solution of using a canvas forces the creation of an intermediate copy (the canvas). Also, to avoid the multiplication of APIs for moving image data between various representations, I would like to suggest an alternative: using the ImageBitmap interface as a hub. The creation of ImageBitmaps is already asynchronous (createImageBitmap returns a promise), and it has overloads for acquiring images from img, video, canvas, url, blob, imagedata. All that is missing are a few methods for getting data directly out of an ImageBitmap. So I think adding toBlob and toImageData (both async) to ImageBitmap is a more succinct proposal that would address your use cases, and many additional ones at the same time.
Detecting image encoding/decoding support
While drafting https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I realised that there is no way to tell in JS what image formats the browser can decode (with Image or ImageBitmap objects) or encode (with canvas toDataURL()/toBlob() or the ImageBitmap.toBlob() method I was proposing). HTMLMediaElement.canPlayType() can give an indication if an audio or video format can be decoded. MediaRecorder.canRecordMimeType() can give an indication if an audio or video format can be encoded. Should we not have the same for images? For example there are a number of image formats beyond PNG and JPEG which some browsers support, such as: - JPEG 2000 (Safari) - JPEG XR (IE/Edge) - WebP (Blink) - APNG (Firefox, Safari) - any other future formats Most client-side detection I've seen involves either trying to load a data URI included in the source, or using canvas' toDataURL() and seeing if the resulting data URI includes the expected MIME type. I think there should be methods on some object (be it Image, ImageBitmap or Canvas) for: - canDecodeType(): indicate if an image format can be encoded by Image/ImageBitmap objects - canEncodeType(): indicate if an image format can be encoded by Canvas toDataURL/toBlob (or my proposed ImageBitmap.toBlob()) Thoughts?
Re: Async Image - ImageData conversion
Sorry for the confusion. Yes, the latest URL is: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html I'm new to specs and WebIDL, my intent was to say those are new methods on ImageBitmap. Is partial interface ImageBitmap the correct way to say that? (I updated the URL to say that instead) The wording of undue latency in the ImageBitmap spec does not appear to rule out ImageBitmap storing its encoded image data, and lazily decoding it. However I'm not sure what the point of that would be, since it seems to be how drawing HTMLImageElements to a canvas already works. I see a couple of options: - allow lazy decoding in ImageBitmap, and therefore converting to ImageData is an explicit request to decompress - require ImageBitmap to decompress its contents, and make it available through a read-only Uint8ClampedArray like ImageData has. Therefore, converting to ImageData indicates the intent to modify this content, therefore a copy is warranted. - provide a way to neuter the ImageBitmap when converting it to ImageData, transferring its contents and avoiding any copy, which is useful if the ImageBitmap is a temporary object only being created for the purposes of getting the ImageData. I guess this would be similar to transferrable objects with workers. - add some kind of load method on ImageBitmap. It may store its contents in compressed form, but calling load causes it to be decompressed (or loaded from somewhere else). The ImageBitmap is only guaranteed to be drawn with undue latency after the load call. Therefore for the use case of low-latency rendering load can always be called immediately after creation, but for conversion purposes not calling load avoids duplicating memory. Note for devices with discrete GPUs, it's possible that ImageBitmap stores the decompressed image data in a GPU texture but does not have it in system RAM. In this case making a copy for the ImageData is also warranted. I'm a web developer proposing this because it would be useful for my purposes, I've no idea which of these would be favoured by implementors or the spec process. I'm happy to get involved in spec work though so please let me know if you have any feedback/suggestions on anything I'm doing here. Ashley On 24 June 2015 at 19:12, Boris Zbarsky bzbar...@mit.edu wrote: On 6/19/15 5:43 AM, Ashley Gullen wrote: I've not done this before, so I've no idea if this is the right/useful approach, but I drafted a spec for it here: https://www.scirra.com/labs/specs/imagedata-blob-extensions.html Ashley, We at Mozilla were just discussing this proposal, and we have a concern. With this proposal, going from an img to an ImageData requires conversion of an intermediate ImageBitmap. Unfortunately, an ImageBitmap is specified to be paintable without undue latency, which we interpret to mean it needs to have decoded image data, and the ImageData will end up needing to copy said data in practice (because it needs an editable copy). That creates two copies of the decoded image data in memory, which seems fairly undesirable on memory-constrained devices. -Boris
Re: Async Image - ImageData conversion
I've not done this before, so I've no idea if this is the right/useful
approach, but I drafted a spec for it here:
https://www.scirra.com/labs/specs/imagedata-blob-extensions.html
Let me know if you have any feedback on this.
On 18 June 2015 at 22:27, Travis Leithead travis.leith...@microsoft.com
wrote:
Cool. Want to write it up as a small extension spec? We have a new
Incubator Community Group coming up soon that would be an ideal place to
drop a small spec into and let implementers experiment with it and get
broader community feedback on the idea.
*From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
Gullen
*Sent:* Wednesday, June 17, 2015 2:06 PM
*To:* Travis Leithead
*Cc:* public-webapps@w3.org
*Subject:* Re: Async Image - ImageData conversion
That seems like a good start. I suppose there should be a
putImageDataAsync counterpart too? Then we can do:
Blob - Image via: load blob URL
Blob - ImageData via: load blob URL - Canvas drawImage
- getImageDataAsync
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - getImageDataAsync
ImageData - Blob via: Canvas putImageDataAsync - Canvas toBlob
ImageData - Image via: Canvas putImageDataAsync - Canvas toBlob - load
blob URL
I think the potential problems with this are:
- putImageDataAsync is the logical counterpart to getImageDataAsync, but
what happens if you make other synchronous draw calls while
putImageDataAsync is processing? What actually ends up in the canvas and
how is this defined?
- some of the conversion steps seem pretty long winded, in particular Blob
- ImageData and ImageData - Image. If implementors think they can
optimise these to be as efficient as direct conversion that's fine, but is
that actually doable with such a roundabout API?
If ImageData has:
PromiseBlob toBlob();
PromiseImageData fromBlob(blob);
then we can use the following conversions without new canvas functions:
Blob - Image via: load blob URL
Blob - ImageData via: ImageData.fromBlob
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - Canvas toBlob -
ImageData.fromBlob
ImageData - Blob via: ImageData.toBlob
ImageData - Image via: ImageData.toBlob - load blob URL
That looks like a much more reasonable set of conversions to me, and is
all async.
I was thinking about toImage and fromImage, but it seems to be a bigger
step to add a new way to make images, and there's the question of what the
src for the image returned by toImage should be.
On 17 June 2015 at 18:59, Travis Leithead travis.leith...@microsoft.com
wrote:
I think solving at least the first-order problem of extracting data from
the Canvas async is do-able.
Something like:
PromiseImageData getImageDataAsync(x,y,w,h);
seems sensible to add J
*From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
Gullen
*Sent:* Wednesday, June 17, 2015 10:00 AM
*To:* public-webapps@w3.org
*Subject:* Async Image - ImageData conversion
I was wondering if there is anything on the standards track to
asynchronously get an ImageData object from an Image?
We have a web app dealing with large sprite sheets (up to 2048x2048), and
at some point we need to get an ImageData for this. Currently the only way
I know of doing this is via a canvas 2d context like so:
function ImageToImageData(img)
{
let w = img.width;
let h = img.height;
let canvas = document.createElement(canvas);
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext(2d);
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};
This whole function is synchronous. With large images, in Chrome the
getImageData call can jank for well over 100ms on a high-end desktop
machine. Maybe this can be faster but I really doubt this whole function is
likely to be done in 16ms on low-end devices, so I believe there ought to
be an async alternative. As far as I am aware there is none.
Ideally we could have an async ImageData.fromImage(img) method or similar.
It would also be useful if we could decode directly from Blob to ImageData
without having to go via an Image as well (perhaps ImageData.fromBlob(blob)
?).
More broadly, it would be good to have a comprehensive set of async
conversion functions between Blob, Image, and ImageData. Blob - Image can
already be done by setting the image src to a blob URL, but basically
everything else requires going synchronously through a 2D canvas, which
janks apps which have to process lots of large images. Running a canvas in
a worker could possibly help, but that would require both Blob and Image to
be transferable to be useful.
Ashley Gullen
Scirra.com
Re: Async Image - ImageData conversion
Gosh, I keep putting the wrong terms in and it really confuses things. I need to get better at writing clearly! I mentioned transferToImageBitmap when I meant transferToImageData. The parargraph from my second-last email should read: I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageData() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. I understand that ImageBitmaps were not intended to be mutated, but the OffscreenCanvas proposal is already suggesting that there be a close() method (to immediately release it), and that they be transferrable (which mutates them on the thread they are transferred from). These appear to be essential requirements for the OffscreenCanvas proposal to work efficiently. If we are going to have this anyway, then it seems reasonable to use the same mechanism for ImageBitmap.transferToImageData(). Also there is still no direct access to the ImageBitmap pixel data, it's only mutable in the sense it can be reset to empty. The same thing could happen for ImageData.transferToImageBitmap(), where the ImageData is neutered and it returns an ImageBitmap which can share the same storage, but I guess this is less controversial since ImageData is already mutable. On 26 June 2015 at 14:10, Justin Novosad ju...@google.com wrote: On Fri, Jun 26, 2015 at 7:07 AM, Ashley Gullen ash...@scirra.com wrote: I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If I am not mistaken, ImageBitmaps were not intended to be mutated. So far I always assumed the intended behavior of transferToImageBitmap() was that it created a new ImageBitmap. The difference with createImageBitmap is that the transfer semantic guarantees zero copying, which implies that the source object has to be neutered. transferToImageBitmap should only be necessary for transferring from objects that have mutable pixels data (ImageData, canvas). If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
That seems like a good start. I suppose there should be a putImageDataAsync
counterpart too? Then we can do:
Blob - Image via: load blob URL
Blob - ImageData via: load blob URL - Canvas drawImage
- getImageDataAsync
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - getImageDataAsync
ImageData - Blob via: Canvas putImageDataAsync - Canvas toBlob
ImageData - Image via: Canvas putImageDataAsync - Canvas toBlob - load
blob URL
I think the potential problems with this are:
- putImageDataAsync is the logical counterpart to getImageDataAsync, but
what happens if you make other synchronous draw calls while
putImageDataAsync is processing? What actually ends up in the canvas and
how is this defined?
- some of the conversion steps seem pretty long winded, in particular Blob
- ImageData and ImageData - Image. If implementors think they can
optimise these to be as efficient as direct conversion that's fine, but is
that actually doable with such a roundabout API?
If ImageData has:
PromiseBlob toBlob();
PromiseImageData fromBlob(blob);
then we can use the following conversions without new canvas functions:
Blob - Image via: load blob URL
Blob - ImageData via: ImageData.fromBlob
Image - Blob via: Canvas drawImage - Canvas toBlob
Image - ImageData via: Canvas drawImage - Canvas toBlob -
ImageData.fromBlob
ImageData - Blob via: ImageData.toBlob
ImageData - Image via: ImageData.toBlob - load blob URL
That looks like a much more reasonable set of conversions to me, and is all
async.
I was thinking about toImage and fromImage, but it seems to be a bigger
step to add a new way to make images, and there's the question of what the
src for the image returned by toImage should be.
On 17 June 2015 at 18:59, Travis Leithead travis.leith...@microsoft.com
wrote:
I think solving at least the first-order problem of extracting data from
the Canvas async is do-able.
Something like:
PromiseImageData getImageDataAsync(x,y,w,h);
seems sensible to add J
*From:* a...@scirra.com [mailto:a...@scirra.com] *On Behalf Of *Ashley
Gullen
*Sent:* Wednesday, June 17, 2015 10:00 AM
*To:* public-webapps@w3.org
*Subject:* Async Image - ImageData conversion
I was wondering if there is anything on the standards track to
asynchronously get an ImageData object from an Image?
We have a web app dealing with large sprite sheets (up to 2048x2048), and
at some point we need to get an ImageData for this. Currently the only way
I know of doing this is via a canvas 2d context like so:
function ImageToImageData(img)
{
let w = img.width;
let h = img.height;
let canvas = document.createElement(canvas);
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext(2d);
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};
This whole function is synchronous. With large images, in Chrome the
getImageData call can jank for well over 100ms on a high-end desktop
machine. Maybe this can be faster but I really doubt this whole function is
likely to be done in 16ms on low-end devices, so I believe there ought to
be an async alternative. As far as I am aware there is none.
Ideally we could have an async ImageData.fromImage(img) method or similar.
It would also be useful if we could decode directly from Blob to ImageData
without having to go via an Image as well (perhaps ImageData.fromBlob(blob)
?).
More broadly, it would be good to have a comprehensive set of async
conversion functions between Blob, Image, and ImageData. Blob - Image can
already be done by setting the image src to a blob URL, but basically
everything else requires going synchronously through a 2D canvas, which
janks apps which have to process lots of large images. Running a canvas in
a worker could possibly help, but that would require both Blob and Image to
be transferable to be useful.
Ashley Gullen
Scirra.com
Async Image - ImageData conversion
I was wondering if there is anything on the standards track to
asynchronously get an ImageData object from an Image?
We have a web app dealing with large sprite sheets (up to 2048x2048), and
at some point we need to get an ImageData for this. Currently the only way
I know of doing this is via a canvas 2d context like so:
function ImageToImageData(img)
{
let w = img.width;
let h = img.height;
let canvas = document.createElement(canvas);
canvas.width = w;
canvas.height = h;
let ctx = canvas.getContext(2d);
ctx.drawImage(img, 0, 0);
return ctx.getImageData(0, 0, w, h);
};
This whole function is synchronous. With large images, in Chrome the
getImageData call can jank for well over 100ms on a high-end desktop
machine. Maybe this can be faster but I really doubt this whole function is
likely to be done in 16ms on low-end devices, so I believe there ought to
be an async alternative. As far as I am aware there is none.
Ideally we could have an async ImageData.fromImage(img) method or similar.
It would also be useful if we could decode directly from Blob to ImageData
without having to go via an Image as well (perhaps ImageData.fromBlob(blob)
?).
More broadly, it would be good to have a comprehensive set of async
conversion functions between Blob, Image, and ImageData. Blob - Image can
already be done by setting the image src to a blob URL, but basically
everything else requires going synchronously through a 2D canvas, which
janks apps which have to process lots of large images. Running a canvas in
a worker could possibly help, but that would require both Blob and Image to
be transferable to be useful.
Ashley Gullen
Scirra.com
Re: Async Image - ImageData conversion
I thought it would be more confusing to have separate competing proposals in one document? They are more like three revisions of the same document anyway, the latest one including all feedback and rationale so far: https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html On 3 July 2015 at 13:04, Anne van Kesteren ann...@annevk.nl wrote: On Fri, Jul 3, 2015 at 1:51 PM, Ashley Gullen ash...@scirra.com wrote: These conversions are covered by what I proposed in my previous email, so I drafted another spec with them: https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html Could you please put them all in a single document? That would make it a bit easier to follow what is being proposed. And perhaps not use Respec or figure out some way to get them in a style similar to the original HTML Standard. This non-algorithmic way of describing methods is error prone. A new toBlob() should probably also take a dictionary as second argument rather than the any approach. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
On 2 July 2015 at 21:36, Jeff Muizelaar jmuizel...@mozilla.com wrote:
The important ones of these are:
Blob - ImageData
HTMLImageElement - ImageData
ImageData - Blob
These conversions are covered by what I proposed in my previous email, so I
drafted another spec with them:
https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html
I also mentioned canEncodeType and canDecodeType in another email thread,
but I think they are important to the same use cases, so I've put them in
the draft as well to provoke discussion.
Ashley
There are a bunch of conversions that aren't covered directly by these but
I'm
not sure it makes us much sense to short circut them beyond what
functionality
is currently exposed. e.g.
Going HTMLVideoElement - ImageBitmap - Blob
isn't really much worse than doing:
HTMLVideoElement - CanvasRenderContext2D - Blob
-Jeff
On Thu, Jul 2, 2015 at 3:05 PM, Justin Novosad ju...@google.com wrote:
Making ImageData the hub would imply more copies in memory in many
cases.
Because ImageData is mutable (not to mention the alpha multiplication
issues
which are also a factor), it cannot share its image buffer with the
source
it was created from, unlike ImageBitmap. Immutability is a significant
advantage of ImageBitmap, which allows for zero-copy code paths in many
cases, which helps with both performance and memory consumption.
On Tue, Jun 30, 2015 at 4:17 PM, Ashley Gullen ash...@scirra.com
wrote:
If it's difficult to make ImageBitmap both an efficient drawing source
and
a conversion intermediary, then we could add conversion functions to
each
object that can be converted, e.g.:
HTMLImageElement.toBlob()
HTMLImageElement.toImageData()
Blob.toImageData()
ImageData.toBlob()
This seems inconsistent. For example to convert a Blob to an Image, you
should create a URL and set an Image's src to it; to convert a Blob to
an
ImageBitmap, you should use createImageBitmap(blob); and then under this
proposal there's a third approach to convert a Blob to an ImageData,
using
Blob.toImageData(). It also has a larger surface area, requiring
changes to
several interfaces.
So perhaps it would be better to make ImageData the hub of conversion,
more like the first proposal I made. If ImageBitmap is a GPU-hosted
premultiplied resource, then ImageData is an expressly not-on-GPU
not-premultiplied alternative. That would mean adding something like
this to
ImageData:
typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
partial interface ImageData {
static PromiseImageData create(ImageDataSource source)
PromiseBlob toBlob()
}
ImageData can also be converted to HTMLImageElement via toBlob,
ImageBitmap via createImageBitmap, or a canvas via putImageData (which
is
still synchronous, but no longer needs to be done purely for conversion
reasons, so probably means you really want it to appear in the canvas
and
therefore should remain synchronous).
This covers all the conversion use cases I can think of without
complicating ImageBitmap's without undue latency requirement. There's
no
more transferring going on either, which I think is now unnecessary
since
you can get from HTMLImageElement to ImageData with one call. I think
it's
more future-proof too since future types can be added to ImageDataSource
allowing new types to be converted without a new method being added.
Does this approach sound better?
Ashley
On 26 June 2015 at 16:37, Boris Zbarsky bzbar...@mit.edu wrote:
On 6/26/15 4:07 AM, Ashley Gullen wrote:
I don't think we should extend HTMLImageElement because it is not
available in workers. Adding the conversion methods to ImageBitmap
allows workers to perform conversions using Blob (compressed image
data)
in the place of HTMLImageElement.
Maybe I wasn't clear. I was suggesting that we have the methods on
both
HTMLImageElement and ImageBitmap (and possibly on any other things we
feel
should have the methods directly).
I like the suggestion that ImageBitmap be the hub of image
conversion,
I agree that it sounds appealing, but it means ImageBitmap now has to
serve two masters: it has to be something that you can paint from
quickly
(premultiplied, probably lives on the GPU) _and_ it needs to be
something
you can transferToImageData efficiently (better to not live on the GPU
for
this).
Maybe that's OK; it's just a bit of a warning flag from my point of
view
when a single object is meant to do multiple quite different things; it
makes it harder to have it be good at all of them...
-Boris
Re: Async Image - ImageData conversion
I'm also wondering that if we have ImageBitmap.transferToImageData(), should we not also have ImageData.transferToImageBitmap()? Currently createImageBitmap(imageData) appears to require making a copy. On 26 June 2015 at 12:07, Ashley Gullen ash...@scirra.com wrote: I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
I've updated the draft with an ImageBitmap.transferToImageData() method: https://www.scirra.com/labs/specs/imagebitmap-conversion-extensions.html It's also used in Example 2 to demonstrate converting a Blob to an ImageData without redundantly copying the pixel data. I don't think we should extend HTMLImageElement because it is not available in workers. Adding the conversion methods to ImageBitmap allows workers to perform conversions using Blob (compressed image data) in the place of HTMLImageElement. I like the suggestion that ImageBitmap be the hub of image conversion, which is why I think it makes sense to add these methods to ImageBitmap and not create() methods on other objects. In particular transferToImageBitmap() seems like it ought to be an ImageBitmap method since it mutates the ImageBitmap, whereas something like ImageData.transferFrom(imageBitmap) seems unintuitive if it mutates its parameter. If transferToImageData() belongs to ImageBitmap, I think toImageData() logically should belong to ImageBitmap too. ImageBitmap.toBlob() is also consistent with HTMLCanvasElement.toBlob()/toDataURL(), where the method belongs to the object which represents the data source. Ashley On 25 June 2015 at 22:10, Anne van Kesteren ann...@annevk.nl wrote: On Thu, Jun 25, 2015 at 1:57 PM, Boris Zbarsky bzbar...@mit.edu wrote: The drawback of adding toBlob/toImageData to the various things ImageData can be constructed from is that it's a bit more spec complexity, but I don't see that as a showstopper, necessarily. We should probably stick to the pattern of either having factory methods or putting methods directly on various objects. Or maybe, change them all to be static methods on the various classes: ImageBitmap.create(...) ImageData.create(...) ... I would personally prefer this last pattern for creating instances where we need to go through a promise. -- https://annevankesteren.nl/
Re: Async Image - ImageData conversion
If it's difficult to make ImageBitmap both an efficient drawing source and
a conversion intermediary, then we could add conversion functions to each
object that can be converted, e.g.:
HTMLImageElement.toBlob()
HTMLImageElement.toImageData()
Blob.toImageData()
ImageData.toBlob()
This seems inconsistent. For example to convert a Blob to an Image, you
should create a URL and set an Image's src to it; to convert a Blob to an
ImageBitmap, you should use createImageBitmap(blob); and then under this
proposal there's a third approach to convert a Blob to an ImageData, using
Blob.toImageData(). It also has a larger surface area, requiring changes to
several interfaces.
So perhaps it would be better to make ImageData the hub of conversion,
more like the first proposal I made. If ImageBitmap is a GPU-hosted
premultiplied resource, then ImageData is an expressly not-on-GPU
not-premultiplied alternative. That would mean adding something like this
to ImageData:
typedef (HTMLImageElement or Blob or ImageBitmap) ImageDataSource;
partial interface ImageData {
static PromiseImageData create(ImageDataSource source)
PromiseBlob toBlob()
}
ImageData can also be converted to HTMLImageElement via toBlob, ImageBitmap
via createImageBitmap, or a canvas via putImageData (which is still
synchronous, but no longer needs to be done purely for conversion reasons,
so probably means you really want it to appear in the canvas and therefore
should remain synchronous).
This covers all the conversion use cases I can think of without
complicating ImageBitmap's without undue latency requirement. There's no
more transferring going on either, which I think is now unnecessary since
you can get from HTMLImageElement to ImageData with one call. I think it's
more future-proof too since future types can be added to ImageDataSource
allowing new types to be converted without a new method being added.
Does this approach sound better?
Ashley
On 26 June 2015 at 16:37, Boris Zbarsky bzbar...@mit.edu wrote:
On 6/26/15 4:07 AM, Ashley Gullen wrote:
I don't think we should extend HTMLImageElement because it is not
available in workers. Adding the conversion methods to ImageBitmap
allows workers to perform conversions using Blob (compressed image data)
in the place of HTMLImageElement.
Maybe I wasn't clear. I was suggesting that we have the methods on both
HTMLImageElement and ImageBitmap (and possibly on any other things we feel
should have the methods directly).
I like the suggestion that ImageBitmap be the hub of image conversion,
I agree that it sounds appealing, but it means ImageBitmap now has to
serve two masters: it has to be something that you can paint from quickly
(premultiplied, probably lives on the GPU) _and_ it needs to be something
you can transferToImageData efficiently (better to not live on the GPU for
this).
Maybe that's OK; it's just a bit of a warning flag from my point of view
when a single object is meant to do multiple quite different things; it
makes it harder to have it be good at all of them...
-Boris
Re: Async Image - ImageData conversion
I'm new to writing specs and when I first looked ReSpec had templates and a guide - is there something similar for the HTML standard style spec? I changed toBlob to use a dictionary (still at https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html). Ashley On 7 July 2015 at 10:48, Anne van Kesteren ann...@annevk.nl wrote: On Mon, Jul 6, 2015 at 2:38 PM, Ashley Gullen ash...@scirra.com wrote: I thought it would be more confusing to have separate competing proposals in one document? They are more like three revisions of the same document anyway, the latest one including all feedback and rationale so far: https://www.scirra.com/labs/specs/imagedata-conversion-extensions.html Sorry, my bad. It seems the other comments still apply though. -- https://annevankesteren.nl/
