Re: [whatwg] Offscreen canvas (or canvas for web workers).
On Mar 14, 2010, at 6:22 PM, Jonas Sicking wrote: One way to do it would be to have an function somewhere, not necessarily on the 2D context, which given a Blob, returns an ImageData object. However this still results in the image being loaded twice into memory, so would only really help if you want to operate on an ImageData object directly. I think loading via an img element in order to draw the image is fine. An HTMLImageElement is just as drawable as an ImageData and already has all the logic for asynchronous loading of images and processing I agree that the number of steps is not important for responsiveness or performance (though it is for complexity). However several of those steps seemed to involved non-trivial amount of CPU usage, that was the concern expressed in my initial mail. At the very least I think we have a skewed proposal. The main use cases that has been brought up are scaling and rotating images. However the proposal is far from optimal for fulfilling that use case. For scaling, it's fairly complex and uses more CPU cycles, both on the main thread, and in total, than would be needed with an API more optimized for that use case. For rotating it doesn't do anything. You're assuming a scale and a rotate are both less expensive than two blits. Since no one else has provided perf data, I made my own test: http://webkit.org/demos/canvas-perf/canvas.html === Results === = Safari (w/ WebKit trunk) = Direct image copy: 39ms Indirect copy with (via ImageData): 138.5ms Copy with 2x scale: 717ms Copy with 0.5x scale: 59.5ms Copy with rotate:142.5ms = Chrome dev 5.0.322.2 = Direct image copy: 63ms Indirect copy with (via ImageData): 161.5ms Copy with 2x scale: 1376.5ms Copy with 0.5x scale: 70.5ms Copy with rotate: 259ms = Opera 10.5 alpha = Direct image copy: 89ms Indirect copy with (via ImageData): 428.5ms Copy with 2x scale: 963.5ms Copy with 0.5x scale: 61ms Copy with rotate: 150ms = Firefox 3.6 = Direct image copy: 81ms Indirect copy with (via ImageData): 693.5ms Copy with 2x scale: 1703.5ms Copy with 0.5x scale: 284.5ms Copy with rotate: 568.5ms === Summary of Data === 1) In all browsers tested, copying to an ImageData and then back to a canvas (two blits) is faster than a 2x scale. 2) In all browsers tested, twice the cost of a canvas-to-canvas blit is considerably less than the cost of copy to and back from ImageData. 3) In all browsers tested, twice the cost of a canvas-to-canas blit is still less than the cost of a single 0.5x scale or a rotate. === Conclusions === 1) For scaling an image up 2x, copying to an ImageData and back for processing on a Worker would improve responsiveness, relative to just doing the scale on the main thread. 2) Copying from one canvas to another is much faster than copying to/ from ImageData. To make copying to a Worker worthwhile as a responsiveness improvement for rotations or downscales, in addition to the OffscreenCanvas proposal we would need a faster way to copy image data to a Worker. One possibility is to allow an OffscreenCanvas to be copied to and from a background thread. It seems this would be much much faster than copying via ImageData. Regards, Maciej
Re: [whatwg] Offscreen canvas (or canvas for web workers).
I agree that the number of steps is not important for responsiveness or performance (though it is for complexity). However several of those steps seemed to involved non-trivial amount of CPU usage, that was the concern expressed in my initial mail. At the very least I think we have a skewed proposal. The main use cases that has been brought up are scaling and rotating images. However the proposal is far from optimal for fulfilling that use case. For scaling, it's fairly complex and uses more CPU cycles, both on the main thread, and in total, than would be needed with an API more optimized for that use case. For rotating it doesn't do anything. You're assuming a scale and a rotate are both less expensive than two blits. Since no one else has provided perf data, I made my own test: http://webkit.org/demos/canvas-perf/canvas.html === Results === = Safari (w/ WebKit trunk) = Direct image copy: 39ms Indirect copy with (via ImageData): 138.5ms Copy with 2x scale: 717ms Copy with 0.5x scale: 59.5ms Copy with rotate:142.5ms = Chrome dev 5.0.322.2 = Direct image copy: 63ms Indirect copy with (via ImageData): 161.5ms Copy with 2x scale: 1376.5ms Copy with 0.5x scale: 70.5ms Copy with rotate: 259ms = Opera 10.5 alpha = Direct image copy: 89ms Indirect copy with (via ImageData): 428.5ms Copy with 2x scale: 963.5ms Copy with 0.5x scale: 61ms Copy with rotate: 150ms = Firefox 3.6 = Direct image copy: 81ms Indirect copy with (via ImageData): 693.5ms Copy with 2x scale: 1703.5ms Copy with 0.5x scale: 284.5ms Copy with rotate: 568.5ms === Summary of Data === 1) In all browsers tested, copying to an ImageData and then back to a canvas (two blits) is faster than a 2x scale. 2) In all browsers tested, twice the cost of a canvas-to-canvas blit is considerably less than the cost of copy to and back from ImageData. 3) In all browsers tested, twice the cost of a canvas-to-canas blit is still less than the cost of a single 0.5x scale or a rotate. === Conclusions === 1) For scaling an image up 2x, copying to an ImageData and back for processing on a Worker would improve responsiveness, relative to just doing the scale on the main thread. 2) Copying from one canvas to another is much faster than copying to/from ImageData. To make copying to a Worker worthwhile as a responsiveness improvement for rotations or downscales, in addition to the OffscreenCanvas proposal we would need a faster way to copy image data to a Worker. One possibility is to allow an OffscreenCanvas to be copied to and from a background thread. It seems this would be much much faster than copying via ImageData. We're clearly going in circles here. My point is this: The two main use cases that has brought up have been scaling and rotating images off the main thread in order to improve responsiveness. The proposed solution addresses these use cases fairly poorly. Both in that APIs could be designed that makes these things simpler, and in that APIs could be designed that perform better both by putting less work on the main thread, and by doing less work in general. This does not take away from the fact that the proposal can be (based on your data) be used to improve performance when doing scaling. / Jonas
Re: [whatwg] Offscreen canvas (or canvas for web workers).
On Mar 15, 2010, at 12:28 AM, Jonas Sicking wrote: === Conclusions === 1) For scaling an image up 2x, copying to an ImageData and back for processing on a Worker would improve responsiveness, relative to just doing the scale on the main thread. 2) Copying from one canvas to another is much faster than copying to/from ImageData. To make copying to a Worker worthwhile as a responsiveness improvement for rotations or downscales, in addition to the OffscreenCanvas proposal we would need a faster way to copy image data to a Worker. One possibility is to allow an OffscreenCanvas to be copied to and from a background thread. It seems this would be much much faster than copying via ImageData. We're clearly going in circles here. My point is this: The two main use cases that has brought up have been scaling and rotating images off the main thread in order to improve responsiveness. The proposed solution addresses these use cases fairly poorly. Both in that APIs could be designed that makes these things simpler, and in that APIs could be designed that perform better both by putting less work on the main thread, and by doing less work in general. Do you have a specific proposal for how to handle those particular use cases? (Side note: I didn't test how efficient it would be to use WebGL to scale or rotate images, in part because I'm not sure how to do it. If you know how to code it, I'll gladly add it to my test case.) BTW although no one has provided specific use cases along these lines, I can imagine that Photoshop-style image processing effects may be compute-intensive enough that you want to do them off the main thread. At least, I think there's some photoshop filters that take noticeable time to complete even as native-compiled C++. Maybe WebGL could be used to do some or all of those things, it's hard to tell. It seems like ImageData is *not* a great way to do them if you can help it, since turning a large image into an ImageData is so expensive. This does not take away from the fact that the proposal can be (based on your data) be used to improve performance when doing scaling. It looks to me like it could improve performance quite a lot if we add a more efficient way Actually, given that ImageData is already copiable to a background thread, it seems like a good idea to add some form of image data that can be copied to a Worker with less work on the main thread. That seems valuable even if there is no actual graphics API on the background thread. Regards, Maciej
Re: [whatwg] Offscreen canvas (or canvas for web workers).
On Mon, Mar 15, 2010 at 7:05 AM, Maciej Stachowiak m...@apple.com wrote: Copying from one canvas to another is much faster than copying to/from ImageData. To make copying to a Worker worthwhile as a responsiveness improvement for rotations or downscales, in addition to the OffscreenCanvas proposal we would need a faster way to copy image data to a Worker. One possibility is to allow an OffscreenCanvas to be copied to and from a background thread. It seems this would be much much faster than copying via ImageData. Maybe this indicates that implementations of getImageData/putImageData ought to be optimised? e.g. do the expensive multiplications and divisions in the premultiplication code with SIMD. (A seemingly similar thing at http://bugzilla.openedhand.com/show_bug.cgi?id=1939 suggests SSE2 makes things 3x as fast). That would avoid the need to invent new API, and would also benefit anyone who wants to use ImageData for other purposes. -- Philip Taylor exc...@gmail.com
Re: [whatwg] Offscreen canvas (or canvas for web workers).
On Mar 15, 2010, at 3:46 AM, Philip Taylor wrote: On Mon, Mar 15, 2010 at 7:05 AM, Maciej Stachowiak m...@apple.com wrote: Copying from one canvas to another is much faster than copying to/ from ImageData. To make copying to a Worker worthwhile as a responsiveness improvement for rotations or downscales, in addition to the OffscreenCanvas proposal we would need a faster way to copy image data to a Worker. One possibility is to allow an OffscreenCanvas to be copied to and from a background thread. It seems this would be much much faster than copying via ImageData. Maybe this indicates that implementations of getImageData/putImageData ought to be optimised? e.g. do the expensive multiplications and divisions in the premultiplication code with SIMD. (A seemingly similar thing at http://bugzilla.openedhand.com/show_bug.cgi?id=1939 suggests SSE2 makes things 3x as fast). That would avoid the need to invent new API, and would also benefit anyone who wants to use ImageData for other purposes. It might be possible to make getImageData/putImageData faster than they are currently, certainly the browsers at the slower end of the ImageData performance spectrum must have a lot of headroom. But they probably also probably have room to optimize drawImage. (Looking back at my data I noticed that getImageData + putImageData in Safari is about as fast or faster than two drawImage calls in the other browsers tested). In the end, though, I doubt that it's possible for getImageData or putImageData to be as fast as drawImage, since drawImage doesn't have to do any conversion of the pixel format. Regards, Maciej
Re: [whatwg] Storage quota introspection and modification
Am 11. März 2010 14:50 schrieb Michael Nordman micha...@google.com: On Thu, Mar 11, 2010 at 1:29 PM, Tab Atkins Jr. jackalm...@gmail.comwrote: 2010/3/11 Ian Fette (イアンフェッティ) ife...@google.com: Yes, but I think there may be uses of things like storage for non-offline uses (pre-fetching email attachments, saving an email that is in a draft state etc.) If it's relatively harmless, like 1mb usage, I don't want to pop up an infobar, I just want to allow it. So, I don't really want to have an infobar each time a site uses one of these features for the first time, I'd like to allow innocuous use if possible. But at the same time, I want apps to be able to say up front, at a time when the user is thinking about it (because they just clicked something on the site, presumably) here's what I am going to need. This is precisely my preferred interaction model as well. Absolutely silent use of a relatively small amount of resources, just like cookies are done today, but with a me-initiated ability to authorize it to act like a full app with unlimited resources (or at least much larger resources). In addition to more storage, another difference that ideally would come along with the full-app-privilege is for the user agent to avoid evicting that data. So stronger promises about keeping that data around relative to unprivileged app data. Also, this is being discussed in terms of apps. Can more than one app be hosted on the same site? And if so, how can their be stored resources be distinquished? Given that we have no way to enforce it in the UA (minus any radical changes to same origin policy), I am leaning towards saying no to anything that would actually be enforced. Perhaps we can make it easier for an app to define what resources it owns, so that it can be broken down in a bit more friendly way in the UI, but I really don't find it that interesting to be honest given how few people I expect to ever go to the UI or even care. ~TJ
Re: [whatwg] Storage quota introspection and modification
Am 11. März 2010 14:35 schrieb Mike Shaver mike.sha...@gmail.com: 2010/3/11 Ian Fette (イアンフェッティ) ife...@google.com: AFAIK most browsers are setting a default quota for storage options that is on the order of megabytes. Could well be, indeed. It sounded like you'd done some thinking about the size, and I was curious about how you came up with that number (versus some %age of available disk, for example). To be honest, I don't think it was all that scientific. More like If someone discovered that random.com was storing 1GB of stuff on their disk and they didn't know, would they (rationally or not) get pissed off about it? and that seemed to be a yes. 5Mb? no. We could go for a percentage of the disk, but again, how much to take? If someone discovers that their disk is half full, are they going to think to open up their browser to clear stuff out? Do they even know where their browser's profile directory is? Or are they just going to get frustrated and curse. Yes, but I think there may be uses of things like storage for non-offline uses (pre-fetching email attachments, saving an email that is in a draft state etc.) If it's relatively harmless, like 1mb usage, I don't want to pop up an infobar, I just want to allow it. So, I don't really want to have an infobar each time a site uses one of these features for the first time, I'd like to allow innocuous use if possible I think of an infobar as relatively innocuous, and a good balance of user awareness versus flow interruption, but I repeat my lack of interaction design credentials! But I really don't want to see infobars everywhere :) (Your attachment example is an interesting one, I think: do I get the request if I request too-big an attachment, but not if it's a small one? Or if it's saving a blog post draft that has a bunch of images in it, vs. one that's 140 chars long.) exactly But at the same time, I want apps to be able to say up front, at a time when the user is thinking about it (because they just clicked something on the site, presumably) here's what I am going to need. OK, I see. What if we had the named-subquota stuff, and the way you triggered that request was creation of a named subquota? That would also encourage app developers to provide a description of the quota, and perhaps the sort of necessary for offline operation vs improves performance vs supports additional features. The named subquota creation request could give an initial requested size and estimated upper bound for the size. An async event delivered back (or a callback called) could tell the app what quota it was granted, if any (or maybe just that it was granted some, but the size limit wasn't specified). My initial reaction was that I don't know how much I buy into the subquota part (vs named quota in general). E.g. if we can't enforce any of the subquota distinctions beyond a same-origin level, it seems of limited use. Upon further thought though, if you assume apps you trust are well behaved, then this may actually be a good idea. Would make displaying this information to users easier as well, even if relatively few users ever do go into options UI. At this point, if named subquota would meet the requirements I initially put forth (request a set of resource quotas that I think I need, and get a callback if I get them), and ideally be able to interrogate some sort of information about the named subquota (be it how many bytes are free vs what are you reasonably sure I can store I really don't care), I am all for it ;-) Is there some named subquota thread that I need to +1? Mike
Re: [whatwg] Offscreen canvas (or canvas for web workers).
On 3/15/2010 4:22 AM, Maciej Stachowiak wrote: On Mar 15, 2010, at 3:46 AM, Philip Taylor wrote: On Mon, Mar 15, 2010 at 7:05 AM, Maciej Stachowiak m...@apple.com wrote: Copying from one canvas to another is much faster than copying to/from ImageData. To make copying to a Worker worthwhile as a responsiveness improvement for rotations or downscales, in addition to the OffscreenCanvas proposal we would need a faster way to copy image data to a Worker. One possibility is to allow an OffscreenCanvas to be copied to and from a background thread. It seems this would be much much faster than copying via ImageData. Maybe this indicates that implementations of getImageData/putImageData ought to be optimised? e.g. do the expensive multiplications and divisions in the premultiplication code with SIMD. (A seemingly similar thing at http://bugzilla.openedhand.com/show_bug.cgi?id=1939 suggests SSE2 makes things 3x as fast). That would avoid the need to invent new API, and would also benefit anyone who wants to use ImageData for other purposes. It might be possible to make getImageData/putImageData faster than they are currently, certainly the browsers at the slower end of the ImageData performance spectrum must have a lot of headroom. But they probably also probably have room to optimize drawImage. (Looking back at my data I noticed that getImageData + putImageData in Safari is about as fast or faster than two drawImage calls in the other browsers tested). In the end, though, I doubt that it's possible for getImageData or putImageData to be as fast as drawImage, since drawImage doesn't have to do any conversion of the pixel format. This is true -- getImageData/putImageData unfortunately saddled us with two performance-killing bits: 1) clamping on assignment. Not so bad, but doesn't help. 2) Unpremultiplied alpha. This is the biggest chunk. We have more optimized code in nightly builds of Firefox now that uses a lookup table and gets a pretty significant speedup for this part of put/get, but it's not going to be as fast as drawImage. Also, canvas is often (or can be) backed by actual hardware surfaces, and drawImage from one to another is going to be much faster than reading the data into system memory and then drawing from there back to the hardware surface. If we wanted to support this across workers (and I think it would be helpful to figure out how to do so), something like saying that if a canvas object was passed (somehow) between workers, it would be a copy -- and internally it could be implemented using copy-on-write semantics. - Vlad
Re: [whatwg] Offscreen canvas (or canvas for web workers).
On Mar 15, 2010, at 2:24 PM, Vladimir Vukicevic wrote: If we wanted to support this across workers (and I think it would be helpful to figure out how to do so), something like saying that if a canvas object was passed (somehow) between workers, it would be a copy -- and internally it could be implemented using copy-on-write semantics. I had been thinking the same -- it would allow the general case (eg. data is only expected to be used in the worker) to get a mostly free copy. - Vlad --Oliver
[whatwg] Define MessagePort.isConnected or MessagePort.ondisconnect
Hi all,
Consider a case where I have a SharedWorker script like below, and I
open two tabs that use this SharedWorker. Now myPorts.length is 2. If
I reload one of the two tabs, then myPorts.length is 3, isn't it? But
one of the three ports is already disconnected from the counterpart,
so postMessage'ing to the port is meaningless and I want to discard
reference to that port.
=== JS ===
var myPorts = [];
onconnect = function(e) {
var port = e.ports[0];
myPorts.push(port);
port.onmessage = function(e) {
myPorts.forEach(function(p) {
if (p !== port)
p.postMessage = e.data;
});
}
}
=== /JS ===
It seems like the only way to know if a MessagePort is connected is to
actually send a message and wait for a reply. So
MessagePort.isConnected or MessagePort.ondisconnect would be nice to
have.
A. TAKAYAMA
Re: [whatwg] Define MessagePort.isConnected or MessagePort.ondisconnect
On Mon, 15 Mar 2010, ATSUSHI TAKAYAMA wrote:
Consider a case where I have a SharedWorker script like below, and I
open two tabs that use this SharedWorker. Now myPorts.length is 2. If I
reload one of the two tabs, then myPorts.length is 3, isn't it? But one
of the three ports is already disconnected from the counterpart, so
postMessage'ing to the port is meaningless and I want to discard
reference to that port.
=== JS ===
var myPorts = [];
onconnect = function(e) {
var port = e.ports[0];
myPorts.push(port);
port.onmessage = function(e) {
myPorts.forEach(function(p) {
if (p !== port)
p.postMessage = e.data;
});
}
}
=== /JS ===
It seems like the only way to know if a MessagePort is connected is to
actually send a message and wait for a reply. So MessagePort.isConnected
or MessagePort.ondisconnect would be nice to have.
We used to have that (onclose would fire on the port), but we removed it
in the first version because it exposes garbage collection behaviour,
which we try pretty hard not to expose, so as to prevent sites from
accidentally depending on a particular GC strategy and thus forcing all
browsers to implement (or fake) the same GC strategy.
If you don't need to ever broadcast something to all the ports, you can
avoid keeping track of the ports altogether, and then you won't have a
problem. If you do need to broadcast, it's hard not to slowly leak at the
moment. We might add an object that can handle broadcast without leaking
at some future point (basically an opaque array that you push ports into
and which you can ask to iterate over the ports calling a function), but
it doesn't exist currently.
--
Ian Hickson U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,.
Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
Re: [whatwg] Define MessagePort.isConnected or MessagePort.ondisconnect
Agreed, there's not a good way to determine that a port is
disentangled. Currently the main solution I know of is to have your document
post a message to your shared worker in their onunload handler.
I think some kind of MessagePort.onclose event or entangled attribute
could be useful - this was originally part of the spec, and the issue with
that was that it's hard to define onclose in such a way that doesn't make it
highly dependent on garbage collection.
As an example:
var channel = new MessageChannel();
channel.port1.onclose = channel.port2.onclose = function() {alert(port
closed);};
channel = null;
What should happen in this case? At what point (if ever) should the onclose
handler be invoked?
I'm just leery of any situation where the garbage collected state of an
unreferenced object is exposed to script, as it seems like this causes
interoperability issues. For example, if you ran the script above in Chrome,
the onclose handler would likely not be invoked until the parent Document
was closed. In Safari, it would get invoked when the JS heap is next garbage
collected. An application that relied on onclose() being called in a timely
manner would break on Chrome.
The only option that comes to mind that doesn't expose compatibility issues
would be to only issue onclose events if close() is explicitly called on the
entangled port, but if you're doing that you might as well just have the
code calling close() post a I'm closing message first.
-atw
On Mon, Mar 15, 2010 at 5:13 PM, ATSUSHI TAKAYAMA
taka.atsu...@googlemail.com wrote:
Hi all,
Consider a case where I have a SharedWorker script like below, and I
open two tabs that use this SharedWorker. Now myPorts.length is 2. If
I reload one of the two tabs, then myPorts.length is 3, isn't it? But
one of the three ports is already disconnected from the counterpart,
so postMessage'ing to the port is meaningless and I want to discard
reference to that port.
=== JS ===
var myPorts = [];
onconnect = function(e) {
var port = e.ports[0];
myPorts.push(port);
port.onmessage = function(e) {
myPorts.forEach(function(p) {
if (p !== port)
p.postMessage = e.data;
});
}
}
=== /JS ===
It seems like the only way to know if a MessagePort is connected is to
actually send a message and wait for a reply. So
MessagePort.isConnected or MessagePort.ondisconnect would be nice to
have.
A. TAKAYAMA
Re: [whatwg] Define MessagePort.isConnected or MessagePort.ondisconnect
Thanks for replies, I understood what's the problem here. The only option that comes to mind that doesn't expose compatibility issues would be to only issue onclose events if close() is explicitly called on the entangled port, but if you're doing that you might as well just have the code calling close() post a I'm closing message first. -atw This would mean that all web pages using a SharedWorker (and keep reference to MessagePort inside the SharedWorker) have to set unload event handlers to call port.close() so that references to the ports in the SharedWorker don't get accumulated. That is not desirable. pagehide handler may not be sufficient for this purpose. I think, as Hixie suggested, an array like object to track references of all connected ports would be nice *for SharedWorkers*. For ports created dynamically by new MessageChannel, it doesn't seem to work well. A. TAKAYAMA
Re: [whatwg] Define MessagePort.isConnected or MessagePort.ondisconnect
If you don't need to ever broadcast something to all the ports, you can
avoid keeping track of the ports altogether, and then you won't have a
problem. If you do need to broadcast, it's hard not to slowly leak at the
moment.
Even with the example below, port and port.onmessage will be
created every time I reload the tab, and they will never be cleaned as
long as I don't close all the pages.
=== JS ==
onconnect = function(e) {
var port = e.ports[0];
port.onmessage = function() {
port.postMessage('yes');
}
}
=== /JS ==
Re: [whatwg] Define MessagePort.isConnected or MessagePort.ondisconnect
On Mon, 15 Mar 2010, ATSUSHI TAKAYAMA wrote:
If you don't need to ever broadcast something to all the ports, you can
avoid keeping track of the ports altogether, and then you won't have a
problem. If you do need to broadcast, it's hard not to slowly leak at the
moment.
Even with the example below, port and port.onmessage will be
created every time I reload the tab, and they will never be cleaned as
long as I don't close all the pages.
=== JS ==
onconnect = function(e) {
var port = e.ports[0];
port.onmessage = function() {
port.postMessage('yes');
}
}
=== /JS ==
In the example above, 'port' will be garbage collected once the
corresponding page has been closed, even if the shared worker continues.
--
Ian Hickson U+1047E)\._.,--,'``.fL
http://ln.hixie.ch/ U+263A/, _.. \ _\ ;`._ ,.
Things that are impossible just take longer. `._.-(,_..'--(,_..'`-.;.'
