[whatwg] Event loop processing model, and current time

[Simon Fraser]

https://html.spec.whatwg.org/multipage/webappapis.html#processing-model-9 says:

1. Let now be the value that would be returned by the Performance object's 
now() method
2. Let docs be the list of Document objects associated with the event loop in 
question…
...
4. For each fully active Document in docs, run the resize steps for that 
Document, passing in now as the timestamp
...

This makes no sense, as performance.now() is per-document (it’s relative to the 
document start time), so passing the same value to all documents in the 
browsing context is bogus.

What may be intended is to “freeze” the performance.now() time in all documents 
before processing those documents, but give each document its own 
performance.now() time.

Simon

Re: [whatwg] Canvas resetClip() API

[Simon Fraser]

On Oct 7, 2013, at 8:55 PM, Rashmi Shyamasundar  wrote:

> Hi All,
> 
> This is regarding the resetClip() API for canvas. Please take a look at my
> patches to the bug https://bugs.webkit.org/show_bug.cgi?id=82801.
> resetClip() can be implemented in Cairo, without any overhead, since Cairo
> supports resetClip.
> 
> I tried to implement the API in CG, using the below approach :- (Please
> check https://bugs.webkit.org/show_bug.cgi?id=82801#c9 )

I don’t like the resetClip() API, and don’t think it should be added to canvas.

The reason resetClip() is a bad API is that it breaks the ability to robustly 
modularize
drawing code. For example, say that you have the following code:

context.save();
// … set up a path to clip to
context.clip();
someJSLibrary.drawSomething();
context.restore();

In the absence of resetClip(), you are guaranteed that 
someJSLibrary.drawSomething()
can’t draw outside of the clip you specified. With resetClip(), that JS library 
can draw
wherever the heck it wants, which may totally break your canvas drawing.

If you really need resetClip() in your own drawing code, you’re probably doing 
it wrong.

Simon

Re: [whatwg] isPointInPath v. set of pixels in canvas hit regions

[Simon Fraser]

On Jul 5, 2012, at 2:25 PM, Ian Hickson  wrote:

> On Thu, 5 Jul 2012, Edward O'Connor wrote:
>> 
>> As things currently stand in the spec, implementations basically need to 
>> keep N+1 bitmaps per canvas, where N is the number of hit regions. I 
>> doubt any implementors would be enthusiastic to implement hit regions 
>> like this. From a WebKit perspective, we'd much prefer keeping a Path 
>> for each hit region, and then simply using isPointInPath for hit 
>> testing. This also implies that the current piggybacking of "Clear 
>> regions that cover the pixels" in clearRect() could go away. Yay! :)
> 
> You only need one bitmap to implement the hit testing.
> 
> (Or you can do it using paths, sure. The effect is the same, but it's 
> probably quicker to use a bitmap.)

I don't think the spec should be written with a particular implementation in 
mind, nor should it dictate one.

I also strongly object to having to update this hit testing bitmap and/or path 
set on drawing operations like clearRect(). That will potentially hurt 
performance. I think its up to the author to manage their set of paths 
appropriately, independently from the drawing operations.

Simon

[whatwg] Please rename this thread (Was: Re: whatwg Digest, Vol 82, Issue 10)

[Simon Fraser]

Please don't start or continue threads without a useful subject line.

Simon

Re: [whatwg] CSS canvas() function

[Simon Fraser]

On Dec 1, 2010, at 5:37 PM, Robert O'Callahan wrote:

> On Thu, Dec 2, 2010 at 2:25 PM, Tab Atkins Jr.  wrote:
> On Wed, Dec 1, 2010 at 5:20 PM, Robert O'Callahan  
> wrote:
> > In the absence of compelling use cases, I'd just leave it at , 
> > and  and whitelist in more elements later if necessary.
> 
> ?  / seem to have roughly equivalent
> use-cases to  (though perhaps we just want to encourage
> ).
> 
>  can contain fallback content which might get tricky.

So can .

It's a shame to disallow HTML elements with known width and height, e.g. if you 
want to render a small disconnected  subtree into a  or map it 
onto a WebGL texture. You'd have to decide how to resolve style (maybe assume 
it's a child of the body?).

Simon

Re: [whatwg] Canvas gradients color interpolation - change to premultiplied?

[Simon Fraser]

On Nov 26, 2010, at 6:38 PM, Boris Zbarsky  wrote:

> On 11/26/10 4:09 PM, Simon Fraser wrote:
>> This would be hard for WebKit, which relies on Core Graphics for gradients
>> on some platforms. CG doesn't allow us to interpolate in premultiplied
>> space.
> 
> But CSS gradients are already requiring interpolation in premutiplied space, 
> right?

I think you're thinking of CSS Transitions, which we decided should run in 
premultiplied.

Simon

Re: [whatwg] Canvas gradients color interpolation - change to premultiplied?

[Simon Fraser]

On Nov 23, 2010, at 12:43 PM, Tab Atkins Jr. wrote:

>  Implementors, does
> this sounds like a change you can get behind?  We already changed
> canvas shadows to match behavior with CSS shadows; this is a much
> smaller change for spec-equivalence.

This would be hard for WebKit, which relies on Core Graphics for gradients
on some platforms. CG doesn't allow us to interpolate in premultiplied space.

[Of course, some of the new radial gradient stuff is also not doable using
CG, so maybe we'll have to have a slower non-CG gradient code
path anyway].

Simon

Re: [whatwg] Processing the zoom level - MS extensions to window.screen

[Simon Fraser]

On Nov 20, 2010, at 7:46 AM, Ojan Vafai wrote:

> On Fri, Nov 19, 2010 at 9:21 PM, Robert O'Callahan  
> wrote:
> Most of the use cases for script access to the exact device pixel ratio that 
> I've heard boil down to "interfere with the user's ability to zoom", which is 
> why I haven't been eager to make it easier.
> 
> To be clear, chrome.tabs.getZoomPercentage is a Chrome extension API. Having 
> extensions that can mess with zoom seems like a legit use-case. But I agree, 
> I can't think of good use-cases for the general web being able to.

The only one I can think of is sizing the canvas backing store to get a sharp 
image on high-resolution displays, and possibly swapping in different image 
assets from JS.

Simon

Re: [whatwg] Timeouts and monotonic vs clock time

[Simon Fraser]

On Nov 3, 2010, at 8:01 PM, and-py wrote:

> Here's a curious little issue.
> 
> When you use `setTimeout` or `setInterval`, the HTML5 spec seems to say
> that the callback should occur after a certain amount of actual time has
> elapsed.
> 
> But what browsers might do is take the system clock, add the given
> number of milliseconds and call back when that system clock time is
> reached. The firing time will differ from the actual-elapsed-time, if
> the system clock is changed in between the setting and the calling of
> the callback.

Another issue to consider here is what happens when the machine goes to sleep,
and then wakes up. This is a much more common scenario than the user
changing the system clock.

Simon

Re: [whatwg] VIDEO Timeupdate event frequency.

[Simon Fraser]

On Sep 10, 2010, at 10:07 AM, Tab Atkins Jr. wrote:

> On Fri, Sep 10, 2010 at 9:58 AM, Simon Fraser  wrote:
>> The problem with a 'newFrame' callback is what to do if the callback
>> takes longer than the duration of a single frame. Does the video engine
>> start dropping frames, or does the video lag?
> 
> Dropping frames would be the better solution, for all the uses I'd put
> it to.  (Or rather, dropping newFrame events.)
> 
> 
>> In WebKit on Mac, video playback is hardware-accelerated, and the
>> presentation of video frames is disconnected from the web page
>> drawing machinery. A newFrame callback would force us to drop
>> back into software rendering, which is significantly more CPU intensive.
>> I don't support the general use of a 'newFrame' callback except in
>> the context of video processing via canvas.
> 
> In general, video processing via canvas is going to require dropping
> into software rendering, right?  I think that's what I was hearing
> from our dudes putting hardware-accelerated video into Chrome.  So at
> least in the case that I can see this often being put towards, you
> don't lose anything.

My concern would be pages registering for newFrame events just
to do stuff like updating a controller, which will vastly increase CPU
usage.

Simon

Re: [whatwg] VIDEO Timeupdate event frequency.

[Simon Fraser]

On Sep 10, 2010, at 7:53 AM, Tab Atkins Jr. wrote:

> On Fri, Sep 10, 2010 at 4:05 AM, Silvia Pfeiffer
>  wrote:
>> On Fri, Sep 10, 2010 at 7:28 PM, Biju  wrote:
>>> 
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=571822
 Firefox fires the timeupdate event once per frame.
 Safari 5 and Chrome 6 fire every 250ms.  Opera 10.50 fires every 200ms.
>>> 
>>> 
>>> Now in firefox bug 571822 they are changing Firefox fires the
>>> timeupdate event at every 250ms
>>> 
>>> But this takes away control of somebody who want to do some image
>>> process on every frame, as well as miss frames.
>>> 
>>> So can we have a "newFrame" event and/or a "minTimeupdate" property to
>>> say what should be the minimum time interval between consecutive
>>> timeupdate event.
>> 
>> If we have a newFrame event, might it be an idea to actually hand over the
>> frame data (audio + video) in the event? I would think that only ppl that
>> want to do manipulations on the media data want to have that kind of
>> resolution and it might be more efficient to just provide the data with the
>> event?
> 
> That would actually be a rather useful property.  I have several
> examples of video/canvas integration that I show off regularly at
> talks (and will have an article about on html5doctors.com soon), where
> I just listen to the play event and start running a function every
> 20ms, stopping when I see that the video is stopped or paused.  Just
> being able to register the function with a newFrame event instead
> would be useful in terms of avoiding unnecessary computation, and
> getting the data directly rather than having to draw the video into a
> backing canvas and then ask for its ImageData would shave some of the
> complexity off of the code.

The problem with a 'newFrame' callback is what to do if the callback
takes longer than the duration of a single frame. Does the video engine
start dropping frames, or does the video lag?

In WebKit on Mac, video playback is hardware-accelerated, and the
presentation of video frames is disconnected from the web page
drawing machinery. A newFrame callback would force us to drop
back into software rendering, which is significantly more CPU intensive.
I don't support the general use of a 'newFrame' callback except in
the context of video processing via canvas.

Simon

Re: [whatwg] Feedback on the Mozilla FullScreen API proposal

[Simon Fraser]

On Aug 5, 2010, at 5:56 PM, Robert O'Callahan wrote:

> On Fri, Aug 6, 2010 at 10:17 AM, Simon Fraser  wrote:
> This is feedback on the Mozilla FullScreen API proposal here:
> <https://wiki.mozilla.org/index.php?title=Gecko:FullScreenAPI>
> 
> > The user agent may transition a Document into or out of the
> > fullscreen state at any time, whether or not script has requested
> > it. User agents are encouraged to provide standard UI to exit the
> > fullscreen state, for example if the user presses the Escape key.
> 
> Does this imply that the UA has a "fullscreen" button, when
> the user uses this to make the window fullscreen, then the
> CSS pseudoclasses described in this document are applied?
> 
> Yes, that's the idea. We want to be able to integrate fullscreen UI with 
> scripted fullscreen.
>  
> I'm not sure that's always desirable; I think it might result
> in unexpected behavior for users who use the fullscreen button
> as an equivalent to maximizing the window.
> I'd suggest that the CSS pseudoclasses are only applied when the
> fullscreen JS API has been called by the web content.
> 
> Well, that's up to the UA; you can have a fullscreen mode that doesn't put 
> the document into the fullscreen state, if you want.
> 
> Note that the API lets authors distinguish "fullscreen with a specific 
> targeted element" vs "fullscreen with no targeted element", and a basic 
> fullscreen UI would only enter the latter mode. So I don't think you'll get 
> unexpected behavior in practice if they're integrated.
> 
> Personally I think they should be integrated. For example, if you have a 
> canvas game that occupies most of the window except for some instruction 
> text, and the game has a fullscreen command that makes the canvas fill the 
> window and resizes the canvas buffer to the window size, I think it would be 
> nice for the UA's fullscreen UI to trigger the same behavior. Or maybe you 
> have a slide viewer and you want the UA's fullscreen UI to trigger fullscreen 
> presentation mode. This API makes that possible.

OK, I think it's good to leave this up to the UA.
> 
> >  * void cancelFullScreen()
> 
> I think "exit" would be better than "cancel".
> 
> The only problem with "exit" is that you might call it when you're not 
> actually in fullscreen state, since requests are asynchronous. I think of it 
> more as cancelling the request than actually forcing a change.

That explains the method name, but I think the most common usage will be to 
exit fullscreen, rather than to cancel a recent request to enter fullscreen.
> 
> > New methods of Document:
> >  * void requestFullScreen()
> >  * void requestFullScreenWithKeys()
> 
> > New methods of Element:
> >  * void requestFullScreen()
> >  * void requestFullScreenWithKeys()
> 
> Do we really need to add these to both Document and Element?
> It seems like the Document ones could be removed; if an
> author wants to take the Documemnt fullscreen, they simply
> call the methods on document.documentElement.
> 
> Yes, we can do that. This means getting rid of the concept of not having a 
> fullscreen element, and instead special-casing the case where the fullscreen 
> element is the root element in a few places. But I think that's probably a 
> win. Thanks.

Cool.

> 
> I'm not fan of "WithKeys" in the method names. Out of
> context, it's not clear what "keys" means. I think
> "keyboardInput" or "keyInput" would be more understandable.
> 
> OK.
>  
> Also, why not just use a parameter:
> 
>void requestFullScreen(bool allowKeyboardInput)
> 
> boolean parameters are bad style IMHO. It's hard to tell at the call site 
> what they mean.
> 
> or have some constants for behavior:
> 
>const unsigned short ALLOW_KEYBOARD_INPUT = 1;
>void requestFullScreen(unsigned short behavior)
> 
> This would be extensible, and would allow us to permit other
> behaviors later.
> 
> Sure.

Yes, I like the 'behavior' version.

> 
> Why don't requestFullScreen()/requestFullScreenWithKeys() return
> a boolean value indicating whether the UA will allow the request
> to proceed? The author has no information about whether fullscreen
> is going to happen after making this call, and UAs will certainly
> want to deny fullscreen in various situations.
> 
> The UA may not be able to make a decision synchronously. Permitting 
> asynchronous decisions about whether to permit fullscreen was a key goal 
> here. For example that gives UAs the option of presenting p

[whatwg] Feedback on the Mozilla FullScreen API proposal

[Simon Fraser]

This is feedback on the Mozilla FullScreen API proposal here:


> The user agent may transition a Document into or out of the
> fullscreen state at any time, whether or not script has requested
> it. User agents are encouraged to provide standard UI to exit the
> fullscreen state, for example if the user presses the Escape key.

Does this imply that the UA has a "fullscreen" button, when
the user uses this to make the window fullscreen, then the
CSS pseudoclasses described in this document are applied?
I'm not sure that's always desirable; I think it might result
in unexpected behavior for users who use the fullscreen button
as an equivalent to maximizing the window.

I'd suggest that the CSS pseudoclasses are only applied when the
fullscreen JS API has been called by the web content.

>  * void cancelFullScreen()

I think "exit" would be better than "cancel".

> New methods of Document:
>  * void requestFullScreen()
>  * void requestFullScreenWithKeys()

> New methods of Element:
>  * void requestFullScreen()
>  * void requestFullScreenWithKeys()

Do we really need to add these to both Document and Element?
It seems like the Document ones could be removed; if an
author wants to take the Documemnt fullscreen, they simply
call the methods on document.documentElement.

I'm not fan of "WithKeys" in the method names. Out of
context, it's not clear what "keys" means. I think
"keyboardInput" or "keyInput" would be more understandable.
Also, why not just use a parameter:

void requestFullScreen(bool allowKeyboardInput)

or have some constants for behavior:

const unsigned short ALLOW_KEYBOARD_INPUT = 1;
void requestFullScreen(unsigned short behavior)

This would be extensible, and would allow us to permit other
behaviors later.

Why don't requestFullScreen()/requestFullScreenWithKeys() return
a boolean value indicating whether the UA will allow the request
to proceed? The author has no information about whether fullscreen
is going to happen after making this call, and UAs will certainly
want to deny fullscreen in various situations.

> New content attribute of the  element:
>  * allowfullscreen
> This is a boolean attribute. When this attribute is not set, UAs
> should ignore fullscreen requests in the iframe or its descendants.

Descendant nodes, or descendant iframes?

> * :full-screen-doc

I don't like the abbreviation of "document" here. Would "page"
be a better term to use than "document" for these?

>  * :full-screen-doc
> While a Document is in the fullscreen state, the 'full-screen-doc'
> pseudoclass applies to the root element of that Document.
> 
>  * :full-screen-doc-with-target
> While a Document is in the fullscreen state and the document's
> current fullscreen element is an element in the document, the
> 'full-screen-doc-with-target' pseudoclass applies to the root
> element of that Document.

Just to be clear, if a document is fullscreen, and the fullscreen
element is not the document element, then the document element
will get both these pseudoclasses? What happens when the
fullscreen element is the the document element?

I'd like to see the proposal fleshed out to address the following
scenarios:

* the document is fullscreen, and navigation happens
* the document is fullscreen, and the content calls requestFullScreen()
  again (possibly with a different element, possibly inside an iframe).
* the document is fullscreen, and the fullscreen element is removed
  from the DOM
* the document is fullscreen, and the fullscreen element has 
  display:none set on it.
* consideration of whether the fullscreen API can be called at
  any time (risk of "drive-by-fullscreening").
* affect, if any, on calls to focus() on elements which are not children
  of the fullscreen element.
  
Thanks
Simon

Re: [whatwg] suggestion for HTML5 spec

[Simon Fraser]

On May 2, 2010, at 2:20 am, Eduard Pascual wrote:

> On Sun, May 2, 2010 at 5:55 AM, Tab Atkins Jr.  wrote:
>> On Sat, May 1, 2010 at 8:25 PM,   wrote:
>>> My suggestion for the HTML5 spec is that the video tag should have a feature
>>> that can enable GPU acceleration on a user's graphics card, so it will take
>>> some stress off the CPU.
>>> 
>>> Do you like my suggestion?
>> 
>> Nothing is stopping browsers from using the GPU to help speed up
>>  playback right now, and in fact I think that some browsers are
>> either already doing so, or are planning to do so in the relatively
>> near future.
> 
> As a matter of fact, Microsoft already announced that IE9's rendering
> engine would use GPU acceleration as much as possible (leveraging MS
> Windows' DirectX technolgies) [1]. Some benchmarking of the platform
> preview vs. IE8 show a more than welcome improve on performance [2].
> Thus, it's not a matter of accelerating just . UA vendors are
> allowed to use any available technology, such as GPU processing, to
> improve the user experience on all aspects; and IE9's approach will
> benefit not only , but everything that the browser renders (for
> example, heavy JS-based animations may take huge profit from this).
> I'm not a big fan of Microsoft's IE software, but I must concede that
> the Redmond guys have taken a step on the right direction with this.
> Especially having seen how IE8 was painfully slower than its older
> siblings (IE6 and 7). I wouldn't be surprised if other browsers follow
> suit.

WebKit already has a "fast path" for video on Mac, and it's being worked
on for Windows.

The only reason I can think of why this should be exposed to web authors
is if a browser is unable to use GPU accelerated for video without side
effects (like always rendering on top of the rest of the content). Such an 
option would
be similar to the "wmode" param on plugins, which is a huge mess, and should
be avoided at all costs. One of the benefits of video being a real HTML element
is that it respects CSS properties like any other element.

Simon

Re: [whatwg] Fullscreen for HTML5 Video element

[Simon Fraser]

On Mar 9, 2010, at 8:01 PM, Robert O'Callahan wrote:

> On Wed, Mar 10, 2010 at 3:59 PM, Michael Dale  wrote:
> I was part of the initial thread that was left unresolved. I would just 
> re-iterate that its important the fullscreen system does not deprive the web 
> designer of flexibility to do DOM overlays / layouts.
> 
> You mean the thread "api for fullscreen()"? Actually I thought that thread 
> ended with reasonably strong consensus on what the API should be.

Agreed.

This thread is about API specifically on the video element, for taking the 
video (and only the video) fullscreen. WebKit has implemented this recently:


It allows WebKit to do a high-performance fullscreen implementation with a high 
quality cinematic, both of which are hard to guarantee with a more generic 
fullscreen API.

However, we thing there are use cases for both, which is why we also interested 
in the more generic API as well. If you need to do DOM overlays with video, 
then the generic fullscreen API is what you want.

Simon

Re: [whatwg] Canvas size and double buffering.

[Simon Fraser]

On Feb 3, 2010, at 7:59 AM, Boris Zbarsky wrote:

> On 2/3/10 9:05 AM, Tim Hutt wrote:
>> 1. You can only set the size exactly in pixels. It is very hard to get
>> a resizable canvas that fills the page. You *can* set the size in CSS,
>> but it doesn't work very well (e.g. using left,right-margin: auto; to
>> centre the canvas doesn't work. Also it scales the canvas rather than
>> resizes it.
> 
> That's right, because resizing the canvas (as in, changing the number of 
> pixels in the backing store) involves clearing it at the moment.  Your 
> proposal would mean that if the user changes the window size your canvas will 
> immediately get cleared.  Is that what you're after?  If not, how do you 
> envision the backing-store resize being performed?

Canvas drawing should have been callback-based from the start. Maybe we can 
retrofit this by having a way the author can specify that the canvas backing 
store resizes with the element, and firing an event when the backing store size 
changes.

>> 2. There doesn't appear to be any kind of double buffering capability.
>> This is pretty critical for animations where you might need to clear
>> the canvas and then draw stuff. You currently get flickering if you
>> try.
> 
> You do?  Under what circumstances?  At least in Gecko, run-to-completion 
> means that while you script is running the canvas won't be painted. What UAs 
> are you seeing flicker in, and under what circumstances?

Same with WebKit. As long as you clear and redraw at the same time, there 
should not be flicker.

Simon

Re: [whatwg] api for fullscreen() - security issues

[Simon Fraser]

On Feb 1, 2010, at 1:14 AM, Henri Sivonen wrote:

> On Jan 31, 2010, at 05:08, Simon Fraser wrote:
> 
>> * disallow enterFullscreen() from a frame or iframe
> 
> This might be a problem if video sites transition their embedding boilerplate 
> to an iframe in order to be able to be able to serve HTML5, Flash, ActiveX, 
> etc. depending on UA without requiring the embedders to copy and paste 
> anything fancy.

Perhaps we'd enforce a same-origin rule where the iframe contents have to be 
from the same domain as the main page, then?
> 
>> * show an hard-to-spoof overlay with some text that tells the user that they 
>> can use the Escape key to exit fullscreen, and prevent the page from 
>> capturing this keypress.
> 
> IIRC, it has been shown that at least as implemented in Flash Player, it is 
> possible to draw enough distractions to make the users unable to read this 
> message.

That's why I said "hard to spoof". The Flash overlay makes the mistake of not 
being contrasty enough. An improvement would be to dim out the rest of the 
content while showing this overlay.

> Also, when the site is legitimate, it's quite annoying to have the overlay 
> there.
> 
> Personally, I'd rather have to click through a once per-Origin authorization 
> bar (like geolocation in Firefox) than watch the "press esc" overlay every 
> time.

That's a possibility, yes.

> 
>> * make the location field available to the user so that they can see the URL 
>> even when in fullscreen
> 
> This defeats the point of full screen. If I want a 16:9 video to go full 
> screen on a 16:9 display, I want all screen pixels to be used for the video.

I said "make available", not "should be visible at all times". I was thinking 
of a behavior where it shows up at the top of the screen if the user moves the 
mouse close to the top.

>> * drop out of fullscreen if navigating to another page
> 
> This would constrain slide shows do be unnecessarily Ajaxy and less linkable 
> with per-slide JavaScriptless URLs.

Maybe a same-origin test could be used here too? But personally I think it's OK 
to force a fullscreen "application" to rely on AJAX and twiddling 
location.hash, rather than loading new pages.

Simon

Re: [whatwg] api for fullscreen() - security issues

[Simon Fraser]

On Jan 28, 2010, at 6:42 pm, Robert O'Callahan wrote:

> On Fri, Jan 29, 2010 at 12:51 PM, Simon Fraser  wrote:
> We have been discussing a more general fullscreen API that lets you take the 
> page fullscreen (perhaps with the ability to focus on a single element), as 
> Maciej mentions. We have not decided on a final form for this API, nor have 
> we resolved whether it's possible to do some nice transition between the two 
> modes. We have talked at some length about the security issues.
> 
> Input on what people would like from this API is welcome, as are ideas on how 
> the transitions should work.
> 
> 1) Should be convenient for authors to make any element in a page display 
> fullscreen
> 2) Should support in-page activation UI for discoverability
> 3) Should support changing the layout of the element when you enter/exit 
> fullscreen mode. For example, authors probably want some controls to be fixed 
> size while other content fills the screen.
> 4) Should accommodate potential UA security concerns, e.g. by allowing the 
> transition to fullscreen mode to happen asynchronously after the user has 
> confirmed permission
> 
> *** WARNING: totally half-baked proposal ahead! ***
> 
> New API for all elements:
> void enterFullscreen(optional boolean enableKeys);
> void exitFullscreen();
> boolean attribute supportsFullscreen;
> boolean attribute displayingFullscreen;
> "beginfullscreen" and "endfullscreen" events
> 
> The enableKeys parameter to enterFullscreen is a hint to the UA that the 
> application would like to be able to receive arbitrary keyboard input. 
> Otherwise the UA is likely to disable alphanumeric keyboard input. If 
> enableKeys is specified, the UA might require more severe confirmation UI.

I'd like to start a discussion on steps that the UA may take to mitigate the 
risks of using the fullscreen API for phishing attacks. I'm not sure how much 
should be required of UAs in the spec, but I could imagine that steps that the 
UA could take may include some or all of the following:

* require that enterFullscreen() is being called inside a user-event handler 
(e.g. click or keypress) to avoid drive-by fullscreen annoyances.
* disallow enterFullscreen() from a frame or iframe
* show an animation as the window enters fullscreen so the user can see the 
transition taking place
* show an hard-to-spoof overlay with some text that tells the user that they 
can use the Escape key to exit fullscreen, and prevent the page from capturing 
this keypress.
* show an affordance to allow the user to exit fullscreen (e.g. a close button) 
when the user moves the mouse
* make the location field available to the user so that they can see the URL 
even when in fullscreen
* drop out of fullscreen if navigating to another page
* if focussed on an element, drop out of fullscreen if that element is removed 
from the DOM
* limit arbitrary keyboard input unless 'enableKeys' is true
* disallow window.open etc. while in fullscreen

I'm just not sure which of these would be requirements in the spec.

Simon

Re: [whatwg] api for fullscreen()

[Simon Fraser]

On Jan 30, 2010, at 1:24 PM, Anne van Kesteren wrote:

> On Sat, 30 Jan 2010 22:12:47 +0100, Simon Fraser  wrote:
>> On Jan 29, 2010, at 9:54 PM, Robert O'Callahan wrote:
>>> So how about a Window API with an optional element component:
>>> void enterFullscreen(optional DOMElement element, optional boolean 
>>> enableKeys);
>>> void exitFullscreen();
>>> boolean attribute supportsFullscreen;
>>> boolean attribute displayingFullscreen;
>>> "beginfullscreen" and "endfullscreen" events
>>> 
>>> Where "beginfullscreen" and "endfullscreen" are targeted at the element if 
>>> one was provided, or else at the window, and bubble. While a window is 
>>> fullscreen, the root element and the designated fullscreen element, if any, 
>>> are given a pseudoclass "fullscreen". Then you can have some default rules 
>>> in the UA style sheet:
>>> *:root:fullscreen { overflow:hidden; }
>>> *:not(:root):fullscreen { position:fixed; left:0; top:0; bottom:0; right:0; 
>>> }
>> 
>> I'll go for that.
>> 
>> Another good argument to have this API on the window object is that 
>> fullscreen is effectively a state on the window. It would not make any sense 
>> to go fullscreen on one element, and then fullscreen on another element.
>> 
>> Should enterFullscreen() throw an exception if already fullscreen?
> 
> To stop polluting the Window object, might it make sense to put the new 
> members (other than event handler attributes) on window.screen?

This would require that the current window object is passed implicitly to the 
method, so the UA knows which window to take fullscreen, right?

This talk of going into fullscreen on different screens at the same time makes 
me think that the screen should be an optional argument:

void enterFullscreen(optional DOMElement element, optional Screen, optional 
boolean enableKeys);

with the default screen being either the main screen, or window.screen.

Simon

Re: [whatwg] api for fullscreen()

[Simon Fraser]

On Jan 29, 2010, at 9:54 PM, Robert O'Callahan wrote:

> So how about a Window API with an optional element component:
> void enterFullscreen(optional DOMElement element, optional boolean 
> enableKeys);
> void exitFullscreen();
> boolean attribute supportsFullscreen;
> boolean attribute displayingFullscreen;
> "beginfullscreen" and "endfullscreen" events
> 
> Where "beginfullscreen" and "endfullscreen" are targeted at the element if 
> one was provided, or else at the window, and bubble. While a window is 
> fullscreen, the root element and the designated fullscreen element, if any, 
> are given a pseudoclass "fullscreen". Then you can have some default rules in 
> the UA style sheet:
> *:root:fullscreen { overflow:hidden; }
> *:not(:root):fullscreen { position:fixed; left:0; top:0; bottom:0; right:0; }

I'll go for that.

Another good argument to have this API on the window object is that fullscreen 
is effectively a state on the window. It would not make any sense to go 
fullscreen on one element, and then fullscreen on another element.

Should enterFullscreen() throw an exception if already fullscreen?

Simon

Re: [whatwg] api for fullscreen()

[Simon Fraser]

On Jan 28, 2010, at 6:42 PM, Robert O'Callahan wrote:

> On Fri, Jan 29, 2010 at 12:51 PM, Simon Fraser  wrote:
> We have been discussing a more general fullscreen API that lets you take the 
> page fullscreen (perhaps with the ability to focus on a single element), as 
> Maciej mentions. We have not decided on a final form for this API, nor have 
> we resolved whether it's possible to do some nice transition between the two 
> modes. We have talked at some length about the security issues.
> 
> Input on what people would like from this API is welcome, as are ideas on how 
> the transitions should work.
> 
> 1) Should be convenient for authors to make any element in a page display 
> fullscreen
> 2) Should support in-page activation UI for discoverability

I agree with Boris that we should leave this up to the page author.

> 3) Should support changing the layout of the element when you enter/exit 
> fullscreen mode. For example, authors probably want some controls to be fixed 
> size while other content fills the screen.
> 4) Should accommodate potential UA security concerns, e.g. by allowing the 
> transition to fullscreen mode to happen asynchronously after the user has 
> confirmed permission

Agreed.
> 
> New API for all elements:
> void enterFullscreen(optional boolean enableKeys);
> void exitFullscreen();
> boolean attribute supportsFullscreen;
> boolean attribute displayingFullscreen;
> "beginfullscreen" and "endfullscreen" events

Those seem reasonable for a proposal where a single element is taken fullscreen.

> While an element is fullscreen, the UA imposes CSS style "position:fixed; 
> left:0; top:0; right:0; bottom:0" on the element and aligns the viewport of 
> its DOM window with the screen. Only the element and its children are 
> rendered, as a single CSS stacking context.

So this makes it a very element-focused API (as does the enterFullscreen() 
method on Element that you propose above).

Another approach would be to leave it entirely up to the page author to style 
their page differently when in fullscreen, and not have the API force them to 
focus on one element. Then the API would probably be on the Window object, and 
the UA would simply transition the view to a fullscreen presentation. There 
could be a pseudo-class to the body, or a way to use media queries to allow the 
author can apply different styles for fullscreen.

In this scenario the author is not forced to nest all their fullscreen content 
under one element, and can continue to show the rest of the page content (maybe 
dimmed out by a semi-transparent overlay div) in the background.

The disadvantage of this approach is that it makes it much harder for the UA to 
do any kind of automatic transitions between the two presentations (though it's 
not clear to me that nice automatic transitions will be possible with the 
element-based API either).

Simon

Re: [whatwg] api for fullscreen()

[Simon Fraser]

On Jan 28, 2010, at 3:38 PM, Robert O'Callahan wrote:

> On Thu, Jan 28, 2010 at 8:34 PM, Henri Sivonen  wrote:
> I haven't seen a proposal, but it looks like code has landed:
> http://trac.webkit.org/changeset/50893
> 
> Demo: http://jilion.com/sublime/video
> (option-click the full screen button in a fresh WebKit nigthly)
>  
> Interesting, but I'd still like to see the proposal. This should definitely 
> be something that works on more than just media elements.

The code referenced in that commit, and used by jilion.com, is NOT our proposal 
for a generic fullscreen mechanism. Rather, it's a new, webkit-prefixed method 
on the HTMLMediaElement which does the equivalent of the user clicking on the 
fullscreen button in the built-in controls. This is highly desirable for page 
that want to user their own custom controls, but would still like the user to 
go fullscreen.

We have been discussing a more general fullscreen API that lets you take the 
page fullscreen (perhaps with the ability to focus on a single element), as 
Maciej mentions. We have not decided on a final form for this API, nor have we 
resolved whether it's possible to do some nice transition between the two 
modes. We have talked at some length about the security issues.

Input on what people would like from this API is welcome, as are ideas on how 
the transitions should work.

Simon

Re: [whatwg] : A 3D Equivalent to

[Simon Fraser]

On Nov 2, 2009, at 4:26 PM, Brian Blakely wrote:


* Though it does not have properties for clipping, Webkit's proposed
implementation of 3D CSS does have them for perspective.  Clipping,
lighting, texture stretching and additional considerations could also
be a part of that spec, but those are discussions for the CSS WG.

Without a 3D media element, none of that work can be done.


Implementing your proposal would require that the model and surrounding
CSS-transformed content be implemented via the same 3D engine, sharing
a common coordinate system. That vastly increases the burden placed on
an implementor of 3D transforms: suddenly wafers in space are non longer
sufficient, and you need a new engine with support for all the features
required by your  content (which also needs to be specified  
somewhere).
You can't just glue a 3D-rendered model into an environment with  
CSS-3D-transformed

HTML elements and expect them to share a common 3D space.


* This proposal of a model  can be considered a direct 3D analog to a
PNG, and its height and width could certainly be modified in the X-Y
axes on which 2D elements live, affecting document flow in that
fashion.


A 3D analog for  is a fine goal, but the primary issue is that  
there is
no 3D file format which is accepted as a standard for this kind of  
use. Even
if there were, the analogy breaks down because 3D is structured data;  
the author

wants to be able to address certain objects in the 3D scene in order to
animate them, or do click handling on them, or whatever. There is much  
more
complexity here than there is with images. In this sense,  is  
as much

a black box for 3D as WebGL on a  is.

A further issue with this proposal is that it doesn't address another  
request,
which is to integrate HTML content into a true 3D scene (e.g. a   
mapped

onto a sphere, with operational hit testing etc).


* Webkit's implementation of CSS does not remove 3D-ified elements
from the document flow, the perspective of that flow is merely changed
only for the affected elements.  3D and 2D elements can exist
side-by-side that way as well.


Simon

Re: [whatwg] : A 3D Equivalent to

[Simon Fraser]

On Nov 2, 2009, at 3:41 PM, Brian Blakely wrote:


Additional clarification on this proposal:

A "model" Element Never Becomes a Wafer
---

Right now, if you try to act on conventional HTML elements with 3D
CSS, those elements become wafers.

See here: http://webkit.org/blog/386/3d-transforms/

One of the primary goals of embedding a 3D object (even a static,
non-interactive one) is that it can be rotated and translated in CSS
without losing its dimensional properties.  "globe.xml" will always
appear round, at any rotation degree, rather than like a cracker, as a
 or  would.


I disagree. In order to do this, you have to have some 3D space in which
the model element lives, and you need to be able to specify clipping  
planes,

perspective etc.

If elements are rotated *inside* the model, then of course they maintain
their 3D space. But the model element itself has to be planar, and has  
to

interact with other elements on the page just as any other HTML element
does.

Simon

Re: [whatwg] : A 3D Equivalent to

[Simon Fraser]

On Oct 30, 2009, at 6:09 PM, Brian Blakely wrote:

To ensure HTML remains semantic as the web makes its gradual  
transition to 3D rich interfaces and content, I am submitting a  
proposal for WHATWG's consideration.  The below examples contain  
HTML as it exists now, the current working standard, and finally, a  
combination of both concepts.



Please lend your thoughts and let's discuss a more semantic and  
accessible use of 3D.  The fictional HTML and CSS above are just  
proposals, and I am not at all asserting these ideas are the best  
possible solution, only that there is a very real problem.


Are you aware of the X3D and O3D efforts?




Simon

Re: [whatwg] Orientation event in Firefox

[Simon Fraser]

I'm a little concerned about use of the word "orientation" for these  
kinds of events.


WebKit on iPhone already uses the term "orientation" to mean "which  
way up is the device", i.e. in portrait or landscape, right-way-up or  
upside-down:



What you're talking about here is getting data from the accelerometer  
to describe arbitrary positions, so I think it would be clearer if  
these were called "accelerometer" events, or something else that  
distinguishes them from the usage that only applies to the cardinal  
angles.


Simon

On Sep 2, 2009, at 2:26 PM, Dave Burke wrote:

On Mon, Aug 31, 2009 at 11:06 PM, Doug Turner  
 wrote:

Thanks Dave for your comments.

I was thinking about update frequency.  right now, it is up to the  
UA (we are using 50ms).


To give an explicit example, if you want to run a high pass (say to  
detect only sudden movements), you might code a simple k-order FIR  
filter of the form:


   y[n] = b0.x[n] + b1.x[n-1] + ... +  bk.x[n-k]

The coefficient values b0 ... bk are a function of the sampling rate  
("update frequency") and hence needs to be settable or at least  
known to allow code portability.



units -- yes.  in g.  This is best.

magnetometer -- but in a different API, right?  I do not see  
directly how you would combined the two.


It's a different API, but (can be) closely related. You can present  
magnetometer information either as x, y, z of the magnetic field  
vector or as the special case of compass orientation. For example,  
Android has the same SensorEvent for both Sensor.TYPE_ACCELEROMETER  
and Sensor.TYPE_MAGNETIC_FIELD. Lots of interesting use-cases for  
compass ranging from map tile orientation, navigation, augmented  
reality, etc.



Also see: http://dougt.org/wordpress/2009/08/orientation-update/ for  
some demos.

Re: [whatwg] and acceleration

[Simon Fraser]

[Speaking with my Apple hat on.]

We agree that upgrading the hardware is not an acceptable answer to  
this question. Many web-enabled devices have much less CPU power than  
the average laptop or desktop (think mobile devices), but many have  
quite capable GPUs. On desktop machines too, hardware acceleration of  
video playback is essential for optimum performance, and for  
acceptable CPU usage with large videos in non-trivial content (e.g.  
content with CSS effects on the video, or lots of expensive-to-render  
content under or over the video).


Ignoring rotations, overlays etc. in order to get reasonable  
performance is an option, but should be considered a bug in the  
implementation.


You could also imagine an attribute on the video element akin to the  
"wmode" attribute for plug-ins, which allow the page author to opt-in  
to faster rendering, with the expectation that the visual rendering  
will suffer. However, we feel strongly that the video spec should not  
be encumbered by an attribute of this kind.


Taking the video full-screen is an approach that makes a lot of sense  
for mobile devices. It's unfortunate that the spec shies away from the  
full-screen issue.


In an ideal world, hardware acceleration of video playback would "just  
work", and the spec would have to say no more about it. We believe  
it's possible to do hardware acceleration of video (and other  
animating web content) and preserve rendering in many cases. WebKit  
actually has some experimental code for this.


I'm not sure how much an HTML spec will be able to say about triggers  
that may cause the video rendering to fall off the hardware- 
accelerated path, since many of those triggers will be CSS-related,  
and implementation details will differ between browsers. From our  
experience in WebKit, those triggers may include:

* clipping (via overflow on the video or an ancestor)
* overlapping elements
* blending (opacity)
* being affected by transforms (via SVG, or CSS transforms on video or  
an ancestor)

* CSS masks or reflections
* CSS box decorations (border, background)

If the spec does say something about performance of , I think  
it should be no more than a note that performance may differ across  
browsers, and can be affected in various ways that may be non-obvious  
to the page author, related to the layout and styling of the video and  
other elements on the page.


Simon

On Apr 28, 2009, at 6:07 PM, Ian Fette wrote:

Upgrade the hardware is not an acceptable answer. Video acceleration  
is meant to offload work from CPU (especially on constrained  
devices, e.g. mobile). You want to be able to do compositing on  
video card, so that you don't have to read the video out of the  
video card's memory, transfer it over the bus, to the CPU, do some  
transforms/overlays/..., and then send it back to the video card for  
display. Doing that absolutely kills framerate.


As we (browsers) implement  I think a lot of us are starting  
with software rendering. Certainly we want to be able to do hardware  
acceleration at some point. Perhaps some things we will still be  
able to do in hardware, e.g. overlays of HTML or certain transforms  
(if the video device supports saying "take this, translate it, and  
composite" and the rendering engine only needs geometry data.) Other  
things we might not be able to do in hardware (e.g. if you have  
"transparent" flash video on top, and Flash wants to know what  
pixels are underneath it, then we would have to read that data off  
of the video card, send it to CPU, ...)


I think what would be helpful is for browsers who are implementing  
 with hardware acceleration to publish information on what  
would make them fall back to software rendering. If it turns out  
that list is roughly similar across implementations, perhaps it  
could be added as a note in the spec that doing the following  
certain things may cause performance implications. We're probably  
not ready to do that yet given that we don't have enough  
implementation experience, but that would be my suggestion for how  
to move forward.


-Ian

On Tue, Apr 28, 2009 at 5:59 PM, Ian Hickson  wrote:
On Sat, 28 Mar 2009, Benjamin M. Schwartz wrote:
>
> The  tag has great potential to be useful on low-powered
> computers and computing devices, where current internet video  
streaming

> solutions (such as Adobe's Flash) are too computationally expensive.
> My personal experience is with OLPC XO-1, on which Flash (and  
Gnash) are

> terribly slow for any purpose, but Theora+Vorbis playback is quite
> smooth at reasonable resolutions and bitrates.
>
> The  standard allows arbitrary manipulations of the video  
stream
> within the HTML renderer.  To permit this, the initial  
implementations

> (such as the one in Firefox 3.5) will perform all video decoding
> operations on the CPU, including the tremendously expensive YUV->RGB
> conversion and scaling.  This is viable only for moderate  
resolutions

> and

Re: [whatwg] Video : Slow motion, fast forward effects

[Simon Fraser]

On Aug 7, 2008, at 12:23 PM, Charles Iliya Krempeaux wrote:


Hello,

On Thu, Aug 7, 2008 at 12:11 PM, Jonas Sicking <[EMAIL PROTECTED]>  
wrote:

Dave Singer wrote:
At 20:10  +1200 7/08/08, Chris Double wrote:
On Thu, Aug 7, 2008 at 6:20 PM, Ian Hickson <[EMAIL PROTECTED]> wrote:
 On Thu, 7 Aug 2008, Biju [EMAIL PROTECTED] wrote:

 On Thu, Aug 7, 2008 at 1:49 AM, Ian Hickson <[EMAIL PROTECTED]> wrote:
 > playbackRate is the right way to do it, but maybe Firefox doesn't  
yet

 > support it.

 So can I assume HTML5 spec also allow playbackRate  to be negative  
value.

 ie to support go backward at various speed

 Yes.

Would you expect the audio to be played backwards too?

I think that's extra credit and optional.  As you say, even with  
audio coded in independent frames you have to flip the samples,  
which is a pain.  For audio with forward dependencies, correct  
decoding means decoding forwards and then flipping whole chunks of  
timeline (though AAC doesn't suffer too badly if you don't do this,  
by the way).


Honestly, this seems useless enough that the spec should just say  
that when playback is less than 0 sound should be turned off. I'd  
hate to see engineers working on this just because "the spec says it  
should work that way".


I don't think turning sound off is a good idea.

This feature would be used to implement "scrubing".  Like what you  
see in Non-Linear Editors... for making movies, etc.  (I.e.,  
grabbing the "position handle" of the player, and moving it forwards  
and backwards through the video, and varying speeds, to find what  
you are looking for.)


In those types of applications, the audio is on.  And it is  
important for usability, for the video editor to hear the sound.


But those applications are not naïvely playing the sound at the  
current playback rate; they are grabbing little snippets of audio  
samples from the current playhead position, and playing them at normal  
rate. I don't think the spec should go into this much detail about  
what happens to audio when rate != 1.


Simon

[whatwg] Focus management

[Simon Fraser]

Some web applications may need more control over focus than is offered  
by

the existing focus management proposal:


Specifically, it's hard to write JavaScript that has more explicit  
control

over focus changes. Functionality that is missing (but often included in
native UI toolkits) includes:

* Focus chain queries
need the ability to ask the document what the next/previous  
focusable

element is from a given element, or from null (first/last focusable
element).

* Is element focusable
need to be able to ask an element if it can take focus.  
Focusability is

currently some function of element type, tabindex, visibility,
contentEditability, UA preferenes etc, and it's hard to write JS
that computes this.

* Explicit advance/rewind focus
need to be able to move focus to the next/previous focusable
element without having explicit knowledge of what the next
element is (this mirrors what happens when the user hits the Tab
or Shift-Tab keys).

(If nextFocus()/previousFocus() are available, this could be  
achieved with

document.nextFocus(document.activeElement).focus(),
 but that seems a little long-winded.)

In addition, I'd like to see a few clarifications in the "Focus  
management"

section of the Web Applications draft:

* What does focus() do on an unfocusable element? Does the previously
  focused element remain focused?

* document.activeElement has some ambiguities:
- Is it valid when the window does not have focus?
- IE has a setActive() method that changes the activeElement, but
does not change the focus. So the activeElement is not always
the focused element.

* how does display: none or visibility: hidden interact with focus?
- if a focused element becomes unrendered, does focus move to the
  next focusable element?
- can non-rendered elements be focused?

Finally I'd like to see some discussion around focus() and window  
activation.
window.focus() obviously (and annoyingly) activates the window, but  
should

focussing an element inside a window raise the window? What happens if
that window is a hidden tab in a tabbed browser?

Simon