Re: [webvr] [gamepad] Missing VRPose for tracked controllers

[Brandon Jones]

Position/Orientation/Acceleration can always be reported as NULL, so that's
what I would expect from controllers that don't support a particular pose
value. That's a bit of a pain to use for feature detection purposes,
though, so maybe a more explicit caps would be nice?

As for determining what the controllers are relative to, that's a good
question. We'll have the vrDisplayID for controllers that are associated
with an HMD, which I think implies that they're in the same space as the
HMD. Not sure what makes sense for, say, a Wii controller. Pitch and Roll
are always relative to gravity, but Yaw probably just initializes to
whatever position the user started with.

On Tue, May 24, 2016 at 2:39 PM Justin Rogers <just...@gmail.com> wrote:

> Apologies if this comes across twice - the mailing list rejected my first
> attempt :-)
>
> Do we need to know if a given Gamepad is orientation only versus
> position+orientation? I'm seeing a lot of devices that only know their
> orientation and we could detect this by a 0,0,0 position, but having a flag
> telling me if position will change could be useful. I just did a quick
> review and I didn't see anything that would help me here.
>
> Also, what about devices where the controllers report relative to the HMD
> and the HMD is relative to world space? Do we want any new prose on VRPose
> that covers these potentially more complicated scenarios?
> On Tue, May 24, 2016 at 11:12 AM, Kearwood "Kip" Gilbert <
> kgilb...@mozilla.com> wrote:
>
>> Brandon Jones (Google) has made some experimental WebVR Chromium builds
>> that include support for the HTC Vive VR motion controls.  We would like to
>> propose this as a standard and update to the GamePad API.
>>
>> Brandon has started a thread on w3.org that gives some background:
>>
>> https://lists.w3.org/Archives/Public/public-webapps/2016AprJun/0052.html
>>
>> The WebVR API 1.0 has already extended the GamePad API by adding a
>> “displayID” attribute:
>>
>> https://mozvr.com/webvr-spec/#interface-gamepad
>>
>> The proposal is to take this further and expose an optional “pose”
>> attribute, which would return the same attributes within the VRPose:
>>
>> https://mozvr.com/webvr-spec/#interface-vrpose
>>
>> Cheers,
>>  - Kearwood “Kip” Gilbert
>>Platform Engineer, Mozilla VR Team
>>
>> On May 24, 2016, at 12:22 AM, Sven Neuhaus <sven...@sven.de> wrote:
>>
>> Hello Florian,
>>
>> Thanks for pointing out the WebVR spec draft.
>>
>> The WebVR draft dated April 1st contains a Gamepad interface expansion
>> (§2.11), however it only extends it by a DisplayId.
>> It should also add a VRPose for tracked controllers.
>>
>> I think adding a VRPose could have benefits for non-VR applications as
>> well (think about the Nintendo Wii controllers!), however. So my
>> suggestion to add it to the Gamepad API still stands.
>>
>> Regards,
>> -Sven Neuhaus
>>
>> Am 23.05.2016 um 15:52 schrieb Florian Bösch:
>>
>> The WebVR API models HMD pose and will model the gesture controllers.
>> https://mozvr.com/webvr-spec/
>>
>> On Tue, May 17, 2016 at 9:41 AM, Sven Neuhaus <sven...@sven.de
>> <mailto:sven...@sven.de>> wrote:
>>
>>
>>I read the gamepad API description at
>>https://developer.mozilla.org/en-US/docs/Web/API/Gamepad
>>
>>I think the gamepad API should support a VRpose for gamepad controllers
>>like the ones included with the HTC Vive and the upcoming Oculus Touch
>>controllers.
>>
>>I suggest that you add a getPose() method that returns a VRPose object
>>for controllers that support tracking.
>>
>>The "orientation" property of the VRPose object could also be useful
>> for
>>some gamepads that include IMUs for orientation tracking.
>>
>>
>> ___
>> web-vr-discuss mailing list
>> web-vr-disc...@mozilla.org
>> https://mail.mozilla.org/listinfo/web-vr-discuss
>>
>>
>>
>> ___
>> web-vr-discuss mailing list
>> web-vr-disc...@mozilla.org
>> https://mail.mozilla.org/listinfo/web-vr-discuss
>>
>> ___
> web-vr-discuss mailing list
> web-vr-disc...@mozilla.org
> https://mail.mozilla.org/listinfo/web-vr-discuss
>

Re: [gamepad] New feature proposals: pose, touchpad, vibration

[Brandon Jones]

Regarding the vibration portion of the API, I'm still not sure how much
control we should provide. Don't want to get too closely tied to any
specific native implementation. To that end I've put together a brief
document comparing the APIs that I'm aware of:
https://docs.google.com/document/d/1Ms4RFTI8bz1dTtympPmkb81FLhPdls7n-mfwQoAeyAI/edit?usp=sharing

Comments are welcome, especially if there's a Gamepad API that I've missed.

On Mon, Apr 25, 2016 at 9:03 AM Brandon Jones <bajo...@google.com> wrote:

> Lots of good feedback! Replies inline.
>
> > Do we want to know the value of the press? Does the API for the Steam
> Controller, for example, return the force of the press as a float?
> > Likewise, Pointer Events have "pressure"
>
> GamepadButton already has a "value" property, which I think would be
> appropriate to use here. Today it's primarily used for things like analog
> triggers or pressure sensitive buttons, but I can't see any reason why it
> wouldn't work for touchpads as well. Put another way: while plenty of
> touchpads have both a touched and a clicked state, I can't think of a
> scenario where a touchpad would have a separate concept of "touch force"
> and "pressed amount".
>
> > Just as a quick API surface comment, Float32Array? is not a very good
> way of representing a vector (or quaternion).
> > Web VR seems to have changed from DOMPointReadOnly to Float32Array? for
> some reason. Maybe they can explain?
>
> I should probably explain that I'm an author of the WebVR spec who also
> happens to be the current primary Gamepad developer on Chrome. :) Of
> course, that makes my opinions on cross pollination between the two APIs
> very non-neutral, which is why I think it's important to gather feedback
> like this. Thanks!
>
> The WebVR API switched from DOMPoint to Float32Arrays for a couple of
> reasons, probably the biggest being that the DOMGeometry types still seem
> to be in spec limbo. This is especially true of the DOMMatrix type, which
> appears to be fairly contentious from reading through the mailing lists. We
> were reluctant to continue relying on a spec with an unclear future or
> timeline.
>
> A common concern raised when talking about these types is that they may
> not be as performant as developers are expecting them to given their
> "native" implementations, since the overhead of transitioning in and out of
> the Javascript VM may swamp out the perf benefits you would otherwise see.
> Counterintuitively, in order to get the best performance it may be best to
> do all of your vector and matrix math in Javascript.
>
> To that end, libraries already exist that operate on Float32Arrays
> directly for vector/matrix math (http://glmatrix.net). And SIMD JS, which
> can improve the performance of these operations up to 4X in some cases,
> expects values to be packed into TypedArrays anyway. It was those potential
> benefits and the desire to not create another explicit vector and matrix
> type to compete against DOMGeometry that lead us to use flat arrays in
> WebVR.
>
> > Why are you interested in creating a competing extension to the Gamepad
> API, instead of working with the Web VR folks on their interface?
>
> I've already explained my relationship to WebVR, so I'll point out that I
> would view this as a complimentary extension, not a competing one. In a
> prior iteration of the WebVR spec things were abstracted out in such a way
> that you could have exposed 6DoF information about gamepads via WebVR
> instead of the Gamepad API. This proved to make using the API very
> confusing for developers. Not only would it have been easy to mistake a
> tracked gamepad for a headset, it required some awkward references between
> APIs to say "this set of pose information is associated with this gamepad
> index at this timestamp". And in some cases (Wii remote) it wouldn't be
> obvious why you had to route through WebVR at all in order to get what
> should be intrinsic properties of the gamepad itself.
>
> I wouldn't be opposed at all to having Pose be a shared interface between
> APIs, used by both WebVR and the Gamepad API. I do feel like it's quite a
> bit more natural for developers to have each context surface their own pose
> values, though.
>
> I know I haven't fully covered everyone's comments so far, but I wanted to
> clarify my relationship to the WebVR API and explain some of the logic
> behind it's choices. I have some thoughts on the vibration concerns as
> well, but I want to gather a bit more reference material before I dive into
> that. Thanks for everyone's input thus far!
>
> --Brandon
>

Re: [gamepad] New feature proposals: pose, touchpad, vibration

[Brandon Jones]

Lots of good feedback! Replies inline.

> Do we want to know the value of the press? Does the API for the Steam
Controller, for example, return the force of the press as a float?
> Likewise, Pointer Events have "pressure"

GamepadButton already has a "value" property, which I think would be
appropriate to use here. Today it's primarily used for things like analog
triggers or pressure sensitive buttons, but I can't see any reason why it
wouldn't work for touchpads as well. Put another way: while plenty of
touchpads have both a touched and a clicked state, I can't think of a
scenario where a touchpad would have a separate concept of "touch force"
and "pressed amount".

> Just as a quick API surface comment, Float32Array? is not a very good way
of representing a vector (or quaternion).
> Web VR seems to have changed from DOMPointReadOnly to Float32Array? for
some reason. Maybe they can explain?

I should probably explain that I'm an author of the WebVR spec who also
happens to be the current primary Gamepad developer on Chrome. :) Of
course, that makes my opinions on cross pollination between the two APIs
very non-neutral, which is why I think it's important to gather feedback
like this. Thanks!

The WebVR API switched from DOMPoint to Float32Arrays for a couple of
reasons, probably the biggest being that the DOMGeometry types still seem
to be in spec limbo. This is especially true of the DOMMatrix type, which
appears to be fairly contentious from reading through the mailing lists. We
were reluctant to continue relying on a spec with an unclear future or
timeline.

A common concern raised when talking about these types is that they may not
be as performant as developers are expecting them to given their "native"
implementations, since the overhead of transitioning in and out of the
Javascript VM may swamp out the perf benefits you would otherwise see.
Counterintuitively, in order to get the best performance it may be best to
do all of your vector and matrix math in Javascript.

To that end, libraries already exist that operate on Float32Arrays directly
for vector/matrix math (http://glmatrix.net). And SIMD JS, which can
improve the performance of these operations up to 4X in some cases, expects
values to be packed into TypedArrays anyway. It was those potential
benefits and the desire to not create another explicit vector and matrix
type to compete against DOMGeometry that lead us to use flat arrays in
WebVR.

> Why are you interested in creating a competing extension to the Gamepad
API, instead of working with the Web VR folks on their interface?

I've already explained my relationship to WebVR, so I'll point out that I
would view this as a complimentary extension, not a competing one. In a
prior iteration of the WebVR spec things were abstracted out in such a way
that you could have exposed 6DoF information about gamepads via WebVR
instead of the Gamepad API. This proved to make using the API very
confusing for developers. Not only would it have been easy to mistake a
tracked gamepad for a headset, it required some awkward references between
APIs to say "this set of pose information is associated with this gamepad
index at this timestamp". And in some cases (Wii remote) it wouldn't be
obvious why you had to route through WebVR at all in order to get what
should be intrinsic properties of the gamepad itself.

I wouldn't be opposed at all to having Pose be a shared interface between
APIs, used by both WebVR and the Gamepad API. I do feel like it's quite a
bit more natural for developers to have each context surface their own pose
values, though.

I know I haven't fully covered everyone's comments so far, but I wanted to
clarify my relationship to the WebVR API and explain some of the logic
behind it's choices. I have some thoughts on the vibration concerns as
well, but I want to gather a bit more reference material before I dive into
that. Thanks for everyone's input thus far!

--Brandon

[gamepad] New feature proposals: pose, touchpad, vibration

[Brandon Jones]

I'd like to propose the addition of several new features to the Gamepad
API, primarily born from needs that have been identified while developing
the WebVR spec , but which also cover
topics that have been under discussion in the past but deemed lower
priority for one reason or another.

First, I'm proposing that we add an optional "pose" object to the the
Gamepad interface that would expose all the information necessary to track
a controller with 3 Degree of Freedom or 6 Degree of Freedom tracking
capabilities. The primary motivator for supporting this information is to
allow devices like HTC Vive controllers or Oculus Touch to be exposed, but
the same structure could also expose motion and orientation data for
devices such as Wii, PS3, and PS4 controllers, as well as anything else
that had a gyroscope and/or accelerometer.

interface *GampadPose* {
  readonly attribute Float32Array? position;
  readonly attribute Float32Array? linearVelocity;
  readonly attribute Float32Array? linearAcceleration;

  readonly attribute Float32Array? orientation;
  readonly attribute Float32Array? angularVelocity;
  readonly attribute Float32Array? angularAcceleration;
};

partial interface *GampadPose* {
  readonly attribute GamepadPose? pose;
};

Position is given in meters from a device-specific origin. Velocity and
acceleration are given it meters per second, and orientation is a
quaternion. If provided all arrays should have 3 elements to except the
orientation, which has 4.

The proposed pose interface largely mirrors the VRPose
 interface from the
WebVR spec, but drops the timestamp since that's already provided on the
Gamepad interface itself. The VRPose interface could conceivably be used
directly, but I'm assuming that we'd rather not have a dependency on a
separate work-in-progress spec.

Second, I'd like to see the API more explicitly support devices that have
touchpads. It's my feeling that the existing axis array is sufficient for
exposing the touch position, but we need a way to indicate if the touchpad
is currently being touched to disambiguate a value of zero from no input.
This can be handled currently by assigning a button to report pressed =
true when the touchpad is touched, but since many touchpads also can be
clicked as a button you end up with two buttons associated with a single
input. I'd like to proposed that we extend GamepadButton with a "touched"
boolean to make this case explicit.

interface *GampadButton* {
  readonly attribute boolean pressed;
  readonly attribute double value;
  readonly attribute boolean touched;
};

For non-touch controls it seems sensible that this would always report
false.

Finally, I'd like to propose that we add gamepad vibration controls. This
has been talked about in the past as well, and the thinking has typically
been that the Gamepad API may want to try and use the Vibration API
 somehow instead of exposing a new
interface. However that API doesn't currently have a way to target anything
other than the device's default motor (usually for a phone), and I don't
see there being much interest in extending it to do so. Still, I don't see
much sense in re-inventing this wheel so I suggest exposing the same
interface on the Gamepad object.

partial interface *Gampad* {
boolean vibrate (VibratePattern
 pattern);
};

This is pretty basic as far as gamepad haptics go, but it's also provides a
lowest common denominator interface that should be supportable by anything
with a motor in it. It gives up some granularity (For example, it doesn't
allow developers to trigger the two motors in an Xbox controller
individually), but it provides a clean, universal interface that's easy to
use and mimics existing features on the web platform.

I've included several developers that have been participating in WebVR's
development to provide some feedback from that end of things. These are
some reasonably large additions, but they can be layered on to the API as
it stands today without breaking backwards compatibility and cover features
that have been discussed as possible additions previously. Hopefully we can
figure out a way to incorporate them that works for everyone.

--Brandon

Re: [gamepad] Allow standard-mapped devices to have extra buttons or axes allow multiple mappings per device

[Brandon Jones]

Re: change #1, with standard gamepad mappings today (at least in Chrome) we
map any buttons that don't correspond to the official standard mapping to
button[17] and up. This, of course, depends on which buttons are actually
visible to the browser (many devices expose buttons that can't be seen by
normal applications, like the home button on Xbox controllers). A good
example is the Touchpad on the PS4 controller. We can't see actual touch
inputs without writing a custom driver, but we detect clicks on the
touchpad and report them as button[17]. The same would go for axes, but
I've yet to encounter a standard-looking gamepad that exposed  four axes
worth of usable data. Perhaps the spec should be slightly reworded in this
regard, since it currently states that The standard gamepad has 4 axes,
and up to 17 buttons, which could be read as explicitly disallowing
buttons or axes beyond that.

I'm not very supportive of suggestion #2 since we don't actually HAVE any
other mappings defined (joystick/flightstick would be the next most natural
candidate, I think, but they can vary significantly). If we did, I'm not
sure how you would reasonably stack mappings without breaking an awful
lot of code. The mappings field would have to be either space/comma
delimited (which breaks things) or turned into an array (which breaks
things) or you would expose the same device twice under different mappings
(which is really weird and probably breaks things.) Beyond that, there's no
way that the layouts of two mapping types could every overlay completely
cleanly, and we really shouldn't be claiming that something that looks like
a racing wheel is actually a standard gamepad anyway. I just can't see
how it would work.

--Brandon



On Sun Oct 12 2014 at 11:59:16 PM Chad Austin caus...@gmail.com wrote:

 Hi all,

 I recently was asked to review the Gamepad API draft specification.  My
 background is games though I've done some scientific computing with
 alternate input devices too.

 http://chadaustin.me/2014/10/the-gamepad-api/

 I'd like to suggest two simple changes to the Gamepad API.  In tandem,
 these APIs would allow applications to have automatic compatibility with a
 broader range of devices.

 The changes are:

 1. Allow standard-mapped devices to have extra buttons or axes
 2. Add support for multiple standard mappings per device.

 Together, these changes would allow a simple application that only needs a
 d-pad and a few buttons to work automatically with a Wiimote, XInput
 device, and a racing wheel (many racing wheels have d-pads on them).

 Even without the proposal #2, proposal #1 would allow optional access to
 extra capabilities on a device without having to sacrifice the standard
 mapping.

 This feels noncontroversial to me because mappings only assign meaning to
 otherwise neutral indices in the buttons[] and axes[] arrays.  For a given
 device, there can be many such meanings.

 --
 Chad Austin
 http://chadaustin.me

Re: [Gamepad] Liveness of Gamepad objects

[Brandon Jones]

I don't think that API design should be driven by implementation details,
but it may be useful to provide some background on Chrome's Gamepad
implementation here:

The current polling-only API Chrome uses is largely due to practicality
with the multiprocess architecture. The gamepad data is acquired in the
central browser process, placed in shared memory, and then polled by the
tab processes when getGamepads() is called. This cuts down on a lot of IPC
overhead that would otherwise result from constant Hey, Gamepad N
changed! messages broadcast to all tabs.

Given that, if we decide that the gamepad objects should be live and not
snapshots I think that Chrome will probably put a system in place where the
first access of a gamepad object property does a behind-the-scenes poll and
then does not poll again until control has been given back to the browser.
(Which is probably not an issue since many games will be processing input
in a rAF loop.) I can't think of any scenario where a forced policy of one
update per frame would be detrimental, but I'm open to feedback on the
matter.

If it were my call I'd vote for gamepad objects being snapshots simply
because it will add some additional overhead to make them live and it avoid
concerns about observing GC, but I don't really object to either design.

--Brandon

On Tue Apr 29 2014 at 10:18:24 AM, Brendan Eich bren...@secure.meer.net
wrote:

 Boris Zbarsky wrote:
  For what it's worth, the way Gecko implements this is by setting up
  that lifetime guarantee only when an expando is added to the object
  (or some other things, like use as a WeakMap key, happen).  Until then
  we allow it to be GCed.

 What do other engines do in general? Unobservable GC is a requirement,
 if you ask me and Mark Miller.

 /be

Re: [Gamepad] Liveness of Gamepad objects

[Brandon Jones]

On Tue Apr 29 2014 at 10:24:48 AM, Mark S. Miller erig...@google.com
wrote:

 I don't know anything about Gamepad. Could someone provide enough context
 that I can understand the question? Thanks.


The Gamepad API returns an array of Gamepad state objects when you call
getGamepads(), like so:

requestAnimationFrame(function() {
var gamepads = navigator.getGamepads();
var playerGamepad = gamepads[0];
if (playerGamepad.buttons[0].pressed) {
player.jump();
}
}

That code currently would work in both Firefox and Chrome. The Liveness
question concerns the following code:

var gamepads = navigator.getGamepads();
var playerGamepad = gamepads[0];
requestAnimationFrame(function() {
if (playerGamepad.buttons[0].pressed) {
player.jump();
}
}

In Firefox this code will be functionally equivalent to the first code
snippet, since it continues to update the values Gamepad state object after
you've acquired it. AKA: The object is live.

In Chrome, however, the gamepad state returned is a snapshot and as such
the second code snippet will not observe changes to the gamepad input.

The spec is currently ambiguous on the matter, so we want to clarify
whether or not the second code snippet is valid or not and update the
browser implementations accordingly.

--Brandon


 On Tue, Apr 29, 2014 at 10:16 AM, Brendan Eich bren...@secure.meer.netwrote:

 Boris Zbarsky wrote:

 For what it's worth, the way Gecko implements this is by setting up that
 lifetime guarantee only when an expando is added to the object (or some
 other things, like use as a WeakMap key, happen).  Until then we allow it
 to be GCed.


 What do other engines do in general? Unobservable GC is a requirement, if
 you ask me and Mark Miller.

 /be




 --
 Cheers,
 --MarkM

Re: [Gamepad] Liveness of Gamepad objects

[Brandon Jones]

On Tue Apr 29 2014 at 4:28:31 PM, Glenn Maynard gl...@zewt.org wrote:

 (That said, I'm confused--where's the event to tell the user that the
 gamepad has changed?  Surely this API doesn't require the developer to
 poll, which would lose inputs at the slightest GC skip and could never give
 high resolution timing.)


This is slightly off topic, but worth addressing. The spec does not, in
fact, have any change notifications. Firefox has some experimental ones you
can enable but they're not official. This does indeed provide an
opportunity for input loss, but I'm not aware of anyone who's actually
found it to be a problem. (Given the sparse number of apps using the API,
though, that doesn't say much.)

This is good feedback, though. It may be that snapshot + change events is
the most effective way to go?


 No, the object should be mutable, since most web API objects have mutable
 properties.  If I want to do things like

 gamepadState = getGamepads()[0]
 // Run the input logic, but pretend the A button isn't pressed:
 gamepadState.aButton = false;
 checkInput(gamepadState);

  then I should be allowed to.  There's nothing strange about this in a JS
 API.


No complaints here. A good use-case for mutable properties may be a JS
script that intercepts gamepad snapshots and re-maps them in-place to a
standard mapping if the browser has not already done so.

Re: Pointer Lock Status

[Brandon Jones]

On http://status.modern.ie/ Microsoft lists Pointer Lock as still Under
Consideration


On Thu, Apr 3, 2014 at 1:38 PM, Vincent Scheib sch...@google.com wrote:

 Implementation status:
 Chrome: Implemented with prefix, in stable version.
 Firefox: Implemented with prefix, in stable version.
 IE: No implementation progress known. Issue:
 http://connect.microsoft.com/IE/feedback/details/793718/ie11-feature-request-support-for-pointer-lock-api
 Safari: No implementation progress known.
 Opera: No implementation progress known.

 Test suite status:
 Two basic tests exist:
 http://w3c-test.org/pointerlock/constructor.html
 http://w3c-test.org/pointerlock/idlharness.html
 which fail in Chrome and Firefox due to prefixing, and likely correctness
 issues are masked by this.

 Plan to complete test suite:
 - Implementations need to start using W3C tests that exist
   - Thus implementations need to support unprefixed pointer lock.
   - Chromium issue:
 https://code.google.com/p/chromium/issues/detail?id=359740
 - More tests are needed.
   - Chromium and Mozilla have possible candidates for upstreaming.
 - I sense little prioritization from either organization to do this
 soon. (Correct me if Mozilla has anyone available to do this)

 Additional specification discussion:
 Artillery started a discussion Problems with mouse-edge scrolling and
 games in public-webapps Feb 21 2014 raising the topic of limiting pointer
 movement to a rectangular area. This is addressed in the spec FAQ for why
 it is postponed from the initial version, and in the recent thread the
 result was a call for prototype. I don't have resources to build it out,
 though would accept others doing so. If this was done rapidly and Mozilla
 was interested in incorporating changes as well there's the possibility
 that Pointer Lock spec should be updated to include clipping. Otherwise, I
 believe a follow up specification at a later time is more appropriate,
 keeping Pointer Lock narrow and moving to complete the specification.

Re: Proposal Virtual Reality View Lock Spec

[Brandon Jones]

So there's a few things to consider regarding this. For one, I think your
ViewEvent structure would need to look more like this:

interface ViewEvent : UIEvent {
readonly attribute Quaternion orientation; // Where Quaternion is 4
floats. Prevents gimble lock.
readonly attribute float offsetX; // offset X from the calibrated
center 0 in millimeters
readonly attribute float offsetY; // offset Y from the calibrated
center 0 in millimeters
readonly attribute float offsetZ; // offset Z from the calibrated
center 0 in millimeters
readonly attribute float accelerationX; // Acceleration along X axis in
m/s^2
readonly attribute float accelerationY; // Acceleration along Y axis in
m/s^2
readonly attribute float accelerationZ; // Acceleration along Z axis in
m/s^2
}

You have to deal with explicit units for a case like this and not
clamped/normalized values. What would a normalized offset of 1.0 mean? Am I
slightly off center? At the other end of the room? It's meaningless without
a frame of reference. Same goes for acceleration. You can argue that you
can normalize to 1.0 == 9.8 m/s^2 but the accelerometers will happily
report values outside that range, and at that point you might as well just
report in a standard unit.

As for things like eye position and such, you'd want to query that
separately (no sense in sending it with every device), along with other
information about the device capabilities (Screen resolution, FOV, Lens
distortion factors, etc, etc.) And you'll want to account for the scenario
where there are more than one device connected to the browser.

Also, if this is going to be a high quality experience you'll want to be
able to target rendering to the HMD directly and not rely on OS mirroring
to render the image. This is a can of worms in and of itself: How do you
reference the display? Can you manipulate a DOM tree on it, or is it
limited to WebGL/Canvas2D? If you can render HTML there how do the
appropriate distortions get applied, and how do things like depth get
communicated? Does this new rendering surface share the same Javascript
scope as the page that launched it? If the HMD refreshes at 90hz and your
monitor refreshes at 60hz, when does requestAnimationFrame fire? These are
not simple questions, and need to be considered carefully to make sure that
any resulting API is useful.

Finally, it's worth considering that for a VR experience to be effective it
needs to be pretty low latency. Put bluntly: Browser suck at this.
Optimizing for scrolling large pages of flat content, text, and images is
very different from optimizing for realtime, super low latency I/O. If you
were to take an Oculus Rift and plug it into one of the existing
browser/Rift demos https://github.com/Instrument/oculus-bridge with
Chrome, you'll probably find that in the best case the rendering lags
behind your head movement by about 4 frames. Even if your code is rendering
at a consistent 60hz that means you're seeing ~67ms of lag, which will
result in a motion-sickness-inducing swimming effect where the world is
constantly catching up to your head position. And that's not even taking
into account the question of how well Javascript/WebGL can keep up with
rendering two high resolution views of a moderately complex scene,
something that even modern gaming PCs can struggle with.

That's an awful lot of work for technology that, right now, does not have a
large user base and for which the standards and conventions are still being
defined. I think that you'll have a hard time drumming up support for such
an API until the technology becomes a little more widespread.

(Disclaimer: I'm very enthusiastic about current VR research. If I sound
negative it's because I'm being practical, not because I don't want to see
this happen)

--Brandon


On Wed, Mar 26, 2014 at 12:34 AM, Brandon Andrews 
warcraftthre...@sbcglobal.net wrote:

 I searched, but I can't find anything relevant in the archives. Since
 pointer lock is now well supported, I think it's time to begin thinking
 about virtual reality APIs. Since this is a complex topic I think any spec
 should start simple. With that I'm proposing we have a discussion on adding
 a head tracking. This should be very generic with just position and
 orientation information. So no matter if the data is coming from a webcam,
 a VR headset, or a pair of glasses with eye tracking in the future the
 interface would be the same. This event would be similar to mouse move with
 a high sample rate (which is why in the event the head tracking and eye
 tracking are in the same event representing a user's total view).

 interface ViewEvent : UIEvent {
 readonly attribute float roll; // radians, positive is slanting the
 head to the right
 readonly attribute float pitch; // radians, positive is looking up
 readonly attribute float yaw; // radians, positive is looking to the
 right
 readonly attribute float offsetX; // offset X from the calibrated
 center 0 in the 

Re: [gamepad] preventing default/capturing controller

[Brandon Jones]

Sorry, meant to respond to this earlier but wanted to make sure the
relevant details were already public.

Valve's upcoming Steam
controllerhttp://store.steampowered.com/livingroom/SteamController/provides
a good use case for an API like this. The controller's touchpads
can be used to emulate mouse and keyboard input, making it possible to
navigate the browser with it. This could be a really nice way to use a
browser in a living room setting or similar. The controller can also be
switched to a gamepad emulation mode, however, where the touchpads emulate
standard joysticks. These modes can be switched via software by sending the
appropriate USB reports (Or using the right Steamworks API, but that'
probably a non-starter for inclusion in a browser). The timing of that
switch, however, would be difficult to predict. Having an explicit gamepad
lock like Patrick described would help this scenario quite a bit.

--Brandon


On Tue, Mar 18, 2014 at 3:39 AM, Matt Gaunt mattga...@chromium.org wrote:

 Adding a best approach to the spec is a good idea.

 Originally I thought that having a gamepadconnected callback (or some use
 of the Gamepad API) would be enough of an indicator to prevent the UA from
 manipulating the key presses / cursor movement.

 The main scenarios I'm struggling to see as a good fit for this:

 - The developer has a settings screen and would rather use the UA's
 manipulated keys within an event listener, rather than handle gamepad state
 in an animation frame. A separate API would probably be cleaner for
 developers to follow and presumably easier to implement in the UA side.
 - [Much more niche] There are native components around the edge of the UA
 which could be interacted with, Firefox on the Ouya is a great example.
 Should this be something which is catered for? Ideally the game would want
 to be in fullscreen mode and the user would have to leave fullscreen to
 interact with the native elements, this way the UA has a clear point to
 break out of the gamepad mode.

 I'll try and get hold of Scott Graham and see if he has any thoughts /
 comments.

Re: [gamepad] Haptic Feedback/Controller Vibration

[Brandon Jones]

Mozilla has played around with gamepad vibration a bit already:
https://bugzilla.mozilla.org/show_bug.cgi?id=680289
There's been some mixed messages, but so far the basic idea seems to be to
either extend the existing Vibration
APIhttp://www.w3.org/TR/2014/WD-vibration-20140211/to handle
multiple motors (and presumably associate it back to a specific
pad) or to add a very similar looking vibrate function to the gamepad
API. Something like:

vibrate(int motorIndex, float intensity);

Personally I think that there's got to be a little more to it than that.
For one, you'll want to be able to query not only how many motors a device
has but have a way of expressing what type of feedback that motor provides
(is it a heavy motor, etc.) and possibly some description of it's locality
(The Xbox One controller has motors in each trigger, and the Steam
controller has motors under each touchpad.) I have no idea how to concisely
and robustly communicate that information. This also assumes that the
device has simple X-Input style feedback, not the more complicated joystick
force feedback that was all the rage in the 90s. (Which is far more
complicated and fairly dead, so I doubt it's worth addressing at all.)

As for controller speakers, I see the argument that it's a type of
controller feedback, but I don't think it should be conflated with
controller vibration. I see them as two very distinct features. It *is* an
interesting feature to consider on it's own, though.



On Sun, Mar 23, 2014 at 11:19 AM, Patrick Martin patrick.marti...@gmail.com
 wrote:

 Hello,

 Is there a reason that haptic feedback support is missing from the gamepad
 api? I feel that this would prevent developers from fully utilizing modern
 gamepads which nearly always include some form of a haptic feedback
 mechanism, (from the speaker on the wiimote to the dual vibrators in an
 xbox 360 controller). In order for this to work however, we need a
 mechanism for detecting supported haptic feedback modes which could then
 be used on the gamepad with the WebAudio API.

 Here is an example of one such mechanism:

 var ctx = new AudioContext();
 var rumble_audio = document.getElementById(rumbleAudio);
 var rumble_audio_source = ctx.createMediaElementSource(rumble_audio)
 var modes = gamepad.getSupportedFeedbackModes();
 var feedback_node = null;
 if (typeof(modes.RUMBLE_STEREO) !== undefined){ // like the xbox 360
 feedback_node = gamepad.getFeedbackNode(ctx,modes.RUMBLE_STEREO);
 }else if(typeof(modes.RUMBLE_MONO) !==undefined){ // like some pc
 gamepads
 feedback_node = gamepad.getFeedbackNode(ctx,modes.RUMBLE_MONO);
 }else if(typeof(modes.GAMEPAD_AUDIO)){ // like the wii remote
 feedback_node = gamepad.getFeedbackNode(ctx,modes.GAMEPAD_AUDIO);
 }else{
 feedback_node = ctx.destination
 }
 rumble_audio_source.connect(feedback_node);
 rumble_audio.play();

 A system like this would allow both greater access to the controller's
 functionality and allow feedback to be dynamically generated or modified by
 the WebAudio API.


 I hope this suggestion helps,
 Patrick Martin

Re: [fullscreen] Problems with mouse-edge scrolling and games

[Brandon Jones]

Several points to add:

- it's possible to theme the OS cursor using custom images with CSS.
https://developer.mozilla.org/en-US/docs/Web/CSS/cursor/url
- The reason the cursor is hidden when the pointer is locked is that some
OSes don't have the ability to report relative mouse movement correctly at
screen edges. This requires the cursors to constantly be reset to the
center of the screen, which obviously would look strange if the cursor was
visible.
- You already mentioned some issues with synthetic mouse events, but
unfortunately it's worse than you suspect. For example: sending a synthetic
click event to a checkbox doesn't actually change it's checked state. (Not
the last time I tried anyway) select controls also have a hard time with
synthetic events, and there's a whole host of other sub rely broken things.
:(

Thanks for the tests!

On Sunday, February 23, 2014, Florian Bösch pya...@gmail.com wrote:

 On Sun, Feb 23, 2014 at 9:55 AM, Florian Bösch 
 pya...@gmail.comjavascript:_e(%7B%7D,'cvml','pya...@gmail.com');
  wrote:


- Inability to use DOM elements with mouse events for a game
overlay/HUD.

 The test I've written here
 http://codeflow.org/issues/software-cursor.html also tests mouse event
 synthesis (as hinted at by an example in the pointerlock spec) and it works
 satisfactory in chrome and firefox on linux. To get all subtleties right
 one would also have to do most other mouse events I guess (mouseover,
 mouseout, mousemove etc.).


 I did some more testing on this. Synthetic mouse events are limited in
 that they cannot trigger some state like hover styles. Other things like
 focus/blur require special handling. It's also left to the synthetic event
 creator to synthesize compound events like over/out/enter/leave and their
 semantic as supplying the primitive mouse events (click, mousemove) doesn't
 automatically synthesize the others.

Re: [fullscreen] Problems with mouse-edge scrolling and games

[Brandon Jones]

Pointer lock does indeed provide a workaround for this, but it's worth
noting that with the current architecture of Chrome a software cursor will
experience visible lag, which these types of games are highly sensitive to.
I'm not sure about the latency of Firefox or others, but in general a
hardware cursor is preferable any time you have a visible cursor. As a
result this is still a use case worth considering.

--Brandon

On Saturday, February 22, 2014, Florian Bösch pya...@gmail.com wrote:

 Pointerlock should solve these problems in the following fashion:

- When the user clicks into the app, request pointerlock
- Use it to give him a cursor drawn by you
- That way you can keep the interaction inside your game and
accurately detect borders etc.

 I run http://webglstats.com/ and It also records the availability of
 pointerlock. This is relative to everybody who has webgl and it currently
 stands at 94.1%.

 Caveat, I think Firefoxes implementaiton of the Pointerlock API might not
 follow the specification yet to make it possible to have pointerlock
 without fullscreen. I know that it's possible to get pointerlock in google
 chrome without fullscreen.

 Links:

- http://www.html5rocks.com/en/tutorials/pointerlock/intro/
- http://www.w3.org/TR/pointerlock/




 On Sat, Feb 22, 2014 at 2:00 AM, Ian Langworth 
 i...@artillery.comjavascript:_e(%7B%7D,'cvml','i...@artillery.com');
  wrote:

 Hi everyone,

 We're building a console- and native-quality game in the browser using
 JavaScript and WebGL. You can see a very early version of the game in this
 video: https://youtu.be/NiCy5igO9-I . It's a realtime strategy (RTS)
 game, like StarCraft[1] or Command  Conquer[2], and moving the cursor to
 the edge of the screen is the primary way that users move around the map.

 We have a single requirement: Moving the cursor to, and possibly past,
 the edge of the window or screen should pan the camera in that direction.

 In windowed mode there are a couple of problems: Players might click
 outside the window accidentally, pixels outside the window are wasted
 screen real estate, mousemove events aren't fired when the cursor is
 outside the window, and sometimes the cursor moves so quickly that it
 misses the area of the window where we do edge detection.

 The better option is to go fullscreen with the Fullscreen API, but this
 has problems as well. There are certain behaviors that occur when moving
 the cursor to the top and bottom of the screen, which I've illustrated in
 the following videos using a quick edge-scrolling demo[3].. (Please ignore
 the problems with cursor detection in the Windows videos. It only occurred
 while the screen recording software was active.):

Chrome, Windows 7
https://www.dropbox.com/s/t9kyo8s5am76ezg/chrome%20win7%20edges.avi
The fullscreen notice appears every time the cursor is moved to the
 top.

Chrome, Mac OS X
https://www.dropbox.com/s/ke1mr5te2dwgvor/chrome%20osx%20edges.mov
A poor experience with the menu and address bars appearing at the top
 and Dock appearing at the bottom.

Firefox, Windows 7

 https://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges.avihttps://www.dropbox.com/s/iku8croaphsgcwd/firefox%20win7%20edges..avi
No problems with the fullscreen experience!

Firefox, Mac OS X
https://www.dropbox.com/s/0bkdx71ir0yhw88/firefox%20osx%20edges.mov
Menu bar appears at top, along with some mysterious window chrome, and
 the Dock appears at the bottom as well.

 These behaviors are to remind the user about fullscreen and lessen the
 chance of phishing. But, as you might imagine, they are troublesome when
 you're in the heat of battle and are trying to crush your opponent. I would
 like to solicit the list for opinions on how we can improve the fullscreen
 UI for games and give players the best experience possible. Some options
 that come to mind:

   (A) Provide an alternate fullscreen API or option which provides a more
 complete experience, but with more dire warnings to the user. For example,
 pass in a flag to requestFullscreen() which gives the page full control
 over the screen (no fullscreen reminder notifications, menu bars, or Docks)
 and even the keyboard (so games can use the Escape key), but entering this
 mode presents a much more intense warning to the user like the invalid
 certificate warnings.

   (B) Maintain the current API, but put an option in the preferences or
 flags which makes the browser's fullscreen mode more complete and without
 interruption. Hardcore gamers will likely accept this extra step if it
 means getting an optimal experience.

   (C) Propose a new Mouse-Edge Detection API, which might solve these
 problems and provide games with better cursor detection overall.

 I appreciate any and all feedback.

 [1] http://youtu.be/Qb0VzbxdP4U?t=10m14s

 [2] http://youtu.be/l41hG-fVDN4?t=3m34s

 [3] http://jsfiddle.net/statico/F8sjw/show/

[gamepad] Reasoning behind GamepadButton struct

[Brandon Jones]

I've been working with the Gamepad API implementation in Chrome recently,
trying to bring it up-to-date with the latest editors draft of the spec (
https://dvcs.w3.org/hg/gamepad/raw-file/default/gamepad.html). While
generally I feel like it's a good update to the working draft I'm puzzled
by the decision to change the gamepad buttons from an array of floats to an
array of GamepadButton structures.

Specifically, it appears to add complexity to the interface and
implementation without providing new information to developers. I can see
that if a piece of hardware provided custom deadzones on axes which would
eventually be interpreted as buttons that it may be convenient to use that
to determine the pressed state, but no hardware I know of actually does
so. (XInput suggests a trigger deadzone of 30/255, but that's an arbitrary
guideline and probably works well for most similar hardware.) If there is
concern about input noise providing false positives presses for these
types of buttons it would seem pretty trivial for the browser
implementation to apply the deadzone internally, and only provide non-zero
values above the desired threshold.

I recognize that Mozilla has already implemented the API as described in
the editors draft, and I don't want to place additional burden on them to
change it unnecessarily. I'm happy to update Chrome's implementation to
match, but I want to ensure this is the right direction before I do. It's
also worth noting that many of the existing web apps that use the API
(though they are admittedly few) expect the buttons to be exposed as
floats, and as such updating Chrome's implementation will break these use
cases. (New fields like mapping and connected, on the other hand, can
be exposed without requiring changes from the app).

I apologize if this has been addressed before, as I've only recently
started working on the implementation and am not familiar with the full
spec history. I haven't seen this particular point discussed in the
archives, however, so I figured it was worth bringing up.

--Brandon Jones