Re: [gamepad] New feature proposals: pose, touchpad, vibration

[Kostiainen, Anssi]

Hi Brandon,

> On 22 Apr 2016, at 08:14, Brandon Jones  wrote:

[...]

> Finally, I'd like to propose that we add gamepad vibration controls. This has 
> been talked about in the past as well, and the thinking has typically been 
> that the Gamepad API may want to try and use the Vibration API somehow 
> instead of exposing a new interface. However that API doesn't currently have 
> a way to target anything other than the device's default motor (usually for a 
> phone), and I don't see there being much interest in extending it to do so. 
> Still, I don't see much sense in re-inventing this wheel so I suggest 
> exposing the same interface on the Gamepad object.
> 
> partial interface Gampad {
> boolean vibrate (VibratePattern pattern);
> };

If you have suggestions for improvements you'd like to see in the Vibration 
API, please feel free to open issues in GitHub:

https://github.com/w3c/vibration

Some specs already reuse parts of the Vibration API. For example, the 
Notifications API reuses and references the vibration pattern, and its 
'validate and normalize' and 'perform vibration' algorithms. We actually added 
some hooks to the Vibration API spec to make that integration possible.

> This is pretty basic as far as gamepad haptics go, but it's also provides a 
> lowest common denominator interface that should be supportable by anything 
> with a motor in it. It gives up some granularity (For example, it doesn't 
> allow developers to trigger the two motors in an Xbox controller 
> individually), but it provides a clean, universal interface that's easy to 
> use and mimics existing features on the web platform.

At the time the spec was crafted, we evaluated devices that were widely 
available and took an intersection of the feature set. That said, we should 
re-evaluate the feature set from time to time and make sure it matches the 
capabilities of the today's devices.

Thanks,

-Anssi (the Vibration API editor)

Re: [gamepad] New feature proposals: pose, touchpad, vibration

[Brandon Jones]

Regarding the vibration portion of the API, I'm still not sure how much
control we should provide. Don't want to get too closely tied to any
specific native implementation. To that end I've put together a brief
document comparing the APIs that I'm aware of:
https://docs.google.com/document/d/1Ms4RFTI8bz1dTtympPmkb81FLhPdls7n-mfwQoAeyAI/edit?usp=sharing

Comments are welcome, especially if there's a Gamepad API that I've missed.

On Mon, Apr 25, 2016 at 9:03 AM Brandon Jones  wrote:

> Lots of good feedback! Replies inline.
>
> > Do we want to know the value of the press? Does the API for the Steam
> Controller, for example, return the force of the press as a float?
> > Likewise, Pointer Events have "pressure"
>
> GamepadButton already has a "value" property, which I think would be
> appropriate to use here. Today it's primarily used for things like analog
> triggers or pressure sensitive buttons, but I can't see any reason why it
> wouldn't work for touchpads as well. Put another way: while plenty of
> touchpads have both a touched and a clicked state, I can't think of a
> scenario where a touchpad would have a separate concept of "touch force"
> and "pressed amount".
>
> > Just as a quick API surface comment, Float32Array? is not a very good
> way of representing a vector (or quaternion).
> > Web VR seems to have changed from DOMPointReadOnly to Float32Array? for
> some reason. Maybe they can explain?
>
> I should probably explain that I'm an author of the WebVR spec who also
> happens to be the current primary Gamepad developer on Chrome. :) Of
> course, that makes my opinions on cross pollination between the two APIs
> very non-neutral, which is why I think it's important to gather feedback
> like this. Thanks!
>
> The WebVR API switched from DOMPoint to Float32Arrays for a couple of
> reasons, probably the biggest being that the DOMGeometry types still seem
> to be in spec limbo. This is especially true of the DOMMatrix type, which
> appears to be fairly contentious from reading through the mailing lists. We
> were reluctant to continue relying on a spec with an unclear future or
> timeline.
>
> A common concern raised when talking about these types is that they may
> not be as performant as developers are expecting them to given their
> "native" implementations, since the overhead of transitioning in and out of
> the Javascript VM may swamp out the perf benefits you would otherwise see.
> Counterintuitively, in order to get the best performance it may be best to
> do all of your vector and matrix math in Javascript.
>
> To that end, libraries already exist that operate on Float32Arrays
> directly for vector/matrix math (http://glmatrix.net). And SIMD JS, which
> can improve the performance of these operations up to 4X in some cases,
> expects values to be packed into TypedArrays anyway. It was those potential
> benefits and the desire to not create another explicit vector and matrix
> type to compete against DOMGeometry that lead us to use flat arrays in
> WebVR.
>
> > Why are you interested in creating a competing extension to the Gamepad
> API, instead of working with the Web VR folks on their interface?
>
> I've already explained my relationship to WebVR, so I'll point out that I
> would view this as a complimentary extension, not a competing one. In a
> prior iteration of the WebVR spec things were abstracted out in such a way
> that you could have exposed 6DoF information about gamepads via WebVR
> instead of the Gamepad API. This proved to make using the API very
> confusing for developers. Not only would it have been easy to mistake a
> tracked gamepad for a headset, it required some awkward references between
> APIs to say "this set of pose information is associated with this gamepad
> index at this timestamp". And in some cases (Wii remote) it wouldn't be
> obvious why you had to route through WebVR at all in order to get what
> should be intrinsic properties of the gamepad itself.
>
> I wouldn't be opposed at all to having Pose be a shared interface between
> APIs, used by both WebVR and the Gamepad API. I do feel like it's quite a
> bit more natural for developers to have each context surface their own pose
> values, though.
>
> I know I haven't fully covered everyone's comments so far, but I wanted to
> clarify my relationship to the WebVR API and explain some of the logic
> behind it's choices. I have some thoughts on the vibration concerns as
> well, but I want to gather a bit more reference material before I dive into
> that. Thanks for everyone's input thus far!
>
> --Brandon
>

Re: [gamepad] New feature proposals: pose, touchpad, vibration

[Brandon Jones]

Lots of good feedback! Replies inline.

> Do we want to know the value of the press? Does the API for the Steam
Controller, for example, return the force of the press as a float?
> Likewise, Pointer Events have "pressure"

GamepadButton already has a "value" property, which I think would be
appropriate to use here. Today it's primarily used for things like analog
triggers or pressure sensitive buttons, but I can't see any reason why it
wouldn't work for touchpads as well. Put another way: while plenty of
touchpads have both a touched and a clicked state, I can't think of a
scenario where a touchpad would have a separate concept of "touch force"
and "pressed amount".

> Just as a quick API surface comment, Float32Array? is not a very good way
of representing a vector (or quaternion).
> Web VR seems to have changed from DOMPointReadOnly to Float32Array? for
some reason. Maybe they can explain?

I should probably explain that I'm an author of the WebVR spec who also
happens to be the current primary Gamepad developer on Chrome. :) Of
course, that makes my opinions on cross pollination between the two APIs
very non-neutral, which is why I think it's important to gather feedback
like this. Thanks!

The WebVR API switched from DOMPoint to Float32Arrays for a couple of
reasons, probably the biggest being that the DOMGeometry types still seem
to be in spec limbo. This is especially true of the DOMMatrix type, which
appears to be fairly contentious from reading through the mailing lists. We
were reluctant to continue relying on a spec with an unclear future or
timeline.

A common concern raised when talking about these types is that they may not
be as performant as developers are expecting them to given their "native"
implementations, since the overhead of transitioning in and out of the
Javascript VM may swamp out the perf benefits you would otherwise see.
Counterintuitively, in order to get the best performance it may be best to
do all of your vector and matrix math in Javascript.

To that end, libraries already exist that operate on Float32Arrays directly
for vector/matrix math (http://glmatrix.net). And SIMD JS, which can
improve the performance of these operations up to 4X in some cases, expects
values to be packed into TypedArrays anyway. It was those potential
benefits and the desire to not create another explicit vector and matrix
type to compete against DOMGeometry that lead us to use flat arrays in
WebVR.

> Why are you interested in creating a competing extension to the Gamepad
API, instead of working with the Web VR folks on their interface?

I've already explained my relationship to WebVR, so I'll point out that I
would view this as a complimentary extension, not a competing one. In a
prior iteration of the WebVR spec things were abstracted out in such a way
that you could have exposed 6DoF information about gamepads via WebVR
instead of the Gamepad API. This proved to make using the API very
confusing for developers. Not only would it have been easy to mistake a
tracked gamepad for a headset, it required some awkward references between
APIs to say "this set of pose information is associated with this gamepad
index at this timestamp". And in some cases (Wii remote) it wouldn't be
obvious why you had to route through WebVR at all in order to get what
should be intrinsic properties of the gamepad itself.

I wouldn't be opposed at all to having Pose be a shared interface between
APIs, used by both WebVR and the Gamepad API. I do feel like it's quite a
bit more natural for developers to have each context surface their own pose
values, though.

I know I haven't fully covered everyone's comments so far, but I wanted to
clarify my relationship to the WebVR API and explain some of the logic
behind it's choices. I have some thoughts on the vibration concerns as
well, but I want to gather a bit more reference material before I dive into
that. Thanks for everyone's input thus far!

--Brandon

RE: [gamepad] New feature proposals: pose, touchpad, vibration

[Domenic Denicola]

From: Brandon Jones [mailto:bajo...@google.com]

>   readonly attribute Float32Array? position;

Just as a quick API surface comment, Float32Array? is not a very good way of 
representing a vector (or quaternion). You want either separate x/y/z/w 
properties, or to use a DOMPoint(ReadOnly).

Web VR seems to have changed from DOMPointReadOnly to Float32Array? for some 
reason. Maybe they can explain? It seems like a big downgrade, especially since 
Flaot32Arrays are mutable.

> First, I'm proposing that we add an optional "pose" object to the the Gamepad 
> interface that would expose all the information necessary to track a 
> controller with 3 Degree of Freedom or 6 Degree of Freedom tracking 
> capabilities. The primary motivator for supporting this information is to 
> allow devices like HTC Vive controllers or Oculus Touch to be exposed, but 
> the same structure could also expose motion and orientation data for devices 
> such as Wii, PS3, and PS4 controllers, as well as anything else that had a 
> gyroscope and/or accelerometer.
>
> ...
>
> The proposed pose interface largely mirrors the 
> http://mozvr.github.io/webvr-spec/#interface-vrpose interface from the WebVR 
> spec, but drops the timestamp since that's already provided on the Gamepad 
> interface itself. The VRPose interface could conceivably be used directly, 
> but I'm assuming that we'd rather not have a dependency on a separate 
> work-in-progress spec.

It seems to me like the Web VR spec is very specifically designed to support 
these use cases. Why are you interested in creating a competing extension to 
the Gamepad API, instead of working with the Web VR folks on their interface? 
We certainly shouldn't have both on the platform, so you need to either talk to 
them about moving portions of their spec into the Gamepad API, or go help them 
work on their spec.

In practice the only non-political difference will be whether it's on 
navigator.getGamepads() or navigator.getVRDisplays(). I don't have enough 
game/VR-developer experience to tell which of these would be more intuitive for 
developers.

Maybe you are already collaborating, as seen by 
https://github.com/MozVR/webvr-spec/issues/31. It's not really clear.

Re: [gamepad] New feature proposals: pose, touchpad, vibration

[Florian Bösch]

On Mon, Apr 25, 2016 at 4:31 AM, Chris Van Wiemeersch 
wrote:
>
> If you take a look at all the content libraries out there for the Gamepad
> API, there's a ridiculous amount of logic and special casing web developers
> are having to do just between the Firefox and Chrome implementations - and
> between Windows and Mac.
>
The existing specification is not adhered to, or contains no fixed language
how things are done (where browsers do differ), maybe it should be added?

Re: [gamepad] New feature proposals: pose, touchpad, vibration

[Patrick H. Lauke]

On 25/04/2016 03:31, Chris Van Wiemeersch wrote:

Do we want to know the value of the press? Does the API for the Steam
Controller, for example, return the force of the press as a float? The
reason I ask is because with Touch events, there's a `force` property on
the `Touch` interface:
https://developer.mozilla.org/en-US/docs/Web/API/Touch/force


Likewise, Pointer Events have "pressure" 
https://w3c.github.io/pointerevents/#widl-PointerEventInit-pressure


P


We should probably introduce something similar to the Gamepad API.

The `Gamepad.vibrate` proposal generally looks good, but I have a few
concerns. `navigator.vibrate(pattern)` works well for pulse vibrations
of varying durations and pause intervals, but the API doesn't allow the
following:

1. vibrating at a higher/lower _strength_ (i.e., every vibration is
emitted at the same strength, but you might want to do neat effects such
as fading in and out)
2. selecting which vibration motors (haptic axes) to vibrate (e.g., with
controllers that allow haptic feedback in different regions, such as
left/right/top/bottom; a perfect example is the PlayStation DualShock
controllers)

 > This is pretty basic as far as gamepad haptics go, but it's also
provides a lowest common denominator interface that should be
supportable by anything with a motor in it. It gives up some granularity
(For example, it doesn't allow developers to trigger the two motors in
an Xbox controller individually), but it provides a clean, universal
interface that's easy to use and mimics existing features on the web
platform.

Brandon, you addressed my #2 above, but my concern is once the API is
changed, it's going to be hard to change it again. If you take a look at
all the content libraries out there for the Gamepad API, there's a
ridiculous amount of logic and special casing web developers are having
to do just between the Firefox and Chrome implementations - and between
Windows and Mac.

We already have `GamepadButton`, but perhaps we should have a
`GamepadAxis` and _there_ is where we should add a `vibrate` property
for each haptic axis. There are probably other ways of handling this,
but just a thought.

Those are my initial thoughts. If I think of anything else, I'll let
y'all know.


On Thu, Apr 21, 2016 at 10:14 PM, Brandon Jones > wrote:

I'd like to propose the addition of several new features to the
Gamepad API, primarily born from needs that have been identified
while developing the WebVR spec
, but which also cover topics
that have been under discussion in the past but deemed lower
priority for one reason or another.

First, I'm proposing that we add an optional "pose" object to the
the Gamepad interface that would expose all the information
necessary to track a controller with 3 Degree of Freedom or 6 Degree
of Freedom tracking capabilities. The primary motivator for
supporting this information is to allow devices like HTC Vive
controllers or Oculus Touch to be exposed, but the same structure
could also expose motion and orientation data for devices such as
Wii, PS3, and PS4 controllers, as well as anything else that had a
gyroscope and/or accelerometer.

interface *GampadPose* {
   readonly attribute Float32Array? position;
   readonly attribute Float32Array? linearVelocity;
   readonly attribute Float32Array? linearAcceleration;

   readonly attribute Float32Array? orientation;
   readonly attribute Float32Array? angularVelocity;
   readonly attribute Float32Array? angularAcceleration;
};

partial interface *GampadPose* {
   readonly attribute GamepadPose? pose;
};

Position is given in meters from a device-specific origin. Velocity
and acceleration are given it meters per second, and orientation is
a quaternion. If provided all arrays should have 3 elements to
except the orientation, which has 4.

The proposed pose interface largely mirrors the VRPose
 interface from
the WebVR spec, but drops the timestamp since that's already
provided on the Gamepad interface itself. The VRPose interface could
conceivably be used directly, but I'm assuming that we'd rather not
have a dependency on a separate work-in-progress spec.

Second, I'd like to see the API more explicitly support devices that
have touchpads. It's my feeling that the existing axis array is
sufficient for exposing the touch position, but we need a way to
indicate if the touchpad is currently being touched to disambiguate
a value of zero from no input. This can be handled currently by
assigning a button to report pressed = true when the touchpad is
touched, but since many touchpads also can be clicked as a button
you end up with two buttons associated with a single input. I'd like
to proposed that we 

Re: [gamepad] New feature proposals: pose, touchpad, vibration

[Chris Van Wiemeersch]

Do we want to know the value of the press? Does the API for the Steam
Controller, for example, return the force of the press as a float? The
reason I ask is because with Touch events, there's a `force` property on
the `Touch` interface:
https://developer.mozilla.org/en-US/docs/Web/API/Touch/force

We should probably introduce something similar to the Gamepad API.

The `Gamepad.vibrate` proposal generally looks good, but I have a few
concerns. `navigator.vibrate(pattern)` works well for pulse vibrations of
varying durations and pause intervals, but the API doesn't allow the
following:

1. vibrating at a higher/lower _strength_ (i.e., every vibration is emitted
at the same strength, but you might want to do neat effects such as fading
in and out)
2. selecting which vibration motors (haptic axes) to vibrate (e.g., with
controllers that allow haptic feedback in different regions, such as
left/right/top/bottom; a perfect example is the PlayStation DualShock
controllers)

> This is pretty basic as far as gamepad haptics go, but it's also provides
a lowest common denominator interface that should be supportable by
anything with a motor in it. It gives up some granularity (For example, it
doesn't allow developers to trigger the two motors in an Xbox controller
individually), but it provides a clean, universal interface that's easy to
use and mimics existing features on the web platform.

Brandon, you addressed my #2 above, but my concern is once the API is
changed, it's going to be hard to change it again. If you take a look at
all the content libraries out there for the Gamepad API, there's a
ridiculous amount of logic and special casing web developers are having to
do just between the Firefox and Chrome implementations - and between
Windows and Mac.

We already have `GamepadButton`, but perhaps we should have a `GamepadAxis`
and _there_ is where we should add a `vibrate` property for each haptic
axis. There are probably other ways of handling this, but just a thought.

Those are my initial thoughts. If I think of anything else, I'll let y'all
know.


On Thu, Apr 21, 2016 at 10:14 PM, Brandon Jones  wrote:

> I'd like to propose the addition of several new features to the Gamepad
> API, primarily born from needs that have been identified while developing
> the WebVR spec , but which also cover
> topics that have been under discussion in the past but deemed lower
> priority for one reason or another.
>
> First, I'm proposing that we add an optional "pose" object to the the
> Gamepad interface that would expose all the information necessary to track
> a controller with 3 Degree of Freedom or 6 Degree of Freedom tracking
> capabilities. The primary motivator for supporting this information is to
> allow devices like HTC Vive controllers or Oculus Touch to be exposed, but
> the same structure could also expose motion and orientation data for
> devices such as Wii, PS3, and PS4 controllers, as well as anything else
> that had a gyroscope and/or accelerometer.
>
> interface *GampadPose* {
>   readonly attribute Float32Array? position;
>   readonly attribute Float32Array? linearVelocity;
>   readonly attribute Float32Array? linearAcceleration;
>
>   readonly attribute Float32Array? orientation;
>   readonly attribute Float32Array? angularVelocity;
>   readonly attribute Float32Array? angularAcceleration;
> };
>
> partial interface *GampadPose* {
>   readonly attribute GamepadPose? pose;
> };
>
> Position is given in meters from a device-specific origin. Velocity and
> acceleration are given it meters per second, and orientation is a
> quaternion. If provided all arrays should have 3 elements to except the
> orientation, which has 4.
>
> The proposed pose interface largely mirrors the VRPose
>  interface from the
> WebVR spec, but drops the timestamp since that's already provided on the
> Gamepad interface itself. The VRPose interface could conceivably be used
> directly, but I'm assuming that we'd rather not have a dependency on a
> separate work-in-progress spec.
>
> Second, I'd like to see the API more explicitly support devices that have
> touchpads. It's my feeling that the existing axis array is sufficient for
> exposing the touch position, but we need a way to indicate if the touchpad
> is currently being touched to disambiguate a value of zero from no input.
> This can be handled currently by assigning a button to report pressed =
> true when the touchpad is touched, but since many touchpads also can be
> clicked as a button you end up with two buttons associated with a single
> input. I'd like to proposed that we extend GamepadButton with a "touched"
> boolean to make this case explicit.
>
> interface *GampadButton* {
>   readonly attribute boolean pressed;
>   readonly attribute double value;
>   readonly attribute boolean touched;
> };
>
> For non-touch controls it seems sensible that this would always