Re: [gamepad] Allow standard-mapped devices to have extra buttons or axes & allow multiple mappings per device

[Brandon Jones]

Whew... trying to process all of Katelyn's email. Forgive me if I overlook
some things, but I'm going to try and hit a few points that have been on my
mind:

On Wed Oct 15 2014 at 12:43:14 AM Katelyn Gadd  wrote:
>
> If there is one standard mapping, and it covers most users, this means
> the natural path for developers is to only support the standard
> mapping. They won't add keybinding/button remapping, they won't add
> support for multiple layouts, they might not even add support for
> choosing which of your gamepads you want to use. This is not an
> unrealistic fear: It is the status quo in virtually all console games
> and in many PC games. In some cases you can rebind by editing .ini
> files, but that is out of the question for web games since their
> configuration typically lies in not-user-editable JS on the server or
> in blobs stored somewhere (localStorage, indexed db, cookies) that are
> not easily user-editable. This 'developers only support the easiest
> thing' future is relatively likely and is actually worse than the
> previous status quo of raw-controllers-only.
>

To be brutally honest I wish that even _this_ was the status quo. In my
experience thus far:

 * Very few games on the web bother with the gamepad API, which I think
boils down to a marketing problem. I've actually had game devs say things
like "So, do you think the web will ever have a way to interact with
gamepads/joysticks/wheels/etc?" If nobody knows the API exists it's hard to
complain much about it's deficiencies. Unfortunately I'm not sure how we
solve this, but assuming that we do...

* Many of the games that I DO see utilizing the API seem to take the
approach of "Well, the dev had a KnockoffBox720Pad, so everything is hard
coded to whatever that reported." It would be great if devs even bothered
looking at the mapping field, but I don't see too many cases where they do.
Again, this feels like a marketing problem. :P


> I think the solution to this is simple: Ensure that there is more than
> one standard mapping. Find a way to make sure that developers have to
> do at least a basic amount of effort to handle things correctly. If
> the simple hack isn't sufficient for wider audiences, they will have
> an incentive to do a good job. This doesn't have to be a punitive
> measure - there are diverse controllers out there that people might
> want to use, from standard gamepads to racing wheels to flight sticks.
> Something as simple as supporting the additional inputs of the
> dualshock 3 & 4 is a good rationale here (they both have gyros &
> accelerometers; the 4 has a touchpad and touchpanel button.)
>

I'm not against adding a new mapping type. "flightstick" seems like the
obvious choice there, though the button layouts on those can vary pretty
radically. At the very least you could standardize axes layout, primary
fire, secondary fire, and hat switch layout and allow for whatever button
order seems natural after that. (Slight tangent: Does anyone know if
the HOTAS-style
controllers  report
both the throttle and stick as a single device?)

I do agree that having more than one standard mapping encourages more
robust usage, assuming we overcome the previously mentioned problems of API
visibility.


> we have the *miserable* constraint where you have to push a button to
> make your controller show up. This changes the ordinal identifiers of
> gamepads from session to session, and - even worse - means that a
> controller cannot be activated by analog stick inputs, which is a real
> problem for games that start with a menu (like two of my major ports).
>

I think we could actually use axes as activation inputs with a little more
processing. The big reason for not doing it right now is that on a lot of
devices inputs that report as an axis (triggers on XInput) have a resting
value other than 0. This may also be a problem if a gamepad has been set
down in such a way that a stick is offcenter (though to be fair the same
problem exists for buttons. I have a gamepad with a stuck "X" button that
immediately activates whenever I go to a gamepad-enabled site.)

My proposed solution would be to not report an input's value unless it has
reported 0 or at least below a deadzone (post-user-agent-mapping) at least
once. This would mean that a joystick axes wouldn't report unless they had
been in their resting state at least once since the page started reporting,
and a stuck button would be a dead input instead of a continuously firing
one. The benefit is that we can then treat any non-zero inputs after that
point as intentional, including axes movement. Thus picking up a controller
and pushing the stick up/down to navigate a menu would properly activate
the gamepad API. It also wouldn't require any API change, which is
generally preferable in any case where the API has been out in the wild.

This defies user expectations and is just plain confusing. Polling an
> actual device to get its st

Re: [gamepad] Allow standard-mapped devices to have extra buttons or axes & allow multiple mappings per device

[Florian Bösch]

I've been looking into mapping input devices for a fair while, and I think
it's not a straighforward problem to solve for a UA (or even a OS).

Difficulties include, but are not limited to:

   - Buttons reported by the driver as axes
   - Axes reported by the driver as buttons
   - Buttons which are also axes (on the device)
   - What an app-developer would want to map an input as
   - Combos
   - New devices and browser update cycles (do you really want to
   disadvantage browsers with slower update cycles from using newer devices?)
   - Differences in how different OSes report devices
   - Devices which cannot be fit into standard mapping schemes (basically
   everything but gamepads, and even there it doesn't work)

It is my belief that UAs should leave mapping alone, and if mapping is
appplied, always make it possible for a developer to get the unmapped
inputs, so that the marketplace (of libraries/APIs/Services) can take care
of that problem.

Re: [gamepad] Allow standard-mapped devices to have extra buttons or axes & allow multiple mappings per device

[Katelyn Gadd]

Hi,

I agree with Chad's suggestions: the spec should be expanded to expose
more than the core set of axes & buttons, and richer metadata would be
extremely useful to developers.

I've been using the gamepad API (as shipped in Chrome, at least - I
haven't had time to adapt to Firefox's implementation, because I used
gamepad.js and it's a broken pile...) for a while now, in a few
shipped JSIL-based game ports. My experiences & opinions inform the
following.

I apologize if this is a bit hard to read, by the way - all this
crunch time for the game I'm working on has fried my brain a bit. :-)

To try and express my PoV on this (previously stated to WG
members/participants, but not directly on this list - sorry, too much
on my plate to add another workgroup to my subscriptions):

I think best-effort mapping to 'standard' layouts is a great idea. I
have some objections to how it is done, but it is an improvement over
what we had before. My main concern is that the current design for
remapping sets us up for a descent into a quagmire that can't easily
be fixed. The scenario I anticipate happening:

Gamepads only have one standard mapping, supported by the vast
majority of available controllers (let's just say 360 pads and
dualshock 3/4 controllers, since those put together probably cover at
least 90% of the market). If you expand the remapping to all XInput
devices, that widens it even further - though this is problematic in a
way I will get to later.

If there is one standard mapping, and it covers most users, this means
the natural path for developers is to only support the standard
mapping. They won't add keybinding/button remapping, they won't add
support for multiple layouts, they might not even add support for
choosing which of your gamepads you want to use. This is not an
unrealistic fear: It is the status quo in virtually all console games
and in many PC games. In some cases you can rebind by editing .ini
files, but that is out of the question for web games since their
configuration typically lies in not-user-editable JS on the server or
in blobs stored somewhere (localStorage, indexed db, cookies) that are
not easily user-editable. This 'developers only support the easiest
thing' future is relatively likely and is actually worse than the
previous status quo of raw-controllers-only.

I think the solution to this is simple: Ensure that there is more than
one standard mapping. Find a way to make sure that developers have to
do at least a basic amount of effort to handle things correctly. If
the simple hack isn't sufficient for wider audiences, they will have
an incentive to do a good job. This doesn't have to be a punitive
measure - there are diverse controllers out there that people might
want to use, from standard gamepads to racing wheels to flight sticks.
Something as simple as supporting the additional inputs of the
dualshock 3 & 4 is a good rationale here (they both have gyros &
accelerometers; the 4 has a touchpad and touchpanel button.)

When the common case is slightly more complicated, it forces
developers to at least *understand* the breadth of the problem and
make rational choices. It also encourages the development of
high-quality, userspace JS wrappers/abstraction layers, so fixes &
improvements can happen in those libraries. I would argue that the
success of XInput is not due to its excessive, somewhat-detrimental
'least common denominator' design, but because it aggressively
targeted & fixed the most common problems with DirectInput.

OK, so, there's my rationale for why the current approach to mapping
should change. Now to dive in deeper to some other issues I see with
the gamepad API.

My first objection to the design & implementation is that it is based
on passive enumeration & device acquisition. You have to poll to find
out what gamepads there are and interact with them. This obviously
produces the threat of fingerprinting, and to mitigate fingerprinting
we have the *miserable* constraint where you have to push a button to
make your controller show up. This changes the ordinal identifiers of
gamepads from session to session, and - even worse - means that a
controller cannot be activated by analog stick inputs, which is a real
problem for games that start with a menu (like two of my major ports).
This defies user expectations and is just plain confusing. Polling an
actual device to get its state is perfectly reasonable, but polling
the overall state of your input devices is not so. Device availability
is something that should be event-based (it is at a driver level
anyway) and it should be exposed to content as events.

The fingerprinting problem is addressed with the solution I proposed
here: Content should *request* the input it needs, at which point the
useragent is able to figure out how to provide it. To mitigate
fingerprinting concerns, this can be done based on user consent, and
the input presented to the content can end up fully sanitized, because
the user content has no need to

Re: [gamepad] Allow standard-mapped devices to have extra buttons or axes & allow multiple mappings per device

[Chad Austin]

On Mon, Oct 13, 2014 at 3:55 PM, Brandon Jones  wrote:

> Re: change #1, with standard gamepad mappings today (at least in Chrome)
> we map any buttons that don't correspond to the official standard mapping
> to button[17] and up. This, of course, depends on which buttons are
> actually visible to the browser (many devices expose buttons that can't be
> "seen" by normal applications, like the home button on Xbox controllers). A
> good example is the Touchpad on the PS4 controller. We can't see actual
> touch inputs without writing a custom driver, but we detect clicks on the
> touchpad and report them as button[17]. The same would go for axes, but
> I've yet to encounter a standard-looking gamepad that exposed > four axes
> worth of usable data. Perhaps the spec should be slightly reworded in this
> regard, since it currently states that "The standard gamepad has 4 axes,
> and up to 17 buttons", which could be read as explicitly disallowing
> buttons or axes beyond that.


Great!  Then I agree - the behavior of Chrome then doesn't match the exact
wording of the spec.  The spec should be updated to reflect reality.

I'm not very supportive of suggestion #2 since we don't actually HAVE any
> other mappings defined (joystick/flightstick would be the next most natural
> candidate, I think, but they can vary significantly). If we did, I'm not
> sure how you would reasonably "stack" mappings without breaking an awful
> lot of code. The "mappings" field would have to be either space/comma
> delimited (which breaks things) or turned into an array (which breaks
> things) or you would expose the same device twice under different mappings
> (which is really weird and probably breaks things.) Beyond that, there's no
> way that the layouts of two mapping types could every overlay completely
> cleanly, and we really shouldn't be claiming that something that looks like
> a racing wheel is actually a "standard gamepad" anyway. I just can't see
> how it would work.
>

What's the status of the gamepad API?  It reads like it's in a draft state,
yet it has some implementations in the wild...  Is it de facto frozen?

Assuming it's not de facto frozen -- and temporarily sidestepping the
problem of maintaining compatibility with existing implementations -- the
issue with the mapping API _as written today_ is that there's only one
standard mapping, meaning that if you plug in any non-XInput-style device
that has a d-pad, you'll need the customer to specify their own bindings.
"Your device is nonstandard.  Press left.  Press up.  Press right.  Press
down.  Press OK."

Beyond that, there's no way that the layouts of two mapping types could
> every overlay completely cleanly, and we really shouldn't be claiming that
> something that looks like a racing wheel is actually a "standard gamepad"
> anyway. I just can't see how it would work.
>

The concern that two mappings that each specify a directional pad would not
have overlapping button IDs is valid.

Fundamentally, what I'd like is access to a richer form of per-button and
per-axis metadata than is provided by the API today.  The "standard
mapping" API provided today is rigid and limiting.  However, I don't have
time to come up with a proposal right now - perhaps coming soon.

+Katelyn as she expressed similar concerns.

Thanks,
Chad

Re: [gamepad] Allow standard-mapped devices to have extra buttons or axes & allow multiple mappings per device

[Brandon Jones]

Re: change #1, with standard gamepad mappings today (at least in Chrome) we
map any buttons that don't correspond to the official standard mapping to
button[17] and up. This, of course, depends on which buttons are actually
visible to the browser (many devices expose buttons that can't be "seen" by
normal applications, like the home button on Xbox controllers). A good
example is the Touchpad on the PS4 controller. We can't see actual touch
inputs without writing a custom driver, but we detect clicks on the
touchpad and report them as button[17]. The same would go for axes, but
I've yet to encounter a standard-looking gamepad that exposed > four axes
worth of usable data. Perhaps the spec should be slightly reworded in this
regard, since it currently states that "The standard gamepad has 4 axes,
and up to 17 buttons", which could be read as explicitly disallowing
buttons or axes beyond that.

I'm not very supportive of suggestion #2 since we don't actually HAVE any
other mappings defined (joystick/flightstick would be the next most natural
candidate, I think, but they can vary significantly). If we did, I'm not
sure how you would reasonably "stack" mappings without breaking an awful
lot of code. The "mappings" field would have to be either space/comma
delimited (which breaks things) or turned into an array (which breaks
things) or you would expose the same device twice under different mappings
(which is really weird and probably breaks things.) Beyond that, there's no
way that the layouts of two mapping types could every overlay completely
cleanly, and we really shouldn't be claiming that something that looks like
a racing wheel is actually a "standard gamepad" anyway. I just can't see
how it would work.

--Brandon



On Sun Oct 12 2014 at 11:59:16 PM Chad Austin  wrote:

> Hi all,
>
> I recently was asked to review the Gamepad API draft specification.  My
> background is games though I've done some scientific computing with
> alternate input devices too.
>
> http://chadaustin.me/2014/10/the-gamepad-api/
>
> I'd like to suggest two simple changes to the Gamepad API.  In tandem,
> these APIs would allow applications to have automatic compatibility with a
> broader range of devices.
>
> The changes are:
>
> 1. Allow standard-mapped devices to have extra buttons or axes
> 2. Add support for multiple standard mappings per device.
>
> Together, these changes would allow a simple application that only needs a
> d-pad and a few buttons to work automatically with a Wiimote, XInput
> device, and a racing wheel (many racing wheels have d-pads on them).
>
> Even without the proposal #2, proposal #1 would allow optional access to
> extra capabilities on a device without having to sacrifice the standard
> mapping.
>
> This feels noncontroversial to me because mappings only assign meaning to
> otherwise neutral indices in the buttons[] and axes[] arrays.  For a given
> device, there can be many such meanings.
>
> --
> Chad Austin
> http://chadaustin.me
>

[gamepad] Allow standard-mapped devices to have extra buttons or axes & allow multiple mappings per device

[Chad Austin]

Hi all,

I recently was asked to review the Gamepad API draft specification.  My
background is games though I've done some scientific computing with
alternate input devices too.

http://chadaustin.me/2014/10/the-gamepad-api/

I'd like to suggest two simple changes to the Gamepad API.  In tandem,
these APIs would allow applications to have automatic compatibility with a
broader range of devices.

The changes are:

1. Allow standard-mapped devices to have extra buttons or axes
2. Add support for multiple standard mappings per device.

Together, these changes would allow a simple application that only needs a
d-pad and a few buttons to work automatically with a Wiimote, XInput
device, and a racing wheel (many racing wheels have d-pads on them).

Even without the proposal #2, proposal #1 would allow optional access to
extra capabilities on a device without having to sacrifice the standard
mapping.

This feels noncontroversial to me because mappings only assign meaning to
otherwise neutral indices in the buttons[] and axes[] arrays.  For a given
device, there can be many such meanings.

-- 
Chad Austin
http://chadaustin.me