On 4/14/2014 10:39 AM, Alice Boxhall wrote:
[Re-sending my message from April 6 which was emailed but is not
showing up in the web interface: apologies to anybody who sees this
twice!]
My apologies for taking so long to respond.
Hi Eric,
I'm an engineer on Chrome, and I've been working with the Polymer team
on making sure that we address Custom Element accessibility.
I agree with you that it is critically important that we don't leave
users with disabilities behind in the Web Components world. I am
cautiously optimistic, however.
Firstly, web developers are already a long way down the path of
creating complicated, mouse-driven UIs which lack semantic
information. Web Components gives us an opportunity to formalize
techniques for creating robust custom elements with good accessibility.
I'll probable repeat myself on this point multiple times, and I
apologize in advance but if you talking about elements on the screen,
you're already too late. The information you need to do a good interface
has degraded from the original enough that you can't do the right thing.
We'll get to the definition of "right thing" later.
For example, the ARIA working group are also currently discussing
proposals for allowing ARIA to be extended by developers (see the
discussion beginning with this email:
http://lists.w3.org/Archives/Public/public-pfwg/2014Apr/0034.html) to
better support novel types of UI. One problem with ARIA is that it is
not supported by the dominant voice control software, Dragon, which is
still using very outdated technology - we do have work to do on the
standards side, but we need those standards to be supported by the
relevant tools as well.
The kindest thing I can say about Nuance is that they have their head up
their ass. Those of us dependent on it, only use it as a speech
recognition dictation and command and control engine. We use
natlink+python (aka natpython) to build high-level toolkits which allow
us to build good speech interfaces. I was hoping to have my programming
by speech tool little more advanced but in a nutshell, natpython makes
it possible for me to create a speech programming environment.
There is also work underway to formalize the possible actions and
events which may occur on an element:
http://www.w3.org/TR/indie-ui-events/ - although this is currently in
the early stages and in need of more support from the community.
Secondly, Web Components gives developers a chance to try out novel
ways of supporting accessibility-related use cases, as argued by Steve
Faulkner:
http://blog.paciellogroup.com/2014/04/usability-accessibility-opportunities-web-compenent-world/
- which could, in the future, possibly become part of the standard
HTML spec.
Our recent article,
http://www.polymer-project.org/articles/accessible-web-components.html, outlines
the strategies the Polymer team are using to address accessibility.
Okay, here's my example of how focusing on the graphical elements fails.
bring up LO writer. (Yes, I know this is not a web app but the concepts
are the same)
Dictate some text such as "this is some simple text". The task is put
that text in a box. I'm choosing the simple route of turning on the
borders in a paragraph.
*By hand* *by speech by graphical element*
*by speech with deeper application knowledge
*
Move mouse to beginning of text,
hold down mouse button,
drag to end of text
Right-click text
move down menu to paragraph
left click paragraph
Click tab 'borders'
click second from the left line arrangement
click okay
move mouse the beginning of text
"hold left mouse button"
"drag right"
...wait for it...
"stop"
"right-click"
...counting menu items..
"move down seven"
"mouseclick"
move mouse over Borders tab
"mouseclick"
move mouse over the line arrangement box (second to the left)
"mouseclick"
move mouse to start of text,
"leave Mark",
move mouse to end of text
turn on all borders
This is a reasonable facsimile of what the user experience would be
using mouse, speech as a mouse replacement, and a speech interface.some
of the nuance tools such as mouse grid and mouse movement are broken, as
I just discovered. Like I said, they have their head up their ass.
Now change the task to increasing the size of the box gets even more
interesting because with a speech interface, you could say "expand box
vertically by five" and the underlying code would read the current
height, at five, and replace the value of changing the box height. By
hand, it's a bit more complicated requiring touching multiple tabs and
count user elements.
One important thing to remember, in this concept, I see the ability to
create speech user interfaces or, for that matter, any accessibility
user interface should reside in the hands of the user, not the
application developer. I noticed a harsh comment but I can't think of
any other way to express it quite yet. The GUI interfaces I've seen and
suffered through do not give me any hope that accessibility interface
will be any better. Leave in the hands of the user so they can create
the interface they need.
Here is how we are trying to address your four points using the
technology available today:
> 1) read the state of anything that can be displayed or changed via a
> GUI. this is a getter function
We annotate custom elements with ARIA role, name, state and value
properties. This provides the state information which can be queried
by speech recognition technology, via platform APIs like IAccessible2,
UI Automation, NSAccessible etc., and allows you to query the
interface by name.
Yeah, solving the wrong problem. As I showed above, you don't want touch
the buttons. You want to touch the data or code the buttons refer to.
For example, you don't know want speak incrementing and decrementing of
value for margin spacing. You want to say "margin spacing is 0.57
inches". You don't need to know where it lays out on the screen, you
don't care if the user interface changes because you're touching
underneath the user interface.
> 2) change the state of anything that can be changed by a GUI. This is a
> putter function.
This is where there is currently a gap in specifications, and authors
are forced to implement their own interactions. The IndieUI spec
proposes one possible mechanism for addressing this:
http://www.w3.org/TR/indie-ui-events/#intro-example-valuechangerequest
. To fill this gap for now, we suggest using data binding to translate
user gestures via keyboard or mouse events into changes to attributes
on the custom elements, which are then reflected in the ARIA
attributes accessible via platform APIs.
That's another dead loss. For example, if I want to automate my bank
login sequence, I have to be able to query what's being displayed and
find out what security question is. That text is simply part of a
paragraph. once I can read the text, then I need to generate a different
response. Doing so by injecting keystrokes has serious reliability
problems. I've learned with 20 years experience with speech recognition
that keystroke injection or GUI elements automationnnever fails to
disappoint.
> 3) do something. This is usually the action associated with a link or
> button but can also drive mouse over or any other event causing an
action.
Similarly, acting on an element is currently handled via keyboard and
mouse events, and this could be supported at a higher level by
something like IndieUI actions
(http://www.w3.org/TR/indie-ui-events/#actions). Currently, Polymer
elements listen for mouse and keyboard events which are used to drive
actions on the element. As you say, these events can be simulated by
assistive technology via the platform accessibility APIs. We do
recommend having a robust keyboard story and ensuring that elements
are focusable, to avoid having to perform fiddly mouse interactions.
In the example I gave above, I eliminated many element activations
thereby increasing reliability and decreasing time to execute. I don't
have to go through a series of selections and mouse and keyboard events
simulations to get to where I need to be.
> 4) tell me when something changes. These event notifications allow you
> to use hand/mouse at the same time as speech and it lets the speech
> system stay in sync with what's being displayed.
ARIA provides an aria-live attribute, which is also implicit in
certain role values, to notify users when the content of certain
regions changes.
Sounds useful. This is the kind of thing you would use when making a
change in one window and transforming the results into another window.
For example, there may things you want to do with speech but you can't
speak because it's not the right form. So, for the moment, imagine a
simple text window with an input filter and an output filter. The
filters convert to and from a speakable form. When you're in the text
window, you can use all the normal text editing commands
NaturallySpeaking users have come to expect. When you're done, you push
the text over to the application through the output filter and, in
theory, everything will be nice inside the application environment.
This notification system could be useful to keep the application buffer
and the speakable text window synchronized
I would greatly appreciate it if you could let us know if you see any
areas where we could improve this strategy to address your needs.
Also, it would help to hear more about your specific experience: what
browser and assistive technology are you using? Are there any sites
which work well for you now which could be used as a model for best
practices?
For the most part, I have almost zero accessibility when it comes to
websites. I just suffer and my hands burn because is no other way to use
speech recognition with browsers without burning out my voice. The only
accessibility I have is plaintext dictation if I dictate into a
dictation box or a speech-enabled editing tool. That's it. Nuance has
some extensions for chrome and Firefox that theoretically provide some
of the capabilities you've described but, remember, head-up-ass, they
release the extensions and didn't bother to notice that it causes system
failures (browser lockup, core dumps etc.) anytime flash is enabled. Who
would've thought that flash would be running on the same system as
speech recognition???
Thanks,
Alice
Thanks for continuing the dialogue. I have 20 years of frustration about
how badly accessibility interfaces are implemented and as someone who
lives the life, I get even more frustrated when the simple intent is
replaced with the complex incomplete solution.
I know some of my UI ideas are nontraditional. I started with the Wang
freestyle project and I would have gone father in the UI world because I
seem to have a talent for creating simple, intent-based UI designs.
Sadly, my hands busted 1994 and I've spent the past 20 years trying to
climb the hill of making it possible to get back to doing UI work using
speech recognition.
Follow Polymer on Google+: plus.google.com/107187849809354688692
---
You received this message because you are subscribed to the Google Groups "Polymer" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/polymer-dev/534CD01F.8010205%40esjworks.com.
For more options, visit https://groups.google.com/d/optout.
