Re: [chromium-dev] Keyboard codes to use for a chrome extension api that enables automation?
The problem I have with using only explicitly defined action functions
is that the whole point of simulating keyboard events is to test
whether or not the user interface is accessible using the keyboard!
(This is not just an issue for people using assistive technology like
a screen reader: there are a few things in Chrome that are impossible
to access without using a mouse now.)
I want to be able to write tests that try to edit bookmarks using the
Bookmark Manager, or change settings using the Options dialog, for
example. Having functions like selectNextBookmark() wouldn't help test
that they're actually accessible via the keyboard.
There aren't that many different keys needed for keyboard
accessibility: off the top of my head, you mostly need tab, shift-tab,
space, enter, and arrow keys - plus a few shortcuts to open a menu or
move the focus to a certain panel. What about having functions for
each of these keys plus other actions that can be triggered via a
keyboard shortcut?
tabToNextControl()
tabToPreviousControl()
pressSpaceBar()
pressEnterKey()
pressEscapeKey()
arrowUp()
arrowDown()
arrowLeft()
arrowRight()
openOptionsDialog()
openBookmarksManager()
focusLocationBar()
...
Then, in order to enter text into a text box in a dialog, perhaps a
function that pastes unicode text at the current cursor position would
be much easier than trying to deal with key codes and input methods.
- Dominic
On Fri, Jan 15, 2010 at 3:24 AM, James Su su...@google.com wrote:
Hi Nico,
I totally agree with your point. Using fake keyboard events for
accessibility things would lead to compatibility nightmare. We should
consider the approach of using explicitly defined action functions rather
than emulating keyboard events. The keyboard events approach has at least
following unsolvable problems:
1. Compatibility issue among different OS platforms.
2. Compatibility issue caused by user customized shortcuts.
3. Supports of different keyboard layouts.
4. Supports of inputting non-Latin text.
Regards
James Su
2010/1/15 Nico Weber tha...@chromium.org
On Thu, Jan 14, 2010 at 8:37 AM, Dominic Mazzoni dmazz...@google.com
wrote:
On Thu, Jan 14, 2010 at 8:19 AM, Nico Weber tha...@google.com wrote:
It probably depends if you want to use this for text input or action
inputs. A text-to-speech extension would probably want to set unicode
characters, while something that (say) hits cmd-f to open the find bar
probably wants to use keycodes.
Text-to-speech is a different problem; this is for actions.
For example, on OS X hitting cmd-f with a hiragana keyboard layout
opens the findbar, while just pressing f opens an IME.
If you want to use this for actons, I would find it more useful to
have a performUiAction() function instead though, since keyboard
shortcuts are different across platforms (all mac shortcuts use cmd
instead of ctrl for example, and the letters for a few shortcuts are
different on OS X too for various reasons).
I completely agree; I would like to add many functions along the lines
of performUiAction. However, there are a few actions for which typing
a key really is the right thing to do - for example pressing Tab to
move to the next control.
Maybe performUiAction('focusNextViewInKeyLoop')?
(By the way, OS X has a system-level setting that configures if tab
should focus only text fields or all controls such as buttons etc. I
don't think Chrome/Mac honors it, but I think there are plans to
support it. Stuff like this is surprising for a caller that sends
tab characters, while an explicit focus next element function
could have clearly defined semantics).
There's also testing. This function will allow you to use only
javascript to test that a particular capability can be accessed using
only the keyboard. I'm already using this for testing this API, by
simulating pressing Ctrl+L to focus the location bar and then checking
to see that the proper focus event is generated.
Some (most?) accessibility shortcuts will vary by platform too. For
example, focus toolbar is ctrl-f5 on OS X, but only if full keyboard
access has been enabled (ctrl-f1). Cocoa the OS X framework don't
always trigger with synthetic keyboard events. (I'm not an a11y expert
– Avi, comments?)
So the way I look at it is: there are some great uses for this
function. There are also things that this function could be used for
but there'd be a better way. It could also be abused to do things that
make no sense.
I don't think that's an argument not to allow the function; I think it
just means we should make sure it's defined properly for the intended
use, and over time try to provide better alternatives for other
possible uses.
- Dominic
Nico
On Thu, Jan 14, 2010 at 7:26 AM, Dominic Mazzoni dmazz...@google.com
wrote:
Hi Darin,
Erik suggested you might have some thoughts. In my proposed extension
api for accessibility
Re: [chromium-dev] Keyboard codes to use for a chrome extension api that enables automation?
What's the motivation of implementing such tests as extensions? The
automation framework already supports simulating keyboard events, can we
just use it for such kind of tests?
Regards
James Su
2010/1/16 Dominic Mazzoni dmazz...@google.com
The problem I have with using only explicitly defined action functions
is that the whole point of simulating keyboard events is to test
whether or not the user interface is accessible using the keyboard!
(This is not just an issue for people using assistive technology like
a screen reader: there are a few things in Chrome that are impossible
to access without using a mouse now.)
I want to be able to write tests that try to edit bookmarks using the
Bookmark Manager, or change settings using the Options dialog, for
example. Having functions like selectNextBookmark() wouldn't help test
that they're actually accessible via the keyboard.
There aren't that many different keys needed for keyboard
accessibility: off the top of my head, you mostly need tab, shift-tab,
space, enter, and arrow keys - plus a few shortcuts to open a menu or
move the focus to a certain panel. What about having functions for
each of these keys plus other actions that can be triggered via a
keyboard shortcut?
tabToNextControl()
tabToPreviousControl()
pressSpaceBar()
pressEnterKey()
pressEscapeKey()
arrowUp()
arrowDown()
arrowLeft()
arrowRight()
openOptionsDialog()
openBookmarksManager()
focusLocationBar()
...
Then, in order to enter text into a text box in a dialog, perhaps a
function that pastes unicode text at the current cursor position would
be much easier than trying to deal with key codes and input methods.
- Dominic
On Fri, Jan 15, 2010 at 3:24 AM, James Su su...@google.com wrote:
Hi Nico,
I totally agree with your point. Using fake keyboard events for
accessibility things would lead to compatibility nightmare. We should
consider the approach of using explicitly defined action functions rather
than emulating keyboard events. The keyboard events approach has at least
following unsolvable problems:
1. Compatibility issue among different OS platforms.
2. Compatibility issue caused by user customized shortcuts.
3. Supports of different keyboard layouts.
4. Supports of inputting non-Latin text.
Regards
James Su
2010/1/15 Nico Weber tha...@chromium.org
On Thu, Jan 14, 2010 at 8:37 AM, Dominic Mazzoni dmazz...@google.com
wrote:
On Thu, Jan 14, 2010 at 8:19 AM, Nico Weber tha...@google.com
wrote:
It probably depends if you want to use this for text input or
action
inputs. A text-to-speech extension would probably want to set unicode
characters, while something that (say) hits cmd-f to open the find
bar
probably wants to use keycodes.
Text-to-speech is a different problem; this is for actions.
For example, on OS X hitting cmd-f with a hiragana keyboard layout
opens the findbar, while just pressing f opens an IME.
If you want to use this for actons, I would find it more useful to
have a performUiAction() function instead though, since keyboard
shortcuts are different across platforms (all mac shortcuts use cmd
instead of ctrl for example, and the letters for a few shortcuts are
different on OS X too for various reasons).
I completely agree; I would like to add many functions along the lines
of performUiAction. However, there are a few actions for which typing
a key really is the right thing to do - for example pressing Tab to
move to the next control.
Maybe performUiAction('focusNextViewInKeyLoop')?
(By the way, OS X has a system-level setting that configures if tab
should focus only text fields or all controls such as buttons etc. I
don't think Chrome/Mac honors it, but I think there are plans to
support it. Stuff like this is surprising for a caller that sends
tab characters, while an explicit focus next element function
could have clearly defined semantics).
There's also testing. This function will allow you to use only
javascript to test that a particular capability can be accessed using
only the keyboard. I'm already using this for testing this API, by
simulating pressing Ctrl+L to focus the location bar and then checking
to see that the proper focus event is generated.
Some (most?) accessibility shortcuts will vary by platform too. For
example, focus toolbar is ctrl-f5 on OS X, but only if full keyboard
access has been enabled (ctrl-f1). Cocoa the OS X framework don't
always trigger with synthetic keyboard events. (I'm not an a11y expert
– Avi, comments?)
So the way I look at it is: there are some great uses for this
function. There are also things that this function could be used for
but there'd be a better way. It could also be abused to do things that
make no sense.
I don't think that's an argument not to allow the function; I think it
just means we should make sure
[chromium-dev] Keyboard codes to use for a chrome extension api that enables automation?
Hi Darin,
Erik suggested you might have some thoughts. In my proposed extension
api for accessibility (http://codereview.chromium.org/402099) one of
the functions is to simulate a key press. How should the client
express the key they would like to press? The current proposed
function prototype is:
{
name: simulateKeyPress,
type: function,
description: Simulate pressing a key.,
parameters: [
{
type: object,
name: keyInfo,
properties: {
key: {type: integer, description: The code of
the key to press, corresponding to event.keyCode.},
control: {type: boolean, optional: true,
description: True if the control key is down.},
shift: {type: boolean, optional: true,
description: True if the shift key is down.},
alt: {type: boolean, optional: true,
description: True if the alt key is down.}
}
}
]
}
What do you think? Should the key be a keyCode? A charCode?
Should it be cross-platform, or should it match what would be returned
by an onKeyDown handler? Are those even mutually exclusive?
My current thinking is: the symmetry of using the same key codes
returned by onKeyDown is appealing. Also, even though there are some
differences, the most common keys needed to automate the UI (tab,
enter, arrows, alphanumerics) are already consistent across platforms.
Thanks,
- Dominic
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
Re: [chromium-dev] Keyboard codes to use for a chrome extension api that enables automation?
The easiest thing for someone who's attempting to use the
accessibility API may be to avoid using keycodes, and instead allow
the input of the desired displayed value. The advantage of this would
be to allow the input of internationalized characters that would
otherwise need IME to be input.
Regards,
Simon
On Thu, Jan 14, 2010 at 3:26 PM, Dominic Mazzoni dmazz...@google.com wrote:
Hi Darin,
Erik suggested you might have some thoughts. In my proposed extension
api for accessibility (http://codereview.chromium.org/402099) one of
the functions is to simulate a key press. How should the client
express the key they would like to press? The current proposed
function prototype is:
{
name: simulateKeyPress,
type: function,
description: Simulate pressing a key.,
parameters: [
{
type: object,
name: keyInfo,
properties: {
key: {type: integer, description: The code of
the key to press, corresponding to event.keyCode.},
control: {type: boolean, optional: true,
description: True if the control key is down.},
shift: {type: boolean, optional: true,
description: True if the shift key is down.},
alt: {type: boolean, optional: true,
description: True if the alt key is down.}
}
}
]
}
What do you think? Should the key be a keyCode? A charCode?
Should it be cross-platform, or should it match what would be returned
by an onKeyDown handler? Are those even mutually exclusive?
My current thinking is: the symmetry of using the same key codes
returned by onKeyDown is appealing. Also, even though there are some
differences, the most common keys needed to automate the UI (tab,
enter, arrows, alphanumerics) are already consistent across platforms.
Thanks,
- Dominic
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
Re: [chromium-dev] Keyboard codes to use for a chrome extension api that enables automation?
On Thu, Jan 14, 2010 at 8:01 AM, Simon Stewart
simon.m.stew...@gmail.com wrote:
The easiest thing for someone who's attempting to use the
accessibility API may be to avoid using keycodes, and instead allow
the input of the desired displayed value. The advantage of this would
be to allow the input of internationalized characters that would
otherwise need IME to be input.
While this could be really useful for international characters, the
main use I had in mind was action keystrokes, like tab, arrow keys,
enter, Ctrl-+, etc. - so for these I think a keycode is needed, right?
- Dominic
Regards,
Simon
On Thu, Jan 14, 2010 at 3:26 PM, Dominic Mazzoni dmazz...@google.com wrote:
Hi Darin,
Erik suggested you might have some thoughts. In my proposed extension
api for accessibility (http://codereview.chromium.org/402099) one of
the functions is to simulate a key press. How should the client
express the key they would like to press? The current proposed
function prototype is:
{
name: simulateKeyPress,
type: function,
description: Simulate pressing a key.,
parameters: [
{
type: object,
name: keyInfo,
properties: {
key: {type: integer, description: The code of
the key to press, corresponding to event.keyCode.},
control: {type: boolean, optional: true,
description: True if the control key is down.},
shift: {type: boolean, optional: true,
description: True if the shift key is down.},
alt: {type: boolean, optional: true,
description: True if the alt key is down.}
}
}
]
}
What do you think? Should the key be a keyCode? A charCode?
Should it be cross-platform, or should it match what would be returned
by an onKeyDown handler? Are those even mutually exclusive?
My current thinking is: the symmetry of using the same key codes
returned by onKeyDown is appealing. Also, even though there are some
differences, the most common keys needed to automate the UI (tab,
enter, arrows, alphanumerics) are already consistent across platforms.
Thanks,
- Dominic
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
Re: [chromium-dev] Keyboard codes to use for a chrome extension api that enables automation?
It probably depends if you want to use this for text input or action
inputs. A text-to-speech extension would probably want to set unicode
characters, while something that (say) hits cmd-f to open the find bar
probably wants to use keycodes.
For example, on OS X hitting cmd-f with a hiragana keyboard layout
opens the findbar, while just pressing f opens an IME.
If you want to use this for actons, I would find it more useful to
have a performUiAction() function instead though, since keyboard
shortcuts are different across platforms (all mac shortcuts use cmd
instead of ctrl for example, and the letters for a few shortcuts are
different on OS X too for various reasons).
On Thu, Jan 14, 2010 at 7:26 AM, Dominic Mazzoni dmazz...@google.com wrote:
Hi Darin,
Erik suggested you might have some thoughts. In my proposed extension
api for accessibility (http://codereview.chromium.org/402099) one of
the functions is to simulate a key press. How should the client
express the key they would like to press? The current proposed
function prototype is:
{
name: simulateKeyPress,
type: function,
description: Simulate pressing a key.,
parameters: [
{
type: object,
name: keyInfo,
properties: {
key: {type: integer, description: The code of
the key to press, corresponding to event.keyCode.},
control: {type: boolean, optional: true,
description: True if the control key is down.},
shift: {type: boolean, optional: true,
description: True if the shift key is down.},
alt: {type: boolean, optional: true,
description: True if the alt key is down.}
}
}
]
}
What do you think? Should the key be a keyCode? A charCode?
Should it be cross-platform, or should it match what would be returned
by an onKeyDown handler? Are those even mutually exclusive?
My current thinking is: the symmetry of using the same key codes
returned by onKeyDown is appealing. Also, even though there are some
differences, the most common keys needed to automate the UI (tab,
enter, arrows, alphanumerics) are already consistent across platforms.
Thanks,
- Dominic
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
Re: [chromium-dev] Keyboard codes to use for a chrome extension api that enables automation?
On Thu, Jan 14, 2010 at 8:19 AM, Nico Weber tha...@google.com wrote:
It probably depends if you want to use this for text input or action
inputs. A text-to-speech extension would probably want to set unicode
characters, while something that (say) hits cmd-f to open the find bar
probably wants to use keycodes.
Text-to-speech is a different problem; this is for actions.
For example, on OS X hitting cmd-f with a hiragana keyboard layout
opens the findbar, while just pressing f opens an IME.
If you want to use this for actons, I would find it more useful to
have a performUiAction() function instead though, since keyboard
shortcuts are different across platforms (all mac shortcuts use cmd
instead of ctrl for example, and the letters for a few shortcuts are
different on OS X too for various reasons).
I completely agree; I would like to add many functions along the lines
of performUiAction. However, there are a few actions for which typing
a key really is the right thing to do - for example pressing Tab to
move to the next control.
There's also testing. This function will allow you to use only
javascript to test that a particular capability can be accessed using
only the keyboard. I'm already using this for testing this API, by
simulating pressing Ctrl+L to focus the location bar and then checking
to see that the proper focus event is generated.
So the way I look at it is: there are some great uses for this
function. There are also things that this function could be used for
but there'd be a better way. It could also be abused to do things that
make no sense.
I don't think that's an argument not to allow the function; I think it
just means we should make sure it's defined properly for the intended
use, and over time try to provide better alternatives for other
possible uses.
- Dominic
Nico
On Thu, Jan 14, 2010 at 7:26 AM, Dominic Mazzoni dmazz...@google.com wrote:
Hi Darin,
Erik suggested you might have some thoughts. In my proposed extension
api for accessibility (http://codereview.chromium.org/402099) one of
the functions is to simulate a key press. How should the client
express the key they would like to press? The current proposed
function prototype is:
{
name: simulateKeyPress,
type: function,
description: Simulate pressing a key.,
parameters: [
{
type: object,
name: keyInfo,
properties: {
key: {type: integer, description: The code of
the key to press, corresponding to event.keyCode.},
control: {type: boolean, optional: true,
description: True if the control key is down.},
shift: {type: boolean, optional: true,
description: True if the shift key is down.},
alt: {type: boolean, optional: true,
description: True if the alt key is down.}
}
}
]
}
What do you think? Should the key be a keyCode? A charCode?
Should it be cross-platform, or should it match what would be returned
by an onKeyDown handler? Are those even mutually exclusive?
My current thinking is: the symmetry of using the same key codes
returned by onKeyDown is appealing. Also, even though there are some
differences, the most common keys needed to automate the UI (tab,
enter, arrows, alphanumerics) are already consistent across platforms.
Thanks,
- Dominic
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
Re: [chromium-dev] Keyboard codes to use for a chrome extension api that enables automation?
On Thu, Jan 14, 2010 at 8:37 AM, Dominic Mazzoni dmazz...@google.com wrote:
On Thu, Jan 14, 2010 at 8:19 AM, Nico Weber tha...@google.com wrote:
It probably depends if you want to use this for text input or action
inputs. A text-to-speech extension would probably want to set unicode
characters, while something that (say) hits cmd-f to open the find bar
probably wants to use keycodes.
Text-to-speech is a different problem; this is for actions.
For example, on OS X hitting cmd-f with a hiragana keyboard layout
opens the findbar, while just pressing f opens an IME.
If you want to use this for actons, I would find it more useful to
have a performUiAction() function instead though, since keyboard
shortcuts are different across platforms (all mac shortcuts use cmd
instead of ctrl for example, and the letters for a few shortcuts are
different on OS X too for various reasons).
I completely agree; I would like to add many functions along the lines
of performUiAction. However, there are a few actions for which typing
a key really is the right thing to do - for example pressing Tab to
move to the next control.
Maybe performUiAction('focusNextViewInKeyLoop')?
(By the way, OS X has a system-level setting that configures if tab
should focus only text fields or all controls such as buttons etc. I
don't think Chrome/Mac honors it, but I think there are plans to
support it. Stuff like this is surprising for a caller that sends
tab characters, while an explicit focus next element function
could have clearly defined semantics).
There's also testing. This function will allow you to use only
javascript to test that a particular capability can be accessed using
only the keyboard. I'm already using this for testing this API, by
simulating pressing Ctrl+L to focus the location bar and then checking
to see that the proper focus event is generated.
Some (most?) accessibility shortcuts will vary by platform too. For
example, focus toolbar is ctrl-f5 on OS X, but only if full keyboard
access has been enabled (ctrl-f1). Cocoa the OS X framework don't
always trigger with synthetic keyboard events. (I'm not an a11y expert
– Avi, comments?)
So the way I look at it is: there are some great uses for this
function. There are also things that this function could be used for
but there'd be a better way. It could also be abused to do things that
make no sense.
I don't think that's an argument not to allow the function; I think it
just means we should make sure it's defined properly for the intended
use, and over time try to provide better alternatives for other
possible uses.
- Dominic
Nico
On Thu, Jan 14, 2010 at 7:26 AM, Dominic Mazzoni dmazz...@google.com wrote:
Hi Darin,
Erik suggested you might have some thoughts. In my proposed extension
api for accessibility (http://codereview.chromium.org/402099) one of
the functions is to simulate a key press. How should the client
express the key they would like to press? The current proposed
function prototype is:
{
name: simulateKeyPress,
type: function,
description: Simulate pressing a key.,
parameters: [
{
type: object,
name: keyInfo,
properties: {
key: {type: integer, description: The code of
the key to press, corresponding to event.keyCode.},
control: {type: boolean, optional: true,
description: True if the control key is down.},
shift: {type: boolean, optional: true,
description: True if the shift key is down.},
alt: {type: boolean, optional: true,
description: True if the alt key is down.}
}
}
]
}
What do you think? Should the key be a keyCode? A charCode?
Should it be cross-platform, or should it match what would be returned
by an onKeyDown handler? Are those even mutually exclusive?
My current thinking is: the symmetry of using the same key codes
returned by onKeyDown is appealing. Also, even though there are some
differences, the most common keys needed to automate the UI (tab,
enter, arrows, alphanumerics) are already consistent across platforms.
Thanks,
- Dominic
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
Re: [chromium-dev] Keyboard codes to use for a chrome extension api that enables automation?
On Thu, Jan 14, 2010 at 9:54 AM, Erik Arvidsson a...@google.com wrote:
Keyboard events in browsers has and is a compatibility nightmare. I
feel like the current working draft for DOM level 3 Events is making
good progress in cleaning this up and it is worth a read if you are
planning to do anything related to keyboard events in a browser.
http://www.w3.org/TR/DOM-Level-3-Events/
Here are some random points based on your initial proposal.
Thanks.
I would far prefer to follow an existing standard rather than
re-invent the wheel here, and this spec looks very complete. Would it
make sense for this method to simply take an object of type
keyboardEvent, as defined in the DOM level 3 event draft? Do we
already have support for turning a keyboardEvent into a native event
on each platform, or is there a logical place to put such code?
Writing new keyboard event conversion code specifically for this
extension api seems like the wrong idea - I'd far prefer to tie this
to existing effort in Chrome or WebKit...
- Dominic
- You also need 'meta' modifier support (meta maps to the command key on Mac).
- You need to distinguish and allow both left and right modifier keys.
This is important for things like Bidi switching and language
switching
- There needs to be some way to do keydown and keyup (as well as textInput)
- Using keyIdentifier seems like the way to go.
- You probably also need to support something similar to keyLocation
(which covers left and right shift)
- Also look at DOM textInput event which is the attempt to fix the
keypress event and it handles things like IME and other methods for
inputting text.
erik
On Thu, Jan 14, 2010 at 07:26, Dominic Mazzoni dmazz...@google.com wrote:
Hi Darin,
Erik suggested you might have some thoughts. In my proposed extension
api for accessibility (http://codereview.chromium.org/402099) one of
the functions is to simulate a key press. How should the client
express the key they would like to press? The current proposed
function prototype is:
{
name: simulateKeyPress,
type: function,
description: Simulate pressing a key.,
parameters: [
{
type: object,
name: keyInfo,
properties: {
key: {type: integer, description: The code of
the key to press, corresponding to event.keyCode.},
control: {type: boolean, optional: true,
description: True if the control key is down.},
shift: {type: boolean, optional: true,
description: True if the shift key is down.},
alt: {type: boolean, optional: true,
description: True if the alt key is down.}
}
}
]
}
What do you think? Should the key be a keyCode? A charCode?
Should it be cross-platform, or should it match what would be returned
by an onKeyDown handler? Are those even mutually exclusive?
My current thinking is: the symmetry of using the same key codes
returned by onKeyDown is appealing. Also, even though there are some
differences, the most common keys needed to automate the UI (tab,
enter, arrows, alphanumerics) are already consistent across platforms.
Thanks,
- Dominic
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
--
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
Re: [chromium-dev] Keyboard codes to use for a chrome extension api that enables automation?
On Thu, Jan 14, 2010 at 4:15 PM, Dominic Mazzoni dmazz...@google.com wrote: On Thu, Jan 14, 2010 at 8:01 AM, Simon Stewart simon.m.stew...@gmail.com wrote: The easiest thing for someone who's attempting to use the accessibility API may be to avoid using keycodes, and instead allow the input of the desired displayed value. The advantage of this would be to allow the input of internationalized characters that would otherwise need IME to be input. While this could be really useful for international characters, the main use I had in mind was action keystrokes, like tab, arrow keys, enter, Ctrl-+, etc. - so for these I think a keycode is needed, right? The keycode is essentially an indication of which physical key was pressed on the keyboard, not which letter the user means to input (where the charcode may be more useful) Even on a standard US keyboard, it's possible to get the same letter via different keycodes just by changing the layout used. Throw international keyboards (such as my UK one, or my colleagues German ones) and you're in a whole world of pain and discomfort. Add choices such as mapping the caps lock key to ctl and the picture becomes muddier still. Worse, unless you provide constants, not many people know which keycode normally maps to which key. Which is a painfully long way of saying that I really think that the keycode is a poor choice. Fortunately, there are at least two pieces of prior art here: 1) MS's SendInput function, which takes a formatted string 2) WebDriver's mechanism for performing similar action keystrokes. I wrote WebDriver's implementation in order to allow testing of a complex application which demanded high fidelity simulation of user input. This seems to broadly overlap your stated goal, so here's the gist of what we do: * All keyboard input is modeled as normal strings * In the simple case of a lower-case string, we do a direct mapping from character to keystroke * In the case where a capital letter is shown, we automatically simulate holding down the shift key * The same applies to any character on the current keyboard layout which can be typed by holding down the shift key * We have defined constants in a unicode PUA for meta characters, such as control, shift, tab, escape and so on. * Meta characters are assumed to be sticky, that is, if we see the ctl character, we assume that the control key is held down until we see another ctl character (where we release the key) or the end of the string or a release all meta keys character is seen, in which case all meta keys are released in the reverse order they were held down in (so most recently pressed is released first) * For characters which have well known mappings in strings (\n, \t) we simulate using the expected key The advantage of this scheme is that the common case (of a user simply typing into a textarea or input element) is easy to read and makes sense to someone not familiar with the test or the API. The more complex case of a user wanting to simulate pressing a normal keyboard shortcut is also handled intuitively because of the behaviour seen when we reach the end of a string. Finally, it is possible to encode relatively complex chording and interactions using this mechanism. Example usages of this API can be seen here: http://selenium.googlecode.com/svn/trunk/common/test/js/typing_test.html FWIW, webdriver's API is already available for Firefox via JS, and there is already work to get this working with Chrome. I'd be happy to provide more information. Regards, Simon -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
