Hi shawn, Try UI browser which checks every UI component in your interface for any information it can extract, in arrays of values that work similarly to a collection in a json or hive table (this is the closest I can illustrate it to be though it probablhy does not adopt the same API). This can be used in conjunction with UIActions (from the same developer) which allows you to change virtual data sets from actions performed on an invisible interface.
The javascript implementation for applescript is rather recent and I’m not sure what it can perform but considering js can be wrapped around applescript, and the underlying user interface components made in objc or java (for those platform independent user interfaces), more information on the elements, their positioning (for custom groupings of controls) and mouse events can be parsed thus scripted. Though this is a welcome addition to the applescript API, there isn’t much documentation in this regards. but I can speculate that some 3rd party modules can be imported such as those scanning for attribute/value changes. The tedious part for this is to populate the value table and figuring or investigating what each key/value pair contains, whether it is editable or not. From there you will have to perform a comparison between possible controls on the element, if child nodes are present, establishing data linking between the invisible interface and an interpreter that can also drill into menus, their sub elements, tables and cell values, etc. But I would also think that custom interfaces are mainly graphically oriented thus don’t encapsulate a lot of information for each node therein which can simply be sent a “say” or NSSpeechSynthesis command such as for the python bridging module pyobjc (which you can find with brew or one of the other repositories). Again, profiling the interface will be a massive task unless you know how to compare/connnect events and interface control dynamics. There’s also the issue of doing this for as many interpretations of customisability for each developer, requiring to ask them what base framework they used, what dynamic link library native to mac OS (carbon? coco bridge? some arbitrary wrapper?) they use for that specific application. This is why I think pushing some standards for each platform is the better option, and considering privacy concerns for the mac os, a lot of reference values are probably closed unless you check nvram for memory mappings, their memory space data types and the values taken from them, in which order (little or big indian) etc. Its an interesting case of memory reading and understanding the memory spaces allocated per thread in a given program, but this is a hack, and definetely not the best performing way of going about it. I tried to do this with the world of warcraft interface which has an external data struct as well as an internal one which cannot be passed from the game and written dynamically on log files or listener scripts. The only thing I can see there is using something like a visual computing library to scan the most important parts of an interface by pixel space which is not where I want to go at this point. It would avoid breaking the user agreement, as opening this would lead to a lot of potential for game exploits, but how to communicate this need for massive online player games to be accessible to blind players is one of my big concerns right now. Anyway, hope some points can be used for further investigation, and good luck. > On 22/06/2015, at 9:28 PM, Sean Murphy <[email protected]> wrote: > > Hello, > > > do you have any more info on what you have outlined below? since this is an > area of interest for myself. Any pointers would be helpful and grateful. > > Sean >> On 22 Jun 2015, at 3:27 am, Yuma Decaux <[email protected]> wrote: >> >> The current OS has python, applescript and jscript that can send information >> directly to voice over, but I haven’t touched the scripts pertaining to >> these in a while as there are 3rd party applications that allow to do this >> very well. Also, there are a bunch of CL tools that also allow some lesser >> inherent amount of manipulation of the GUIs, provided they are rendered >> accessible. >> >> I’m not sure what happens on the windows side, but I’ve taken it to myself >> to do everything a coder would do on the mac, and so far it’s getting >> smoother everytime. And much much faster too. Can’t say more. >> >> Cheers, >> >> >> >>> On 21/06/2015, at 11:21 AM, Sean Murphy <[email protected]> wrote: >>> >>> On either platform, a developer can use their own controls still. They do >>> not have to use the provided objects in the Coco library or Windows >>> development tool. This occurs equally on both platforms for the reasons >>> that have already been raised. The struggle anyone has even if you work >>> within the vendor is to get them to use the necessary resources to fix it. >>> >>> The biggest challenge I find with Voice-Over is if an application is not >>> using the Coco library controls (objects) that do have accessibility built >>> into them or the developer is using their own method. Then that product is >>> not usable on the Mac. If this was the case on Windows, you do have the >>> ability of addressing it via the screen readers. On the flip side, this is >>> a disadvantage as well because you are not educating the developer best >>> practise to use the right accessibility libraries in the first place. There >>> are applications on Windows which the outlined solution available on >>> windows screen readers do not work. In saying this, you can go a lot >>> further then you can on the Mac. >>> >>> One thing I want to validate and I suspect it can be done but is a lot >>> harder on the Mac due to you having to know a lot more. Is it possible to >>> interrogate an application like Pages via Perl, Python, Coco (Swift) and >>> then send this information directly to Voice-Over. For example: You create >>> a little app that grabs the current style being used with all the relevant >>> properties. The info is sent to voice-over and the app can turn the feature >>> on or off. The reason for asking this is, if it can be done then I can see >>> how to really improve the experience of a lot of different applications via >>> this method. Since most developers will not provide this enhance feature. >>> Window screen reader’s provide this type of enhance capability via their >>> scripting environments and you do not have to learn an complete development >>> environment. >>> >>> How much control via scripts does someone have over Voice-Over using the >>> above languages? >>> >>> Sean >>> >>> >>> >>>> On 21 Jun 2015, at 3:28 am, Yuma Decaux <[email protected]> wrote: >>>> >>>> Hi, >>>> >>>> Let me just add that the wwdc this year has put a highlight on >>>> accessibility to the developers attending or downloading the workshop >>>> videos, so saying that apple has put accessibility to the bottom of the >>>> list is untrue. In addition, now all standard UI controls in xcode are by >>>> default accessible. This has engendered an effort from oracle with its >>>> javafx framework to render all their controls accessible for both mac and >>>> windows. >>>> >>>> When it comes to custom APIs, I remember there used to be a tool called >>>> accessibility inspector which allows the developer to check the hierarchy >>>> of controls and scenes in an application to see which control gives what >>>> accessible output, or what voice over will announce. Usualy custom >>>> controls are 3D based or image based, which requires an effort from the >>>> developer to handle what is said, how and what triggers it, which is a set >>>> of coding behaviours that need to be placed on top of the custom control. >>>> >>>> Cheers, >>>> >>>> >>>>> On 21/06/2015, at 1:06 AM, Littlefield, Tyler <[email protected]> wrote: >>>>> >>>>> -----BEGIN PGP SIGNED MESSAGE----- >>>>> Hash: SHA1 >>>>> >>>>> Hello: >>>>> First, it's always a really hard discussion when someone with little >>>>> to no development experience talks about how things "should" be done >>>>> because they're usually way off. So I'll explain how things work >>>>> currently. >>>>> >>>>> A long long time ago, in a galaxy far far away Windows and OSX >>>>> developers all used different APIs than they do now. A majority of >>>>> windows applications were written in C/C++ or delphy, with a couple >>>>> others thrown in. OSX used C mainly (maybe c++) and they used the >>>>> Carbon API. >>>>> >>>>> Carbon and Win32 basically let people do the same stuff, create >>>>> windows, etc. but where the problems came in was that it gave a lot of >>>>> control to the developer, so the developer could basically write up >>>>> all sorts of cool new things in their controls. Now that people >>>>> started moving more toward using vb.net, c# etc for Windows >>>>> development (because using the win32 API is not fun--I have a media >>>>> player I'm writing mostly in c), as well as people moving toward >>>>> Cocoah, this problem is going away for the most part. >>>>> >>>>> Now the problems we face are a lot smaller. Windows had MSAA and UIAA >>>>> for providing information to the screen readers, and Cocoah has its' >>>>> own accessibility frameworks. As you're using a framework they >>>>> control, it's really easy to make most things accessible by default. >>>>> But again, the issue is with custom controls. Many times a developer >>>>> wants something new in their app, because it looks cool or because the >>>>> control can provide custom functionality that another one does not. >>>>> This, I believe is the issue you are running into. So rather than >>>>> Apple do what Windows does as they're already doing that, there needs >>>>> to be something that tells developers when their controls are broken. >>>>> This really should be up to the developer to fix, mostly because Apple >>>>> can either say nothing with custom controls gets into the app store >>>>> (bad idea), or they would have to fix it themselves. Maybe some sort >>>>> of testing framework could be created to determine if everything works >>>>> as it should. The goal would be to create an instance of that control >>>>> and determine if it has the requisite properties. >>>>> >>>>> Sorry for the rambling jumble here, no coffee yet! >>>>> HTH, >>>>> On 6/20/2015 8:58 AM, Devin Prater wrote: >>>>>> Hi all. So I’ve been thinking about the accessibility of both the >>>>>> Mac and Windows apps. While Apple has clearly laid out the details >>>>>> of how the accessibility API works, developers usually don’t know >>>>>> them because either its way down in the developer guides or the >>>>>> developers just don’t worry about that kind of stuff. This isn’t >>>>>> just complex apps, these are little apps too, the apps you’d expect >>>>>> to work flawlessly, like Atlantis, the MUD client for mac. I’d love >>>>>> to be able to use it, but nope. Why not, I ask Apple. “Its up to >>>>>> developers to make their apps accessible.” Why? Why should it be >>>>>> the developer’s fault if an app they make can’t be used by the >>>>>> system screen reader? I think that the accessibility engineers have >>>>>> been going about this the wrong way. First of all, if a developer >>>>>> uses a custom development that doesn’t support accessibility, there >>>>>> is no way of fixing that, and we can’t expect developers to rebuild >>>>>> apps just for us. Take the Alter-aeon MUD app for example. Now, >>>>>> maybe an app is pretty accessible but maybe just needs a little >>>>>> more tweaking that the developers just won’t be bothered with? Or >>>>>> maybe an app like open-emu, where the preferences dialog is almost >>>>>> accessible but the tabs along the top of the window cannot be >>>>>> reached via keyboard. We can’t expect developers to get it all >>>>>> right. I think that voiceover should copy what other windows screen >>>>>> readers have done in the past and has made countless apps >>>>>> accessible. Just get information about what’s on the screen and >>>>>> make that available to voiceover as well as the os x API’s. >>>>>> >>>>> >>>>> >>>>> - -- >>>>> Take care, >>>>> Ty >>>>> twitter: @sorressean >>>>> web:http://tysdomain.com >>>>> pubkey: http://tysdomain.com/files/pubkey.asc >>>>> -----BEGIN PGP SIGNATURE----- >>>>> Version: GnuPG v2 >>>>> >>>>> iQEcBAEBAgAGBQJVhYF5AAoJEAdP60+BYxejDoEH+wVpVznk5HoG3cBrFB8/J3Ew >>>>> isf6MYlGPVyC/n6WAiJhtndM3Ih91oXPmsqh/V9sUjPZNY+VDo52byHcxBW5bAWL >>>>> 4sNaU6p3d5a/jOicTYJAlP5GBsyHcMmTalSpNwZUVNlp17rfAe6TnsOodgiVvWUR >>>>> PkhlO5OmYfnJMa++7i8KnAjVdsGi1OHN1AWlHwknnPp1Qu1CAPOWHe/ZGPfrD44H >>>>> JcaTGFnGcDS93Gz/j7qPvjr5NI37mE8pIUnyotTx5H+okTpt9CWjOgMVmTwKDxxb >>>>> PjHk4u85/3Og2uxVyE8BcCVafmpKHAfX6jOTrJJoMKJ+8fZxA8ZRVue+dasqCWg= >>>>> =PAQw >>>>> -----END PGP SIGNATURE----- >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "MacVisionaries" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send an >>>>> email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at http://groups.google.com/group/macvisionaries. >>>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "MacVisionaries" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at http://groups.google.com/group/macvisionaries. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "MacVisionaries" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at http://groups.google.com/group/macvisionaries. >>> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "MacVisionaries" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at http://groups.google.com/group/macvisionaries. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "MacVisionaries" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/macvisionaries. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "MacVisionaries" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/macvisionaries. For more options, visit https://groups.google.com/d/optout.
