Re: Problem using AppleScript of "system events" to click a location on screen
You didn’t mention the kind of mac you are running and whether either the
standalone or ide is running in rosetta or not :)
To be fair, that part was about the exec error rather than the expected outcome
though…
The reason you are seeing the effect you are when not doing the request from a
separate process is that executing applescript is a modal operation (Ie you
can’t get an application to use AppleScript to control itself) - and I suspect
the accessibility framework is timing out internally and just doing the best it
can in that situation.
Warmest Regards,
Mark.
Sent from my iPhone
> On 27 Apr 2023, at 19:22, Ben Rubinstein via use-livecode
> wrote:
>
> Update to note:
>
> 1) Difference in LiveCode versions.
>
> Under the IDE in LiveCode 9.6.7 behaviour is as per the standalone; no click,
> and 'the result' is the path to the control. Under the IDE in LiveCode 9.6.8
> or LiveCode 10.0.0dp5, 'the result' is "execution error".
>
> 2) The browser widget is a red herring; relevant only insofar as if it was
> some other spot on the stack, I could just the LiveCode "click at ".
> Whether over the browser widget or some other part of the stack window,
> invoking the applescript returns the identity of the control at that
> location, rather than clicking it (in LC 9.6.7 IDE, or standalone built from
> any version); or execution error in later IDEs.
>
>> On 27/04/2023 16:42, Ben Rubinstein via use-livecode wrote:
>> I had a need to click on an element in a web page loaded in a browser widget
>> on a card.
>> There might be an elegant way to do this using javascript injected into the
>> widget, but I'm too ignorant to figure it out.
>> So to save time (hah!), I though I could use the accessibility functionality
>> to just click on that bit of the screen. I could get the location to click
>> within the stack, then use globalLoc to convert it to screen coordinates.
>> (This is just a personal stack to achieve an objective, nothing that's ever
>> going to be shared with anyone else, so any filthy/fragile method is OK if
>> it works.)
>> I tested this in Script Debugger:
>> tell application "System Events"
>> click at {917, 667}
>> end tell
>> It worked fine.
>> So then I tried the same script in my stack, via
>> do ... as "applescript"
>> And after a brief spinning pizza, got the dry result "execution error".
>> I expected that this would happen because I needed to give it permission to
>> control the computer; but after granting that in System Preferences, there
>> was no change.
>> Just for fun, I tried building a standalone from my stack. This demonstrated
>> a different effect: first I got the same result "execution error". Then I
>> granted it permission for this standalone app to 'control the computer'.
>> Now, there's still no evidence that it sends a click; but it gives a
>> different result, which I think is the path to the UI element corresponding
>> to that location. If the point is over a native control in the stack, it's
>> something like
>> window "Ben Test Stack (1)" of application process "Ben Test Stack"
>> of application "System Events"
>> (I don't think LiveCode controls are recognisable by the accessibility
>> system.) If the point is over the browser widget, 'the result' is something
>> like:
>> group 2 of UI element 1 of scroll area 1 of group 1 of group 1
>> of window "Ben Test Stack (1)" of application process "Ben Test Stack"
>> of application "System Events"
>> ("Ben Test Stack" is the name of my stack and of the standalone. My guess
>> about the complicated control path is that it reflects the complicated web
>> page loaded in the browser widget.)
>> So that does suggest that there is something which doesn't quite work about
>> giving the IDE permission to use the Accessibility Framework, but which does
>> for a standalone. But still doesn't explain what the problem is.
>> I finally solved my problem with an even uglier hack: compiling the script
>> from Script Debugger as an 'application', and then in my stack, instead of
>> executing the applescript directly, using "launch" on that application.
>> Yeuchh.
>> Can anyone shed light, either on how to grant the IDE permission to use the
>> Accessibility controls; or on why the applescript wouldn't work?
>> TIA,
>> Ben
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Re: Problem using AppleScript of "system events" to click a location on screen
Update to note:
1) Difference in LiveCode versions.
Under the IDE in LiveCode 9.6.7 behaviour is as per the standalone; no click,
and 'the result' is the path to the control. Under the IDE in LiveCode 9.6.8
or LiveCode 10.0.0dp5, 'the result' is "execution error".
2) The browser widget is a red herring; relevant only insofar as if it was
some other spot on the stack, I could just the LiveCode "click at ".
Whether over the browser widget or some other part of the stack window,
invoking the applescript returns the identity of the control at that location,
rather than clicking it (in LC 9.6.7 IDE, or standalone built from any
version); or execution error in later IDEs.
On 27/04/2023 16:42, Ben Rubinstein via use-livecode wrote:
I had a need to click on an element in a web page loaded in a browser widget
on a card.
There might be an elegant way to do this using javascript injected into the
widget, but I'm too ignorant to figure it out.
So to save time (hah!), I though I could use the accessibility functionality
to just click on that bit of the screen. I could get the location to click
within the stack, then use globalLoc to convert it to screen coordinates.
(This is just a personal stack to achieve an objective, nothing that's ever
going to be shared with anyone else, so any filthy/fragile method is OK if it
works.)
I tested this in Script Debugger:
tell application "System Events"
click at {917, 667}
end tell
It worked fine.
So then I tried the same script in my stack, via
do ... as "applescript"
And after a brief spinning pizza, got the dry result "execution error".
I expected that this would happen because I needed to give it permission to
control the computer; but after granting that in System Preferences, there was
no change.
Just for fun, I tried building a standalone from my stack. This demonstrated a
different effect: first I got the same result "execution error". Then I
granted it permission for this standalone app to 'control the computer'. Now,
there's still no evidence that it sends a click; but it gives a different
result, which I think is the path to the UI element corresponding to that
location. If the point is over a native control in the stack, it's something like
window "Ben Test Stack (1)" of application process "Ben Test Stack"
of application "System Events"
(I don't think LiveCode controls are recognisable by the accessibility
system.) If the point is over the browser widget, 'the result' is something like:
group 2 of UI element 1 of scroll area 1 of group 1 of group 1
of window "Ben Test Stack (1)" of application process "Ben Test Stack"
of application "System Events"
("Ben Test Stack" is the name of my stack and of the standalone. My guess
about the complicated control path is that it reflects the complicated web
page loaded in the browser widget.)
So that does suggest that there is something which doesn't quite work about
giving the IDE permission to use the Accessibility Framework, but which does
for a standalone. But still doesn't explain what the problem is.
I finally solved my problem with an even uglier hack: compiling the script
from Script Debugger as an 'application', and then in my stack, instead of
executing the applescript directly, using "launch" on that application. Yeuchh.
Can anyone shed light, either on how to grant the IDE permission to use the
Accessibility controls; or on why the applescript wouldn't work?
TIA,
Ben
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Problem using AppleScript of "system events" to click a location on screen
On 2023-04-27 16:42, Ben Rubinstein via use-livecode wrote: Can anyone shed light, either on how to grant the IDE permission to use the Accessibility controls; or on why the applescript wouldn't work? Not immediately - no :) However, three questions... What version of LC are you using? Are you using an Apple architecture mac? If so, is the IDE running under Rosetta, and the standalone native, or vice-versa (or both the same)? -- Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/ LiveCode: Build Amazing Things ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Problem using AppleScript of "system events" to click a location on screen
I had a need to click on an element in a web page loaded in a browser widget
on a card.
There might be an elegant way to do this using javascript injected into the
widget, but I'm too ignorant to figure it out.
So to save time (hah!), I though I could use the accessibility functionality
to just click on that bit of the screen. I could get the location to click
within the stack, then use globalLoc to convert it to screen coordinates.
(This is just a personal stack to achieve an objective, nothing that's ever
going to be shared with anyone else, so any filthy/fragile method is OK if it
works.)
I tested this in Script Debugger:
tell application "System Events"
click at {917, 667}
end tell
It worked fine.
So then I tried the same script in my stack, via
do ... as "applescript"
And after a brief spinning pizza, got the dry result "execution error".
I expected that this would happen because I needed to give it permission to
control the computer; but after granting that in System Preferences, there was
no change.
Just for fun, I tried building a standalone from my stack. This demonstrated a
different effect: first I got the same result "execution error". Then I
granted it permission for this standalone app to 'control the computer'. Now,
there's still no evidence that it sends a click; but it gives a different
result, which I think is the path to the UI element corresponding to that
location. If the point is over a native control in the stack, it's something like
window "Ben Test Stack (1)" of application process "Ben Test Stack"
of application "System Events"
(I don't think LiveCode controls are recognisable by the accessibility
system.) If the point is over the browser widget, 'the result' is something like:
group 2 of UI element 1 of scroll area 1 of group 1 of group 1
of window "Ben Test Stack (1)" of application process "Ben Test Stack"
of application "System Events"
("Ben Test Stack" is the name of my stack and of the standalone. My guess
about the complicated control path is that it reflects the complicated web
page loaded in the browser widget.)
So that does suggest that there is something which doesn't quite work about
giving the IDE permission to use the Accessibility Framework, but which does
for a standalone. But still doesn't explain what the problem is.
I finally solved my problem with an even uglier hack: compiling the script
from Script Debugger as an 'application', and then in my stack, instead of
executing the applescript directly, using "launch" on that application. Yeuchh.
Can anyone shed light, either on how to grant the IDE permission to use the
Accessibility controls; or on why the applescript wouldn't work?
TIA,
Ben
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode
