Re: RFR: 8345538: Robot.mouseMove doesn't clamp bounds on macOS when trying to move mouse off screen [v3]

[Alexey Ivanov]

On Fri, 20 Dec 2024 20:59:52 GMT, Alexey Ivanov <aiva...@openjdk.org> wrote:

>>> Displays: 2 Placement: Side-by-side, with the bottom edge aligned as below
>>> 
>>> <img alt="image" width="478" 
>>> src="https://private-user-images.githubusercontent.com/95945681/397856615-7ea5be69-989f-42b5-ac26-631d170d8db0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzQ3Mjc0NjMsIm5iZiI6MTczNDcyNzE2MywicGF0aCI6Ii85NTk0NTY4MS8zOTc4NTY2MTUtN2VhNWJlNjktOTg5Zi00MmI1LWFjMjYtNjMxZDE3MGQ4ZGIwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjIwVDIwMzkyM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWZhYWUzZWNkZWZjOGFiZTRjMWE4ODgxMjhhYjE3N2E4NWRkOTA2MmI0MzMyZWJmNmExYTJmODA0ZjgzZGJmMmUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.vGpCyq1K_5DGxJ-rZPPp5PosfvP4Q90l-3hIAWb1E7M";>
>>> Screen bounds: 
>>> Win32GraphicsDevice[screen=0]:java.awt.Rectangle[x=-1920,y=363,width=1280,height=720]
>>>  
>>> Win32GraphicsDevice[screen=1]:java.awt.Rectangle[x=0,y=0,width=2293,height=960]
>>> 
>>> Stack Trace: ----------System.err:(11/959)---------- 
>>> java.lang.NullPointerException: Cannot invoke 
>>> "java.awt.PointerInfo.getLocation()" because the return value of 
>>> "java.awt.MouseInfo.getPointerInfo()" is null at 
>>> MouseMoveOffScreen.main(MouseMoveOffScreen.java:57) at 
>>> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
>>>  at java.base/java.lang.reflect.Method.invoke(Method.java:565) at 
>>> com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
>>>  at java.base/java.lang.Thread.run(Thread.java:1447)
>>> 
>>> JavaTest Message: Test threw exception: java.lang.NullPointerException: 
>>> Cannot invoke "java.awt.PointerInfo.getLocation()" because the return value 
>>> of "java.awt.MouseInfo.getPointerInfo()" is null JavaTest Message: shutting 
>>> down test
>> 
>> The stack trace wasn't as revealing as I hoped. But the monitor config may 
>> tell us something.
>> (0, 200) is a location that is off the top of screen 1, so not within the 
>> virtual bounds.
>> But it is interesting that you get NPE before the fix as well .. perhaps the 
>> windows clamping isn't happening as Alisen said. I think we need to 
>> understand that as the next step.
>> 
>> After that , since it looks to me like the bounds of all screens need to be 
>> looked at to clamp because we can't rely on a single rectangle defining it 
>> even if there's a single virtual screen, and need to look at the union.
>
> Now I ran the test on macOS Sequoia 15.1.1. With the current `Robot` 
> implementation, it's able to move mouse across both displays. With Alisen's 
> updated implementation, the coordinates are limited to the main screen.
> 
> <details>
> <summary>Updated test</summary>
> 
> 
>     private static void moveMouseAndCheck(final Robot robot,
>                                           final int x, final int y) {
>         System.out.println(x + ", " + y);
>         robot.mouseMove(x, y);
>         robot.delay(1000);
> 
>         PointerInfo cursor = MouseInfo.getPointerInfo();
>         if (cursor != null) {
>             System.out.println(cursor.getLocation() + " - "
>                                + cursor.getDevice().getIDstring());
>         }
>     }
> 
>     public static void main(String[] args) throws Exception {
>         Robot robot = new Robot();
> 
>         GraphicsDevice[] screens = getLocalGraphicsEnvironment()
>                                    .getScreenDevices();
>         for (GraphicsDevice screen : screens) {
>             GraphicsConfiguration gc = screen.getDefaultConfiguration();
>             System.out.println(screen.getIDstring());
>             System.out.println("\t" + gc.getBounds());
>             System.out.println("\t" + gc.getDefaultTransform());
>         }
>         System.out.println("\n");
> 
>         for (GraphicsDevice screen : screens) {
>             System.out.println(screen.getIDstring());
>             Rectangle bounds = screen.getDefaultConfiguration()
>                                      .getBounds();
> 
>             moveMouseAndCheck(robot, bounds.x, bounds.y);
>             moveMouseAndCheck(robot, bounds.x + bounds.width, bounds.y);
>             moveMouseAndCheck(robot, bounds.x, bounds.y + bounds.height);
>             moveMouseAndCheck(robot, bounds.x + bounds.width, bounds.y + 
> bounds.height);
>             moveMouseAndCheck(robot, bounds.x + bounds.width / 2, bounds.y + 
> bounds.height / 2);
>         }
> 
> </details>
> 
> Displays are arranged side by side: the built-in Display is on the left, the 
> main display is on the right.
> 
> <details>
> <summary>JDK 21 without the fix</summary>
> 
> **JDK 21 without the fix**
> 
> Display 69734208
>       java.awt.Rectangle[x=-1440,y=0,width=1440,height=900]
>       AffineTransform[[2.0, 0.0, 0.0], [0.0, 2.0, 0.0]]
> Display 725353101
>       java.awt.Rectangle[x=0,y=0,width=1920,height=1080]
>       AffineTransform[[2.0, 0.0, 0.0], [0.0, 2.0, 0.0]]
> 
> 
> Display 69734208
> -1440, 0
> java.awt.Point[x=-1440,y=0] - Display 69734208
> 0, 0
> java.awt.Point[x=0,y=0] - Display 725353101
> -1440, 900
> cursor == null
> ...

Note that even with Alisen's fix, **the test fails to get the mouse pointer 
location after moving it off the screen!**

The coordinates are expected to be within the screen bounds.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/22781#discussion_r1894391146