Re: RFR: 8345538: Robot.mouseMove doesn't clamp bounds on macOS when trying to move mouse off screen [v3]

[Alexey Ivanov]

On Fri, 20 Dec 2024 20:51:09 GMT, Phil Race <p...@openjdk.org> wrote:

>> I ran it on Windows, sorry I should've said it explicitly.
>
>> Displays: 2 Placement: Side-by-side, with the bottom edge aligned as below
>> 
>> <img alt="image" width="478" 
>> src="https://private-user-images.githubusercontent.com/95945681/397856615-7ea5be69-989f-42b5-ac26-631d170d8db0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzQ3Mjc0NjMsIm5iZiI6MTczNDcyNzE2MywicGF0aCI6Ii85NTk0NTY4MS8zOTc4NTY2MTUtN2VhNWJlNjktOTg5Zi00MmI1LWFjMjYtNjMxZDE3MGQ4ZGIwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDEyMjAlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQxMjIwVDIwMzkyM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWZhYWUzZWNkZWZjOGFiZTRjMWE4ODgxMjhhYjE3N2E4NWRkOTA2MmI0MzMyZWJmNmExYTJmODA0ZjgzZGJmMmUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.vGpCyq1K_5DGxJ-rZPPp5PosfvP4Q90l-3hIAWb1E7M";>
>> Screen bounds: 
>> Win32GraphicsDevice[screen=0]:java.awt.Rectangle[x=-1920,y=363,width=1280,height=720]
>>  
>> Win32GraphicsDevice[screen=1]:java.awt.Rectangle[x=0,y=0,width=2293,height=960]
>> 
>> Stack Trace: ----------System.err:(11/959)---------- 
>> java.lang.NullPointerException: Cannot invoke 
>> "java.awt.PointerInfo.getLocation()" because the return value of 
>> "java.awt.MouseInfo.getPointerInfo()" is null at 
>> MouseMoveOffScreen.main(MouseMoveOffScreen.java:57) at 
>> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
>>  at java.base/java.lang.reflect.Method.invoke(Method.java:565) at 
>> com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
>>  at java.base/java.lang.Thread.run(Thread.java:1447)
>> 
>> JavaTest Message: Test threw exception: java.lang.NullPointerException: 
>> Cannot invoke "java.awt.PointerInfo.getLocation()" because the return value 
>> of "java.awt.MouseInfo.getPointerInfo()" is null JavaTest Message: shutting 
>> down test
> 
> The stack trace wasn't as revealing as I hoped. But the monitor config may 
> tell us something.
> (0, 200) is a location that is off the top of screen 1, so not within the 
> virtual bounds.
> But it is interesting that you get NPE before the fix as well .. perhaps the 
> windows clamping isn't happening as Alisen said. I think we need to 
> understand that as the next step.
> 
> After that , since it looks to me like the bounds of all screens need to be 
> looked at to clamp because we can't rely on a single rectangle defining it 
> even if there's a single virtual screen, and need to look at the union.

Now I ran the test on macOS Sequoia 15.1.1. With the current `Robot` 
implementation, it's able to move mouse across both displays. With Alisen's 
updated implementation, the coordinates are limited to the main screen.

<details>
<summary>Updated test</summary>


    private static void moveMouseAndCheck(final Robot robot,
                                          final int x, final int y) {
        System.out.println(x + ", " + y);
        robot.mouseMove(x, y);
        robot.delay(1000);

        PointerInfo cursor = MouseInfo.getPointerInfo();
        if (cursor != null) {
            System.out.println(cursor.getLocation() + " - "
                               + cursor.getDevice().getIDstring());
        }
    }

    public static void main(String[] args) throws Exception {
        Robot robot = new Robot();

        GraphicsDevice[] screens = getLocalGraphicsEnvironment()
                                   .getScreenDevices();
        for (GraphicsDevice screen : screens) {
            GraphicsConfiguration gc = screen.getDefaultConfiguration();
            System.out.println(screen.getIDstring());
            System.out.println("\t" + gc.getBounds());
            System.out.println("\t" + gc.getDefaultTransform());
        }
        System.out.println("\n");

        for (GraphicsDevice screen : screens) {
            System.out.println(screen.getIDstring());
            Rectangle bounds = screen.getDefaultConfiguration()
                                     .getBounds();

            moveMouseAndCheck(robot, bounds.x, bounds.y);
            moveMouseAndCheck(robot, bounds.x + bounds.width, bounds.y);
            moveMouseAndCheck(robot, bounds.x, bounds.y + bounds.height);
            moveMouseAndCheck(robot, bounds.x + bounds.width, bounds.y + 
bounds.height);
            moveMouseAndCheck(robot, bounds.x + bounds.width / 2, bounds.y + 
bounds.height / 2);
        }

</details>

Displays are arranged side by side: the built-in Display is on the left, the 
main display is on the right.

<details>
<summary>JDK 21 without the fix</summary>

**JDK 21 without the fix**

Display 69734208
        java.awt.Rectangle[x=-1440,y=0,width=1440,height=900]
        AffineTransform[[2.0, 0.0, 0.0], [0.0, 2.0, 0.0]]
Display 725353101
        java.awt.Rectangle[x=0,y=0,width=1920,height=1080]
        AffineTransform[[2.0, 0.0, 0.0], [0.0, 2.0, 0.0]]


Display 69734208
-1440, 0
java.awt.Point[x=-1440,y=0] - Display 69734208
0, 0
java.awt.Point[x=0,y=0] - Display 725353101
-1440, 900
cursor == null
0, 900
java.awt.Point[x=0,y=900] - Display 725353101
-720, 450
java.awt.Point[x=-720,y=450] - Display 69734208
Display 725353101
0, 0
java.awt.Point[x=0,y=0] - Display 725353101
1920, 0
cursor == null
0, 1080
cursor == null
1920, 1080
cursor == null
960, 540
java.awt.Point[x=960,y=540] - Display 725353101
Exception in thread "main" java.lang.NullPointerException: Cannot invoke 
"java.awt.PointerInfo.getLocation()" because the return value of 
"java.awt.MouseInfo.getPointerInfo()" is null
        at MouseMoveOffScreen.main(MouseMoveOffScreen.java:90)

</details>


<details>
<summary>With the fix</summary>

**With the fix**

Display 69734208
        java.awt.Rectangle[x=-1440,y=0,width=1440,height=900]
        AffineTransform[[2.0, 0.0, 0.0], [0.0, 2.0, 0.0]]
Display 725353101
        java.awt.Rectangle[x=0,y=0,width=1920,height=1080]
        AffineTransform[[2.0, 0.0, 0.0], [0.0, 2.0, 0.0]]


Display 69734208
-1440, 0
2024-12-20 20:39:09.383 java[94658:1065578] +[IMKClient subclass]: chose 
IMKClient_Modern
2024-12-20 20:39:09.383 java[94658:1065578] +[IMKInputSession subclass]: chose 
IMKInputSession_Modern
java.awt.Point[x=0,y=0] - Display 725353101
0, 0
java.awt.Point[x=0,y=0] - Display 725353101
-1440, 900
java.awt.Point[x=0,y=900] - Display 725353101
0, 900
java.awt.Point[x=0,y=900] - Display 725353101
-720, 450
java.awt.Point[x=0,y=450] - Display 725353101
Display 725353101
0, 0
java.awt.Point[x=0,y=0] - Display 725353101
1920, 0
cursor == null
0, 1080
cursor == null
1920, 1080
cursor == null
960, 540
java.awt.Point[x=960,y=540] - Display 725353101
Exception in thread "main" java.lang.NullPointerException: Cannot invoke 
"java.awt.PointerInfo.getLocation()" because the return value of 
"java.awt.MouseInfo.getPointerInfo()" is null
        at MouseMoveOffScreen.main(MouseMoveOffScreen.java:90)

</details>

In both cases, the points at the right and bottom edges of the screen aren't 
available, `MouseInfo.getPointerInfo()` returns `null`. I should've subtracted 
1 when adding `bounds.width` or `bounds.height`.

-------------

PR Review Comment: https://git.openjdk.org/jdk/pull/22781#discussion_r1894385239