On Wed, May 27, 2026 at 09:24:46AM -0400, Mark Higgins wrote: > I've tried using the "computer use" stuff in AIs to automatically push > buttons in the XG app. It doesn't work, because the AI needs to know > what position is displayed on the screen, and they're still really bad at > counting checkers in an image of a board - they can't reliably identify the > position.
Maybe you don't need to do it this way. Folks that try to transcribe live matches recorded with a camera have indeed to manage with only the image of the board, but scraping a GUI you usually have the moves list displayed beside the board. It might be easier to read it and update your representation of the position incrementally rather than interpret the current board.
