It turned out I was trying to specify the resolution in a wrong way - using "resolution=WxH" instead of "resolution WxH".
Still, an attempt to start ROTT the correct way - $ rott-commercial window resolution 800x600 (with any supported width and height) fails in exactly the same manner, as indicated in the original report.