You clearly don't understand the distinction between SaaSS and connecting to servers in general. Browsing the web is not SaaSS, but under your definition ('anything that involves connecting to a server you do not directly control') it would be. The difference is that in the case of, say, Google Drive word processing, you are using a service hosted on a server to do your own computing for you, instead of simply running a free word processor of your own. The ethical issue here is that, as with proprietary software, the user is not in control of her own computing- those who run the server are. Requesting files from a server, however, is not SaaSS because those files are simply being retrieved and then viewed in a program on the user's own computer. Sure, potential abuses exist, as they always do, but this does not make the practice of browsing the web unethical. It is important to note that in both these cases the user is connecting to servers running software they have no direct control over- yet there is a clear difference in how both practices (browsing the web and SaaSS) actually function, and the ethical consequences therein.
My case is that I do not consider playing, say, Nexuiz online over someone else's server a case of SaaSS- and I don't think Stallman does either. A game of Nexuiz usually requires both a client and a server. Usually (in single-player) these are both hosted on the same machine; one client, one server. In the case of online play, however, multiple users have decided to share one server between their many clients, communicating over a network.