Basically, X has an additional layer of abstraction; apps aren't talking directly to the hardware or even using a relatively low-level interface; instead they have to talk to X via a special protocol. This means that X applications can be run over the network ("network transparent").
Another contributing factor is that many XFree86 servers aren't optimized as well as the corresponding Windows drivers, but they are improving over time. XFree86 has some new technologies that should begin to address this problem, such as better acceleration. Havoc