I highly recommend the talk by Daniel Stone who used to be a core X.org
developer. He explains it quite well how X is used currently, and why
it has problems and why they are considered so hard to fix that Wayland
(and Mir) was created.
One interesting point he made was the X server is no longer network
transparent like it used to be. It is network capable now but when used
in that way (ssh forwarding), it's essentially done in the same way as
VNC, but more poorly because of the way X11 is architected.