> The trouble I've had in the past wasn't so much needing X later, but > needing dependencies in the X packages later, FWIW.
This is true also. More and more packages have quite well-hidden dependencies on X. It is easier for the ports developers and users if X is always there. Choose to not install X at your own risk.
