AFAIK, that only works when usecanonical names is on. Otherwise, it just uses what the client sends.
Wouldn't adding the port to ServerName be what you need?
ServerName example.com:80
Iirc it is even suggested in the docs.
-- Brian Akins Senior Systems Engineer CNN Internet Technologies