I'm definitely not on a crusade in this matter. Risk assessment will vary with each individual organization's requirements. As Postfix offers excellent means of using both encryption and authentication, I have not yet come across a situation where relying on IP addresses alone was the sole option available. Then again, I usually work in environments where distributing SSL keys in a safe manner is usually not a problem.
-Ralph