spamd deny users from using the SMTP

[Edy]

Hi,

Spamd is running fine now but when an user trying to send out email 
using the SMTP he is getting Server  temporary failure please try again 
later error message.
They have to retry many times before the IP address is whitelisted.
Any workaround to that?
We would not be able to add the IP address into the whitelist before 
hand because users could be connected from anywhere.

Thanks,
Edy