I couldn't find any options in 'man ld' to *remove* a directory from the library search path. I hope I simply missed it - it would seem a rather odd omission otherwise.
Well, how exactly do you expect to tell ld which libraries to remove from the search path? (Canonicalization, anyone?)
IIRC, you use the order in which you add libraries to the search path to override any potential library references you don't want.