Never heard of ears swelling due to fever, though an allergic reaction could very well cause both to happen as a result of the allergy. Are you giving him any new ear drops, or using transdermal medicines on his ears?
One of my cats has heavier cartilage in his ears than all the others cats I have ever known, they are very thick and very stiff. But they stay that way, it doesn't come and go. Phaewryn http://ucat.us Adopt a New England FIV+ cat: http://ucat.us/FIVadopt.html Special Needs Cat Links (and feline info library): http://ucat.us/domesticcatlinks.html
