If your OS supports S/Key, you might find that a useful option. S/Key
authentication uses one-time-use passwords; much safer than general use
password protection. You (and any others who need it) can keep a collection
of S/Key passphrases in a wallet, and use them even with public-access
PCs in internet cafes or libraries -- notorious for having keyboard capture
virii and other ills.
To support S/Key -- if the OS allows it -- one merely needs
ChallengeResponseAuthentication yes
in sshd_config.
I use OpenBSD. In that environement, to use S/Key, I merely connect with
"<user>:skey" as the login userid to enable the challenge and response. I
get a prompt that shows me which passphrase to use from my list of passphrases.
If I don't use the "...:skey" login 'style' then the only other allowable
authentication is RSA public key.
When using S/Key, I just need to ensure no one else can read the passphrase
list when I remove it from my wallet. :)