You can forward your USB controllers to a VM OR Disable them in the BIOS
It is very easy to re-write a USB drive firmware via another virus on a poorly secured different computer so this doesn't really need physical access not that it would be difficult to simply have someone cause a scene and then have someone else walk by and insert a drive in to your laptop for a few seconds while you were distracted if you were a high profile target (politician, ceo, lawyer etc)
