Hello. The ARP cache timeout used to be 1200 seconds or 20 minutes, hard coded. Now, it looks like it's either 1200 seconds or 300 seconds, I'm not sure after a quick romp through the kernel source. In any case, The fact that you're getting regular delays on your pings suggests there is a delay between the time when the arp cache times out and when it gets refreshed. As a consequence of that delay, if you have a high speed stream running when the cache times out, it's possible the send buffer of the sending process, i.e. sshd, is filling up before that cache gets refreshed and the packets can flow again. What is the value of net.inet.tcp.sendbuf_max on your dom0? also, is net.inet.tcp.sendbuf_auto set to 1? If not, try setting that to 1 with sysctl(8) and see if that changes the behavior at all.
-thanks -Brian