Setting RuntimeDirectoryPreserve=yes solves the issue for me.
- Bug#934663: socket activated sshd sometimes com... Marc Haber
- Bug#934663: socket activated sshd sometime... Marc Haber
- Bug#934663: socket activated sshd sometime... WGH
- Processed: Bug#934663 marked as pending in... Debian Bug Tracking System
- Bug#934663: marked as done (socket activat... Debian Bug Tracking System

