Maybe that's because x11-display take a few milliseconds to start, and
that xmodmap actually starts before x11-display is actually done
loading.
I've encountered this type of issue with emacs-server, where I used this
solution on RDE, but it's emacs/service specific :
I recently set up home-xmodmap-service-type so I could change the
PrtSc key on my ThinkPad into a Hyper modifier. This is my
configuration:
(service home-xmodmap-service-type
(home-xmodmap-configuration
(key-map '("clear mod3"
("remove mod4" .