I think I have found the issue.
The problem was a circlular reference in some class that managed
the creation and destruction of PartyAnimals. Now I still don't
know how that triggered the behaviour that I saw, but I do have a
theory. When a certain size of memory is allocated, the GC may
have said that it's time to clean up. If there was an instance
created prior to the actual code being run, e.g. through a
unittest, it would get a problem while cleaning up.
So the problem essentially was that my unittests weren't
extensive enough, as they did not cover deallocation of some
instances. Ironically, by reducing the unittest to a few lines, I
found which one was actually triggering the problem.
TL;DR: not a D bug, problem caused by my own incompetence.