As an update to this, I have not been able to figure out or fix the
issue as of yet, so I have had to continue using Stretch for now which
seems to be working fine.
The problem appears to be kernel related, if I use the kernel from
Stretch backports the problem occurs. I guess that's not much of a
surprise since the Stretch backports kernel is the same release as Buster.
I have left one of my servers on Buster in a broken state for testing
sometime in future if I get time. If I do find out a fix I'll post a
dnsdist mailing list