Did a few more things to try and debug this problem.
- Tried to remove 60-latin.conf and apparently the Noto fonts are still loaded (not the DejaVu I chose in my config)
- Tried to move 50-user.conf to 01-user.conf and to 99-user.conf, and the still the Noto fonts are loaded instead of DejaVu
Any idea what might be going on here, or how to debug it? It might be a bug in my configuration and not in the package then.