So I just did a conversion from single to dup and got the following
$btrfs balance start -mconvert=dup /mnt/data
RAID5/6 support has known problems and is strongly discouraged
to be used besides testing or evaluation. It is recommended that
you use one of the other RAID profiles.
The operation will continue in 10 seconds.
Use Ctrl-C to stop.
I let it run and it's dup now, so the conversion itself works correctly.
Looks like this was introduced in  but offhand I don't see what's wrong.