The simplest workaround I found is to include "clk_ignore_unused" in the kernel parameters when the installer boots.
This also works with GRUB, there press "e" key when GRUB menu appears, edit the line starts with "linux" e.g.: linux /boot/vmlinuz clk_ignore_unused ro quiet Since the clock driver kernel modules included in the arm64 installer initrd.gz, it can load it successfully. Can be closed.