Non-portable packages are required to fail to build on architectures
where they can't work (
https://www.debian.org/doc/manuals/developers-reference/ch05.en.html#packages-arch-specific
). A build-dependency automatically does that (e.g. nodejs is arch:any,
libv8 isn't), but one can't build-depend on virtual packages.
A third option would be
Architecture: any
Build-Depends: pocl-opencl-icd [!armhf !kfreebsd-i386]
which would also allow running the test suite (pocl is the CPU ICD, and
hence the only one likely to work on a buildd).
(Not sure whether including ppc64el is allowed, as main can't depend on
non-free)
