You shouldn't need libpoclu-dev to build OpenCL-using code, only ocl-icd-opencl-dev (which is available on all architectures).
What does require an ICD is _running_ OpenCL code, including in the test suite (if any); as buildds are unlikely to have a GPU, build-time OpenCL-using tests can only be run on architectures with pocl-opencl-icd. You could build but not test the OpenCL-using part on the other architectures, but you may well not want to. There isn't a standard for how to turn off OpenCL (beyond the conventions applying to compile-time-optional features in general): check the git log for how you turned it on, and/or upstream's documentation.