You shouldn't need libpoclu-dev to build OpenCL-using code, only
ocl-icd-opencl-dev (which is available on all architectures).
What does require an ICD is _running_ OpenCL code, including in the test
suite (if any); as buildds are unlikely to have a GPU, build-time
OpenCL-using tests can only be run on architectures with pocl-opencl-icd.
You could build but not test the OpenCL-using part on the other
architectures, but you may well not want to.
There isn't a standard for how to turn off OpenCL (beyond the conventions
applying to compile-time-optional features in general): check the git log
for how you turned it on, and/or upstream's documentation.