Yes. In practical terms, most of the times, it all comes down to a matter of
trust in the entity that ultimately provides you the product.
Very few people also inspect the source code of the programs they use, and
compare it with the compiled form in which such programs are distributed.
(Because, most of them, are not even able to, in terms of knowledge.) And,
merely choose a distro they think they can trust in.
But, the same way that I count on this happening for the software... If a
curious person, who knows a lot about the subject and that has the ability
(or, in this case, the material capacity) to make such inspection, were to
find something wrong, it could warn the others.
And, with "free hardware", you wouldn't necessarily be dependent on some
shady corporation, you wouldn't trust in, that would be the only entity
authorized to make such a hardware. And, you could improve, a lot, the
confidence on your equipment, if it was possible for such a hardware to be
also made by another entity you knew you could trust in.