A binary language is a language with two letters that are usually denoted "0"
and "1". Two letters always are enough: a mapping can be defined from any
other alphabet to (long-enough) sequences of "0" and "1". Most digital
systems use that language: on a CD, 500 nm wide pits and lands represent the
two letters, most digital circuits (in particular in a processor) only
consider two signals: high or low voltage; etc.
All that said, when we write about "a binary", we are actually talking about
a file whose content is directly executable by the processor, i.e., a file
only containing extremely simple (i.e., low-level) instructions. Those
instructions are the assembly language. Like any other language, the
instructions (the letters of the language) can be mapped to sequences of "0"
and "1". But that is not the point.
The point is: humans can only read/write tiny programs in an assembly
language. Most of the programs we use are written in high-level languages
(mixes of English and mathematics), that are far easier to work with.
Programs in high-level languages can then be automatically translated (a
process called "compilation") to the assembly language the processor
understands... but that human beings cannot understand anymore (unless the
program is tiny). Also, it basically is impossible do do any significant
modification to a binary.
Almost all proprietary programs are only distributed as binaries, i.e., human
beings cannot read/write them. On the contrary, a free software program
grants access to the source code (in the high-level language the program was
actually written in): the user can study the program and can modify it.