No you have not. You only have to remove the latest kernel (about three clicks in the Synaptics package manager) and reboot. The previous kernel will be used.
And you can try other versions of Linux-libre as easily (Trisquel's repository has versions 3.13, 3.16, 3.19 and 4.2; Jxself's repository has versions 3.4, 3.14, 4.1 and 4.5).
