The first thing to study is the board that you're testing. You should have a good basic understanding of its operation. Whfen you functionally test something, you are attempting to recreate the normal working environment of the board and verify that it meets it's design specifications. You will need a detailed test specification that lists all setup and stimulus/response conditions along with tolerances on stimulus and measurements. Once you have all of that, you can begin to design the test equipment. What, if any signal conditioning required will depend on the specs of the measurments and the specs of the test instruments. For example, if you're measuring an analog signal, is it AC or DC, what is the expected amplitude, what is the frequency and are all of these with the range of you instrument? A bed of nails fixture is typically used only for low frequency so you may have to do something there.
When designing the software, you want to consider who is going to run the program and what is the purpose. Do you need to provide the ability to run a single test out of many? Do you need to log data? What happens when a failure occurs - stop, continue, pause? I hope that's gives you a start. It's kind of hard to condense years of experience into a couple of paragraphs.
