Re: [IxDA Discuss] User Testing and Recruiting Advice
My company does recruiting. We have done remote as well as on site testing. Many of our clients use WebEx for testing. We have also had clients use web cameras. One of the keys to recruiting success is having the recruiters understand who you are looking for. Most recruiting companies have a room full of drones who aren't even trusted enough to give them computers. In house recruiting is certainly possible but you'll need a way to track respondents, and if you want them to be fresh you'll need to develop a database that tracks participation, notes, demographics and so on. Jared's comments that We deal with many participants that can't, by law, get paid for helping us. This is a different kind of recruiting than general consumers. General consumers should absolutely be paid well for their opinions, just as you are. We tell participants they are being paid for doing a job and should view the appointment in the same way they would view showing up at work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=38251 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] User Testing and Recruiting Advice
Regarding your question about remote testing - I've done a great deal of remote testing and have had excellent results. I've even run pseudo-paper prototypes remotely using PhotoShop layers to handle the paper switching. PowerPoint can be useful for that as well As for software to use, I've had the best results by far with GoToMeeting. It's simple to use and the performance is very good. It also has some handy pen highlighter tools that are useful when having discussions about the UI with the participant on the other end. Remote testing is great because people can squeeze in tests right from their desk whenever they have a spare hour or so, and geographical location is virtually irrelevant - though time zone language may be for international testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=38251 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] User Testing and Recruiting Advice
I concur with all of Rob's comments. Certainly finding the right incentive is key. One of my clients recently said that as the world is in economic hard times, then we should be able to pay people less. I defended this in the context of he cost of the test, i.e. paying 5 people $20 less each is a tiny saving in the context of the whole session (for which a majority attendance is required), and the risk it introduces. Other pointers include: - collecting people's cellphone numbers - telling people you'll remind them beforehand (ensuring of course you do) - telling people the incentives are cash if this is the case - letting people know that there's only a small number of attendees so it's really important they attend - informing them that it's a one-to-one, not a group (i.e. they WILL be missed) - giving them ALL the information to ensure their arrival on time. This can include nearby parking, rates, payment methods, and how much additional time to allow for this - give clear instructions on how to find the facility. Many testing facilities are hidden inside a nondescript buildling Basically the key is the incentivize enough and mitigate all the little things that can make a participant late The other thing that helps is scheduling sessions with gaps between, rather than back to back. This allows particularly insightful sessions to extend, but also late showing participants a chance of still doing a full session. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=38251 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
[IxDA Discuss] User Testing and Recruiting Advice
My company is looking to make in-house user testing a monthly practice for it's many brands. We have the testing location, we have the necessary equipment, we have usability practitioners on staff. What we are lacking, is successful recruiting practices (many users cancel at the last minute or do not show up). I was hoping to get some feedback/advice from the group on the following: 1) Tips for successful recruiting using in house resources, as we can not use an outside recruiting agency for cost reasons? 2) We have considered forming a Customer Council. A group of dedicated users that represent defined personas for each brand. The idea is to utilize these individuals to get their feedback/opinion on website enhancements, etc. Does anyone have an experience using a Customer Council model? 3) Has anyone had success with remote testing? Any preferred software or methods that you can recommend? 4) Netflix and Microsoft (to name a few) have incorporated rapid user testing into their process. Does anyone on the list happen to work for either of those companies? If so, would you be willing to share your experiences, what has and has not worked for you? Thanks in advance for your opinions and expertise, it is very much appreciated. Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] User Testing and Recruiting Advice
I agree with Rob, but in house recruiting is time consuming, which means it costs the company money, but it is a hidden cost. It costs about $100 for scheduled participant to use a recruiting service and they take on the burden of replacing no shows and such. If you find a recruiting agency you like, stick with them. Give them a very detailed user profile and provide the qualifying criteria for each study and let them do the work so you can focus on designing and administering the tests. That is what Microsoft does because they need thousands of participants every year; they contract with vendors to do the work to get butts into chairs. I have done lots of remote testing, and it often works well, depending on what you are testing or evaluating. But I have had even more no-shows with remote testing and I think this is because they have no sense that anyone will miss them, perhaps. And when Rob says pay more money, that means paying people what they feel their time is worth. You are taking 1, 2, or 3 hours of their time in order to improve a product and make a higher profit. They know that and I think participants sometimes feel we take advantage of them. Never expect people to show up out of altruism. Mary Deaton Manager, STC Usability and User Experience On Fri, Feb 6, 2009 at 7:39 AM, Rob Tannen rtan...@bresslergroup.comwrote: Vicki - Some recommended bast practices for participant recruitment and retention: -Expect a 25% cancellation/no-show rate. So for every four slots you need to fill, you need to recruit 5 people. You can schedule extra slots or use floaters -- people scheduled to serve as substitutes overlapping with specific time slots. If you're running ahead of schedule you can include the floaters as well. -Remind recruited participants ahead of time (week before, then day before). Tell people to come in 15 minutes before the actual testing slow and then call them on their cell phones if they are not there at that time. -Pay more money. Increase your participant incentive by 10-20% can be a cost-effective way to improve show-up rates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=38251 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help -- Mary Deaton Yes we can. Yes we did. Yes we will Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] User Testing and Recruiting Advice
Hi Vicki, I'll take a stab at answering these two, since the others are very broad and could result in endless threads of their own on this list. On Feb 6, 2009, at 6:44 AM, Vicki Splaine wrote: 1) Tips for successful recruiting using in house resources, as we can not use an outside recruiting agency for cost reasons? I co-wrote an entire report on just this subject called Recruiting Without Fear (http://is.gd/iFbB). It's 43 pages of tips on how to do your own recruiting. In our research, the most effective organizations do their own recruiting, because it's a great way to extend your research capabilities. It's not hard to do, but there is a fair amount of administration involved, so you have to be prepared to put the resources into it. Shirking on the resources will reduce the effectiveness, increase your costs, and take a lot more time in the long run. 2) We have considered forming a Customer Council. A group of dedicated users that represent defined personas for each brand. The idea is to utilize these individuals to get their feedback/opinion on website enhancements, etc. Does anyone have an experience using a Customer Council model? We've used customer councils and advisory boards in research with many of our clients. When done well, it's very powerful. A couple of caveats: 1) Not all councils / boards are the same. Before invitations are sent out, the team really needs to know what they want to get from the council. If you just invite folks without having clear goals, it reduces the likelihood that the council/board will produce long-term useful results, beyond just enhancing the power of the echo chamber. 2) Make sure you're not confusing personas (which are behavior archetypes) with market segments (which are demographic/psychographic groupings). For most organizations, it would be virtually impossible to form a council around personas, since well-formed personas don't map into specific individuals or groups. 3) Look beyond focus groups. Far beyond focus groups. Like, if someone says, Hey, let's just invite them together for a sort of focus group, take that individual out and slap them around. (I really think we don't take enough advantage of natural selection. Sometimes we just need to thin the herd, know what I mean?) If the primary execution of your council/board is to gather them together and have them discuss designs, don't bother. It's not going to provide any more insight than you currently have and, as found in many documented cases, is likely to lead your team in the wrong direction. The council/board will be of most use if you use them as a base for behavioral work, which probably means going to them and watching, not discussing. Hope that helps, Jared Jared M. Spool User Interface Engineering 510 Turnpike St., Suite 102, North Andover, MA 01845 e: jsp...@uie.com p: +1 978 327 5561 http://uie.com Blog: http://uie.com/brainsparks Twitter: jmspool UIE Web App Summit, 4/19-4/22: http://webappsummit.com Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] User Testing and Recruiting Advice
Rob, With all due love and respect, I disagree with these recommendations. At least, two of them. On Feb 6, 2009, at 7:39 AM, Rob Tannen wrote: Vicki - Some recommended bast practices for participant recruitment and retention: -Expect a 25% cancellation/no-show rate. So for every four slots you need to fill, you need to recruit 5 people. You can schedule extra slots or use floaters -- people scheduled to serve as substitutes overlapping with specific time slots. If you're running ahead of schedule you can include the floaters as well. 25%!!! We have less than a 2% no-show rate across thousands of participants (including neurosurgeons!) over the last 20 years. (We don't get paid if participants don't show up, so we don't have any tolerance for no-shows.) If you have a 25% no-show rate, you've got something really wrong with your recruiting practice. There's no excuse that 1 out of 4 folks don't show up. In most studies, you never have to recruit floaters. For the occasional high-priority study, you might include a couple of extra participants in the original schedule to be safe. We do that for one out of every ten studies we do. -Remind recruited participants ahead of time (week before, then day before). Tell people to come in 15 minutes before the actual testing slow and then call them on their cell phones if they are not there at that time. This is better. Though, since we almost never have no-shows, we don't bother with the 15-minute before thing. No sense in wasting anyone's time when everyone is where they are supposed to be. -Pay more money. Increase your participant incentive by 10-20% can be a cost-effective way to improve show-up rates. Huh. We deal with many participants that can't, by law, get paid for helping us. They still show up when they are supposed to. Money to get people to show us is a trap that will likely lead to lower quality sessions. You really don't want people who are there for the money. (The money is really just to prove to them that you're serious and not going to turn it into a timesharing sales session.) It really sounds to me like your recruitment process could use an overhaul if this is what you're seeing for results. That's my $0.02. Jared Jared M. Spool User Interface Engineering 510 Turnpike St., Suite 102, North Andover, MA 01845 e: jsp...@uie.com p: +1 978 327 5561 http://uie.com Blog: http://uie.com/brainsparks Twitter: jmspool UIE Web App Summit, 4/19-4/22: http://webappsummit.com Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] User Testing and Recruiting Advice
Jared - You're one of four people (the other three being my wife and daughters) who I can't win an argument with, so I won't bother. But I am curious as to whether others have recruitment experiences that are more similar to yours or mine. Maybe I'll have to download your report. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=38251 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] User Testing and Recruiting Advice
In addition to our report, I should've also mentioned that Dana Chisnell has been writing on this topic on her blog: Yes or No: Make Your Recruiter Smarter http://is.gd/iFrx Why Your Screener Isn't Working http://is.gd/iFrV Her firm, Usability Works, are ideal recruiters and understand why UX recruiting is not the same as market research recruiting. They also have very low no-show rates and produce high-quality participants, even for very unusual project requirements. They're on the top of my list for outside recruiters. (Dana will also help you set up an effective internal recruitment process, if that's how you want to go.) Jared On Feb 6, 2009, at 12:26 PM, Jared Spool wrote: Hi Vicki, I'll take a stab at answering these two, since the others are very broad and could result in endless threads of their own on this list. On Feb 6, 2009, at 6:44 AM, Vicki Splaine wrote: 1) Tips for successful recruiting using in house resources, as we can not use an outside recruiting agency for cost reasons? I co-wrote an entire report on just this subject called Recruiting Without Fear (http://is.gd/iFbB). It's 43 pages of tips on how to do your own recruiting. In our research, the most effective organizations do their own recruiting, because it's a great way to extend your research capabilities. It's not hard to do, but there is a fair amount of administration involved, so you have to be prepared to put the resources into it. Shirking on the resources will reduce the effectiveness, increase your costs, and take a lot more time in the long run. 2) We have considered forming a Customer Council. A group of dedicated users that represent defined personas for each brand. The idea is to utilize these individuals to get their feedback/opinion on website enhancements, etc. Does anyone have an experience using a Customer Council model? We've used customer councils and advisory boards in research with many of our clients. When done well, it's very powerful. A couple of caveats: 1) Not all councils / boards are the same. Before invitations are sent out, the team really needs to know what they want to get from the council. If you just invite folks without having clear goals, it reduces the likelihood that the council/board will produce long-term useful results, beyond just enhancing the power of the echo chamber. 2) Make sure you're not confusing personas (which are behavior archetypes) with market segments (which are demographic/ psychographic groupings). For most organizations, it would be virtually impossible to form a council around personas, since well- formed personas don't map into specific individuals or groups. 3) Look beyond focus groups. Far beyond focus groups. Like, if someone says, Hey, let's just invite them together for a sort of focus group, take that individual out and slap them around. (I really think we don't take enough advantage of natural selection. Sometimes we just need to thin the herd, know what I mean?) If the primary execution of your council/board is to gather them together and have them discuss designs, don't bother. It's not going to provide any more insight than you currently have and, as found in many documented cases, is likely to lead your team in the wrong direction. The council/board will be of most use if you use them as a base for behavioral work, which probably means going to them and watching, not discussing. Hope that helps, Jared Jared M. Spool User Interface Engineering 510 Turnpike St., Suite 102, North Andover, MA 01845 e: jsp...@uie.com p: +1 978 327 5561 http://uie.com Blog: http://uie.com/brainsparks Twitter: jmspool UIE Web App Summit, 4/19-4/22: http://webappsummit.com Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] User Testing and Recruiting Advice
On Feb 6, 2009, at 12:56 PM, Rob Tannen wrote: Jared - You're one of four people (the other three being my wife and daughters) who I can't win an argument with, so I won't bother. But I am curious as to whether others have recruitment experiences that are more similar to yours or mine. Maybe I'll have to download your report. I know the feeling. *I* can't even win an article with *me* somedays. And it really pisses me off. Seriously. Download the report. If it doesn't help, I'll refund the money. And feel free to contact me to talk about how to get your no-show rate down. It'll really make your life easier and get you better results in the long run. Jared Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
Re: [IxDA Discuss] User Testing and Recruiting Advice
In thinking more on this topic and after absorbing Jared's comments, another possible problem came to mind. You have to be flexible and accommodating to the users you really want to bring in. If you have found truly representative users for your study, work with their schedule. Many people with 9-5 jobs cannot be at a study at 10am. Offer evening and weekend opportunities in addition to traditional business hours. Find out if they have young children. I've had studies postponed at the last minute because of childcare difficulties. Knowing this might happened allowed me to communicate with users so they know it's okay to let me know if something comes up. Rather than becoming a no-show or cancelation, it's a reschedule for another day. Look at how you promote the opportunity. If it doesn't sound fun or lacks an impact on products the user actually uses, it could be hard to convince them to help without much incentive. Show the user how their feedback could impact the design. Make them feel like they have a say in something important. This also works really well for internal studies on enterprise applications. Employees like having a say in the software they use everyday. Samantha . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Posted from the new ixda.org http://www.ixda.org/discuss?post=38251 Welcome to the Interaction Design Association (IxDA)! To post to this list ... disc...@ixda.org Unsubscribe http://www.ixda.org/unsubscribe List Guidelines http://www.ixda.org/guidelines List Help .. http://www.ixda.org/help
