Jared, As one of the people developing one of the other tools on the market, webnographer <http://webnographer.com> , I will point out some corrections to the points that you raise.
The first point is that there is a remote data collection tool, and a remote method. Collecting the data is only the first part. The data collecting part is easy, getting the method right is hard, and it has taken us a long time to get right. For the data gathering part there is a long history of academic projects that use the concept of using a proxy and injecting Javascript. It was first done with WebQuilt <http://portal.acm.org/citation.cfm?doid=502115.502118>application at Berkeley and then by UsaProxy <http://fnuked.de/usaproxy/publications.htm> from the University of Munich. I think it may be a bit voodoo like to say that "I have seen one, I know them all.". When we have run tests remote vs lab we have managed to get smiler findings from both, and according to our clients who tell us that the cost runs at about 1/3rd of a traditional lab based test. For the example you gave, the differences in findings is due to this difference participant's knowledge and intention, not the difference in methods used! With our tool we often carry out tests using real users that are about to carry out a purchase. You cannot compare two methods if participants don't have the exact scenario - and demographic, and for your example the key difference was participant's knowledge and intention. The Lab users in your test have been thinking about buying a computer for at least a week, while remote users did not. This explains the difference in your findings. You mention the challenge of participant recruitment. Remote is easier. The participant can do the study from home, therefore they do need such a high incentive, as when they do a test from a lab. But getting the right participant is one of the most overlooked areas for both lab, and remote. We find out that many issues are down to the configuration of the users machine. These issues you can not discover in a lab, where the machine is normally the same. The sort of issues we have found are down to Screen Size, having a pdf reader installed, and browser issues. We also find many issues that are cultural. How many web sites have an International Scope, but are only tested with participants in one town, let alone throughout a whole country. You mention that you are limited in the tasks your participants can perform on remote. This is not the case with Webnograpther. Our system does not require that every user take the same path through the system. James http://blog.feralabs.com PS: If you want to debate which method is more voodoo I am up for a public Oxford style debate in Brighton when you are in the UK in July. 2009/4/8 Jared Spool <[email protected]> > > On Apr 7, 2009, at 11:04 PM, Toby Biddle wrote: > > A new online, unmoderated user testing tool has recently launched - >> www.loop11.com. Anyone used it have any thoughts? >> > > We haven't used this tool in particular, but, from their site, it looks > similar to a slew of other tools on the market. > > These tools are limited in value because of four key factors: > > 1) The pool of invited participants is critically important. In Loop11, it > seems you have to invite your own pool , which means you have to use > standard recruitment techniques to source, schedule, and incent participants > in the study. This will probably triple (or more) the costs. (Many > unmoderated tools offer their own pre-recruited pools, which keeps costs > down, but are often low quality participants, such as people who only > participate to get the incentive and don't really use the design.) > > 2) You are limited in the tasks your participants can perform. For the > software to work, the site has to know when a task is completed. For > example, when evaluating a travel site, you have to know what page the user > will end up on. If the confirmation page for a trip booking is computer > generated, this might not be possible. Even if it is, can the system tell if > all the values were properly entered? > > 3) We know from our research at UIE that participants who are actually > interested in the task (for example, currently planning a vacation in Paris) > will behave substantially differently than those who are asked to pretend to > do a task. They take more time, are more discriminating on the results, are > more likely to be frustrated when key information is missing, and are more > likely to be delighted when the design meets their needs. Yet, these systems > usually require that every user take the same path through the system, which > means recruiting people with identical interests (every participant has to > be actively planning their vacation to Paris and desiring the same dates & > hotel requirements). > > 4) The site reports standard analytic measures: time on task, "fail pages", > common navigation paths. But it's extremely difficult to come to the correct > inference based on these measures. For example, does longer time-on-task or > time-on-page imply frustration or interest? Does a deviation from the common > navigation path imply clicking on the wrong element or curious exploration > of additional features? Without talking to the individual, it's hard to even > know if a reported measure is good or bad, let alone the action the team > should take based on the reported result. > > In the ten years since I first started seeing these tools on the market. > I've never seen results from a study that the team could actually interpret > and act on. In one study a few years back with a major electronics retailer, > we conducted an in-lab study with 18 highly-qualified participants that was > comparable to a 60-participant Netraker (a Loop11 competitor from the past). > The task was to find the laptop computer of your dreams and put it in the > cart. > > In our study, all 18 participants were in the market to buy laptops, had > spent at least a week thinking about the laptop they wanted and its > requirements, and were given the cash to make the purchase (they would keep > the laptop after the study). In the Netraker study, they 60 randomly > selected participants from a panel of thousands who reportedly were in the > demographic groups of the site (unverfiable) and hadn't thought about laptop > purchases until the instructions for the test had popped up. > > In the Netraker results, 94% of the participants completed the tasks and > the average time was 1m 18s. In our study, only 33% of the participants > completed the task and the average time was 18 minutes. > > Why do you think there were such striking differences? Which study would > you pay more attention to? > > Beware of VooDoo measurement techniques. > > Hope that helps, > > Jared > > Jared M. Spool > User Interface Engineering > 510 Turnpike St., Suite 102, North Andover, MA 01845 > e: [email protected] p: +1 978 327 5561 > http://uie.com Blog: http://uie.com/brainsparks Twitter: jmspool > UIE Web App Summit, 4/19-4/22: http://webappsummit.com > > ________________________________________________________________ > Welcome to the Interaction Design Association (IxDA)! > To post to this list ....... [email protected] > Unsubscribe ................ http://www.ixda.org/unsubscribe > List Guidelines ............ http://www.ixda.org/guidelines > List Help .................. http://www.ixda.org/help > ________________________________________________________________ Welcome to the Interaction Design Association (IxDA)! To post to this list ....... [email protected] Unsubscribe ................ http://www.ixda.org/unsubscribe List Guidelines ............ http://www.ixda.org/guidelines List Help .................. http://www.ixda.org/help
