azinck3 wrote: > Very interesting results. Thank you for the time and care you spent > doing this. > > I am not a researcher, just an amateur like yourself :), but I have a > few methodological concerns: > > 1) You bundled the files into two complete bundles (all mp3s were in > group A, all lossless in B). In doing so it becomes much more difficult > to draw any conclusions about the detectability/preference of mp3 > encoding as it pertains to any one pair of files. It also allows people > to focus intently only on the files they care to listen to, or those > which they came across first, and then to make conclusions about the > whole group. Additionally, it prevents you from changing the order of > the files (selection of group A by most respondents may have been > primacy effect, to some degree). Asking about confidence on a per-file > basis as you did would seem to mitigate some of these concerns but I > don't think it fully addresses the potential problems. > > 2) There's no control group. This could have helped identify any > possible the primacy effect, too (if you'd had each individual file > pairing done independently it would have been possible to have two > identical files as one of the pairs). > > 3) Your main question "which set sounded inferior" had 3 answers: A, B, > or "no audible difference". This, paired with the question about > confidence does a decent job of answering the question "which sounds > better" but I wonder if it does a good enough job of answering the > question "is mp3 distinguishable from lossless". There could be a > subset of people who had a hard time developing an opinion on which one > sounded inferior, but an easy time distinguishing between the files. > These people would not have wanted to answer "no audible difference" so > may have taken a guess for the question "which set sounded inferior", > but for "how difficult was it to come to your conclusion" they might > have said "easy" (since it was easy for them to distinguish the files). > Maybe I'm splitting hairs here; these are just some thoughts that came > to mind while reading the results. > > 4) I also wonder about the decision to use such an unorthodox mp3 > encoding technique. I understand your rationale for doing so, but in > the end it seems that your conclusion necessarily becomes "people tend > to prefer this unusual method of audio processing over the original > files". If you'd used a more typical encoding method then your > conclusions could be more useful by applying more broadly to mp3s in the > "real" world. > > > These all sound like harsh criticisms. They're not. Your survey is, to > me, exactly the kind of stuff audio publications should be doing. You > clearly put a lot of though into this; it was a great read!
Thanks azinck, I agree with your points and appreciate you putting them down. In preparation for the test, I wanted to make sure that it was first and foremost "doable" in the sense of being simple enough to perform and not onerous for those wanting to partake. I was already a bit concerned about the 75MB file size for example. 1. Out of simplicity I bundled the songs together. I agree that people would likely only pick the ones they can bear to listen to! This was why 3 songs were provided spanning a few genres... I was also worried about confusion and error if I were to mix-and-match especially "out in the wild" where the respondents can become confused if they had to respond with something like "I liked Church_A, Time_B, KeineZeit_A". 2. Don't know how I could have done this unless I provided at least 2 ZIP files; 1 being just MP3 A&B or lossless A&B to guage the serial position effect (primacy effect vs. recency effect). Again, we'd be looking at more complexity and the survey would have to allow people to identify which test they downloaded... 3. & 4. Good points and I think the most important criticisms of the methodology. Interestingly, nobody said the test was "easy" but voted for "no difference" (good that didn't happen! :-). Yes the unorthodox MP3 encoding was what bothered me the most but I considered it a "necessary evil"! However, if it's that easy to create a preference for MP3, that's meaningful as well! ------------------------------------------------------------------------ Archimago's Profile: http://forums.slimdevices.com/member.php?userid=2207 View this thread: http://forums.slimdevices.com/showthread.php?t=98010 _______________________________________________ audiophiles mailing list [email protected] http://lists.slimdevices.com/mailman/listinfo/audiophiles
