A "synthesist" (that's my scenario), who can't hear *anything* before the end of the pipe, would be very sensitive to jitter (the deviation, or "sloppyness", between triggers and actual sound) and would therefore require the smallest of small buffers.
Trading jitter for constant latency can work well for some, too.
Unfortunately the increasingly popular MIDI over USB connections already introduce a millisecond of jitter, so if you're already working at a blocksize of 64 at 44.1 or faster it's not worth the effort.
Cheers, Tim
