I've been using the msharpen filter quite a bit lately. Yesterday, I tried to use the "mask" option, and the output was badly mangled. I didn't have time to do any testing, but I suspect it is incompatible with any resizing/rescaling options. The result looked like what you get when you force incorrect input geometry.
People targeting dvd output with high bitrate might be interested in trying "-J msharpen=strength=20".
