Another revision was required, available at: http://cr.openjdk.java.net/~ceisserer/fix12/webrev.04/
When the destination-area is quadratic, no geometry is stored inside the mask and therefore the composition-window is used for clipping and it's size has to be rounded properly. In cases where it is not quadrant transformed, the code makes now sure no pixels are clipped away unintentionally. Thanks for your patience, Clemens