Based on an idea of jhMikeS I modified svn code (i2s-pp.c) in the following way:
// IISFIFO_CFG |= 0x33; /* 12 slots full/empty config */
/* Bit2-3, Bit6-7 seem to have no meaning */
IISFIFO_CFG |= 0x22; /* 8 slots full/empty config */
which solves the channel swapping issues. The modification triggers the
interrupt already when 8 slots are empty, giving the system additional 45.3usec
to execute the fiq without the FIFO running empty. With current frequency
setting the cache releases the bus in time to execute the next fiq. The
drawback is, that 50% more fiq calls are issued. But the fiq handler itself has
some performance tuning potential.