Thanks everyone for taking the time to dig into this issue. I've done more testing and it turns out that my initial analysis was wrong. There are no threads magically appearing and disappearing (it was all caused by the monitoring tools I used). It rather seems that there is an issue with terminating the test threads - I've added a lot of logging to the original test and was able to observe that sometimes the new test threads were started before the terminating test threads have disappeared.
So I've added more rigorous check for the threads termination - checking the thread states instead of just comparing the thread counts. By doing this I was able to decrease the chances of failing but it still seems that there is some discrepancy between the numbers reported by the mbean and eg. the result of Thread.getAllStackTraces(). I am logging all the threads reported by Thread.getAllStackTraces() before the call to mbean.getThreadCount() and after the call and sometimes it just happens that mbean.getThreadCount() reports the thread count which is off by 1 in regards to both Thread.getAllStackTraces() calls. I will try the "thread.join()" suggestion from Daniel. -JB-
