> > So shouldn't this test just be marked @headful ? > I can mark it headful if Sergey is ok with it.
This will skip the test in our nightly, but actually does not solve the existed bug in the test. > fails last time due to some update dialog or antivirus popup) > javax/swing/JButton/8151303/PressedIconTest.java 8198689 macosx-all* > javax/swing/JWindow/ShapedAndTranslucentWindows/ShapedPerPixelTranslucentGradient.java > >> I guess there will be many more tests that will fail if we try to > run > >> for 100 iterations and Problemlist will swell, so I guess we should > > >> keep only those tests in ProblemList that fail if we run > individually. The ProblemList excludes the tests from the nightly and testing after commit, the rest of tests(which are headless and not in this list) are quite stable. > > Automated tests that are being run on a regular basis need to pass > > reliably. > But we do not run each test in an batch for 100 iterations to find out > > if it is reliable (should we?). I do that, our sqe run the tests for 1000 iterations. > I guess we just run it individually in standalone mode and if it > pass > everytime, we consider it reliable, and all these 5 tests pass if run I guess the problem here is that these tests were added to the problem list without investigation of root cause, at least It was not checked that it was a setup issue like wrong "Display color profile". As a solution I suggest to check the source code of the tests, and try to find some issues which may affect the stability of the tests, then run these tests in a bash loop for 100 iterations to prove that they really stable.
