Hi, From: Subrata Modak <[email protected]> Subject: Re: [LTP] [PATCH] Disable Broken Test Cases Date: Mon, 9 Mar 2009 15:44:45 +0530
> Hi, > > On Sun, Mar 8, 2009 at 9:59 PM, CAI Qian <[email protected]> wrote: > >> Hi, >> >> This patch temporarily disables broken test cases, >> >> nanosleep01 >> nanosleep02 >> pselect01 >> pselect01_64 >> >> because the ways to calculate the syscall execution time looks like >> invalid. For example, in pselect01 test case, >> >> start = time(&t); >> retval = pselect(0, &readfds, NULL, NULL, (struct timespec *)&tv,NULL); >> end = time(&t); >> >> /* Changed total_sec compare to an at least vs an exact compare */ >> >> if(((end - start) >= total_sec) && ((end - start) <= total_sec + 1)) >> tst_resm(TPASS,"Sleep time was correct"); >> else >> tst_resm(TFAIL,"Sleep time was incorrect:%d != %d",total_sec,(end - >> start)); >> } >> >> The CPU scheduler can suspend the execution of the program just after >> time(), but before pselect(). As the result, (end - start) is not the >> actual syscall execution time, which introduces a false test failure >> here. The same problem exists in nanosleep01 and nanosleep02 as well, >> >> /* Note down the current time */ >> gettimeofday(&otime, 0); >> /* >> * Call nanosleep() to suspend child process >> * for specified time. >> */ >> TEST(nanosleep(&timereq, NULL)); >> >> /* time after child resumes execution */ >> gettimeofday(&ntime, 0); >> >> >> Signed-off-by: CAI Qian <[email protected]> > > > I am not sure whether it would be correct to disable tests when they are > broken (supposed to be). The inherent reason being that once a/some > test/tests are disabled, they just go out of the perview of the testers > memory, as, he/she can no more see the test results of the same tests. > Already overburdened to handle/analyze so many tests results, he simply > forgets that such-and-such tests - disabled some time ago due to broken > issue(s) - needs to be fixed. > > So, i would prefer to keep them, so that it occasionally pricks into > somebody(s) eye(s), and he/she is generous enough to give a fix someday in > future. > Sorry, I can't agree with you. Those tests are broken by design. It wastes everybody's time by using broken tests, and then need to figure it out why failed. Why do we release broken tests. I understand your point by not disable them, so hopefully people may think of fixing it some day. However, you can achieve the same by put those tests to the list of known broken tests or TODO list somewhere obvious like in top directory or homepage. I am afraid I don't know how to fix those test cases at the moment. Considering a little bit further, I guess stable/unstable branches might be a way to go, so we can add new or working-in-progress tests to unstable branches until they have enough time to be mature. We are maintaining LTP inside Red Hat as well at the moment. There are some patches like those to disable some working-in-progress tests, so it will give us more reliable testing results, and then we can actually use it for regression testing of kernels. I am looking for what is the best to merge those patches upstream, so we can avoid further gaps between two repositories. CAI Qian > Regards-- > Subrata > > >> >> --- runtest/syscalls.orig 2009-03-09 00:04:17.000000000 +0800 >> +++ runtest/syscalls 2009-03-09 00:15:49.000000000 +0800 >> @@ -622,8 +622,10 @@ >> munmap02 munmap02 >> munmap03 munmap03 >> >> -nanosleep01 nanosleep01 >> -nanosleep02 nanosleep02 >> +# The following test cases are broken by design with the syscall time >> +# calculations. >> +#nanosleep01 nanosleep01 >> +#nanosleep02 nanosleep02 >> nanosleep03 nanosleep03 >> nanosleep04 nanosleep04 >> >> @@ -699,8 +701,10 @@ >> >> profil01 profil01 >> >> -pselect01 pselect01 >> -pselect01_64 pselect01_64 >> +# The following test cases are broken by design with the syscall time >> +# calculations. >> +#pselect01 pselect01 >> +#pselect01_64 pselect01_64 >> >> ptrace01 ptrace01 >> ptrace02 ptrace02 >> >> >> ------------------------------------------------------------------------------ >> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, >> CA >> -OSBC tackles the biggest issue in open source: Open Sourcing the >> Enterprise >> -Strategies to boost innovation and cut costs with open source >> participation >> -Receive a $600 discount off the registration fee with the source code: >> SFAD >> http://p.sf.net/sfu/XcvMzF8H >> _______________________________________________ >> Ltp-list mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/ltp-list >> > > > > -- > Regards & Thanks-- > Subrata ------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ Ltp-list mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ltp-list
