Hi, From: Subrata Modak <[email protected]> Subject: Re: [LTP] [PATCH] Disable Broken Test Cases Date: Thu, 12 Mar 2009 17:06:30 +0530
> On Thu, 2009-03-12 at 05:10 +0800, CAI Qian wrote: >> Hi, >> >> From: Subrata Modak <[email protected]> >> Subject: Re: [LTP] [PATCH] Disable Broken Test Cases >> Date: Mon, 9 Mar 2009 15:44:45 +0530 >> >> > Hi, >> > >> > On Sun, Mar 8, 2009 at 9:59 PM, CAI Qian <[email protected]> wrote: >> > >> >> Hi, >> >> >> >> This patch temporarily disables broken test cases, >> >> >> >> nanosleep01 >> >> nanosleep02 >> >> pselect01 >> >> pselect01_64 >> >> >> >> because the ways to calculate the syscall execution time looks like >> >> invalid. For example, in pselect01 test case, >> >> >> >> start = time(&t); >> >> retval = pselect(0, &readfds, NULL, NULL, (struct timespec *)&tv,NULL); >> >> end = time(&t); >> >> >> >> /* Changed total_sec compare to an at least vs an exact compare */ >> >> >> >> if(((end - start) >= total_sec) && ((end - start) <= total_sec + 1)) >> >> tst_resm(TPASS,"Sleep time was correct"); >> >> else >> >> tst_resm(TFAIL,"Sleep time was incorrect:%d != %d",total_sec,(end - >> >> start)); >> >> } >> >> >> >> The CPU scheduler can suspend the execution of the program just after >> >> time(), but before pselect(). As the result, (end - start) is not the > > Yes, that can definitely happen. And even a suspend can happen after > pselect() and before then next call to time(). > >> >> actual syscall execution time, which introduces a false test failure >> >> here. The same problem exists in nanosleep01 and nanosleep02 as well, > > Then, what is the best way to determine the exact time taken by a > syscall to execute ?? > I don't know. Looks like "strace" is using a similar way to measure system call. As Jiri said it is unreliable, but probably the best we can do at the moment. > A part of this fix was introduced by Craig Meier. Noting that even there > might have been 100 interruptions between the first and last time() > call, still (end - begin) is within (actual time to execute the syscall > + 1). If this time exceeds, means something notably wrong with the > scheduler to schedule this process taking notably context switching. > In theory, you are correct. However, the real world situation is more complicated. Apart from the situation Jiri mentioned, I have also seen those tests failed when running with some background load. I'll add some comments in those tests to warn people for those situations, so hopefully can save them a little bit time to debug the failures. CAI Qian > Regards-- > Subrata > >> >> >> >> /* Note down the current time */ >> >> gettimeofday(&otime, 0); >> >> /* >> >> * Call nanosleep() to suspend child process >> >> * for specified time. >> >> */ >> >> TEST(nanosleep(&timereq, NULL)); >> >> >> >> /* time after child resumes execution */ >> >> gettimeofday(&ntime, 0); >> >> >> >> >> >> Signed-off-by: CAI Qian <[email protected]> >> > >> > >> > I am not sure whether it would be correct to disable tests when they are >> > broken (supposed to be). The inherent reason being that once a/some >> > test/tests are disabled, they just go out of the perview of the testers >> > memory, as, he/she can no more see the test results of the same tests. >> > Already overburdened to handle/analyze so many tests results, he simply >> > forgets that such-and-such tests - disabled some time ago due to broken >> > issue(s) - needs to be fixed. >> > >> > So, i would prefer to keep them, so that it occasionally pricks into >> > somebody(s) eye(s), and he/she is generous enough to give a fix someday in >> > future. >> > >> >> Sorry, I can't agree with you. Those tests are broken by design. It >> wastes everybody's time by using broken tests, and then need to figure >> it out why failed. Why do we release broken tests. I understand your >> point by not disable them, so hopefully people may think of fixing it >> some day. However, you can achieve the same by put those tests to the >> list of known broken tests or TODO list somewhere obvious like in top >> directory or homepage. I am afraid I don't know how to fix those test >> cases at the moment. >> >> Considering a little bit further, I guess stable/unstable branches might >> be a way to go, so we can add new or working-in-progress tests to >> unstable branches until they have enough time to be mature. We are >> maintaining LTP inside Red Hat as well at the moment. There are some >> patches like those to disable some working-in-progress tests, so it will >> give us more reliable testing results, and then we can actually use it >> for regression testing of kernels. I am looking for what is the best to >> merge those patches upstream, so we can avoid further gaps between two >> repositories. >> >> CAI Qian >> >> > Regards-- >> > Subrata >> > >> > >> >> >> >> --- runtest/syscalls.orig 2009-03-09 00:04:17.000000000 +0800 >> >> +++ runtest/syscalls 2009-03-09 00:15:49.000000000 +0800 >> >> @@ -622,8 +622,10 @@ >> >> munmap02 munmap02 >> >> munmap03 munmap03 >> >> >> >> -nanosleep01 nanosleep01 >> >> -nanosleep02 nanosleep02 >> >> +# The following test cases are broken by design with the syscall time >> >> +# calculations. >> >> +#nanosleep01 nanosleep01 >> >> +#nanosleep02 nanosleep02 >> >> nanosleep03 nanosleep03 >> >> nanosleep04 nanosleep04 >> >> >> >> @@ -699,8 +701,10 @@ >> >> >> >> profil01 profil01 >> >> >> >> -pselect01 pselect01 >> >> -pselect01_64 pselect01_64 >> >> +# The following test cases are broken by design with the syscall time >> >> +# calculations. >> >> +#pselect01 pselect01 >> >> +#pselect01_64 pselect01_64 >> >> >> >> ptrace01 ptrace01 >> >> ptrace02 ptrace02 >> >> >> >> >> >> ------------------------------------------------------------------------------ >> >> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, >> >> CA >> >> -OSBC tackles the biggest issue in open source: Open Sourcing the >> >> Enterprise >> >> -Strategies to boost innovation and cut costs with open source >> >> participation >> >> -Receive a $600 discount off the registration fee with the source code: >> >> SFAD >> >> http://p.sf.net/sfu/XcvMzF8H >> >> _______________________________________________ >> >> Ltp-list mailing list >> >> [email protected] >> >> https://lists.sourceforge.net/lists/listinfo/ltp-list >> >> >> > >> > >> > >> > -- >> > Regards & Thanks-- >> > Subrata >> >> ------------------------------------------------------------------------------ >> Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are >> powering Web 2.0 with engaging, cross-platform capabilities. Quickly and >> easily build your RIAs with Flex Builder, the Eclipse(TM)based development >> software that enables intelligent coding and step-through debugging. >> Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com >> _______________________________________________ >> Ltp-list mailing list >> [email protected] >> https://lists.sourceforge.net/lists/listinfo/ltp-list > ------------------------------------------------------------------------------ Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are powering Web 2.0 with engaging, cross-platform capabilities. Quickly and easily build your RIAs with Flex Builder, the Eclipse(TM)based development software that enables intelligent coding and step-through debugging. Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com _______________________________________________ Ltp-list mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ltp-list
