Hi,
On Thursday 12 March 2009 12:36:30 Subrata Modak wrote:
> On Thu, 2009-03-12 at 05:10 +0800, CAI Qian wrote:
> > From: Subrata Modak <[email protected]>
> > Subject: Re: [LTP] [PATCH] Disable Broken Test Cases
> > Date: Mon, 9 Mar 2009 15:44:45 +0530
> >
> > > On Sun, Mar 8, 2009 at 9:59 PM, CAI Qian <[email protected]> wrote:
> > >
> > >> This patch temporarily disables broken test cases,
> > >>
> > >> nanosleep01
> > >> nanosleep02
> > >> pselect01
> > >> pselect01_64
> > >>
> > >> because the ways to calculate the syscall execution time looks like
> > >> invalid. For example, in pselect01 test case,
> > >>
> > >> start = time(&t);
> > >> retval = pselect(0, &readfds, NULL, NULL, (struct timespec *)&tv,NULL);
> > >> end = time(&t);
> > >>
> > >> /* Changed total_sec compare to an at least vs an exact compare */
> > >>
> > >> if(((end - start) >= total_sec) && ((end - start) <= total_sec + 1))
> > >> tst_resm(TPASS,"Sleep time was correct");
> > >> else
> > >> tst_resm(TFAIL,"Sleep time was incorrect:%d != %d",total_sec,(end -
> > >> start));
> > >> }
> > >>
> > >> The CPU scheduler can suspend the execution of the program just after
> > >> time(), but before pselect(). As the result, (end - start) is not the
>
> Yes, that can definitely happen. And even a suspend can happen after
> pselect() and before then next call to time().
I think these scenarios are rather improbable, because it would mean the
process got extremely short timeslice. I think it's much more likely that the
process is woken up (eg. state=S), but then waits for the scheduler to actually
run it.
> > >> actual syscall execution time, which introduces a false test failure
> > >> here. The same problem exists in nanosleep01 and nanosleep02 as well,
>
> Then, what is the best way to determine the exact time taken by a
> syscall to execute ??
I think there isn't any. Running the process with realtime priority might help
a lot, but still not guarantee success.
Still, I think it's better to have unreliable tests for a given requirement,
than none at all.
> A part of this fix was introduced by Craig Meier. Noting that even there
> might have been 100 interruptions between the first and last time()
> call, still (end - begin) is within (actual time to execute the syscall
> + 1). If this time exceeds, means something notably wrong with the
> scheduler to schedule this process taking notably context switching.
It might happen eg. if you run it concurrently with hackbench or msgctl11,
which are kind of fork bombs. When msgctl11 runs on my system, there are
moments when there are like 2000 processes, 1000 running, 100 sleeping, 900
zombie. So you can guess what is the responsiveness under these conditions.
Regards
Jiri Palecek
> > >>
> > >> /* Note down the current time */
> > >> gettimeofday(&otime, 0);
> > >> /*
> > >> * Call nanosleep() to suspend child process
> > >> * for specified time.
> > >> */
> > >> TEST(nanosleep(&timereq, NULL));
> > >>
> > >> /* time after child resumes execution */
> > >> gettimeofday(&ntime, 0);
> > >>
> > >>
> > >> Signed-off-by: CAI Qian <[email protected]>
> > >
> > >
> > > I am not sure whether it would be correct to disable tests when they are
> > > broken (supposed to be). The inherent reason being that once a/some
> > > test/tests are disabled, they just go out of the perview of the testers
> > > memory, as, he/she can no more see the test results of the same tests.
> > > Already overburdened to handle/analyze so many tests results, he simply
> > > forgets that such-and-such tests - disabled some time ago due to broken
> > > issue(s) - needs to be fixed.
> > >
> > > So, i would prefer to keep them, so that it occasionally pricks into
> > > somebody(s) eye(s), and he/she is generous enough to give a fix someday in
> > > future.
> > >
> >
> > Sorry, I can't agree with you. Those tests are broken by design. It
> > wastes everybody's time by using broken tests, and then need to figure
> > it out why failed. Why do we release broken tests. I understand your
> > point by not disable them, so hopefully people may think of fixing it
> > some day. However, you can achieve the same by put those tests to the
> > list of known broken tests or TODO list somewhere obvious like in top
> > directory or homepage. I am afraid I don't know how to fix those test
> > cases at the moment.
> >
> > Considering a little bit further, I guess stable/unstable branches might
> > be a way to go, so we can add new or working-in-progress tests to
> > unstable branches until they have enough time to be mature. We are
> > maintaining LTP inside Red Hat as well at the moment. There are some
> > patches like those to disable some working-in-progress tests, so it will
> > give us more reliable testing results, and then we can actually use it
> > for regression testing of kernels. I am looking for what is the best to
> > merge those patches upstream, so we can avoid further gaps between two
> > repositories.
> >
> > CAI Qian
> >
> > > Regards--
> > > Subrata
> > >
> > >
> > >>
> > >> --- runtest/syscalls.orig 2009-03-09 00:04:17.000000000 +0800
> > >> +++ runtest/syscalls 2009-03-09 00:15:49.000000000 +0800
> > >> @@ -622,8 +622,10 @@
> > >> munmap02 munmap02
> > >> munmap03 munmap03
> > >>
> > >> -nanosleep01 nanosleep01
> > >> -nanosleep02 nanosleep02
> > >> +# The following test cases are broken by design with the syscall time
> > >> +# calculations.
> > >> +#nanosleep01 nanosleep01
> > >> +#nanosleep02 nanosleep02
> > >> nanosleep03 nanosleep03
> > >> nanosleep04 nanosleep04
> > >>
> > >> @@ -699,8 +701,10 @@
> > >>
> > >> profil01 profil01
> > >>
> > >> -pselect01 pselect01
> > >> -pselect01_64 pselect01_64
> > >> +# The following test cases are broken by design with the syscall time
> > >> +# calculations.
> > >> +#pselect01 pselect01
> > >> +#pselect01_64 pselect01_64
> > >>
> > >> ptrace01 ptrace01
> > >> ptrace02 ptrace02
> > >>
> > >>
> > >> ------------------------------------------------------------------------------
> > >> Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco,
> > >> CA
> > >> -OSBC tackles the biggest issue in open source: Open Sourcing the
> > >> Enterprise
> > >> -Strategies to boost innovation and cut costs with open source
> > >> participation
> > >> -Receive a $600 discount off the registration fee with the source code:
> > >> SFAD
> > >> http://p.sf.net/sfu/XcvMzF8H
> > >> _______________________________________________
> > >> Ltp-list mailing list
> > >> [email protected]
> > >> https://lists.sourceforge.net/lists/listinfo/ltp-list
> > >>
> > >
> > >
> > >
> > > --
> > > Regards & Thanks--
> > > Subrata
> >
> > ------------------------------------------------------------------------------
> > Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
> > powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
> > easily build your RIAs with Flex Builder, the Eclipse(TM)based development
> > software that enables intelligent coding and step-through debugging.
> > Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
> > _______________________________________________
> > Ltp-list mailing list
> > [email protected]
> > https://lists.sourceforge.net/lists/listinfo/ltp-list
>
>
>
>
------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list
