On Mon, Feb 24, 2020 at 6:32 AM Xi Ruoyao via lfs-dev
<lfs-dev@lists.linuxfromscratch.org> wrote:
>
> On 2020-02-23 16:21 -0600, Bruce Dubbs via lfs-dev wrote:
> > On 2/23/20 10:46 AM, Tadeus Prastowo via lfs-dev wrote:
> > > On Sun, Feb 23, 2020 at 5:20 PM Xi Ruoyao via lfs-dev
> > > <lfs-dev@lists.linuxfromscratch.org> wrote:
> > > > We don't consider XPASS a failure. There are many XPASS in LFS packages
> > > > (>20 in
> > > > total, I think) and it does not make sense to document them one by one.
> > > >
> > > > I'd never seen a XPASS which is a "bad thing". Once a grep maintainer
> > > > believed
> > > > "XPASS is bad" so he made the entire test suite to FAIL if there was any
> > > > XPASS.
> > > > Then after a Glibc upgrading, a XPASS in grep forced us to use "-k" for
> > > > grep
> > > > "make check". It's stupid IMO - a test suite should not be a fragile
> > > > "status
> > > > change detector".
> > >
> > > Very interesting. So, XPASS here does not mean something that is
> > > really expected to fail as in the unit test "test to open a lock with
> > > the wrong key", which is expected to fail and with an XPASS really
> > > means something goes wrong?
>
> > If they expected it to do one thing and it didn't it would FAIL. If
> > something normally fails, that is XFAIL. If it actually passes due to
> > an update to a library or the kernel, then it would be an XPASS.
> >
> > Really the only thing to be concened about is a lot of FAILs. Quite a
> > few packages have some small number of failures.
> >
> > -- Bruce
> >
> >
> >
> > > > We can add a discussion about the meaning of PASS/FAIL/XPASS/XFAIL in
> > > > "4.6
> > > > About
> > > > the Test Suites", though.
> > >
> > > Seconded, and please include the case of "test to open a lock with the
> > > wrong key" to define the semantics of XPASS. Thank you.
>
> There is no such case. "test to open a lock with the wrong key" should be
> wrote
> like:
>
> void test_open_a_lock_with_the_wrong_key(struct test_driver *t)
> {
> struct lock l1, l2;
> struct key k1, k2;
> int ret1, ret2;
>
> gen_key_lock_pair(&l1, &k1);
> gen_key_lock_pair(&l2, &k2);
>
> ret1 = try_open(&l1, &k2);
> ret2 = try_open(&l2, &k1);
> if (ret1 == FAIL_MISMATCH && ret2 == FAIL_MISMATCH)
> t.pass(); /* PASS, not XFAIL */
> else
> t.fail(); /* FAIL, not XPASS */
> }
>
> XFAIL should never be used to indicate "this function call should return an
> error". In that case you should check if the returned error code is expected.
> If it's expected, this unit test should PASS, otherwise FAIL.
>
> XFAIL is used to indicate that "we know there may be a bug but it's not fatal
> and we can't fix it now (because of complications or buggy external
> dependencies)".
> --
> Xi Ruoyao <xry...@mengyan1223.wang>
> School of Aerospace Science and Technology, Xidian University
Thank you very much for taking the time to explain the concept of
XPASS. I really appreciate it.
It would indeed be great if the information could be made available in
the LFS book. Thank you.
--
Best regards,
Tadeus
--
http://lists.linuxfromscratch.org/listinfo/lfs-dev
FAQ: http://www.linuxfromscratch.org/faq/
Unsubscribe: See the above information page
