On 2020-02-23 16:21 -0600, Bruce Dubbs via lfs-dev wrote:
> On 2/23/20 10:46 AM, Tadeus Prastowo via lfs-dev wrote:
> > On Sun, Feb 23, 2020 at 5:20 PM Xi Ruoyao via lfs-dev
> > <lfs-dev@lists.linuxfromscratch.org> wrote:
> > > We don't consider XPASS a failure. There are many XPASS in LFS packages
> > > (>20 in
> > > total, I think) and it does not make sense to document them one by one.
> > >
> > > I'd never seen a XPASS which is a "bad thing". Once a grep maintainer
> > > believed
> > > "XPASS is bad" so he made the entire test suite to FAIL if there was any
> > > XPASS.
> > > Then after a Glibc upgrading, a XPASS in grep forced us to use "-k" for
> > > grep
> > > "make check". It's stupid IMO - a test suite should not be a fragile
> > > "status
> > > change detector".
> >
> > Very interesting. So, XPASS here does not mean something that is
> > really expected to fail as in the unit test "test to open a lock with
> > the wrong key", which is expected to fail and with an XPASS really
> > means something goes wrong?
> If they expected it to do one thing and it didn't it would FAIL. If
> something normally fails, that is XFAIL. If it actually passes due to
> an update to a library or the kernel, then it would be an XPASS.
>
> Really the only thing to be concened about is a lot of FAILs. Quite a
> few packages have some small number of failures.
>
> -- Bruce
>
>
>
> > > We can add a discussion about the meaning of PASS/FAIL/XPASS/XFAIL in "4.6
> > > About
> > > the Test Suites", though.
> >
> > Seconded, and please include the case of "test to open a lock with the
> > wrong key" to define the semantics of XPASS. Thank you.
There is no such case. "test to open a lock with the wrong key" should be wrote
like:
void test_open_a_lock_with_the_wrong_key(struct test_driver *t)
{
struct lock l1, l2;
struct key k1, k2;
int ret1, ret2;
gen_key_lock_pair(&l1, &k1);
gen_key_lock_pair(&l2, &k2);
ret1 = try_open(&l1, &k2);
ret2 = try_open(&l2, &k1);
if (ret1 == FAIL_MISMATCH && ret2 == FAIL_MISMATCH)
t.pass(); /* PASS, not XFAIL */
else
t.fail(); /* FAIL, not XPASS */
}
XFAIL should never be used to indicate "this function call should return an
error". In that case you should check if the returned error code is expected.
If it's expected, this unit test should PASS, otherwise FAIL.
XFAIL is used to indicate that "we know there may be a bug but it's not fatal
and we can't fix it now (because of complications or buggy external
dependencies)".
--
Xi Ruoyao <xry...@mengyan1223.wang>
School of Aerospace Science and Technology, Xidian University
--
http://lists.linuxfromscratch.org/listinfo/lfs-dev
FAQ: http://www.linuxfromscratch.org/faq/
Unsubscribe: See the above information page
