On 10/1/19 2:44 PM, Max Reitz wrote:
> On 28.09.19 01:35, John Snow wrote:
>>
>>
>> On 9/23/19 9:09 AM, Max Reitz wrote:
>>> On 18.09.19 01:45, John Snow wrote:
>>>> verify_platform will check an explicit whitelist and blacklist instead.
>>>> The default will now be assumed to be allowed to run anywhere.
>>>>
>>>> For tests that do not specify their platforms explicitly, this has the
>>>> effect of
>>>> enabling these tests on non-linux platforms. For tests that always
>>>> specified
>>>> linux explicitly, there is no change.
>>>>
>>>> For Python tests on FreeBSD at least; only seven python tests fail:
>>>> 045 147 149 169 194 199 211
>>>>
>>>> 045 and 149 appear to be misconfigurations,
>>>> 147 and 194 are the AF_UNIX path too long error,
>>>> 169 and 199 are bitmap migration bugs, and
>>>> 211 is a bug that shows up on Linux platforms, too.
>>>>
>>>> This is at least good evidence that these tests are not Linux-only. If
>>>> they aren't suitable for other platforms, they should be disabled on a
>>>> per-platform basis as appropriate.
>>>>
>>>> Therefore, let's switch these on and deal with the failures.
>>>
>>> What exactly do you mean by “deal with the failures”? Do you have a
>>> reference to patches that deal with them, or are you or is someone else
>>> working on them...?
>>>
>>> Apart from that, I am rather hesitant to take a patch through my tree
>>> that not only may cause test failures on platforms that I will not or
>>> actually cannot run tests on (like MacOS or Windows), but that actually
>>> does introduce new failures as you describe.
>>>
>>> Well, at least it doesn’t introduce build failures because it appears
>>> there is no Python test that’s in the auto group, so I suppose “rather
>>> hesitant” is not an “I won’t”.
>>>
>>
>> Think of it more like this: The failures were always there, but we hid
>> them. I'm not "introducing new failures" as such O:-)
>
> That is incorrect.
>
> As I have said, the conceptual problem is that the iotests now run as
> part of make check. As such, allowing auto tests to run on non-Linux
> platforms may introduce build failures that I cannot do anything about.
>
> And those are very much new failures.
>
>> I think that I have demonstrated sufficiently that it's not correct to
>> prohibit python tests from running on other platforms wholesale, so I'd
>> prefer we don't do that anymore.
>
> You have not.
>
I feel as if I have.
The tests can run on other platforms and I proved that most of them
work. There's not good evidence that any of these tests are indeed
"Linux-only", as only a handful experience problems not more easily
explained by other factors. There's nothing inherent to the framework
itself that makes it Linux-only.
I think it's more than accurate to say that "it's not correct to
prohibit python tests from running on other platforms wholesale."
Maybe you want to talk about whether or not we should gate on these
tests on those platforms, and that's fair, but it's not what got written.
> The actual argument to convince me is “This does not affect any tests in
> the auto group, so it will not introduce build failures at this time”.
>
>> Further, iotests on FreeBSD already weren't 100% green, so I'm not
>> causing a regression in that sense, either.
>
> My problem is twofold:
>
> (1) You claim “Sure, there are failures, but let’s just deal with them”
> and then you do not deal with them. Seems wrong to me.
>
> I’m fine with the argument “Sorry, royal ‘we’. But it just doesn’t help
> anyone to hide the errors. If someone’s on BSD and wants to run the
> iotests, let them.”
>
> That sounds good to me.
>
That's more or less what I want to convey. Enable them and see where the
chips fall, but don't gate pull requests on non-Linux platforms.
In the event that an "auto" group test did fail on an esoteric platform,
I wanted to offer being the point of contact for that so I wasn't
foisting work onto you.
"Royal 'we', but with the expectation that I'd likely just re-blacklist
certain tests we don't have the capacity to support."
I felt like the burden wouldn't be too severe there, as all of the
'auto' tests appeared to pass without much of a fuss.
> (2) Maybe someone adds a Python test in the future that is in auto and
> that does not specify Linux as the only supported platform. Then I send
> a pull request and it breaks on macOS. Now what? Remove it from auto?
> Blindly put "macOS" in unsupported platforms?
>
> In any case, it’ll be a problem for no good reason.
>
I don't agree with the sentiment that it's "no good reason".
We claim to support these platforms. If tests fail on them, I think we
rather want to know.
The fear, I think, is that it will be mostly boring platform differences
that are unimportant to the actual functioning of QEMU: that it's just
good at finding bugs in the test instead of the binary.
I think that's a legitimate concern, but it doesn't move me enough to
agree that we shouldn't run tests on platforms we claim to support.
Not that I wanted to wage this war anyway. I just wanted to fix some
Python logging.
> More on that in the next chunk.
>
>> I'm going to meekly push and ask that we stage this as-is, and when
>> something bad happens you can remind me that I wanted this and make me
>> do it.
>
> Make you do what? Deal with the fact that a pull request is rejected
> because a test fails on macOS?
>
> This is precisely the kind of problem I already had with adding the
> iotests to make check, and I’m already very much not happy about it.
> (In that case it was $DISPLAY not being set on Peter’s test system.)
>
Make me "deal with it", is what I meant. I am willing to own
multiplatform problems for the block layer, but I'm also constrained by
some other simple facts like "I don't have a Mac OSX machine."
I am having a difficult time advocating for the idea that we have
supported platforms on which we don't actually test. If we disable tests
for those platforms, I think it's not right to say those platforms are
supported.
I will say, though, you make an extremely convincing case for the idea
that, actually, our only supported platform is Linux.
>
> I’ll let you make the deduction of “The problem isn’t allowing the
> iotests to run on non-Linux platforms, but the fact that they run in
> make check” yourself, so that I no longer feel like I’m the only one who
> considers that having been a mistake.
>
I'd be OK with not running them in "make check" on non-Linux anymore,
because it matches my feelings on them not being robust enough to gate
on for that platform.
It looks like we're just going to drop the idea entirely, though. As
long as the iotests are getting run regularly I will be happy anyway,
and I suppose we're looking at Travis for that now.
With that in mind, it's probably fine to remove the restrictions here, I
hope. The alternative is working it back the other way and adding new
restrictions to existing tests to force it back to Linux-only, and I'd
prefer not to do that.
https://i.kym-cdn.com/photos/images/original/001/365/753/94c.jpg
--js
