Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Tue, Sep 12, 2017 at 2:27 AM, Uwe Kleine-König  
wrote:
>> BTW, if I want to get a PPC64 machine for Linux testing purpose, is the
>> used apple G5 a good place to start?
>
> Honestly I don't know. https://wiki.debian.org/ppc64el tells
>
> Debian/ppc64el requires, at minimum, a POWER8 processor machine.
> Although Debian was initially bootstrapped on a POWER7 set of
> servers. this class of server is not supported anymore, and you
> are not able to run Debian/ppc64el on a POWER7 processor without
> hitting an illegal instruction fault.
>
> Hm https://en.wikipedia.org/wiki/POWER8 tells:
>
> Systems based on POWER8 became available from IBM in June
> 2014. Systems and POWER8 processor designs made by other
> OpenPOWER members was available in early 2015.
>
> So I think this rules out a G5.

Thanks for the tip. I am glad I asked before I pull the trigger.

> https://wiki.debian.org/ppc64el/Installation mentions you can run this
> under qemu however.

Good idea. I will give that a try too.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Wed, Sep 6, 2017 at 11:36 AM, Uwe Kleine-König  
wrote:
> There is https://dsa.debian.org/doc/guest-account/ which would give you
> the possibility to access some Debian machines. Other than that I intend

Sorry for the delay. Thanks for the pointer of the guest account.

> to upload 0.5.1 to Debian soon and then can provide you links to build
> failures in the build server farm :-) And if you have a patch, I can
> volunteer to make the test monkey for you.

BTW, if I want to get a PPC64 machine for Linux testing purpose, is the
used apple G5 a good place to start?

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Wed, Sep 6, 2017 at 10:44 AM, Uwe Kleine-König  
wrote:
>
> and while it's ok to test the core stuff and not wanting the system
> includes to interfere, there should also be tests that check "ordinary"
> userspace programs which naturally depend on the system headers.
>

There is one. The "selfcheck" target was checking sparse on its own
source file. That will definitively use the system header file. However,
there are some warning trigger in the system header file can't be fixed
in the sparse source code. It need to change the system header to make
some warning go away, or disable that warning.

>
> No, it's not solved. But given that it is somehow known that sparse
> (without cgcc) fails to work well on userspace stuff, I think the
> following would be fine for the Debian side:
>
>  a) let horst use cgcc instead of sparse
>  b) downgrade bug to important or even normal pointing out that cgcc
> should be used for userspace programs

That seems to be the right thing to do for now. That is until
sparse are smarter on user space header regarding architecture stuff.

> For sparse it would be cool to:
>
>  c) drop the #weak_define of __amd64__ to make this "problem" more
> apparent. (Assuming this doesn't break e.g. a kernel build.)

You mean remove define of "__x86_64__".

It will likely break some other stuff. For the record the "selfcheck" target
already using cgcc. We still need to fix the breakage.

Any suggestion how to test sparse running on other platform headfile
without have to get access to ppc64 for example? I think that is the biggest
obstacle right now. I can make some changes, but I don't have a good way
to test it other than x86 platform.

>  d) fix the test suite, at least mark the tests that are expected to
> fail on !x86 as such to make $(make check) succeed. (Otherwise I'd
> have to disable or ignore the testsuite which isn't that great.)

We can make the test-suite not depend on system header files.
That seems to be the right think to do. I also send out a patch
to let the llvm back end test-suite use cgcc last week. Removing
system header usage in test suite is better.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Sun, Sep 3, 2017 at 5:14 PM, Luc Van Oostenryck
 wrote:
>
> I really think that the testsuite should not depend on system or library
> header.

I think that is a good point. We can start cleaning up the system header
file dependency in the existing test suite. See how it goes.

>
> Otherwise, I'm not at all opposed to sparse being universal but I would like
> to note that things can become very quickly very very messy.
> For example, for the current problem here I understood that it was
> at least partially based on the lack of a definition of _CALL_ELF
> but do we need to define it to 1 or to 2, in other words, do we need
> to support the ELFv1 ABI or the ELFv2? GCC has some flags for this
> (-mabi=elfv[12]) but what default value do we want? ELFv1 is the default

I think we can just sparse default to as late as the latest release
version of gcc.

> for big-endian platform and ELFv2 for little-endian platform, so yes,
> we need a flag for the endianness but which endianness we want as default?

I am tempting to make the endianness the same as the host gcc by default.
Then it can be overwrite by architecture flags.

>
> Things become even more fun when taking in account the difference
> between GCC version. Do we want to be universal there too (and thus
> have some flags for to specify which gcc's version we want to mimick)?
> What about other compilers?

I purpose just sync to the latest gcc version (or a late enough version
we can agree on. e.g. the one that supported by kernel compile.) Sparse
current try to sync to the latest gcc attributes already.

> I think that part of the needed info can be auto-extracted from GCC
> when doing a native build. Using some sort of spec file or a .sparserc

Is there a way to do auto-extract? That would be a good starting point.

> I also note that currently, sparse is already largely universal *because*
> it *doesn't* need those platform details (or only the very minimal: word 
> size).

Sparse is not universal, it just support a very small sub set of the C
source file that haven't expose to those platform detail macros. Adding
those architecture macro support will not make it any less universal.
Might slow things down, that is some thing we need to watch out for.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Fri, Sep 1, 2017 at 3:02 AM, Josh Triplett  wrote:
>> First of all. It is not very trivial to teach sparse about the architecture
>> stuff. To my mind, we need to move all the cgcc logic into sparse.
>
> Related to that: while it would mean we couldn't necessarily just rely
> entirely on GCC's definitions for a target platform, I think in an ideal
> world we could have a sparse binary that understood *all* target
> platforms at once, such that you could ask Sparse on x86_64 to "compile"

Yes, that is what I want to have. It is list as one of the project in
project idea document as well. I have  a related question. How do we
test the different architecture handling without actually run sparse
on different platform? I am thinking maybe using gcc cross platform
compiler and compare some macro against the sparse one.

> as though targeting any arbitrary architecture. That would also have the
> major advantage of making it easy to run the Sparse testsuite for
> *every* target architecture without needing compilers for every such
> architecture.

Another way to fix the test suite for now would be let testsuite
specify using cgcc instead of sparse directly, for the test source
that needs it. That will buy us some time. Fixing sparse properly
in the long term obvious would be better.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Fri, Sep 1, 2017 at 3:46 AM, Uwe Kleine-König  wrote:
\>
> Nearly right. I'm responsible for the sparse Debian package and the
> problem at hand is https://bugs.debian.org/873508. This bug report has
> "Severity: serious" wihch might eventually result in the removal of
> sparse from the Debian archive.
>
> @anarcat: Given that cgcc seems to work, would you agree to apply the
> following patch to horst:
>
> diff --git a/Makefile b/Makefile
> index 4f924fa..d563652 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -110,7 +110,7 @@ $(NAME): $(OBJS)
>  $(OBJS): .buildflags
>
>  check:
> -   sparse $(CFLAGS) *.[ch]
> +   cgcc -no-compile $(CFLAGS) *.[ch]

You patch definitely works.
I think it is the better way to fix it.
Sparse "selfcheck" target was doing the same thing.
It is using "cgcc -no-compile" already.

I suggest in your patch you can do some thing
similar to "selfcheck" target:

CHECKER = cgcc -no-compile

 $(CHECKER) $(CFLAGS) *.[ch]

Later when we update sparse to handle architecture
properly. We can just invoke make with "CHECKER="
to test.


> and downgrade the bug to "important"? That would be a compromise that
> buys us a bit of time.

Agree.

>
> This only solves a part of the problem because the bug report is about
> sparse itself, not it's llvm part.

I agree with that too.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Thu, Aug 31, 2017 at 6:43 PM, Ramsay Jones
>>  - move sparse to /usr/lib
>>  - teach cgcc about the move of sparse
>>  - make /usr/bin/sparse call cgcc -no-compile "$@"
>
> Hmm, I don't think that would be a good idea ...
>

Agree.

>
> Anyway, if you were to un-install llvm, sparse-llvm etc., would not
> be built, and the tests would not be run ... ;-)

I think Uwe already confirm using cgcc to invoke sparse will
fix the problem.

In this test, we just need to use cgcc to invoke sparse-llvm.
I have a patch to fix the usage in test-suite in the other email.

Chris

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Thu, Aug 31, 2017 at 4:55 PM, Uwe Kleine-König  Yes
that works. So to address the Debian bug I can do:
>
>  - move sparse to /usr/lib
>  - teach cgcc about the move of sparse
>  - make /usr/bin/sparse call cgcc -no-compile "$@"

I don't like that. It means the user can't invoke sparse directly.

>
> or is it easier to teach sparse about the architecture stuff?

First of all. It is not very trivial to teach sparse about the architecture
stuff. To my mind, we need to move all the cgcc logic into sparse.

For this case, I think it is easier to teach the sparse validation
code to use cgcc on those back end testing. Most validation don't
need to include system header file at all so it does not have
this problem.

How about this patch?
I know my patch is white space damaged in email.
Git branch is at:
https://git.kernel.org/pub/scm/devel/sparse/chrisl/sparse.git/log/?h=llvm-cgcc

Please let me know if that fix your problem. It pass check
on my local machine running x86_64. I don't have ppc64 to
test with.

Chris

diff --git a/sparsec b/sparsec
index 9dc96c9..2990d26 100755
--- a/sparsec
+++ b/sparsec
@@ -32,7 +32,8 @@ done
 TMPLLVM=`mktemp -t tmp.XX`".llvm"
 TMPFILE=`mktemp -t tmp.XX`".o"

-$DIRNAME/sparse-llvm $SPARSEOPTS > $TMPLLVM
+env CHECK=$DIRNAME/sparse-llvm $DIRNAME/cgcc -no-compile \
+   $SPARSEOPTS > $TMPLLVM

 LLC=`"${LLVM_CONFIG:-llvm-config}" --bindir`/llc

diff --git a/sparsei b/sparsei
index 3431a9f..3abd00f 100755
--- a/sparsei
+++ b/sparsei
@@ -10,4 +10,4 @@ if [ $# -eq 0 ]; then
   exit 1
 fi

-$DIRNAME/sparse-llvm $@ | $LLI
+env CHECK=$DIRNAME/sparse-llvm $DIRNAME/cgcc -no-compile $@ | $LLI

Bug#873508: sparse test failures on ppc32le (and other not so common archs)

[Christopher Li]

On Wed, Aug 30, 2017 at 1:36 PM, Uwe Kleine-König  
wrote:
>> >
>> > diff --git a/validation/backend/sum.c b/validation/backend/sum.c
>> > index 0604299..d0be8dd 100644
>> > --- a/validation/backend/sum.c
>> > +++ b/validation/backend/sum.c
>> > @@ -1,3 +1,5 @@
>> > +#define __powerpc64__
>> > +#define _CALL_ELF 2
>> >  #include 
>> >  #include 
>> >
>> >
>>
>> Yep, sparse/sparsec do not define various macros that gcc/clang define
>> by default on a given architecture. This is a known problem (that I have
>> been meaning to fix ...). The 'workaround' for the time being is to use
>> the cgcc front-end to sparse. (for example 'make CC=cgcc', or perhaps
>> 'cgcc -no-compile').
>>
>> [You didn't mention your usage - is this for a kernel build?]
>
> This problem became visible during the make check phase when creating packaged
> on the listed archs for horst[1]. You can see a build logs at
>
> 
> https://buildd.debian.org/status/fetch.php?pkg=horst=s390x=5.0-1=1503905687=0
> 
> https://buildd.debian.org/status/fetch.php?pkg=horst=ppc64el=5.0-1=1503906226=0
>
> The error message looks identical (I checked the ppc64el log) to the
> problem with backend/sum.c:
>
> sparse -g -O2 -fdebug-prefix-map=/<>=. 
> -fstack-protector-strong -Wformat -Werror=format-security -std=gnu99 -Wall 
> -Wextra -g -I. -DDO_DEBUG -I/usr/include/libnl3 *.[ch]
> /usr/include/powerpc64le-linux-gnu/gnu/stubs.h:8:12: error: unable to 
> open 'gnu/stubs-32.h'

That is very much like on x86_64 missing define "#weak_define __x86_64__ 1"

Does cgcc work for you? In the future we do want to move the archetecture
related define from cgcc into sparse by itself. For now you can set
"sparse" as "cgcc -no-compile"

Chris