[Tinycc-devel] Reproducible builds

2016-10-20 Thread avih 
I tried to create a reproducible windows build of tcc. I chose windows because:
- I'm more familiar with Windows as a tcc build target.- tcc targeting windows 
is self contained (has all headers/libs to create executables)
- tcc is distributed as binary for windows, so it would be nice if users could 
reproduce the official build on any OS.
- It might be able to detect problems with tcc.
- I wanted to make it easy to build tcc for windows without installing big 
environments.

What I came up with is a simple procedure which is compiler and 
operating-system agnostic*:
- Use a native compiler to create a native tcc executable which targets windows.
- Use this new tcc to build tcc for windows (this is how the normal makefile 
builds the libs for the windows cross compilers).
- It then outputs a single combined signature of all the output files such that 
it can be compared easily between environments.
- For simplicity, I used ONE_SOURCE (which still supports -run, but no libtcc).

I ended up with a script which builds 4 windows tcc compilers: native 32 and 
64, and cross 32->64 and 64->32.
Its only input is a working native compiler, and it requires a posix-y shell, 
and works in busybox, dash, bash, etc.
To use it, place the attached file at the win32 dir of tcc, and then:
CC=gcc ./build-tcc-reproducible.sh

For the currently latest mob commit (3054a76   i386-gen: use EBX as 4th 
register) the build signature is 52155cb6a03fcfd2632dc649984875e1

This script completes on all the environments which I tried and produces 
identical binaries (and signature) regardless of the system and the initial 
compiler.

Tested environments/compilers which produce identical outputs:
- Ubuntu 16.04 32/64 (gcc 5.4 and clang-3.8, glibc)
- FreeBSD* (GhostBSD) 32/64 (gcc 4.4, glibc?)
- OSX* el capitan (Apple LLVM 8)
- Alpine** linux 64 (gcc 6.2, musl-libc)
- Windows: Msys2 32/64 (gcc 6.2), MSVC 2013 32/64
- And of course each of the 4 output tcc compilers which this script generates 
can generate identical 4 output compilers.

On windows, it's enough to use a binary distribution of tcc (zipped, but not 
0.9.26) and busybox to generate identical outputs.

I didn't run the resulting tcc's through the existing tests, but I did use them 
as inputs to the script itself (generates identical outputs) and as input for 
the normal tcc configure, make, make test - where the generated tcc's do pass 
all tests.

I'm not sure how much value this has in general, but at the very least it's 
interesting IMO.
I _think_ it would be possible to generate identical compilers which target 
other systems too, even if they won't be as functional as the windows outputs, 
just to see how tcc "reacts" on different systems where it's supposed to 
generate eventual identical outputs, but I'm not familiar enough with the 
subject to really assess how feasible or useful it would be.

---

* On Alpine/OSX/BSD I wasn't able to build native tcc with configure+make, but 
the script still works and generates cross and then identical win-native tcc's.

** On GhostBSD 32/64 there's a tcc bug where the object files (for libtcc1.a) 
use BSD bits at elf header even if the target if not BSD. I worked around it 
but didn't push to mob since I don't think it's the best fix. The offending 
line is at (the current) tccelf.c:2577 :

#if defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
    ehdr.e_ident[EI_OSABI] = ELFOSABI_FREEBSD;
#endif

And my workaround was to check if the target is not windows, which was good 
enough to generate identical signature as the other envs. There are actually a 
lot of similar target-unconditional BSD code, but changing this one alone was 
enough to generate identical output.



build-tcc-reproducible.sh
Description: Binary data
___
Tinycc-devel mailing list
Tinycc-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

Re: [Tinycc-devel] core dump on FreeBSD with last commit "configure: --triplet= option, Makefile: cleanup"

2016-10-20 Thread Christian Jullien 
I'm sure you'll not believe me, but it's NOT an April Fool.

$ uname -a
FreeBSD freebsd64.eligis.com 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0
r306420: Thu Sep 29 01:43:23 UTC 2016
r...@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
$ gcc -m32 --print-file-name=crt1.o
/usr/lib/crt1.o
$ gcc -m64 --print-file-name=crt1.o
/usr/lib/crt1.o
$ gcc --print-file-name=crt1.o
/usr/lib/crt1.o
$ find /usr -name crt1.o
/usr/lib32/crt1.o
/usr/lib/crt1.o

-Original Message-
From: Tinycc-devel [mailto:tinycc-devel-bounces+eligis=orange...@nongnu.org]
On Behalf Of Michael Matz
Sent: jeudi 20 octobre 2016 17:23
To: tinycc-devel@nongnu.org
Subject: Re: [Tinycc-devel] core dump on FreeBSD with last commit
"configure: --triplet= option, Makefile: cleanup"

Hello Christian,

On Wed, 19 Oct 2016, Christian Jullien wrote:

> x86_64:
> $ uname -m
> amd64
> $ gcc --print-file-name=crt1.o
> /usr/lib/crt1.o
> $ find /usr -name crt1.o
> /usr/lib32/crt1.o
> /usr/lib/crt1.o

Aha!  So they chose the opposite way to the linuxes.  Okay, that helps,
thanks.  Can you verify that

% gcc -m32 --print-file-name=crt1.o

gives /usr/lib32/crt1.o ?


Ciao,
Michael.

___
Tinycc-devel mailing list
Tinycc-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/tinycc-devel


___
Tinycc-devel mailing list
Tinycc-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

Re: [Tinycc-devel] Weird bitfield size handling, discrepancy with gcc

2016-10-20 Thread David Mertens 
On Tue, Oct 18, 2016 at 11:37 AM, Michael Matz  wrote:

> Hi,
>
> On Tue, 18 Oct 2016, David Mertens wrote:
>
> > According to Christian, we have at least one major compiler (VC++) whose
> > behavior matches tcc's current behavior and another (GCC) whose behavior
> > differs.
>
> Yes, I've implemented the GCC way privately already (necessary for e.g.
> linux kernel).  Need to clean this up, sigh.
>
> > While it would be nice to just pick one implementation and go with it, I
> > am personally much more concerned with binary compatibility with one's
> > major compiler of choice.
>
> Exactly.  Actually also GCC itself has different implementations.  E.g.
> when configured for windows (or with -mms-bitfields) it implements the
> microsoft way, otherwise it defaults to the old PCC layout, which GCC uses
> on most UNIXen and other operating systems.  There're also minor
> variations for the latter e.g. on ppc64, but those wouldn't concern us.
>

Should we add this as a compiler flag to tcc, perhaps?


> > Here is a patch that makes the bit field size logic match gcc. I can
> > confirm that the original alignment bug is gone (though other alignment
> > bugs may yet be around). I am not 100% sure if this is correct, but I
> > throw it out there for folks to mull over. Note that indentation may not
> > be consistent; the indentation pattern in the existing code was weird.
> > Finally, it might be more efficient to use (bit_size & 7) rather than
> > (bit_size % 8), or optimizing compilers might do that for us anyway; I
> > leave that to the gurus.
>
> >  } else {
> > +   size = bit_size / 8;
> > +if (bit_size % 8) size++; /* round up */
> > +lbit_pos = bit_pos;
> >  /* we do not have enough room ?
> > did the type change?
> > is it a union? */
> >  if ((bit_pos + bit_size) > bsize ||
> > -bt != prevbt || a == TOK_UNION)
> > -bit_pos = 0;
> > -lbit_pos = bit_pos;
> > +bt != prevbt || a == TOK_UNION) {
> > +lbit_pos = bit_pos = 0;
>
> For instance, this is wrong for PCC layout.  MS layout indeed switches
> storage containers when the base type changes, i.e. here:
>
> struct S {
>   char  c:1;
>   short s:1;
>   char foo;
> };
>
> PCC layout has no such provisions.  But the base type _does_ influence
> alignment of the containing struct nevertheless.  Except if the bitfield
> has :0 size, then it _does_ force the next field to be placed into a new
> storage container.  It's all quite terrible and twisted :-/
>

I question the wisdom of targeting the binary layout of pcc. It's a neat
compiler, with an aim similar to tcc, but developer time and resources are
limited. I'd vote for just MSVC and GCC compatibility as a start.
(Presumably we get clang for free with that.) If anybody later comes along
with inclination, they can extend it to handle PCC layout.

David

-- 
 "Debugging is twice as hard as writing the code in the first place.
  Therefore, if you write the code as cleverly as possible, you are,
  by definition, not smart enough to debug it." -- Brian Kernighan
___
Tinycc-devel mailing list
Tinycc-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

Re: [Tinycc-devel] core dump on FreeBSD with last commit "configure: --triplet= option, Makefile: cleanup"

2016-10-20 Thread Michael Matz 
Hi,

On Tue, 18 Oct 2016, grischka wrote:

> Honestly, in my book the cross compilers are just to test
> compilation,  they are not assumed to work out of the box.
> (Except the windows compiles where everything you need comes
> with the source and therefor is well known where it is).

I generally sympathize with this, except for one case: where the cross 
compilers aren't really "cross" :)  E.g. on my linux x86-64 system I'd 
expect x86_64-tcc and ./tcc be basically the same.  Also the i386-tcc 
should be basically a native compiler that acts like gcc -m32.  I agree 
that for e.g. ./arm-tcc on such system all bets are off.

> There is no support whatsoever from our configure either.
> People tried to "fix" that but I was not able to detect
> any underlying concept.

Right, there were only hacks over hacks that made it sort of work for most 
people :)  I also agree that cleaning this up would be better.

> As to the lib/lib64 issue, whatever you can figure out I'm fine.

Okay, perhaps on the weekend.

> Except I'd prefer to have the logic in only one location, that
> is NOT in both the Makefile AND configure, because that is just
> confusing.

Yeah.

> FYI: on my (old 8.10 from 2008) ubuntu 64, there is /usr/lib64,
> but it is a link to /usr/lib.  There is also /usr/lib/x86_64-linux-gnu,
> but it is empty.  I was under the impression that lib64 was an
> old misguided concept from the early 64 bit days. Maybe I was wrong.

Yeah, no, it wasn't really misguided.  Simply the first approach to deal 
with a multi-arch system; at the time we did the x86-64 port there were 
already some systems that used similar models: ia64 with its /libx86, and 
Solaris with /usr/lib/64.  Of course once you do something like that 
you're bound to never change it again because it'd break all userspace :-/

The debian multi-arch way was invented much later to solve a larger 
problem (that of _really_ having cross architecture libs on one system).  
The debian based distros eventually picked this up (though, of course, the 
older /lib64 way is still supported), the rpm based ones didn't bother.

I'll try to come up with something.


Ciao,
Michael.

___
Tinycc-devel mailing list
Tinycc-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

Re: [Tinycc-devel] core dump on FreeBSD with last commit "configure: --triplet= option, Makefile: cleanup"

2016-10-20 Thread Michael Matz 
Hello Christian,

On Wed, 19 Oct 2016, Christian Jullien wrote:

> x86_64:
> $ uname -m
> amd64
> $ gcc --print-file-name=crt1.o
> /usr/lib/crt1.o
> $ find /usr -name crt1.o
> /usr/lib32/crt1.o
> /usr/lib/crt1.o

Aha!  So they chose the opposite way to the linuxes.  Okay, that helps, 
thanks.  Can you verify that

% gcc -m32 --print-file-name=crt1.o

gives /usr/lib32/crt1.o ?


Ciao,
Michael.

___
Tinycc-devel mailing list
Tinycc-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/tinycc-devel

Re: [Tinycc-devel] asmtest problem

2016-10-20 Thread Michael Matz 
Hello grischka,

On Wed, 19 Oct 2016, grischka wrote:

> There seems to be a asm problem in tcctest.c with the bts
> instruction on x86-64, which crashes it under circumstances.
> 
> __asm__("bts %1,%0" : "=m"(*pset) : "Ir"(_sig - 1) : "cc");
> 
> From dump:
>   3d: 8b 45 d0mov-0x30(%rbp),%eax <-- address
>   Here it should load the addres into %rax, not %eax

Ah yes, that one.  The attached patch should solve this, which also is 
part of that mythical larger series that makes us compile the linux x86-64 
kernel.


Ciao,
Michael.

commit 70a1dc38908b6ff679f15cf033828afda885d35f
Author: Michael Matz 
Date:   Wed Jun 29 18:06:40 2016 +0200

inline asm: Fix 'm' constraints

See testcase.  The m constraint actually passes the address
and hence is of pointer type.

diff --git a/i386-asm.c b/i386-asm.c
index c0dd435..cc73320 100644
--- a/i386-asm.c
+++ b/i386-asm.c
@@ -1291,7 +1291,12 @@ ST_FUNC void subst_asm_operand(CString *add_str,
 if (reg >= VT_CONST)
 tcc_error("internal compiler error");
 snprintf(buf, sizeof(buf), "(%%%s)",
- get_tok_str(TOK_ASM_eax + reg, NULL));
+#ifdef TCC_TARGET_X86_64
+ get_tok_str(TOK_ASM_rax + reg, NULL)
+#else
+ get_tok_str(TOK_ASM_eax + reg, NULL)
+#endif
+);
 cstr_cat(add_str, buf, -1);
 } else {
 /* register case */
@@ -1392,7 +1397,8 @@ ST_FUNC void asm_gen_code(ASMOperand *operands, int 
nb_operands,
output cases) */
 SValue sv;
 sv = *op->vt;
-sv.r = (sv.r & ~VT_VALMASK) | VT_LOCAL;
+sv.r = (sv.r & ~VT_VALMASK) | VT_LOCAL | VT_LVAL;
+   sv.type.t = VT_PTR;
 load(op->reg, );
 } else if (i >= nb_outputs || op->is_rw) {
 /* load value in register */
diff --git a/tccasm.c b/tccasm.c
index 55feecf..ad9580a 100644
--- a/tccasm.c
+++ b/tccasm.c
@@ -1043,7 +1043,8 @@ static void parse_asm_operands(ASMOperand *operands, int 
*nb_operands_ptr,
 skip('(');
 gexpr();
 if (is_output) {
-test_lvalue();
+   if (!(vtop->type.t & VT_ARRAY))
+   test_lvalue();
 } else {
 /* we want to avoid LLOCAL case, except when the 'm'
constraint is used. Note that it may come from
diff --git a/tests/tcctest.c b/tests/tcctest.c
index 28ba263..0922d77 100644
--- a/tests/tcctest.c
+++ b/tests/tcctest.c
@@ -2583,12 +2583,33 @@ int fls64(unsigned long long x)
 }
 #endif
 
+struct struct123 {
+int a;
+int b;
+};
+struct struct1231 {
+unsigned long addr;
+};
+
+unsigned long mconstraint_test(struct struct1231 *r)
+{
+unsigned long ret;
+unsigned int a[2];
+a[0] = 0;
+__asm__ volatile ("leaq %2,%0; movl 4(%0),%k0; addl %2,%k0; movl $51,%2; 
movl $52,4%2; movl $63,%1"
+ : "=" (ret), "=m" (a)
+ : "m" (*(struct struct123 *)r->addr));
+return ret + a[0];
+}
+
 unsigned int set;
 
 void asm_test(void)
 {
 char buf[128];
 unsigned int val;
+struct struct123 s1;
+struct struct1231 s2 = { (unsigned long) };
 
 printf("inline asm:\n");
 
@@ -2610,6 +2631,10 @@ void asm_test(void)
 printf("mul64=0x%Lx\n", mul64(0x12345678, 0xabcd1234));
 printf("inc64=0x%Lx\n", inc64(0x12345678));
 
+s1.a = 42;
+s1.b = 43;
+printf("mconstraint: %d", mconstraint_test());
+printf(" %d %d\n", s1.a, s1.b);
 set = 0xff;
 sigdelset1(, 2);
 sigaddset1(, 16);

___
Tinycc-devel mailing list
Tinycc-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/tinycc-devel