hello,
I joined as co-maintainer for syft after the recent todo item[1] and modernized
the PKGBUILD.
Based on the repro-get package[2] (which did reproduce successfully) I've
changed the syft build instructions to this:
```
export CGO_CPPFLAGS="${CPPFLAGS}"
export CGO_CFLAGS="${CFLAGS}"
export CGO_CXXFLAGS="${CXXFLAGS}"
export CGO_LDFLAGS="${LDFLAGS}"
export CGO_REQUIRED="1"
mkdir -p build
go build \
-o build/ \
-buildmode=pie \
-mod=readonly \
-ldflags "-compressdwarf=false -linkmode=external -X main.version=${pkgver}" \
./...
```
This did not reproduce in our verification system however:
```
│ ├── usr/bin/syft
│ │┄ File has been modified after NT_GNU_BUILD_ID has been applied.
│ │ ├── readelf --wide --notes {}
│ │ │ @@ -1,16 +1,16 @@
│ │ │
│ │ │ Displaying notes found in: .note.gnu.property
│ │ │ Owner Data size Description
│ │ │ GNU 0x00000010 NT_GNU_PROPERTY_TYPE_0
Properties: x86 ISA needed: x86-64-baseline
│ │ │
│ │ │ Displaying notes found in: .note.gnu.build-id
│ │ │ Owner Data size Description
│ │ │ - GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID
bitstring) Build ID: 5d4cf47367358962632d3bfd04530c513f897790
│ │ │ + GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID
bitstring) Build ID: dd9283dbad8ad8c4b742f7ac88f6a0e1f3b9acc1
│ │ │
│ │ │ Displaying notes found in: .note.go.buildid
│ │ │ Owner Data size Description
│ │ │ - Go 0x00000053 GO BUILDID description data: 42 49
56 6f 41 61 39 67 43 50 4e 49 6e 37 70 49 55 2d 76 64 2f 5f 6b 4a 66 6e 31 37 68
6e 4b 42 65 42 66 62 6f 39 74 4d 4b 2f 59 39 66 41 33 4d 74 7a 69 66 56 71 4b 4a
42 42 68 78 7a 57 2f 42 50 73 44 75 79 76 55 49 4c 37 30 47 37 73 6e 67 6a 55 73
│ │ │ + Go 0x00000053 GO BUILDID description data: 50 44
71 65 63 37 36 71 30 4e 52 69 4b 36 69 57 63 31 73 52 2f 5f 6b 4a 66 6e 31 37 68
6e 4b 42 65 42 66 62 6f 39 74 4d 4b 2f 59 39 66 41 33 4d 74 7a 69 66 56 71 4b 4a
42 42 68 78 7a 57 2f 73 6a 72 50 70 62 38 39 32 41 4c 48 4e 62 6c 4d 72 75 4d 46
│ │ │
│ │ │ Displaying notes found in: .note.ABI-tag
│ │ │ Owner Data size Description
│ │ │ GNU 0x00000010 NT_GNU_ABI_TAG (ABI version tag)
OS: Linux, ABI: 4.4.0
│ │ ├── strings --all --bytes=8 {}
│ │ │ @@ -1,8 +1,8 @@
│ │ │
-BIVoAa9gCPNIn7pIU-vd/_kJfn17hnKBeBfbo9tMK/Y9fA3MtzifVqKJBBhxzW/BPsDuyvUIL70G7sngjUs
│ │ │
+PDqec76q0NRiK6iWc1sR/_kJfn17hnKBeBfbo9tMK/Y9fA3MtzifVqKJBBhxzW/sjrPpb892ALHNblMruMF
│ │ │ /lib64/ld-linux-x86-64.so.2
│ │ │ __libc_start_main
│ │ │ __cxa_finalize
│ │ │ __vfprintf_chk
│ │ │ sigfillset
│ │ │ pthread_sigmask
│ │ │ pthread_attr_init
│ │ ├── readelf --wide --decompress --hex-dump=.gnu_debuglink {}
│ │ │ @@ -1,4 +1,4 @@
│ │ │
│ │ │ Hex dump of section '.gnu_debuglink':
│ │ │ - 0x00000000 73796674 2e646562 75670000 ca77ec4e syft.debug...w.N
│ │ │ + 0x00000000 73796674 2e646562 75670000 6b0247ba syft.debug..k.G.
```
From what I understand both build IDs are hashes over other parts of the
binary. Since they are the only difference though, the problem is likely in the
debug symbols that get detached after the compile (and also after this hashing
is done), but before diffoscope is executed on the final release artifact.
Our guidelines for Go[3] mention the -trimpath option (which I forgot to add,
but also wasn't necessary for repro-get), but since we have normalized paths in
our build containers I'm not sure if the claim about Reproducible Builds applies
to non-AUR packages?
Using the data from reproducible.archlinux.org, there's currently 371 packages
that depend on our Go package and have some kind of status attached to it:
% rg -c GOOD resolved-go.txt
240
% rg -c BAD resolved-go.txt
131
Which is about ~65%
In comparison, although many packages still use the `rustc --print` approach,
our Rust build instructions essentially boil down to:
```
cargo fetch --locked --target host-tuple
cargo build --frozen --release
```
Those two commands with no further manually set environment variables gets us:
% rg -c GOOD resolved-rust.txt
669
% rg -c BAD resolved-rust.txt
86
Which is ~89%
Although 20 percent-points doesn't seem like that much, it's a difference of
"every 1 out of 3" vs "every 1 out of 10".
Is there anything we can do to improve this overall situation? Both the amount
of complexity necessary to compile a simple binary, as well as the reproducible
build success ration?
Maybe we could have something like /etc/makepkg.conf.d/rust.conf but for Go?
cheers,
kpcyrd
[1]:
https://archlinux.org/todo/rebuild-packages-signed-by-anatolik-morganamilo-shibumi/
[2]:
https://gitlab.archlinux.org/archlinux/packaging/packages/repro-get/-/blob/main/PKGBUILD?ref_type=heads
[3]: https://wiki.archlinux.org/title/Go_package_guidelines
