So what happens if we actually fix the underlying bug?
I see two ways of doing this. In findfp.c, we allocate an array of FILE * today
like:
g = (struct glue *)malloc(sizeof(*g) + ALIGNBYTES + n * sizeof(FILE));
but that assumes that FILE just has normal pointer alignment requirements.
However,
due to the mbstate having int64_t alignment requirements, this is wrong. Maybe
we
need to do something like
g = (struct glue *)malloc(sizeof(*g) + max(sizeof(int64_t),ALIGNBYTES)
+ n * sizeof(FILE));
which wouldn’t change anything on LP64 systems, but would result in proper
alignment
for ILP32 systems. We’d have to fix the loop that uses ALIGN afterwards to use
roundup. Instead, we’d need to round up to the neared 8-byte aligned offset (or
technically,
the max of ALIGNBYTES and 8, but that’s always 8 on today’s systems. If we do
this,
we can make sure that each file is 8-byte aligned or better. We may need to
round up
sizeof(FILE) to a multiple of 8 as well. I believe that since it has the 8-byte
alignment
for a member, its size must be a multiple of 8, but I’ve not chased that belief
to ground.
If not, we may need another decorator (__aligned(8), I think, spelled with the
ugly
max expression above). That way, the contract we’re making with the compiler
will
always be true. ALIGN BYTES is 4 on Arm anyway, so that bit is clearly wrong.
This wouldn’t be an ABI change, since you can only get a valid FILE * from
fopen (and
friends), plus stdin, stdout, and stderr. Those addresses aren’t hard coded
into binaries,
so even if we have to tweak the last three and deal with some ‘fake’ FILE abuse
in libc
(which I don’t think suffers from this issue, btw, given the alignment
requirements that would
naturally follow from something on the stack), we’d still be ahead. At least
for all CONFORMING
implementations[*]...
TL;DR: Why not make FILE * always 8-byte aligned? The compiler options are a
band-aide.
Warner
[*] There’s at least on popular package that has a copy of the FILE structure
in one of its
.h files and uses that to do unnatural optimization things, but even that’s
cool, I think,
since it never allocates a new one.
> On Dec 25, 2015, at 7:24 AM, Mark Millard <mar...@dsl-only.net> wrote:
>
> [Good News Summary: Rebuilding buildworld/buildkernel for rpi2 11.0-CURRENT
> 292413 from amd64 based on adding -fmax-type-align=4 has so far removed the
> crashes during the toolchain activity: no more misaligned accesses in libc's
> _fseeko or elsewhere.]
>
> On 2015-Dec-25, at 12:31 AM, Mark Millard <mar...@dsl-only.net> wrote:
>
>> On 2015-Dec-24, at 10:39 PM, Mark Millard <mar...@dsl-only.net> wrote:
>>
>>> [I do not know if this partial crash analysis related to on-arm
>>> clang-associated activity is good enough and appropriate to submit or not.]
>>>
>>> The /usr/local/arm-gnueabi-freebsd/bin/ar on the rpi2b involved below came
>>> from pkg install activity instead of port building. Used as-is.
>>>
>>> When I just tried my first from-rpi2b builds (ports for a rpi2b),
>>> /usr/local/arm-gnueabi-freebsd/bin/ar crashed. I believe that the following
>>> suggests an alignment error for the type of instructions that memset for
>>> 128 bytes was translated to (sizeof(mbstate_t)) in the code used by
>>> /usr/local/arm-gnueabi-freebsd/bin/ar. (But I do not know how to check
>>> SCTLR bit[1] to be directly sure that alignment was being enforced.)
>>>
>>> The crash was a Bus error in /usr/local/arm-gnueabi-freebsd/bin/ar :
>>>
>>>> libtool: link: /usr/local/arm-gnueabi-freebsd/bin/ar cru
>>>> .libs/libgnuintl.a bindtextdom.o dcgettext.o dgettext.o gettext.o
>>>> finddomain.o hash-string.o loadmsgcat.o localealias.o textdomain.o
>>>> l10nflist.o explodename.o dcigettext.o dcngettext.o dngettext.o ngettext.o
>>>> pluralx.o plural-exp.o localcharset.o threadlib.o lock.o relocatable.o
>>>> langprefs.o localename.o log.o printf.o setlocale.o version.o xsize.o
>>>> osdep.o intl-compat.o
>>>> Bus error (core dumped)
>>>> *** [libgnuintl.la] Error code 138
>>>
>>> It failed in _fseeko doing a memset that turned into uses of "vst1.64
>>> {d16-d17}, [r0]" instructions, for an address in register r0 that ended in
>>> 0xa4, so was not aligned to 8 byte boundaries. From what I read such "VSTn
>>> (multiple n-element structures)" that have .64 require 8 byte alignment.
>>> The evidence of the code and register value follow.
>>>
>>>> # gdb /usr/local/arm-gnueabi-freebsd/bin/ar
>>>> /usr/obj/portswork/usr/ports/devel/gettext-tools/work/gettext-0.19.6/gettext-tools/intl/ar.core
>>>> . . .
>>>> #0 0x2033adcc in _fseeko (fp=0x20651dcc, offset=<value optimized out>,
>>>> whence=<value optimized out>, ltest=<value optimized out>) at
>>>> /usr/src/lib/libc/stdio/fseek.c:299
>>>> 299 memset(&fp->_mbstate, 0, sizeof(mbstate_t));
>>>> . . .
>>>> (gdb) x/24i 0x2033adb0
>>>> 0x2033adb0 <_fseeko+836>: vmov.i32 q8, #0 ; 0x00000000
>>>> 0x2033adb4 <_fseeko+840>: movw r1, #65503 ; 0xffdf
>>>> 0x2033adb8 <_fseeko+844>: stm r4, {r0, r7}
>>>> 0x2033adbc <_fseeko+848>: ldrh r0, [r4, #12]
>>>> 0x2033adc0 <_fseeko+852>: and r0, r0, r1
>>>> 0x2033adc4 <_fseeko+856>: strh r0, [r4, #12]
>>>> 0x2033adc8 <_fseeko+860>: add r0, r4, #216 ; 0xd8
>>>> 0x2033adcc <_fseeko+864>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033add0 <_fseeko+868>: add r0, r4, #200 ; 0xc8
>>>> 0x2033add4 <_fseeko+872>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033add8 <_fseeko+876>: add r0, r4, #184 ; 0xb8
>>>> 0x2033addc <_fseeko+880>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033ade0 <_fseeko+884>: add r0, r4, #168 ; 0xa8
>>>> 0x2033ade4 <_fseeko+888>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033ade8 <_fseeko+892>: add r0, r4, #152 ; 0x98
>>>> 0x2033adec <_fseeko+896>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033adf0 <_fseeko+900>: add r0, r4, #136 ; 0x88
>>>> 0x2033adf4 <_fseeko+904>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033adf8 <_fseeko+908>: add r0, r4, #120 ; 0x78
>>>> 0x2033adfc <_fseeko+912>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033ae00 <_fseeko+916>: add r0, r4, #104 ; 0x68
>>>> 0x2033ae04 <_fseeko+920>: vst1.64 {d16-d17}, [r0]
>>>> 0x2033ae08 <_fseeko+924>: b 0x2033b070 <_fseeko+1540>
>>>> 0x2033ae0c <_fseeko+928>: cmp r5, #0 ; 0x0
>>>> (gdb) info all-registers
>>>> r0 0x20651ea4 543497892
>>>> r1 0xffdf 65503
>>>> r2 0x0 0
>>>> r3 0x0 0
>>>> r4 0x20651dcc 543497676
>>>> r5 0x0 0
>>>> r6 0x0 0
>>>> r7 0x0 0
>>>> r8 0x20359df4 540384756
>>>> r9 0x0 0
>>>> r10 0x0 0
>>>> r11 0xbfbfb948 -1077954232
>>>> r12 0x2037b208 540520968
>>>> sp 0xbfbfb898 -1077954408
>>>> lr 0x2035a004 540385284
>>>> pc 0x2033adcc 540257740
>>>> f0 0 (raw 0x000000000000000000000000)
>>>> f1 0 (raw 0x000000000000000000000000)
>>>> f2 0 (raw 0x000000000000000000000000)
>>>> f3 0 (raw 0x000000000000000000000000)
>>>> f4 0 (raw 0x000000000000000000000000)
>>>> f5 0 (raw 0x000000000000000000000000)
>>>> f6 0 (raw 0x000000000000000000000000)
>>>> f7 0 (raw 0x000000000000000000000000)
>>>> fps 0x0 0
>>>> cpsr 0x60000010 1610612752
>>>
>>> The syntax in use for vst1.64 instructions does not explicitly have the
>>> alignment notation. Presuming that the decoding is correct then from what I
>>> read the following applies:
>>>
>>>> Home > NEON and VFP Programming > NEON load and store element and
>>>> structure instructions > Alignment restrictions in load and store, element
>>>> and structure instructions
>>>>
>>>> . . . When the alignment is not specified in the instruction, the
>>>> alignment restriction is controlled by the A bit (SCTLR bit[1]):
>>>> • if the A bit is 0, there are no alignment restrictions (except
>>>> for strongly ordered or device memory, where accesses must be element
>>>> aligned or the result is unpredictable)
>>>> • if the A bit is 1, accesses must be element aligned.
>>>> If an address is not correctly aligned, an alignment fault occurs.
>>>
>>> So if at the time the "A bit" (SCTLR bit[1]) is 1 then the Bus error would
>>> have the context to happen because of the mis-alignment.
>>>
>>> The following shows the make.conf context that explains how
>>> /usr/local/arm-gnueabi-freebsd/bin/ar came to be invoked:
>>>
>>>> # more /etc/make.conf
>>>> WRKDIRPREFIX=/usr/obj/portswork
>>>> WITH_DEBUG=
>>>> WITH_DEBUG_FILES=
>>>> MALLOC_PRODUCTION=
>>>> #
>>>> TO_TYPE=armv6
>>>> TOOLS_TO_TYPE=arm-gnueabi
>>>> CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/
>>>> .if ${.MAKE.LEVEL} == 0
>>>> CC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>>>> CXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>>>> CPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi
>>>> -march=armv7a
>>>> .export CC
>>>> .export CXX
>>>> .export CPP
>>>> AS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as
>>>> AR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar
>>>> LD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld
>>>> NM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm
>>>> OBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy
>>>> OBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump
>>>> RANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib
>>>> SIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size
>>>> #NO-SUCH: STRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings
>>>> STRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings
>>>> .export AS
>>>> .export AR
>>>> .export LD
>>>> .export NM
>>>> .export OBJCOPY
>>>> .export OBJDUMP
>>>> .export RANLIB
>>>> .export SIZE
>>>> .export STRINGS
>>>> .endif
>>>
>>>
>>> Other context:
>>>
>>>> # freebsd-version -ku; uname -aKU
>>>> 11.0-CURRENT
>>>> 11.0-CURRENT
>>>> FreeBSD rpi2 11.0-CURRENT FreeBSD 11.0-CURRENT #0 r292413M: Tue Dec 22
>>>> 22:02:21 PST 2015
>>>> root@FreeBSDx64:/usr/obj/clang/arm.armv6/usr/src/sys/RPI2-NODBG arm
>>>> 1100091 1100091
>>>
>>>
>>>
>>> I will note that world and kernel are my own build of -r292413 (earlier
>>> experiment) --a build made from an amd64 host context and put in place via
>>> DESTDIR=. My expectation would be that the amd64 context would not be
>>> likely to have similar alignment restrictions involved in its ar activity
>>> (or other activity). That would explain how I got this far using such a
>>> clang 3.7 related toolchain for targeting an rpi2 before finding such a
>>> problem.
>>
>>
>> I realized re-reading the all above that it seems to suggest that the
>> _fseeko code involved is from /usr/local/arm-gnueabi-freebsd/bin/ar but that
>> was not my intent.
>>
>> libc.so.7 is from my buildworld, including the fseeko implementation:
>>
>> Reading symbols from /lib/libc.so.7...Reading symbols from
>> /usr/lib/debug//lib/libc.so.7.debug...done.
>> done.
>> Loaded symbols for /lib/libc.so.7
>>
>>
>> head/sys/sys/_types.h has:
>>
>> /*
>> * mbstate_t is an opaque object to keep conversion state during multibyte
>> * stream conversions.
>> */
>> typedef union {
>> char __mbstate8[128];
>> __int64_t _mbstateL; /* for alignment */
>> } __mbstate_t;
>>
>> suggesting an implicit alignment of the union to whatever the implementation
>> defines for __int64_t --which need not be 8 byte alignment (in the abstract,
>> general case). But 8 byte alignment is a possibility as well (in the
>> abstract).
>>
>> But printing *fp in gdb for the fp argument to _fseeko reports the same
>> not-8-byte aligned address for __mbstate8 that was in r0:
>>
>>> (gdb) bt
>>> #0 0x2033adcc in _fseeko (fp=0x20651dcc, offset=<value optimized out>,
>>> whence=<value optimized out>, ltest=<value optimized out>) at
>>> /usr/src/lib/libc/stdio/fseek.c:299
>>> #1 0x2033b108 in fseeko (fp=0x20651dcc, offset=18571438587904, whence=0)
>>> at /usr/src/lib/libc/stdio/fseek.c:82
>>> #2 0x00016138 in ?? ()
>>> (gdb) print fp
>>> $2 = (FILE *) 0x20651dcc
>>> (gdb) print *fp
>>> $3 = {_p = 0x2069a240 "", _r = 0, _w = 0, _flags = 5264, _file = 36, _bf =
>>> {_base = 0x2069a240 "", _size = 32768}, _lbfsize = 0, _cookie = 0x20651dcc,
>>> _close = 0x20359dfc <__sclose>,
>>> _read = 0x20359de4 <__sread>, _seek = 0x20359df4 <__sseek>, _write =
>>> 0x20359dec <__swrite>, _ub = {_base = 0x0, _size = 0}, _up = 0x0, _ur = 0,
>>> _ubuf = 0x20651e0c "", _nbuf = 0x20651e0f "", _lb = {
>>> _base = 0x0, _size = 0}, _blksize = 32768, _offset = 0, _fl_mutex = 0x0,
>>> _fl_owner = 0x0, _fl_count = 0, _orientation = 0, _mbstate = {__mbstate8 =
>>> 0x20651e34 "", _mbstateL = 0}, _flags2 = 0}
>>
>> The overall FILE struct containing the _mbstate field is also not 8-byte
>> aligned. But the offset from the start of the FILE struct to __mbstate8 is a
>> multiple of 8 bytes.
>>
>> It is my interpretation that there is nothing here to justify the memset
>> implementation combination:
>>
>> SCTLR bit[1]==1
>>
>> mixed with
>>
>> vst1.64 instructions
>>
>> I.e.: one or both needs to change unless some way for forcing 8-byte
>> alignment is introduced.
>>
>> I have not managed to track down anything that would indicate FreeBSD's
>> intent for SCTLR bit[1]. I do not even know if it is required by the design
>> to be constant (once initialized).
>
>
> I have (so far) removed the build tool crashes based on adding
> -fmax-type-align=4 to avoid the misaligned accesses. Details follow.
>
> src.conf on amd64 for the rpi2 targeting buildworld/buildkernel now looks
> like:
>
>> # more ~/src.configs/src.conf.rpi2-clang.amd64-host
>> TO_TYPE=armv6
>> TOOLS_TO_TYPE=arm-gnueabi
>> FROM_TYPE=amd64
>> TOOLS_FROM_TYPE=x86_64
>> VERSION_CONTEXT=11.0
>> #
>> KERNCONF=RPI2-NODBG
>> TARGET=arm
>> .if ${.MAKE.LEVEL} == 0
>> TARGET_ARCH=${TO_TYPE}
>> .export TARGET_ARCH
>> .endif
>> #
>> WITHOUT_CROSS_COMPILER=
>> #
>> # For WITH_BOOT= . . .
>> # arm-gnueabi-freebsd/bin/ld reports bootinfo.o: relocation
>> R_ARM_MOVW_ABS_NC against `a local symbol' can not be used when making a
>> shared object; recompile with -fPIC
>> WITHOUT_BOOT=
>> #
>> WITH_FAST_DEPEND=
>> WITH_LIBCPLUSPLUS=
>> WITH_CLANG=
>> WITH_CLANG_IS_CC=
>> WITH_CLANG_FULL=
>> WITH_LLDB=
>> WITH_CLANG_EXTRAS=
>> #
>> WITHOUT_LIB32=
>> WITHOUT_GCC=
>> WITHOUT_GNUCXX=
>> #
>> NO_WERROR=
>> MALLOC_PRODUCTION=
>> #CFLAGS+= -DELF_VERBOSE
>> #
>> WITH_DEBUG=
>> WITH_DEBUG_FILES=
>> #
>> # TOOLS_TO_TYPE based on ${TO_TYPE}-xtoolchain-gcc related bintutils...
>> #
>> #CROSS_TOOLCHAIN=${TO_TYPE}-gcc
>> X_COMPILER_TYPE=clang
>> CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/
>> .if ${.MAKE.LEVEL} == 0
>> XCC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>> -fmax-type-align=4
>> XCXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>> -fmax-type-align=4
>> XCPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi
>> -march=armv7a -fmax-type-align=4
>> .export XCC
>> .export XCXX
>> .export XCPP
>> XAS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as
>> XAR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar
>> XLD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld
>> XNM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm
>> XOBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy
>> XOBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump
>> XRANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib
>> XSIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size
>> #NO-SUCH: XSTRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings
>> XSTRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings
>> .export XAS
>> .export XAR
>> .export XLD
>> .export XNM
>> .export XOBJCOPY
>> .export XOBJDUMP
>> .export XRANLIB
>> .export XSIZE
>> .export XSTRINGS
>> .endif
>> #
>> # Host compiler stuff:
>> .if ${.MAKE.LEVEL} == 0
>> CC=/usr/bin/clang -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin
>> CXX=/usr/bin/clang++ -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin
>> CPP=/usr/bin/clang-cpp -B/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin
>> .export CC
>> .export CXX
>> .export CPP
>> AS=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/as
>> AR=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ar
>> LD=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ld
>> NM=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/nm
>> OBJCOPY=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/objcopy
>> OBJDUMP=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/objdump
>> RANLIB=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/ranlib
>> SIZE=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/size
>> #NO-SUCH: STRINGS=/usr/local/${TOOLS_FROM_TYPE}-freebsd/bin/strings
>> STRINGS=/usr/local/bin/${TOOLS_FROM_TYPE}-freebsd-strings
>> .export AS
>> .export AR
>> .export LD
>> .export NM
>> .export OBJCOPY
>> .export OBJDUMP
>> .export RANLIB
>> .export SIZE
>> .export STRINGS
>> .endif
>
> make.conf for during the on-rpi2 port builds now looks like:
>
>> $ more /etc/make.conf
>> WRKDIRPREFIX=/usr/obj/portswork
>> WITH_DEBUG=
>> WITH_DEBUG_FILES=
>> MALLOC_PRODUCTION=
>> #
>> TO_TYPE=armv6
>> TOOLS_TO_TYPE=arm-gnueabi
>> CROSS_BINUTILS_PREFIX=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/
>> .if ${.MAKE.LEVEL} == 0
>> CC=/usr/bin/clang -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>> -fmax-type-align=4
>> CXX=/usr/bin/clang++ -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>> -fmax-type-align=4
>> CPP=/usr/bin/clang-cpp -target ${TO_TYPE}--freebsd11.0-gnueabi -march=armv7a
>> -fmax-type-align=4
>> .export CC
>> .export CXX
>> .export CPP
>> AS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/as
>> AR=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ar
>> LD=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ld
>> NM=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/nm
>> OBJCOPY=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objcopy
>> OBJDUMP=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/objdump
>> RANLIB=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/ranlib
>> SIZE=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/size
>> #NO-SUCH: STRINGS=/usr/local/${TOOLS_TO_TYPE}-freebsd/bin/strings
>> STRINGS=/usr/local/bin/${TOOLS_TO_TYPE}-freebsd-strings
>> .export AS
>> .export AR
>> .export LD
>> .export NM
>> .export OBJCOPY
>> .export OBJDUMP
>> .export RANLIB
>> .export SIZE
>> .export STRINGS
>> .endif
>
>
>
> ===
> Mark Millard
> markmi at dsl-only.net
>
>
>
> _______________________________________________
> freebsd-toolchain@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-toolchain
> To unsubscribe, send any mail to "freebsd-toolchain-unsubscr...@freebsd.org"
signature.asc
Description: Message signed with OpenPGP using GPGMail
