Re: Recent OBJDIR changes can cause host /etc files to leak into DESTDIR -- Fixed in r325416

[Bryan Drewery]

On 11/4/2017 8:27 PM, Bryan Drewery wrote:
> For people doing upgrades of remote systems or building images, you
> should upgrade past r325288 to fix the problem.

I of course meant r325416 from the subject.



signature.asc
Description: OpenPGP digital signature

Recent OBJDIR changes can cause host /etc files to leak into DESTDIR -- Fixed in r325416

[Bryan Drewery]

With the change in r325288, Wed Nov 1 21:22:05 2017 +,
'make distribution' would copy files (listed in /usr/src/etc/*/Makefile)
from the host's /etc/
to the provided DESTDIR rather than from the source tree's etc/
directory.  Generated files like master.passwd and ssh keys were not
prone to this problem.  Files in /etc/* were not copied, only ones in
sub-directories of /etc such as /etc/defaults.  Commands like
mergemaster and etcupdate could hit this problem too.

This was fixed today in r325416.

I also added an assertion to ensure this doesn't happen again in r325405.

This is the same root cause as the recent 'Head build unsafe for /etc
today' thread [1].  Until today I thought the problem was limited to the
AUTO_OBJ builds which I disabled ASAP the other day.  The 'rm -rf /etc'
case was limited to AUTO_OBJ default on introduced on Friday but the
'copy files from host /etc' was with independent of AUTO_OBJ.  It was
not until today that Peter Wemm noticed some bad timestamps on an image
he built and I realized the problem existed.  Note that this bad commit
was tested extensively by me and another person, and reviewed.  This
problem was very subtle.

For people doing upgrades from head on their system the impact is
minimal.  You may have some stale files in /etc.
For people doing upgrades of remote systems or building images, you
should upgrade past r325288 to fix the problem.

I am confident this is the end of the /etc problem, especially with the
assertion that has been added.

[1] The root cause is complex but basically a sub-make starting with a
MAKEOBJDIR in the environment that expanded to /etc/ rather than
/usr/obj/usr/src//etc/.  Because etc/Makefile does not list
its subdirectories in SUBDIR, 'make obj' never creates object
directories for them.  When it installs files for its subdirectories it
just forwards on the 'make distribution' command to them.  Once
src.sys.obj.mk was parsed the MAKEOBJDIR properly expanded to the
expected place but .OBJDIR was not updated internally because the
expected .OBJDIR was never created.  Thus the bogus /etc/ was left
in .OBJDIR rather than expected .CURDIR (like /usr/src/etc/defaults) to
find the files to install from the source checkout instead.

Humble apologies,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: release(7) targets busted

[Bryan Drewery]

On 11/4/2017 9:35 AM, Bryan Drewery wrote:
> On 11/3/2017 9:35 PM, Bryan Drewery wrote:
>>> *** FATAL ERROR: Cannot 'cd' to /root/git/freebsd/release/.. and install 
>>> files to
>>> the temproot environment
>>> 
>>>  
>>
>> If you run into an issue like that with the make release targets, here's
>> a fix:
>> http://people.freebsd.org/~bdrewery/patches/release-objdir-fix.diff
>>
>> I still need to finish some testing of unrelated patches and more cases
>> involving this patch.  I will commit tomorrow.
> 
> Yeah this one breaks the rescue build.
> 
>>
>> If that one has problems an alternative fix is:
>> https://people.freebsd.org/~bdrewery/patches/mm-mtree-makelevel.diff
>>
> 
> This one is safe for now but won't be what I commit.
> 

Fixed in r325412.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: Head build unsafe for /etc today

[Cy Schubert]

In message <201711041723.va4hnanj001...@pdx.rh.cn85.dnsmgr.net>, "Rodney W. 
Gri
mes" writes:
> > On 11/2/2017 8:58 PM, Warner Losh wrote:
> > > FreeBSD has grown too big to test every possible thing before you commit.
> 
> I would say that it has always been too big to test every possible
> thing before a commit.  Breakage is just going to happen, we make
> great efforts to minimize it, but like all risks sooner or later
> your gona have a failure.
> 
> > 
> > The build itself is massive.  I usually forget about release/ and the
> > new 'make packages', external toolchain, "old style" kernel builds, etc.
> Good starting list for "make build-regresion" ?
> 
> > 
> > Steve's concerns have validity.   I do think it's time we have an
> > automated suite to test most build cases for things like bmake upgrades
> > or other high risk changes like META_MODE.
> > 
> > 
> > I'll think about this and add to my list of things to implement.
> 
> I would even go so far as to say this may be what we should be
> running in (a) Jenkins.  Or perhaps a deeper exp-run?

Yes.


-- 
Cheers,
Cy Schubert 
FreeBSD UNIX:     Web:  http://www.FreeBSD.org

The need of the many outweighs the greed of the few.


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: Head build unsafe for /etc today

[Rodney W. Grimes]

> On 11/2/2017 8:58 PM, Warner Losh wrote:
> > FreeBSD has grown too big to test every possible thing before you commit.

I would say that it has always been too big to test every possible
thing before a commit.  Breakage is just going to happen, we make
great efforts to minimize it, but like all risks sooner or later
your gona have a failure.

> 
> The build itself is massive.  I usually forget about release/ and the
> new 'make packages', external toolchain, "old style" kernel builds, etc.
Good starting list for "make build-regresion" ?

> 
> Steve's concerns have validity.   I do think it's time we have an
> automated suite to test most build cases for things like bmake upgrades
> or other high risk changes like META_MODE.
> 
> 
> I'll think about this and add to my list of things to implement.

I would even go so far as to say this may be what we should be
running in (a) Jenkins.  Or perhaps a deeper exp-run?

-- 
Rod Grimes rgri...@freebsd.org
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r325320 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs [breaks lld on zfs: lld uses fallocate]

[Mark Millard]

On 2017-Nov-4, at 4:58 AM, Ed Maste  wrote:

> On 4 November 2017 at 07:41, Andriy Gapon  wrote:
>> On 04/11/2017 12:32, Mark Millard wrote:
>>>  if (int Err = ::posix_fallocate(FD, 0, Size)) {
>>>if (Err != EOPNOTSUPP)
>>>  return std::error_code(Err, std::generic_category());
>>>  }
>> 
>> The commit message that you didn't include into your reply contains some 
>> useful
>> information that authors / maintainers of this code should probably take into
>> account:
>> 
>>>  Please note that EINVAL is used to report that the underlying file system
>>>  does not support the operation (POSIX.1-2008).
>> 
>> Here is a link for that:
>> http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_fallocate.html
> 
> I have no idea how they decided EINVAL was a reasonable errno for this case.

I think they viewed it as a bad fd argument: a reference
into a wrong file system, much like a wrong len (<0) or
offset (<0).

That there is no other means of run-time classifying the
file system(s)(?) was not viewed as sufficient reason
to give it a different classification.

But it is just a guess.

===
Mark Millard
markmi at dsl-only.net

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r325320 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs [breaks lld on zfs: lld uses fallocate]

[Mark Millard]

[The patch allowed the amd64 -> aarch64 cross-buildworld
to complete instead of failing in lld.]

On 2017-Nov-4, at 10:13 AM, Mark Millard  wrote:

> On 2017-Nov-4, at 10:02 AM, Mark Millard  wrote:
> 
> 
>> On 2017-Nov-4, at 4:58 AM, Ed Maste  wrote:
>> 
>>> On 4 November 2017 at 07:41, Andriy Gapon  wrote:
 On 04/11/2017 12:32, Mark Millard wrote:
> if (int Err = ::posix_fallocate(FD, 0, Size)) {
>  if (Err != EOPNOTSUPP)
>return std::error_code(Err, std::generic_category());
> }
 
 The commit message that you didn't include into your reply contains some 
 useful
 information that authors / maintainers of this code should probably take 
 into
 account:
 
> Please note that EINVAL is used to report that the underlying file system
> does not support the operation (POSIX.1-2008).
 
 Here is a link for that:
 http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_fallocate.html
>>> 
>>> I have no idea how they decided EINVAL was a reasonable errno for this case.
>>> 
>>> Mark, can you give this patch a try:
>>> 
>>> diff --git a/contrib/llvm/lib/Support/Unix/Path.inc
>>> b/contrib/llvm/lib/Support/Unix/Path.inc
>>> index 45097eb918b7..67edb46f0025 100644
>>> --- a/contrib/llvm/lib/Support/Unix/Path.inc
>>> +++ b/contrib/llvm/lib/Support/Unix/Path.inc
>>> @@ -427,7 +427,7 @@ std::error_code resize_file(int FD, uint64_t Size) {
>>> // If we have posix_fallocate use it. Unlike ftruncate it always allocates
>>> // space, so we get an error if the disk is full.
>>> if (int Err = ::posix_fallocate(FD, 0, Size)) {
>>> -if (Err != EOPNOTSUPP)
>>> +if (Err != EINVAL && Err != EOPNOTSUPP)
>>> return std::error_code(Err, std::generic_category());

This change allowed the amd64 -> aarch64 cross-buildworld
(and buildkernel) to finish.

Thanks.


===
Mark Millard
markmi at dsl-only.net


___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: nanobsd: g_handleattr: md0 bio_length 24 len 31 -> EFAULT

[Warner Losh]

Can you share your config file?

Warner

On Nov 4, 2017 12:04 PM, "O. Hartmann"  wrote:

> Writing nanobsd (GPT) images into a memory disk results in a lot of
> console errors like
> those shown below on CURRENT ( 12.0-CURRENT FreeBSD 12.0-CURRENT #8
> r325371: Fri Nov  3
> 22:47:33 CET 2017 amd64):
>
> g_handleattr: md0 bio_length 24 len 31 -> EFAULT
> g_handleattr: md0 bio_length 24 len 31 -> EFAULT
> g_handleattr: md0 bio_length 24 len 31 -> EFAULT
>
> I also have errors when writing to the /cfg partition on a running system:
>
> checksum failed: cg 27, cgp: 0xbd184b61 != bp: 0xf751d361
>
> --
> O. Hartmann
>
> Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
> Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

nanobsd: g_handleattr: md0 bio_length 24 len 31 -> EFAULT

[O. Hartmann]

Writing nanobsd (GPT) images into a memory disk results in a lot of console 
errors like
those shown below on CURRENT ( 12.0-CURRENT FreeBSD 12.0-CURRENT #8 r325371: 
Fri Nov  3
22:47:33 CET 2017 amd64): 

g_handleattr: md0 bio_length 24 len 31 -> EFAULT
g_handleattr: md0 bio_length 24 len 31 -> EFAULT
g_handleattr: md0 bio_length 24 len 31 -> EFAULT

I also have errors when writing to the /cfg partition on a running system:

checksum failed: cg 27, cgp: 0xbd184b61 != bp: 0xf751d361

-- 
O. Hartmann

Ich widerspreche der Nutzung oder Übermittlung meiner Daten für
Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG).


pgpkEcPhHooaq.pgp
Description: OpenPGP digital signature

Re: svn commit: r325320 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs [breaks lld on zfs: lld uses fallocate]

[Mark Millard]

On 2017-Nov-4, at 10:02 AM, Mark Millard  wrote:


> On 2017-Nov-4, at 4:58 AM, Ed Maste  wrote:
> 
>> On 4 November 2017 at 07:41, Andriy Gapon  wrote:
>>> On 04/11/2017 12:32, Mark Millard wrote:
 if (int Err = ::posix_fallocate(FD, 0, Size)) {
   if (Err != EOPNOTSUPP)
 return std::error_code(Err, std::generic_category());
 }
>>> 
>>> The commit message that you didn't include into your reply contains some 
>>> useful
>>> information that authors / maintainers of this code should probably take 
>>> into
>>> account:
>>> 
 Please note that EINVAL is used to report that the underlying file system
 does not support the operation (POSIX.1-2008).
>>> 
>>> Here is a link for that:
>>> http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_fallocate.html
>> 
>> I have no idea how they decided EINVAL was a reasonable errno for this case.
>> 
>> Mark, can you give this patch a try:
>> 
>> diff --git a/contrib/llvm/lib/Support/Unix/Path.inc
>> b/contrib/llvm/lib/Support/Unix/Path.inc
>> index 45097eb918b7..67edb46f0025 100644
>> --- a/contrib/llvm/lib/Support/Unix/Path.inc
>> +++ b/contrib/llvm/lib/Support/Unix/Path.inc
>> @@ -427,7 +427,7 @@ std::error_code resize_file(int FD, uint64_t Size) {
>>  // If we have posix_fallocate use it. Unlike ftruncate it always allocates
>>  // space, so we get an error if the disk is full.
>>  if (int Err = ::posix_fallocate(FD, 0, Size)) {
>> -if (Err != EOPNOTSUPP)
>> +if (Err != EINVAL && Err != EOPNOTSUPP)
>>  return std::error_code(Err, std::generic_category());
> 
> I've got a simple buildworld going but I expect that
> it will end up using lld in a form that runs into
> the problem. But I may luck out since I can link a
> trivial main to produce an a.out for amd64.

Actually I take that back: I no longer have
WITH_LLD_IS_LD= as part of my normal amd64
environment. (I did for a time.)

So I will not get the problem.

> It may be appropriate to have notes somewhere about
> what to do for folks that land in the range -r325320
> to whatever revision the updated
> contrib/llvm/lib/Support/Unix/Path.inc ends up at
> and that also have a zfs filesystem context involved.

Explicitly adding to that context-requirement for
having the problem for amd64: and that one has
WITH_LLD_IS_LD= in use.

Of course, for arm64.aarch64 WITH_LLD_IS_LD= is the
normal case and so would be more likely to catch
folks. So this too should be explicit.

> I'll let you know if the build completes vs. not. It
> takes a while since llvm materials are rebuilding.

It should complete since binutils's ld is in use:
I'm not building on aarch64 and reverted to normal
for amd64 some time ago relateive to WITH_LLD_IS_LD= .

===
Mark Millard
markmi at dsl-only.net

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r325320 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs [breaks lld on zfs: lld uses fallocate]

[Mark Millard]

On 2017-Nov-4, at 4:58 AM, Ed Maste  wrote:

> On 4 November 2017 at 07:41, Andriy Gapon  wrote:
>> On 04/11/2017 12:32, Mark Millard wrote:
>>>  if (int Err = ::posix_fallocate(FD, 0, Size)) {
>>>if (Err != EOPNOTSUPP)
>>>  return std::error_code(Err, std::generic_category());
>>>  }
>> 
>> The commit message that you didn't include into your reply contains some 
>> useful
>> information that authors / maintainers of this code should probably take into
>> account:
>> 
>>>  Please note that EINVAL is used to report that the underlying file system
>>>  does not support the operation (POSIX.1-2008).
>> 
>> Here is a link for that:
>> http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_fallocate.html
> 
> I have no idea how they decided EINVAL was a reasonable errno for this case.
> 
> Mark, can you give this patch a try:
> 
> diff --git a/contrib/llvm/lib/Support/Unix/Path.inc
> b/contrib/llvm/lib/Support/Unix/Path.inc
> index 45097eb918b7..67edb46f0025 100644
> --- a/contrib/llvm/lib/Support/Unix/Path.inc
> +++ b/contrib/llvm/lib/Support/Unix/Path.inc
> @@ -427,7 +427,7 @@ std::error_code resize_file(int FD, uint64_t Size) {
>   // If we have posix_fallocate use it. Unlike ftruncate it always allocates
>   // space, so we get an error if the disk is full.
>   if (int Err = ::posix_fallocate(FD, 0, Size)) {
> -if (Err != EOPNOTSUPP)
> +if (Err != EINVAL && Err != EOPNOTSUPP)
>   return std::error_code(Err, std::generic_category());

I've got a simple buildworld going but I expect that
it will end up using lld in a form that runs into
the problem. But I may luck out since I can link a
trivial main to produce an a.out for amd64.

It may be appropriate to have notes somewhere about
what to do for folks that land in the range -r325320
to whatever revision the updated
contrib/llvm/lib/Support/Unix/Path.inc ends up at
and that also have a zfs filesystem context involved.

I'll let you know if the build completes vs. not. It
takes a while since llvm materials are rebuilding.

===
Mark Millard
markmi at dsl-only.net

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r325320 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs [breaks lld on zfs: lld uses fallocate]

[Mark Millard]

On 2017-Nov-4, at 5:04 AM, Andriy Gapon  wrote:

> On 04/11/2017 13:41, Andriy Gapon wrote:
>> On 04/11/2017 12:32, Mark Millard wrote:
>>>  if (int Err = ::posix_fallocate(FD, 0, Size)) {
>>>if (Err != EOPNOTSUPP)
>>>  return std::error_code(Err, std::generic_category());
>>>  }
>> 
>> The commit message that you didn't include into your reply contains some 
>> useful
>> information that authors / maintainers of this code should probably take into
>> account:
>> 
>>>  Please note that EINVAL is used to report that the underlying file system
>>>  does not support the operation (POSIX.1-2008).
>> 
>> Here is a link for that:
>> http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_fallocate.html

I had no doubt about the change being required or
per-specification and did not think to list the
reference.

It was more that world's use of fallocate was not
reviewed and possibly adjusted first that caused
problems.

FYI:

It appears that sqlite3 might use fallocate.
And heimdal (via its sqlite3.c). I've not
reviewed the details for any of the below
that do not trace to the lld issue:

# grep -rl "fallocate" /usr/src/* | more
/usr/src/contrib/sqlite3/configure.ac
/usr/src/contrib/sqlite3/configure
/usr/src/contrib/sqlite3/sqlite3.c
/usr/src/contrib/netbsd-tests/lib/libc/sys/t_posix_fallocate.c
/usr/src/contrib/compiler-rt/lib/dfsan/libc_ubuntu1404_abilist.txt
/usr/src/contrib/compiler-rt/include/sanitizer/linux_syscall_hooks.h
/usr/src/contrib/llvm/lib/Support/Unix/Path.inc
/usr/src/contrib/pjdfstest/configure.ac
/usr/src/contrib/pjdfstest/tests/misc.sh
/usr/src/contrib/pjdfstest/pjdfstest.c
/usr/src/contrib/openbsm/NEWS
/usr/src/contrib/openbsm/etc/audit_event
/usr/src/crypto/heimdal/lib/sqlite/sqlite3.c
/usr/src/lib/libc/sys/posix_fallocate.2
/usr/src/lib/libc/sys/Symbol.map
/usr/src/lib/libc/sys/Makefile.inc
/usr/src/lib/libc/tests/sys/Makefile
/usr/src/lib/clang/include/llvm/Config/config.h
/usr/src/share/man/man9/VOP_ALLOCATE.9
/usr/src/sys/compat/freebsd32/syscalls.master
/usr/src/sys/compat/freebsd32/freebsd32_syscalls.c
/usr/src/sys/compat/freebsd32/freebsd32_sysent.c
/usr/src/sys/compat/freebsd32/capabilities.conf
/usr/src/sys/compat/freebsd32/freebsd32_misc.c
/usr/src/sys/compat/freebsd32/freebsd32_proto.h
/usr/src/sys/compat/freebsd32/freebsd32_syscall.h
/usr/src/sys/compat/freebsd32/freebsd32_systrace_args.c
/usr/src/sys/compat/cloudabi/cloudabi_file.c
/usr/src/sys/compat/linux/linux_file.c
/usr/src/sys/i386/linux/linux_sysent.c
/usr/src/sys/i386/linux/linux_proto.h
/usr/src/sys/i386/linux/linux_syscalls.c
/usr/src/sys/i386/linux/linux_systrace_args.c
/usr/src/sys/i386/linux/linux_syscall.h
/usr/src/sys/i386/linux/syscalls.master
/usr/src/sys/sys/fcntl.h
/usr/src/sys/sys/syscall.h
/usr/src/sys/sys/syscall.mk
/usr/src/sys/sys/syscallsubr.h
/usr/src/sys/sys/sysproto.h
/usr/src/sys/kern/init_sysent.c
/usr/src/sys/kern/syscalls.master
/usr/src/sys/kern/systrace_args.c
/usr/src/sys/kern/capabilities.conf
/usr/src/sys/kern/vfs_syscalls.c
/usr/src/sys/kern/syscalls.c
/usr/src/sys/amd64/linux32/linux32_syscall.h
/usr/src/sys/amd64/linux32/linux32_systrace_args.c
/usr/src/sys/amd64/linux32/syscalls.master
/usr/src/sys/amd64/linux32/linux32_syscalls.c
/usr/src/sys/amd64/linux32/linux32_sysent.c
/usr/src/sys/amd64/linux32/linux32_proto.h
/usr/src/sys/amd64/linux/linux_proto.h
/usr/src/sys/amd64/linux/syscalls.master
/usr/src/sys/amd64/linux/linux_syscalls.c
/usr/src/sys/amd64/linux/linux_systrace_args.c
/usr/src/sys/amd64/linux/linux_syscall.h
/usr/src/sys/amd64/linux/linux_sysent.c
/usr/src/tests/sys/pjdfstest/config.h


> My response above is quite dry, so I want to add this.
> Thank you very much for the deep analysis.
> I am sorry for the trouble that my change caused, but I think that its root
> cause lies elsewhere (lld, posix).

You are welcome.

-- 
Andriy Gapon

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: release(7) targets busted

[Bryan Drewery]

On 11/3/2017 9:35 PM, Bryan Drewery wrote:
>> *** FATAL ERROR: Cannot 'cd' to /root/git/freebsd/release/.. and install 
>> files to
>> the temproot environment
>>  
>> 
> 
> If you run into an issue like that with the make release targets, here's
> a fix:
> http://people.freebsd.org/~bdrewery/patches/release-objdir-fix.diff
> 
> I still need to finish some testing of unrelated patches and more cases
> involving this patch.  I will commit tomorrow.

Yeah this one breaks the rescue build.

> 
> If that one has problems an alternative fix is:
> https://people.freebsd.org/~bdrewery/patches/mm-mtree-makelevel.diff
> 

This one is safe for now but won't be what I commit.

-- 
Regards,
Bryan Drewery



signature.asc
Description: OpenPGP digital signature

Re: Head build unsafe for /etc today

[Pete Wright]

On 11/03/2017 13:14, Bryan Drewery wrote:

On 11/2/2017 8:58 PM, Warner Losh wrote:

FreeBSD has grown too big to test every possible thing before you commit.

The build itself is massive.  I usually forget about release/ and the
new 'make packages', external toolchain, "old style" kernel builds, etc.

Steve's concerns have validity.   I do think it's time we have an
automated suite to test most build cases for things like bmake upgrades
or other high risk changes like META_MODE.


I'll think about this and add to my list of things to implement.

hey Bryan - this sounds like a pretty awesome task to take on, i'd be 
willing to help out with this effort as well.  if/when you come up with 
a plan would it be possible to ping freebsd-current@ so that i can see 
if there is anything i'll be able to help out with (assuming there isn't 
already a wiki or list somewhere i can reference to see where i may be 
able to chip in)?


cheers!
-pete

--
Pete Wright
p...@nomadlogic.org
@nomadlogicLA

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

[base packages] [r325303 -> r325351] install: hcsecd.conf: Permission denied

[Boris Samorodov]

Hi All,

I create packages (as an ordinary user with command "make packages")
for HEAD once a day. Today I've got a regression:
--- rev. 325351 regression ---
install -U -M /usr/obj/usr/src/amd64.amd64/worldstage//METALOG -D
/usr/obj/usr/src/amd64.amd64/worldstage -T package=runtime -o root  -g
wheel -m 600  hcsecd.conf
/usr/obj/usr/src/amd64.amd64/worldstage/etc/bluetooth/hcsecd.conf
install: hcsecd.conf: Permission denied
*** Error code 71

Stop.
make[9]: stopped in /usr/src/etc/bluetooth
---

Files in question are:
---
% ls -ld /usr/obj/usr/src/amd64.amd64/worldstage/etc/bluetooth
drwxr-xr-x  2 bsam  wheel  512  3 нояб. 06:35
/usr/obj/usr/src/amd64.amd64/worldstage/etc/bluetooth

% ls -l /usr/obj/usr/src/amd64.amd64/worldstage/etc/bluetooth
total 12
-rw---  1 bsam  wheel  1343  3 нояб. 06:35 hcsecd.conf
-rw-r--r--  1 bsam  wheel   305  3 нояб. 06:35 hosts
-r--r--r--  1 bsam  wheel  1110  3 нояб. 06:35 protocols
---

Yesterday rev. 325303 worked fine.

-- 
WBR, bsam
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r325320 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs [breaks lld on zfs: lld uses fallocate]

[Andriy Gapon]

On 04/11/2017 13:58, Ed Maste wrote:
> I have no idea how they decided EINVAL was a reasonable errno for this case.

I completely agree.  That's a weird choice that I have not seen for any other 
API.

-- 
Andriy Gapon
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r325320 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs [breaks lld on zfs: lld uses fallocate]

[Andriy Gapon]

On 04/11/2017 13:41, Andriy Gapon wrote:
> On 04/11/2017 12:32, Mark Millard wrote:
>>   if (int Err = ::posix_fallocate(FD, 0, Size)) {
>> if (Err != EOPNOTSUPP)
>>   return std::error_code(Err, std::generic_category());
>>   }
> 
> The commit message that you didn't include into your reply contains some 
> useful
> information that authors / maintainers of this code should probably take into
> account:
> 
>>   Please note that EINVAL is used to report that the underlying file system
>>   does not support the operation (POSIX.1-2008).
> 
> Here is a link for that:
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_fallocate.html
> 

My response above is quite dry, so I want to add this.
Thank you very much for the deep analysis.
I am sorry for the trouble that my change caused, but I think that its root
cause lies elsewhere (lld, posix).

-- 
Andriy Gapon
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r325320 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs [breaks lld on zfs: lld uses fallocate]

[Ed Maste]

On 4 November 2017 at 07:41, Andriy Gapon  wrote:
> On 04/11/2017 12:32, Mark Millard wrote:
>>   if (int Err = ::posix_fallocate(FD, 0, Size)) {
>> if (Err != EOPNOTSUPP)
>>   return std::error_code(Err, std::generic_category());
>>   }
>
> The commit message that you didn't include into your reply contains some 
> useful
> information that authors / maintainers of this code should probably take into
> account:
>
>>   Please note that EINVAL is used to report that the underlying file system
>>   does not support the operation (POSIX.1-2008).
>
> Here is a link for that:
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_fallocate.html

I have no idea how they decided EINVAL was a reasonable errno for this case.

Mark, can you give this patch a try:

diff --git a/contrib/llvm/lib/Support/Unix/Path.inc
b/contrib/llvm/lib/Support/Unix/Path.inc
index 45097eb918b7..67edb46f0025 100644
--- a/contrib/llvm/lib/Support/Unix/Path.inc
+++ b/contrib/llvm/lib/Support/Unix/Path.inc
@@ -427,7 +427,7 @@ std::error_code resize_file(int FD, uint64_t Size) {
   // If we have posix_fallocate use it. Unlike ftruncate it always allocates
   // space, so we get an error if the disk is full.
   if (int Err = ::posix_fallocate(FD, 0, Size)) {
-if (Err != EOPNOTSUPP)
+if (Err != EINVAL && Err != EOPNOTSUPP)
   return std::error_code(Err, std::generic_category());
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r325320 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs [breaks lld on zfs: lld uses fallocate]

[Andriy Gapon]

On 04/11/2017 12:32, Mark Millard wrote:
>   if (int Err = ::posix_fallocate(FD, 0, Size)) {
> if (Err != EOPNOTSUPP)
>   return std::error_code(Err, std::generic_category());
>   }

The commit message that you didn't include into your reply contains some useful
information that authors / maintainers of this code should probably take into
account:

>   Please note that EINVAL is used to report that the underlying file system
>   does not support the operation (POSIX.1-2008).

Here is a link for that:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/posix_fallocate.html

-- 
Andriy Gapon
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: ld (a.k.a. ld.lld) for amd64 -r325369 -> aarch64 cross buildworld is messed up: ld just says "Invalid argument" [vs. binutils ld: R_AARCH64_ABS64 used with TLS symbol]

[Mark Millard]

It turns out that svn commit: r325320
breaks lld.

lld has code that uses fallocate and
now can fail (stop the link) on zfs.

I've sent a separate reply to the
notice below that gives the details.
I added the [...] part of the title.

Re: svn commit: r325320 -  head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs 
[breaks lld on zfs: lld uses fallocate]

See, for example,

https://lists.freebsd.org/pipermail/freebsd-toolchain/2017-November/003413.html

===
Mark Millard
markmi at dsl-only.net




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: svn commit: r325320 - head/sys/cddl/contrib/opensolaris/uts/common/fs/zfs [breaks lld on zfs: lld uses fallocate]

[Mark Millard]

> Author: avg
> Date: Thu Nov  2 13:49:08 2017
> New Revision: 325320
> URL: 
> https://svnweb.freebsd.org/changeset/base/325320
> 
> 
> Log:
>   Disable posix_fallocate(2) for ZFS
> . . .

Turns out lld uses fallocate and so can fail
on zfs now.

The following is the lld for a amd64 -> aarch64
cross-buildworld.

/usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp/usr/bin/ld: 
error: cannot open output file a.out: Invalid argument

This resulted from:

Breakpoint 5, 0x00cf1cd1 in llvm::sys::fs::resize_file(int, unsigned 
long) ()
(gdb) disass
Dump of assembler code for function _ZN4llvm3sys2fs11resize_fileEim:
. . .
   0x00cf1ce5 <+21>:callq  0x1ad5880 
. . .

via the error status return value handling. It ends up with:

Breakpoint 3, 0x0041c6e4 in lld::elf::error(llvm::Twine const&) ()
(gdb) bt
#0  0x0041c6e4 in lld::elf::error(llvm::Twine const&) ()
#1  0x004113b1 in void 
lld::elf::LinkerDriver::link >(llvm::opt::InputArgList&) ()
#2  0x0040be3f in lld::elf::LinkerDriver::main(llvm::ArrayRef, bool) ()
#3  0x0040ae89 in lld::elf::link(llvm::ArrayRef, bool, 
llvm::raw_ostream&) ()
#4  0x0054cd61 in main ()


Progressing from posix_fallocate's call to its caller
and so on:

# grep -r "fallocate" /usr/src/contrib/llvm/ | more
/usr/src/contrib/llvm/lib/Support/Unix/Path.inc:  // If we have posix_fallocate 
use it. Unlike ftruncate it always allocates
/usr/src/contrib/llvm/lib/Support/Unix/Path.inc:  if (int Err = 
::posix_fallocate(FD, 0, Size)) {

Is called by:

std::error_code resize_file(int FD, uint64_t Size) {
#if defined(HAVE_POSIX_FALLOCATE)
  // If we have posix_fallocate use it. Unlike ftruncate it always allocates
  // space, so we get an error if the disk is full.
  if (int Err = ::posix_fallocate(FD, 0, Size)) {
if (Err != EOPNOTSUPP)
  return std::error_code(Err, std::generic_category());
  }
#endif
  // Use ftruncate as a fallback. It may or may not allocate space. At least on
  // OS X with HFS+ it does.
  if (::ftruncate(FD, Size) == -1)
return std::error_code(errno, std::generic_category());

  return std::error_code();
}

# grep -r "resize_file" /usr/src/contrib/llvm/ | more
/usr/src/contrib/llvm/lib/Support/FileOutputBuffer.cpp:  EC = 
sys::fs::resize_file(FD, Size);
/usr/src/contrib/llvm/lib/Support/Unix/Path.inc:std::error_code resize_file(int 
FD, uint64_t Size) {
/usr/src/contrib/llvm/lib/Support/Windows/Path.inc:std::error_code 
resize_file(int FD, uint64_t Size) {
/usr/src/contrib/llvm/include/llvm/Support/FileSystem.h:std::error_code 
resize_file(int FD, uint64_t Size);

Is called by:

ErrorOr
FileOutputBuffer::create(StringRef FilePath, size_t Size, unsigned Flags) {
  // Check file is not a regular file, in which case we cannot remove it.
. . .
#ifndef LLVM_ON_WIN32
  // . . .
  EC = sys::fs::resize_file(FD, Size);
  if (EC)
return EC;
#endif

Is called by:

std::error_code elf::tryCreateFile(StringRef Path) {
  if (Path.empty())
return std::error_code();
  return FileOutputBuffer::create(Path, 1).getError();
}

Is called by:

template  void LinkerDriver::link(opt::InputArgList ) {
  SymbolTable Symtab;
  elf::Symtab::X = 
  Target = getTarget();

  Config->MaxPageSize = getMaxPageSize(Args);
  Config->ImageBase = getImageBase(Args);

  // Default output filename is "a.out" by the Unix tradition.
  if (Config->OutputFile.empty())
Config->OutputFile = "a.out";

  // Fail early if the output file or map file is not writable. If a user has a
  // long link, e.g. due to a large LTO link, they do not wish to run it and
  // find that it failed because there was a mistake in their command-line.
  if (auto E = tryCreateFile(Config->OutputFile))
error("cannot open output file " + Config->OutputFile + ": " + E.message());


And the error call is then made once
tryCreateFile returns.

===
Mark Millard
markmi at dsl-only.net

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: ld (a.k.a. ld.lld) for amd64 -r325369 -> aarch64 cross buildworld is messed up: ld just says "Invalid argument" [vs. binutils ld: R_AARCH64_ABS64 used with TLS symbol]

[Mark Millard]

[Top post: contrast with a combination
using aarch64-binutils-2.28,1 that does
build.]

I've found one combination that works:
Use of /usr/local/aarch64-freebsd/bin/*
binutils materials mixed with use of
WITHOUT_DEBUG_FILES= 

With that it was able to build libc.so.7
and, so, continue with the build --all
the way to completion.

For ld.lld, use of WITHOUT_DEBUG_FILES=
still failed the same way as the original
use of WITH_DEBUG_FILES= .


Prior notes:

> On 2017-Nov-3, at 10:36 PM, Mark Millard  wrote:
> 
> [Forcing use of /usr/local/aarch64-freebsd/bin/ld
> and other aarch64-binutils-2.28,1 material for the
> aarch64 links: that fails too, but with messages
> that are more informative (but might be
> independent): "R_AARCH64_ABS64 used with TLS
> symbol".]
> 
>> On 2017-Nov-3, at 9:24 PM, Mark Millard  wrote:
>> 
>> Context:
>> 
>> # uname -apKU
>> FreeBSD FreeBSDx64OPC 12.0-CURRENT FreeBSD 12.0-CURRENT  r325369M  amd64 
>> amd64 1200052 1200052
>> 
>> 
>> Problem:
>> 
>> While I show the buildworld error report later,
>> even simple ld commands that list a input file
>> to the amd64 -> aarch64 cross build's ld (a.k.a.
>> ld.lld) file get messages like:
>> 
>> /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp/usr/bin/ld: 
>> error: cannot open output file a.out: Invalid argument
>> 
>> The "cannot open output file" is always part of
>> the message and tracks any -o NAME given. But
>> not listing -o NAME at all gives the above:
>> a reference to a.out .
> 
> Well I tried forcing the use of
> aarch64-binutils-2.28,1
> ( /usr/local/aarch64-freebsd/bin/* )
> materials. The result was failure at
> the same place but with messages that
> look to be more informative:
> 
> --- libc.so.7.full ---
> building shared library libc.so.7
> /usr/local/aarch64-freebsd/bin/ld: getutxent.pico(.debug_info+0x3c): 
> R_AARCH64_ABS64 used with TLS symbol udb
> /usr/local/aarch64-freebsd/bin/ld: getutxent.pico(.debug_info+0x59): 
> R_AARCH64_ABS64 used with TLS symbol uf
> /usr/local/aarch64-freebsd/bin/ld: utxdb.pico(.debug_info+0x5b): 
> R_AARCH64_ABS64 used with TLS symbol futx_to_utx.ut
> /usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.pico(.debug_info+0x3d): 
> R_AARCH64_ABS64 used with TLS symbol __je_tsd_tls
> /usr/local/aarch64-freebsd/bin/ld: jemalloc_tsd.pico(.debug_info+0x134c): 
> R_AARCH64_ABS64 used with TLS symbol __je_tsd_initialized
> /usr/local/aarch64-freebsd/bin/ld: 
> cxa_thread_atexit_impl.pico(.debug_info+0x3c): R_AARCH64_ABS64 used with TLS 
> symbol dtors
> /usr/local/aarch64-freebsd/bin/ld: xlocale.pico(.debug_info+0x407): 
> R_AARCH64_ABS64 used with TLS symbol __thread_locale
> /usr/local/aarch64-freebsd/bin/ld: setrunelocale.pico(.debug_info+0x3d): 
> R_AARCH64_ABS64 used with TLS symbol _ThreadRuneLocale
> cc: error: linker command failed with exit code 1 (use -v to see invocation)
> 
> Of course it may be an independent issue.
> In fact this seems familiar for .debug_info
> material (from before use of ld.lld ).
> 
>> The above message is after eliminating all command line options
>> and it still got such a message. For reference:
>> 
>> # file 
>> /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp/usr/bin/ld
>> /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp/usr/bin/ld: 
>> symbolic link to ld.lld
>> 
>> # file 
>> /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp/usr/bin/ld.lld
>> /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp/usr/bin/ld.lld:
>>  ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), statically linked, 
>> for FreeBSD 12.0 (1200052), FreeBSD-style, not stripped
>> 
>> (Could statically linked programs still
>> have problems?)
>> 
>> --- libc.so.7.full ---
>> building shared library libc.so.7
>> /usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp/usr/bin/ld: 
>> error: cannot open output file libc.so.7.full: Invalid argument
>> cc: error: linker command failed with exit code 1 (use -v to see invocation)
>> *** [libc.so.7.full] Error code 1
>> 
>> make[4]: stopped in /usr/src/lib/libc
>> .ERROR_TARGET='libc.so.7.full'
>> .ERROR_META_FILE='/usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/lib/libc/libc.so.7.full.meta'
>> .MAKE.LEVEL='4'
>> MAKEFILE=''
>> .MAKE.MODE='meta missing-filemon=yes missing-meta=yes silent=yes verbose'
>> _ERROR_CMD='@echo building shared library libc.so.7; @rm -f libc.so.7 
>> libc.so; cc -mcpu=cortex-a53 -target aarch64-unknown-freebsd12.0 
>> --sysroot=/usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp 
>> -B/usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/tmp/usr/bin  
>> -nodefaultlibs -Wl,--version-script=Version.map  -shared -Wl,-x 
>> -Wl,--fatal-warnings -Wl,--warn-shared-textrel  -o libc.so.7.full 
>> -Wl,-soname,libc.so.7  `NM='nm' NMFLAGS='' lorder machdep_ldisQ.pico 
>> bt_close.pico bt_conv.pico bt_debug.pico bt_delete.pico bt_get.pico 
>> bt_open.pico bt_overflow.pico