[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[Paul D. Smith]

Update of bug #52922 (project make):

  Status:None => Fixed  
 Open/Closed:Open => Closed 
   Fixed Release:None => SCM

___

Follow-up Comment #13:

This should be fixed with the latest changes pushed today.


___

Reply to this item at:

  

___
Message sent via Savannah
https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[Paul D. Smith]

Follow-up Comment #12, bug #52922 (project make):

I'm looking at this and Dmitry's change in bug #60774 this week.  I got
sidetracked by some issues being raised for POSIX Issue 8.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[Thomas Munro]

Follow-up Comment #11, bug #52922 (project make):

[comment #10 comment #10:]
> Is there a number of ports which have gnu make specific makefiles?

Yes.  Depending how you count there's about ~30k ports and about ~4k of them
have a build-time dependency on GNU make (I searched the FreeBSD ports tree
for "USES=.*gmake").

I'm not a ports maintainer though, I work on PostgreSQL which brought me here.
 It happens to use GNU makefiles (BSD make definitely doesn't like them), and
gets built and tested after every commit on about 10 operating systems:
https://buildfarm.postgresql.org/cgi-bin/show_status.pl

Before patches reach that stage, developer machines and CI systems also run on
several operating systems including Macs and FreeBSD, and it'd be nice to be
able to use eg make check-world -jN -Otarget to speed things up without
garbling the output.

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[Dmitry Goncharov]

Follow-up Comment #10, bug #52922 (project make):

> I considered fixing that in FreeBSD, but I'm running build farm stuff across
a bunch of OSes that all have this problem hence desire to fix it here.


Gentlemen, can you please describe the state of affairs in the bsd world in
regards to gnu make? Is there a number of ports which have gnu make specific
makefiles? Is there a number of ports which have portable makefiles and you
prefer to build them with gnu make?
In other words, what causes you to use gnu make to build bsd ports?

___

Reply to this item at:

  

___
  Сообщение отправлено по Savannah
  https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[John Hood]

Follow-up Comment #9, bug #52922 (project make):

I am still posting too late, but I think what I'm seeing agrees with this--
output directed to a pipe fails but output to a file is OK.  I'll check more
carefully tomorrow and update if I'm still wrong.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[Thomas Munro]

Follow-up Comment #8, bug #52922 (project make):

Yeah, it's implemented at VFS level, so it needs a vnode:

https://github.com/freebsd/freebsd-src/blob/7326e8589cc21431d62f25802eac7c5dd6f74122/sys/kern/kern_descrip.c#L617

I considered fixing that in FreeBSD, but I'm running build farm stuff across a
bunch of OSes that all have this problem hence desire to fix it here.

I haven't look at that other patch yet (at a quick glimpse I'm confused about
why it's testing every time, but I probably need to actually try it out to
have a useful opinion), but I'll be very happy however this gets fixed!

___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[John Hood]

Follow-up Comment #7, bug #52922 (project make):

Oops, I'm posting too late at night.  The problem is still there on FreeBSD.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[John Hood]

Follow-up Comment #6, bug #52922 (project make):

For the record, this issue seems to be gone on FreeBSD 13.0-- the FreeBSD port
is now using GNU Make 4.3, and I don't see this issue now.  The port still
doesn't have any patches that seem relevant to this.

It's possible it got fixed on the kernel side, I don't know.


___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[Dmitry Goncharov]

Follow-up Comment #5, bug #52922 (project make):

> Maybe this enhancement (using a separate lock file for submakes that are
redirected) is something that simply won't be supported on those systems as
the effort is not worth the reward.

This confuses users on linux as well.

___

Reply to this item at:

  

___
  Сообщение отправлено по Savannah
  https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[Dmitry Goncharov]

Follow-up Comment #4, bug #52922 (project make):

There was a recent discussion about this. See
https://savannah.gnu.org/bugs/?60774 and the mailing list.

___

Reply to this item at:

  

___
  Сообщение отправлено по Savannah
  https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[Thomas Munro]

Follow-up Comment #3, bug #52922 (project make):

Here's a draft patch to create a lock file if an initial test of the output
descriptor fails.  Tested on FreeBSD.

I don't currently have any good ideas for how to detect redirection and use a
separate lock file.


(file #51900)
___

Additional Item Attachment:

File name: 0001-Fix-O-portability.patch   Size:4 KB
   




___

Reply to this item at:

  

___
  Message sent via Savannah
  https://savannah.gnu.org/

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[John Hood]

Follow-up Comment #2, bug #52922 (project make):

I spent a while longer looking at this, and being reminded of how messy UNIX
locking is.

* On FreeBSD, none of fcntl-locks, flock(), and lockf() work on pipes.

* POSIX reminds me that write() is atomic on regular files.  If make were to
carefully accumulate and concatenate output for stdout and stderr into
buffers, and then write() entire buffers, make might not need locking at all
for regular files.  Of course, buffers of unbounded size would be a problem
here.

* Network file systems have varying support for locking.

I think having either a lock file or a POSIX semaphore to share with children
is probably the best way to handle this.  This loses the
redirected-children-dont-lock property, but in output_dump() make is pretty
much just copying the temporary output and doing fairly simple sprintfs for
its own output.  That should go fairly fast most of the time, and I'd guess
concurrency wouldn't be a major problem, except for really pathological cases
such as very large numbers of targets or very large output from targets.


___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[Paul D. Smith]

Follow-up Comment #1, bug #52922 (project make):

Yes, I'm aware of this as I've seen it happen on MacOS systems myself.  Thanks
for filing the bug, as I never got around to it.

I've been considering different ways to address the issue including using a
lock file on systems where locking of pipes fails.  Using a file that all
makes have open is itself not an issue, because the parent make can simply
open the file and leave it open when invoking submakes.  We just need to
create a way for the parent to notify the submakes which FD to use for
locking.  We already do similar things for, for example, the jobserver pipes
so this is not breaking new ground.

However, using stdout has the nice feature of automatically working correctly
if you redirect output (for example, if you invoke a submake and redirect it's
output to a file--that submake doesn't need to participate in the same output
locking that the parent does and since the stdout is a different file
descriptor that just magically happens).

So far I haven't found the time to sit down and really think about how this
could work correctly using a lock file instead of stdout, or what extra effort
make needs to go to.  Maybe this enhancement (using a separate lock file for
submakes that are redirected) is something that simply won't be supported on
those systems as the effort is not worth the reward.

___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make

[bug #52922] -Otarget does not function properly and errors on FreeBSD, with stdout to a pipe

[John Hood]

URL:
  

 Summary: -Otarget does not function properly and errors on
FreeBSD, with stdout to a pipe
 Project: make
Submitted by: cgull
Submitted on: Thu 18 Jan 2018 02:35:34 AM UTC
Severity: 3 - Normal
  Item Group: Bug
  Status: None
 Privacy: Public
 Assigned to: None
 Open/Closed: Open
 Discussion Lock: Any
   Component Version: 4.2.1
Operating System: POSIX-Based
   Fixed Release: None
   Triage Status: None

___

Details:

FreeBSD 11.1, GNU Make version 4.2.1 (installed from FreeBSD packages; the
distribution patches are small and don't seem relevant here).

On this system, if GNU Make is invoked with options '-j32 -Otarget' and
standard output is redirected to a pipe, Make prints the following errors:

jhood@jhood_lfs:/tmp 0 $ gmake -j32 -Otarget | cat
fcntl(): Bad file descriptor
touch foo
fcntl(): Bad file descriptor
touch bar
jhood@jhood_lfs:/tmp 0 $ uname -a

This is because FreeBSD only supports fcntl(..., F_SETLKW, ...) on a few types
of files, including regular files and ttys.  Pipes and sockets are not
supported.

Locking for other than regular files is not required for POSIX compliance, see
http://pubs.opengroup.org/onlinepubs/9699919799/

I think the impact is fairly minimal.  The error message appears, and locking
doesn't happen, but since GNU make gathers output and writes it more
atomically in this mode, -Otarget should still mostly work.

This isn't a problem on Linux, which does seem to support F_SETLKW on pipes
and sockets at least.  I do see the problem on macOS, and I would expect that
the problem exists on most/all BSD-derived systems and probably various other
POSIX environments as well.

I don't see an obvious easy solution; I think this is intended to work with
child invocations of make, and so it's best to use a file that all of them
already have open, like stdout.  BSD systems could probably use flock()
instead, but choosing flock() over fcntl() in Configure would require figuring
out which is the better implementation on a given system.





___

Reply to this item at:

  

___
  Message sent via/by Savannah
  http://savannah.gnu.org/


___
Bug-make mailing list
Bug-make@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-make