[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290
--- Comment #18 from Tatsuki Makino ---
(In reply to Alastair Hogge from comment #17)
>(In reply to Ivan Rozhuk from comment #15)
>> $${PORTSDIR -> ${PORTSDIR
$${PORTSDIR:-/usr/ports} -> ${PORTSDIR:U/usr/ports}
It seems that the way default values are given has been changed from sh to make
side?
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290
--- Comment #17 from Alastair Hogge ---
(In reply to Ivan Rozhuk from comment #15)
> $${PORTSDIR -> ${PORTSDIR
Results in:
> MAKE="make" sh /usr/src/sys/conf/newvers.sh KOTO
> --- vers.o ---
> cc -target x86_64-unknown-freebsd15.0
> --sysroot=/net/fafnir/obj/usr/src/amd64.amd64/tmp
> -B/net/fafnir/obj/usr/src/amd64.amd64/tmp/usr/bin -c -O2 -pipe
> -fno-strict-aliasing -nostdinc -I. -I/usr/src/sys
> -I/usr/src/sys/contrib/ck/include -I/usr/src/sys/contrib/libfdt -D_KERNEL
> -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common
> -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer
> -fdebug-prefix-map=./machine=/usr/src/sys/amd64/include
> -fdebug-prefix-map=./x86=/usr/src/sys/x86/include
> -fdebug-prefix-map=./i386=/usr/src/sys/i386/include -mcmodel=kernel
> -mno-red-zone -mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables
> -ffreestanding -fwrapv -fstack-protector -Wall -Wstrict-prototypes
> -Wmissing-prototypes -Wpointer-arith -Wcast-qual -Wundef -Wno-pointer-sign
> -D__printf__=__freebsd_kprintf__ -Wmissing-include-dirs
> -fdiagnostics-show-option -Wno-unknown-pragmas
> -Wno-error=tautological-compare -Wno-error=empty-body
> -Wno-error=parentheses-equality -Wno-error=unused-function
> -Wno-error=pointer-sign -Wno-error=shift-negative-value
> -Wno-address-of-packed-member -Wno-format-zero-length -mno-aes -mno-avx
> -std=gnu99 -Werror versc
> --- kernel ---
> linking kernel
> objcopy --strip-debug kernel
> text data bssdechex filename
> 11990200 1052822 3137536 16180558 0xf6e54e kernel
> make[2]: Unknown modifier "-/usr/ports"
> make[2]: Unknown modifier "-/usr/ports"
> make[2]: Unknown modifier "-/usr/ports"
> --- all ---
> ===> Ports module graphics/drm-515-kmod (all)
> cd /graphics/drm-515-kmod; env -u CC -u CXX -u CPP -u MAKESYSPATH -u
> MK_AUTO_OBJ -u MAKEOBJDIR MAKEFLAGS="-D NO_MODULES_OBJ KERNEL=kernel
> TARGET=amd64 TARGET_ARCH=amd64" SYSDIR=/usr/src/sys
> PATH=/net/fafnir/obj/usr/src/amd64.amd64/tmp/bin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/usr/sbin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/usr/bin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/legacy/bin:/net/fafnir/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin
> SRC_BASE=/usr/src OSVERSION=157
> WRKDIRPREFIX=/net/fafnir/obj/usr/src/amd64.amd64/sys/KOTO make clean build
> cd: /graphics/drm-515-kmod: No such file or directory
> *** [all] Error code 2
>
> make[2]: stopped in /net/fafnir/obj/usr/src/amd64.amd64/sys/KOTO
> 1 error
>
> make[2]: stopped in /net/fafnir/obj/usr/src/amd64.amd64/sys/KOTO
> 42.12 real 281.49 user 132.20 sys
>
> make[1]: stopped in /usr/src
>
> make: stopped in /usr/src
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 Ivan Rozhuk changed: What|Removed |Added Attachment #234277|maintainer-approval?| Flags|| Attachment #234277|0 |1 is obsolete|| Attachment #247166||maintainer-approval? Flags|| --- Comment #16 from Ivan Rozhuk --- Created attachment 247166 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=247166=edit patch 14/stable -- You are receiving this mail because: You are the assignee for the bug.
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290
--- Comment #15 from Ivan Rozhuk ---
(In reply to Alastair Hogge from comment #14)
$${PORTSDIR -> ${PORTSDIR
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290
--- Comment #14 from Alastair Hogge ---
Created attachment 247153
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=247153=edit
patch for 15-CURRENT
Hello,
Attached patch based on Tatsuki Makino's. Having ${PORTS_MODULES} handle
flavours is a massive Quality of Life (QOL) improvement for those of use
building on one host, and distributing system updates via NFS.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290
--- Comment #13 from Tatsuki Makino ---
Created attachment 234307
--> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=234307=edit
patch for stable/12
This is a patch for stable/12, but the buildkernel and installkernel on
stable/12 went fine.
The flat plain of load average 1.00 did not even occur when -j 3 buildkernel.
The ports build was invoked as follows.
===> Ports module graphics/drm-fbsd12.0-kmod (all)
cd ${PORTSDIR:-/usr/ports}/graphics/drm-fbsd12.0-kmod; env -u CC -u CXX -u
CPP -u MAKESYSPATH -u MK_AUTO_OBJ -u MAKEOBJDIR MAKEFLAGS="-D
NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNCONF=GENERIC KERNEL=kernel
TARGET=amd64 TARGET_ARCH=amd64" SYSDIR=/usr/src/sys
PATH=/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin
SRC_BASE=/usr/src OSVERSION=1203507
WRKDIRPREFIX=/usr/obj/usr/src/amd64.amd64/sys/GENERIC make clean build
===> Ports module graphics/gpu-firmware-radeon-kmod@caicos (all)
cd ${PORTSDIR:-/usr/ports}/graphics/gpu-firmware-radeon-kmod; env -u CC -u
CXX -u CPP -u MAKESYSPATH -u MK_AUTO_OBJ -u MAKEOBJDIR MAKEFLAGS="-D
NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNCONF=GENERIC KERNEL=kernel
TARGET=amd64 TARGET_ARCH=amd64" SYSDIR=/usr/src/sys
PATH=/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin
SRC_BASE=/usr/src OSVERSION=1203507
WRKDIRPREFIX=/usr/obj/usr/src/amd64.amd64/sys/GENERIC make FLAVOR=caicos clean
build
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290
Tatsuki Makino changed:
What|Removed |Added
CC||tatsuki_mak...@hotmail.com
--- Comment #12 from Tatsuki Makino ---
If the -B of make -B is only to prevent parallel execution, I think it is
excessive.
For example, the value of .MAKE.MODE changes from normal to compat.
Just make -- and make -j 1 also makes a difference.
Options like -J 15,16 come across in the MAKEFLAGS environment variable.
Wouldn't it be better to pass MAKEFLAGS with the -[Jj] option dropped in the
environment variable?
MAKEFLAGS="${MAKEFLAGS:M*:C/((^| )-[Jjm]) /\1/gW:N-[Jjm]*:NMK_AUTO_OBJ=*}"
The number of jobs when building ports is left to MAKE_JOBS_NUMBER on the ports
side.
If make buildkernel is executed with -j 1, but MAKE_JOBS_NUMBER is 4, then
build the port with 4 jobs.
(In reply to Ivan Rozhuk from comment #11)
:M*@*:C/.*@/FLAVOR=/g is great :)
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 Ivan Rozhuk changed: What|Removed |Added Attachment #205980|maintainer-approval?| Flags|| Attachment #205980|0 |1 is obsolete|| Attachment #234277||maintainer-approval? Flags|| --- Comment #11 from Ivan Rozhuk --- Created attachment 234277 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=234277=edit add FLAVOR support Now allow to use FLAVOR: PORTS_MODULES= graphics/gpu-firmware-radeon-kmod@caicos -- You are receiving this mail because: You are the assignee for the bug.
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290
--- Comment #10 from rozhuk...@gmail.com ---
(In reply to Bryan Drewery from comment #9)
Probably there is misunderstanding.
MAKEFLAGS_PORTS=${MAKEFLAGS:M*:tW:S/^-m /-m_/g:S/ -m / -m_/g:S/^-j /-m_/g:S/ -j
/ -m_/g:S/^-J /-m_/g:S/ -J / -m_/g:tw:N-m_*:NMK_AUTO_OBJ=*}
As I remember this thing removes -j args from flags.
MAKE_JOBS_NUMBER=${.MAKE.JOBS} - is passed to ports after my patch.
--
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 --- Comment #9 from Bryan Drewery --- Passing -j1 instead of -B seems to work when using a proof of concept test. Can you try that please? -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290
--- Comment #8 from Bryan Drewery ---
(In reply to rozhuk.im from comment #3)
What's a terrible idea is making a change you don't understand. The -j flag
has 0 impact on the ports build except to BREAK it. -j is not how ports
are allowed to use make jobs. These fields are:
# MAKE_JOBS_UNSAFE
# - Disallow multiple jobs even when user set a global override.
# To be used with known bad ports.
# DISABLE_MAKE_JOBS
# - Set to disable the multiple jobs feature. User settable.
# MAKE_JOBS_NUMBER
# - Override the number of make jobs to be used. User settable.
# MAKE_JOBS_NUMBER_LIMIT
# - Set a limit for maximum number of make jobs allowed to be
# used.
77 # Read ports defaults.
78 PORTSDIR?= $(${MAKE} -V PORTSDIR)
79 DISTDIR?= $(${MAKE} -V DISTDIR)
80 PORT_DBDIR?=$(${MAKE} -V PORT_DBDIR)
81 OPTIONS_SET?= $(${MAKE} -V OPTIONS_SET)
82 OPTIONS_UNSET?= $(${MAKE} -V OPTIONS_UNSET)
83 NO_CCACHE?= $(${MAKE} -V NO_CCACHE)
84 CCACHE_DIR?=$(${MAKE} -V CCACHE_DIR)
85 WITH_CCACHE_BUILD?= $(${MAKE} -V WITH_CCACHE_BUILD)
86 DEFAULT_VERSIONS?= $(${MAKE} -V DEFAULT_VERSIONS)
87 WITH_DEBUG_PORTS?= $(${MAKE} -V WITH_DEBUG_PORTS)
This is completely redundant and not needed.
This is no such thing as maintainer timeout on this code as it is not ports.
I suggest if you want this fixed you explore why the MAKE_JOBS settings are not
used and why those env vars were needed when they are already available in
ports.
--
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 rozhuk...@gmail.com changed: What|Removed |Added Flags||maintainer-feedback- -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 --- Comment #7 from rozhuk...@gmail.com --- maintainer timeout -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 rozhuk...@gmail.com changed: What|Removed |Added Attachment #205980||maintainer-approval? Flags|| -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 rozhuk...@gmail.com changed: What|Removed |Added Version|12.0-STABLE |CURRENT -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 rozhuk...@gmail.com changed: What|Removed |Added Attachment #203755|maintainer-approval?| Flags|| Attachment #203755|0 |1 is obsolete|| --- Comment #6 from rozhuk...@gmail.com --- Created attachment 205980 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=205980=edit more env vars to ports Pass more ports specific settings via env. Now all ports build into kernel OBJ dir. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 rozhuk...@gmail.com changed: What|Removed |Added Attachment #203705|maintainer-approval?| Flags|| Attachment #203705|0 |1 is obsolete|| Attachment #203755||maintainer-approval? Flags|| --- Comment #5 from rozhuk...@gmail.com --- Created attachment 203755 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=203755=edit more clean MAKEFLAGS src MAKEFLAGS: -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64 Passed to ports: MAKE_JOBS_NUMBER: 8 MAKEFLAGS: -s -D ALWAYS_CHECK_MAKE -s -D ALWAYS_CHECK_MAKE -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64 -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 --- Comment #4 from rozhuk...@gmail.com --- (In reply to rozhuk.im from comment #3) MAKEFLAGS: -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -j 8 -s -D ALWAYS_CHECK_MAKE -J 15,16 -m /usr/src/share/mk -D NO_MODULES_OBJ .MAKE.LEVEL.ENV=MAKELEVEL KERNEL=kernel TARGET=amd64 TARGET_ARCH=amd64 -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290
--- Comment #3 from rozhuk...@gmail.com ---
(In reply to Bryan Drewery from comment #2)
Keeping -B is terrible idea: compilation takes too many time.
-j can be filtered out here:
MAKEFLAGS="${MAKEFLAGS:M*:tW:S/^-m /-m_/g:S/ -m /
-m_/g:tw:N-m_*:NMK_AUTO_OBJ=*}"
also I can extract "-j N" N arg and pass it to MAKE_JOBS_NUMBER.
I dont know why -m filered out here.
Now MAKEFLAGS contains -j 8 -s -J15,16 and some other strange staff.
Probably better not pass MAKEFLAGS or keep only known good flags, like -s.
--
You are receiving this mail because:
You are the assignee for the bug.
___
freebsd-bugs@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 --- Comment #2 from Bryan Drewery --- Don't remove -B please. We do not want to pass -j down to the ports build. Ports does not support -j. It uses an internal mechanism of MAKE_JOBS. Some ports support it and others do not. Passing down -j doesn't do what you would expect. Separate issue, but it's also not supported to build multiple ports at once simply via 'make'. Something like Poudriere is needed for that. # MAKE_JOBS_NUMBER # - Override the number of make jobs to be used. User settable. # MAKE_JOBS_NUMBER_LIMIT # - Set a limit for maximum number of make jobs allowed to be # used. MAKE_JOBS_NUMBER defaults to the number of CPU threads on the system as long as you do not set DISABLE_MAKE_JOBS and the port isn't MAKE_JOBS_UNSAFE. -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 Mark Linimon changed: What|Removed |Added Keywords||patch -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 rozhuk...@gmail.com changed: What|Removed |Added Attachment #203705||maintainer-approval? Flags|| -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
[Bug 237290] [PATCH] build(7): PORTS_MODULES does not build in parallel
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237290 rozhuk...@gmail.com changed: What|Removed |Added Summary|build(7): PORTS_MODULES |[PATCH] build(7): |does not build in parallel |PORTS_MODULES does not ||build in parallel -- You are receiving this mail because: You are the assignee for the bug. ___ freebsd-bugs@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-bugs To unsubscribe, send any mail to "freebsd-bugs-unsubscr...@freebsd.org"
