On Jun 16, 2025, at 08:22, Mark Millard <mark...@yahoo.com> wrote: > On Jun 16, 2025, at 04:50, Ronald Klop <ronald-li...@klop.ws> wrote: > >> Van: Mark Millard <mark...@yahoo.com> >> Datum:vrijdag, 30 mei 2025 20:58 >> Aan:"Dag-Erling Smørgrav" <d...@freebsd.org> >> CC:ronald-li...@klop.ws, FreeBSD Mailing List <freebsd-po...@freebsd.org> >> Onderwerp:Re: error -9 on build cluster? >> On May 30, 2025, at 11:29, Mark Millard <mark...@yahoo.com> wrote: >> >>> On May 30, 2025, at 10:43, Dag-Erling Smørgrav <d...@freebsd.org> wrote: >>> >>>> Mark Millard <mark...@yahoo.com> writes: >>>>> Dag-Erling Smørgrav <d...@freebsd.org> writes: >>>>>> This is preceded by an 800-line list of warnings from a single source >>>>>> file (multi_index_block.cpp). This list is clearly truncated as it's >>>>>> missing the final “N warnings generated” line, so my guess is there is >>>>>> at least one error there as well which just doesn't get displayed. >>>>> My guess: the failure occurred while the warnings were otherwise >>>>> still being generated and that is what stopped the warnings. >>>> >>>> That's possible. If we assume that the error code returned by scons is >>>> an errno, it corresponds to EBADF. Perhaps a bug in scons causes it to >>>> lose track of the pipe it uses to read the compiler's stderr. >>> >>> devel/scans has not been updated after 2024-Jun-06: >>> >>> QUOTE >>> author Po-Chuan Hsieh <sunp...@freebsd.org> 2024-06-06 22:03:56 +0000 >>> committer Po-Chuan Hsieh <sunp...@freebsd.org> 2024-06-06 22:03:56 +0000 >>> . . . >>> devel/scons: Update to 4.7.0 >>> Changes: https://github.com/SCons/scons/releases >>> https://scons.org/tag/releases.html PR: 279488 Exp-run by: antoine >>> END QUOTE >>> >>> That would mean that quarterly got the update in 2024 as well. >>> >>> https://pkg-status.freebsd.org/ampere1/data/141arm64-quarterly/2e2d801cf977/logs/mongodb60-6.0.18.log >>> >>> did not fail but shows the warnings. >>> >>> devel/scons does not seem to be the source of the change in beavior. >> >> >> Worked: >> >> https://pkg-status.freebsd.org/ampere3/data/134arm64-default/1afdc808e67b/logs/mongodb60-6.0.20_1.log >> build started at Sun Jan 5 17:35:49 UTC 2025 >> >> The next 134arm64-default build Failed: >> >> https://pkg-status.freebsd.org/ampere3/data/134arm64-default/86e28c84a7e0/logs/errors/mongodb60-6.0.20_1.log >> build started at Sun Jan 19 11:21:16 UTC 2025 >> >> That at least gives a time frame to investigate. >> >> More details: >> >> =>> Building databases/mongodb60 >> build started at Sun Jan 5 17:35:49 UTC 2025 >> port directory: /usr/ports/databases/mongodb60 >> package name: mongodb60-6.0.20_1 >> building for: FreeBSD 134arm64-default-job-06 13.4-RELEASE-p2 FreeBSD >> 13.4-RELEASE-p2 arm64 >> maintained by: ron...@freebsd.org >> Makefile datestamp: -rw-r--r-- 1 root wheel 5845 Jan 4 01:02 >> /usr/ports/databases/mongodb60/Makefile >> Ports top last git commit: 1afdc808e67 >> Ports top unclean checkout: no >> Port dir last git commit: 3ab18c284b0 >> Port dir unclean checkout: no >> Poudriere version: poudriere-git-3.4.2 >> Host OSVERSION: 1500028 >> Jail OSVERSION: 1304000 >> >> vs. >> >> =>> Building databases/mongodb60 >> build started at Sun Jan 19 11:21:16 UTC 2025 >> port directory: /usr/ports/databases/mongodb60 >> package name: mongodb60-6.0.20_1 >> building for: FreeBSD 134arm64-default-job-13 13.4-RELEASE-p2 FreeBSD >> 13.4-RELEASE-p2 arm64 >> maintained by: ron...@freebsd.org >> Makefile datestamp: -rw-r--r-- 1 root wheel 5845 Jan 4 01:02 >> /usr/ports/databases/mongodb60/Makefile >> Ports top last git commit: 86e28c84a7e >> Ports top unclean checkout: no >> Port dir last git commit: 3ab18c284b0 >> Port dir unclean checkout: no >> Poudriere version: poudriere-git-3.4.2 >> Host OSVERSION: 1500028 >> Jail OSVERSION: 1304000 >> >> Same Host and Jail OSVERSION's. Same "Port dir last git commit". >> >> But different "Ports top last git commit". So. some change in >> another package that is involved? >> >> === >> Mark Millard >> marklmi at yahoo.com >> >> >> >> Hi, >> >> The builder got the same error on mongodb70. > > Well recent 60 vs. the 70: > > scons: *** [build/59f4f0dd/mongo/db/catalog/multi_index_block.o] Error -9 > vs.: > scons: *** [build/opt/mongo/db/query/canonical_query_encoder.o] Error -9 > > It is not at an analogous point in the build. Looking at the build times: > > build time: 00:45:05 > vs.: > build time: 05:05:52 > > > Looking at the 60 history . . . > > scons: *** [build/59f4f0dd/mongo/db/catalog/multi_index_block.o] Error -9 > scons: building terminated because of errors. > build/59f4f0dd/mongo/db/catalog/multi_index_block.o failed: Error -9 > > scons: *** [build/59f4f0dd/mongo/db/views/view_catalog_helpers.o] Error -9 > . . . > scons: building terminated because of errors. > build/59f4f0dd/mongo/db/views/view_catalog_helpers.o failed: Error -9 > > (The above 2 are amd64, the below older ones are all arm64:) > > src/mongo/db/commands/collection_to_capped.cpp > . . . > *** Signal 9 > > build/59f4f0dd/mongo/db/commands/feature_compatibility_version_document_gen.cpp > . . . > *** Signal 9 > > build/59f4f0dd/mongo/db/storage/storage_parameters_gen.cpp > . . . > *** Signal 9 > > (The older ones only have the report available now, not the log:) > > scons: building terminated because of errors. > build/59f4f0dd/mongo/db/s/config/sharding_catalog_manager_zone_operations.o > failed: Error -9 > > scons: building terminated because of errors. > build/59f4f0dd/mongo/db/commands/mr_common.o failed: Error -9 > > scons: building terminated because of errors. > build/59f4f0dd/mongo/db/query/find.o failed: Error -9 > > src/third_party/mozjs/platform/aarch64/freebsd/build/Unified_cpp_js_src16.cpp > *** Signal 9 > > (The above goes back to 2025-02-05 17:50.) > > The first is 6.0.23_1. The 2nd is 6.0.22_1. The last 2 are 6.0.20_1. > The rest are 6.0.20_2. > > There is no uniformity to where the errors occur. There is also > variability for "Signal 9" vs. "Error -9". A guess would be that > "Error -9" is another way of reporting "Signal 9".
Speaking of variability: https://pkg-status.freebsd.org/beefy18/build.html?mastername=main-amd64-default&build=p056670b6dfea_s1749465947a shows building all the amd64 mongodb[5678]0* just fine. That build started at: 05 Jun 2025 01:11:32 GMT on beefy18 . As does the beefy8 13 May 2025 09:01:49 GMT: https://pkg-status.freebsd.org/beefy8/build.html?mastername=142amd64-default&build=2025-05-13_09h01m49s and the beefy16 24 Apr 2025 07:24:46 GMT: https://pkg-status.freebsd.org/beefy16/build.html?mastername=134amd64-default&build=dc1d367f2961 and the beefy14 12 Apr 2025 01:03:08 GMT: https://pkg-status.freebsd.org/beefy14/build.html?mastername=134amd64-quarterly&build=37270c265761 mongodb[5678]0 does not reliably fail. > Back in 2022-Jul there was this to avoid "Signal 9" problems: > > A commit in branch main references this bug: > > URL: > https://cgit.FreeBSD.org/ports/commit/?id=0aabc02be18794a56c436233e59379ebaa86c2e0 > > commit 0aabc02be18794a56c436233e59379ebaa86c2e0 > Author: Daniel Engberb <dii...@freebsd.org> > AuthorDate: 2022-07-14 14:43:02 +0000 > Commit: Fernando Apesteguía <fern...@freebsd.org> > CommitDate: 2022-07-16 15:11:58 +0000 > > databases/mongodb50: Disable LTO by default. > > Building mongo with LTO enabled takes a ridiculously amount of memory > (32GB in amd64). > Disable -ggdb so memory requirements don't sky rocket (around 10-12GB) > > PR: 264340 > Reported by: ulassaygi...@gmail.com > Approved by: ronald-li...@klop.ws (maintainer) > > databases/mongodb50/files/patch-SConstruct | 15 +++++++++++---- > 1 file changed, 11 insertions(+), 4 deletions(-) > > > There was also a comment reporting console log text like: > > swap_pager: out of swap space > swp_pager_getswapspace(2): failed > swap_pager: out of swap space > swp_pager_getswapspace(2): failed > swp_pager_getswapspace(5): failed > swp_pager_getswapspace(1): failed > swp_pager_getswapspace(23): failed > pid 73953 (ld.lld), jid 0, uid 0, was killed: failed to reclaim memory > > But, to my knowledge, such information is not publicly > available and would be not easily tied to match the > timing of mongodb* build failures. > > So personal/local builds may be needed to figure out if > such might be involved. > >> But this build does not have all the warnings. I still don't see the cause >> of Error -9. >> >> https://pkg-status.freebsd.org/beefy22/data/142amd64-default/02e9dc3b903b/logs/mongodb70-7.0.21_1.log >> >> https://portsfallout.com/fallout?port=databases%2Fmongodb70%24 >> >> My next idea is that this occurs because a process gets killed (-9), which >> should be visible in the logs of the cluster maintainers. Does this makes >> sense? === Mark Millard marklmi at yahoo.com
