The problem is that you have installed a copy of llvm15 that was compiled for
FreeBSD 13.3 ("FreeBSD_version: 1303001" in the pkg info). That depends on
the new symbol _ZNSt3__122__libcpp_verbose_abortEPKcz from the FreeBSD 13.3
version of /usr/lib/libc++.so.1, but your jail will have the FreeBSD 13.2
version of /usr/lib/libc++.so.1 without that symbol.
I suspect that you installed llvm15 using pkg install after FreeBSD 13.2 was
EoL. At that point, the 13.x packages on the FreeBSD pkg servers will be
compiled for FreeBSD 13.3 so you will get the wrong version. The FreeBSD pkg
servers don't support EoL versions of FreeBSD, but I would expect pkg install
to give a warning about this.
The reason why it has only just failed is probably because jit is only used if
PostgreSQL calculates that it is useful:
https://www.postgresql.org/docs/15/jit-decision.html
__Martin
>>>>> On Wed, 13 Nov 2024 15:25:27 -0500, B Smith said:
>
> Sudo pkg check -s found no issues.
>
> Here is the pkg info and ldd outputs:
> # pkg info llvm15
> llvm15-15.0.7_10
> Name : llvm15
> Version : 15.0.7_10
> Installed on :
> Shared Libs required:
> libzstd.so.1
> libpython3.11.so.1.0
> liblua-5.3.so
> libedit.so.0
> Shared Libs provided:
> libomptarget.so.15
> libomptarget.rtl.amdgpu.so.15
> libomp.so
> liblldb.so.15
> libclang_rt.ubsan_standalone-x86_64.so
> libclang_rt.ubsan_standalone-i386.so
> libclang_rt.ubsan_minimal-x86_64.so
> libclang_rt.ubsan_minimal-i386.so
> libclang_rt.tsan-x86_64.so
> libclang_rt.dyndd-x86_64.so
> libclang_rt.asan-x86_64.so
> libclang_rt.asan-i386.so
> libclang.so.15
> libclang-cpp.so.15
> libarcher.so
> libRemarks.so.15
> libLTO.so.15
> libLLVM-15.so
> Annotations :
> FreeBSD_version: 1303001
> build_timestamp: 2024-07-09T01:34:12+0000
> built_by : poudriere-git-3.4.1-30-g79e3edcd
> cpe : cpe:2.3:a:llvm:llvm:15.0.7:::::freebsd13:x64:10
> flavor : default
> port_checkout_unclean: no
> port_git_hash : be013aced
> ports_top_checkout_unclean: no
> ports_top_git_hash: be013aced
> repo_type : binary
> repository : FreeBSD
> Flat size : 966MiB
> Description :
> The LLVM Project is a collection of modular and reusable compiler and
> toolchain technologies.
>
> This port includes Clang (a C/C++/Objective-C compiler), LLD (a linker),
> LLDB (a debugger), an OpenMP runtime library, and the LLVM infrastructure
> these are built on.
>
> # ldd /usr/local/llvm15/lib/libLLVM-15.so
> /usr/local/llvm15/lib/libLLVM-15.so:
> librt.so.1 => /usr/lib/librt.so.1 (0x801486000)
> libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x80148f000)
> libthr.so.3 => /lib/libthr.so.3 (0x801495000)
> libm.so.5 => /lib/libm.so.5 (0x8014c2000)
> libz.so.6 => /lib/libz.so.6 (0x8014fe000)
> libzstd.so.1 => /usr/local/lib/libzstd.so.1 (0x80151b000)
> libc++.so.1 => /usr/lib/libc++.so.1 (0x8084a7000)
> libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x8085ab000)
> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x8085cd000)
> libc.so.7 => /lib/libc.so.7 (0x801083000)
> libelf.so.2 => /lib/libelf.so.2 (0x801068000)
>
> Thanks,
>
> B. Smith
>
> On Wed, Nov 13, 2024 at 7:40 AM Martin Simmons <mar...@lispworks.com> wrote:
>
> > What is the output of these commands in the jail?
> >
> > pkg info llvm15
> >
> > ldd /usr/local/llvm15/lib/libLLVM-15.so
> >
> > __Martin
> >
> >
> > >>>>> On Tue, 12 Nov 2024 17:19:34 -0500, B Smith said:
> > >
> > > Yes, I should probably upgrade it to 13.4. I'm slow to upgrade my jails
> > > when they are functioning well and they are single-task with no
> > > outward-facing components (i.e., the Bacula jail is only for running
> > > Bacula).
> > >
> > > I should have mentioned in my original email that the library is still
> > > there. It was installed via pkg.
> > > Running the same commands you did:
> > > # locate llvmjit.so
> > > /usr/local/lib/postgresql/llvmjit.so
> > > # pkg which /usr/local/lib/postgresql/llvmjit.so
> > > /usr/local/lib/postgresql/llvmjit.so was installed by package
> > > postgresql15-server-15.7
> > >
> > > Neither llvmjit.so nor libLLVM-15.so show any modification since July 9
> > > (when they were installed).
> > >
> > > Thanks.
> > >
> > >
> > > On Tue, Nov 12, 2024 at 5:08 PM Dan Langille <d...@langille.org> wrote:
> > >
> > > >
> > > > On Tue, Nov 12, 2024, at 4:13 PM, B. Smith wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'm running Bacula in a FreeBSD 13.2 jail under TrueNAS Core. This
> > weekend
> > > > all my backups proceeded fine, but I noticed that a backup that should
> > have
> > > > been suspended started up, so I stopped the job. A few other
> > incremental
> > > > jobs ran without incident, but then I got an error in a job after
> > those:
> > > >
> > > >
> > > > side note, time to get that jail update to fix known vulns.
> > > >
> > > >
> > > > Fatal error: backup.c:343 [DE0028] Query failed: DECLARE _bac_cursor
> > > > CURSOR FOR SELECT Path, Filename, FileIndex, JobId, LStat, DeltaSeq,
> > MD5,
> > > > JobTDate FROM ((SELECT Path.Path, T1.Filename, T1.FileIndex, T1.JobId,
> > > > LStat, DeltaSeq, MD5, JobTDate FROM ( SELECT DISTINCT ON (Filename,
> > PathId)
> > > > JobTDate, JobId, FileId, FileIndex, PathId, Filename, LStat, MD5,
> > DeltaSeq
> > > > FROM (SELECT FileId, JobId, PathId, Filename, FileIndex, LStat, MD5,
> > > > DeltaSeq FROM File WHERE JobId IN
> > > >
> > (1561,1633,1642,1750,1759,2001,2010,2019,2028,2037,2046,2055,2064,2073,2082,2091,2100,2109,2118,2127,2136,2145,2182,2191,2200,2209,2218,2227,2236,2245,2254,2263,2272,2281,2340,2471,2517,2755)
> > > > UNION ALL SELECT File.FileId, File.JobId, PathId, Filename,
> > File.FileIndex,
> > > > LStat, MD5, DeltaSeq FROM BaseFiles JOIN File USING (FileId) WHERE
> > > > BaseFiles.JobId IN
> > > >
> > (1561,1633,1642,1750,1759,2001,2010,2019,2028,2037,2046,2055,2064,2073,2082,2091,2100,2109,2118,2127,2136,2145,2182,2191,2200,2209,2218,2227,2236,2245,2254,2263,2272,2281,2340,2471,2517,2755)
> > > > ) AS T JOIN Job USING (JobId) ORDER BY Filename, PathId, JobTDate
> > DESC )
> > > > AS T1 JOIN Path ON (Path.PathId = T1.PathId) WHERE FileIndex > 0 ) )
> > AS U1
> > > > ORDER BY JobTDate, FileIndex ASC: ERR=ERROR: could not load library
> > > > "/usr/local/lib/postgresql/llvmjit.so":
> > > > /usr/local/llvm15/lib/libLLVM-15.so: Undefined symbol
> > > > "_ZNSt3__122__libcpp_verbose_abortEPKcz"
> > > >
> > > > I shutdown Bacula, restarted the jail, and still had the same issue. I
> > did
> > > > not initiate any updates or changes to the system between the working
> > and
> > > > non-working jobs, and no real job (other than BackupCatalog) would run
> > > > without the error. Before restoring my database from backup, I used
> > > > postgres to turn off JIT:
> > > >
> > > > alter system set jit=off;
> > > >
> > > >
> > > > JIT is not familiar. Reading
> > > > https://www.freshports.org/databases/postgresql16-server it is
> > > >
> > > > LLVM=on: Build with support for JIT-compiling expressions
> > > >
> > > > Looking on my PostgreSQL 16 server, I find:
> > > >
> > > > [22:01 pg03 dvl ~] % locate llvmjit.so
> > > > /usr/local/lib/postgresql/llvmjit.so
> > > > [22:05 pg03 dvl ~] % pkg which /usr/local/lib/postgresql/llvmjit.so
> > > > /usr/local/lib/postgresql/llvmjit.so was installed by package
> > > > postgresql16-server-16.4
> > > >
> > > > It sounds like something deleted the file from your system and that's
> > how
> > > > it became upset.
> > > >
> > > > How was PostgreSQL installed there?
> > > >
> > > > ALTER SYSTEM
> > > >
> > > >
> > > > Doing this immediately fixed the problem. But I am mystified as to how
> > it
> > > > suddenly became a problem in the first place. Any suggestions as to why
> > > > this suddenly popped up, and whether JIT is even important for Bacula
> > > > performance?
> > > >
> > > >
> > > > I can't help you there.
> > > >
> > > > --
> > > > Dan Langille
> > > > d...@langille.org
> > > >
> > > >
> > > >
> > >
> >
>
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
