Re: PG vs LLVM 12 on seawasp, next round
On 1/24/21 11:06 PM, Tom Lane wrote: > Noah Misch writes: >> On Mon, Jan 18, 2021 at 09:29:53PM +0100, Fabien COELHO wrote: >>> ... Last two months were a >>> little overworked for me so I let slip quite a few things. If you want to >>> disable the animal as Tom suggests, do as you want. >> Perhaps he was suggesting that you (buildfarm owner) disable the cron job >> that >> initiates new runs. That's what I do when one of my animals needs my >> intervention. > Indeed. I'm not sure there even is a provision to block an animal on the > buildfarm-server side. If there is, you'd have to request that it be > manually undone after you get around to fixing the animal. Frankly, > if I were the BF admin, I would be in about as much hurry to do that > as you've been to fix it. > > It's actually very easy, but it's something I usually reserve for people who are very unresponsive to emails. As noted, disabling the crontab entry on the client side is a preferable solution. cheers andrew -- Andrew Dunstan EDB: https://www.enterprisedb.com
Re: PG vs LLVM 12 on seawasp, next round
I've added an explicit LD_LIBRARY_PATH, which will be triggered at some point later. This seems to have fixed the issue. I'm sorry for the noise and quite baffled anyway, because according to my change logs it does not seem that I modified anything from my side about the dynamic library path when compiling with clang. At least I do not see a trace of that. You can also do something like LDFLAGS="$LDFLAGS -Wl,-rpath,$(llvm-config --libdir)" or such. I've resorted to just hardcode LD_LIBRARY_PATH alongside PATH when compiling with clang in my buildfarm script. Thanks for the tip anyway. And thanks Thomas for pointing out the fix! -- Fabien.
Re: PG vs LLVM 12 on seawasp, next round
Hi, On 2021-02-15 10:05:32 +0100, Fabien COELHO wrote: > > show? What do you have in seawap's LD_LIBRARY_PATH, and is the > > problem that you need to add /home/fabien/clgtk/lib to it > > Argh. Would it be so stupid? :-( I thought the configuration stuff would > manage the link path automatically, which may be quite naïve, indeed. The only way to do that is to add an rpath annotation, and e.g. distros don't like that - there's some security implications. > I've added an explicit LD_LIBRARY_PATH, which will be triggered at some > point later. You can also do something like LDFLAGS="$LDFLAGS -Wl,-rpath,$(llvm-config --libdir)" or such. Greetings, Andres Freund
Re: PG vs LLVM 12 on seawasp, next round
Hello Thomas, Thanks for looking at this, I'm currently far behind on many things and not very responsive:-( Here is the creation of llvmjit.so: g++ ... -o llvmjit.so ... -L/home/fabien/clgtk/lib ... -lLLVMOrcJIT ... That'd be from llvm-config --ldflags or similar, from this binary: checking for llvm-config... (cached) /home/fabien/clgtk/bin/llvm-config So what does ls -slap /home/fabien/clgtk/lib show? Plenty files and links, eg: 0 lrwxrwxrwx 1 fabien fabien 21 janv. 23 09:40 /home/fabien/clgtk/lib/libLLVMMCJIT.so -> libLLVMMCJIT.so.12git 2140 -rw-r--r-- 1 fabien fabien 2190824 janv. 23 09:28 /home/fabien/clgtk/lib/libLLVMMCJIT.so.12git 0 lrwxrwxrwx 1 fabien fabien 22 janv. 23 09:40 /home/fabien/clgtk/lib/libLLVMOrcJIT.so -> libLLVMOrcJIT.so.12git 40224 -rw-r--r-- 1 fabien fabien 41187208 janv. 23 09:34 /home/fabien/clgtk/lib/libLLVMOrcJIT.so.12git Hmmm, clang recompilation has been failing for some weeks. Argh, this is due to the recent "master" to "main" branch renaming. What does ldd /home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so No such file of directory:-) because the directory is cleaned up. show? What do you have in seawap's LD_LIBRARY_PATH, and is the problem that you need to add /home/fabien/clgtk/lib to it Argh. Would it be so stupid? :-( I thought the configuration stuff would manage the link path automatically, which may be quite naïve, indeed. PS Could you try blowing away the accache directory so we can rule out bad cached configure stuff? Hmmm. I've tried that before. I can do it again. I've added an explicit LD_LIBRARY_PATH, which will be triggered at some point later. -- Fabien Coelho - CRI, MINES ParisTech
Re: PG vs LLVM 12 on seawasp, next round
On Sat, Dec 12, 2020 at 8:45 AM Fabien COELHO wrote: > > +ERROR: could not load library > > "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": > > libLLVMOrcJIT.so.12git: cannot open shared object file: No such file or > > directory Bonjour Fabien, Here is the creation of llvmjit.so: g++ ... -o llvmjit.so ... -L/home/fabien/clgtk/lib ... -lLLVMOrcJIT ... That'd be from llvm-config --ldflags or similar, from this binary: checking for llvm-config... (cached) /home/fabien/clgtk/bin/llvm-config So what does ls -slap /home/fabien/clgtk/lib show? What does ldd /home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so show? What do you have in seawap's LD_LIBRARY_PATH, and is the problem that you need to add /home/fabien/clgtk/lib to it (or is it supposed to be making it into the rpath, or is it in your ld.so.conf)? PS Could you try blowing away the accache directory so we can rule out bad cached configure stuff?
Re: PG vs LLVM 12 on seawasp, next round
Noah Misch writes: > On Mon, Jan 18, 2021 at 09:29:53PM +0100, Fabien COELHO wrote: >> ... Last two months were a >> little overworked for me so I let slip quite a few things. If you want to >> disable the animal as Tom suggests, do as you want. > Perhaps he was suggesting that you (buildfarm owner) disable the cron job that > initiates new runs. That's what I do when one of my animals needs my > intervention. Indeed. I'm not sure there even is a provision to block an animal on the buildfarm-server side. If there is, you'd have to request that it be manually undone after you get around to fixing the animal. Frankly, if I were the BF admin, I would be in about as much hurry to do that as you've been to fix it. regards, tom lane
Re: PG vs LLVM 12 on seawasp, next round
On Mon, Jan 18, 2021 at 09:29:53PM +0100, Fabien COELHO wrote: > >>>The "no such file" error seems more like a machine local issue to me. > >> > >>I'll look into it when I have time, which make take some time. Hopefully > >>over the holidays. > > > >This is still happening ... Any chance you can have a look at it? > > Indeed. I'll try to look (again) into it soon. I had a look but did not find > anything obvious in the short time frame I had. Last two months were a > little overworked for me so I let slip quite a few things. If you want to > disable the animal as Tom suggests, do as you want. Perhaps he was suggesting that you (buildfarm owner) disable the cron job that initiates new runs. That's what I do when one of my animals needs my intervention.
Re: PG vs LLVM 12 on seawasp, next round
Hello Alvaro, The "no such file" error seems more like a machine local issue to me. I'll look into it when I have time, which make take some time. Hopefully over the holidays. This is still happening ... Any chance you can have a look at it? Indeed. I'll try to look (again) into it soon. I had a look but did not find anything obvious in the short time frame I had. Last two months were a little overworked for me so I let slip quite a few things. If you want to disable the animal as Tom suggests, do as you want. -- Fabien.
Re: PG vs LLVM 12 on seawasp, next round
Alvaro Herrera writes: > On 2020-Dec-11, Fabien COELHO wrote: >> I'll look into it when I have time, which make take some time. Hopefully >> over the holidays. > This is still happening ... Any chance you can have a look at it? If you don't have time to debug it, perhaps you could just disable the buildfarm animal till you do. It's cluttering the buildfarm failure report without providing useful info ... regards, tom lane
Re: PG vs LLVM 12 on seawasp, next round
On 2020-Dec-11, Fabien COELHO wrote: > > I hadn't checked that before, but for the last few days there's been a > > different failure than the one I saw earlier: > > > > +ERROR: could not load library > > "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": > > libLLVMOrcJIT.so.12git: cannot open shared object file: No such file or > > directory > > The "no such file" error seems more like a machine local issue to me. > > I'll look into it when I have time, which make take some time. Hopefully > over the holidays. This is still happening ... Any chance you can have a look at it? -- Álvaro Herrera39°49'30"S 73°17'W
Re: PG vs LLVM 12 on seawasp, next round
Hello Andres, I hadn't checked that before, but for the last few days there's been a different failure than the one I saw earlier: +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git: cannot open shared object file: No such file or directory whereas what I fixed is about: +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": /home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so: undefined symbol: LLVMInitializeX86Target Changed somewhere between https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp=2020-11-20%2009%3A17%3A10 and https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp=2020-11-21%2023%3A17%3A11 The "no such file" error seems more like a machine local issue to me. I'll look into it when I have time, which make take some time. Hopefully over the holidays. -- Fabien.
Re: PG vs LLVM 12 on seawasp, next round
Hi, On 2020-12-07 19:38:19 -0800, Andres Freund wrote: > On 2020-12-01 12:08:10 -0800, Andres Freund wrote: > > On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote: > > > Andres investigated a few days ago, managed to reproduce the issue > > > locally, > > > and has one line patch. I'm unsure if it should be prevently back-patched, > > > though. > > > > I see no reason not to backpatch - it's more correct for past versions > > of LLVM as well. > > I pushed that now. I hadn't checked that before, but for the last few days there's been a different failure than the one I saw earlier: +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git: cannot open shared object file: No such file or directory whereas what I fixed is about: +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": /home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so: undefined symbol: LLVMInitializeX86Target Changed somewhere between https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp=2020-11-20%2009%3A17%3A10 and https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=seawasp=2020-11-21%2023%3A17%3A11 The "no such file" error seems more like a machine local issue to me. Greetings, Andres Freund
Re: PG vs LLVM 12 on seawasp, next round
Hi, On 2020-12-01 12:08:10 -0800, Andres Freund wrote: > On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote: > > Andres investigated a few days ago, managed to reproduce the issue locally, > > and has one line patch. I'm unsure if it should be prevently back-patched, > > though. > > I see no reason not to backpatch - it's more correct for past versions > of LLVM as well. I pushed that now. - Andres
Re: PG vs LLVM 12 on seawasp, next round
Hi, On 2020-12-01 21:04:44 +0100, Fabien COELHO wrote: > Andres investigated a few days ago, managed to reproduce the issue locally, > and has one line patch. I'm unsure if it should be prevently back-patched, > though. I see no reason not to backpatch - it's more correct for past versions of LLVM as well. Greetings, Andres Freund
Re: PG vs LLVM 12 on seawasp, next round
Hello Thomas, Since seawasp's bleeding edge LLVM installation moved to "trunk 20201114 c8f4e06b 12.0.0" ~16 days ago, it has been red. Further updates didn't help it and it's now on "trunk 20201127 6ee22ca6 12.0.0". I wonder if there is something in Fabien's scripting that needs to be tweaked, perhaps a symlink name or similar. The compiler compilation script is quite straightforward (basically, get sources, configure and compile), even for a such a moving target… The right approach is to wait for some time before looking at the issue, typically one week for the next recompilation, in case the problem evaporates, so you were right not to jump on it right away:-) Andres investigated a few days ago, managed to reproduce the issue locally, and has one line patch. I'm unsure if it should be prevently back-patched, though. -- Fabien.
Re: PG vs LLVM 12 on seawasp, next round
Hi, On 2020-12-01 17:35:49 +1300, Thomas Munro wrote: > Since seawasp's bleeding edge LLVM installation moved to "trunk > 20201114 c8f4e06b 12.0.0" ~16 days ago, it has been red. Further > updates didn't help it and it's now on "trunk 20201127 6ee22ca6 > 12.0.0". I wonder if there is something in Fabien's scripting that > needs to be tweaked, perhaps a symlink name or similar. I don't > follow LLVM development but I found my way to a commit[1] around the > right time that mentions breaking up the OrcJIT library, so *shrug* > maybe that's a clue. > > +ERROR: could not load library > "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": > libLLVMOrcJIT.so.12git: cannot open shared object file: No such file > or directory It's a change in how LLVM dependencies are declared internally. Previously the 'native' component was - unintentionally - transitively included via the 'orcjit' component, but now that's not the case anymore. The attached patch should fix it, I think? Greetings, Andres Freund diff --git i/config/llvm.m4 w/config/llvm.m4 index a5f4a9af448..3a75cd8b4df 100644 --- i/config/llvm.m4 +++ w/config/llvm.m4 @@ -76,6 +76,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT], debuginfodwarf) pgac_components="$pgac_components $pgac_component";; orcjit) pgac_components="$pgac_components $pgac_component";; passes) pgac_components="$pgac_components $pgac_component";; + native) pgac_components="$pgac_components $pgac_component";; perfjitevents) pgac_components="$pgac_components $pgac_component";; esac done; diff --git i/configure w/configure index ffcd0c5b1d4..2a03ed0a018 100755 --- i/configure +++ w/configure @@ -5168,6 +5168,7 @@ fi debuginfodwarf) pgac_components="$pgac_components $pgac_component";; orcjit) pgac_components="$pgac_components $pgac_component";; passes) pgac_components="$pgac_components $pgac_component";; + native) pgac_components="$pgac_components $pgac_component";; perfjitevents) pgac_components="$pgac_components $pgac_component";; esac done;
PG vs LLVM 12 on seawasp, next round
Hi Since seawasp's bleeding edge LLVM installation moved to "trunk 20201114 c8f4e06b 12.0.0" ~16 days ago, it has been red. Further updates didn't help it and it's now on "trunk 20201127 6ee22ca6 12.0.0". I wonder if there is something in Fabien's scripting that needs to be tweaked, perhaps a symlink name or similar. I don't follow LLVM development but I found my way to a commit[1] around the right time that mentions breaking up the OrcJIT library, so *shrug* maybe that's a clue. +ERROR: could not load library "/home/fabien/pg/build-farm-11/buildroot/HEAD/pgsql.build/tmp_install/home/fabien/pg/build-farm-11/buildroot/HEAD/inst/lib/postgresql/llvmjit.so": libLLVMOrcJIT.so.12git: cannot open shared object file: No such file or directory [1] https://github.com/llvm/llvm-project/commit/1d0676b54c4e3a517719220def96dfdbc26d8048
