Re: JIT compiling with LLVM v11
Hi, On 2018-03-15 19:14:09 +0100, Catalin Iacob wrote: > For RHEL proper, I would guess that PG11 is too late for RHEL8 which, > according to history, should be coming soon. Yea. > For RHEL9 I would really expect RedHat to add llvm and clang to proper > RHEL and build/run against those, even if they add it only for > Postgres (like they did for mesa). By the looks of what's going to come for RHEL8 I think it already contains a suitable LLVM and clang (i.e. >= 3.9)? > As far as I know clang does promise gcc compatibility in > the sense that one can link together .o files compiled with both so I > expect the combination not to cause issues (assuming the other > compiler flags affecting binary compatibility are aligned). Right. But that's not even needed, as we just use plain old C ABI via dlsym(). Nothing needs to be linked together outside of dlsym(), so I'm not too concerned about that aspect. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On Thu, Mar 15, 2018 at 6:19 PM, Andres Freund wrote: > What we were talking about in this subthread was about a depency on > clang, not LLVM. And that's just needed at buildtime, to generate the > bitcode files (including synchronizing types / function signatures). I was actually thinking of both the buildtime and runtime dependency because I did not realize the PGDG packages already depend on EPEL. > For the yum.pg.o, which already depends on EPEL, there's a new enough > LLVM version. There's a new enough version in RHEL proper, but it > appears to only be there for mesa (llvm-private). Indeed RHEL 7 comes with llvm-private for mesa but that doesn't seem kosher to use for other things. When I said packagers I was only thinking of PGDG. I was thinking the software collections would be the likely solution for the PGDG packages for both buildtime and runtime. But it seems using clang from software collections and LLVM from EPEL is also a possibility, assuming that the newer clang generates IR that the older libraries are guaranteed to be able to load. For RHEL proper, I would guess that PG11 is too late for RHEL8 which, according to history, should be coming soon. For RHEL9 I would really expect RedHat to add llvm and clang to proper RHEL and build/run against those, even if they add it only for Postgres (like they did for mesa). I really don't see them shipping without a major speedup for a major DB, also because in the meantime the JIT in PG will have matured. That's also why I find it important to support gcc and not restrict JIT to clang builds as I expect that RedHat and all other Linux distros want to build everything with gcc and asking them to switch to clang or give up JIT will put them in a hard spot. As far as I know clang does promise gcc compatibility in the sense that one can link together .o files compiled with both so I expect the combination not to cause issues (assuming the other compiler flags affecting binary compatibility are aligned).
Re: JIT compiling with LLVM v11
On 2018-03-15 12:42:54 -0400, Tom Lane wrote: > Andres Freund writes: > > It'd be a build not runtime dependency, doesn't that change things? > > How could it not be a runtime dependency? What we were talking about in this subthread was about a depency on clang, not LLVM. And that's just needed at buildtime, to generate the bitcode files (including synchronizing types / function signatures). For the yum.pg.o, which already depends on EPEL, there's a new enough LLVM version. There's a new enough version in RHEL proper, but it appears to only be there for mesa (llvm-private). > You're not proposing that we'd embed all of LLVM into a Postgres > package are you? No. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
Andres Freund writes: > On 2018-03-15 12:33:08 -0400, Tom Lane wrote: >> The "software collections" stuff was still in its infancy when I left >> Red Hat, so things might've changed, but I'm pretty sure at the time >> it was verboten for any mainstream package to depend on an SCL one. > But we won't get PG 11 into RHEL7.x either way, no? Well, they've been known to back-port newer releases of PG into older RHEL; I wouldn't necessarily assume it'd happen for 11, but maybe 12 or beyond could be made available for RHEL7 at some point. >> But they very probably wouldn't want postgresql depending on a >> compiler package even if the dependency was mainstream, so I rather >> doubt that you'll ever see an --enable-jit PG build out of there, >> making this most likely moot as far as the official RH package goes. >> I don't know what Devrim's opinion might be about PGDG. > It'd be a build not runtime dependency, doesn't that change things? How could it not be a runtime dependency? You're not proposing that we'd embed all of LLVM into a Postgres package are you? If you are, be assured that Red Hat will *never* ship that. Static linking/embedding of one package in another is forbidden for obvious maintainability reasons. I would think that other distros have similar policies. regards, tom lane
Re: JIT compiling with LLVM v11
On 2018-03-15 12:33:08 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-03-15 17:19:23 +0100, Catalin Iacob wrote: > >> Indeed. It might be a bit awkward for packagers to depend on something > >> from Software Collections, for example because they come as separate > >> trees in /opt that are by default not in your path or dynamic loader > >> path - one needs to run everything via a scl wrapper or source the > >> /opt/rh/llvm-toolset-7/enable file to get the appropriate PATH and > >> LD_LIBRARY_PATH settings, But it seems doable. > > > It'd be just for clang, and they're not *forced* to do it, it's an > > optional dependency. So I think I'm ok with that. > > The "software collections" stuff was still in its infancy when I left > Red Hat, so things might've changed, but I'm pretty sure at the time > it was verboten for any mainstream package to depend on an SCL one. But we won't get PG 11 into RHEL7.x either way, no? > But they very probably wouldn't want postgresql depending on a > compiler package even if the dependency was mainstream, so I rather > doubt that you'll ever see an --enable-jit PG build out of there, > making this most likely moot as far as the official RH package goes. > I don't know what Devrim's opinion might be about PGDG. It'd be a build not runtime dependency, doesn't that change things? Greetings, Andres Freund
Re: JIT compiling with LLVM v11
Andres Freund writes: > On 2018-03-15 17:19:23 +0100, Catalin Iacob wrote: >> Indeed. It might be a bit awkward for packagers to depend on something >> from Software Collections, for example because they come as separate >> trees in /opt that are by default not in your path or dynamic loader >> path - one needs to run everything via a scl wrapper or source the >> /opt/rh/llvm-toolset-7/enable file to get the appropriate PATH and >> LD_LIBRARY_PATH settings, But it seems doable. > It'd be just for clang, and they're not *forced* to do it, it's an > optional dependency. So I think I'm ok with that. The "software collections" stuff was still in its infancy when I left Red Hat, so things might've changed, but I'm pretty sure at the time it was verboten for any mainstream package to depend on an SCL one. But they very probably wouldn't want postgresql depending on a compiler package even if the dependency was mainstream, so I rather doubt that you'll ever see an --enable-jit PG build out of there, making this most likely moot as far as the official RH package goes. I don't know what Devrim's opinion might be about PGDG. regards, tom lane
Re: JIT compiling with LLVM v11
Hi, On 2018-03-15 17:19:23 +0100, Catalin Iacob wrote: > On Thu, Mar 15, 2018 at 1:20 AM, Andres Freund wrote: > > I don't really live in the RHEL world, but I wonder if > > https://developers.redhat.com/blog/2017/10/04/red-hat-adds-go-clangllvm-rust-compiler-toolsets-updates-gcc/ > > is relevant? > > Indeed. It might be a bit awkward for packagers to depend on something > from Software Collections, for example because they come as separate > trees in /opt that are by default not in your path or dynamic loader > path - one needs to run everything via a scl wrapper or source the > /opt/rh/llvm-toolset-7/enable file to get the appropriate PATH and > LD_LIBRARY_PATH settings, But it seems doable. It'd be just for clang, and they're not *forced* to do it, it's an optional dependency. So I think I'm ok with that. > I just installed llvm-toolset-7 (the LLVM version is 4.0.1) on RHEL > 7.4 and did a build of your tree at > 475b4da439ae397345ab3df509e0e8eb26a8ff39. make installcheck passes for > both the default config and a server forced to jit everything (I > think) via: > jit_above_cost = '0' > jit_inline_above_cost = '0' > jit_optimize_above_cost = '0' > > As a side note, this increases the runtime from approx 4 min to 18 > min. Sure, that jits everything, which is obviously pointless to do for performancereasons. Especially SQL functions play very badly, because they're replanned every execution. But it's good for testing ;) > Disabling jit completely with -1 in all of the above yields 3 min > 48s, close to the default question raising maybe the question of how > much coverage does jit get with the default config. A bit, but not hugely so. I'm not too concerned about that. I plan to stand up a few buildfarm animals testing JITing with everything on w/ various LLVM versions. > The build was with the newer gcc 7.2.1 from the aforementioned > collections, I'll try the system gcc as well. I run a buildfarm animal > (katydid) on this RHEL. When JIT gets committed I'll make it use > --with-llvm against this Software Collections LLVM. Cool! Thanks for testing! Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On Thu, Mar 15, 2018 at 1:20 AM, Andres Freund wrote: > I don't really live in the RHEL world, but I wonder if > https://developers.redhat.com/blog/2017/10/04/red-hat-adds-go-clangllvm-rust-compiler-toolsets-updates-gcc/ > is relevant? Indeed. It might be a bit awkward for packagers to depend on something from Software Collections, for example because they come as separate trees in /opt that are by default not in your path or dynamic loader path - one needs to run everything via a scl wrapper or source the /opt/rh/llvm-toolset-7/enable file to get the appropriate PATH and LD_LIBRARY_PATH settings, But it seems doable. I just installed llvm-toolset-7 (the LLVM version is 4.0.1) on RHEL 7.4 and did a build of your tree at 475b4da439ae397345ab3df509e0e8eb26a8ff39. make installcheck passes for both the default config and a server forced to jit everything (I think) via: jit_above_cost = '0' jit_inline_above_cost = '0' jit_optimize_above_cost = '0' As a side note, this increases the runtime from approx 4 min to 18 min. Disabling jit completely with -1 in all of the above yields 3 min 48s, close to the default question raising maybe the question of how much coverage does jit get with the default config. The build was with the newer gcc 7.2.1 from the aforementioned collections, I'll try the system gcc as well. I run a buildfarm animal (katydid) on this RHEL. When JIT gets committed I'll make it use --with-llvm against this Software Collections LLVM. > Appears to be available on centos too > https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/ Indeed they are available for CentOS as well.
Re: JIT compiling with LLVM v11
Hi, On 2018-03-14 22:36:52 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-03-13 15:29:33 -0700, Andres Freund wrote: > >> On 2018-03-14 10:32:40 +1300, Thomas Munro wrote: > >>> It looks > >>> like -fexcess-precision-standard is coming from a configure test that > >>> was run against ${CC}, not against ${CLANG}, so I hacked the generated > >>> src/Makefile.global to remove that too, just to see if I could get > >>> past that. > > >> Yea, I'd hoped we could avoid duplicating all the configure tests, but > >> maybe not :(. > > > I've mostly done that now (not pushed). I've created new > > PGAC_PROG_VARCC_VARFLAGS_OPT(compiler variable, flag variable, testflag) > > function, which now is used to implement PGAC_PROG_CC_CFLAGS_OPT and > > PGAC_PROG_CC_VAR_OPT (similar for CXX). That makes it reasonable to > > test the variables clang recognizes separately. > > Meh. Why? The necessary configure code isn't that large: # Test for behaviour changing compiler flags, to keep compatibility # with compiler used for normal postgres code. XXX expand if test "$with_llvm" = yes ; then PGAC_PROG_VARCC_VARFLAGS_OPT(CLANG, BITCODE_CFLAGS, [-fno-strict-aliasing]) PGAC_PROG_VARCC_VARFLAGS_OPT(CLANG, BITCODE_CFLAGS, [-fwrapv]) PGAC_PROG_VARCC_VARFLAGS_OPT(CLANG, BITCODE_CFLAGS, [-fexcess-precision=standard]) AC_SUBST(BITCODE_CFLAGS, $BITCODE_CFLAGS) fi If the relevant clang version doesn't understand, say -fno-strict-aliasing, then we'd in trouble already if it's required. After all we do support compiling postgres with clang. > I agree with Thomas' concern that it's not clear we can or should > just ignore discrepancies between the -f options supported by the C > and CLANG compilers. What's the precise concern here? We pass these flags to work around compiler issues / "defining our standard". As I said above, if we do not know the right flags to make clang behave sensibly, we're in trouble already. For a good part of the code we already want to be compatible with compiling postgres with one compiler, and linking to libraries compiled with something else. > Is it really so necessary to bring a second compiler into the mix for > this? Why not just insist that JIT is only supported if the main build > is done with clang, too? My experience with mixing results from different > compilers is, eh, not positive. I don't like that option. It doesn't really buy us much, a few lines of config code, and one additional configure option that should normally be autodected from the environment. Requiring a specific compiler will be terrible on windows, seems out of line how we do development, requires using clang which is still generates a bit slower code, prevent getting gcc warnings etc. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
Andres Freund writes: > On 2018-03-13 15:29:33 -0700, Andres Freund wrote: >> On 2018-03-14 10:32:40 +1300, Thomas Munro wrote: >>> It looks >>> like -fexcess-precision-standard is coming from a configure test that >>> was run against ${CC}, not against ${CLANG}, so I hacked the generated >>> src/Makefile.global to remove that too, just to see if I could get >>> past that. >> Yea, I'd hoped we could avoid duplicating all the configure tests, but >> maybe not :(. > I've mostly done that now (not pushed). I've created new > PGAC_PROG_VARCC_VARFLAGS_OPT(compiler variable, flag variable, testflag) > function, which now is used to implement PGAC_PROG_CC_CFLAGS_OPT and > PGAC_PROG_CC_VAR_OPT (similar for CXX). That makes it reasonable to > test the variables clang recognizes separately. Meh. I agree with Thomas' concern that it's not clear we can or should just ignore discrepancies between the -f options supported by the C and CLANG compilers. Is it really so necessary to bring a second compiler into the mix for this? Why not just insist that JIT is only supported if the main build is done with clang, too? My experience with mixing results from different compilers is, eh, not positive. regards, tom lane
Re: JIT compiling with LLVM v11
Hi, On 2018-03-13 15:29:33 -0700, Andres Freund wrote: > On 2018-03-14 10:32:40 +1300, Thomas Munro wrote: > > I decided to try this on a CentOS 7.2 box. It has LLVM 3.9 in the > > 'epel' package repo, but unfortunately it only has clang 3.4. > > That's a bit odd, given llvm and clang really live in the same repo... I don't really live in the RHEL world, but I wonder if https://developers.redhat.com/blog/2017/10/04/red-hat-adds-go-clangllvm-rust-compiler-toolsets-updates-gcc/ is relevant? Appears to be available on centos too https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/ I checked it out and supporting 3.4 would be a bit painful due to not being able to directly emit module summaries. We could support that by building the summaries separately using LLVM, but that'd be either slower for everyone, or we'd need somewhat finnicky conditionals. > > clang: error: unknown argument: '-fexcess-precision=standard' > > clang: error: unknown argument: '-flto=thin' > > > > Ok, so I hacked src/Makefile.global.in to remove -flto=thin. > > I think I can get actually rid of that entirely. Err, no, not really. Would increase overhead due to separate module summary generation, so I'd rather not do it. > > It looks > > like -fexcess-precision-standard is coming from a configure test that > > was run against ${CC}, not against ${CLANG}, so I hacked the generated > > src/Makefile.global to remove that too, just to see if I could get > > past that. > > Yea, I'd hoped we could avoid duplicating all the configure tests, but > maybe not :(. I've mostly done that now (not pushed). I've created new PGAC_PROG_VARCC_VARFLAGS_OPT(compiler variable, flag variable, testflag) function, which now is used to implement PGAC_PROG_CC_CFLAGS_OPT and PGAC_PROG_CC_VAR_OPT (similar for CXX). That makes it reasonable to test the variables clang recognizes separately. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
Hi, On 2018-03-14 10:32:40 +1300, Thomas Munro wrote: > I decided to try this on a CentOS 7.2 box. It has LLVM 3.9 in the > 'epel' package repo, but unfortunately it only has clang 3.4. That's a bit odd, given llvm and clang really live in the same repo... > clang: error: unknown argument: '-fexcess-precision=standard' > clang: error: unknown argument: '-flto=thin' > > Ok, so I hacked src/Makefile.global.in to remove -flto=thin. I think I can get actually rid of that entirely. > It looks > like -fexcess-precision-standard is coming from a configure test that > was run against ${CC}, not against ${CLANG}, so I hacked the generated > src/Makefile.global to remove that too, just to see if I could get > past that. Yea, I'd hoped we could avoid duplicating all the configure tests, but maybe not :(. > Then I could build successfully and make check passed. I did see one warning: > > In file included from execExpr.c:39: > ../../../src/include/jit/jit.h:36:3: warning: redefinition of typedef > 'JitProviderCallbacks' is a C11 feature [-Wtypedef-redefinition] > } JitProviderCallbacks; > ^ > ../../../src/include/jit/jit.h:22:37: note: previous definition is here > typedef struct JitProviderCallbacks JitProviderCallbacks; > ^ Yep. Removed the second superflous / redundant typedef. Will push a heavily rebased version in a bit, will include fix for this. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On Thu, Mar 1, 2018 at 9:02 PM, Andres Freund wrote: > Biggest changes: > - LLVM 3.9 - master are now supported. This includes a good chunk of > work by Pierre Ducroquet. I decided to try this on a CentOS 7.2 box. It has LLVM 3.9 in the 'epel' package repo, but unfortunately it only has clang 3.4. I suppose it's important to make this work for RHEL7 using only dependencies that can be met by the vendor package repos? Maybe someone who knows more about CentOS/RHE could tell me if I'm mistaken and there is a way to get a more modern clang from a reputable repo that our packages could depend on, though I release that clang is only a build dependency, not a runtime one. I'm unsure how that constrains things. clang: "clang version 3.4.2 (tags/RELEASE_34/dot2-final)" gcc and g++: "gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)" llvm: "3.9.1" First problem: clang: error: unknown argument: '-fexcess-precision=standard' clang: error: unknown argument: '-flto=thin' Ok, so I hacked src/Makefile.global.in to remove -flto=thin. It looks like -fexcess-precision-standard is coming from a configure test that was run against ${CC}, not against ${CLANG}, so I hacked the generated src/Makefile.global to remove that too, just to see if I could get past that. I don't know if there was another way to control floating point precision in ancient clang before they adopted the GCC-compatible flag, but it would seem slightly fishy to have .o files and .bc files compiled with different floating point settings because then you could get different answers depending on whether your expression is JITted. Then I could build successfully and make check passed. I did see one warning: In file included from execExpr.c:39: ../../../src/include/jit/jit.h:36:3: warning: redefinition of typedef 'JitProviderCallbacks' is a C11 feature [-Wtypedef-redefinition] } JitProviderCallbacks; ^ ../../../src/include/jit/jit.h:22:37: note: previous definition is here typedef struct JitProviderCallbacks JitProviderCallbacks; ^ That's a legit complaint. -- Thomas Munro http://www.enterprisedb.com
Re: JIT compiling with LLVM v11
Andres Freund writes: > On 2018-03-13 14:36:44 -0400, Robert Haas wrote: >> I realize that EXPLAIN (JIT OFF) may sound like it's intended to >> disable JIT itself > Yea, that's what I'm concerned about. >> , but I think it's pretty clear that EXPLAIN (BUFFERS OFF) does not >> disable the use of actual buffers, only the display of buffer-related >> information. > Hm. FWIW, I agree with Robert's preference for just JIT here. The "info" bit isn't conveying anything. And we've never had any EXPLAIN options that actually change the behavior of the explained command, only ones that change the amount of info displayed. I don't see why we'd consider JIT an exception to that. regards, tom lane
Re: JIT compiling with LLVM v11
Hi, On 2018-03-13 14:36:44 -0400, Robert Haas wrote: > On Mon, Mar 12, 2018 at 5:04 PM, Andres Freund wrote: > > Currently a handful of explain outputs in the regression tests change > > output when compiled with JITing. Therefore I'm thinking of adding > > JITINFO or such option, which can be set to false for those tests? > > Can we spell that JIT or at least JIT_INFO? The latter works, I don't have a strong opinion on that. For now I've just tied it to COSTS off. > I realize that EXPLAIN (JIT OFF) may sound like it's intended to > disable JIT itself Yea, that's what I'm concerned about. > , but I think it's pretty clear that EXPLAIN (BUFFERS OFF) does not > disable the use of actual buffers, only the display of buffer-related > information. Hm. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On Mon, Mar 12, 2018 at 5:04 PM, Andres Freund wrote: > Currently a handful of explain outputs in the regression tests change > output when compiled with JITing. Therefore I'm thinking of adding > JITINFO or such option, which can be set to false for those tests? Can we spell that JIT or at least JIT_INFO? I realize that EXPLAIN (JIT OFF) may sound like it's intended to disable JIT itself, but I think it's pretty clear that EXPLAIN (BUFFERS OFF) does not disable the use of actual buffers, only the display of buffer-related information. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
Re: JIT compiling with LLVM v11
On 2018-03-13 10:25:49 -0400, Peter Eisentraut wrote: > On 3/12/18 17:04, Andres Freund wrote: > > │ JIT: > > │ > > │ Functions: 4 > > │ > > │ Inlining: false > > │ > > │ Inlining Time: 0.000 > > │ > > │ Optimization: false > > │ > > │ Optimization Time: 5.023 > > │ > > │ Emission Time: 34.987 > > │ > > The time quantities need some units. > > > │ Execution time: 46.277 ms > > │ > > like this :) Yea, I know. I was planning to start a thread about that. explain.c is littered with code like if (es->format == EXPLAIN_FORMAT_TEXT) appendStringInfo(es->str, "Planning time: %.3f ms\n", 1000.0 * plantime); else ExplainPropertyFloat("Planning Time", 1000.0 * plantime, 3, es); which, to me, is bonkers. I think we should add add 'const char *unit' parameter to at least ExplainProperty{Float,Integer,Long}? Or a *Unit version of them doing so, allowing a bit more gradual change? Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On 3/12/18 17:04, Andres Freund wrote: > │ JIT: >│ > │ Functions: 4 >│ > │ Inlining: false >│ > │ Inlining Time: 0.000 >│ > │ Optimization: false >│ > │ Optimization Time: 5.023 >│ > │ Emission Time: 34.987 >│ The time quantities need some units. > │ Execution time: 46.277 ms >│ like this :) -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: JIT compiling with LLVM v11
Hi, On 2018-03-12 17:14:00 -0400, Tom Lane wrote: > Andres Freund writes: > > Currently a handful of explain outputs in the regression tests change > > output when compiled with JITing. Therefore I'm thinking of adding > > JITINFO or such option, which can be set to false for those tests? > > Maintaining duplicate output for them seems painful. Better ideas? > > Why not just suppress that info when COSTS OFF is specified? I wondered about that too. But that'd mean it'd be harder to write a test that tests the planning bits of JITing (i.e. decision whether to use optimization & inlining or not) . Not sure if it's worth adding complexity to be able to do so. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
Andres Freund writes: > Currently a handful of explain outputs in the regression tests change > output when compiled with JITing. Therefore I'm thinking of adding > JITINFO or such option, which can be set to false for those tests? > Maintaining duplicate output for them seems painful. Better ideas? Why not just suppress that info when COSTS OFF is specified? regards, tom lane
Re: JIT compiling with LLVM v11
On 2018-03-09 13:08:36 -0800, Andres Freund wrote: > On 2018-03-09 15:42:24 -0500, Peter Eisentraut wrote: > > What I'd quite like is if EXPLAIN or EXPLAIN ANALYZE showed something > > about what kind of JIT processing was done, if any, to help with this > > kind of testing. > > Yea, I like that. I think we can only show that when timing is on, > because otherwise the tests will not be stable depending on --with-jit > being specified or not. > > So I'm thinking of displaying it similar to the "Planning time" piece, > i.e. depending on es->summary being enabled. It'd be good to display the > inline/optimize/emit times too. I think we can just store it in the > JitContext. But the inline/optimize/emission times will only be > meaningful when the query is actually executed, I don't see a way around > that... Not yet really happy with how it exactly looks, but here's my current state: tpch_10[20923][1]=# ;explain (format text, analyze, timing off) SELECT relkind, relname FROM pg_class pgc WHERE relkind = 'r'; ┌┐ │ QUERY PLAN │ ├┤ │ Seq Scan on pg_class pgc (cost=0.00..15.70 rows=77 width=65) (actual rows=77 loops=1) │ │ Filter: (relkind = 'r'::"char") │ │ Rows Removed by Filter: 299 │ │ Planning time: 0.187 ms │ │ JIT: │ │ Functions: 4 │ │ Inlining: false │ │ Optimization: false │ │ Execution time: 72.229 ms │ └┘ (9 rows) tpch_10[20923][1]=# ;explain (format text, analyze, timing on) SELECT relkind, relname FROM pg_class pgc WHERE relkind = 'r'; ┌┐ │ QUERY PLAN │ ├┤ │ Seq Scan on pg_class pgc (cost=0.00..15.70 rows=77 width=65) (actual time=40.570..40.651 rows=77 loops=1) │ │ Filter: (relkind = 'r'::"char") │ │ Rows Removed by Filter: 299 │ │ Planning time: 0.138 ms │ │ JIT: │ │ Functions: 4 │ │ Inlining: false │ │ Inlining Time: 0.000 │ │ Optimization: false │ │ Optimization Time: 5.023 │ │ Emission Time: 34.987 │ │ Execution time: 46.277 ms │ └┘ (12 rows) json (excerpt): │ "Triggers": [ ↵│ │ ], ↵│ │ "JIT": {↵│ │ "Functions": 4, ↵│ │ "Inlining": false,↵│ │ "Inlining Time": 0.000, ↵│ │ "Optimization": false,↵│ │ "Optimization Time": 9.701, ↵│ │ "Emission Time": 52.951 ↵│ │ }, ↵│ │ "Execution Time": 70.292↵│ I'm not at all wedded to the current format, but I feel like that's the basic functionality needed? Right now the JIT bit will only be displayed if at least one JITed function has been emitted. Otherwise we'll just create noise for everyone. Currently a handful of explain outputs in
Re: JIT compiling with LLVM v11
On 3/12/18 13:05, Andres Freund wrote: >> will *not* use JIT, becaue jit_expressions applies at execution time. > Right. It'd be easy to change that so jit_expressions=off wouldn't have > an effect there anymore. But I'm not sure we want that? I don't have a > strong feeling about this, except that I think jit_above_cost etc should > apply at plan, not execution time. I lean toward making everything apply at plan time. Not only is that easier in the current code structure, but over time we'll probably want to add more detailed planner knobs, e.g., perhaps an alternative cpu_tuple_cost, and all of that would be a planner setting. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: JIT compiling with LLVM v11
On 2018-03-12 11:21:36 -0400, Peter Eisentraut wrote: > On 3/11/18 14:25, Andres Freund wrote: > >> It's perhaps a bit confusing that some of the jit_* settings take effect > >> at plan time and some at execution time. At the moment, this mainly > >> affects me reading the code ;-), but it would also have some effect on > >> prepared statements and such. > > Not quite sure what you mean? > > I haven't tested this, but what appears to be the case is that > > SET jit_above_cost = 0; > PREPARE foo AS SELECT ; > SET jit_above_cost = infinity; > EXECUTE foo; > > will use JIT, because jit_above_cost applies at plan time, whereas > > SET jit_expressions = on; > PREPARE foo AS SELECT ; > SET jit_expressions = off; > EXECUTE foo; > > will *not* use JIT, becaue jit_expressions applies at execution time. Right. It'd be easy to change that so jit_expressions=off wouldn't have an effect there anymore. But I'm not sure we want that? I don't have a strong feeling about this, except that I think jit_above_cost etc should apply at plan, not execution time. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On 3/11/18 14:25, Andres Freund wrote: >> It's perhaps a bit confusing that some of the jit_* settings take effect >> at plan time and some at execution time. At the moment, this mainly >> affects me reading the code ;-), but it would also have some effect on >> prepared statements and such. > Not quite sure what you mean? I haven't tested this, but what appears to be the case is that SET jit_above_cost = 0; PREPARE foo AS SELECT ; SET jit_above_cost = infinity; EXECUTE foo; will use JIT, because jit_above_cost applies at plan time, whereas SET jit_expressions = on; PREPARE foo AS SELECT ; SET jit_expressions = off; EXECUTE foo; will *not* use JIT, becaue jit_expressions applies at execution time. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: JIT compiling with LLVM v11
On 2018-03-11 13:19:57 -0400, Peter Eisentraut wrote: > On 3/9/18 15:56, Andres Freund wrote: > > I think that's largely that unnecessary trivial queries get JITed and > > optimized, because the stats are entirely completely off. > > Right. I instrumented this a bit, and there are indeed two handfuls of > queries that exceed the default JIT thresholds, as well as a few that > trigger JIT because they disable some enable_* planner setting, as > previously discussed. > > Should we throw in some ANALYZEs to avoid this? Hm, I'd actually lean to just leave it as is for now. JITing halfway random queries isn't actually that bad... If we get fed up with the additional time after a while, we can do something then? > It's perhaps a bit confusing that some of the jit_* settings take effect > at plan time and some at execution time. At the moment, this mainly > affects me reading the code ;-), but it would also have some effect on > prepared statements and such. Not quite sure what you mean? > Also, jit_tuple_deforming is apparently used only when jit_expressions > is on. Right. I've not found a good place to hook into that has enough context to do JITed deforming otherwise. I'm inclined to just relegate jit_tuple_deforming to debugging status (i.e. exclude from show all, docs etc) for now. > So, we should work toward more clarity on all these different settings, > what they are useful for, when to set them, how they interact. Yep. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On 3/9/18 15:42, Peter Eisentraut wrote: > The default of jit_above_cost = 50 seems pretty good. I constructed > a query that cost about 45 where the run time with and without JIT > were about even. This is obviously very limited testing, but it's a > good start. Actually, the default in your latest code is 10, which per my analysis would be too low. Did you arrive at that setting based on testing? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: JIT compiling with LLVM v11
On 3/9/18 15:56, Andres Freund wrote: > On 2018-03-09 15:28:19 -0500, Peter Eisentraut wrote: >> On 3/6/18 15:16, Andres Freund wrote: >>> 2) Don't load the JIT provider until fully needed. Right now >>>jit_compile_expr() will load the jit provider even if not really >>>needed. We should probably move the first two return blocks in >>>llvm_compile_expr() into jit_compile_expr(), to avoid that. >> >> I see that you have implemented that, but it doesn't seem to have helped >> with my make installcheck times. > > What's the exact comparison you're looking at? I'm just running `time make installcheck` with default settings, as described in my message from March 6. > I think that's largely that unnecessary trivial queries get JITed and > optimized, because the stats are entirely completely off. Right. I instrumented this a bit, and there are indeed two handfuls of queries that exceed the default JIT thresholds, as well as a few that trigger JIT because they disable some enable_* planner setting, as previously discussed. Should we throw in some ANALYZEs to avoid this? If I set jit_expressions = off, then the timings match again. It's perhaps a bit confusing that some of the jit_* settings take effect at plan time and some at execution time. At the moment, this mainly affects me reading the code ;-), but it would also have some effect on prepared statements and such. Also, jit_tuple_deforming is apparently used only when jit_expressions is on. So, we should work toward more clarity on all these different settings, what they are useful for, when to set them, how they interact. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: JIT compiling with LLVM v11
On 2018-03-09 15:42:24 -0500, Peter Eisentraut wrote: > For jit_optimize_above_cost, in my testing, any query where JIT payed > off was even faster with optimizing. So right now I don't see a need to > make this a separate setting. Maybe just make it an on/off setting for > experimenting. I'd prefer to be more defensive here. The time needed for JITing without optimization is roughly linear, whereas optimization is definitely not linear with input size. > For inlining, I haven't been able to get a clear picture. It's a bit > faster perhaps, but the optimizing dominates it. I don't have a clear > mental model for what kind of returns to expect from this. Yea, you need longrunning queries to benefit significantly. There's a *lot* more potential once some structural issues with the expression format (both with and without JIT) are fixed. > What I'd quite like is if EXPLAIN or EXPLAIN ANALYZE showed something > about what kind of JIT processing was done, if any, to help with this > kind of testing. Yea, I like that. I think we can only show that when timing is on, because otherwise the tests will not be stable depending on --with-jit being specified or not. So I'm thinking of displaying it similar to the "Planning time" piece, i.e. depending on es->summary being enabled. It'd be good to display the inline/optimize/emit times too. I think we can just store it in the JitContext. But the inline/optimize/emission times will only be meaningful when the query is actually executed, I don't see a way around that... Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On 2018-03-09 15:28:19 -0500, Peter Eisentraut wrote: > On 3/6/18 15:16, Andres Freund wrote: > > 2) Don't load the JIT provider until fully needed. Right now > >jit_compile_expr() will load the jit provider even if not really > >needed. We should probably move the first two return blocks in > >llvm_compile_expr() into jit_compile_expr(), to avoid that. > > I see that you have implemented that, but it doesn't seem to have helped > with my make installcheck times. What's the exact comparison you're looking at? I think that's largely that unnecessary trivial queries get JITed and optimized, because the stats are entirely completely off. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On 3/6/18 10:29, Peter Eisentraut wrote: > I think taking the total cost as the triggering threshold is probably > good enough for a start. The cost modeling can be refined over time. I looked into this a bit more. The default of jit_above_cost = 50 seems pretty good. I constructed a query that cost about 45 where the run time with and without JIT were about even. This is obviously very limited testing, but it's a good start. For jit_optimize_above_cost, in my testing, any query where JIT payed off was even faster with optimizing. So right now I don't see a need to make this a separate setting. Maybe just make it an on/off setting for experimenting. For inlining, I haven't been able to get a clear picture. It's a bit faster perhaps, but the optimizing dominates it. I don't have a clear mental model for what kind of returns to expect from this. What I'd quite like is if EXPLAIN or EXPLAIN ANALYZE showed something about what kind of JIT processing was done, if any, to help with this kind of testing. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: JIT compiling with LLVM v11
On 3/6/18 15:16, Andres Freund wrote: > 2) Don't load the JIT provider until fully needed. Right now >jit_compile_expr() will load the jit provider even if not really >needed. We should probably move the first two return blocks in >llvm_compile_expr() into jit_compile_expr(), to avoid that. I see that you have implemented that, but it doesn't seem to have helped with my make installcheck times. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: JIT compiling with LLVM v11
On Fri, Mar 9, 2018 at 9:12 AM, Andres Freund wrote: > Or even in core LLVM, which has this nice comment: > > // If user asked for the 'native' CPU, we need to autodetect features. > // This is necessary for x86 where the CPU might not support all the > // features the autodetected CPU name lists in the target. For example, > // not all Sandybridge processors support AVX. > if (MCPU == "native") { > > which pretty much describes the issue you're apparently hitting. > > I've pushed an attempted fix (needs a comment, but works here). === All 186 tests passed. === That did the trick. Thanks! -- Thomas Munro http://www.enterprisedb.com
Re: JIT compiling with LLVM v11
On 2018-03-08 11:58:41 -0800, Andres Freund wrote: > I think we can easily fix this by behaving like clang, which uses > llvm::sys::getHostCPUFeatures(HostFeatures) to built the feature list: > > // If -march=native, autodetect the feature list. > if (const Arg *A = Args.getLastArg(clang::driver::options::OPT_march_EQ)) { > if (StringRef(A->getValue()) == "native") { > llvm::StringMap HostFeatures; > if (llvm::sys::getHostCPUFeatures(HostFeatures)) > for (auto &F : HostFeatures) > Features.push_back( > Args.MakeArgString((F.second ? "+" : "-") + F.first())); > } > } > > which seems easy enough. Or even in core LLVM, which has this nice comment: // If user asked for the 'native' CPU, we need to autodetect features. // This is necessary for x86 where the CPU might not support all the // features the autodetected CPU name lists in the target. For example, // not all Sandybridge processors support AVX. if (MCPU == "native") { which pretty much describes the issue you're apparently hitting. I've pushed an attempted fix (needs a comment, but works here). Greetings, Andres Freund
Re: JIT compiling with LLVM v11
Hi, On 2018-03-09 00:33:03 +1300, Thomas Munro wrote: > On Wed, Mar 7, 2018 at 3:49 PM, Thomas Munro > wrote: > > make check at today's HEAD of your jit branch crashes on my FreeBSD > > box. The first thing to crash is this query from point.sql: > > > > LOG: server process (PID 87060) was terminated by signal 4: Illegal > > instruction > > DETAIL: Failed process was running: SELECT '' AS thirtysix, p1.f1 AS > > point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dist > >FROM POINT_TBL p1, POINT_TBL p2 > >ORDER BY dist, p1.f1[0], p2.f1[0]; > > Hmm. It's trying to execute an AVX instruction. Ah, that's interesting. > I am not sure if that is real though, because the stack is immediately > corrupted. I don't think the stack is corrupted at all, it's just that lldb can't unwind with functions it doesn't know. To add that capability I've a pending LLVM patch. > So either func is not really a function, or it is but was > compiled for the wrong target. I see that you call > LLVMCreateTargetMachine() with the result of LLVMGetHostCPUName() as > cpu. For me that's "ivybridge", so I tried hard coding "generic" > instead and it didn't help. Hm. > I see that you say "" for features, where > is where one would normally put "avx" to turn on AVX instructions, so > I think perhaps that theory is entirely bogus. Could you try a -avx in features and see whether it fixes things? This kinda suggests an LLVM bug or at least an oddity, but I'll try to drill down more into this. Is this a native machine or a VM? I think we can easily fix this by behaving like clang, which uses llvm::sys::getHostCPUFeatures(HostFeatures) to built the feature list: // If -march=native, autodetect the feature list. if (const Arg *A = Args.getLastArg(clang::driver::options::OPT_march_EQ)) { if (StringRef(A->getValue()) == "native") { llvm::StringMap HostFeatures; if (llvm::sys::getHostCPUFeatures(HostFeatures)) for (auto &F : HostFeatures) Features.push_back( Args.MakeArgString((F.second ? "+" : "-") + F.first())); } } which seems easy enough. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On Wed, Mar 7, 2018 at 3:49 PM, Thomas Munro wrote: > make check at today's HEAD of your jit branch crashes on my FreeBSD > box. The first thing to crash is this query from point.sql: > > LOG: server process (PID 87060) was terminated by signal 4: Illegal > instruction > DETAIL: Failed process was running: SELECT '' AS thirtysix, p1.f1 AS > point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dist >FROM POINT_TBL p1, POINT_TBL p2 >ORDER BY dist, p1.f1[0], p2.f1[0]; Hmm. It's trying to execute an AVX instruction. * thread #1, stop reason = breakpoint 1.1 frame #0: llvmjit.so`ExecRunCompiledExpr(state=0x000801de4880, econtext=0x000801de3560, isNull="") at llvmjit_expr.c:432 429 430 state->evalfunc = func; 431 -> 432 return func(state, econtext, isNull); 433 } 434 435 static void emit_lifetime_end(ExprState *state, LLVMModuleRef mod, LLVMBuilderRef b); (lldb) s Process 44513 stopped * thread #1, stop reason = signal SIGILL: privileged instruction frame #0: 0x000801157193 -> 0x801157193: vmovsd (%rax), %xmm0 ; xmm0 = mem[0],zero 0x801157197: vmovsd 0x8(%rax), %xmm1 ; xmm1 = mem[0],zero 0x80115719c: vsubsd (%rcx), %xmm0, %xmm2 0x8011571a0: vsubsd 0x8(%rcx), %xmm1, %xmm0 (lldb) bt * thread #1, stop reason = signal SIGILL: privileged instruction * frame #0: 0x000801157193 This is running on a "Intel(R) Celeron(R) CPU G1610T @ 2.30GHz" with no AVX. I am not sure if that is real though, because the stack is immediately corrupted. So either func is not really a function, or it is but was compiled for the wrong target. I see that you call LLVMCreateTargetMachine() with the result of LLVMGetHostCPUName() as cpu. For me that's "ivybridge", so I tried hard coding "generic" instead and it didn't help. I see that you say "" for features, where is where one would normally put "avx" to turn on AVX instructions, so I think perhaps that theory is entirely bogus. -- Thomas Munro http://www.enterprisedb.com
Re: JIT compiling with LLVM v11
On Tue, Mar 6, 2018 at 10:39 PM, Andres Freund wrote: > [more commits] +* OSX prefixes all object level symbols with an underscore. But neither "macOS" (see commit da6c4f6c and all mentions since). make check at today's HEAD of your jit branch crashes on my FreeBSD box. The first thing to crash is this query from point.sql: LOG: server process (PID 87060) was terminated by signal 4: Illegal instruction DETAIL: Failed process was running: SELECT '' AS thirtysix, p1.f1 AS point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dist FROM POINT_TBL p1, POINT_TBL p2 ORDER BY dist, p1.f1[0], p2.f1[0]; Unfortunately when I tried to load the core file into lldb, the stack is like this: * thread #1, name = 'postgres', stop reason = signal SIGILL * frame #0: 0x000800e7c1ea Apparently the generated code is nuking the stack and executing garbage? I don't have time to investigate right now, and this may indicate something busted in my environment, but I thought this might tell you something. These variants of that query don't crash (even though I set jit_above_cost = 0 and checked that it's actually JIT-ing), which might be clues: -- no p1.f1 <-> p2.f1 SELECT p1.f1 AS point1, p2.f1 AS point2 FROM POINT_TBL p1, POINT_TBL p2 ORDER BY p1.f1[0], p2.f1[0]; -- no join SELECT p1.f1 <-> p1.f1 AS dist FROM POINT_TBL p1 ORDER BY 1; These variants do crash: -- p1.f1 <-> p2.f1 in order by, but not select list SELECT p1.f1 AS point1, p2.f1 AS point2 FROM POINT_TBL p1, POINT_TBL p2 ORDER BY p1.f1 <-> p2.f1, p1.f1[0], p2.f1[0]; -- p1.f1 <-> p2.f1 in select list, but not in order by SELECT p1.f1 AS point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dist FROM POINT_TBL p1, POINT_TBL p2 ORDER BY p1.f1[0], p2.f1[0]; -- simple, with a join SELECT p1.f1 <-> p1.f1 AS dist FROM POINT_TBL p1, POINT_TBL p2 ORDER BY 1; I build it like this: ./configure \ --prefix=$HOME/install/ \ --enable-tap-tests \ --enable-cassert \ --enable-debug \ --enable-depend \ --with-llvm \ CC="ccache cc" CFLAGS="-O0" CXX="ccache c++" CXXFLAGS="-std=c++11" \ CLANG=/usr/local/llvm50/bin/clang \ LLVM_CONFIG=/usr/local/llvm50/bin/llvm-config \ --with-libraries="/usr/local/lib" \ --with-includes="/usr/local/include" -- Thomas Munro http://www.enterprisedb.com
Re: JIT compiling with LLVM v11
Andres Freund writes: > I'm not too worried about that scenario. If, for a cheap plan, the > planner ends up with a seqscan despite it being disabled, you're pretty > close to randomly choosing plans already, as the pruning doesn't work > well anymore (as the %1 percent fuzz factor in > compare_path_costs_fuzzily() swamps the actual plan costs). Something I've wanted to do for awhile is to get rid of disable_cost in favor of pruning disabled plans through logic rather than costing. I've looked at this once or twice, and it seems doable but not entirely trivial --- the sticky bits are places where you do need to allow a disabled plan type because there's no other alternative. But if we could get that done, it'd help with this sort of problem. regards, tom lane
Re: JIT compiling with LLVM v11
On 2018-03-06 12:16:01 -0800, Andres Freund wrote: > > I ran some performance assessments: > > > > merge base (0b1d1a038babff4aadf0862c28e7b667f1b12a30) > > > > make installcheck 3.14s user 3.34s system 17% cpu 37.954 total > > > > jit branch default settings > > > > make installcheck 3.17s user 3.30s system 13% cpu 46.596 total > > > > jit_above_cost=0 > > > > make installcheck 3.30s user 3.53s system 5% cpu 1:59.89 total > > > > jit_optimize_above_cost=0 (and jit_above_cost=0) > > > > make installcheck 3.44s user 3.76s system 1% cpu 8:12.42 total > > > > jit_inline_above_cost=0 (and jit_above_cost=0) > > > > make installcheck 3.32s user 3.62s system 2% cpu 5:35.58 total > > > > One can see the CPU savings quite nicely. > > I'm not quite sure what you mean by that. > > > > One obvious problem is that with the default settings, the test suite > > run gets about 15% slower. (These figures are reproducible over several > > runs.) Is there some debugging stuff turned on that would explain this? > > Or would just loading the jit module in each session cause this? > > I suspect it's loading the module. There's also another issue: For a lot queries in the tests the stats are way way way off because the relevant tables have never been analyzed. There's a few cases where costs are off by like 5-7 orders of magnitude... Greetings, Andres Freund
Re: JIT compiling with LLVM v11
Hi, On 2018-03-06 10:29:47 -0500, Peter Eisentraut wrote: > I think taking the total cost as the triggering threshold is probably > good enough for a start. The cost modeling can be refined over time. Cool. > We should document that both jit_optimize_above_cost and > jit_inline_above_cost require jit_above_cost to be set, or otherwise > nothing happens. Yea, that's a good plan. We could also change it so it would, but I don't think there's much point? > One problem I see is that if someone sets things like > enable_seqscan=off, the artificial cost increase created by those > settings would quite likely bump the query over the jit threshold, which > would alter the query performance characteristics in a way that the user > would not have intended. I don't have an idea how to address this right > now. I'm not too worried about that scenario. If, for a cheap plan, the planner ends up with a seqscan despite it being disabled, you're pretty close to randomly choosing plans already, as the pruning doesn't work well anymore (as the %1 percent fuzz factor in compare_path_costs_fuzzily() swamps the actual plan costs). > I ran some performance assessments: > > merge base (0b1d1a038babff4aadf0862c28e7b667f1b12a30) > > make installcheck 3.14s user 3.34s system 17% cpu 37.954 total > > jit branch default settings > > make installcheck 3.17s user 3.30s system 13% cpu 46.596 total > > jit_above_cost=0 > > make installcheck 3.30s user 3.53s system 5% cpu 1:59.89 total > > jit_optimize_above_cost=0 (and jit_above_cost=0) > > make installcheck 3.44s user 3.76s system 1% cpu 8:12.42 total > > jit_inline_above_cost=0 (and jit_above_cost=0) > > make installcheck 3.32s user 3.62s system 2% cpu 5:35.58 total > > One can see the CPU savings quite nicely. I'm not quite sure what you mean by that. > One obvious problem is that with the default settings, the test suite > run gets about 15% slower. (These figures are reproducible over several > runs.) Is there some debugging stuff turned on that would explain this? > Or would just loading the jit module in each session cause this? I suspect it's loading the module. There's two pretty easy avenues to improve this: 1) Attempt to load the JIT provider in postmaster, thereby avoiding a lot of redundant dynamic linker work if already installed. That's ~5-10 lines or such. I basically refrained from that because it's convenient to not have to restart the server during development (one can just reconnect and get a newer jit plugin). 2) Don't load the JIT provider until fully needed. Right now jit_compile_expr() will load the jit provider even if not really needed. We should probably move the first two return blocks in llvm_compile_expr() into jit_compile_expr(), to avoid that. > From the other results, we can see that one clearly needs quite a big > database to see a solid benefit from this. Right, until we've got caching this'll only be beneficial for ~1s+ analytics queries. Unfortunately caching requires some larger planner & executor surgery, so I don't want to go there at the same time (I'm already insane enough). > Do you have any information gathered about this so far? Any scripts > to create test databases and test queries? Yes. I've used tpc-h. Not because it's the greatest, but because it's semi conveniently available and a lot of others have experience with it already. Do you mean whether I've run a couple benchmarks? If so, yes. I'll schedule some more later - am on battery power rn. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
With the build issues in check, I'm looking at the configuration settings. I think taking the total cost as the triggering threshold is probably good enough for a start. The cost modeling can be refined over time. We should document that both jit_optimize_above_cost and jit_inline_above_cost require jit_above_cost to be set, or otherwise nothing happens. One problem I see is that if someone sets things like enable_seqscan=off, the artificial cost increase created by those settings would quite likely bump the query over the jit threshold, which would alter the query performance characteristics in a way that the user would not have intended. I don't have an idea how to address this right now. I ran some performance assessments: merge base (0b1d1a038babff4aadf0862c28e7b667f1b12a30) make installcheck 3.14s user 3.34s system 17% cpu 37.954 total jit branch default settings make installcheck 3.17s user 3.30s system 13% cpu 46.596 total jit_above_cost=0 make installcheck 3.30s user 3.53s system 5% cpu 1:59.89 total jit_optimize_above_cost=0 (and jit_above_cost=0) make installcheck 3.44s user 3.76s system 1% cpu 8:12.42 total jit_inline_above_cost=0 (and jit_above_cost=0) make installcheck 3.32s user 3.62s system 2% cpu 5:35.58 total One can see the CPU savings quite nicely. One obvious problem is that with the default settings, the test suite run gets about 15% slower. (These figures are reproducible over several runs.) Is there some debugging stuff turned on that would explain this? Or would just loading the jit module in each session cause this? >From the other results, we can see that one clearly needs quite a big database to see a solid benefit from this. Do you have any information gathered about this so far? Any scripts to create test databases and test queries? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: JIT compiling with LLVM v11
On 3/6/18 04:39, Andres Freund wrote: > I did, and reproduced. Turned out I just missed the error in the above > test. > > The bug was caused by one ifdef in get_LifetimeEnd() being wrong > (function is is overloaded starting in 5 rather than 4). The comment > above it even had it right... OK, it's fixed for me now. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: JIT compiling with LLVM v11
On 2018-03-05 14:01:05 -0800, Andres Freund wrote: > On 2018-03-05 13:36:04 -0800, Andres Freund wrote: > > On 2018-03-05 16:19:52 -0500, Peter Eisentraut wrote: > > > Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ... > > > > > > The build works out of the box with whatever the default system packages > > > are. > > > > > > Regression tests crash many times. One backtrace looks like this: > > > > > > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 > > > #1 0x7fd5b1730231 in __GI_abort () at abort.c:79 > > > #2 0x55c10a1555e3 in ExceptionalCondition > > > (conditionName=conditionName@entry=0x7fd5a245c2d8 > > > "!(LLVMGetIntrinsicID(fn))", > > > errorType=errorType@entry=0x7fd5a245bb1d "FailedAssertion", > > > fileName=fileName@entry=0x7fd5a245c294 "llvmjit_expr.c", > > > lineNumber=lineNumber@entry=193) at assert.c:54 > > > #3 0x7fd5a245510f in get_LifetimeEnd (mod=mod@entry=0x55c10b1db670) > > > at llvmjit_expr.c:193 > > > #4 0x7fd5a24553c8 in get_LifetimeEnd (mod=0x55c10b1db670) at > > > llvmjit_expr.c:233 > > > #5 BuildFunctionCall (context=context@entry=0x55c10b0ca340, > > > builder=builder@entry=0x55c10b225160, > > > mod=mod@entry=0x55c10b1db670, fcinfo=0x55c10b1a08b0, > > > v_fcinfo_isnull=v_fcinfo_isnull@entry=0x7ffc701f5c60) > > > at llvmjit_expr.c:244 > > > > Hm, that should be trivial to fix. Which version of llvm are you > > building against? There appear to be a lot of them in testing: > > https://packages.debian.org/search?keywords=llvm+dev&searchon=names&suite=testing§ion=all > > On Debian unstable, I built against a wide variety of branches: > > for v in 3.9 4.0 5.0 6.0;do rm -f ../config.cache;CLANG="ccache clang-$v" > LLVM_CONFIG=/usr/lib/llvm-$v/bin/llvm-config ../config.sh --with-llvm && make > -j16 -s install && make -s check;done > > All of those pass. I'll create a testing chroot. I did, and reproduced. Turned out I just missed the error in the above test. The bug was caused by one ifdef in get_LifetimeEnd() being wrong (function is is overloaded starting in 5 rather than 4). The comment above it even had it right... Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On 2018-03-05 13:36:04 -0800, Andres Freund wrote: > On 2018-03-05 16:19:52 -0500, Peter Eisentraut wrote: > > Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ... > > > > The build works out of the box with whatever the default system packages > > are. > > > > Regression tests crash many times. One backtrace looks like this: > > > > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 > > #1 0x7fd5b1730231 in __GI_abort () at abort.c:79 > > #2 0x55c10a1555e3 in ExceptionalCondition > > (conditionName=conditionName@entry=0x7fd5a245c2d8 > > "!(LLVMGetIntrinsicID(fn))", > > errorType=errorType@entry=0x7fd5a245bb1d "FailedAssertion", > > fileName=fileName@entry=0x7fd5a245c294 "llvmjit_expr.c", > > lineNumber=lineNumber@entry=193) at assert.c:54 > > #3 0x7fd5a245510f in get_LifetimeEnd (mod=mod@entry=0x55c10b1db670) > > at llvmjit_expr.c:193 > > #4 0x7fd5a24553c8 in get_LifetimeEnd (mod=0x55c10b1db670) at > > llvmjit_expr.c:233 > > #5 BuildFunctionCall (context=context@entry=0x55c10b0ca340, > > builder=builder@entry=0x55c10b225160, > > mod=mod@entry=0x55c10b1db670, fcinfo=0x55c10b1a08b0, > > v_fcinfo_isnull=v_fcinfo_isnull@entry=0x7ffc701f5c60) > > at llvmjit_expr.c:244 > > Hm, that should be trivial to fix. Which version of llvm are you > building against? There appear to be a lot of them in testing: > https://packages.debian.org/search?keywords=llvm+dev&searchon=names&suite=testing§ion=all On Debian unstable, I built against a wide variety of branches: for v in 3.9 4.0 5.0 6.0;do rm -f ../config.cache;CLANG="ccache clang-$v" LLVM_CONFIG=/usr/lib/llvm-$v/bin/llvm-config ../config.sh --with-llvm && make -j16 -s install && make -s check;done All of those pass. I'll create a testing chroot. Regards, Andres
Re: JIT compiling with LLVM v11
Hi, On 2018-03-05 12:17:30 -0800, Andres Freund wrote: > Writing up a patch that I can actually push. Thanks both to Thomas and > Peter for pointing me towards this issue! After screwing the first attempt at a fix, the second one seems to work nicely. With optimizations, inlining, etc all core tests pass on Thomas' machine. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On 2018-03-05 16:19:52 -0500, Peter Eisentraut wrote: > Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ... > > The build works out of the box with whatever the default system packages > are. > > Regression tests crash many times. One backtrace looks like this: > > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 > #1 0x7fd5b1730231 in __GI_abort () at abort.c:79 > #2 0x55c10a1555e3 in ExceptionalCondition > (conditionName=conditionName@entry=0x7fd5a245c2d8 > "!(LLVMGetIntrinsicID(fn))", > errorType=errorType@entry=0x7fd5a245bb1d "FailedAssertion", > fileName=fileName@entry=0x7fd5a245c294 "llvmjit_expr.c", > lineNumber=lineNumber@entry=193) at assert.c:54 > #3 0x7fd5a245510f in get_LifetimeEnd (mod=mod@entry=0x55c10b1db670) > at llvmjit_expr.c:193 > #4 0x7fd5a24553c8 in get_LifetimeEnd (mod=0x55c10b1db670) at > llvmjit_expr.c:233 > #5 BuildFunctionCall (context=context@entry=0x55c10b0ca340, > builder=builder@entry=0x55c10b225160, > mod=mod@entry=0x55c10b1db670, fcinfo=0x55c10b1a08b0, > v_fcinfo_isnull=v_fcinfo_isnull@entry=0x7ffc701f5c60) > at llvmjit_expr.c:244 Hm, that should be trivial to fix. Which version of llvm are you building against? There appear to be a lot of them in testing: https://packages.debian.org/search?keywords=llvm+dev&searchon=names&suite=testing§ion=all Greetings, Andres Freund
Re: JIT compiling with LLVM v11
Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ... The build works out of the box with whatever the default system packages are. Regression tests crash many times. One backtrace looks like this: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x7fd5b1730231 in __GI_abort () at abort.c:79 #2 0x55c10a1555e3 in ExceptionalCondition (conditionName=conditionName@entry=0x7fd5a245c2d8 "!(LLVMGetIntrinsicID(fn))", errorType=errorType@entry=0x7fd5a245bb1d "FailedAssertion", fileName=fileName@entry=0x7fd5a245c294 "llvmjit_expr.c", lineNumber=lineNumber@entry=193) at assert.c:54 #3 0x7fd5a245510f in get_LifetimeEnd (mod=mod@entry=0x55c10b1db670) at llvmjit_expr.c:193 #4 0x7fd5a24553c8 in get_LifetimeEnd (mod=0x55c10b1db670) at llvmjit_expr.c:233 #5 BuildFunctionCall (context=context@entry=0x55c10b0ca340, builder=builder@entry=0x55c10b225160, mod=mod@entry=0x55c10b1db670, fcinfo=0x55c10b1a08b0, v_fcinfo_isnull=v_fcinfo_isnull@entry=0x7ffc701f5c60) at llvmjit_expr.c:244 ... #16 0x55c10a0433ad in exec_simple_query ( query_string=0x55c10b096358 "SELECT COUNT(*) FROM test_tsquery WHERE keyword < 'new & york';") at postgres.c:1082 -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: JIT compiling with LLVM v11
Hi, On 2018-03-04 21:00:06 -0800, Andres Freund wrote: > > Looking at llvm_get_function(), the function that raises that error, I > > see that there are a few different paths here. I don't have > > HAVE_DECL_LLVMORCGETSYMBOLADDRESSIN defined, and I don't have LLVM < > > 5, so I should be getting the symbol address with > > LLVMOrcGetSymbolAddress(llvm_opt0_orc, &addr, mangled) or > > LLVMOrcGetSymbolAddress(llvm_opt3_orc, &addr, mangled), but clearly > > those are returning NULL. > > Yep. I wonder if this is some symbol naming issue or such, because > emitting and relocating the object worked without an error. Thanks to Thomas helping get access to an OSX machine I was able to discover what the issue is. OSX prepends, for reason unbeknownst to me, a leading underscore to all function names. That lead to two issues: First JITed functions do not have that underscore (making us look up a non-existing symbol, because llvm_get_function applied mangling). Secondly, llvm_resolve_symbol failed looking up symbol names, because for $reason dlsym() etc do *not* have the names prefixed by the underscore. Easily enough fixed. After that I discovered another problem, the bitcode files for core pg / contrib modules weren't installed. That turned out to be a make version issue, I'd used define install_llvm_module = # body but older make only like define install_llvm_module # body Writing up a patch that I can actually push. Thanks both to Thomas and Peter for pointing me towards this issue! Greetings, Andres Freund
Re: JIT compiling with LLVM v11
Hi, On 2018-03-05 17:32:09 +1300, Thomas Munro wrote: > I tried out your "jit" branch on my macOS 10.13.3 system. Vendor "cc" > and "c++" are version "Apple LLVM version 9.0.0 (clang-900.0.39.2)". > I used MacPorts (whereas Peter E is using HomeBrew) to install LLVM > with "sudo port install llvm-5.0". Thanks for checking! > warning: ignoring debug info with an invalid version (70003) in > /Users/munro/install/postgres/lib/llvmjit_types.bc That's harmless, log output aside. Should strip the debug info there, to remove the potential for that issue. > Looking at llvm_get_function(), the function that raises that error, I > see that there are a few different paths here. I don't have > HAVE_DECL_LLVMORCGETSYMBOLADDRESSIN defined, and I don't have LLVM < > 5, so I should be getting the symbol address with > LLVMOrcGetSymbolAddress(llvm_opt0_orc, &addr, mangled) or > LLVMOrcGetSymbolAddress(llvm_opt3_orc, &addr, mangled), but clearly > those are returning NULL. Yep. I wonder if this is some symbol naming issue or such, because emitting and relocating the object worked without an error. > Not sure what's happening yet... Hm. :/ Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On Sun, Mar 4, 2018 at 8:39 AM, Andres Freund wrote: > On 2018-03-03 09:37:35 -0500, Peter Eisentraut wrote: >> [discussion of making this work on a Mac] I tried out your "jit" branch on my macOS 10.13.3 system. Vendor "cc" and "c++" are version "Apple LLVM version 9.0.0 (clang-900.0.39.2)". I used MacPorts (whereas Peter E is using HomeBrew) to install LLVM with "sudo port install llvm-5.0". First, I built it like this: ./configure --prefix=$HOME/install/postgres \ --enable-debug --enable-cassert --enable-depend --with-llvm --with-openssl \ --enable-tap-tests \ --with-includes="/opt/local/include" --with-libraries="/opt/local/lib" \ CC="ccache cc" CXX="ccache c++" LLVM_CONFIG=/opt/local/bin/llvm-config-mp-5.0 The build succeeded, initdb ran, the server started up, and then I tried the sequence Andres showed: set jit_above_cost = 0; set client_min_messages=debug2; SELECT pg_jit_available(); On that last command I got: DEBUG: probing availability of llvm for JIT at /Users/munro/install/postgres/lib/llvmjit.so DEBUG: successfully loaded LLVM in current session DEBUG: time to opt: 0.001s DEBUG: time to emit: 0.034s ERROR: failed to JIT: evalexpr_0_0 Looking at the server output I saw: warning: ignoring debug info with an invalid version (70003) in /Users/munro/install/postgres/lib/llvmjit_types.bc 2018-03-05 16:50:05.888 NZDT [14797] ERROR: failed to JIT: evalexpr_0_0 2018-03-05 16:50:05.888 NZDT [14797] STATEMENT: SELECT pg_jit_available(); I could see that llvmjit_types.bc had been produced by this command: /usr/bin/clang -Wno-ignored-attributes -Wno-unknown-warning-option -Wno-ignored-optimization-argument -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -g -O0 -Wall -Werror -O1 -I../../../../src/include -I/opt/local/include -flto=thin -emit-llvm -c -o pseudotypes.bc pseudotypes.c So I tried installing a later clang with "sudo port install clang-5.0" and setting CLANG=/pt/local/bin/clang-mp-5.0. It builds and uses that clang to generate the .bc files, but gives the same error, this time without the "warning" message. Looking at llvm_get_function(), the function that raises that error, I see that there are a few different paths here. I don't have HAVE_DECL_LLVMORCGETSYMBOLADDRESSIN defined, and I don't have LLVM < 5, so I should be getting the symbol address with LLVMOrcGetSymbolAddress(llvm_opt0_orc, &addr, mangled) or LLVMOrcGetSymbolAddress(llvm_opt3_orc, &addr, mangled), but clearly those are returning NULL. Not sure what's happening yet... -- Thomas Munro http://www.enterprisedb.com
Re: JIT compiling with LLVM v11
Hi, On 2018-03-03 09:37:35 -0500, Peter Eisentraut wrote: > On 3/2/18 19:29, Andres Freund wrote: > >> Using my standard set of CC=gcc-7 and CXX=g++-7, the build fails with > >> > >> g++-7: error: unrecognized command line option '-stdlib=libc++' > > > It's actually already filtered, I just added -std*, because of selecting > > the c++ standard, I guess I need to filter more aggressively. This is > > fairly fairly annoying. > > I see you already filter llvm-config --cflags by picking only -I and -D. > Why not do the same for --cxxflags? Any other options that we need > like -f* should be discovered using the normal > does-the-compiler-support-this-option tests. Well, some -f obtions are ABI / behaviour influencing. You can't, to my knowledge, mix/match code build with -fno-rtti with code built with it (influences symbol names). LLVM builds without rtti by default, but a lot of distros enable it... I narrowed the filter to -std= (from -std), which should take care of the -stdlib bit. I also dropped -fno-exceptions being copied since that should not conflict. > >> It seems that it was intended that way anyway, since llvmjit.h contains > >> its own provisions for extern C. > > > > Hrmpf, yea, I broke that the third time now. I'm actually inclined to > > add an appropriate #ifdef ... #error so it's not repeated, what do you > > think? > > Not sure. Why not just move the line and not move it again? ;-) Heh, done ;). Let's see how long it takes... > > Does putting an > > override COMPILER = $(CXX) $(CFLAGS) > > > > into src/backend/jit/llvm/Makefile work? It does force the use of CXX > > for all important platforms if I see it correctly. Verified that it > > works on linux. > > Your latest HEAD builds out of the box for me now using the system compiler. Cool. > >> configure didn't find any of the LLVMOrc* symbols it was looking for. > >> Is that a problem? They seem to be for some debugging support. > > > > That's not a problem, except that the symbols won't be registered with > > the debugger, which is a bit annoying for backtraces. I tried to have > > configure throw errors in cases llvm is too old or such. > > Where does one get those then? I have LLVM 5.0.1. Is there something > even newer? I've submitted them upstream, but they're not yet released. > > Hm, I'll switch them on in the development branch. Independent of the > > final decision that's definitely the right thing for now. The "full > > capability" of the patchset is used if you turn on these three GUCs: > > > > -c jit_expressions=1 > > -c jit_tuple_deforming=1 > > -c jit_perform_inlining=1 > > The last one doesn't seem to exist anymore. Yup, as discussed in the earlier reply to you, I decided it's not particularly useful to have. As also threatened in that reply, I've switched the defaults so you shouldn't have to change them anymore. > If I turn on either of the first two, then make installcheck fails. See > attached diff. Hm, so there's definitely something going on here that I don't yet understand. I've pushed something that I've a slight hunch about (dropping the dots from the symbol names, some tooling doesn't seem to like that). I'd to rebase to fix a few issues, but I've left the changes made since the last push as separate commits. Could you run something like: regression[18425][1]=# set jit_above_cost = 0; SET regression[18425][1]=# set client_min_messages=debug2; SET regression[18425][1]=# SELECT pg_jit_available(); DEBUG: 0: probing availability of llvm for JIT at /home/andres/build/postgres/dev-assert/install/lib/llvmjit.so LOCATION: provider_init, jit.c:83 DEBUG: 0: successfully loaded LLVM in current session LOCATION: provider_init, jit.c:107 DEBUG: 0: time to opt: 0.001s LOCATION: llvm_compile_module, llvmjit.c:435 DEBUG: 0: time to emit: 0.014s LOCATION: llvm_compile_module, llvmjit.c:481 ┌──┐ │ pg_jit_available │ ├──┤ │ t│ └──┘ (1 row) regression[18425][1]=# select now(); DEBUG: 0: time to opt: 0.001s LOCATION: llvm_compile_module, llvmjit.c:435 DEBUG: 0: time to emit: 0.008s LOCATION: llvm_compile_module, llvmjit.c:481 ┌───┐ │ now │ ├───┤ │ 2018-03-03 11:33:13.776947-08 │ └───┘ (1 row) regression[18425][1]=# SET jit_dump_bitcode = 1; SET regression[18425][1]=# select now(); DEBUG: 0: time to opt: 0.002s LOCATION: llvm_compile_module, llvmjit.c:435 DEBUG: 0: time to emit: 0.018s LOCATION: llvm_compile_module, llvmjit.c:481 ┌───┐ │ now │ ├───┤ │ 2018-03-03 11:33:23.508875-08 │ └───┘ (1 row) The last command should have dumped something into your data directory, even if it failed like your regression test output showed. Could attach the two files something li
Re: JIT compiling with LLVM v11
On 2018-03-02 16:29:54 -0800, Andres Freund wrote: > > #include > > #include > > > > It seems that it was intended that way anyway, since llvmjit.h contains > > its own provisions for extern C. > > Hrmpf, yea, I broke that the third time now. I'm actually inclined to > add an appropriate #ifdef ... #error so it's not repeated, what do you > think? Hm, don't think that's easily possible :( - Andres
Re: JIT compiling with LLVM v11
Hi, On 2018-03-02 19:13:01 -0500, Peter Eisentraut wrote: > On 3/1/18 03:02, Andres Freund wrote: > > I've pushed a revised version of my JIT patchset. > > The git tree is at > > https://git.postgresql.org/git/users/andresfreund/postgres.git > > in the jit branch > > > > https://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/jit > > (testing 2e15e8b8100a61ec092a1e5b2db4a93f07a64cbd) > > I'm having an interesting time getting this to build on macOS. Sorry for that... > First, you need to use a CXX that is reasonably similar to the CC. > Otherwise, the CXX will complain about things like attributes not > being supported etc. That's not surprising, but it's a support issue > that we'll have to prepare ourselves for. Right. > Using my standard set of CC=gcc-7 and CXX=g++-7, the build fails with > > g++-7: error: unrecognized command line option '-stdlib=libc++' > > That comes from llvm-config --cxxflags, which was apparently made for > /usr/bin/cc (which is clang). > I see here the same problems as we had in the olden days with Perl, > where it gave us a bunch of compiler flags that applied to the system > compiler but not the compiler currently in use. We should just take the > flags that we really need, like -I and -L. Maybe we don't need it at all. It's actually already filtered, I just added -std*, because of selecting the c++ standard, I guess I need to filter more aggressively. This is fairly fairly annoying. > Using this patch gets it past that: > > diff --git a/src/backend/jit/llvm/llvmjit_inline.cpp > b/src/backend/jit/llvm/llvmjit_inline.cpp > index d4204d2cd2..ad87cfd2d9 100644 > --- a/src/backend/jit/llvm/llvmjit_inline.cpp > +++ b/src/backend/jit/llvm/llvmjit_inline.cpp > @@ -22,7 +22,6 @@ > extern "C" > { > #include "postgres.h" > -#include "jit/llvmjit.h" > > #include > #include > @@ -35,6 +34,8 @@ extern "C" > #include "storage/fd.h" > } > > +#include "jit/llvmjit.h" > + > #include > #include > > It seems that it was intended that way anyway, since llvmjit.h contains > its own provisions for extern C. Hrmpf, yea, I broke that the third time now. I'm actually inclined to add an appropriate #ifdef ... #error so it's not repeated, what do you think? > Then, I'm getting this error: > It seems the problem here is linking C++ with the C compiler. If I > hack in to use c++ in the above command, it continues, and the build > completes. Yea, I was afraid of that, even if I didn't see it locally. Unfortunately Makefile.shlib has a bunch of references both to $(COMPILER) and $(CC). Most of the relevant platforms (using llvmjit on hpux seems like and edge case somebody desiring it can fix) use $(COMPILER). Does putting an override COMPILER = $(CXX) $(CFLAGS) into src/backend/jit/llvm/Makefile work? It does force the use of CXX for all important platforms if I see it correctly. Verified that it works on linux. > configure didn't find any of the LLVMOrc* symbols it was looking for. > Is that a problem? They seem to be for some debugging support. That's not a problem, except that the symbols won't be registered with the debugger, which is a bit annoying for backtraces. I tried to have configure throw errors in cases llvm is too old or such. > So, how do I turn this on then? I see a bunch of new GUC settings > that are all off by default. Which ones turn the feature(s) on? Hm, I'll switch them on in the development branch. Independent of the final decision that's definitely the right thing for now. The "full capability" of the patchset is used if you turn on these three GUCs: -c jit_expressions=1 -c jit_tuple_deforming=1 -c jit_perform_inlining=1 If you set -c log_min_messages=debug one and run a query you'd see something like: 2018-03-02 16:27:19.717 PST [11077][3/8] DEBUG: time to inline: 0.087s 2018-03-02 16:27:19.724 PST [11077][3/8] DEBUG: time to opt: 0.007s 2018-03-02 16:27:19.750 PST [11077][3/8] DEBUG: time to emit: 0.027s I think I should just remove jit_tuple_deforming=1, jit_perform_inlining=1, they're better done via the cost settings (-1 disables). I think having -c jit_expressions is helpful leaving the cost settings aside, because it allows to enable/disable jitting wholesale without changing cost settings, which seems good. Greetings, Andres Freund
Re: JIT compiling with LLVM v11
On 3/1/18 03:02, Andres Freund wrote: > I've pushed a revised version of my JIT patchset. > The git tree is at > https://git.postgresql.org/git/users/andresfreund/postgres.git > in the jit branch > > https://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/jit (testing 2e15e8b8100a61ec092a1e5b2db4a93f07a64cbd) I'm having an interesting time getting this to build on macOS. First, you need to use a CXX that is reasonably similar to the CC. Otherwise, the CXX will complain about things like attributes not being supported etc. That's not surprising, but it's a support issue that we'll have to prepare ourselves for. Using my standard set of CC=gcc-7 and CXX=g++-7, the build fails with g++-7: error: unrecognized command line option '-stdlib=libc++' That comes from llvm-config --cxxflags, which was apparently made for /usr/bin/cc (which is clang). I see here the same problems as we had in the olden days with Perl, where it gave us a bunch of compiler flags that applied to the system compiler but not the compiler currently in use. We should just take the flags that we really need, like -I and -L. Maybe we don't need it at all. Trying it again then with CC=/usr/bin/cc and CXX=/usr/bin/c++, it fails with In file included from llvmjit_inline.cpp:25: In file included from ../../../../src/include/jit/llvmjit.h:16: In file included from /usr/local/Cellar/llvm/5.0.1/include/llvm-c/Types.h:17: In file included from /usr/local/Cellar/llvm/5.0.1/include/llvm/Support/DataTypes.h:33: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/cmath:555:1: error: templates must have C++ linkage Using this patch gets it past that: diff --git a/src/backend/jit/llvm/llvmjit_inline.cpp b/src/backend/jit/llvm/llvmjit_inline.cpp index d4204d2cd2..ad87cfd2d9 100644 --- a/src/backend/jit/llvm/llvmjit_inline.cpp +++ b/src/backend/jit/llvm/llvmjit_inline.cpp @@ -22,7 +22,6 @@ extern "C" { #include "postgres.h" -#include "jit/llvmjit.h" #include #include @@ -35,6 +34,8 @@ extern "C" #include "storage/fd.h" } +#include "jit/llvmjit.h" + #include #include It seems that it was intended that way anyway, since llvmjit.h contains its own provisions for extern C. Then, I'm getting this error: /usr/bin/cc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -Wno-unused-command-line-argument -g -O2 -Wno-deprecated-declarations -Werror -bundle -multiply_defined suppress -o llvmjit.so llvmjit.o llvmjit_error.o llvmjit_inline.o llvmjit_wrap.o llvmjit_expr.o llvmjit_deform.o -L../../../../src/port -L../../../../src/common -L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -L/usr/local/Cellar/libxml2/2.9.7/lib -L/usr/local/Cellar/llvm/5.0.1/lib -L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -Wl,-dead_strip_dylibs -Werror -lLLVMPasses -lLLVMipo -lLLVMInstrumentation -lLLVMVectorize -lLLVMLinker -lLLVMIRReader -lLLVMAsmParser -lLLVMOrcJIT -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMCodeGen -lLLVMScalarOpts -lLLVMInstCombine -lLLVMTransformUtils -lLLVMBitWriter -lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMMCJIT -lLLVMExecutionEngine -lLLVMTarget -lLLVMAnalysis -lLLVMProfileData -lLLVMRuntimeDyld -lLLVMDebugInfoDWARF -lLLVMObject -lLLVMMCParser -lLLVMMC -lLLVMBitReader -lLLVMCore -lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -lcurses -lz -lm -bundle_loader ../../../../src/backend/postgres Undefined symbols for architecture x86_64: "std::__1::error_code::message() const", referenced from: _LLVMPrintModuleToFile in libLLVMCore.a(Core.cpp.o) _LLVMCreateMemoryBufferWithContentsOfFile in libLLVMCore.a(Core.cpp.o) _LLVMCreateMemoryBufferWithSTDIN in libLLVMCore.a(Core.cpp.o) _LLVMTargetMachineEmitToFile in libLLVMTarget.a(TargetMachineC.cpp.o) llvm::errorToErrorCode(llvm::Error) in libLLVMSupport.a(Error.cpp.o) llvm::ECError::log(llvm::raw_ostream&) const in libLLVMSupport.a(Error.cpp.o) llvm::SectionMemoryManager::finalizeMemory(std::__1::basic_string, std::__1::allocator >*) in libLLVMExecutionEngine.a(SectionMemoryManager.cpp.o) [snipped about 900 lines] It seems the problem here is linking C++ with the C compiler. If I hack in to use c++ in the above command, it continues, and the build completes. configure didn't find any of the LLVMOrc* symbols it was looking for. Is that a problem? They seem to be for some debugging support. So, how do I turn this on then? I see a bunch of new GUC settings that are all off by default. Which ones turn the feature(s) on? -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training &
JIT compiling with LLVM v11
Hi, I've pushed a revised version of my JIT patchset. The git tree is at https://git.postgresql.org/git/users/andresfreund/postgres.git in the jit branch https://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/jit Biggest changes: - LLVM 3.9 - master are now supported. This includes a good chunk of work by Pierre Ducroquet. Doing so I found that the patches Pierre provided didn't work when a query was expensive enough to warrant inlining. Turns out LLVM < 5 can't combine the summaries of multiple thin module summaries. But that actually turned out to be a good thing, because it made me think about symbol resolution preferences. Previously it was basically arbitrary whether a function with conflicting names would be choosen from core postgres or one of the extension libs providing it. This is now rewritten so we don't build a combined module summary for core postgres and extensions at backend start. Instead summaries for core pg and extensions are loaded separately, and the correct one for a symbol is used. - Functions in extension libraries are now not referred to with their C symbol in LLVM IR, instead we generate a fictious symbol that includes the library path. E.g. hstore's hstore_from_record is now referenced as @"pgextern.$libdir/hstore.hstore_from_record". Both symbol resolution and inlining knows how to properly resolve those. - As hinted to above, the inlining support has evolved considerably. Instead of a combined index built at backend start we now have individual indexes for each extension / shlib. Symbols are searched with a search path (internal functions just in the 'postgres' index, for extensions it's main 'postgres', 'extension'), and symbols that explicitly reference a function are just looked up within that extension. This has the nice advantage that we don't have to process indexes for extensions that aren't used, which in turn means that extensions can be installed on the system level while a backend is running, and JITing will work even for old backends once the extension is created (or rather functions in it). Additionally the inline costing logic has improved, the super verbose logging is #ifdef'ed out ('ilog' wrapper that's just (void) 0). - The installation of bitcode is now a nice separate make function. pgxs (including contrib's kinda use of pgxs) now automatically generate & install bitcode when the server was compiled --with-llvm. I learned some about make I didn't know ;). - bunch of compilation issues (older clang, -D_NDEBUG from llvm-config being used for all of postgres, ...) have been fixed. - Two bigger prerequisite patches have been merged. - lotsa smaller stuff. Regards, Andres