Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-15 19:14:09 +0100, Catalin Iacob wrote:
> For RHEL proper, I would guess that PG11 is too late for RHEL8 which,
> according to history, should be coming soon.

Yea.


> For RHEL9 I would really expect RedHat to add llvm and clang to proper
> RHEL and build/run against those, even if they add it only for
> Postgres (like they did for mesa).

By the looks of what's going to come for RHEL8 I think it already
contains a suitable LLVM and clang (i.e. >= 3.9)?


> As far as I know clang does promise gcc compatibility in
> the sense that one can link together .o files compiled with both so I
> expect the combination not to cause issues (assuming the other
> compiler flags affecting binary compatibility are aligned).

Right. But that's not even needed, as we just use plain old C ABI via
dlsym(). Nothing needs to be linked together outside of dlsym(), so I'm
not too concerned about that aspect.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Catalin Iacob]

On Thu, Mar 15, 2018 at 6:19 PM, Andres Freund  wrote:
> What we were talking about in this subthread was about a depency on
> clang, not LLVM. And that's just needed at buildtime, to generate the
> bitcode files (including synchronizing types / function signatures).

I was actually thinking of both the buildtime and runtime dependency
because I did not realize the PGDG packages already depend on EPEL.

> For the yum.pg.o, which already depends on EPEL, there's a new enough
> LLVM version.  There's a new enough version in RHEL proper, but it
> appears to only be there for mesa (llvm-private).

Indeed RHEL 7 comes with llvm-private for mesa but that doesn't seem
kosher to use for other things.

When I said packagers I was only thinking of PGDG. I was thinking the
software collections would be the likely solution for the PGDG
packages for both buildtime and runtime. But it seems using clang from
software collections and LLVM from EPEL is also a possibility,
assuming that the newer clang generates IR that the older libraries
are guaranteed to be able to load.

For RHEL proper, I would guess that PG11 is too late for RHEL8 which,
according to history, should be coming soon.

For RHEL9 I would really expect RedHat to add llvm and clang to proper
RHEL and build/run against those, even if they add it only for
Postgres (like they did for mesa). I really don't see them shipping
without a major speedup for a major DB, also because in the meantime
the JIT in PG will have matured. That's also why I find it important
to support gcc and not restrict JIT to clang builds as I expect that
RedHat and all other Linux distros want to build everything with gcc
and asking them to switch to clang or give up JIT will put them in a
hard spot. As far as I know clang does promise gcc compatibility in
the sense that one can link together .o files compiled with both so I
expect the combination not to cause issues (assuming the other
compiler flags affecting binary compatibility are aligned).

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-15 12:42:54 -0400, Tom Lane wrote:
> Andres Freund  writes:
> > It'd be a build not runtime dependency, doesn't that change things?
> 
> How could it not be a runtime dependency?

What we were talking about in this subthread was about a depency on
clang, not LLVM. And that's just needed at buildtime, to generate the
bitcode files (including synchronizing types / function signatures).

For the yum.pg.o, which already depends on EPEL, there's a new enough
LLVM version.  There's a new enough version in RHEL proper, but it
appears to only be there for mesa (llvm-private).


> You're not proposing that we'd embed all of LLVM into a Postgres
> package are you?

No.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Tom Lane]

Andres Freund  writes:
> On 2018-03-15 12:33:08 -0400, Tom Lane wrote:
>> The "software collections" stuff was still in its infancy when I left
>> Red Hat, so things might've changed, but I'm pretty sure at the time
>> it was verboten for any mainstream package to depend on an SCL one.

> But we won't get PG 11 into RHEL7.x either way, no?

Well, they've been known to back-port newer releases of PG into older
RHEL; I wouldn't necessarily assume it'd happen for 11, but maybe 12
or beyond could be made available for RHEL7 at some point.

>> But they very probably wouldn't want postgresql depending on a
>> compiler package even if the dependency was mainstream, so I rather
>> doubt that you'll ever see an --enable-jit PG build out of there,
>> making this most likely moot as far as the official RH package goes.
>> I don't know what Devrim's opinion might be about PGDG.

> It'd be a build not runtime dependency, doesn't that change things?

How could it not be a runtime dependency?  You're not proposing that
we'd embed all of LLVM into a Postgres package are you?  If you are, be
assured that Red Hat will *never* ship that.  Static linking/embedding of
one package in another is forbidden for obvious maintainability reasons.
I would think that other distros have similar policies.

regards, tom lane

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-15 12:33:08 -0400, Tom Lane wrote:
> Andres Freund  writes:
> > On 2018-03-15 17:19:23 +0100, Catalin Iacob wrote:
> >> Indeed. It might be a bit awkward for packagers to depend on something
> >> from Software Collections, for example because they come as separate
> >> trees in /opt that are by default not in your path or dynamic loader
> >> path - one needs to run everything via a scl wrapper or source the
> >> /opt/rh/llvm-toolset-7/enable file to get the appropriate PATH and
> >> LD_LIBRARY_PATH settings, But it seems doable.
> 
> > It'd be just for clang, and they're not *forced* to do it, it's an
> > optional dependency. So I think I'm ok with that.
> 
> The "software collections" stuff was still in its infancy when I left
> Red Hat, so things might've changed, but I'm pretty sure at the time
> it was verboten for any mainstream package to depend on an SCL one.

But we won't get PG 11 into RHEL7.x either way, no?


> But they very probably wouldn't want postgresql depending on a
> compiler package even if the dependency was mainstream, so I rather
> doubt that you'll ever see an --enable-jit PG build out of there,
> making this most likely moot as far as the official RH package goes.
> I don't know what Devrim's opinion might be about PGDG.

It'd be a build not runtime dependency, doesn't that change things?

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Tom Lane]

Andres Freund  writes:
> On 2018-03-15 17:19:23 +0100, Catalin Iacob wrote:
>> Indeed. It might be a bit awkward for packagers to depend on something
>> from Software Collections, for example because they come as separate
>> trees in /opt that are by default not in your path or dynamic loader
>> path - one needs to run everything via a scl wrapper or source the
>> /opt/rh/llvm-toolset-7/enable file to get the appropriate PATH and
>> LD_LIBRARY_PATH settings, But it seems doable.

> It'd be just for clang, and they're not *forced* to do it, it's an
> optional dependency. So I think I'm ok with that.

The "software collections" stuff was still in its infancy when I left
Red Hat, so things might've changed, but I'm pretty sure at the time
it was verboten for any mainstream package to depend on an SCL one.

But they very probably wouldn't want postgresql depending on a
compiler package even if the dependency was mainstream, so I rather
doubt that you'll ever see an --enable-jit PG build out of there,
making this most likely moot as far as the official RH package goes.
I don't know what Devrim's opinion might be about PGDG.

regards, tom lane

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-15 17:19:23 +0100, Catalin Iacob wrote:
> On Thu, Mar 15, 2018 at 1:20 AM, Andres Freund  wrote:
> > I don't really live in the RHEL world, but I wonder if
> > https://developers.redhat.com/blog/2017/10/04/red-hat-adds-go-clangllvm-rust-compiler-toolsets-updates-gcc/
> > is relevant?
> 
> Indeed. It might be a bit awkward for packagers to depend on something
> from Software Collections, for example because they come as separate
> trees in /opt that are by default not in your path or dynamic loader
> path - one needs to run everything via a scl wrapper or source the
> /opt/rh/llvm-toolset-7/enable file to get the appropriate PATH and
> LD_LIBRARY_PATH settings, But it seems doable.

It'd be just for clang, and they're not *forced* to do it, it's an
optional dependency. So I think I'm ok with that.


> I just installed llvm-toolset-7 (the LLVM version is 4.0.1) on RHEL
> 7.4 and did a build of your tree at
> 475b4da439ae397345ab3df509e0e8eb26a8ff39. make installcheck passes for
> both the default config and a server forced to jit everything (I
> think) via:
> jit_above_cost = '0'
> jit_inline_above_cost = '0'
> jit_optimize_above_cost = '0'
> 
> As a side note, this increases the runtime from approx 4 min to 18
> min.

Sure, that jits everything, which is obviously pointless to do for
performancereasons. Especially SQL functions play very badly, because
they're replanned every execution.  But it's good for testing ;)


> Disabling jit completely with -1 in all of the above yields 3 min
> 48s, close to the default question raising maybe the question of how
> much coverage does jit get with the default config.

A bit, but not hugely so. I'm not too concerned about that. I plan to
stand up a few buildfarm animals testing JITing with everything on w/
various LLVM versions.


> The build was with the newer gcc 7.2.1 from the aforementioned
> collections, I'll try the system gcc as well. I run a buildfarm animal
> (katydid) on this RHEL. When JIT gets committed I'll make it use
> --with-llvm against this Software Collections LLVM.

Cool! Thanks for testing!

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Catalin Iacob]

On Thu, Mar 15, 2018 at 1:20 AM, Andres Freund  wrote:
> I don't really live in the RHEL world, but I wonder if
> https://developers.redhat.com/blog/2017/10/04/red-hat-adds-go-clangllvm-rust-compiler-toolsets-updates-gcc/
> is relevant?

Indeed. It might be a bit awkward for packagers to depend on something
from Software Collections, for example because they come as separate
trees in /opt that are by default not in your path or dynamic loader
path - one needs to run everything via a scl wrapper or source the
/opt/rh/llvm-toolset-7/enable file to get the appropriate PATH and
LD_LIBRARY_PATH settings, But it seems doable.

I just installed llvm-toolset-7 (the LLVM version is 4.0.1) on RHEL
7.4 and did a build of your tree at
475b4da439ae397345ab3df509e0e8eb26a8ff39. make installcheck passes for
both the default config and a server forced to jit everything (I
think) via:
jit_above_cost = '0'
jit_inline_above_cost = '0'
jit_optimize_above_cost = '0'

As a side note, this increases the runtime from approx 4 min to 18
min. Disabling jit completely with -1 in all of the above yields 3 min
48s, close to the default question raising maybe the question of how
much coverage does jit get with the default config.

The build was with the newer gcc 7.2.1 from the aforementioned
collections, I'll try the system gcc as well. I run a buildfarm animal
(katydid) on this RHEL. When JIT gets committed I'll make it use
--with-llvm against this Software Collections LLVM.

> Appears to be available on centos too
> https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/

Indeed they are available for CentOS as well.

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-14 22:36:52 -0400, Tom Lane wrote:
> Andres Freund  writes:
> > On 2018-03-13 15:29:33 -0700, Andres Freund wrote:
> >> On 2018-03-14 10:32:40 +1300, Thomas Munro wrote:
> >>> It looks
> >>> like -fexcess-precision-standard is coming from a configure test that
> >>> was run against ${CC}, not against ${CLANG}, so I hacked the generated
> >>> src/Makefile.global to remove that too, just to see if I could get
> >>> past that.
> 
> >> Yea, I'd hoped we could avoid duplicating all the configure tests, but
> >> maybe not :(.
> 
> > I've mostly done that now (not pushed).  I've created new
> > PGAC_PROG_VARCC_VARFLAGS_OPT(compiler variable, flag variable, testflag)
> > function, which now is used to implement PGAC_PROG_CC_CFLAGS_OPT and
> > PGAC_PROG_CC_VAR_OPT (similar for CXX).  That makes it reasonable to
> > test the variables clang recognizes separately.
> 
> Meh.

Why?  The necessary configure code isn't that large:

# Test for behaviour changing compiler flags, to keep compatibility
# with compiler used for normal postgres code. XXX expand
if test "$with_llvm" = yes ; then
  PGAC_PROG_VARCC_VARFLAGS_OPT(CLANG, BITCODE_CFLAGS, [-fno-strict-aliasing])
  PGAC_PROG_VARCC_VARFLAGS_OPT(CLANG, BITCODE_CFLAGS, [-fwrapv])
  PGAC_PROG_VARCC_VARFLAGS_OPT(CLANG, BITCODE_CFLAGS, 
[-fexcess-precision=standard])

  AC_SUBST(BITCODE_CFLAGS, $BITCODE_CFLAGS)
fi

If the relevant clang version doesn't understand, say
-fno-strict-aliasing, then we'd in trouble already if it's
required. After all we do support compiling postgres with clang.


> I agree with Thomas' concern that it's not clear we can or should
> just ignore discrepancies between the -f options supported by the C
> and CLANG compilers.

What's the precise concern here? We pass these flags to work around
compiler issues / "defining our standard". As I said above, if we do not
know the right flags to make clang behave sensibly, we're in trouble
already.

For a good part of the code we already want to be compatible with
compiling postgres with one compiler, and linking to libraries compiled
with something else.


> Is it really so necessary to bring a second compiler into the mix for
> this?  Why not just insist that JIT is only supported if the main build
> is done with clang, too?  My experience with mixing results from different
> compilers is, eh, not positive.

I don't like that option. It doesn't really buy us much, a few lines of
config code, and one additional configure option that should normally be
autodected from the environment.  Requiring a specific compiler will be
terrible on windows, seems out of line how we do development, requires
using clang which is still generates a bit slower code, prevent getting
gcc warnings etc.


Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Tom Lane]

Andres Freund  writes:
> On 2018-03-13 15:29:33 -0700, Andres Freund wrote:
>> On 2018-03-14 10:32:40 +1300, Thomas Munro wrote:
>>> It looks
>>> like -fexcess-precision-standard is coming from a configure test that
>>> was run against ${CC}, not against ${CLANG}, so I hacked the generated
>>> src/Makefile.global to remove that too, just to see if I could get
>>> past that.

>> Yea, I'd hoped we could avoid duplicating all the configure tests, but
>> maybe not :(.

> I've mostly done that now (not pushed).  I've created new
> PGAC_PROG_VARCC_VARFLAGS_OPT(compiler variable, flag variable, testflag)
> function, which now is used to implement PGAC_PROG_CC_CFLAGS_OPT and
> PGAC_PROG_CC_VAR_OPT (similar for CXX).  That makes it reasonable to
> test the variables clang recognizes separately.

Meh.  I agree with Thomas' concern that it's not clear we can or should
just ignore discrepancies between the -f options supported by the C
and CLANG compilers.

Is it really so necessary to bring a second compiler into the mix for
this?  Why not just insist that JIT is only supported if the main build
is done with clang, too?  My experience with mixing results from different
compilers is, eh, not positive.

regards, tom lane

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-13 15:29:33 -0700, Andres Freund wrote:
> On 2018-03-14 10:32:40 +1300, Thomas Munro wrote:
> > I decided to try this on a CentOS 7.2 box.  It has LLVM 3.9 in the
> > 'epel' package repo, but unfortunately it only has clang 3.4.
> 
> That's a bit odd, given llvm and clang really live in the same repo...

I don't really live in the RHEL world, but I wonder if
https://developers.redhat.com/blog/2017/10/04/red-hat-adds-go-clangllvm-rust-compiler-toolsets-updates-gcc/
is relevant?
Appears to be available on centos too
https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/


I checked it out and supporting 3.4 would be a bit painful due to not
being able to directly emit module summaries. We could support that by
building the summaries separately using LLVM, but that'd be either
slower for everyone, or we'd need somewhat finnicky conditionals.


> > clang: error: unknown argument: '-fexcess-precision=standard'
> > clang: error: unknown argument: '-flto=thin'
> >
> > Ok, so I hacked src/Makefile.global.in to remove -flto=thin.
> 
> I think I can get actually rid of that entirely.

Err, no, not really. Would increase overhead due to separate module
summary generation, so I'd rather not do it.


> > It looks
> > like -fexcess-precision-standard is coming from a configure test that
> > was run against ${CC}, not against ${CLANG}, so I hacked the generated
> > src/Makefile.global to remove that too, just to see if I could get
> > past that.
> 
> Yea, I'd hoped we could avoid duplicating all the configure tests, but
> maybe not :(.

I've mostly done that now (not pushed).  I've created new
PGAC_PROG_VARCC_VARFLAGS_OPT(compiler variable, flag variable, testflag)
function, which now is used to implement PGAC_PROG_CC_CFLAGS_OPT and
PGAC_PROG_CC_VAR_OPT (similar for CXX).  That makes it reasonable to
test the variables clang recognizes separately.


Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-14 10:32:40 +1300, Thomas Munro wrote:
> I decided to try this on a CentOS 7.2 box.  It has LLVM 3.9 in the
> 'epel' package repo, but unfortunately it only has clang 3.4.

That's a bit odd, given llvm and clang really live in the same repo...


> clang: error: unknown argument: '-fexcess-precision=standard'
> clang: error: unknown argument: '-flto=thin'
>
> Ok, so I hacked src/Makefile.global.in to remove -flto=thin.

I think I can get actually rid of that entirely.


> It looks
> like -fexcess-precision-standard is coming from a configure test that
> was run against ${CC}, not against ${CLANG}, so I hacked the generated
> src/Makefile.global to remove that too, just to see if I could get
> past that.

Yea, I'd hoped we could avoid duplicating all the configure tests, but
maybe not :(.



> Then I could build successfully and make check passed.  I did see one warning:
> 
> In file included from execExpr.c:39:
> ../../../src/include/jit/jit.h:36:3: warning: redefinition of typedef
> 'JitProviderCallbacks' is a C11 feature [-Wtypedef-redefinition]
> } JitProviderCallbacks;
>   ^
> ../../../src/include/jit/jit.h:22:37: note: previous definition is here
> typedef struct JitProviderCallbacks JitProviderCallbacks;
> ^

Yep. Removed the second superflous / redundant typedef.  Will push a
heavily rebased version in a bit, will include fix for this.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Tom Lane]

Andres Freund  writes:
> On 2018-03-13 14:36:44 -0400, Robert Haas wrote:
>> I realize that EXPLAIN (JIT OFF) may sound like it's intended to
>> disable JIT itself

> Yea, that's what I'm concerned about.

>> , but I think it's pretty clear that EXPLAIN (BUFFERS OFF) does not
>> disable the use of actual buffers, only the display of buffer-related
>> information.

> Hm.

FWIW, I agree with Robert's preference for just JIT here.  The "info"
bit isn't conveying anything.  And we've never had any EXPLAIN options
that actually change the behavior of the explained command, only ones
that change the amount of info displayed.  I don't see why we'd
consider JIT an exception to that.

regards, tom lane

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-13 14:36:44 -0400, Robert Haas wrote:
> On Mon, Mar 12, 2018 at 5:04 PM, Andres Freund  wrote:
> > Currently a handful of explain outputs in the regression tests change
> > output when compiled with JITing. Therefore I'm thinking of adding
> > JITINFO or such option, which can be set to false for those tests?
> 
> Can we spell that JIT or at least JIT_INFO?

The latter works, I don't have a strong opinion on that. For now I've
just tied it to COSTS off.


> I realize that EXPLAIN (JIT OFF) may sound like it's intended to
> disable JIT itself

Yea, that's what I'm concerned about.


> , but I think it's pretty clear that EXPLAIN (BUFFERS OFF) does not
> disable the use of actual buffers, only the display of buffer-related
> information.

Hm.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Robert Haas]

On Mon, Mar 12, 2018 at 5:04 PM, Andres Freund  wrote:
> Currently a handful of explain outputs in the regression tests change
> output when compiled with JITing. Therefore I'm thinking of adding
> JITINFO or such option, which can be set to false for those tests?

Can we spell that JIT or at least JIT_INFO?

I realize that EXPLAIN (JIT OFF) may sound like it's intended to
disable JIT itself, but I think it's pretty clear that EXPLAIN
(BUFFERS OFF) does not disable the use of actual buffers, only the
display of buffer-related information.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-13 10:25:49 -0400, Peter Eisentraut wrote:
> On 3/12/18 17:04, Andres Freund wrote:
> > │ JIT:  
> >  │
> > │   Functions: 4
> >  │
> > │   Inlining: false 
> >  │
> > │   Inlining Time: 0.000
> >  │
> > │   Optimization: false 
> >  │
> > │   Optimization Time: 5.023
> >  │
> > │   Emission Time: 34.987   
> >  │
> 
> The time quantities need some units.
> 
> > │ Execution time: 46.277 ms 
> >  │
> 
> like this :)

Yea, I know. I was planning to start a thread about that. explain.c is
littered with code like
if (es->format == EXPLAIN_FORMAT_TEXT)
appendStringInfo(es->str, "Planning time: %.3f ms\n",
 1000.0 * plantime);
else
ExplainPropertyFloat("Planning Time", 1000.0 * 
plantime, 3, es);
which, to me, is bonkers.  I think we should add add 'const char *unit'
parameter to at least ExplainProperty{Float,Integer,Long}? Or a *Unit
version of them doing so, allowing a bit more gradual change?

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

On 3/12/18 17:04, Andres Freund wrote:
> │ JIT:
>│
> │   Functions: 4  
>│
> │   Inlining: false   
>│
> │   Inlining Time: 0.000  
>│
> │   Optimization: false   
>│
> │   Optimization Time: 5.023  
>│
> │   Emission Time: 34.987 
>│

The time quantities need some units.

> │ Execution time: 46.277 ms   
>│

like this :)

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-12 17:14:00 -0400, Tom Lane wrote:
> Andres Freund  writes:
> > Currently a handful of explain outputs in the regression tests change
> > output when compiled with JITing. Therefore I'm thinking of adding
> > JITINFO or such option, which can be set to false for those tests?
> > Maintaining duplicate output for them seems painful. Better ideas?
> 
> Why not just suppress that info when COSTS OFF is specified?

I wondered about that too.  But that'd mean it'd be harder to write a
test that tests the planning bits of JITing (i.e. decision whether to
use optimization & inlining or not) .  Not sure if it's worth adding
complexity to be able to do so.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Tom Lane]

Andres Freund  writes:
> Currently a handful of explain outputs in the regression tests change
> output when compiled with JITing. Therefore I'm thinking of adding
> JITINFO or such option, which can be set to false for those tests?
> Maintaining duplicate output for them seems painful. Better ideas?

Why not just suppress that info when COSTS OFF is specified?

regards, tom lane

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-09 13:08:36 -0800, Andres Freund wrote:
> On 2018-03-09 15:42:24 -0500, Peter Eisentraut wrote:
> > What I'd quite like is if EXPLAIN or EXPLAIN ANALYZE showed something
> > about what kind of JIT processing was done, if any, to help with this
> > kind of testing.
> 
> Yea, I like that. I think we can only show that when timing is on,
> because otherwise the tests will not be stable depending on --with-jit
> being specified or not.
> 
> So I'm thinking of displaying it similar to the "Planning time" piece,
> i.e. depending on es->summary being enabled. It'd be good to display the
> inline/optimize/emit times too. I think we can just store it in the
> JitContext. But the inline/optimize/emission times will only be
> meaningful when the query is actually executed, I don't see a way around
> that...

Not yet really happy with how it exactly looks, but here's my current
state:

tpch_10[20923][1]=# ;explain (format text, analyze, timing off) SELECT relkind, 
relname FROM pg_class pgc WHERE relkind = 'r';
┌┐
│   QUERY PLAN  
 │
├┤
│ Seq Scan on pg_class pgc  (cost=0.00..15.70 rows=77 width=65) (actual rows=77 
loops=1) │
│   Filter: (relkind = 'r'::"char") 
 │
│   Rows Removed by Filter: 299 
 │
│ Planning time: 0.187 ms   
 │
│ JIT:  
 │
│   Functions: 4
 │
│   Inlining: false 
 │
│   Optimization: false 
 │
│ Execution time: 72.229 ms 
 │
└┘
(9 rows)

tpch_10[20923][1]=# ;explain (format text, analyze, timing on) SELECT relkind, 
relname FROM pg_class pgc WHERE relkind = 'r';

┌┐
│ QUERY PLAN
 │
├┤
│ Seq Scan on pg_class pgc  (cost=0.00..15.70 rows=77 width=65) (actual 
time=40.570..40.651 rows=77 loops=1) │
│   Filter: (relkind = 'r'::"char") 
 │
│   Rows Removed by Filter: 299 
 │
│ Planning time: 0.138 ms   
 │
│ JIT:  
 │
│   Functions: 4
 │
│   Inlining: false 
 │
│   Inlining Time: 0.000
 │
│   Optimization: false 
 │
│   Optimization Time: 5.023
 │
│   Emission Time: 34.987   
 │
│ Execution time: 46.277 ms 
 │
└┘
(12 rows)

json (excerpt):
│ "Triggers": [   ↵│
│ ],  ↵│
│ "JIT": {↵│
│   "Functions": 4,   ↵│
│   "Inlining": false,↵│
│   "Inlining Time": 0.000,   ↵│
│   "Optimization": false,↵│
│   "Optimization Time": 9.701,   ↵│
│   "Emission Time": 52.951   ↵│
│ },  ↵│
│ "Execution Time": 70.292↵│


I'm not at all wedded to the current format, but I feel like that's the
basic functionality needed?

Right now the JIT bit will only be displayed if at least one JITed
function has been emitted. Otherwise we'll just create noise for
everyone.

Currently a handful of explain outputs 

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

On 3/12/18 13:05, Andres Freund wrote:
>> will *not* use JIT, becaue jit_expressions applies at execution time.
> Right.  It'd be easy to change that so jit_expressions=off wouldn't have
> an effect there anymore.  But I'm not sure we want that? I don't have a
> strong feeling about this, except that I think jit_above_cost etc should
> apply at plan, not execution time.

I lean toward making everything apply at plan time.  Not only is that
easier in the current code structure, but over time we'll probably want
to add more detailed planner knobs, e.g., perhaps an alternative
cpu_tuple_cost, and all of that would be a planner setting.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-12 11:21:36 -0400, Peter Eisentraut wrote:
> On 3/11/18 14:25, Andres Freund wrote:
> >> It's perhaps a bit confusing that some of the jit_* settings take effect
> >> at plan time and some at execution time.  At the moment, this mainly
> >> affects me reading the code ;-), but it would also have some effect on
> >> prepared statements and such.
> > Not quite sure what you mean?
> 
> I haven't tested this, but what appears to be the case is that
> 
> SET jit_above_cost = 0;
> PREPARE foo AS SELECT ;
> SET jit_above_cost = infinity;
> EXECUTE foo;
> 
> will use JIT, because jit_above_cost applies at plan time, whereas
> 
> SET jit_expressions = on;
> PREPARE foo AS SELECT ;
> SET jit_expressions = off;
> EXECUTE foo;
> 
> will *not* use JIT, becaue jit_expressions applies at execution time.

Right.  It'd be easy to change that so jit_expressions=off wouldn't have
an effect there anymore.  But I'm not sure we want that? I don't have a
strong feeling about this, except that I think jit_above_cost etc should
apply at plan, not execution time.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

On 3/11/18 14:25, Andres Freund wrote:
>> It's perhaps a bit confusing that some of the jit_* settings take effect
>> at plan time and some at execution time.  At the moment, this mainly
>> affects me reading the code ;-), but it would also have some effect on
>> prepared statements and such.
> Not quite sure what you mean?

I haven't tested this, but what appears to be the case is that

SET jit_above_cost = 0;
PREPARE foo AS SELECT ;
SET jit_above_cost = infinity;
EXECUTE foo;

will use JIT, because jit_above_cost applies at plan time, whereas

SET jit_expressions = on;
PREPARE foo AS SELECT ;
SET jit_expressions = off;
EXECUTE foo;

will *not* use JIT, becaue jit_expressions applies at execution time.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-11 13:19:57 -0400, Peter Eisentraut wrote:
> On 3/9/18 15:56, Andres Freund wrote:
> > I think that's largely that unnecessary trivial queries get JITed and
> > optimized, because the stats are entirely completely off.
> 
> Right.  I instrumented this a bit, and there are indeed two handfuls of
> queries that exceed the default JIT thresholds, as well as a few that
> trigger JIT because they disable some enable_* planner setting, as
> previously discussed.
> 
> Should we throw in some ANALYZEs to avoid this?

Hm, I'd actually lean to just leave it as is for now. JITing halfway
random queries isn't actually that bad... If we get fed up with the
additional time after a while, we can do something then?


> It's perhaps a bit confusing that some of the jit_* settings take effect
> at plan time and some at execution time.  At the moment, this mainly
> affects me reading the code ;-), but it would also have some effect on
> prepared statements and such.

Not quite sure what you mean?


> Also, jit_tuple_deforming is apparently used only when jit_expressions
> is on.

Right. I've not found a good place to hook into that has enough context
to do JITed deforming otherwise.  I'm inclined to just relegate
jit_tuple_deforming to debugging status (i.e. exclude from show all,
docs etc) for now.


> So, we should work toward more clarity on all these different settings,
> what they are useful for, when to set them, how they interact.

Yep.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

On 3/9/18 15:42, Peter Eisentraut wrote:
> The default of jit_above_cost = 50 seems pretty good.  I constructed
> a query that cost about 45 where the run time with and without JIT
> were about even.  This is obviously very limited testing, but it's a
> good start.

Actually, the default in your latest code is 10, which per my
analysis would be too low.  Did you arrive at that setting based on testing?

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

On 3/9/18 15:56, Andres Freund wrote:
> On 2018-03-09 15:28:19 -0500, Peter Eisentraut wrote:
>> On 3/6/18 15:16, Andres Freund wrote:
>>> 2) Don't load the JIT provider until fully needed. Right now
>>>jit_compile_expr() will load the jit provider even if not really
>>>needed. We should probably move the first two return blocks in
>>>llvm_compile_expr() into jit_compile_expr(), to avoid that.
>>
>> I see that you have implemented that, but it doesn't seem to have helped
>> with my make installcheck times.
> 
> What's the exact comparison you're looking at?

I'm just running `time make installcheck` with default settings, as
described in my message from March 6.

> I think that's largely that unnecessary trivial queries get JITed and
> optimized, because the stats are entirely completely off.

Right.  I instrumented this a bit, and there are indeed two handfuls of
queries that exceed the default JIT thresholds, as well as a few that
trigger JIT because they disable some enable_* planner setting, as
previously discussed.

Should we throw in some ANALYZEs to avoid this?

If I set jit_expressions = off, then the timings match again.

It's perhaps a bit confusing that some of the jit_* settings take effect
at plan time and some at execution time.  At the moment, this mainly
affects me reading the code ;-), but it would also have some effect on
prepared statements and such.

Also, jit_tuple_deforming is apparently used only when jit_expressions
is on.

So, we should work toward more clarity on all these different settings,
what they are useful for, when to set them, how they interact.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-09 15:42:24 -0500, Peter Eisentraut wrote:
> For jit_optimize_above_cost, in my testing, any query where JIT payed
> off was even faster with optimizing.  So right now I don't see a need to
> make this a separate setting.  Maybe just make it an on/off setting for
> experimenting.

I'd prefer to be more defensive here. The time needed for JITing without
optimization is roughly linear, whereas optimization is definitely not
linear with input size.


> For inlining, I haven't been able to get a clear picture.  It's a bit
> faster perhaps, but the optimizing dominates it.  I don't have a clear
> mental model for what kind of returns to expect from this.

Yea, you need longrunning queries to benefit significantly. There's a
*lot* more potential once some structural issues with the expression
format (both with and without JIT) are fixed.


> What I'd quite like is if EXPLAIN or EXPLAIN ANALYZE showed something
> about what kind of JIT processing was done, if any, to help with this
> kind of testing.

Yea, I like that. I think we can only show that when timing is on,
because otherwise the tests will not be stable depending on --with-jit
being specified or not.

So I'm thinking of displaying it similar to the "Planning time" piece,
i.e. depending on es->summary being enabled. It'd be good to display the
inline/optimize/emit times too. I think we can just store it in the
JitContext. But the inline/optimize/emission times will only be
meaningful when the query is actually executed, I don't see a way around
that...

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-09 15:28:19 -0500, Peter Eisentraut wrote:
> On 3/6/18 15:16, Andres Freund wrote:
> > 2) Don't load the JIT provider until fully needed. Right now
> >jit_compile_expr() will load the jit provider even if not really
> >needed. We should probably move the first two return blocks in
> >llvm_compile_expr() into jit_compile_expr(), to avoid that.
> 
> I see that you have implemented that, but it doesn't seem to have helped
> with my make installcheck times.

What's the exact comparison you're looking at?

I think that's largely that unnecessary trivial queries get JITed and
optimized, because the stats are entirely completely off.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

On 3/6/18 10:29, Peter Eisentraut wrote:
> I think taking the total cost as the triggering threshold is probably
> good enough for a start.  The cost modeling can be refined over time.

I looked into this a bit more.

The default of jit_above_cost = 50 seems pretty good.  I constructed
a query that cost about 45 where the run time with and without JIT
were about even.  This is obviously very limited testing, but it's a
good start.

For jit_optimize_above_cost, in my testing, any query where JIT payed
off was even faster with optimizing.  So right now I don't see a need to
make this a separate setting.  Maybe just make it an on/off setting for
experimenting.

For inlining, I haven't been able to get a clear picture.  It's a bit
faster perhaps, but the optimizing dominates it.  I don't have a clear
mental model for what kind of returns to expect from this.

What I'd quite like is if EXPLAIN or EXPLAIN ANALYZE showed something
about what kind of JIT processing was done, if any, to help with this
kind of testing.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

On 3/6/18 15:16, Andres Freund wrote:
> 2) Don't load the JIT provider until fully needed. Right now
>jit_compile_expr() will load the jit provider even if not really
>needed. We should probably move the first two return blocks in
>llvm_compile_expr() into jit_compile_expr(), to avoid that.

I see that you have implemented that, but it doesn't seem to have helped
with my make installcheck times.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: JIT compiling with LLVM v11

[Thomas Munro]

On Fri, Mar 9, 2018 at 9:12 AM, Andres Freund  wrote:
> Or even in core LLVM, which has this nice comment:
>
>   // If user asked for the 'native' CPU, we need to autodetect features.
>   // This is necessary for x86 where the CPU might not support all the
>   // features the autodetected CPU name lists in the target. For example,
>   // not all Sandybridge processors support AVX.
>   if (MCPU == "native") {
>
> which pretty much describes the issue you're apparently hitting.
>
> I've pushed an attempted fix (needs a comment, but works here).

===
 All 186 tests passed.
===

That did the trick.  Thanks!

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-08 11:58:41 -0800, Andres Freund wrote:
> I think we can easily fix this by behaving like clang, which uses
> llvm::sys::getHostCPUFeatures(HostFeatures) to built the feature list:
> 
>   // If -march=native, autodetect the feature list.
>   if (const Arg *A = Args.getLastArg(clang::driver::options::OPT_march_EQ)) {
> if (StringRef(A->getValue()) == "native") {
>   llvm::StringMap HostFeatures;
>   if (llvm::sys::getHostCPUFeatures(HostFeatures))
> for (auto  : HostFeatures)
>   Features.push_back(
>   Args.MakeArgString((F.second ? "+" : "-") + F.first()));
> }
>   }
> 
> which seems easy enough.

Or even in core LLVM, which has this nice comment:

  // If user asked for the 'native' CPU, we need to autodetect features.
  // This is necessary for x86 where the CPU might not support all the
  // features the autodetected CPU name lists in the target. For example,
  // not all Sandybridge processors support AVX.
  if (MCPU == "native") {

which pretty much describes the issue you're apparently hitting.

I've pushed an attempted fix (needs a comment, but works here).

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-09 00:33:03 +1300, Thomas Munro wrote:
> On Wed, Mar 7, 2018 at 3:49 PM, Thomas Munro
>  wrote:
> > make check at today's HEAD of your jit branch crashes on my FreeBSD
> > box.  The first thing to crash is this query from point.sql:
> >
> > LOG:  server process (PID 87060) was terminated by signal 4: Illegal 
> > instruction
> > DETAIL:  Failed process was running: SELECT '' AS thirtysix, p1.f1 AS
> > point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dist
> >FROM POINT_TBL p1, POINT_TBL p2
> >ORDER BY dist, p1.f1[0], p2.f1[0];
> 
> Hmm.  It's trying to execute an AVX instruction.

Ah, that's interesting.



> I am not sure if that is real though, because the stack is immediately
> corrupted.

I don't think the stack is corrupted at all, it's just that lldb can't
unwind with functions it doesn't know. To add that capability I've a
pending LLVM patch.


> So either func is not really a function, or it is but was
> compiled for the wrong target.  I see that you call
> LLVMCreateTargetMachine() with the result of LLVMGetHostCPUName() as
> cpu.  For me that's "ivybridge", so I tried hard coding "generic"
> instead and it didn't help.

Hm.


> I see that you say "" for features, where
> is where one would normally put "avx" to turn on AVX instructions, so
> I think perhaps that theory is entirely bogus.

Could you try a -avx in features and see whether it fixes things?

This kinda suggests an LLVM bug or at least an oddity, but I'll try to
drill down more into this. Is this a native machine or a VM?

I think we can easily fix this by behaving like clang, which uses
llvm::sys::getHostCPUFeatures(HostFeatures) to built the feature list:

  // If -march=native, autodetect the feature list.
  if (const Arg *A = Args.getLastArg(clang::driver::options::OPT_march_EQ)) {
if (StringRef(A->getValue()) == "native") {
  llvm::StringMap HostFeatures;
  if (llvm::sys::getHostCPUFeatures(HostFeatures))
for (auto  : HostFeatures)
  Features.push_back(
  Args.MakeArgString((F.second ? "+" : "-") + F.first()));
}
  }

which seems easy enough.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Thomas Munro]

On Tue, Mar 6, 2018 at 10:39 PM, Andres Freund  wrote:
> [more commits]

+* OSX prefixes all object level symbols with an underscore. But neither

"macOS" (see commit da6c4f6c and all mentions since).

make check at today's HEAD of your jit branch crashes on my FreeBSD
box.  The first thing to crash is this query from point.sql:

LOG:  server process (PID 87060) was terminated by signal 4: Illegal instruction
DETAIL:  Failed process was running: SELECT '' AS thirtysix, p1.f1 AS
point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dist
   FROM POINT_TBL p1, POINT_TBL p2
   ORDER BY dist, p1.f1[0], p2.f1[0];

Unfortunately when I tried to load the core file into lldb, the stack
is like this:

* thread #1, name = 'postgres', stop reason = signal SIGILL
  * frame #0: 0x000800e7c1ea

Apparently the generated code is nuking the stack and executing
garbage?  I don't have time to investigate right now, and this may
indicate something busted in my environment, but I thought this might
tell you something.

These variants of that query don't crash (even though I set
jit_above_cost = 0 and checked that it's actually JIT-ing), which
might be clues:

  -- no p1.f1 <-> p2.f1
  SELECT p1.f1 AS point1, p2.f1 AS point2
   FROM POINT_TBL p1, POINT_TBL p2
   ORDER BY p1.f1[0], p2.f1[0];

  -- no join
  SELECT p1.f1 <-> p1.f1 AS dist
   FROM POINT_TBL p1
   ORDER BY 1;

These variants do crash:

  -- p1.f1 <-> p2.f1 in order by, but not select list
  SELECT p1.f1 AS point1, p2.f1 AS point2
   FROM POINT_TBL p1, POINT_TBL p2
   ORDER BY p1.f1 <-> p2.f1, p1.f1[0], p2.f1[0];

  -- p1.f1 <-> p2.f1 in select list, but not in order by
  SELECT p1.f1 AS point1, p2.f1 AS point2, p1.f1 <-> p2.f1 AS dist
   FROM POINT_TBL p1, POINT_TBL p2
   ORDER BY p1.f1[0], p2.f1[0];

  -- simple, with a join
  SELECT p1.f1 <-> p1.f1 AS dist
   FROM POINT_TBL p1, POINT_TBL p2
   ORDER BY 1;

I build it like this:

./configure \
  --prefix=$HOME/install/ \
  --enable-tap-tests \
  --enable-cassert \
  --enable-debug \
  --enable-depend \
  --with-llvm \
  CC="ccache cc" CFLAGS="-O0" CXX="ccache c++" CXXFLAGS="-std=c++11" \
  CLANG=/usr/local/llvm50/bin/clang \
  LLVM_CONFIG=/usr/local/llvm50/bin/llvm-config \
  --with-libraries="/usr/local/lib" \
  --with-includes="/usr/local/include"

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v11

[Tom Lane]

Andres Freund  writes:
> I'm not too worried about that scenario. If, for a cheap plan, the
> planner ends up with a seqscan despite it being disabled, you're pretty
> close to randomly choosing plans already, as the pruning doesn't work
> well anymore (as the %1 percent fuzz factor in
> compare_path_costs_fuzzily() swamps the actual plan costs).

Something I've wanted to do for awhile is to get rid of disable_cost
in favor of pruning disabled plans through logic rather than costing.
I've looked at this once or twice, and it seems doable but not entirely
trivial --- the sticky bits are places where you do need to allow a
disabled plan type because there's no other alternative.  But if we
could get that done, it'd help with this sort of problem.

regards, tom lane

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-06 12:16:01 -0800, Andres Freund wrote:
> > I ran some performance assessments:
> >
> > merge base (0b1d1a038babff4aadf0862c28e7b667f1b12a30)
> >
> > make installcheck  3.14s user 3.34s system 17% cpu 37.954 total
> >
> > jit branch default settings
> >
> > make installcheck  3.17s user 3.30s system 13% cpu 46.596 total
> >
> > jit_above_cost=0
> >
> > make installcheck  3.30s user 3.53s system 5% cpu 1:59.89 total
> >
> > jit_optimize_above_cost=0 (and jit_above_cost=0)
> >
> > make installcheck  3.44s user 3.76s system 1% cpu 8:12.42 total
> >
> > jit_inline_above_cost=0 (and jit_above_cost=0)
> >
> > make installcheck  3.32s user 3.62s system 2% cpu 5:35.58 total
> >
> > One can see the CPU savings quite nicely.
> 
> I'm not quite sure what you mean by that.
> 
> 
> > One obvious problem is that with the default settings, the test suite
> > run gets about 15% slower.  (These figures are reproducible over several
> > runs.)  Is there some debugging stuff turned on that would explain this?
> >  Or would just loading the jit module in each session cause this?
> 
> I suspect it's loading the module.

There's also another issue: For a lot queries in the tests the stats are
way way way off because the relevant tables have never been analyzed.
There's a few cases where costs are off by like 5-7 orders of
magnitude...

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-06 10:29:47 -0500, Peter Eisentraut wrote:
> I think taking the total cost as the triggering threshold is probably
> good enough for a start.  The cost modeling can be refined over time.

Cool.


> We should document that both jit_optimize_above_cost and
> jit_inline_above_cost require jit_above_cost to be set, or otherwise
> nothing happens.

Yea, that's a good plan. We could also change it so it would, but I
don't think there's much point?


> One problem I see is that if someone sets things like
> enable_seqscan=off, the artificial cost increase created by those
> settings would quite likely bump the query over the jit threshold, which
> would alter the query performance characteristics in a way that the user
> would not have intended.  I don't have an idea how to address this right
> now.

I'm not too worried about that scenario. If, for a cheap plan, the
planner ends up with a seqscan despite it being disabled, you're pretty
close to randomly choosing plans already, as the pruning doesn't work
well anymore (as the %1 percent fuzz factor in
compare_path_costs_fuzzily() swamps the actual plan costs).


> I ran some performance assessments:
>
> merge base (0b1d1a038babff4aadf0862c28e7b667f1b12a30)
>
> make installcheck  3.14s user 3.34s system 17% cpu 37.954 total
>
> jit branch default settings
>
> make installcheck  3.17s user 3.30s system 13% cpu 46.596 total
>
> jit_above_cost=0
>
> make installcheck  3.30s user 3.53s system 5% cpu 1:59.89 total
>
> jit_optimize_above_cost=0 (and jit_above_cost=0)
>
> make installcheck  3.44s user 3.76s system 1% cpu 8:12.42 total
>
> jit_inline_above_cost=0 (and jit_above_cost=0)
>
> make installcheck  3.32s user 3.62s system 2% cpu 5:35.58 total
>
> One can see the CPU savings quite nicely.

I'm not quite sure what you mean by that.


> One obvious problem is that with the default settings, the test suite
> run gets about 15% slower.  (These figures are reproducible over several
> runs.)  Is there some debugging stuff turned on that would explain this?
>  Or would just loading the jit module in each session cause this?

I suspect it's loading the module.  There's two pretty easy avenues to
improve this:

1) Attempt to load the JIT provider in postmaster, thereby avoiding a
   lot of redundant dynamic linker work if already installed. That's
   ~5-10 lines or such.  I basically refrained from that because it's
   convenient to not have to restart the server during development (one
   can just reconnect and get a newer jit plugin).

2) Don't load the JIT provider until fully needed. Right now
   jit_compile_expr() will load the jit provider even if not really
   needed. We should probably move the first two return blocks in
   llvm_compile_expr() into jit_compile_expr(), to avoid that.


> From the other results, we can see that one clearly needs quite a big
> database to see a solid benefit from this.

Right, until we've got caching this'll only be beneficial for ~1s+
analytics queries. Unfortunately caching requires some larger planner &
executor surgery, so I don't want to go there at the same time (I'm
already insane enough).


> Do you have any information gathered about this so far?  Any scripts
> to create test databases and test queries?

Yes. I've used tpc-h. Not because it's the greatest, but because it's
semi conveniently available and a lot of others have experience with it
already.  Do you mean whether I've run a couple benchmarks? If so, yes.
I'll schedule some more later - am on battery power rn.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

With the build issues in check, I'm looking at the configuration settings.

I think taking the total cost as the triggering threshold is probably
good enough for a start.  The cost modeling can be refined over time.

We should document that both jit_optimize_above_cost and
jit_inline_above_cost require jit_above_cost to be set, or otherwise
nothing happens.

One problem I see is that if someone sets things like
enable_seqscan=off, the artificial cost increase created by those
settings would quite likely bump the query over the jit threshold, which
would alter the query performance characteristics in a way that the user
would not have intended.  I don't have an idea how to address this right
now.

I ran some performance assessments:

merge base (0b1d1a038babff4aadf0862c28e7b667f1b12a30)

make installcheck  3.14s user 3.34s system 17% cpu 37.954 total

jit branch default settings

make installcheck  3.17s user 3.30s system 13% cpu 46.596 total

jit_above_cost=0

make installcheck  3.30s user 3.53s system 5% cpu 1:59.89 total

jit_optimize_above_cost=0 (and jit_above_cost=0)

make installcheck  3.44s user 3.76s system 1% cpu 8:12.42 total

jit_inline_above_cost=0 (and jit_above_cost=0)

make installcheck  3.32s user 3.62s system 2% cpu 5:35.58 total

One can see the CPU savings quite nicely.

One obvious problem is that with the default settings, the test suite
run gets about 15% slower.  (These figures are reproducible over several
runs.)  Is there some debugging stuff turned on that would explain this?
 Or would just loading the jit module in each session cause this?

>From the other results, we can see that one clearly needs quite a big
database to see a solid benefit from this.  Do you have any information
gathered about this so far?  Any scripts to create test databases and
test queries?

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

On 3/6/18 04:39, Andres Freund wrote:
> I did, and reproduced. Turned out I just missed the error in the above
> test.
> 
> The bug was caused by one ifdef in get_LifetimeEnd() being wrong
> (function is is overloaded starting in 5 rather than 4). The comment
> above it even had it right...

OK, it's fixed for me now.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-05 14:01:05 -0800, Andres Freund wrote:
> On 2018-03-05 13:36:04 -0800, Andres Freund wrote:
> > On 2018-03-05 16:19:52 -0500, Peter Eisentraut wrote:
> > > Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ...
> > > 
> > > The build works out of the box with whatever the default system packages
> > > are.
> > > 
> > > Regression tests crash many times.  One backtrace looks like this:
> > > 
> > > #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> > > #1  0x7fd5b1730231 in __GI_abort () at abort.c:79
> > > #2  0x55c10a1555e3 in ExceptionalCondition
> > > (conditionName=conditionName@entry=0x7fd5a245c2d8
> > > "!(LLVMGetIntrinsicID(fn))",
> > > errorType=errorType@entry=0x7fd5a245bb1d "FailedAssertion",
> > > fileName=fileName@entry=0x7fd5a245c294 "llvmjit_expr.c",
> > > lineNumber=lineNumber@entry=193) at assert.c:54
> > > #3  0x7fd5a245510f in get_LifetimeEnd (mod=mod@entry=0x55c10b1db670)
> > > at llvmjit_expr.c:193
> > > #4  0x7fd5a24553c8 in get_LifetimeEnd (mod=0x55c10b1db670) at
> > > llvmjit_expr.c:233
> > > #5  BuildFunctionCall (context=context@entry=0x55c10b0ca340,
> > > builder=builder@entry=0x55c10b225160,
> > > mod=mod@entry=0x55c10b1db670, fcinfo=0x55c10b1a08b0,
> > > v_fcinfo_isnull=v_fcinfo_isnull@entry=0x7ffc701f5c60)
> > > at llvmjit_expr.c:244
> > 
> > Hm, that should be trivial to fix.  Which version of llvm are you
> > building against? There appear to be a lot of them in testing:
> > https://packages.debian.org/search?keywords=llvm+dev=names=testing=all
> 
> On Debian unstable, I built against a wide variety of branches:
> 
> for v in 3.9 4.0 5.0 6.0;do rm -f ../config.cache;CLANG="ccache clang-$v" 
> LLVM_CONFIG=/usr/lib/llvm-$v/bin/llvm-config ../config.sh --with-llvm && make 
> -j16 -s install && make -s check;done
> 
> All of those pass. I'll create a testing chroot.

I did, and reproduced. Turned out I just missed the error in the above
test.

The bug was caused by one ifdef in get_LifetimeEnd() being wrong
(function is is overloaded starting in 5 rather than 4). The comment
above it even had it right...

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-05 13:36:04 -0800, Andres Freund wrote:
> On 2018-03-05 16:19:52 -0500, Peter Eisentraut wrote:
> > Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ...
> > 
> > The build works out of the box with whatever the default system packages
> > are.
> > 
> > Regression tests crash many times.  One backtrace looks like this:
> > 
> > #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> > #1  0x7fd5b1730231 in __GI_abort () at abort.c:79
> > #2  0x55c10a1555e3 in ExceptionalCondition
> > (conditionName=conditionName@entry=0x7fd5a245c2d8
> > "!(LLVMGetIntrinsicID(fn))",
> > errorType=errorType@entry=0x7fd5a245bb1d "FailedAssertion",
> > fileName=fileName@entry=0x7fd5a245c294 "llvmjit_expr.c",
> > lineNumber=lineNumber@entry=193) at assert.c:54
> > #3  0x7fd5a245510f in get_LifetimeEnd (mod=mod@entry=0x55c10b1db670)
> > at llvmjit_expr.c:193
> > #4  0x7fd5a24553c8 in get_LifetimeEnd (mod=0x55c10b1db670) at
> > llvmjit_expr.c:233
> > #5  BuildFunctionCall (context=context@entry=0x55c10b0ca340,
> > builder=builder@entry=0x55c10b225160,
> > mod=mod@entry=0x55c10b1db670, fcinfo=0x55c10b1a08b0,
> > v_fcinfo_isnull=v_fcinfo_isnull@entry=0x7ffc701f5c60)
> > at llvmjit_expr.c:244
> 
> Hm, that should be trivial to fix.  Which version of llvm are you
> building against? There appear to be a lot of them in testing:
> https://packages.debian.org/search?keywords=llvm+dev=names=testing=all

On Debian unstable, I built against a wide variety of branches:

for v in 3.9 4.0 5.0 6.0;do rm -f ../config.cache;CLANG="ccache clang-$v" 
LLVM_CONFIG=/usr/lib/llvm-$v/bin/llvm-config ../config.sh --with-llvm && make 
-j16 -s install && make -s check;done

All of those pass. I'll create a testing chroot.

Regards,

Andres

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-05 12:17:30 -0800, Andres Freund wrote:
> Writing up a patch that I can actually push.  Thanks both to Thomas and
> Peter for pointing me towards this issue!

After screwing the first attempt at a fix, the second one seems to work
nicely. With optimizations, inlining, etc all core tests pass on
Thomas' machine.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-05 16:19:52 -0500, Peter Eisentraut wrote:
> Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ...
> 
> The build works out of the box with whatever the default system packages
> are.
> 
> Regression tests crash many times.  One backtrace looks like this:
> 
> #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> #1  0x7fd5b1730231 in __GI_abort () at abort.c:79
> #2  0x55c10a1555e3 in ExceptionalCondition
> (conditionName=conditionName@entry=0x7fd5a245c2d8
> "!(LLVMGetIntrinsicID(fn))",
> errorType=errorType@entry=0x7fd5a245bb1d "FailedAssertion",
> fileName=fileName@entry=0x7fd5a245c294 "llvmjit_expr.c",
> lineNumber=lineNumber@entry=193) at assert.c:54
> #3  0x7fd5a245510f in get_LifetimeEnd (mod=mod@entry=0x55c10b1db670)
> at llvmjit_expr.c:193
> #4  0x7fd5a24553c8 in get_LifetimeEnd (mod=0x55c10b1db670) at
> llvmjit_expr.c:233
> #5  BuildFunctionCall (context=context@entry=0x55c10b0ca340,
> builder=builder@entry=0x55c10b225160,
> mod=mod@entry=0x55c10b1db670, fcinfo=0x55c10b1a08b0,
> v_fcinfo_isnull=v_fcinfo_isnull@entry=0x7ffc701f5c60)
> at llvmjit_expr.c:244

Hm, that should be trivial to fix.  Which version of llvm are you
building against? There appear to be a lot of them in testing:
https://packages.debian.org/search?keywords=llvm+dev=names=testing=all

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ...

The build works out of the box with whatever the default system packages
are.

Regression tests crash many times.  One backtrace looks like this:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x7fd5b1730231 in __GI_abort () at abort.c:79
#2  0x55c10a1555e3 in ExceptionalCondition
(conditionName=conditionName@entry=0x7fd5a245c2d8
"!(LLVMGetIntrinsicID(fn))",
errorType=errorType@entry=0x7fd5a245bb1d "FailedAssertion",
fileName=fileName@entry=0x7fd5a245c294 "llvmjit_expr.c",
lineNumber=lineNumber@entry=193) at assert.c:54
#3  0x7fd5a245510f in get_LifetimeEnd (mod=mod@entry=0x55c10b1db670)
at llvmjit_expr.c:193
#4  0x7fd5a24553c8 in get_LifetimeEnd (mod=0x55c10b1db670) at
llvmjit_expr.c:233
#5  BuildFunctionCall (context=context@entry=0x55c10b0ca340,
builder=builder@entry=0x55c10b225160,
mod=mod@entry=0x55c10b1db670, fcinfo=0x55c10b1a08b0,
v_fcinfo_isnull=v_fcinfo_isnull@entry=0x7ffc701f5c60)
at llvmjit_expr.c:244

...

#16 0x55c10a0433ad in exec_simple_query (
query_string=0x55c10b096358 "SELECT COUNT(*) FROM test_tsquery WHERE
keyword <  'new & york';") at postgres.c:1082

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-04 21:00:06 -0800, Andres Freund wrote:
> > Looking at llvm_get_function(), the function that raises that error, I
> > see that there are a few different paths here.  I don't have
> > HAVE_DECL_LLVMORCGETSYMBOLADDRESSIN defined, and I don't have LLVM <
> > 5, so I should be getting the symbol address with
> > LLVMOrcGetSymbolAddress(llvm_opt0_orc, , mangled) or
> > LLVMOrcGetSymbolAddress(llvm_opt3_orc, , mangled), but clearly
> > those are returning NULL.
> 
> Yep. I wonder if this is some symbol naming issue or such, because
> emitting and relocating the object worked without an error.

Thanks to Thomas helping get access to an OSX machine I was able to
discover what the issue is.  OSX prepends, for reason unbeknownst to me,
a leading underscore to all function names.  That lead to two issues:
First JITed functions do not have that underscore (making us look up a
non-existing symbol, because llvm_get_function applied
mangling). Secondly, llvm_resolve_symbol failed looking up symbol names,
because for $reason dlsym() etc do *not* have the names prefixed by the
underscore.   Easily enough fixed.

After that I discovered another problem, the bitcode files for core pg /
contrib modules weren't installed. That turned out to be a make version
issue, I'd used
define install_llvm_module =
# body
but older make only like
define install_llvm_module
# body

Writing up a patch that I can actually push.  Thanks both to Thomas and
Peter for pointing me towards this issue!

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-05 17:32:09 +1300, Thomas Munro wrote:
> I tried out your "jit" branch on my macOS 10.13.3 system.  Vendor "cc"
> and "c++" are version "Apple LLVM version 9.0.0 (clang-900.0.39.2)".
> I used MacPorts (whereas Peter E is using HomeBrew) to install LLVM
> with "sudo port install llvm-5.0".

Thanks for checking!


> warning: ignoring debug info with an invalid version (70003) in
> /Users/munro/install/postgres/lib/llvmjit_types.bc

That's harmless, log output aside. Should strip the debug info there, to
remove the potential for that issue.


> Looking at llvm_get_function(), the function that raises that error, I
> see that there are a few different paths here.  I don't have
> HAVE_DECL_LLVMORCGETSYMBOLADDRESSIN defined, and I don't have LLVM <
> 5, so I should be getting the symbol address with
> LLVMOrcGetSymbolAddress(llvm_opt0_orc, , mangled) or
> LLVMOrcGetSymbolAddress(llvm_opt3_orc, , mangled), but clearly
> those are returning NULL.

Yep. I wonder if this is some symbol naming issue or such, because
emitting and relocating the object worked without an error.


> Not sure what's happening yet...

Hm. :/

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Thomas Munro]

On Sun, Mar 4, 2018 at 8:39 AM, Andres Freund  wrote:
> On 2018-03-03 09:37:35 -0500, Peter Eisentraut wrote:
>> [discussion of making this work on a Mac]

I tried out your "jit" branch on my macOS 10.13.3 system.  Vendor "cc"
and "c++" are version "Apple LLVM version 9.0.0 (clang-900.0.39.2)".
I used MacPorts (whereas Peter E is using HomeBrew) to install LLVM
with "sudo port install llvm-5.0".

First, I built it like this:

./configure
  --prefix=$HOME/install/postgres \
  --enable-debug --enable-cassert --enable-depend --with-llvm --with-openssl \
  --enable-tap-tests \
  --with-includes="/opt/local/include" --with-libraries="/opt/local/lib" \
  CC="ccache cc" CXX="ccache c++" LLVM_CONFIG=/opt/local/bin/llvm-config-mp-5.0

The build succeeded, initdb ran, the server started up, and then I
tried the sequence Andres showed:

set jit_above_cost = 0;
set client_min_messages=debug2;
SELECT pg_jit_available();

On that last command I got:

DEBUG:  probing availability of llvm for JIT at
/Users/munro/install/postgres/lib/llvmjit.so
DEBUG:  successfully loaded LLVM in current session
DEBUG:  time to opt: 0.001s
DEBUG:  time to emit: 0.034s
ERROR:  failed to JIT: evalexpr_0_0

Looking at the server output I saw:

warning: ignoring debug info with an invalid version (70003) in
/Users/munro/install/postgres/lib/llvmjit_types.bc
2018-03-05 16:50:05.888 NZDT [14797] ERROR:  failed to JIT: evalexpr_0_0
2018-03-05 16:50:05.888 NZDT [14797] STATEMENT:  SELECT pg_jit_available();

I could see that llvmjit_types.bc had been produced by this command:

/usr/bin/clang -Wno-ignored-attributes -Wno-unknown-warning-option
-Wno-ignored-optimization-argument -Wall -Wmissing-prototypes
-Wpointer-arith -Wdeclaration-after-statement -Wendif-labels
-Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
-fwrapv -Wno-unused-command-line-argument -g -O0 -Wall -Werror  -O1
-I../../../../src/include  -I/opt/local/include -flto=thin -emit-llvm
-c -o pseudotypes.bc pseudotypes.c

So I tried installing a later clang with "sudo port install clang-5.0"
and setting CLANG=/pt/local/bin/clang-mp-5.0.  It builds and uses that
clang to generate the .bc files, but gives the same error, this time
without the "warning" message.

Looking at llvm_get_function(), the function that raises that error, I
see that there are a few different paths here.  I don't have
HAVE_DECL_LLVMORCGETSYMBOLADDRESSIN defined, and I don't have LLVM <
5, so I should be getting the symbol address with
LLVMOrcGetSymbolAddress(llvm_opt0_orc, , mangled) or
LLVMOrcGetSymbolAddress(llvm_opt3_orc, , mangled), but clearly
those are returning NULL.

Not sure what's happening yet...

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-03 09:37:35 -0500, Peter Eisentraut wrote:
> On 3/2/18 19:29, Andres Freund wrote:
> >> Using my standard set of CC=gcc-7 and CXX=g++-7, the build fails with
> >>
> >> g++-7: error: unrecognized command line option '-stdlib=libc++'
> 
> > It's actually already filtered, I just added -std*, because of selecting
> > the c++ standard, I guess I need to filter more aggressively.  This is
> > fairly fairly annoying.
> 
> I see you already filter llvm-config --cflags by picking only -I and -D.
>  Why not do the same for --cxxflags?  Any other options that we need
> like -f* should be discovered using the normal
> does-the-compiler-support-this-option tests.

Well, some -f obtions are ABI / behaviour influencing.  You can't, to my
knowledge, mix/match code build with -fno-rtti with code built with it
(influences symbol names).  LLVM builds without rtti by default, but a
lot of distros enable it...

I narrowed the filter to -std= (from -std), which should take care of
the -stdlib bit. I also dropped -fno-exceptions being copied since that
should not conflict.


> >> It seems that it was intended that way anyway, since llvmjit.h contains
> >> its own provisions for extern C.
> > 
> > Hrmpf, yea, I broke that the third time now.  I'm actually inclined to
> > add an appropriate #ifdef ... #error so it's not repeated, what do you
> > think?
> 
> Not sure.  Why not just move the line and not move it again? ;-)

Heh, done ;). Let's see how long it takes...


> > Does putting an
> > override COMPILER = $(CXX) $(CFLAGS)
> > 
> > into src/backend/jit/llvm/Makefile work?  It does force the use of CXX
> > for all important platforms if I see it correctly. Verified that it
> > works on linux.
> 
> Your latest HEAD builds out of the box for me now using the system compiler.

Cool.


> >> configure didn't find any of the LLVMOrc* symbols it was looking for.
> >> Is that a problem?  They seem to be for some debugging support.
> > 
> > That's not a problem, except that the symbols won't be registered with
> > the debugger, which is a bit annoying for backtraces. I tried to have
> > configure throw errors in cases llvm is too old or such.
> 
> Where does one get those then?  I have LLVM 5.0.1.  Is there something
> even newer?

I've submitted them upstream, but they're not yet released.


> > Hm, I'll switch them on in the development branch. Independent of the
> > final decision that's definitely the right thing for now.  The "full
> > capability" of the patchset is used if you turn on these three GUCs:
> > 
> > -c jit_expressions=1
> > -c jit_tuple_deforming=1
> > -c jit_perform_inlining=1
> 
> The last one doesn't seem to exist anymore.

Yup, as discussed in the earlier reply to you, I decided it's not
particularly useful to have.  As also threatened in that reply, I've
switched the defaults so you shouldn't have to change them anymore.


> If I turn on either of the first two, then make installcheck fails.  See
> attached diff.

Hm, so there's definitely something going on here that I don't yet
understand. I've pushed something that I've a slight hunch about
(dropping the dots from the symbol names, some tooling doesn't seem to
like that).

I'd to rebase to fix a few issues, but I've left the changes made since
the last push as separate commits.

Could you run something like:
regression[18425][1]=# set jit_above_cost = 0;
SET
regression[18425][1]=# set client_min_messages=debug2;
SET
regression[18425][1]=# SELECT pg_jit_available();
DEBUG:  0: probing availability of llvm for JIT at 
/home/andres/build/postgres/dev-assert/install/lib/llvmjit.so
LOCATION:  provider_init, jit.c:83
DEBUG:  0: successfully loaded LLVM in current session
LOCATION:  provider_init, jit.c:107
DEBUG:  0: time to opt: 0.001s
LOCATION:  llvm_compile_module, llvmjit.c:435
DEBUG:  0: time to emit: 0.014s
LOCATION:  llvm_compile_module, llvmjit.c:481
┌──┐
│ pg_jit_available │
├──┤
│ t│
└──┘
(1 row)

regression[18425][1]=# select now();
DEBUG:  0: time to opt: 0.001s
LOCATION:  llvm_compile_module, llvmjit.c:435
DEBUG:  0: time to emit: 0.008s
LOCATION:  llvm_compile_module, llvmjit.c:481
┌───┐
│  now  │
├───┤
│ 2018-03-03 11:33:13.776947-08 │
└───┘
(1 row)

regression[18425][1]=# SET jit_dump_bitcode = 1;
SET
regression[18425][1]=# select now();
DEBUG:  0: time to opt: 0.002s
LOCATION:  llvm_compile_module, llvmjit.c:435
DEBUG:  0: time to emit: 0.018s
LOCATION:  llvm_compile_module, llvmjit.c:481
┌───┐
│  now  │
├───┤
│ 2018-03-03 11:33:23.508875-08 │
└───┘
(1 row)


The last command should have dumped something into your data directory,
even if it failed like your regression test output showed. Could attach
the two files something 

Re: JIT compiling with LLVM v11

[Andres Freund]

On 2018-03-02 16:29:54 -0800, Andres Freund wrote:
> >  #include 
> >  #include 
> > 
> > It seems that it was intended that way anyway, since llvmjit.h contains
> > its own provisions for extern C.
> 
> Hrmpf, yea, I broke that the third time now.  I'm actually inclined to
> add an appropriate #ifdef ... #error so it's not repeated, what do you
> think?

Hm, don't think that's easily possible :(

- Andres

Re: JIT compiling with LLVM v11

[Andres Freund]

Hi,

On 2018-03-02 19:13:01 -0500, Peter Eisentraut wrote:
> On 3/1/18 03:02, Andres Freund wrote:
> > I've pushed a revised version of my JIT patchset.
> > The git tree is at
> >   https://git.postgresql.org/git/users/andresfreund/postgres.git
> > in the jit branch
> >   
> > https://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/jit
> 
> (testing 2e15e8b8100a61ec092a1e5b2db4a93f07a64cbd)
> 
> I'm having an interesting time getting this to build on macOS.

Sorry for that...


> First, you need to use a CXX that is reasonably similar to the CC.
> Otherwise, the CXX will complain about things like attributes not
> being supported etc.  That's not surprising, but it's a support issue
> that we'll have to prepare ourselves for.

Right.


> Using my standard set of CC=gcc-7 and CXX=g++-7, the build fails with
> 
> g++-7: error: unrecognized command line option '-stdlib=libc++'
> 
> That comes from llvm-config --cxxflags, which was apparently made for
> /usr/bin/cc (which is clang).

> I see here the same problems as we had in the olden days with Perl,
> where it gave us a bunch of compiler flags that applied to the system
> compiler but not the compiler currently in use.  We should just take the
> flags that we really need, like -I and -L.  Maybe we don't need it at all.

It's actually already filtered, I just added -std*, because of selecting
the c++ standard, I guess I need to filter more aggressively.  This is
fairly fairly annoying.


> Using this patch gets it past that:
> 
> diff --git a/src/backend/jit/llvm/llvmjit_inline.cpp
> b/src/backend/jit/llvm/llvmjit_inline.cpp
> index d4204d2cd2..ad87cfd2d9 100644
> --- a/src/backend/jit/llvm/llvmjit_inline.cpp
> +++ b/src/backend/jit/llvm/llvmjit_inline.cpp
> @@ -22,7 +22,6 @@
>  extern "C"
>  {
>  #include "postgres.h"
> -#include "jit/llvmjit.h"
> 
>  #include 
>  #include 
> @@ -35,6 +34,8 @@ extern "C"
>  #include "storage/fd.h"
>  }
> 
> +#include "jit/llvmjit.h"
> +
>  #include 
>  #include 
> 
> It seems that it was intended that way anyway, since llvmjit.h contains
> its own provisions for extern C.

Hrmpf, yea, I broke that the third time now.  I'm actually inclined to
add an appropriate #ifdef ... #error so it's not repeated, what do you
think?


> Then, I'm getting this error:

> It seems the problem here is linking C++ with the C compiler.  If I
> hack in to use c++ in the above command, it continues, and the build
> completes.

Yea, I was afraid of that, even if I didn't see it locally.
Unfortunately Makefile.shlib has a bunch of references both to
$(COMPILER) and $(CC).  Most of the relevant platforms (using llvmjit on
hpux seems like and edge case somebody desiring it can fix) use
$(COMPILER).

Does putting an
override COMPILER = $(CXX) $(CFLAGS)

into src/backend/jit/llvm/Makefile work?  It does force the use of CXX
for all important platforms if I see it correctly. Verified that it
works on linux.


> configure didn't find any of the LLVMOrc* symbols it was looking for.
> Is that a problem?  They seem to be for some debugging support.

That's not a problem, except that the symbols won't be registered with
the debugger, which is a bit annoying for backtraces. I tried to have
configure throw errors in cases llvm is too old or such.


> So, how do I turn this on then?  I see a bunch of new GUC settings
> that are all off by default.  Which ones turn the feature(s) on?

Hm, I'll switch them on in the development branch. Independent of the
final decision that's definitely the right thing for now.  The "full
capability" of the patchset is used if you turn on these three GUCs:

-c jit_expressions=1
-c jit_tuple_deforming=1
-c jit_perform_inlining=1

If you set -c log_min_messages=debug one and run a query you'd see
something like:
2018-03-02 16:27:19.717 PST [11077][3/8] DEBUG:  time to inline: 0.087s
2018-03-02 16:27:19.724 PST [11077][3/8] DEBUG:  time to opt: 0.007s
2018-03-02 16:27:19.750 PST [11077][3/8] DEBUG:  time to emit: 0.027s


I think I should just remove jit_tuple_deforming=1,
jit_perform_inlining=1, they're better done via the cost settings (-1
disables).  I think having -c jit_expressions is helpful leaving the
cost settings aside, because it allows to enable/disable jitting
wholesale without changing cost settings, which seems good.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v11

[Peter Eisentraut]

On 3/1/18 03:02, Andres Freund wrote:
> I've pushed a revised version of my JIT patchset.
> The git tree is at
>   https://git.postgresql.org/git/users/andresfreund/postgres.git
> in the jit branch
>   
> https://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/jit

(testing 2e15e8b8100a61ec092a1e5b2db4a93f07a64cbd)

I'm having an interesting time getting this to build on macOS.

First, you need to use a CXX that is reasonably similar to the CC.
Otherwise, the CXX will complain about things like attributes not
being supported etc.  That's not surprising, but it's a support issue
that we'll have to prepare ourselves for.

Using my standard set of CC=gcc-7 and CXX=g++-7, the build fails with

g++-7: error: unrecognized command line option '-stdlib=libc++'

That comes from llvm-config --cxxflags, which was apparently made for
/usr/bin/cc (which is clang).

I see here the same problems as we had in the olden days with Perl,
where it gave us a bunch of compiler flags that applied to the system
compiler but not the compiler currently in use.  We should just take the
flags that we really need, like -I and -L.  Maybe we don't need it at all.

Trying it again then with CC=/usr/bin/cc and CXX=/usr/bin/c++, it
fails with

In file included from llvmjit_inline.cpp:25:
In file included from ../../../../src/include/jit/llvmjit.h:16:
In file included from
/usr/local/Cellar/llvm/5.0.1/include/llvm-c/Types.h:17:
In file included from
/usr/local/Cellar/llvm/5.0.1/include/llvm/Support/DataTypes.h:33:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/cmath:555:1:
error: templates must have C++ linkage

Using this patch gets it past that:

diff --git a/src/backend/jit/llvm/llvmjit_inline.cpp
b/src/backend/jit/llvm/llvmjit_inline.cpp
index d4204d2cd2..ad87cfd2d9 100644
--- a/src/backend/jit/llvm/llvmjit_inline.cpp
+++ b/src/backend/jit/llvm/llvmjit_inline.cpp
@@ -22,7 +22,6 @@
 extern "C"
 {
 #include "postgres.h"
-#include "jit/llvmjit.h"

 #include 
 #include 
@@ -35,6 +34,8 @@ extern "C"
 #include "storage/fd.h"
 }

+#include "jit/llvmjit.h"
+
 #include 
 #include 

It seems that it was intended that way anyway, since llvmjit.h contains
its own provisions for extern C.

Then, I'm getting this error:

/usr/bin/cc -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute
-Wformat-security -fno-strict-aliasing -fwrapv
-Wno-unused-command-line-argument -g -O2 -Wno-deprecated-declarations
-Werror   -bundle -multiply_defined suppress -o llvmjit.so  llvmjit.o
llvmjit_error.o llvmjit_inline.o llvmjit_wrap.o llvmjit_expr.o
llvmjit_deform.o -L../../../../src/port -L../../../../src/common
-L/usr/local/lib -L/usr/local/opt/openssl/lib
-L/usr/local/opt/readline/lib -L/usr/local/Cellar/libxml2/2.9.7/lib
-L/usr/local/Cellar/llvm/5.0.1/lib  -L/usr/local/lib
-L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib
-Wl,-dead_strip_dylibs  -Werror  -lLLVMPasses -lLLVMipo
-lLLVMInstrumentation -lLLVMVectorize -lLLVMLinker -lLLVMIRReader
-lLLVMAsmParser -lLLVMOrcJIT -lLLVMX86Disassembler -lLLVMX86AsmParser
-lLLVMX86CodeGen -lLLVMGlobalISel -lLLVMSelectionDAG -lLLVMAsmPrinter
-lLLVMDebugInfoCodeView -lLLVMDebugInfoMSF -lLLVMCodeGen
-lLLVMScalarOpts -lLLVMInstCombine -lLLVMTransformUtils -lLLVMBitWriter
-lLLVMX86Desc -lLLVMMCDisassembler -lLLVMX86Info -lLLVMX86AsmPrinter
-lLLVMX86Utils -lLLVMMCJIT -lLLVMExecutionEngine -lLLVMTarget
-lLLVMAnalysis -lLLVMProfileData -lLLVMRuntimeDyld -lLLVMDebugInfoDWARF
-lLLVMObject -lLLVMMCParser -lLLVMMC -lLLVMBitReader -lLLVMCore
-lLLVMBinaryFormat -lLLVMSupport -lLLVMDemangle -lcurses -lz -lm
-bundle_loader ../../../../src/backend/postgres
Undefined symbols for architecture x86_64:
  "std::__1::error_code::message() const", referenced from:
  _LLVMPrintModuleToFile in libLLVMCore.a(Core.cpp.o)
  _LLVMCreateMemoryBufferWithContentsOfFile in libLLVMCore.a(Core.cpp.o)
  _LLVMCreateMemoryBufferWithSTDIN in libLLVMCore.a(Core.cpp.o)
  _LLVMTargetMachineEmitToFile in libLLVMTarget.a(TargetMachineC.cpp.o)
  llvm::errorToErrorCode(llvm::Error) in libLLVMSupport.a(Error.cpp.o)
  llvm::ECError::log(llvm::raw_ostream&) const in
libLLVMSupport.a(Error.cpp.o)

llvm::SectionMemoryManager::finalizeMemory(std::__1::basic_string*) in
libLLVMExecutionEngine.a(SectionMemoryManager.cpp.o)
[snipped about 900 lines]

It seems the problem here is linking C++ with the C compiler.  If I
hack in to use c++ in the above command, it continues, and the build
completes.

configure didn't find any of the LLVMOrc* symbols it was looking for.
Is that a problem?  They seem to be for some debugging support.


So, how do I turn this on then?  I see a bunch of new GUC settings
that are all off by default.  Which ones turn the feature(s) on?

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7