Re: JIT compiling with LLVM v11

2018-03-15 Thread Andres Freund
Hi, On 2018-03-15 19:14:09 +0100, Catalin Iacob wrote: > For RHEL proper, I would guess that PG11 is too late for RHEL8 which, > according to history, should be coming soon. Yea. > For RHEL9 I would really expect RedHat to add llvm and clang to proper > RHEL and build/run against those, even

Re: JIT compiling with LLVM v11

2018-03-15 Thread Catalin Iacob
On Thu, Mar 15, 2018 at 6:19 PM, Andres Freund wrote: > What we were talking about in this subthread was about a depency on > clang, not LLVM. And that's just needed at buildtime, to generate the > bitcode files (including synchronizing types / function signatures). I was

Re: JIT compiling with LLVM v11

2018-03-15 Thread Andres Freund
On 2018-03-15 12:42:54 -0400, Tom Lane wrote: > Andres Freund writes: > > It'd be a build not runtime dependency, doesn't that change things? > > How could it not be a runtime dependency? What we were talking about in this subthread was about a depency on clang, not LLVM.

Re: JIT compiling with LLVM v11

2018-03-15 Thread Tom Lane
Andres Freund writes: > On 2018-03-15 12:33:08 -0400, Tom Lane wrote: >> The "software collections" stuff was still in its infancy when I left >> Red Hat, so things might've changed, but I'm pretty sure at the time >> it was verboten for any mainstream package to depend on an

Re: JIT compiling with LLVM v11

2018-03-15 Thread Andres Freund
On 2018-03-15 12:33:08 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-03-15 17:19:23 +0100, Catalin Iacob wrote: > >> Indeed. It might be a bit awkward for packagers to depend on something > >> from Software Collections, for example because they come as separate >

Re: JIT compiling with LLVM v11

2018-03-15 Thread Tom Lane
Andres Freund writes: > On 2018-03-15 17:19:23 +0100, Catalin Iacob wrote: >> Indeed. It might be a bit awkward for packagers to depend on something >> from Software Collections, for example because they come as separate >> trees in /opt that are by default not in your path or

Re: JIT compiling with LLVM v11

2018-03-15 Thread Andres Freund
Hi, On 2018-03-15 17:19:23 +0100, Catalin Iacob wrote: > On Thu, Mar 15, 2018 at 1:20 AM, Andres Freund wrote: > > I don't really live in the RHEL world, but I wonder if > >

Re: JIT compiling with LLVM v11

2018-03-15 Thread Catalin Iacob
On Thu, Mar 15, 2018 at 1:20 AM, Andres Freund wrote: > I don't really live in the RHEL world, but I wonder if > https://developers.redhat.com/blog/2017/10/04/red-hat-adds-go-clangllvm-rust-compiler-toolsets-updates-gcc/ > is relevant? Indeed. It might be a bit awkward for

Re: JIT compiling with LLVM v11

2018-03-14 Thread Andres Freund
Hi, On 2018-03-14 22:36:52 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-03-13 15:29:33 -0700, Andres Freund wrote: > >> On 2018-03-14 10:32:40 +1300, Thomas Munro wrote: > >>> It looks > >>> like -fexcess-precision-standard is coming from a configure test that >

Re: JIT compiling with LLVM v11

2018-03-14 Thread Tom Lane
Andres Freund writes: > On 2018-03-13 15:29:33 -0700, Andres Freund wrote: >> On 2018-03-14 10:32:40 +1300, Thomas Munro wrote: >>> It looks >>> like -fexcess-precision-standard is coming from a configure test that >>> was run against ${CC}, not against ${CLANG}, so I hacked

Re: JIT compiling with LLVM v11

2018-03-14 Thread Andres Freund
Hi, On 2018-03-13 15:29:33 -0700, Andres Freund wrote: > On 2018-03-14 10:32:40 +1300, Thomas Munro wrote: > > I decided to try this on a CentOS 7.2 box. It has LLVM 3.9 in the > > 'epel' package repo, but unfortunately it only has clang 3.4. > > That's a bit odd, given llvm and clang really

Re: JIT compiling with LLVM v11

2018-03-13 Thread Andres Freund
Hi, On 2018-03-14 10:32:40 +1300, Thomas Munro wrote: > I decided to try this on a CentOS 7.2 box. It has LLVM 3.9 in the > 'epel' package repo, but unfortunately it only has clang 3.4. That's a bit odd, given llvm and clang really live in the same repo... > clang: error: unknown argument:

Re: JIT compiling with LLVM v11

2018-03-13 Thread Tom Lane
Andres Freund writes: > On 2018-03-13 14:36:44 -0400, Robert Haas wrote: >> I realize that EXPLAIN (JIT OFF) may sound like it's intended to >> disable JIT itself > Yea, that's what I'm concerned about. >> , but I think it's pretty clear that EXPLAIN (BUFFERS OFF) does not

Re: JIT compiling with LLVM v11

2018-03-13 Thread Andres Freund
Hi, On 2018-03-13 14:36:44 -0400, Robert Haas wrote: > On Mon, Mar 12, 2018 at 5:04 PM, Andres Freund wrote: > > Currently a handful of explain outputs in the regression tests change > > output when compiled with JITing. Therefore I'm thinking of adding > > JITINFO or such

Re: JIT compiling with LLVM v11

2018-03-13 Thread Robert Haas
On Mon, Mar 12, 2018 at 5:04 PM, Andres Freund wrote: > Currently a handful of explain outputs in the regression tests change > output when compiled with JITing. Therefore I'm thinking of adding > JITINFO or such option, which can be set to false for those tests? Can we spell

Re: JIT compiling with LLVM v11

2018-03-13 Thread Andres Freund
On 2018-03-13 10:25:49 -0400, Peter Eisentraut wrote: > On 3/12/18 17:04, Andres Freund wrote: > > │ JIT: > > │ > > │ Functions: 4 >

Re: JIT compiling with LLVM v11

2018-03-13 Thread Peter Eisentraut
On 3/12/18 17:04, Andres Freund wrote: > │ JIT: >│ > │ Functions: 4 >│ > │ Inlining: false

Re: JIT compiling with LLVM v11

2018-03-12 Thread Andres Freund
Hi, On 2018-03-12 17:14:00 -0400, Tom Lane wrote: > Andres Freund writes: > > Currently a handful of explain outputs in the regression tests change > > output when compiled with JITing. Therefore I'm thinking of adding > > JITINFO or such option, which can be set to false for

Re: JIT compiling with LLVM v11

2018-03-12 Thread Tom Lane
Andres Freund writes: > Currently a handful of explain outputs in the regression tests change > output when compiled with JITing. Therefore I'm thinking of adding > JITINFO or such option, which can be set to false for those tests? > Maintaining duplicate output for them seems

Re: JIT compiling with LLVM v11

2018-03-12 Thread Andres Freund
On 2018-03-09 13:08:36 -0800, Andres Freund wrote: > On 2018-03-09 15:42:24 -0500, Peter Eisentraut wrote: > > What I'd quite like is if EXPLAIN or EXPLAIN ANALYZE showed something > > about what kind of JIT processing was done, if any, to help with this > > kind of testing. > > Yea, I like that.

Re: JIT compiling with LLVM v11

2018-03-12 Thread Peter Eisentraut
On 3/12/18 13:05, Andres Freund wrote: >> will *not* use JIT, becaue jit_expressions applies at execution time. > Right. It'd be easy to change that so jit_expressions=off wouldn't have > an effect there anymore. But I'm not sure we want that? I don't have a > strong feeling about this, except

Re: JIT compiling with LLVM v11

2018-03-12 Thread Andres Freund
On 2018-03-12 11:21:36 -0400, Peter Eisentraut wrote: > On 3/11/18 14:25, Andres Freund wrote: > >> It's perhaps a bit confusing that some of the jit_* settings take effect > >> at plan time and some at execution time. At the moment, this mainly > >> affects me reading the code ;-), but it would

Re: JIT compiling with LLVM v11

2018-03-12 Thread Peter Eisentraut
On 3/11/18 14:25, Andres Freund wrote: >> It's perhaps a bit confusing that some of the jit_* settings take effect >> at plan time and some at execution time. At the moment, this mainly >> affects me reading the code ;-), but it would also have some effect on >> prepared statements and such. >

Re: JIT compiling with LLVM v11

2018-03-11 Thread Andres Freund
On 2018-03-11 13:19:57 -0400, Peter Eisentraut wrote: > On 3/9/18 15:56, Andres Freund wrote: > > I think that's largely that unnecessary trivial queries get JITed and > > optimized, because the stats are entirely completely off. > > Right. I instrumented this a bit, and there are indeed two

Re: JIT compiling with LLVM v11

2018-03-11 Thread Peter Eisentraut
On 3/9/18 15:42, Peter Eisentraut wrote: > The default of jit_above_cost = 50 seems pretty good. I constructed > a query that cost about 45 where the run time with and without JIT > were about even. This is obviously very limited testing, but it's a > good start. Actually, the default

Re: JIT compiling with LLVM v11

2018-03-11 Thread Peter Eisentraut
On 3/9/18 15:56, Andres Freund wrote: > On 2018-03-09 15:28:19 -0500, Peter Eisentraut wrote: >> On 3/6/18 15:16, Andres Freund wrote: >>> 2) Don't load the JIT provider until fully needed. Right now >>>jit_compile_expr() will load the jit provider even if not really >>>needed. We should

Re: JIT compiling with LLVM v11

2018-03-09 Thread Andres Freund
On 2018-03-09 15:42:24 -0500, Peter Eisentraut wrote: > For jit_optimize_above_cost, in my testing, any query where JIT payed > off was even faster with optimizing. So right now I don't see a need to > make this a separate setting. Maybe just make it an on/off setting for > experimenting. I'd

Re: JIT compiling with LLVM v11

2018-03-09 Thread Andres Freund
On 2018-03-09 15:28:19 -0500, Peter Eisentraut wrote: > On 3/6/18 15:16, Andres Freund wrote: > > 2) Don't load the JIT provider until fully needed. Right now > >jit_compile_expr() will load the jit provider even if not really > >needed. We should probably move the first two return blocks

Re: JIT compiling with LLVM v11

2018-03-09 Thread Peter Eisentraut
On 3/6/18 10:29, Peter Eisentraut wrote: > I think taking the total cost as the triggering threshold is probably > good enough for a start. The cost modeling can be refined over time. I looked into this a bit more. The default of jit_above_cost = 50 seems pretty good. I constructed a query

Re: JIT compiling with LLVM v11

2018-03-09 Thread Peter Eisentraut
On 3/6/18 15:16, Andres Freund wrote: > 2) Don't load the JIT provider until fully needed. Right now >jit_compile_expr() will load the jit provider even if not really >needed. We should probably move the first two return blocks in >llvm_compile_expr() into jit_compile_expr(), to avoid

Re: JIT compiling with LLVM v11

2018-03-08 Thread Thomas Munro
On Fri, Mar 9, 2018 at 9:12 AM, Andres Freund wrote: > Or even in core LLVM, which has this nice comment: > > // If user asked for the 'native' CPU, we need to autodetect features. > // This is necessary for x86 where the CPU might not support all the > // features the

Re: JIT compiling with LLVM v11

2018-03-08 Thread Andres Freund
On 2018-03-08 11:58:41 -0800, Andres Freund wrote: > I think we can easily fix this by behaving like clang, which uses > llvm::sys::getHostCPUFeatures(HostFeatures) to built the feature list: > > // If -march=native, autodetect the feature list. > if (const Arg *A =

Re: JIT compiling with LLVM v11

2018-03-08 Thread Andres Freund
Hi, On 2018-03-09 00:33:03 +1300, Thomas Munro wrote: > On Wed, Mar 7, 2018 at 3:49 PM, Thomas Munro > wrote: > > make check at today's HEAD of your jit branch crashes on my FreeBSD > > box. The first thing to crash is this query from point.sql: > > > > LOG:

Re: JIT compiling with LLVM v11

2018-03-06 Thread Thomas Munro
On Tue, Mar 6, 2018 at 10:39 PM, Andres Freund wrote: > [more commits] +* OSX prefixes all object level symbols with an underscore. But neither "macOS" (see commit da6c4f6c and all mentions since). make check at today's HEAD of your jit branch crashes on my FreeBSD

Re: JIT compiling with LLVM v11

2018-03-06 Thread Tom Lane
Andres Freund writes: > I'm not too worried about that scenario. If, for a cheap plan, the > planner ends up with a seqscan despite it being disabled, you're pretty > close to randomly choosing plans already, as the pruning doesn't work > well anymore (as the %1 percent fuzz

Re: JIT compiling with LLVM v11

2018-03-06 Thread Andres Freund
On 2018-03-06 12:16:01 -0800, Andres Freund wrote: > > I ran some performance assessments: > > > > merge base (0b1d1a038babff4aadf0862c28e7b667f1b12a30) > > > > make installcheck 3.14s user 3.34s system 17% cpu 37.954 total > > > > jit branch default settings > > > > make installcheck 3.17s user

Re: JIT compiling with LLVM v11

2018-03-06 Thread Andres Freund
Hi, On 2018-03-06 10:29:47 -0500, Peter Eisentraut wrote: > I think taking the total cost as the triggering threshold is probably > good enough for a start. The cost modeling can be refined over time. Cool. > We should document that both jit_optimize_above_cost and > jit_inline_above_cost

Re: JIT compiling with LLVM v11

2018-03-06 Thread Peter Eisentraut
With the build issues in check, I'm looking at the configuration settings. I think taking the total cost as the triggering threshold is probably good enough for a start. The cost modeling can be refined over time. We should document that both jit_optimize_above_cost and jit_inline_above_cost

Re: JIT compiling with LLVM v11

2018-03-06 Thread Peter Eisentraut
On 3/6/18 04:39, Andres Freund wrote: > I did, and reproduced. Turned out I just missed the error in the above > test. > > The bug was caused by one ifdef in get_LifetimeEnd() being wrong > (function is is overloaded starting in 5 rather than 4). The comment > above it even had it right... OK,

Re: JIT compiling with LLVM v11

2018-03-06 Thread Andres Freund
On 2018-03-05 14:01:05 -0800, Andres Freund wrote: > On 2018-03-05 13:36:04 -0800, Andres Freund wrote: > > On 2018-03-05 16:19:52 -0500, Peter Eisentraut wrote: > > > Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ... > > > > > > The build works out of the box with whatever

Re: JIT compiling with LLVM v11

2018-03-05 Thread Andres Freund
On 2018-03-05 13:36:04 -0800, Andres Freund wrote: > On 2018-03-05 16:19:52 -0500, Peter Eisentraut wrote: > > Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ... > > > > The build works out of the box with whatever the default system packages > > are. > > > > Regression tests

Re: JIT compiling with LLVM v11

2018-03-05 Thread Andres Freund
Hi, On 2018-03-05 12:17:30 -0800, Andres Freund wrote: > Writing up a patch that I can actually push. Thanks both to Thomas and > Peter for pointing me towards this issue! After screwing the first attempt at a fix, the second one seems to work nicely. With optimizations, inlining, etc all core

Re: JIT compiling with LLVM v11

2018-03-05 Thread Andres Freund
On 2018-03-05 16:19:52 -0500, Peter Eisentraut wrote: > Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ... > > The build works out of the box with whatever the default system packages > are. > > Regression tests crash many times. One backtrace looks like this: > > #0

Re: JIT compiling with LLVM v11

2018-03-05 Thread Peter Eisentraut
Testing 0732ee73cf3ffd18d0f651376d69d4798d351ccc on Debian testing ... The build works out of the box with whatever the default system packages are. Regression tests crash many times. One backtrace looks like this: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51 #1

Re: JIT compiling with LLVM v11

2018-03-05 Thread Andres Freund
Hi, On 2018-03-04 21:00:06 -0800, Andres Freund wrote: > > Looking at llvm_get_function(), the function that raises that error, I > > see that there are a few different paths here. I don't have > > HAVE_DECL_LLVMORCGETSYMBOLADDRESSIN defined, and I don't have LLVM < > > 5, so I should be getting

Re: JIT compiling with LLVM v11

2018-03-04 Thread Andres Freund
Hi, On 2018-03-05 17:32:09 +1300, Thomas Munro wrote: > I tried out your "jit" branch on my macOS 10.13.3 system. Vendor "cc" > and "c++" are version "Apple LLVM version 9.0.0 (clang-900.0.39.2)". > I used MacPorts (whereas Peter E is using HomeBrew) to install LLVM > with "sudo port install

Re: JIT compiling with LLVM v11

2018-03-04 Thread Thomas Munro
On Sun, Mar 4, 2018 at 8:39 AM, Andres Freund wrote: > On 2018-03-03 09:37:35 -0500, Peter Eisentraut wrote: >> [discussion of making this work on a Mac] I tried out your "jit" branch on my macOS 10.13.3 system. Vendor "cc" and "c++" are version "Apple LLVM version 9.0.0

Re: JIT compiling with LLVM v11

2018-03-03 Thread Andres Freund
Hi, On 2018-03-03 09:37:35 -0500, Peter Eisentraut wrote: > On 3/2/18 19:29, Andres Freund wrote: > >> Using my standard set of CC=gcc-7 and CXX=g++-7, the build fails with > >> > >> g++-7: error: unrecognized command line option '-stdlib=libc++' > > > It's actually already filtered, I just

Re: JIT compiling with LLVM v11

2018-03-02 Thread Andres Freund
On 2018-03-02 16:29:54 -0800, Andres Freund wrote: > > #include > > #include > > > > It seems that it was intended that way anyway, since llvmjit.h contains > > its own provisions for extern C. > > Hrmpf, yea, I broke that the third time now. I'm actually inclined to > add an appropriate

Re: JIT compiling with LLVM v11

2018-03-02 Thread Andres Freund
Hi, On 2018-03-02 19:13:01 -0500, Peter Eisentraut wrote: > On 3/1/18 03:02, Andres Freund wrote: > > I've pushed a revised version of my JIT patchset. > > The git tree is at > > https://git.postgresql.org/git/users/andresfreund/postgres.git > > in the jit branch > > > >

Re: JIT compiling with LLVM v11

2018-03-02 Thread Peter Eisentraut
On 3/1/18 03:02, Andres Freund wrote: > I've pushed a revised version of my JIT patchset. > The git tree is at > https://git.postgresql.org/git/users/andresfreund/postgres.git > in the jit branch > >