Re: JIT compiling with LLVM v12.2

[Andres Freund]

On March 30, 2018 10:04:25 AM PDT, Konstantin Knizhnik 
 wrote:
>
>
>On 30.03.2018 18:54, Andres Freund wrote:
>> On 2018-03-30 15:12:05 +0300, Konstantin Knizhnik wrote:
>>> I have repeated performance tests at my computer and find out some
>>> regression comparing with previous JIT version.
>>> Previously JIT provides about 2 times improvement at TPC-H Q1. Now
>the
>>> difference is reduced to 1.4 without parallel execution and 1.3 with
>>> parallel execution:
>> Huh. That's the same computer you did the tests on?
>>
>> There shouldn't have been any, I'll check it out.
>>
>> - Andres
>
>Yes, it is the same computer.
>But sorry, may be it is false alarm.
>I noticed that the time of normal (non-jit) query execution was also 
>faster in the past: for parallel execution 6549 vs. 7550 now, for 
>non-parallel execution 20075 vs. 25100.
>I do not know whether this difference is caused by some changes in 
>Postgres committed since this time (end of January) or just because of 
>different layout of data in memory.

A brief attempt at bisecting would be good. That's quite the regression. 
Possibly it's OS related though. Meltdown / Spectre?

Andres
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: JIT compiling with LLVM v12.2

[Konstantin Knizhnik]

On 30.03.2018 18:54, Andres Freund wrote:

On 2018-03-30 15:12:05 +0300, Konstantin Knizhnik wrote:

I have repeated performance tests at my computer and find out some
regression comparing with previous JIT version.
Previously JIT provides about 2 times improvement at TPC-H Q1. Now the
difference is reduced to 1.4 without parallel execution and 1.3 with
parallel execution:

Huh. That's the same computer you did the tests on?

There shouldn't have been any, I'll check it out.

- Andres


Yes, it is the same computer.
But sorry, may be it is false alarm.
I noticed that the time of normal (non-jit) query execution was also 
faster in the past: for parallel execution 6549 vs. 7550 now, for 
non-parallel execution 20075 vs. 25100.
I do not know whether this difference is caused by some changes in 
Postgres committed since this time (end of January) or just because of 
different layout of data in memory.
But JIT performance improvement is almost the same in both cases: 1.493 
vs 1.434 now.



--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Re: JIT compiling with LLVM v12.2

[Andres Freund]

On 2018-03-30 15:12:05 +0300, Konstantin Knizhnik wrote:
> I have repeated performance tests at my computer and find out some
> regression comparing with previous JIT version.
> Previously JIT provides about 2 times improvement at TPC-H Q1. Now the
> difference is reduced to 1.4 without parallel execution and 1.3 with
> parallel execution:

Huh. That's the same computer you did the tests on?

There shouldn't have been any, I'll check it out.

- Andres

Re: JIT compiling with LLVM v12.2

[Konstantin Knizhnik]

On 30.03.2018 02:14, Andres Freund wrote:

Hi,

On 2018-03-29 19:57:42 +0700, John Naylor wrote:

Hi Andres,
I spent some time over pouring over the JIT README, and I've attached
a patch with some additional corrections as well as some stylistic
suggestions. The latter may be debatable, but I'm sure you can take
and pick as you see fit. If there are cases where I misunderstood your
intent, maybe that's also useful information. :-)

I've picked most of them, and pushed a change including some additional
changes. Thanks!

- Andres

I have repeated performance tests at my computer and find out some 
regression comparing with previous JIT version.
Previously JIT provides about 2 times improvement at TPC-H Q1. Now the 
difference is reduced to 1.4 without parallel execution and 1.3 with 
parallel execution:



max_parallel_workers_per_gather=0
max_parallel_workers_per_gather=4
jit=on
17500
5730
jit=off
25100
7550




Previous my result for JIT was 13440 for sequential execution.
I know that performance is not the high priority now, it is more 
important to commit infrastructure.

Just want to inform that such regression takes place.
It will be nice if you can mark future directions of improving JIT 
performance...

postgres=# explain (analyze,buffers) select
    l_returnflag,
    l_linestatus,
    sum(l_quantity) as sum_qty,
    sum(l_extendedprice) as sum_base_price,
    sum(l_extendedprice*(1-l_discount)) as sum_disc_price,
    sum(l_extendedprice*(1-l_discount)*(1+l_tax)) as sum_charge,
    avg(l_quantity) as avg_qty,
    avg(l_extendedprice) as avg_price,
    avg(l_discount) as avg_disc,
    count(*) as count_order
from
    lineitem
where
    l_shipdate <= '1998-12-01'
group by
    l_returnflag,
    l_linestatus
order by
    l_returnflag,
    l_linestatus;
QUERY PLAN

---
--
 Finalize GroupAggregate  (cost=2064556.89..2064560.47 rows=6 width=60) 
(actual time=6573.905..6573.915 rows=4 loops=1)

   Group Key: l_returnflag, l_linestatus
   Buffers: shared hit=240472
   ->  Gather Merge  (cost=2064556.89..2064559.76 rows=24 width=132) 
(actual time=6573.888..6573.897 rows=20 loops=1)

 Workers Planned: 4
 Workers Launched: 4
 Buffers: shared hit=240472
 ->  Sort  (cost=2063556.83..2063556.85 rows=6 width=132) 
(actual time=6562.256..6562.256 rows=4 loops=5)

   Sort Key: l_returnflag, l_linestatus
   Sort Method: quicksort  Memory: 26kB
   Worker 0:  Sort Method: quicksort  Memory: 26kB
   Worker 1:  Sort Method: quicksort  Memory: 26kB
   Worker 2:  Sort Method: quicksort  Memory: 26kB
   Worker 3:  Sort Method: quicksort  Memory: 26kB
   Buffers: shared hit=1276327
   ->  Partial HashAggregate (cost=2063556.69..2063556.75 
rows=6 width=132) (actual time=6562.222..6562.224 rows=4

 loops=5)
 Group Key: l_returnflag, l_linestatus
 Buffers: shared hit=1276299
 ->  Parallel Seq Scan on lineitem 
(cost=0.00..1463755.41 rows=14995032 width=20) (actual time=312.454..25

20.753 rows=11997210 loops=5)
   Filter: (l_shipdate <= '1998-12-01'::date)
   Buffers: shared hit=1276299
 Planning Time: 0.130 ms
 JIT:
   Functions: 18
   Generation Time: 2.344 ms
   Inlining: true
   Inlining Time: 15.364 ms
   Optimization: true
   Optimization Time: 298.833 ms
   Emission Time: 155.257 ms
 Execution Time: 6807.751 ms
(31 rows)

Time: 6808.216 ms (00:06.808)


--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-29 19:57:42 +0700, John Naylor wrote:
> Hi Andres,
> I spent some time over pouring over the JIT README, and I've attached
> a patch with some additional corrections as well as some stylistic
> suggestions. The latter may be debatable, but I'm sure you can take
> and pick as you see fit. If there are cases where I misunderstood your
> intent, maybe that's also useful information. :-)

I've picked most of them, and pushed a change including some additional
changes. Thanks!

- Andres

Re: JIT compiling with LLVM v12.2

[John Naylor]

Hi Andres,
I spent some time over pouring over the JIT README, and I've attached
a patch with some additional corrections as well as some stylistic
suggestions. The latter may be debatable, but I'm sure you can take
and pick as you see fit. If there are cases where I misunderstood your
intent, maybe that's also useful information. :-)

-John Naylor
diff --git a/src/backend/jit/README b/src/backend/jit/README
index bfed319..7924127 100644
--- a/src/backend/jit/README
+++ b/src/backend/jit/README
@@ -13,12 +13,12 @@ the CPU that just handles that expression, yielding a speedup.
 That this is done at query execution time, possibly even only in cases
 the relevant task is done a number of times, makes it JIT, rather than
 ahead-of-time (AOT). Given the way JIT compilation is used in
-postgres, the lines between interpretation, AOT and JIT are somewhat
+PostgreSQL, the lines between interpretation, AOT and JIT are somewhat
 blurry.
 
 Note that the interpreted program turned into a native program does
 not necessarily have to be a program in the classical sense. E.g. it
-is highly beneficial JIT compile tuple deforming into a native
+is highly beneficial to JIT compile tuple deforming into a native
 function just handling a specific type of table, despite tuple
 deforming not commonly being understood as a "program".
 
@@ -26,7 +26,7 @@ deforming not commonly being understood as a "program".
 Why JIT?
 
 
-Parts of postgres are commonly bottlenecked by comparatively small
+Parts of PostgreSQL are commonly bottlenecked by comparatively small
 pieces of CPU intensive code. In a number of cases that is because the
 relevant code has to be very generic (e.g. handling arbitrary SQL
 level expressions, over arbitrary tables, with arbitrary extensions
@@ -49,11 +49,11 @@ particularly beneficial for removing branches during tuple deforming.
 How to JIT
 ==
 
-Postgres, by default, uses LLVM to perform JIT. LLVM was chosen
+PostgreSQL, by default, uses LLVM to perform JIT. LLVM was chosen
 because it is developed by several large corporations and therefore
 unlikely to be discontinued, because it has a license compatible with
-PostgreSQL, and because its LLVM IR can be generated from C
-using the clang compiler.
+PostgreSQL, and because its IR can be generated from C using the Clang
+compiler.
 
 
 Shared Library Separation
@@ -68,20 +68,20 @@ An additional benefit of doing so is that it is relatively easy to
 evaluate JIT compilation that does not use LLVM, by changing out the
 shared library used to provide JIT compilation.
 
-To achieve this code, e.g. expression evaluation, intending to perform
-JIT, calls a LLVM independent wrapper located in jit.c to do so. If
-the shared library providing JIT support can be loaded (i.e. postgres
-was compiled with LLVM support and the shared library is installed),
-the task of JIT compiling an expression gets handed of to shared
-library. This obviously requires that the function in jit.c is allowed
-to fail in case no JIT provider can be loaded.
+To achieve this, code intending to perform JIT (e.g. expression evaluation)
+calls an LLVM independent wrapper located in jit.c to do so. If the
+shared library providing JIT support can be loaded (i.e. PostgreSQL was
+compiled with LLVM support and the shared library is installed), the task
+of JIT compiling an expression gets handed of to the shared library. This
+obviously requires that the function in jit.c is allowed to fail in case
+no JIT provider can be loaded.
 
 Which shared library is loaded is determined by the jit_provider GUC,
 defaulting to "llvmjit".
 
 Cloistering code performing JIT into a shared library unfortunately
 also means that code doing JIT compilation for various parts of code
-has to be located separately from the code doing so without
+has to be located separately from the code that executes without
 JIT. E.g. the JITed version of execExprInterp.c is located in
 jit/llvm/ rather than executor/.
 
@@ -105,17 +105,21 @@ implementations.
 
 Emitting individual functions separately is more expensive than
 emitting several functions at once, and emitting them together can
-provide additional optimization opportunities. To facilitate that the
-LLVM provider separates function definition from emitting them in an
-executable way.
+provide additional optimization opportunities. To facilitate that, the
+LLVM provider separates function definition (LLVM IR) from function
+emission (executable mmap()ed segments).
 
 Creating functions into the current mutable module (a module
 essentially is LLVM's equivalent of a translation unit in C) is done
 using
+
   extern LLVMModuleRef llvm_mutable_module(LLVMJitContext *context);
+
 in which it then can emit as much code using the LLVM APIs as it
 wants. Whenever a function actually needs to be called
+
   extern void *llvm_get_function(LLVMJitContext *context, const char *funcname);
+
 returns a pointer to it.
 
 E.g. in the expression evaluation case this 

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-25 00:07:11 +1300, Thomas Munro wrote:
> I spotted a couple of typos and some very minor coding details -- see
> please see attached.

Thanks, applying 0001 in a bit.


> From 648e303072c77e781eca2bb06f488f6be9ccac84 Mon Sep 17 00:00:00 2001
> From: Thomas Munro 
> Date: Sat, 24 Mar 2018 23:12:40 +1300
> Subject: [PATCH 2/2] Minor code cleanup for llvmjit_wrap.cpp.
> 
> llvm::sys::getHostCPUName()'s result is a llvm::StringRef.  Its data() member
> function doesn't guarantee a null-terminated result, so we'd better jump
> through an extra hoop to get a C string.

Hm, I checked, and it's fine, I'm not enthusiastic about this...


> It seems better to use LLVMCreateMessage() rather than strdup() to allocate
> the copy returned by LLVMGetHostCPUFeatures() and LLVMGetHostCPUName(),
> since the contract is that the caller should free it with
> LLVMDisposeMessage().  While we can see that LLVMCreateMessage() and
> LLVMDisposeMessage() are currently just wrappers for strdup() and free(),
> using them symmetrically seems like a good idea for future Windows support,
> where DLLs can be using different heap allocators (the same reason we provide
> PQfreemem in libpq).

I just kept it similar to nearby functions in the LLVM code.


> Fix brace style.

I tried to keep this as it's submitted to LLVM, I hope we can get rid of
them for newer version soon...  I think I'll update them to be exactly
the same as soon as the upstream patch is applied.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

Thomas Munro  wrote:
> typos

A dead line.

-- 
Thomas Munro
http://www.enterprisedb.com


0003-Remove-dead-code.patch
Description: Binary data

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

Hi Andres,

I spotted a couple of typos and some very minor coding details -- see
please see attached.

-- 
Thomas Munro
http://www.enterprisedb.com


0001-Correct-some-minor-typos-in-the-new-JIT-code.patch
Description: Binary data


0002-Minor-code-cleanup-for-llvmjit_wrap.cpp.patch
Description: Binary data

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-22 16:09:51 +1300, Thomas Munro wrote:
> On Thu, Mar 22, 2018 at 1:36 PM, Thomas Munro
>  wrote:
> > I've now run out of things to complain about for now.  Nice work!
> 
> I jumped on a POWER8 box.  As expected, the same breakage occurs.  So
> I hacked LLVM 6.0 thusly:
> 
> diff --git a/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
> b/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
> index 68397be..08aa3a8 100644
> --- a/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
> +++ b/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
> @@ -54,7 +54,11 @@ createLocalCompileCallbackManager(const Triple ,
>  std::function
>  createLocalIndirectStubsManagerBuilder(const Triple ) {
>switch (T.getArch()) {
> -default: return nullptr;
> +default:
> +  return [](){
> +return llvm::make_unique<
> +   orc::LocalIndirectStubsManager>();
> +  };
> 
>  case Triple::aarch64:
>return [](){

> I am not qualified to have an opinion on whether this is the correct
> fix for LLVM, but with this change our make check passes, indicating
> that things are otherwise looking good on this architecture.

Yea, that should do the trick, as long as one doesn't rely on indirect
stubs, which we don't.  Kinda wonder if we could hackfix this by putting
your definition of createLocalIndirectStubsManagerBuilder() into
something earlier on the search path...


> So I've now tested your branch on various combinations of:
> FreeBSD/amd64 (including with a weird CPU that lacks AVX),
> Debian/amd64, Debian/i386, Debian/arm64, RHEL/ppc64le, macOS/amd64,
> with LLVM 3.9, 4,0, 5.0, 6.0, with GCC and clang as the main compiler,
> with libstdc++ and libc++ as the C++ standard library.

Many thanks again.


> If I had access to one I'd try it on a big endian machine, but I
> don't.  Anyone?  The elephant in the room is Windows.  I'm not
> personally in the same room as that particular elephant, however.

Hah.  I'm not 100% sure I can MSVC project stuff done for this release,
TBH. Doing it via mingw shouldn't be much trouble. But I'm aiming for
fixing the project generation support too.


> FWIW, your branch doesn't build against LLVM master (future 7.0),
> because the shared module stuff is changing:
> 
> llvmjit.c: In function ‘llvm_compile_module’:
> llvmjit.c:544:4: error: unknown type name ‘LLVMSharedModuleRef’
> LLVMSharedModuleRef smod;
> ^
> llvmjit.c:546:4: warning: implicit declaration of function
> ‘LLVMOrcMakeSharedModule’ [-Wimplicit-function-declaration]
> smod = LLVMOrcMakeSharedModule(context->module);
> ^
> llvmjit.c:548:12: warning: passing argument 3 of
> ‘LLVMOrcAddEagerlyCompiledIR’ makes pointer from integer without a
> cast [enabled by default]
> llvm_resolve_symbol, NULL))
> ^
> In file included from llvmjit.c:31:0:
> /home/thomas.munro/build/llvm/debug/install/include/llvm-c/OrcBindings.h:99:1:
> note: expected ‘LLVMModuleRef’ but argument is of type ‘int’
>  LLVMOrcAddEagerlyCompiledIR(LLVMOrcJITStackRef JITStack,
>  ^
> llvmjit.c:552:4: warning: implicit declaration of function
> ‘LLVMOrcDisposeSharedModuleRef’ [-Wimplicit-function-declaration]
> LLVMOrcDisposeSharedModuleRef(smod);

Yea, I guess we could add branches for that, but 7 just branched and is
a moving target, so I'm inclined to wait a bit.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Thu, Mar 22, 2018 at 1:36 PM, Thomas Munro
 wrote:
> I've now run out of things to complain about for now.  Nice work!

I jumped on a POWER8 box.  As expected, the same breakage occurs.  So
I hacked LLVM 6.0 thusly:

diff --git a/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
b/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
index 68397be..08aa3a8 100644
--- a/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
+++ b/lib/ExecutionEngine/Orc/IndirectionUtils.cpp
@@ -54,7 +54,11 @@ createLocalCompileCallbackManager(const Triple ,
 std::function
 createLocalIndirectStubsManagerBuilder(const Triple ) {
   switch (T.getArch()) {
-default: return nullptr;
+default:
+  return [](){
+return llvm::make_unique<
+   orc::LocalIndirectStubsManager>();
+  };

 case Triple::aarch64:
   return [](){

I am not qualified to have an opinion on whether this is the correct
fix for LLVM, but with this change our make check passes, indicating
that things are otherwise looking good on this architecture.

So I've now tested your branch on various combinations of:
FreeBSD/amd64 (including with a weird CPU that lacks AVX),
Debian/amd64, Debian/i386, Debian/arm64, RHEL/ppc64le, macOS/amd64,
with LLVM 3.9, 4,0, 5.0, 6.0, with GCC and clang as the main compiler,
with libstdc++ and libc++ as the C++ standard library.

If I had access to one I'd try it on a big endian machine, but I
don't.  Anyone?  The elephant in the room is Windows.  I'm not
personally in the same room as that particular elephant, however.

FWIW, your branch doesn't build against LLVM master (future 7.0),
because the shared module stuff is changing:

llvmjit.c: In function ‘llvm_compile_module’:
llvmjit.c:544:4: error: unknown type name ‘LLVMSharedModuleRef’
LLVMSharedModuleRef smod;
^
llvmjit.c:546:4: warning: implicit declaration of function
‘LLVMOrcMakeSharedModule’ [-Wimplicit-function-declaration]
smod = LLVMOrcMakeSharedModule(context->module);
^
llvmjit.c:548:12: warning: passing argument 3 of
‘LLVMOrcAddEagerlyCompiledIR’ makes pointer from integer without a
cast [enabled by default]
llvm_resolve_symbol, NULL))
^
In file included from llvmjit.c:31:0:
/home/thomas.munro/build/llvm/debug/install/include/llvm-c/OrcBindings.h:99:1:
note: expected ‘LLVMModuleRef’ but argument is of type ‘int’
 LLVMOrcAddEagerlyCompiledIR(LLVMOrcJITStackRef JITStack,
 ^
llvmjit.c:552:4: warning: implicit declaration of function
‘LLVMOrcDisposeSharedModuleRef’ [-Wimplicit-function-declaration]
LLVMOrcDisposeSharedModuleRef(smod);
^

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Wed, Mar 21, 2018 at 8:06 PM, Thomas Munro
 wrote:
> "make -C src/interfaces/ecpg/test check" consistently fails on my macOS 
> machine:
>
> test compat_oracle/char_array ... stderr source FAILED

I can't reproduce this anymore on the tip of your jit branch.  I don't
know what caused it or which change fixed it...

I've now run out of things to complain about for now.  Nice work!

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Thu, Mar 22, 2018 at 11:46 AM, Andres Freund  wrote:
> On 2018-03-22 11:36:47 +1300, Thomas Munro wrote:
>> Not sure if we'll want to try to actively identify and avoid known
>> buggy versions or not?
>
> I'm currently not inclined to invest a lot of effort into it, besides
> trying to get the bug fixed.
>
> A possible testcase would be to call
> createLocalIndirectStubsManagerBuilder() and report an error if it
> returns nullptr. But that'd fail once the bug is fixed, because we don't
> actually *need* that functionality, it's just that LLVM instantiates the
> stub manager unconditionally for some reason.

So how about we test createLocalIndirectStubsManagerBuilder(), and if
it's nullptr then we also test the LLVM version number?  For each
major release (3.9, 4.0, 5.0, ... you can see that they did the same
kind of versioning schema change that we did!) there will eventually
be a minor/patch release number where this works even when nullptr is
returned here.

This problem is going to come up on any architecture not covered in
the following code, namely anything but x86, x86_64 and (since 6.0)
aarch64 (aka arm64), so we definitely don't want to leave JIT disabled
once that bug is fixed:

https://github.com/llvm-mirror/llvm/blob/release_39/lib/ExecutionEngine/Orc/IndirectionUtils.cpp#L48
https://github.com/llvm-mirror/llvm/blob/release_60/lib/ExecutionEngine/Orc/IndirectionUtils.cpp#L48

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-22 11:36:47 +1300, Thomas Munro wrote:
> On Thu, Mar 22, 2018 at 10:59 AM, Andres Freund  wrote:
> > On 2018-03-22 10:50:52 +1300, Thomas Munro wrote:
> >> Hmm.  There is no LLVM 6 in backports.
> >
> > I think there now is:
> > https://packages.debian.org/search?keywords=llvm=names=all=stretch-backports
> >
> > Package llvm-6.0-dev
> >
> > stretch-backports (devel): Modular compiler and toolchain technologies, 
> > libraries and headers
> > 1:6.0-1~bpo9+1: amd64
> >
> > It's a recent addition:
> >
> > llvm-toolchain-6.0 (1:6.0-1~bpo9+1) stretch-backports; urgency=medium
> >
> >   * Team upload
> >   * Rebuild for stretch-backports.
> >
> >  -- Anton Gladky   Mon, 12 Mar 2018 18:58:43 +0100
> 
> Huh, it hasn't made it to my mirror yet.

Interesting.


> Anyway, I upgraded and built with LLVM 6 and make check now passes on
> my arm64 system.  Woohoo!

Yay, thanks for testing!


> Via an off-list exchange I learned that Andres suspects a bug in LLVM
> 3.9 on arm64 and will investigate/maybe file a bug report with LLVM.
> Not sure if we'll want to try to actively identify and avoid known
> buggy versions or not?

I'm currently not inclined to invest a lot of effort into it, besides
trying to get the bug fixed.

A possible testcase would be to call
createLocalIndirectStubsManagerBuilder() and report an error if it
returns nullptr. But that'd fail once the bug is fixed, because we don't
actually *need* that functionality, it's just that LLVM instantiates the
stub manager unconditionally for some reason.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Thu, Mar 22, 2018 at 10:59 AM, Andres Freund  wrote:
> On 2018-03-22 10:50:52 +1300, Thomas Munro wrote:
>> Hmm.  There is no LLVM 6 in backports.
>
> I think there now is:
> https://packages.debian.org/search?keywords=llvm=names=all=stretch-backports
>
> Package llvm-6.0-dev
>
> stretch-backports (devel): Modular compiler and toolchain technologies, 
> libraries and headers
> 1:6.0-1~bpo9+1: amd64
>
> It's a recent addition:
>
> llvm-toolchain-6.0 (1:6.0-1~bpo9+1) stretch-backports; urgency=medium
>
>   * Team upload
>   * Rebuild for stretch-backports.
>
>  -- Anton Gladky   Mon, 12 Mar 2018 18:58:43 +0100

Huh, it hasn't made it to my mirror yet.

Anyway, I upgraded and built with LLVM 6 and make check now passes on
my arm64 system.  Woohoo!

Via an off-list exchange I learned that Andres suspects a bug in LLVM
3.9 on arm64 and will investigate/maybe file a bug report with LLVM.
Not sure if we'll want to try to actively identify and avoid known
buggy versions or not?

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Andres Freund]

On 2018-03-22 10:50:52 +1300, Thomas Munro wrote:
> On Thu, Mar 22, 2018 at 10:44 AM, Andres Freund  wrote:
> > On 2018-03-21 23:10:27 +1300, Thomas Munro wrote:
> >> Next up, I have an arm64 system running Debian 9.4.  It bombs in
> >> "make check" and in simple tests:
> >
> > Any chance you could try w/ LLVM 6?  It looks like some parts of ORC
> > only got aarch64 in LLVM 6.  I didn't *think* those were necessary, but
> > given the backtrace it looks like that still might be relevant.
> 
> Hmm.  There is no LLVM 6 in backports.

I think there now is:
https://packages.debian.org/search?keywords=llvm=names=all=stretch-backports

Package llvm-6.0-dev

stretch-backports (devel): Modular compiler and toolchain technologies, 
libraries and headers
1:6.0-1~bpo9+1: amd64

It's a recent addition:

llvm-toolchain-6.0 (1:6.0-1~bpo9+1) stretch-backports; urgency=medium

  * Team upload
  * Rebuild for stretch-backports.

 -- Anton Gladky   Mon, 12 Mar 2018 18:58:43 +0100

Otherwise I think LLVM has a repo with the necessary bits:
http://apt.llvm.org/


But if it's not this, I think we're going to have to indeed build
LLVM. Without proper debugging symbols it's going to be hard to figure
this out otherwise.

FWIW, I build it with:

mkdir -p ~/build/llvm/debug/vpath
cd ~/build/llvm/debug/vpath
cmake -G Ninja ~/src/llvm/ 
-DCMAKE_INSTALL_PREFIX=/home/andres/build/llvm/debug/install 
-DBUILD_SHARED_LIBS=true -DLLVM_TARGETS_TO_BUILD='X86;BPF' 
-DLLVM_CCACHE_BUILD=true
ninja -j8 install

I suspect you'd need to replace X86 with AArch64 (BPF isn't needed,
that's for stuff unrelated to PG).


> If that turns out to be it I guess we'd need to figure out how to
> detect an LLVM with bits missing hand handle it more gracefully?

Yea :/.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Thu, Mar 22, 2018 at 10:50 AM, Thomas Munro
 wrote:
> On Thu, Mar 22, 2018 at 10:44 AM, Andres Freund  wrote:
>> On 2018-03-21 23:10:27 +1300, Thomas Munro wrote:
>>> Next up, I have an arm64 system running Debian 9.4.  It bombs in
>>> "make check" and in simple tests:
>>
>> Any chance you could try w/ LLVM 6?  It looks like some parts of ORC
>> only got aarch64 in LLVM 6.  I didn't *think* those were necessary, but
>> given the backtrace it looks like that still might be relevant.
>
> Hmm.  There is no LLVM 6 in backports.  I'll have to build it, which
> I'm happy to do if I can wrap my brain around its cmake build system
> (or for you to build it if you want), but it may take... who knows, a
> day? on this little thing.

Actually scratch that, I'll just install buster.  More soon.

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Thu, Mar 22, 2018 at 10:44 AM, Andres Freund  wrote:
> On 2018-03-21 23:10:27 +1300, Thomas Munro wrote:
>> Next up, I have an arm64 system running Debian 9.4.  It bombs in
>> "make check" and in simple tests:
>
> Any chance you could try w/ LLVM 6?  It looks like some parts of ORC
> only got aarch64 in LLVM 6.  I didn't *think* those were necessary, but
> given the backtrace it looks like that still might be relevant.

Hmm.  There is no LLVM 6 in backports.  I'll have to build it, which
I'm happy to do if I can wrap my brain around its cmake build system
(or for you to build it if you want), but it may take... who knows, a
day? on this little thing.

If that turns out to be it I guess we'd need to figure out how to
detect an LLVM with bits missing hand handle it more gracefully?

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Thu, Mar 22, 2018 at 10:36 AM, Andres Freund  wrote:
> On 2018-03-21 14:21:01 -0700, Andres Freund wrote:
>> I think it's I that did something wrong not you. And the architecture
>> thing is a non-issue, because we're taking the target triple from the
>> right place.  I think it's a separate issue. Notably the generated code
>> is apparently corrupt, when reading in the generated bitcode:
>>
>> $ opt-6.0 -O3 -S < /tmp/data/6814.1.bc|less
>> opt-6.0: : error: Invalid record (Producer: 'LLVM6.0.0' Reader: 'LLVM 
>> 6.0.0')
>>
>> I suspect there's a 32bit vs 64bit confusion in the expression code
>> somewhere, might've accidentally used a 64bit type for Datum somewhere
>> or such. Will compile an LLVM with assertions enabled, to figure this
>> out (which verifies this kinda thing).
>
> Yup, that's it.  Found it by searching for 64bit references, while LLVM
> was compiling.  I've pushed quickfixes (for the 32 warnings, as well as
> for the 32bit x86 issue, as for configure typos).

Looks good here too.

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-22 09:31:12 +1300, Thomas Munro wrote:
> Aside from whatever problem is causing this, we can see that there is
> no top-level handling of exceptions.  That's probably fine if we are
> in a no throw scenario (unless there is something seriously corrupted,
> as is probably the case here), and it seems that we must be because
> we're accessing this code via its C API.

Yea, it should only happen in abort() type situations. Notably LLVM
doesn't even default to enabling exceptions...

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-21 23:10:27 +1300, Thomas Munro wrote:
> Next up, I have an arm64 system running Debian 9.4.  It bombs in
> "make check" and in simple tests:

Any chance you could try w/ LLVM 6?  It looks like some parts of ORC
only got aarch64 in LLVM 6.  I didn't *think* those were necessary, but
given the backtrace it looks like that still might be relevant.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-22 10:09:23 +1300, Thomas Munro wrote:
> > FWIW, a 32bit chroot, on a 64bit kernel works:
> >
> > 2018-03-21 20:57:56.576 UTC [3708] DEBUG:  successfully loaded LLVM in 
> > current session
> > 2018-03-21 20:57:56.577 UTC [3708] DEBUG:  JIT detected CPU "skylake", with 
> > features 
> > "+sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,+prfchw,+bmi2,+xsavec,+fsgsbase,+popcnt,+aes,-pcommit,+xsaves,-avx512er,-clwb,-avx512f,-pku,+smap,+mmx,-xop,+rdseed,+hle,-sse4a,-avx512bw,+clflushopt,+xsave,-avx512vl,+invpcid,-avx512cd,+avx,+rtm,+fma,+bmi,-mwaitx,+rdrnd,+sse4.1,+sse4.2,+avx2,+sse,+lzcnt,+pclmul,-prefetchwt1,+f16c,+ssse3,+sgx,+cmov,-avx512vbmi,+movbe,+xsaveopt,-sha,+adx,-avx512pf,+sse3"
> > 2018-03-21 20:57:56.579 UTC [3708] DEBUG:  time to inline: 0.000s, opt: 
> > 0.000s, emit: 0.002s
> >
> > that's debian testing though.
> 
> Hmm.  So now I'm doing this:

I've now reproduced this. It actually only fails for *some*
queries, a good number works. Investigating.


As a random aside, our costing is fairly ridiculous here:
┌──┐
│  QUERY PLAN  │
├──┤
│ Sort  (cost=1088314.21..1103119.40 rows=5922075 width=34)│
│   Sort Key: booltbl1.f1, booltbl2.f1 │
│   ->  Nested Loop  (cost=0.00..118524.73 rows=5922075 width=34)  │
│ Join Filter: ((booltbl2.f1 = booltbl1.f1) OR booltbl1.f1)│
│ ->  Seq Scan on booltbl1  (cost=0.00..38.10 rows=2810 width=1)   │
│ ->  Materialize  (cost=0.00..52.15 rows=2810 width=1)│
│   ->  Seq Scan on booltbl2  (cost=0.00..38.10 rows=2810 width=1) │
│ JIT: │
│   Functions: 6   │
│   Inlining: true │
│   Optimization: true │
└──┘


> I wonder what I'm doing wrong... what you're doing is very similar,
> right?  It's a 32 bit user land on a 64 bit kernel whereas mine is a
> 32 bit user land on a 32 bit kernel (on a 64 bit CPU).

I think it's I that did something wrong not you. And the architecture
thing is a non-issue, because we're taking the target triple from the
right place.  I think it's a separate issue. Notably the generated code
is apparently corrupt, when reading in the generated bitcode:

$ opt-6.0 -O3 -S < /tmp/data/6814.1.bc|less
opt-6.0: : error: Invalid record (Producer: 'LLVM6.0.0' Reader: 'LLVM 
6.0.0')

I suspect there's a 32bit vs 64bit confusion in the expression code
somewhere, might've accidentally used a 64bit type for Datum somewhere
or such. Will compile an LLVM with assertions enabled, to figure this
out (which verifies this kinda thing).

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Thu, Mar 22, 2018 at 9:59 AM, Andres Freund  wrote:
> On 2018-03-22 09:51:01 +1300, Thomas Munro wrote:
>> Hah, that makes sense.  I tried setting cpu to "x86", and now it fails
>> differently:
>
> Did you change the variable, or replace the value that's passed to the
> LLVMCreateTargetMachine() calls?  If you did the former, the error
> wouldn't be surprising, because
> LLVMDisposeMessage(cpu);
> cpu = NULL;
> will attempt to free the return value of LLVMGetHostCPUName(), which'll
> obviously not work if you just set to a constant.

Duh.  Right.

> FWIW, a 32bit chroot, on a 64bit kernel works:
>
> 2018-03-21 20:57:56.576 UTC [3708] DEBUG:  successfully loaded LLVM in 
> current session
> 2018-03-21 20:57:56.577 UTC [3708] DEBUG:  JIT detected CPU "skylake", with 
> features 
> "+sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,+prfchw,+bmi2,+xsavec,+fsgsbase,+popcnt,+aes,-pcommit,+xsaves,-avx512er,-clwb,-avx512f,-pku,+smap,+mmx,-xop,+rdseed,+hle,-sse4a,-avx512bw,+clflushopt,+xsave,-avx512vl,+invpcid,-avx512cd,+avx,+rtm,+fma,+bmi,-mwaitx,+rdrnd,+sse4.1,+sse4.2,+avx2,+sse,+lzcnt,+pclmul,-prefetchwt1,+f16c,+ssse3,+sgx,+cmov,-avx512vbmi,+movbe,+xsaveopt,-sha,+adx,-avx512pf,+sse3"
> 2018-03-21 20:57:56.579 UTC [3708] DEBUG:  time to inline: 0.000s, opt: 
> 0.000s, emit: 0.002s
>
> that's debian testing though.

Hmm.  So now I'm doing this:

llvm_opt0_targetmachine =
-   LLVMCreateTargetMachine(llvm_targetref, llvm_triple,
cpu, features,
+   LLVMCreateTargetMachine(llvm_targetref, llvm_triple,
"x86" /*cpu*/, "" /*features*/,

LLVMCodeGenLevelNone,

LLVMRelocDefault,

LLVMCodeModelJITDefault);
llvm_opt3_targetmachine =
-   LLVMCreateTargetMachine(llvm_targetref, llvm_triple,
cpu, features,
+   LLVMCreateTargetMachine(llvm_targetref, llvm_triple,
"x86" /*cpu*/, "" /*features*/,

LLVMCodeGenLevelAggressive,

LLVMRelocDefault,

LLVMCodeModelJITDefault);

And I'm still getting a segfault:

(gdb)
#0  0xac22c453 in llvm::SelectionDAG::getNode(unsigned int,
llvm::SDLoc const&, llvm::EVT, llvm::SDValue) () at
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/CodeGen/SelectionDAG/SelectionDAG.cpp:2898
#1  0xac269c29 in llvm::TargetLowering::SimplifySetCC(llvm::EVT,
llvm::SDValue, llvm::SDValue, llvm::ISD::CondCode, bool,
llvm::TargetLowering::DAGCombinerInfo&, llvm::SDLoc const&) const ()
at 
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/CodeGen/SelectionDAG/TargetLowering.cpp:1480
#2  0xac1163a8 in (anonymous
namespace)::DAGCombiner::visit(llvm::SDNode*) () at
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/CodeGen/SelectionDAG/DAGCombiner.cpp:14438
#3  0xac117f0b in (anonymous
namespace)::DAGCombiner::combine(llvm::SDNode*) () at
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/CodeGen/SelectionDAG/DAGCombiner.cpp:1449
#4  0xac11930e in llvm::SelectionDAG::Combine(llvm::CombineLevel,
llvm::AAResults&, llvm::CodeGenOpt::Level) () at
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/CodeGen/SelectionDAG/DAGCombiner.cpp:1303
#5  0xac245cec in llvm::SelectionDAGISel::CodeGenAndEmitDAG() () at
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:755
#6  0xac246239 in
llvm::SelectionDAGISel::SelectBasicBlock(llvm::ilist_iterator, llvm::ilist_iterator, bool&) ()
at 
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:679
#7  0xac24d66f in
llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) ()
at 
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:1482
#8  0xac25073c in
llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&)
() at 
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/CodeGen/SelectionDAG/SelectionDAGISel.cpp:500
#9  0xad34f414 in (anonymous
namespace)::X86DAGToDAGISel::runOnMachineFunction(llvm::MachineFunction&)
() at 
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/Target/X86/X86ISelDAGToDAG.cpp:175
#10 0xabf53019 in
llvm::MachineFunctionPass::runOnFunction(llvm::Function&) () at
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/CodeGen/MachineFunctionPass.cpp:60
#11 0xabde8aeb in llvm::FPPassManager::runOnFunction(llvm::Function&)
() at 
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/IR/LegacyPassManager.cpp:1526
#12 0xabde8e35 in llvm::FPPassManager::runOnModule(llvm::Module&) ()
at 
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/IR/LegacyPassManager.cpp:1547
#13 0xabde919a in llvm::legacy::PassManagerImpl::run(llvm::Module&) ()
at 
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/IR/LegacyPassManager.cpp:1603
#14 0xabde937f in llvm::legacy::PassManager::run(llvm::Module&) () at
/build/llvm-toolchain-3.9-UOOPrK/llvm-toolchain-3.9-3.9.1/lib/IR/LegacyPassManager.cpp:1737
#15 

Re: JIT compiling with LLVM v12.2

[Andres Freund]

On 2018-03-22 09:51:01 +1300, Thomas Munro wrote:
> On Thu, Mar 22, 2018 at 9:09 AM, Andres Freund  wrote:
> > Hi,
> >
> > On 2018-03-22 09:00:19 +1300, Thomas Munro wrote:
> >> 64 bit CPU, 32 bit OS.  I didn't try Debian multi-arch i386 support on
> >> an amd64 system, but that's probably an easier way to do this if you
> >> already have one of those...
> >
> > Ah, then I think I might know what happend. Does it start to work if you
> > replace the auto-detected cpu with "x86"? I think what might happen is
> > that it generates 64bit code, because of the detected CPU name.
> 
> Hah, that makes sense.  I tried setting cpu to "x86", and now it fails
> differently:

Did you change the variable, or replace the value that's passed to the
LLVMCreateTargetMachine() calls?  If you did the former, the error
wouldn't be surprising, because
LLVMDisposeMessage(cpu);
cpu = NULL;
will attempt to free the return value of LLVMGetHostCPUName(), which'll
obviously not work if you just set to a constant.

> #1  0xb7593806 in _int_free (av=0xae700220
> , p=0xae75f273, have_lock=0) at
> malloc.c:3905
> #2  0xabd05cd8 in LLVMDisposeMessage () from
> /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
> #3  0xae75100b in llvm_session_initialize () at llvmjit.c:636
> #4  llvm_create_context (jitFlags=15) at llvmjit.c:136
> #5  0xae75d3e9 in llvm_compile_expr (state=0x2616e60) at llvmjit_expr.c:132
> #6  0x00650118 in ExecReadyExpr (state=state@entry=0x2616e60) at 
> execExpr.c:627
> #7  0x00652dd7 in ExecInitExpr (node=0x2666bb4, parent=0x261693c) at
> execExpr.c:144
> ...


FWIW, a 32bit chroot, on a 64bit kernel works:

2018-03-21 20:57:56.576 UTC [3708] DEBUG:  successfully loaded LLVM in current 
session
2018-03-21 20:57:56.577 UTC [3708] DEBUG:  JIT detected CPU "skylake", with 
features 
"+sse2,+cx16,-tbm,-avx512ifma,-avx512dq,-fma4,+prfchw,+bmi2,+xsavec,+fsgsbase,+popcnt,+aes,-pcommit,+xsaves,-avx512er,-clwb,-avx512f,-pku,+smap,+mmx,-xop,+rdseed,+hle,-sse4a,-avx512bw,+clflushopt,+xsave,-avx512vl,+invpcid,-avx512cd,+avx,+rtm,+fma,+bmi,-mwaitx,+rdrnd,+sse4.1,+sse4.2,+avx2,+sse,+lzcnt,+pclmul,-prefetchwt1,+f16c,+ssse3,+sgx,+cmov,-avx512vbmi,+movbe,+xsaveopt,-sha,+adx,-avx512pf,+sse3"
2018-03-21 20:57:56.579 UTC [3708] DEBUG:  time to inline: 0.000s, opt: 0.000s, 
emit: 0.002s

that's debian testing though.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Thu, Mar 22, 2018 at 9:09 AM, Andres Freund  wrote:
> Hi,
>
> On 2018-03-22 09:00:19 +1300, Thomas Munro wrote:
>> 64 bit CPU, 32 bit OS.  I didn't try Debian multi-arch i386 support on
>> an amd64 system, but that's probably an easier way to do this if you
>> already have one of those...
>
> Ah, then I think I might know what happend. Does it start to work if you
> replace the auto-detected cpu with "x86"? I think what might happen is
> that it generates 64bit code, because of the detected CPU name.

Hah, that makes sense.  I tried setting cpu to "x86", and now it fails
differently:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  malloc_printerr (action=3, str=0xb7682d00 "free(): invalid
pointer", ptr=0xae75f27b, ar_ptr=0xae700220
) at malloc.c:5036
5036 malloc.c: No such file or directory.
(gdb) bt
#0  malloc_printerr (action=3, str=0xb7682d00 "free(): invalid
pointer", ptr=0xae75f27b, ar_ptr=0xae700220
) at malloc.c:5036
#1  0xb7593806 in _int_free (av=0xae700220
, p=0xae75f273, have_lock=0) at
malloc.c:3905
#2  0xabd05cd8 in LLVMDisposeMessage () from
/usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#3  0xae75100b in llvm_session_initialize () at llvmjit.c:636
#4  llvm_create_context (jitFlags=15) at llvmjit.c:136
#5  0xae75d3e9 in llvm_compile_expr (state=0x2616e60) at llvmjit_expr.c:132
#6  0x00650118 in ExecReadyExpr (state=state@entry=0x2616e60) at execExpr.c:627
#7  0x00652dd7 in ExecInitExpr (node=0x2666bb4, parent=0x261693c) at
execExpr.c:144
...

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Dagfinn Ilmari Mannsåker]

Andres Freund  writes:

> Hi,
>
> On 2018-03-21 20:06:49 +1300, Thomas Munro wrote:
>> On Wed, Mar 21, 2018 at 4:07 PM, Andres Freund  wrote:
>> > Indeed. I've pushed a rebased version now, that basically just fixes the
>> > issue Thomas observed.
>> 
>> I set up a 32 bit i386 virtual machine and installed Debian 9.4.
>> Compiler warnings:
>
> Was that with a 64bit CPU and 32bit OS, or actually a 32bit CPU?
>
>
>> gcc -Wall -Wmissing-prototypes -Wpointer-arith
>> -Wdeclaration-after-statement -Wendif-labels
>> -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
>> -fwrapv -fexcess-precision=standard -g -O2  -fPIC
>> -D__STDC_LIMIT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS
>> -D_GNU_SOURCE -I/usr/lib/llvm-3.9/include  -I../../../../src/include
>> -D_GNU_SOURCE   -c -o llvmjit.o llvmjit.c
>> llvmjit.c: In function ‘llvm_get_function’:
>> llvmjit.c:268:10: warning: cast to pointer from integer of different
>> size [-Wint-to-pointer-cast]
>>return (void *) addr;
>>   ^
>> llvmjit.c:270:10: warning: cast to pointer from integer of different
>> size [-Wint-to-pointer-cast]
>>return (void *) addr;
>>   ^
>> llvmjit.c: In function ‘llvm_resolve_symbol’:
>> llvmjit.c:842:10: warning: cast from pointer to integer of different
>> size [-Wpointer-to-int-cast]
>>addr = (uint64_t) load_external_function(modname, funcname,
>>   ^
>> llvmjit.c:845:10: warning: cast from pointer to integer of different
>> size [-Wpointer-to-int-cast]
>>addr = (uint64_t) LLVMSearchForAddressOfSymbol(symname);
>>   ^
>
> Hrmpf, those need to be fixed.

How about using uintptr_t is for this?  I see configure.in includes
AC_TYPE_UINTPTR_T, which probes for an existing uintptr_t or defines it
as an alias for the appropriate unsinged (long (long)) int type.

- ilmari
-- 
- Twitter seems more influential [than blogs] in the 'gets reported in
  the mainstream press' sense at least.   - Matt McLeod
- That'd be because the content of a tweet is easier to condense down
  to a mainstream media article.  - Calle Dybedahl

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Thu, Mar 22, 2018 at 9:06 AM, Andres Freund  wrote:
> On 2018-03-21 23:10:27 +1300, Thomas Munro wrote:
>> Next up, I have an arm64 system running Debian 9.4.  It bombs in
>> "make check" and in simple tests:
>
> Hum. Is it running a 32bit or 64 bit kernel/os?

checking size of void *... 8

>> ./configure \
>>   --prefix=$HOME/install \
>>   --enable-cassert \
>>   --enable-debug \
>>   --with-llvm \
>>   CC="ccache gcc" \
>>   CXX="ccache g++" \
>>   CLANG="ccache /usr/lib/llvm-3.9/bin/clang" \
>>   LLVM_CONFIG="/usr/lib/llvm-3.9/bin/llvm-config"
>
> I guess you'd swapped out 3.9 for 5.0?

Right, in the second backtrace I showed it was 5.0 (for both clang and
llvm-config).

>> I can provide access to this thing if you think that'd be useful.
>
> Perhaps that's necessary. Before that though, could you check how the
> backtrace looks with LLVM debug symbols installed?

After installing libllvm3.9-dgb:

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0xa1ae1df4 in __GI_abort () at abort.c:89
#2  0x9634ee40 in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#3  0x9634cd4c in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#4  0x9634cd98 in std::terminate() () from
/usr/lib/aarch64-linux-gnu/libstdc++.so.6
#5  0x9634d01c in __cxa_throw () from
/usr/lib/aarch64-linux-gnu/libstdc++.so.6
#6  0x963754bc in std::__throw_bad_function_call() () from
/usr/lib/aarch64-linux-gnu/libstdc++.so.6
warning: Could not find DWO CU
CMakeFiles/LLVMOrcJIT.dir/OrcCBindings.cpp.dwo(0x691f70a1d71f901d)
referenced by CU at offset 0x11ffa [in module
/usr/lib/debug/.build-id/09/04bb3e707305e175216a59bc3598c2b194775a.debug]
#7  0x97697a2c in LLVMOrcCreateInstance () at
/usr/include/c++/6/functional:2126
#8  0x98accdb0 in llvm_session_initialize () at llvmjit.c:643
#9  llvm_create_context (jitFlags=9) at llvmjit.c:136
#10 0x98ad78c8 in llvm_compile_expr (state=0xfce73208) at
llvmjit_expr.c:132
#11 0xc1bd671c in ExecReadyExpr
(state=state@entry=0xfce73208) at execExpr.c:627
#12 0xc1bd97b8 in ExecBuildProjectionInfo
(targetList=, econtext=, slot=, parent=parent@entry=0xfce72de0,
inputDesc=inputDesc@entry=0x0) at execExpr.c:471
#13 0xc1bec028 in ExecAssignProjectionInfo
(planstate=planstate@entry=0xfce72de0,
inputDesc=inputDesc@entry=0x0) at execUtils.c:460
#14 0xc1c08a28 in ExecInitResult
(node=node@entry=0xfcdc11a0, estate=estate@entry=0xfce72bc8,
eflags=eflags@entry=16) at nodeResult.c:221
#15 0xc1be7828 in ExecInitNode (node=0xfcdc11a0,
node@entry=0xfcded630, estate=estate@entry=0xfce72bc8,
eflags=eflags@entry=16) at execProcnode.c:164
#16 0xc1be2a70 in InitPlan (eflags=16,
queryDesc=0xfcde0808) at execMain.c:1051
#17 standard_ExecutorStart (queryDesc=0xfcde0808, eflags=16) at
execMain.c:266
#18 0xc1d39bec in PortalStart (portal=0x400,
portal@entry=0xfce234d8, params=0x274580612ce0a285,
params@entry=0x0, eflags=43690, eflags@entry=0,
snapshot=0xc1fa9f58, snapshot@entry=0x0) at pquery.c:520
#19 0xc1d34b18 in exec_simple_query
(query_string=query_string@entry=0xfcdbf3d8 "select 42;") at
postgres.c:1082
#20 0xc1d366a8 in PostgresMain (argc=,
argv=argv@entry=0xfcdebb90, dbname=,
username=) at postgres.c:4147
#21 0xc1a28dd0 in BackendRun (port=0xfcde0410) at postmaster.c:4409
#22 BackendStartup (port=0xfcde0410) at postmaster.c:4081
#23 ServerLoop () at postmaster.c:1754
#24 0xc1cb7048 in PostmasterMain (argc=,
argv=) at postmaster.c:1362
#25 0xc1a2a7cc in main (argc=3, argv=0xfcdb9f70) at main.c:228

GDB also printed a ton of messages like this for many LLVM .cpp files:

warning: Could not find DWO CU
CMakeFiles/LLVMLibDriver.dir/LibDriver.cpp.dwo(0x117022032f862080)
referenced by CU at offset 0x187da [in module
/usr/lib/debug/.build-id/09/04bb3e707305e175216a59bc3598c2b194775a.debug]
...

We can see that it's missing some debug info, any clues about that?
Here's LLVM 3.9's LLVMOrcCreateInstance function:

https://github.com/llvm-mirror/llvm/blob/6531c3164cb9edbfb9f4b43ca383810a94ca5aa0/lib/ExecutionEngine/Orc/OrcCBindings.cpp#L15

Without digging through more source code I'm not sure which line of
that is invoking our uninvocable std::function...

The server also prints this:

terminate called after throwing an instance of 'std::bad_function_call'
  what():  bad_function_call

Aside from whatever problem is causing this, we can see that there is
no top-level handling of exceptions.  That's probably fine if we are
in a no throw scenario (unless there is something seriously corrupted,
as is probably the case here), and it seems that we must be because
we're accessing this code via its C API.

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-22 09:00:19 +1300, Thomas Munro wrote:
> On Thu, Mar 22, 2018 at 8:47 AM, Andres Freund  wrote:
> > On 2018-03-21 20:06:49 +1300, Thomas Munro wrote:
> >> On Wed, Mar 21, 2018 at 4:07 PM, Andres Freund  wrote:
> >> > Indeed. I've pushed a rebased version now, that basically just fixes the
> >> > issue Thomas observed.
> >>
> >> I set up a 32 bit i386 virtual machine and installed Debian 9.4.
> >> Compiler warnings:
> >
> > Was that with a 64bit CPU and 32bit OS, or actually a 32bit CPU?
> 
> 64 bit CPU, 32 bit OS.  I didn't try Debian multi-arch i386 support on
> an amd64 system, but that's probably an easier way to do this if you
> already have one of those...

Ah, then I think I might know what happend. Does it start to work if you
replace the auto-detected cpu with "x86"? I think what might happen is
that it generates 64bit code, because of the detected CPU name.

Let me set up a chroot, in this case I should be able to emulate this
pretty easily...

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Andres Freund]

On 2018-03-21 23:10:27 +1300, Thomas Munro wrote:
> On Wed, Mar 21, 2018 at 8:06 PM, Thomas Munro
>  wrote:
> > On Wed, Mar 21, 2018 at 4:07 PM, Andres Freund  wrote:
> >> Indeed. I've pushed a rebased version now, that basically just fixes the
> >> issue Thomas observed.
> >
> > I set up a 32 bit i386 virtual machine and installed Debian 9.4.
> 
> Next up, I have an arm64 system running Debian 9.4.  It bombs in
> "make check" and in simple tests:

Hum. Is it running a 32bit or 64 bit kernel/os?



> Program received signal SIGABRT, Aborted.
> __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> 51  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> #1  0x8f65adf4 in __GI_abort () at abort.c:89
> #2  0x83e2de40 in __gnu_cxx::__verbose_terminate_handler() ()
> from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
> #3  0x83e2bd4c in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
> #4  0x83e2bd98 in std::terminate() () from
> /usr/lib/aarch64-linux-gnu/libstdc++.so.6
> #5  0x83e2c01c in __cxa_throw () from
> /usr/lib/aarch64-linux-gnu/libstdc++.so.6
> #6  0x83e544bc in std::__throw_bad_function_call() () from
> /usr/lib/aarch64-linux-gnu/libstdc++.so.6
> #7  0x85176a2c in LLVMOrcCreateInstance () from
> /usr/lib/aarch64-linux-gnu/libLLVM-3.9.so.1
> #8  0x865c4db0 in llvm_session_initialize () at llvmjit.c:643
> #9  llvm_create_context (jitFlags=9) at llvmjit.c:136
> #10 0x865cf8c8 in llvm_compile_expr (state=0xf2300208) at
> llvmjit_expr.c:132

Hm.


> With LLVM 5.0 (from backports) it seemed to get further (?):
> 
> Program terminated with signal SIGABRT, Aborted.
> #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> 51  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
> #1  0xa9642df4 in __GI_abort () at abort.c:89
> #2  0x9d306e40 in __gnu_cxx::__verbose_terminate_handler() ()
> from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
> #3  0x9d304d4c in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
> #4  0x9d304d98 in std::terminate() () from
> /usr/lib/aarch64-linux-gnu/libstdc++.so.6
> #5  0x9d30501c in __cxa_throw () from
> /usr/lib/aarch64-linux-gnu/libstdc++.so.6
> #6  0x9d32d4bc in std::__throw_bad_function_call() () from
> /usr/lib/aarch64-linux-gnu/libstdc++.so.6
> #7  0x9eac7dc4 in ?? () from 
> /usr/lib/aarch64-linux-gnu/libLLVM-5.0.so.1
> #8  0xdd2dced0 in ?? ()
> #9  0x40100401 in ?? ()
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
> (gdb)
> 
> Configure was run like this:
> 
> ./configure \
>   --prefix=$HOME/install \
>   --enable-cassert \
>   --enable-debug \
>   --with-llvm \
>   CC="ccache gcc" \
>   CXX="ccache g++" \
>   CLANG="ccache /usr/lib/llvm-3.9/bin/clang" \
>   LLVM_CONFIG="/usr/lib/llvm-3.9/bin/llvm-config"

I guess you'd swapped out 3.9 for 5.0?


> I can provide access to this thing if you think that'd be useful.

Perhaps that's necessary. Before that though, could you check how the
backtrace looks with LLVM debug symbols installed?

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-21 08:26:28 +0100, Catalin Iacob wrote:
> On Wed, Mar 21, 2018 at 4:07 AM, Andres Freund  wrote:
> > Indeed. I've pushed a rebased version now, that basically just fixes the
> > issue Thomas observed.
> 
> Testing 2d6f2fba from your repository configured --with-llvm I noticed
> some weird things in the configure output.

Thanks!

> Without --enable-debug:
> configure: using compiler=gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
> configure: using CFLAGS=-Wall -Wmissing-prototypes -Wpointer-arith
> -Wdeclaration-after-statement -Wendif-labels
> -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
> -fwrapv -fexcess-precision=standard -O2
> configure: using CPPFLAGS= -D_GNU_SOURCE
> configure: using LDFLAGS= -L/opt/rh/llvm-toolset-7/root/usr/lib64
> -Wl,--as-needed
> configure: using CXX=g++
> configure: using CXXFLAGS=-Wall -Wpointer-arith -Wendif-labels
> -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
> -fwrapv -g -O2
> configure: using CLANG=/opt/rh/llvm-toolset-7/root/usr/bin/clang
> configure: using BITCODE_CFLAGS= -fno-strict-aliasing -fwrapv -O2
> configure: using BITCODE_CXXFLAGS= -fno-strict-aliasing -fwrapv -O2
> BITCODE_CXXFLAGS
> 
> With --enable-debug:
> configure: using compiler=gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
> configure: using CFLAGS=-Wall -Wmissing-prototypes -Wpointer-arith
> -Wdeclaration-after-statement -Wendif-labels
> -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
> -fwrapv -fexcess-precision=standard -g -O2
> configure: using CPPFLAGS= -D_GNU_SOURCE
> configure: using LDFLAGS= -L/opt/rh/llvm-toolset-7/root/usr/lib64
> -Wl,--as-needed
> configure: using CXX=g++
> configure: using CXXFLAGS=-Wall -Wpointer-arith -Wendif-labels
> -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
> -fwrapv -g -g -O2
> configure: using CLANG=/opt/rh/llvm-toolset-7/root/usr/bin/clang
> configure: using BITCODE_CFLAGS= -fno-strict-aliasing -fwrapv -O2
> configure: using BITCODE_CXXFLAGS= -fno-strict-aliasing -fwrapv -O2
> BITCODE_CXXFLAGS
> 
> So I unconditionally get one -g added to CXXFLAGS regardless of
> whether I specify --enable-debug or not. And --enable-debug results in
> -g -g in CXXFLAGS.

Aaah, nice catch.  I was missing an unset CXXFLAGS.


> BTW, you should probably specify -std=c++11 (or whatever you need) as
> various g++ and clang++ versions default to various things. Will the
> required C++ standard be based on the requirements of the C++ code in
> the PG tree or will you take it from LLVM's CXXFLAGS?

It's currently already taken from LLVM's CXXFLAGS if present there, but
just specified for LLVM wrapping files.  Relevant code is in
src/backend/jit/llvm/Makefile:

# All files in this directy link to LLVM.
CFLAGS += $(LLVM_CFLAGS)
CXXFLAGS += $(LLVM_CXXFLAGS)
override CPPFLAGS := $(LLVM_CPPFLAGS) $(CPPFLAGS)
SHLIB_LINK += $(LLVM_LIBS)

Since there's no other C++ code, and I don't forsee anything else, I'm
not planning to set the global CXXFLAGS differently atm. Would just make
it more complicated to use the right flag from llvm's CXXFLAGSS.


> Can --std=c++11 and --std=c++14 compiled .o files be linked together?

Yes, with some limitations. In the PG case all the intra-file calls are
C ABI, there wouldn't be a problem.


> Also, my CXXFLAGS did not get -fexcess-precision=standard neither did
> BITCODE_CFLAGS nor BITCODE_CXXFLAGS.

Yea, that's to be expected, gcc doesn't know that for C++ on most
versions. Some vendors have it patched into.


> 2. Unlike all the other *FLAGS, BITCODE_CXXFLAGS includes itself on
> the right hand side of the equal
> configure: using BITCODE_CXXFLAGS= -fno-strict-aliasing -fwrapv -O2
> BITCODE_CXXFLAGS

Hum, that's definitely a typo bug (missing $ when adding to
BITCODE_CXXFLAGS).


Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Thu, Mar 22, 2018 at 8:47 AM, Andres Freund  wrote:
> On 2018-03-21 20:06:49 +1300, Thomas Munro wrote:
>> On Wed, Mar 21, 2018 at 4:07 PM, Andres Freund  wrote:
>> > Indeed. I've pushed a rebased version now, that basically just fixes the
>> > issue Thomas observed.
>>
>> I set up a 32 bit i386 virtual machine and installed Debian 9.4.
>> Compiler warnings:
>
> Was that with a 64bit CPU and 32bit OS, or actually a 32bit CPU?

64 bit CPU, 32 bit OS.  I didn't try Debian multi-arch i386 support on
an amd64 system, but that's probably an easier way to do this if you
already have one of those...

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-21 20:06:49 +1300, Thomas Munro wrote:
> On Wed, Mar 21, 2018 at 4:07 PM, Andres Freund  wrote:
> > Indeed. I've pushed a rebased version now, that basically just fixes the
> > issue Thomas observed.
> 
> I set up a 32 bit i386 virtual machine and installed Debian 9.4.
> Compiler warnings:

Was that with a 64bit CPU and 32bit OS, or actually a 32bit CPU?


> gcc -Wall -Wmissing-prototypes -Wpointer-arith
> -Wdeclaration-after-statement -Wendif-labels
> -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
> -fwrapv -fexcess-precision=standard -g -O2  -fPIC
> -D__STDC_LIMIT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS
> -D_GNU_SOURCE -I/usr/lib/llvm-3.9/include  -I../../../../src/include
> -D_GNU_SOURCE   -c -o llvmjit.o llvmjit.c
> llvmjit.c: In function ‘llvm_get_function’:
> llvmjit.c:268:10: warning: cast to pointer from integer of different
> size [-Wint-to-pointer-cast]
>return (void *) addr;
>   ^
> llvmjit.c:270:10: warning: cast to pointer from integer of different
> size [-Wint-to-pointer-cast]
>return (void *) addr;
>   ^
> llvmjit.c: In function ‘llvm_resolve_symbol’:
> llvmjit.c:842:10: warning: cast from pointer to integer of different
> size [-Wpointer-to-int-cast]
>addr = (uint64_t) load_external_function(modname, funcname,
>   ^
> llvmjit.c:845:10: warning: cast from pointer to integer of different
> size [-Wpointer-to-int-cast]
>addr = (uint64_t) LLVMSearchForAddressOfSymbol(symname);
>   ^

Hrmpf, those need to be fixed.



> While trying out many combinations of versions of stuff on different
> OSes, I found another way to screw up that I wanted to report here.
> It's obvious that this is doomed if you know what's going on, but I
> thought the failure mode was interesting enough to report here.  There
> is a hazard for people running systems where the vendor ships some
> version (possibly a mystery version) of clang in the PATH but you have
> to get LLVM separately (eg from ports/brew/whatever):

> 1.  If you use macOS High Sierra's current /usr/bin/clang ("9.0.0"),
> ie the default if you didn't set CLANG to something else when you ran
> ./configure, and you build against LLVM 3.9, then llvm-lto gives this
> message during "make install":
> 
> Invalid summary version 3, 1 expected
> error: can't create ModuleSummaryIndexObjectFile for buffer: Corrupted bitcode
> 
> Then it segfaults!

Gah, that's not desirable :/.  It's fine that it doesn't work, but it'd
be better if it didn't segfault. I guess I could just try by corrupting
the file explicitly...



> Neither of these cases should be too surprising, and users of those
> operating systems can easily get a newer LLVM or an older -- it was
> just interesting to see exactly what goes wrong and exactly when.  I
> suppose there could be a configure test to see if your $CLANG can play
> nicely with your $LLVM_CONFIG.

Not precisely sure how.  I think suggesting to use compatible clang is
going to be sufficient for most cases...

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Wed, Mar 21, 2018 at 8:06 PM, Thomas Munro
 wrote:
> On Wed, Mar 21, 2018 at 4:07 PM, Andres Freund  wrote:
>> Indeed. I've pushed a rebased version now, that basically just fixes the
>> issue Thomas observed.
>
> I set up a 32 bit i386 virtual machine and installed Debian 9.4.

Next up, I have an arm64 system running Debian 9.4.  It bombs in
"make check" and in simple tests:

postgres=# set jit_above_cost = 0;
SET
postgres=# select 42;


The stack looks like this:

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x8f65adf4 in __GI_abort () at abort.c:89
#2  0x83e2de40 in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#3  0x83e2bd4c in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#4  0x83e2bd98 in std::terminate() () from
/usr/lib/aarch64-linux-gnu/libstdc++.so.6
#5  0x83e2c01c in __cxa_throw () from
/usr/lib/aarch64-linux-gnu/libstdc++.so.6
#6  0x83e544bc in std::__throw_bad_function_call() () from
/usr/lib/aarch64-linux-gnu/libstdc++.so.6
#7  0x85176a2c in LLVMOrcCreateInstance () from
/usr/lib/aarch64-linux-gnu/libLLVM-3.9.so.1
#8  0x865c4db0 in llvm_session_initialize () at llvmjit.c:643
#9  llvm_create_context (jitFlags=9) at llvmjit.c:136
#10 0x865cf8c8 in llvm_compile_expr (state=0xf2300208) at
llvmjit_expr.c:132
#11 0xb64ca71c in ExecReadyExpr
(state=state@entry=0xf2300208) at execExpr.c:627
#12 0xb64cd7b8 in ExecBuildProjectionInfo
(targetList=, econtext=, slot=, parent=parent@entry=0xf22ffde0,
inputDesc=inputDesc@entry=0x0)
at execExpr.c:471
#13 0xb64e0028 in ExecAssignProjectionInfo
(planstate=planstate@entry=0xf22ffde0,
inputDesc=inputDesc@entry=0x0) at execUtils.c:460
#14 0xb64fca28 in ExecInitResult
(node=node@entry=0xf224e1a0, estate=estate@entry=0xf22ffbc8,
eflags=eflags@entry=16) at nodeResult.c:221
#15 0xb64db828 in ExecInitNode (node=0xf224e1a0,
node@entry=0xf227a610, estate=estate@entry=0xf22ffbc8,
eflags=eflags@entry=16) at execProcnode.c:164
#16 0xb64d6a70 in InitPlan (eflags=16,
queryDesc=0xf226d808) at execMain.c:1051
#17 standard_ExecutorStart (queryDesc=0xf226d808, eflags=16) at
execMain.c:266
#18 0xb662dbec in PortalStart (portal=0x400,
portal@entry=0xf22b04d8, params=0x59004077f060bc65,
params@entry=0x0, eflags=43690, eflags@entry=0,
snapshot=0xb689df58, snapshot@entry=0x0)
at pquery.c:520
#19 0xb6628b18 in exec_simple_query
(query_string=query_string@entry=0xf224c3d8 "select 42;") at
postgres.c:1082
#20 0xb662a6a8 in PostgresMain (argc=,
argv=argv@entry=0xf2278b70, dbname=,
username=) at postgres.c:4147
#21 0xb631cdd0 in BackendRun (port=0xf226d410) at postmaster.c:4409
#22 BackendStartup (port=0xf226d410) at postmaster.c:4081
#23 ServerLoop () at postmaster.c:1754
#24 0xb65ab048 in PostmasterMain (argc=,
argv=) at postmaster.c:1362
#25 0xb631e7cc in main (argc=3, argv=0xf2246f70) at main.c:228

Taking frame 6 at face value, it appears to be trying to call an empty
std::function (that's what the exception std::bad_function_call
means).  No clue how or why though.

With LLVM 5.0 (from backports) it seemed to get further (?):

Program terminated with signal SIGABRT, Aborted.
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0xa9642df4 in __GI_abort () at abort.c:89
#2  0x9d306e40 in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#3  0x9d304d4c in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6
#4  0x9d304d98 in std::terminate() () from
/usr/lib/aarch64-linux-gnu/libstdc++.so.6
#5  0x9d30501c in __cxa_throw () from
/usr/lib/aarch64-linux-gnu/libstdc++.so.6
#6  0x9d32d4bc in std::__throw_bad_function_call() () from
/usr/lib/aarch64-linux-gnu/libstdc++.so.6
#7  0x9eac7dc4 in ?? () from /usr/lib/aarch64-linux-gnu/libLLVM-5.0.so.1
#8  0xdd2dced0 in ?? ()
#9  0x40100401 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

Configure was run like this:

./configure \
  --prefix=$HOME/install \
  --enable-cassert \
  --enable-debug \
  --with-llvm \
  CC="ccache gcc" \
  CXX="ccache g++" \
  CLANG="ccache /usr/lib/llvm-3.9/bin/clang" \
  LLVM_CONFIG="/usr/lib/llvm-3.9/bin/llvm-config"

I can provide access to this thing if you think that'd be useful.

-- 
Thomas Munro

Re: JIT compiling with LLVM v12.2

[Catalin Iacob]

On Wed, Mar 21, 2018 at 4:07 AM, Andres Freund  wrote:
> Indeed. I've pushed a rebased version now, that basically just fixes the
> issue Thomas observed.

Testing 2d6f2fba from your repository configured --with-llvm I noticed
some weird things in the configure output.

Without --enable-debug:
configure: using compiler=gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
configure: using CFLAGS=-Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels
-Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
-fwrapv -fexcess-precision=standard -O2
configure: using CPPFLAGS= -D_GNU_SOURCE
configure: using LDFLAGS= -L/opt/rh/llvm-toolset-7/root/usr/lib64
-Wl,--as-needed
configure: using CXX=g++
configure: using CXXFLAGS=-Wall -Wpointer-arith -Wendif-labels
-Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
-fwrapv -g -O2
configure: using CLANG=/opt/rh/llvm-toolset-7/root/usr/bin/clang
configure: using BITCODE_CFLAGS= -fno-strict-aliasing -fwrapv -O2
configure: using BITCODE_CXXFLAGS= -fno-strict-aliasing -fwrapv -O2
BITCODE_CXXFLAGS

With --enable-debug:
configure: using compiler=gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)
configure: using CFLAGS=-Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels
-Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
-fwrapv -fexcess-precision=standard -g -O2
configure: using CPPFLAGS= -D_GNU_SOURCE
configure: using LDFLAGS= -L/opt/rh/llvm-toolset-7/root/usr/lib64
-Wl,--as-needed
configure: using CXX=g++
configure: using CXXFLAGS=-Wall -Wpointer-arith -Wendif-labels
-Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
-fwrapv -g -g -O2
configure: using CLANG=/opt/rh/llvm-toolset-7/root/usr/bin/clang
configure: using BITCODE_CFLAGS= -fno-strict-aliasing -fwrapv -O2
configure: using BITCODE_CXXFLAGS= -fno-strict-aliasing -fwrapv -O2
BITCODE_CXXFLAGS

So I unconditionally get one -g added to CXXFLAGS regardless of
whether I specify --enable-debug or not. And --enable-debug results in
-g -g in CXXFLAGS.

Didn't get to look at the code yet, maybe that comes from:
$ llvm-config --cxxflags
-I/opt/rh/llvm-toolset-7/root/usr/include -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic
-fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter
-Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic
-Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor
-Wno-comment -std=c++11 -ffunction-sections -fdata-sections -O2 -g
-DNDEBUG  -fno-exceptions -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS
-D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS

But on the other hand there are lots of other flags in there that
don't end up in CXXFLAGS.

BTW, you should probably specify -std=c++11 (or whatever you need) as
various g++ and clang++ versions default to various things. Will the
required C++ standard be based on the requirements of the C++ code in
the PG tree or will you take it from LLVM's CXXFLAGS? Can --std=c++11
and --std=c++14 compiled .o files be linked together? Or in other
words, in case in the future LLVM starts requiring C++14 but the code
in the PG tree you wrote still builds with C++11, will PG upgrade it's
requirement with LLVM or will it stay with the older standard?

Also, my CXXFLAGS did not get -fexcess-precision=standard neither did
BITCODE_CFLAGS nor BITCODE_CXXFLAGS.

In case it's interesting:
$ llvm-config --cflags
-I/opt/rh/llvm-toolset-7/root/usr/include -O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches   -m64 -mtune=generic
-fPIC -Wall -W -Wno-unused-parameter -Wwrite-strings
-Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-comment
-ffunction-sections -fdata-sections -O2 -g -DNDEBUG -D_GNU_SOURCE
-D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS


2. Unlike all the other *FLAGS, BITCODE_CXXFLAGS includes itself on
the right hand side of the equal
configure: using BITCODE_CXXFLAGS= -fno-strict-aliasing -fwrapv -O2
BITCODE_CXXFLAGS

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Wed, Mar 21, 2018 at 4:07 PM, Andres Freund  wrote:
> Indeed. I've pushed a rebased version now, that basically just fixes the
> issue Thomas observed.

I set up a 32 bit i386 virtual machine and installed Debian 9.4.
Compiler warnings:

gcc -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels
-Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
-fwrapv -fexcess-precision=standard -g -O2  -fPIC
-D__STDC_LIMIT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_CONSTANT_MACROS
-D_GNU_SOURCE -I/usr/lib/llvm-3.9/include  -I../../../../src/include
-D_GNU_SOURCE   -c -o llvmjit.o llvmjit.c
llvmjit.c: In function ‘llvm_get_function’:
llvmjit.c:268:10: warning: cast to pointer from integer of different
size [-Wint-to-pointer-cast]
   return (void *) addr;
  ^
llvmjit.c:270:10: warning: cast to pointer from integer of different
size [-Wint-to-pointer-cast]
   return (void *) addr;
  ^
llvmjit.c: In function ‘llvm_resolve_symbol’:
llvmjit.c:842:10: warning: cast from pointer to integer of different
size [-Wpointer-to-int-cast]
   addr = (uint64_t) load_external_function(modname, funcname,
  ^
llvmjit.c:845:10: warning: cast from pointer to integer of different
size [-Wpointer-to-int-cast]
   addr = (uint64_t) LLVMSearchForAddressOfSymbol(symname);
  ^

Then "make check" bombs:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0xac233453 in llvm::SelectionDAG::getNode(unsigned int,
llvm::SDLoc const&, llvm::EVT, llvm::SDValue) () from
/usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
(gdb) bt
#0  0xac233453 in llvm::SelectionDAG::getNode(unsigned int,
llvm::SDLoc const&, llvm::EVT, llvm::SDValue) () from
/usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#1  0xac270c29 in llvm::TargetLowering::SimplifySetCC(llvm::EVT,
llvm::SDValue, llvm::SDValue, llvm::ISD::CondCode, bool,
llvm::TargetLowering::DAGCombinerInfo&, llvm::SDLoc const&) const ()
from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#2  0xac11d3a8 in ?? () from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#3  0xac11ef0b in ?? () from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#4  0xac12030e in llvm::SelectionDAG::Combine(llvm::CombineLevel,
llvm::AAResults&, llvm::CodeGenOpt::Level) () from
/usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#5  0xac24ccec in llvm::SelectionDAGISel::CodeGenAndEmitDAG() () from
/usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#6  0xac24d239 in
llvm::SelectionDAGISel::SelectBasicBlock(llvm::ilist_iterator, llvm::ilist_iterator, bool&) () from
/usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#7  0xac25466f in
llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) ()
from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#8  0xac25773c in
llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&)
() from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#9  0xad356414 in ?? () from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#10 0xabf5a019 in
llvm::MachineFunctionPass::runOnFunction(llvm::Function&) () from
/usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#11 0xabdefaeb in llvm::FPPassManager::runOnFunction(llvm::Function&)
() from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#12 0xabdefe35 in llvm::FPPassManager::runOnModule(llvm::Module&) ()
from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#13 0xabdf019a in llvm::legacy::PassManagerImpl::run(llvm::Module&) ()
from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#14 0xabdf037f in llvm::legacy::PassManager::run(llvm::Module&) ()
from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#15 0xacb3c3de in
std::_Function_handler::_M_invoke(std::_Any_data
const&, llvm::Module&) () from
/usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#16 0xacb37d00 in ?? () from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#17 0xacb384f8 in ?? () from /usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#18 0xacb388d5 in LLVMOrcAddEagerlyCompiledIR () from
/usr/lib/i386-linux-gnu/libLLVM-3.9.so.1
#19 0xae7bb3e4 in llvm_compile_module (context=0x20858a0) at llvmjit.c:539
#20 llvm_get_function (context=0x20858a0, funcname=0x21da818
"evalexpr_2_3") at llvmjit.c:244
#21 0xae7c333e in ExecRunCompiledExpr (state=0x2119634,
econtext=0x211810c, isNull=0xbfdd138e "\207") at llvmjit_expr.c:2563
#22 0x00745e10 in ExecEvalExprSwitchContext (isNull=0xbfdd138e "\207",
econtext=, state=0x2119634) at
../../../src/include/executor/executor.h:305
#23 ExecQual (econtext=, state=0x2119634) at
../../../src/include/executor/executor.h:374
#24 ExecNestLoop (pstate=) at nodeNestloop.c:214
#25 0x00748ddd in ExecProcNode (node=0x2118080) at
../../../src/include/executor/executor.h:239
#26 ExecSort (pstate=0x2117ff4) at nodeSort.c:107
#27 0x0071e9d2 in ExecProcNode (node=0x2117ff4) at
../../../src/include/executor/executor.h:239
#28 ExecutePlan (execute_once=, dest=0x0,
direction=NoMovementScanDirection, numberTuples=,
sendTuples=, operation=CMD_SELECT,
use_parallel_mode=, planstate=0x2117ff4,
estate=0x2117ee8) at execMain.c:1729
#29 standard_ExecutorRun 

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-20 23:03:13 -0400, Stephen Frost wrote:
> Greetings,
> 
> * Andres Freund (and...@anarazel.de) wrote:
> > On 2018-03-20 19:29:55 -0700, Andres Freund wrote:
> > > On 2018-03-21 15:22:08 +1300, Thomas Munro wrote:
> > > > Somehow your configure test correctly concludes that my $CC (clang
> > > > 4.0) doesn't support -fexcess-precision=standard but that my $CXX
> > > > (clang++ 4.0) does, despite producing a nearly identical warning:
> > > 
> > > Yea, there was a copy & pasto (s/ac_c_werror_flag/ac_cxx_werror_flag/),
> > > sorry. If you rebase onto the committed version, it should work?  I'll
> > > push a version rebased of the jit tree soon.
> > 
> > Well, or not. Seems git.pg.o is down atm:
> > 
> > debug1: Next authentication method: publickey
> > debug1: Offering public key: RSA 
> > SHA256:cMbSa8YBm8AgaIeMtCSFvvPDrrrdadCxzQaFiWFe+7c /home/andres/.ssh/id_rsa
> > debug1: Server accepts key: pkalg ssh-rsa blen 277
> > 
> > 
> > Will try tomorrow.
> 
> Andres contacted pginfra over IRC about this, but it seems that it
> resolved itself shortly following (per a comment from Andres to that
> effect), so, afaik, things are working properly.

Indeed. I've pushed a rebased version now, that basically just fixes the
issue Thomas observed.

Thanks,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Stephen Frost]

Greetings,

* Andres Freund (and...@anarazel.de) wrote:
> On 2018-03-20 19:29:55 -0700, Andres Freund wrote:
> > On 2018-03-21 15:22:08 +1300, Thomas Munro wrote:
> > > Somehow your configure test correctly concludes that my $CC (clang
> > > 4.0) doesn't support -fexcess-precision=standard but that my $CXX
> > > (clang++ 4.0) does, despite producing a nearly identical warning:
> > 
> > Yea, there was a copy & pasto (s/ac_c_werror_flag/ac_cxx_werror_flag/),
> > sorry. If you rebase onto the committed version, it should work?  I'll
> > push a version rebased of the jit tree soon.
> 
> Well, or not. Seems git.pg.o is down atm:
> 
> debug1: Next authentication method: publickey
> debug1: Offering public key: RSA 
> SHA256:cMbSa8YBm8AgaIeMtCSFvvPDrrrdadCxzQaFiWFe+7c /home/andres/.ssh/id_rsa
> debug1: Server accepts key: pkalg ssh-rsa blen 277
> 
> 
> Will try tomorrow.

Andres contacted pginfra over IRC about this, but it seems that it
resolved itself shortly following (per a comment from Andres to that
effect), so, afaik, things are working properly.

If anyone has issues with git.p.o, please let us know, but hopefully all
is good now.

Thanks!

Stephen


signature.asc
Description: PGP signature

Re: JIT compiling with LLVM v12.2

[Andres Freund]

On 2018-03-20 19:29:55 -0700, Andres Freund wrote:
> Hi,
> 
> On 2018-03-21 15:22:08 +1300, Thomas Munro wrote:
> > Somehow your configure test correctly concludes that my $CC (clang
> > 4.0) doesn't support -fexcess-precision=standard but that my $CXX
> > (clang++ 4.0) does, despite producing a nearly identical warning:
> 
> Yea, there was a copy & pasto (s/ac_c_werror_flag/ac_cxx_werror_flag/),
> sorry. If you rebase onto the committed version, it should work?  I'll
> push a version rebased of the jit tree soon.

Well, or not. Seems git.pg.o is down atm:

debug1: Next authentication method: publickey
debug1: Offering public key: RSA 
SHA256:cMbSa8YBm8AgaIeMtCSFvvPDrrrdadCxzQaFiWFe+7c /home/andres/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277


Will try tomorrow.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-21 15:22:08 +1300, Thomas Munro wrote:
> Somehow your configure test correctly concludes that my $CC (clang
> 4.0) doesn't support -fexcess-precision=standard but that my $CXX
> (clang++ 4.0) does, despite producing a nearly identical warning:

Yea, there was a copy & pasto (s/ac_c_werror_flag/ac_cxx_werror_flag/),
sorry. If you rebase onto the committed version, it should work?  I'll
push a version rebased of the jit tree soon.

Greetings,

Andres Freund

Re: JIT compiling with LLVM v12.2

[Thomas Munro]

On Wed, Mar 21, 2018 at 1:50 PM, Andres Freund  wrote:
> On 2018-03-20 03:14:55 -0700, Andres Freund wrote:
>> My current plan is to push the first few commits relatively soon, give
>> the BF a few cycles to shake out. Set up a few BF animals with each
>> supported LLVM version. Then continue mergin.
>
> I've done that.  I'll set up a number of BF animals as soon as I've got
> the buildfarm secrets for them.

Somehow your configure test correctly concludes that my $CC (clang
4.0) doesn't support -fexcess-precision=standard but that my $CXX
(clang++ 4.0) does, despite producing a nearly identical warning:

configure:5489: checking whether ccache cc supports
-fexcess-precision=standard, for CFLAGS
configure:5511: ccache cc -c -Wall -Wmissing-prototypes
-Wpointer-arith -Wdeclaration-after-statement -Wendif-labels
-Wmissing-format-attribute -Wformat-security -fno-strict-aliasing
-fwrapv -fexcess-precision=standard  conftest.c >&5
cc: warning: optimization flag '-fexcess-precision=standard' is not
supported [-Wignored-optimization-argument]
configure:5511: $? = 0
configure: failed program was:
...
configure:5521: result: no

configure:5528: checking whether ccache c++ supports
-fexcess-precision=standard, for CXXFLAGS
configure:5556: ccache c++ -c -Wall -Wpointer-arith
-fno-strict-aliasing -fwrapv -fexcess-precision=standard  conftest.cpp
>&5
c++: warning: optimization flag '-fexcess-precision=standard' is not
supported [-Wignored-optimization-argument]
configure:5556: $? = 0
configure:5572: result: yes

So it goes into my $CXXFLAGS and then I get the same warning when
compiling the three .cpp files in the tree.

GCC also doesn't like that in C++ mode, but it seems to report an
error (rather than a warning) so with g++ as your $CXX configure sees
$? = 1 and draws the correct conclusion.

$ gcc -fexcess-precision=standard -c test.c
$ g++ -fexcess-precision=standard -c test.cpp
cc1plus: sorry, unimplemented: -fexcess-precision=standard for C++

-- 
Thomas Munro
http://www.enterprisedb.com

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

On 2018-03-20 03:14:55 -0700, Andres Freund wrote:
> My current plan is to push the first few commits relatively soon, give
> the BF a few cycles to shake out. Set up a few BF animals with each
> supported LLVM version. Then continue mergin.

I've done that.  I'll set up a number of BF animals as soon as I've got
the buildfarm secrets for them.

- Andres

Re: JIT compiling with LLVM v12.2

[Andres Freund]

Hi,

I've pushed a revised and rebased version of my JIT patchset.
The git tree is at
  https://git.postgresql.org/git/users/andresfreund/postgres.git
in the jit branch
  
https://git.postgresql.org/gitweb/?p=users/andresfreund/postgres.git;a=shortlog;h=refs/heads/jit

There's a lot of tiny changes in here:
- doc proofreading, addition of --with-llvm docs
- comments
- pgindent (jit files, not whole tree)
- syncing of order between compiled and interpreted expressions in case
  statement
- line-by-line review of expression compliation
- fix of a small memory leak (missing pfree of the JIT context struct
  itself)
- slight simplification of JIT resowner integration (no need to
  re-associate with parent)

My current plan is to push the first few commits relatively soon, give
the BF a few cycles to shake out. Set up a few BF animals with each
supported LLVM version. Then continue mergin.

Greetings,

Andres Freund