Re: Setting rpath on llvmjit.so?

[Michael Paquier]

On Thu, Apr 26, 2018 at 01:16:09PM -0600, Jason Petersen wrote:
> So the tooling around people using it is there and it’s used by some
> very large and mature projects covering a variety of domains: I don’t
> know if it’s “poorly supported” (the documentation leaves something to
> be desired), but if CMake has trouble down the line a lot of very
> important projects will be in trouble. It seems like a pretty safe bet
> given the sheer inertia of the list above. 

Yes, those would be arguments pushing in favor of cmake.  Yuriy has
mentioned me a couple of times that when he worked on the cmake
integration on Windows he used heavily VS because that was quite
friendly on Windows and their was a good integration work.
--
Michael


signature.asc
Description: PGP signature

Re: Setting rpath on llvmjit.so?

[Jason Petersen]

> On Apr 18, 2018, at 8:57 AM, Tom Lane  wrote:
> 
> I'm wondering whether that will result in expending a lot of effort to move 
> from a poorly-supported build system to a different poorly-supported build 
> system.


I’m not sure whether the former is autoconf/make or cmake, but count me as 
another vote for cmake over meson. CMake has direct support from Microsoft 
(Visual Studio 2017 can auto-ingest CMake-based files as soon as they’re 
checked out) and JetBrains CLion and is in use by LLVM, KDE, Qt, Blender, 
libpng, cURL, LAPACK, MySQL/MariaDB, OpenCV, SDL, the Dolphin Gamecube emulator 
etc.

So the tooling around people using it is there and it’s used by some very large 
and mature projects covering a variety of domains: I don’t know if it’s “poorly 
supported” (the documentation leaves something to be desired), but if CMake has 
trouble down the line a lot of very important projects will be in trouble. It 
seems like a pretty safe bet given the sheer inertia of the list above.

--
Jason Petersen
Software Engineer | Citus Data
303.736.9255
ja...@citusdata.com

Re: Setting rpath on llvmjit.so?

[Yuriy Zhuravlev]

I talked about autoconf build system, /tools/msvc it's extra home build
system.

On Thu, 19 Apr 2018, 00:58 Andres Freund,  wrote:

> On 2018-04-18 15:54:59 +, Yuriy Zhuravlev wrote:
> > Current autoconf system not working on Windows at all, what we talk
> about?
>
> We generate windows project files. See src/tools/msvc/
>
> - Andres
>

Re: Setting rpath on llvmjit.so?

[Yuriy Zhuravlev]

>
> I also politely decline the offer to be forced to use XCode on mac.


Why? We supporting MSVC and not nmake, what difference with xcode? Also,
it's just extra benefit from cmake/meson.

Re: Setting rpath on llvmjit.so?

[Andres Freund]

On 2018-04-18 15:54:59 +, Yuriy Zhuravlev wrote:
> Current autoconf system not working on Windows at all, what we talk about?

We generate windows project files. See src/tools/msvc/

- Andres

Re: Setting rpath on llvmjit.so?

[Yuriy Zhuravlev]

Current autoconf system not working on Windows at all, what we talk about?

On Wed, 18 Apr 2018, 23:57 Robert Haas,  wrote:

> On Tue, Apr 17, 2018 at 4:13 PM, Andres Freund  wrote:
> > I'd not advocate for this solely based on the age of autoconf. But the
> > separate windows buildsystem which makes it very hard to build
> > extensions separately is a good reason on its own. As is the fact that
> > recursive make as we're using it has significant issues. Both of those
> > would be avoided by using cmake or such.
>
> How much of the pain building extensions on Windows comes from our
> build system and how much of it comes from the fact that it's Windows?
>  My guess would have been that the latter is by far the bigger issue,
> but maybe I'm wrong.
>
> There is really nothing keeping us from removing or reducing the use
> of recursive make without switching to cmake.  And it's probably not
> even that much work.
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
>

Re: Setting rpath on llvmjit.so?

[Robert Haas]

On Tue, Apr 17, 2018 at 4:13 PM, Andres Freund  wrote:
> I'd not advocate for this solely based on the age of autoconf. But the
> separate windows buildsystem which makes it very hard to build
> extensions separately is a good reason on its own. As is the fact that
> recursive make as we're using it has significant issues. Both of those
> would be avoided by using cmake or such.

How much of the pain building extensions on Windows comes from our
build system and how much of it comes from the fact that it's Windows?
 My guess would have been that the latter is by far the bigger issue,
but maybe I'm wrong.

There is really nothing keeping us from removing or reducing the use
of recursive make without switching to cmake.  And it's probably not
even that much work.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Setting rpath on llvmjit.so?

[Peter Eisentraut]

On 4/17/18 16:14, Andres Freund wrote:
> I still think cmake is the least unreasonable path going forward.

I would rather try to make Meson work and if needed add features back
into Meson.

-- 
Peter Eisentraut  http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Setting rpath on llvmjit.so?

[Yuriy Zhuravlev]

>  Well, before it does everything, there's little point in reviewing
>  whether it's mergeable or not.

For this significant case, it's not working as you expect.
First, Postgres community should find consensus about migration to CMake
(or alternative). Now, this project too huge to work on it without sure in
importance.
Second, a few main contributors should start helping to fix bugs and deep
into architecture. It's very important because build system very tightly
bound with all source code and one man can't know all rarely cases.
Moreover, it's needed to understand some restriction CMake (and another
project generators) and find consensus in the community about it. For
example why we can't build contrib modules independent on main postgres
with CMake.
Also, you should understand what you can't keep the whole feature set and
behaviors in the new build system, postgres too big for this now (probably
you will have many new features but you will lose too). It's why main
contributors should know new build system to find solutions and consensus
for decisions.

We can open a new thread for discussion about the first question, the
second question should be later.

Re: Setting rpath on llvmjit.so?

[Andres Freund]

Hi,

On 2018-04-16 11:26:14 +0900, Yuriy Zhuravlev wrote:
> My cmake branch still working and I supporting stable postgres releases:
> https://github.com/stalkerg/postgres_cmake

> Anyway, my branch exist, cmake working fine maybe without minor features
> like generating documentation.

> We can start conversation about it again, I am open for this.

Well, before it does everything, there's little point in reviewing
whether it's mergeable or not.

I still think cmake is the least unreasonable path going forward.

Greetings,

Andres Freund

Re: Setting rpath on llvmjit.so?

[Andres Freund]

On 2018-04-17 15:56:14 -0400, Robert Haas wrote:
> On Sat, Apr 14, 2018 at 5:13 PM, Andres Freund  wrote:
> > The half unmaintainedness of autoconf (no release in five years counts
> > as that imo), sure makes it look like a good idea to move on to cmake or
> > such at some point...
> 
> I don't necessarily see that as a reason to move.  If we're hitting
> bugs in autoconf that are causing problems, and we can't get them
> fixed in the upstream, that's a reason.  But switching build systems
> won't be frictionless, so I'm not in favor of it unless there's a
> clear payoff.

I'd not advocate for this solely based on the age of autoconf. But the
separate windows buildsystem which makes it very hard to build
extensions separately is a good reason on its own. As is the fact that
recursive make as we're using it has significant issues. Both of those
would be avoided by using cmake or such.

Greetings,

Andres Freund

Re: Setting rpath on llvmjit.so?

[Tom Lane]

Robert Haas  writes:
> ... But switching build systems
> won't be frictionless,

I think we have a nominee for Understatement of the Year.

regards, tom lane

Re: Setting rpath on llvmjit.so?

[Robert Haas]

On Sat, Apr 14, 2018 at 5:13 PM, Andres Freund  wrote:
> The half unmaintainedness of autoconf (no release in five years counts
> as that imo), sure makes it look like a good idea to move on to cmake or
> such at some point...

I don't necessarily see that as a reason to move.  If we're hitting
bugs in autoconf that are causing problems, and we can't get them
fixed in the upstream, that's a reason.  But switching build systems
won't be frictionless, so I'm not in favor of it unless there's a
clear payoff.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Setting rpath on llvmjit.so?

[Yuriy Zhuravlev]

2018-04-15 6:13 GMT+09:00 Andres Freund :

> On 2018-04-14 17:10:21 -0400, Tom Lane wrote:
> > Andres Freund  writes:
> > > On April 14, 2018 1:56:08 PM PDT, Tom Lane  wrote:
> > >> The project policy is to use exactly the GNU distribution of autoconf.
> >
> > > Fwiw, I see one copyright year related diff with unmodified upstream
> autoconf.
> >
> > Really?  Where did you get autoconf from?  My archived copy of
> > autoconf-2.69.tar.gz, fetched 2013-11-15, has internal file
> > dates of (mostly) 2012-04-24.  That matches the file dates at
> > https://ftp.gnu.org/gnu/autoconf/
> > so I've not actually pulled down fresh bits to compare, but ...
>
> The v2.69 git tag.  So I guess that's set somewhere in the tarball
> building. Hrmpf.
>
> The half unmaintainedness of autoconf (no release in five years counts
> as that imo), sure makes it look like a good idea to move on to cmake or
> such at some point...
>
>
My cmake branch still working and I supporting stable postgres releases:
https://github.com/stalkerg/postgres_cmake

Also, I looked into Meson but unfortunately some features still not exist
for building postgres.
Anyway, my branch exist, cmake working fine maybe without minor features
like generating documentation.
My cmake version also have minor different behaviour but it because cmake
not build your project directly, it more like project generator (same as
automake and meson).

We can start conversation about it again, I am open for this.

Regards

Re: Setting rpath on llvmjit.so?

[Andres Freund]

On 2018-04-14 17:10:21 -0400, Tom Lane wrote:
> Andres Freund  writes:
> > On April 14, 2018 1:56:08 PM PDT, Tom Lane  wrote:
> >> The project policy is to use exactly the GNU distribution of autoconf.
> 
> > Fwiw, I see one copyright year related diff with unmodified upstream 
> > autoconf.
> 
> Really?  Where did you get autoconf from?  My archived copy of
> autoconf-2.69.tar.gz, fetched 2013-11-15, has internal file
> dates of (mostly) 2012-04-24.  That matches the file dates at
> https://ftp.gnu.org/gnu/autoconf/
> so I've not actually pulled down fresh bits to compare, but ...

The v2.69 git tag.  So I guess that's set somewhere in the tarball
building. Hrmpf.

The half unmaintainedness of autoconf (no release in five years counts
as that imo), sure makes it look like a good idea to move on to cmake or
such at some point...

Greetings,

Andres Freund

Re: Setting rpath on llvmjit.so?

[Tom Lane]

Andres Freund  writes:
> On April 14, 2018 1:56:08 PM PDT, Tom Lane  wrote:
>> The project policy is to use exactly the GNU distribution of autoconf.

> Fwiw, I see one copyright year related diff with unmodified upstream autoconf.

Really?  Where did you get autoconf from?  My archived copy of
autoconf-2.69.tar.gz, fetched 2013-11-15, has internal file
dates of (mostly) 2012-04-24.  That matches the file dates at
https://ftp.gnu.org/gnu/autoconf/
so I've not actually pulled down fresh bits to compare, but ...

regards, tom lane

Re: Setting rpath on llvmjit.so?

[Andres Freund]

On April 14, 2018 1:56:08 PM PDT, Tom Lane  wrote:
>Andres Freund  writes:
>> On 2018-04-14 12:32:35 -0700, Jeff Davis wrote:
>>> PS: I also noticed that a autoconf run on master generates some
>diffs
>>> in configure.
>
>> I only see one copyright year related diff. Perhaps you're using a
>> patched distribution autoconf?
>
>The project policy is to use exactly the GNU distribution of autoconf.
>Distros tend to hack it around (especially since it's been so long
>since their last release :-(), so you more or less have to install a
>private copy to get matching results.  Fortunately it's pretty
>trivial to build/install from source.

Fwiw, I see one copyright year related diff with unmodified upstream autoconf.

Andres
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Setting rpath on llvmjit.so?

[Tom Lane]

Andres Freund  writes:
> On 2018-04-14 12:32:35 -0700, Jeff Davis wrote:
>> PS: I also noticed that a autoconf run on master generates some diffs
>> in configure.

> I only see one copyright year related diff. Perhaps you're using a
> patched distribution autoconf?

The project policy is to use exactly the GNU distribution of autoconf.
Distros tend to hack it around (especially since it's been so long
since their last release :-(), so you more or less have to install a
private copy to get matching results.  Fortunately it's pretty
trivial to build/install from source.

regards, tom lane

Re: Setting rpath on llvmjit.so?

[Andres Freund]

Hi,

On 2018-04-14 12:32:35 -0700, Jeff Davis wrote:
> Currently, we don't set rpath for llvmjit.so to include `llvm-config --
> libdir`. That means you need to keep LD_LIBRARY_PATH set properly to
> use any non-system installation of LLVM.

Or pass in the appropriate flags to configure. My configure invoking
script has: LDFLAGS='-Wl,-rpath,/home/andres/build/llvm/6/opt/install/lib' etc
in it.


> This isn't unique to LLVM (we only set rpath to the postgres $libdir),
> so I thought it might be project policy. But I think it will be worse
> for LLVM (and perhaps ICU) where people are more likely to want to run
> specific versions. Comments?

I'm not sure we want to do that - and if so to which libraries. I tend
towards thinking that if we do that, we should do it uniformly, not just
for LLVM. At the very least it'd have to respect --disable-rpath.


> PS: I also noticed that a autoconf run on master generates some diffs
> in configure.

I only see one copyright year related diff. Perhaps you're using a
patched distribution autoconf?

Greetings,

Andres Freund

Setting rpath on llvmjit.so?

[Jeff Davis]

Currently, we don't set rpath for llvmjit.so to include `llvm-config --
libdir`. That means you need to keep LD_LIBRARY_PATH set properly to
use any non-system installation of LLVM.

This isn't unique to LLVM (we only set rpath to the postgres $libdir),
so I thought it might be project policy. But I think it will be worse
for LLVM (and perhaps ICU) where people are more likely to want to run
specific versions. Comments?

Included a simple patch, but it could use some review from someone who
knows more about portable linking and our build system.

Regards,
Jeff Davis

PS: I also noticed that a autoconf run on master generates some diffs
in configure.

diff --git a/config/llvm.m4 b/config/llvm.m4
index e25ffec..582ba51 100644
--- a/config/llvm.m4
+++ b/config/llvm.m4
@@ -90,6 +90,7 @@ AC_DEFUN([PGAC_LLVM_SUPPORT],
   done
 
   LLVM_BINPATH=`$LLVM_CONFIG --bindir`
+  LLVM_LIBPATH=`$LLVM_CONFIG --libdir`
 
   # Check which functionality is present
   SAVE_CPPFLAGS="$CPPFLAGS"
@@ -104,5 +105,6 @@ AC_DEFUN([PGAC_LLVM_SUPPORT],
   AC_SUBST(LLVM_CFLAGS)
   AC_SUBST(LLVM_CXXFLAGS)
   AC_SUBST(LLVM_BINPATH)
+  AC_SUBST(LLVM_LIBPATH)
 
 ])# PGAC_LLVM_SUPPORT
diff --git a/configure b/configure
index a30d7be..b070658 100755
--- a/configure
+++ b/configure
@@ -730,6 +730,7 @@ CPP
 BITCODE_CXXFLAGS
 BITCODE_CFLAGS
 CFLAGS_VECTOR
+LLVM_LIBPATH
 LLVM_BINPATH
 LLVM_CXXFLAGS
 LLVM_CFLAGS
@@ -5005,6 +5006,7 @@ fi
   done
 
   LLVM_BINPATH=`$LLVM_CONFIG --bindir`
+  LLVM_LIBPATH=`$LLVM_CONFIG --libdir`
 
   # Check which functionality is present
   SAVE_CPPFLAGS="$CPPFLAGS"
@@ -5065,6 +5067,7 @@ _ACEOF
 
 
 
+
 fi
 
 
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index 2529202..e3cdc13 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -229,6 +229,7 @@ PTHREAD_LIBS		= @PTHREAD_LIBS@
 
 LLVM_CONFIG = @LLVM_CONFIG@
 LLVM_BINPATH = @LLVM_BINPATH@
+LLVM_LIBPATH = @LLVM_LIBPATH@
 CLANG = @CLANG@
 BITCODE_CFLAGS = @BITCODE_CFLAGS@
 BITCODE_CXXFLAGS = @BITCODE_CXXFLAGS@
diff --git a/src/backend/jit/llvm/Makefile b/src/backend/jit/llvm/Makefile
index e2db4ce..009ff47 100644
--- a/src/backend/jit/llvm/Makefile
+++ b/src/backend/jit/llvm/Makefile
@@ -39,6 +39,8 @@ OBJS=$(WIN32RES)
 OBJS += llvmjit.o llvmjit_error.o llvmjit_inline.o llvmjit_wrap.o
 # Code generation
 OBJS += llvmjit_expr.o llvmjit_deform.o
+# Add LLVM library path to rpath
+rpathdir := $(LLVM_LIBPATH):$(rpathdir)
 
 all: all-shared-lib llvmjit_types.bc