subject:"\[RFC\] building postgres with meson"

Hi,

On 2022-09-27 17:29:27 +1300, Thomas Munro wrote:
> On Tue, Sep 27, 2022 at 2:19 PM Andres Freund  wrote:
> > Subject: [PATCH v17 15/23] windows: Set UMDF_USING_NTSTATUS globally, 
> > include ntstatus.h
> 
> No Windows expertise here, but this looks reasonable.  I originally
> tried to contain UMDF_USING_NTSTATUS to small translation units for
> fear of unintended consequences, but PCH requires you not to mess with
> macros that affect the compilation of a header as seen by different
> translation units, which is an incompatible goal.  If this is passing
> on MSVC and MingGW then +1 from me.

Yes, passes both.


> You mentioned WIN32_NO_STATUS in the commit message -- a mistake?

Argh. An earlier iteration. Works on mingw, but making it work with msvc
required a lot more modifications IIRC.


> Digging out my old emails/notes... that's another way to be allowed to
> include both ntstatus.h and windows.h/etc in the same translation
> unit, but not the one we're using.  I assume it's worse because you
> have to define it and then undefine it, which sounds more antithetical
> to the PCH dream.  Admittedly UMDF_USING_NTSTATUS -- from the
> "User-Mode Driver Framework" -- is a weird thing to be getting tangled
> up with because we aren't writing a driver here, but it seems to be a
> well known and widely used alternative, and is nicer because you only
> have to define it.

It's definitely weird. But it appears to be widely used...


> > Subject: [PATCH v17 16/23] windows: adjust FD_SETSIZE via commandline define
> 
> Right, we have to fix that across translation units for the same
> reason.  But why as -D and not in win32_port.h?  I followed the
> discussion from 9acda73118 to try to find the answer to that and saw
> that Michael wanted to put it there, but wanted to minimise the blast
> radius at the time:
> 
> https://www.postgresql.org/message-id/20190826054000.GE7005%40paquier.xyz

I guess a similar consideration. I was a bit worried about the references to
FD_SETSIZE in src/backend/port/win32/socket.c. Multi kB on-stack arrays in
postmaster seem like they could cause issues.

ISTM we really ought to move away from stuff using FD_SETSIZE on windows...

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-26 Thread Thomas Munro

On Tue, Sep 27, 2022 at 2:19 PM Andres Freund  wrote:
> Subject: [PATCH v17 15/23] windows: Set UMDF_USING_NTSTATUS globally, include 
> ntstatus.h

No Windows expertise here, but this looks reasonable.  I originally
tried to contain UMDF_USING_NTSTATUS to small translation units for
fear of unintended consequences, but PCH requires you not to mess with
macros that affect the compilation of a header as seen by different
translation units, which is an incompatible goal.  If this is passing
on MSVC and MingGW then +1 from me.

You mentioned WIN32_NO_STATUS in the commit message -- a mistake?
Digging out my old emails/notes... that's another way to be allowed to
include both ntstatus.h and windows.h/etc in the same translation
unit, but not the one we're using.  I assume it's worse because you
have to define it and then undefine it, which sounds more antithetical
to the PCH dream.  Admittedly UMDF_USING_NTSTATUS -- from the
"User-Mode Driver Framework" -- is a weird thing to be getting tangled
up with because we aren't writing a driver here, but it seems to be a
well known and widely used alternative, and is nicer because you only
have to define it.

> Subject: [PATCH v17 16/23] windows: adjust FD_SETSIZE via commandline define

Right, we have to fix that across translation units for the same
reason.  But why as -D and not in win32_port.h?  I followed the
discussion from 9acda73118 to try to find the answer to that and saw
that Michael wanted to put it there, but wanted to minimise the blast
radius at the time:

https://www.postgresql.org/message-id/20190826054000.GE7005%40paquier.xyz

Re: [RFC] building postgres with meson - v13

2022-09-26 Thread Peter Geoghegan

On Mon, Sep 26, 2022 at 1:27 PM Andres Freund  wrote:
> > Some feedback:
> > * I gather that "running" as it appears in commands like "meson test
> > --setup running" refers to a particular setup named "running", that
> > you invented as part of creating a meson-ish substitute for
> > installcheck. Can "running" be renamed to something that makes it
> > obvious that it's a Postgres thing, and not a generic meson thing?
>
> Yes. The only caveat is that it makes lines longer, because it's included in
> the printed test line (there's no real requirement to have the test suite and
> the setup named the same,b ut it seems confusing not to)

Probably doesn't have to be too long. And I'm not sure of the details.
Just a small thing from my point of view.

> > * It would be nice if failed tests told me exactly which "diffs" file
> > I needed to look at, without my having to look for the message through
> > the meson log (or running with -v). Is this possible?
>
> You can use --print-errorlogs to print the log output iff a test fails. It's a
> bit painful that some tests have very verbose output :(.  I don't really see a
> way that meson can help us here, this is pretty much on "our" end.

Makes sense. Thanks.

> Hm. I'm not entirely sure what you mean here. The only thing that you can't
> change in a existing build-dir with meson configure is the compiler.

I do understand that it doesn't particularly matter to meson itself.
The point I was making was one about how I personally find it
convenient to set those things that I know will never change in
practice (because they're fundamentally things that I know that I
won't ever need to change) during "meson setup", while doing
everything else using "meson configure". I like to automate everything
using shell scripts. I will very occasionally have to run "meson
setup" via a zsh function anyway, so why not couple that process with
the process of setting "immutable for this build directory" settings?

With autoconf, I will run one of various zsh functions that run
configure in some specific way -- there are various "build types",
such as debug, release, and Valgrind. But with meson it makes sense to
split it in two -- have a generic zsh function for generic once-off
build directory setup (including even the mkdir), that also sets
generic, "immutable" settings, and a specialized zsh function that
changes things in a way that is particular to that kind of build (like
whether asserts are enabled, optimization level, and so on).

> I personally have different types of build dirs set up in parallel
> (e.g. assert, optimize, assert-32, assert-w64). I'll occasionally
> enable/disable

I know that other experienced hackers do it that way. I have found
that ccache works well enough that I don't feel the need for multiple
build directories per branch.

Perhaps I've assumed more than I should about my approach being
broadly representative. It might ultimately be easier to just have
multiple build directories per branch/source directory -- one per
"build type" per branch. That has the advantage of not requiring each
"build type" zsh function to remember to reset anything that might
have been set by one of its sibling zsh functions for some other build
type (there is no need to "reset the setting to its default"). That
approach is more like scripting autoconf/configure would be, in that
you basically never change any settings for a given build directory in
practice (you maybe invent a new kind of build type instead, or you
update the definition of an existing standard build type based on a
new requirement for that build type).

It's really handy that meson lets you quickly change one setting
against an existing build directory. I'm slightly worried that that
will allow me to shoot myself in the foot, though. Perhaps I'll change
some exotic setting in an ad-hoc way, and then forget to unset it
afterwards, leading to (say) a mysterious performance degradation for
what is supposed to be one of my known standard build types. There is
no risk of that with my autoconf/configure workflow, because I'll
rerun the relevant configure zsh function before long anyway, making
it impossible for me to accidentally keep something that I never meant
to keep.

I like being able to throw everything away and quickly rebuild "from
scratch" (in reality rebuild using ccache and a cache for configure)
due to superstition/defensive paranoia/learned helplessness. This has
always worked well enough because ccache works fairly well. I'm not
sure how useful that kind of mindset will be with meson just yet, and
if I'm just thinking about it in the wrong way, so forgive me for
rambling like this.

-- 
Peter Geoghegan

Re: [RFC] building postgres with meson - v13

Hi,

On 2022-09-26 12:47:14 -0700, Peter Geoghegan wrote:
> On Sun, Sep 25, 2022 at 5:38 PM Andres Freund  wrote:
> > # run just the main pg_regress tests against existing server
> > meson test --setup running main/regress-running
> 
> > Peter, would this address your use case?
> 
> I tried out your v16 patchset, which seems to mostly work as I'd hoped
> it would.

Thanks & cool.


> Some feedback:
> * I gather that "running" as it appears in commands like "meson test
> --setup running" refers to a particular setup named "running", that
> you invented as part of creating a meson-ish substitute for
> installcheck. Can "running" be renamed to something that makes it
> obvious that it's a Postgres thing, and not a generic meson thing?

Yes. The only caveat is that it makes lines longer, because it's included in
the printed test line (there's no real requirement to have the test suite and
the setup named the same,b ut it seems confusing not to)


> Maybe some kind of consistent naming convention would work best here.
> This setup could be "pg_against_running_server", or something along
> those lines.



> * It would be nice if failed tests told me exactly which "diffs" file
> I needed to look at, without my having to look for the message through
> the meson log (or running with -v). Is this possible?

You can use --print-errorlogs to print the log output iff a test fails. It's a
bit painful that some tests have very verbose output :(.  I don't really see a
way that meson can help us here, this is pretty much on "our" end.


> BTW the meson wiki page iencourages users to think of "meson setup"
> and "meson configure" as equivalent to autoconf configure. I get why
> you explained it like that, but that confused me at first. What I
> since figured out (which will be absurdly obvious to you) is that you
> really need to decouple the generic from the specific -- very much
> unlike autoconf. I found it useful to separate stuff that I know will
> never change for a given build directory (such as the prefix install
> path) from other things that are variable configuration settings
> (things like the optimization level used by GCC). I now have a
> scripted way of running "meson setup" for the former stuff (which is
> generic), and a scripted way of running "meson configure" for the
> latter set of stuff (with variations for "standard" release and debug
> builds, building Valgrind, etc).

Hm. I'm not entirely sure what you mean here. The only thing that you can't
change in a existing build-dir with meson configure is the compiler.

I personally have different types of build dirs set up in parallel
(e.g. assert, optimize, assert-32, assert-w64). I'll occasionally
enable/disable

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-26 Thread Peter Geoghegan

On Sun, Sep 25, 2022 at 5:38 PM Andres Freund  wrote:
> # run just the main pg_regress tests against existing server
> meson test --setup running main/regress-running

> Peter, would this address your use case?

I tried out your v16 patchset, which seems to mostly work as I'd hoped
it would. Some feedback:

* I gather that "running" as it appears in commands like "meson test
--setup running" refers to a particular setup named "running", that
you invented as part of creating a meson-ish substitute for
installcheck. Can "running" be renamed to something that makes it
obvious that it's a Postgres thing, and not a generic meson thing?

Maybe some kind of consistent naming convention would work best here.
This setup could be "pg_against_running_server", or something along
those lines.

* It would be nice if failed tests told me exactly which "diffs" file
I needed to look at, without my having to look for the message through
the meson log (or running with -v). Is this possible?

To be fair I should probably just be running -v when I run tests
against an existing running server, anyway -- so maybe I'm asking for
the wrong thing. It would at least be slightly better if I always got
to see a path to a .diffs file for failed tests, even without -v. But
it's just a "nice to have" thing -- it's not worth going out of your
way to make it work like that

* Just FYI, there are considerations about the libpq that we link to
here (actually this isn't particularly related to the new installcheck
work, but thought I'd mention it in passing).

I'm using Debian Unstable here. Like Nathan, I found that I needed a
custom -Dextra_lib_dirs just so that binaries would link against the
installation's own libpq, rather than the system libpq. This is
important-ish because linking to the wrong libpq means that I get an
error about not being able to connect via trust authentication to a
unix socket from the directory /var/run/postgresql -- confusion over
where to look for sockets visibly breaks many things.

The workaround that I have is fine, but this still seems like
something that should "just work". I believe that there is a pending
patch for this already, so enough said here.

> I think it'd make sense to add a few toplevel targets to run tests in certain
> ways, but I've not done that here.

I usually run "installcheck-world" (not just installcheck) when I want
to do a generic smoke test with Vaglrind. Sometimes that will fail
relatively early for very silly reasons, for example because I don't
have exactly the expected plan in some harmless way (I try to account
for this by running Valgrind in a shellscript that tries to match
"make check", but that doesn't always work). It is nice that I won't
have to worry about such minor issues derailing everything for a long
running and unsupervised Valgrind test. (Maybe I could have worked
around this before now, but I guess I never tried.)

More generally, I think that we should be encouraging users to think
of the tests as something that you can run in any order. People should
be encouraged to think in terms of the meson abstractions, such as
test setups.

I found that "meson test --setup running --list" will show me what
tests I'll be running if I want to do "installcheck" style testing,
without having to run any tests at all -- another small but important
improvement. This seems worth drawing attention to on the meson Wiki
page as a non-obvious improvement over "installcheck". I might even
find it useful to hard-code some of these tests in a shellscript, that
runs only a subset of "--setup running" tests that happen to be
interesting for Valgrind testing right now.

BTW the meson wiki page iencourages users to think of "meson setup"
and "meson configure" as equivalent to autoconf configure. I get why
you explained it like that, but that confused me at first. What I
since figured out (which will be absurdly obvious to you) is that you
really need to decouple the generic from the specific -- very much
unlike autoconf. I found it useful to separate stuff that I know will
never change for a given build directory (such as the prefix install
path) from other things that are variable configuration settings
(things like the optimization level used by GCC). I now have a
scripted way of running "meson setup" for the former stuff (which is
generic), and a scripted way of running "meson configure" for the
latter set of stuff (with variations for "standard" release and debug
builds, building Valgrind, etc).

-- 
Peter Geoghegan

Re: [RFC] building postgres with meson - v13

Hi,

On 2022-09-26 09:35:16 -0700, Andres Freund wrote:
> > 9c00d355d0e9 meson: Add PGXS compatibility
> >
> > This looks like a reasonable direction to me.  How complete is it?  It
> > says it works for some extensions but not others.  How do we define
> > the target line here?
>
> Yea, those are good questions.
>
>
> > How complete is it?
>
> It's a bit hard to know. I think the most important stuff is there. But
> there's no clear "API" around pgxs. E.g. we don't (yet?) have an exactly
> equivalent definition of 'host', because that's very config.guess specific.
>
> There's lots of shortcuts - e.g. with meson we don't need an equivalent to
> PGAC_CHECK_STRIP, so we need to make up something for Makefile.global.
>
> Noah suggested using $(error something), but that only works if $variable is
> only used in recursively expanded variables - the errors end up confusing.

Looking through a few of the not-nicely-replaced things, I think we can
simplify at least some away:

- RANLIB: most platforms use AROPT = crs, making ranlib unnecessary. {free,
  net, open}bsd don't currently, but all support it from what I know

- with_gnu_ld: this is only used on solaris, to set export_dynamic = -Wl,-E
  when using a gnu ld. How about moving this to configure instead, and just
  checking if -Wl,-E links?

- FLEXFLAGS: As a configure input this is afaict unused and undocumented - and
  it's not clear why it'd be useful? Not that an empty replacement is a
  meaningful effort


I'm not sure what to do about:
- autodepend - I'm inclined to set it to true when using a gcc like
  compiler. I think extension authors won't be happy if suddenly their
  extensions don't rebuild reliably anymore. An --enable-depend like
  setting doesn't make sense for meson, so we don't have anything to source it
  from.
- {LDAP,UUID,ICU}_{LIBS,CFLAGS} - might some extension need them?


For some others I think it's ok to not have replacement. Would be good for
somebody to check my thinking though:

- LIBOBJS, PG_CRC32C_OBJS, TAS: Not needed because we don't build
  the server / PLs with the generated makefile
- ZIC: only needed to build tzdata as part of server build
- MSGFMT et al: translation doesn't appear to be supported by pgxs, correct?
- XMLLINT et al: docs don't seem to be supported by pgxs
- GENHTML et al: supporting coverage for pgxs-in-meson build doesn't seem worth 
it
- WINDRES: I don't think extensions are bothering to generate rc files on 
windows


I'll include an updated pgxs-compat patch in the next post of the series (in a
few hours).

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

Hi,

On 2022-09-26 15:18:29 +0700, John Naylor wrote:
> On Wed, Sep 21, 2022 at 7:11 AM Andres Freund  wrote:
> >
> > I added installation instructions for meson for a bunch of platforms, but
> 
> A couple more things for the wiki:
> 
> 1) /opt/homebrew/ seems to be an "Apple silicon" path?

Yea, it's /usr/local on x86-64, based on what was required to make macos CI
work. I updated the wiki page, half-blindly - it'd be nice if you could
confirm that that works?

I needed something like the below to get (nearly?) all dependencies working:

brewpath="/usr/local"
PKG_CONFIG_PATH="${brewpath}/lib/pkgconfig:${PKG_CONFIG_PATH}"

for pkg in icu4c krb5 openldap openssl zstd ; do
  pkgpath="${brewpath}/opt/${pkg}"
  PKG_CONFIG_PATH="${pkgpath}/lib/pkgconfig:${PKG_CONFIG_PATH}"
  PATH="${pkgpath}/bin:${pkgpath}/sbin:$PATH"
done

export PKG_CONFIG_PATH PATH

meson setup \
  --buildtype=debug \
  -Dextra_include_dirs=${brewpath}/include \
  -Dextra_lib_dirs=${brewpath}/lib \
  -Dcassert=true \
  -Dssl=openssl -Duuid=e2fs -Ddtrace=auto \
  -DPG_TEST_EXTRA="$PG_TEST_EXTRA" \
  build

the per-package stuff is needed because some libraries aren't installed into
/usr/local (or /opt/homebrew), but only in a subdirectory within that.

> Either way it doesn't exist on this machine. I was able to get a working
> build with
> 
> /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig

Hm - what did you need this path for - I don't think that should be needed.

> (My homebrew install doesn't seem to have anything relevant for
> extra_include_dirs or extra_lib_dirs.)

I think libintl.h / libintl.dylib are only in there. With meson that's the
only need for extra_include_dirs / extra_lib_dirs I found on arm apple.

> 2) Also, "ninja -v install" has the same line count as "ninja install" --
> are there versions that do something different?

Yea, that looks like a copy-and-pasto (not even from me :)). Think I fixed it.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

Hi,

On 2022-09-26 15:01:56 +0200, Peter Eisentraut wrote:
> Here is some review of the remaining ones (might not match exactly what you
> attached, I was working off your branch):

Thanks, and makes sense.

> 9f789350a7a7 meson: ci: wip: move compilerwarnings task to meson
>
> This sounds reasonable to me in principle, but I haven't followed the
> cirrus stuff too closely, and it doesn't say why it's "wip".  Perhaps
> others could take a closer look.

It's mostly WIP because it doesn't yet convert all the checks, specifically
headerscheck/cpluspluscheck isn't converted yet.

> ccf20a68f874 meson: ci: Add additional CI coverage
>
> IIUC, this is just for testing your branch, not meant for master?

Yes. I think we might want to add openbsd / netbsd at some point, but that'll
be a separate thread. Until then it just catches a bunch of mistakes more
easily.

> 02d84c21b227 meson: prereq: win: remove date from version number in
> win32ver.rc
>
> do it

The newest version has evolved a bit, changing Project.pm as well.

> 5c42b3e7812e meson: WIP: Add some of the windows resource files
>
> What is the thinking on this now?  What does this change over the
> current state?

The newest commit has a lot more rc files added and has this summary:

meson: Add windows resource files

The generated resource files aren't exactly the same ones as the old
buildsystems generate. Previously "InternalName" and "OriginalFileName" were
mostly wrong / not set (despite being required), but that was hard to fix in
at least the make build. Additionally, the meson build falls back to a
"auto-generated" description when not set, and doesn't set it in a few 
cases -
unlikely that anybody looks at these descriptions in detail.

The only thing missing rc files is the various ecpg libraries. The issue is
that we shouldn't add resource file to static libraries, so we need to split
the definitions. I'll go and do that next.

> 9bc60bccfd10 meson: Add support for relative rpaths, fixing tests on MacOS
> w/ SIP
>
> I suggest a separate thread and/or CF entry for this.  There have been
> various attempts to deal with SIP before, with varying results.  This
> is not part of the meson transition as such.

I think I might need to split this one more time. We don't add all the rpaths
we add with autoconf before this commit, even not on macOS, which is not
great... Nor do we have a --disable-rpath equivalent yet - I suspect we'll
need that.

https://postgr.es/m/2022093729.GA721620%40nathanxps13

> 9f5be26c1215 meson: Add docs for building with meson
>
> I do like the overall layout of this.
>
> The "Supported Platforms" section should be moved back to near the end
> of the chapter.  I don't see a reason to move it forward, at least
> none that is related to the meson issue.
>
> The changes to the "Getting the Source" section are also not
> appropriate for this patch.

We don't really support building from a tarball with meson yet (you'd need to
confiure, maintainer-clean, configure meson), so it does make some sense...

> 9c00d355d0e9 meson: Add PGXS compatibility
>
> This looks like a reasonable direction to me.  How complete is it?  It
> says it works for some extensions but not others.  How do we define
> the target line here?

Yea, those are good questions.

> How complete is it?

It's a bit hard to know. I think the most important stuff is there. But
there's no clear "API" around pgxs. E.g. we don't (yet?) have an exactly
equivalent definition of 'host', because that's very config.guess specific.

There's lots of shortcuts - e.g. with meson we don't need an equivalent to
PGAC_CHECK_STRIP, so we need to make up something for Makefile.global.

Noah suggested using $(error something), but that only works if $variable is
only used in recursively expanded variables - the errors end up confusing.

> It says it works for some extensions but not others.

I think that's slightly outdated - IIRC it was about pgbouncer, but after a
fix the remaining failure is shared between autoconf and meson builds.

> 3fd5e13dcad3 meson: Add postgresql-extension.pc for building extension
> libraries
>
> Separate thread for this as well.  This is good and important, but we
> must also add it to the make build.

Makes sense.

> eb40f6e53104 meson: Add support for building with precompiled headers
>
> Any reason not to enable this by default?  The benefits on non-Windows
> appear to be less dramatic, but they are not zero.  Might be better to
> enable it consistently so that for example any breakage is easier
> caught.

There's no real reason not to - the wins are small on linux, so introducing
PCH didn't seem necessary. I'm also not sure how well pch works across random
compiler versions - it's so crucial on windows that it seems like a more well
worn path there.

linux, gcc 12:

b_pch=false:
real0m16.233s
user6m40.375s
sys 0m48.953s

b_pch=true:
real0m15.983s
user6m20.357s
sys 0m49.967s

freebsd VM,

Re: [RFC] building postgres with meson - v13

Hi,

On 2022-09-26 10:41:01 +0200, Alvaro Herrera wrote:
> On 2022-Sep-25, Andres Freund wrote:
> 
> > From 3eb0ca196084da314d94d1e51c7b775012a4773c Mon Sep 17 00:00:00 2001
> > From: Andres Freund 
> > Date: Wed, 21 Sep 2022 11:03:07 -0700
> > Subject: [PATCH v16 04/16] meson: Add windows resource files
> 
> > diff --git a/src/backend/jit/llvm/meson.build 
> > b/src/backend/jit/llvm/meson.build
> > index de2e624ab58..5fb63768358 100644
> > --- a/src/backend/jit/llvm/meson.build
> > +++ b/src/backend/jit/llvm/meson.build
> > @@ -20,6 +20,12 @@ llvmjit_sources += files(
> >'llvmjit_expr.c',
> >  )
> >  
> > +if host_system == 'windows'
> > +  llvmjit_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
> > +'--NAME', 'llvmjit',
> > +'--FILEDESC', 'llvmjit - JIT using LLVM',])
> > +endif
> 
> This is tediously imperative.  Isn't there a more declarative way to
> have it?

I tried to come up with something better, without success. I think it's
acceptable, even if not great.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-26 Thread Peter Eisentraut


On 24.09.22 20:09, Andres Freund wrote:

On 2022-09-24 13:52:29 -0400, Tom Lane wrote:

... btw, shouldn't the CF entry [1] get closed now?


Unfortunately not - there's quite a few followup patches that haven't been
[fully] reviewed and thus not applied yet.


Here is some review of the remaining ones (might not match exactly what 
you attached, I was working off your branch):



9f789350a7a7 meson: ci: wip: move compilerwarnings task to meson

This sounds reasonable to me in principle, but I haven't followed the
cirrus stuff too closely, and it doesn't say why it's "wip".  Perhaps
others could take a closer look.


ccf20a68f874 meson: ci: Add additional CI coverage

IIUC, this is just for testing your branch, not meant for master?


02d84c21b227 meson: prereq: win: remove date from version number in 
win32ver.rc


do it


5c42b3e7812e meson: WIP: Add some of the windows resource files

What is the thinking on this now?  What does this change over the
current state?


9bc60bccfd10 meson: Add support for relative rpaths, fixing tests on 
MacOS w/ SIP


I suggest a separate thread and/or CF entry for this.  There have been
various attempts to deal with SIP before, with varying results.  This
is not part of the meson transition as such.


9f5be26c1215 meson: Add docs for building with meson

I do like the overall layout of this.

The "Supported Platforms" section should be moved back to near the end
of the chapter.  I don't see a reason to move it forward, at least
none that is related to the meson issue.

The changes to the "Getting the Source" section are also not
appropriate for this patch.

In the section "Building and Installation with meson":

- Remove the "git clone" stuff.

- The "Running tests" section should be moved to Chapter 33. Regression 
Tests.


Some copy-editing will probably be suitable, but I haven't focused on
that yet.


9c00d355d0e9 meson: Add PGXS compatibility

This looks like a reasonable direction to me.  How complete is it?  It
says it works for some extensions but not others.  How do we define
the target line here?


3fd5e13dcad3 meson: Add postgresql-extension.pc for building extension 
libraries


Separate thread for this as well.  This is good and important, but we
must also add it to the make build.


4b5bfa1c19aa meson: Add LLVM bitcode emission

still in progress


eb40f6e53104 meson: Add support for building with precompiled headers

Any reason not to enable this by default?  The benefits on non-Windows
appear to be less dramatic, but they are not zero.  Might be better to
enable it consistently so that for example any breakage is easier
caught.


377bfdea6042 meson: Add xmllint/xsltproc wrapper script to handle 
dependencies automatically


Is this part of the initial transition, required for correctness, or
is it an optional optimization?  Could use more explanation.  Maybe
move to separate thread also?

Re: [RFC] building postgres with meson - v13

2022-09-26 Thread Alvaro Herrera

On 2022-Sep-25, Andres Freund wrote:

> From 3eb0ca196084da314d94d1e51c7b775012a4773c Mon Sep 17 00:00:00 2001
> From: Andres Freund 
> Date: Wed, 21 Sep 2022 11:03:07 -0700
> Subject: [PATCH v16 04/16] meson: Add windows resource files

> diff --git a/src/backend/jit/llvm/meson.build 
> b/src/backend/jit/llvm/meson.build
> index de2e624ab58..5fb63768358 100644
> --- a/src/backend/jit/llvm/meson.build
> +++ b/src/backend/jit/llvm/meson.build
> @@ -20,6 +20,12 @@ llvmjit_sources += files(
>'llvmjit_expr.c',
>  )
>  
> +if host_system == 'windows'
> +  llvmjit_sources += rc_lib_gen.process(win32ver_rc, extra_args: [
> +'--NAME', 'llvmjit',
> +'--FILEDESC', 'llvmjit - JIT using LLVM',])
> +endif

This is tediously imperative.  Isn't there a more declarative way to
have it?

-- 
Álvaro HerreraBreisgau, Deutschland  —  https://www.EnterpriseDB.com/
"Pensar que el espectro que vemos es ilusorio no lo despoja de espanto,
sólo le suma el nuevo terror de la locura" (Perelandra, C.S. Lewis)

Re: [RFC] building postgres with meson - v13

2022-09-26 Thread John Naylor

On Wed, Sep 21, 2022 at 7:11 AM Andres Freund  wrote:
>
> I added installation instructions for meson for a bunch of platforms, but

A couple more things for the wiki:

1) /opt/homebrew/ seems to be an "Apple silicon" path? Either way it
doesn't exist on this machine. I was able to get a working build with

/usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig

(My homebrew install doesn't seem to have anything relevant for
extra_include_dirs or extra_lib_dirs.)

2) Also, "ninja -v install" has the same line count as "ninja install" --
are there versions that do something different?

--
John Naylor
EDB: http://www.enterprisedb.com

RE: [RFC] building postgres with meson - v13

2022-09-26 Thread wangw.f...@fujitsu.com

On Mon, Sep 26, 2022 at 14:47 PM Andres Freund  wrote:
> Hi,
> 
> On 2022-09-26 06:24:42 +, wangw.f...@fujitsu.com wrote:
> > I tried to use meson and ninja and they are really efficient.
> > But when I tried to specify "c_args", it did not take effect.
> 
> They should take effect, but won't be shown in the summary section
> currently. That currently only shows the flags chosen by the configure step,
> rather than user specified ones.
> 
> 
> > After I made the below modifications, the specified "c_args" took effect.
> > ```
> > @@ -2439,6 +2439,10 @@ endif
> >
> >  # Set up compiler / linker arguments to be used everywhere, individual
> targets
> >  # can add further args directly, or indirectly via dependencies
> > +
> > +tmp_c_args = get_option('c_args')
> > +cflags += tmp_c_args
> > +
> >  add_project_arguments(cflags, language: ['c'])
> >  add_project_arguments(cppflags, language: ['c'])
> >  add_project_arguments(cflags_warn, language: ['c'])
> > ```
> 
> That'll likely end up with the same cflags added multiple times. You should
> see them when building with ninja -v.

Thanks for sharing the information.
I saw the user specified CFLAG when building with `ninja -v`.

But, after installing PG with command `ninja -v install`, pg_config does not
show the user specified CFLAG. Should we print this information there?

> How about adding c_args to the summary, in a separate line? I think that'd
> clarify what's happening?

Yes, I think it might be better.

Regards,
Wang wei

Re: [RFC] building postgres with meson - v13

Hi,

On 2022-09-26 06:24:42 +, wangw.f...@fujitsu.com wrote:
> I tried to use meson and ninja and they are really efficient.
> But when I tried to specify "c_args", it did not take effect.

They should take effect, but won't be shown in the summary section
currently. That currently only shows the flags chosen by the configure step,
rather than user specified ones.

> After I made the below modifications, the specified "c_args" took effect.
> ```
> @@ -2439,6 +2439,10 @@ endif
> 
>  # Set up compiler / linker arguments to be used everywhere, individual 
> targets
>  # can add further args directly, or indirectly via dependencies
> +
> +tmp_c_args = get_option('c_args')
> +cflags += tmp_c_args
> +
>  add_project_arguments(cflags, language: ['c'])
>  add_project_arguments(cppflags, language: ['c'])
>  add_project_arguments(cflags_warn, language: ['c'])
> ```

That'll likely end up with the same cflags added multiple times. You should
see them when building with ninja -v.

How about adding c_args to the summary, in a separate line? I think that'd
clarify what's happening?

Greetings,

Andres Freund

RE: [RFC] building postgres with meson - v13

2022-09-26 Thread wangw.f...@fujitsu.com

Hi,

I tried to use meson and ninja and they are really efficient.
But when I tried to specify "c_args", it did not take effect.

Attached my steps:
[In the HEAD (7d708093b7)]
$ meson setup build --prefix /home/wangw/install/parallel_apply/ -Dcassert=true 
-Dtap_tests=enabled -Dicu=enabled -Dc_args='-fno-omit-frame-pointer'

Log:
..
  Compiler Flags
CPP FLAGS  : -D_GNU_SOURCE
C FLAGS, functional: -fno-strict-aliasing -fwrapv 
-fexcess-precision=standard
C FLAGS, warnings  : -Wmissing-prototypes -Wpointer-arith -Werror=vla 
-Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 
-Wcast-function-type -Wformat-security -Wdeclaration-after-statement 
-Wno-format-truncation -Wno-stringop-truncation
..

After I made the below modifications, the specified "c_args" took effect.
```
@@ -2439,6 +2439,10 @@ endif

 # Set up compiler / linker arguments to be used everywhere, individual targets
 # can add further args directly, or indirectly via dependencies
+
+tmp_c_args = get_option('c_args')
+cflags += tmp_c_args
+
 add_project_arguments(cflags, language: ['c'])
 add_project_arguments(cppflags, language: ['c'])
 add_project_arguments(cflags_warn, language: ['c'])
```

I might missed something. Just to confirm is there another way to add CFLAG ?

Regards,
Wang wei

Re: [RFC] building postgres with meson - v13

2022-09-25 Thread Andres Freund

Hi,

On 2022-09-24 17:33:49 -0700, Peter Geoghegan wrote:
> On Sat, Sep 24, 2022 at 5:13 PM Andres Freund  wrote:
> > > One more question about this, that wasn't covered by the Wiki page: is
> > > there some equivalent to "make installcheck" with meson builds?
> >
> > Not yet. Nothing impossible, just not done yet. Partially because 
> > installcheck
> > is so poorly defined (run against an already running server for pg_regress 
> > vs
> > using "system" installed binaries for tap tests).
> 
> Got it. I can work around that by just having an old autoconf-based
> vpath build directory. I'll need to do this when I run Valgrind.
> 
> My workaround would be annoying if I needed to run "installcheck"
> anywhere near as frequently as I run "make check-world". But that
> isn't the case. meson delivers a significant improvement in the metric
> that really matters to me, so I can't really complain.

My gut feeling is that we should use this opportunity to split 'installcheck'
into two. "test a running server" and "test installed binaries".

I think the cleanest way to do this with meson would be to utilize meson
tests's "setups".
$ meson test --setup 'running-server'
would run all [selected] tests compatible with running against a running
server. And
$ meson test --setup 'installed'
would test installed binaries.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-24 Thread Peter Geoghegan

On Sat, Sep 24, 2022 at 5:13 PM Andres Freund  wrote:
> > One more question about this, that wasn't covered by the Wiki page: is
> > there some equivalent to "make installcheck" with meson builds?
>
> Not yet. Nothing impossible, just not done yet. Partially because installcheck
> is so poorly defined (run against an already running server for pg_regress vs
> using "system" installed binaries for tap tests).

Got it. I can work around that by just having an old autoconf-based
vpath build directory. I'll need to do this when I run Valgrind.

My workaround would be annoying if I needed to run "installcheck"
anywhere near as frequently as I run "make check-world". But that
isn't the case. meson delivers a significant improvement in the metric
that really matters to me, so I can't really complain.

-- 
Peter Geoghegan

Re: [RFC] building postgres with meson - v13

2022-09-24 Thread Andres Freund

Hi,

On 2022-09-24 16:56:20 -0700, Peter Geoghegan wrote:
> On Thu, Sep 22, 2022 at 2:50 PM Andres Freund  wrote:
> > meson:
> >
> > time meson test
> > real0m42.178s
> > user7m8.533s
> > sys 2m17.711s
> 
> I find that a more or less comparable test run on my workstation
> (which has a Ryzen 9 5950X) takes just over 38 seconds. I think that
> the improvement is far more pronounced on that machine compared to a
> much older workstation.

Cool!


> One more question about this, that wasn't covered by the Wiki page: is
> there some equivalent to "make installcheck" with meson builds?

Not yet. Nothing impossible, just not done yet. Partially because installcheck
is so poorly defined (run against an already running server for pg_regress vs
using "system" installed binaries for tap tests).

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-24 Thread Peter Geoghegan

On Thu, Sep 22, 2022 at 2:50 PM Andres Freund  wrote:
> meson:
>
> time meson test
> real0m42.178s
> user7m8.533s
> sys 2m17.711s

I find that a more or less comparable test run on my workstation
(which has a Ryzen 9 5950X) takes just over 38 seconds. I think that
the improvement is far more pronounced on that machine compared to a
much older workstation.

One more question about this, that wasn't covered by the Wiki page: is
there some equivalent to "make installcheck" with meson builds?

-- 
Peter Geoghegan

Re: [RFC] building postgres with meson - v13

2022-09-24 Thread Tom Lane

... btw, shouldn't the CF entry [1] get closed now?
The cfbot's unhappy that the last patch no longer applies.

regards, tom lane

[1] https://commitfest.postgresql.org/39/3395/

Re: [RFC] building postgres with meson - v13

2022-09-22 Thread Nathan Bossart

On Thu, Sep 22, 2022 at 01:28:09PM -0700, Andres Freund wrote:
> On 2022-09-22 13:05:33 -0700, Nathan Bossart wrote:
>> * I'm using an Ubuntu-based distribution, and the version of meson that apt
>> installed was not new enough for Postgres.  I ended up cloning meson [0]
>> and using the newest tag.  This is no big deal.
> 
> I assume this is 20.04 LTS? If so, we're missing it by one version of meson
> currently. There's unfortunately a few features that'd be a bit painful to not
> have.

Yes.  I imagine I'll upgrade to 22.04 LTS soon, which appears to provide a
new enough version of meson.

>> * The installed binaries were unable to locate libraries like libpq.  I
>> ended up setting the extra_lib_dirs option to the directory where these
>> libraries were installed to fix this.  This one is probably worth
>> investigating further.
> 
> I think that should be "fixed" in a later commit in the meson tree - any
> chance you could try that?

Yup, after cherry-picking 9bc60bc, this is fixed.

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

Re: [RFC] building postgres with meson - v13

2022-09-22 Thread Peter Geoghegan

On Thu, Sep 22, 2022 at 2:50 PM Andres Freund  wrote:
> I'm likely the most biased person on this, but for me the reliable incremental
> builds and the readability of the test output are big enough wins that the
> answer is pretty clear... Doesn't hurt that running all tests is faster too.

It's nice that things are much more discoverable now. For example, if
you want to run some random test on its own then you just...do it in
the obvious, discoverable way. It took me about 2 minutes to figure
out how to do that, without reading any documentation.

OTOH doing the same thing with the old autoconf-based build system
requires the user to know the exact magical incantation for Postgres
tests. You just have to know to run the 2 or 3 tests that are
undocumented (or poorly documented) dependencies first. That seems
like an enormous usability improvement, especially for those of us
that haven't been working on Postgres for years.

> time to run all tests (cassert, -Og), in a fully built tree:

> time make -j48 -s -Otarget check-world PROVE_FLAGS='-j4'
> real1m1.577s
> user7m32.579s
> sys 2m17.767s

> time meson test
> real0m42.178s
> user7m8.533s
> sys 2m17.711s

Sold!

-- 
Peter Geoghegan

Re: [RFC] building postgres with meson - v13

Hi,

On 2022-09-22 13:21:28 -0700, Peter Geoghegan wrote:
> Is it generally recommended that individual hackers mostly switch over
> to Meson for their day to day work soon? I'm guessing that this
> question doesn't really have a clear answer yet, but thought I'd ask,
> just in case.

It'll probably depend on who you ask ;)

I'm likely the most biased person on this, but for me the reliable incremental
builds and the readability of the test output are big enough wins that the
answer is pretty clear... Doesn't hurt that running all tests is faster too.

The currently existing limitations are imo mostly around making it usable for
production, particularly on windows.

time to run all tests (cassert, -Og), in a fully built tree:

make:

time make -j48 -s -Otarget check-world
real2m44.206s
user6m29.121s
sys 1m54.069s

time make -j48 -s -Otarget check-world PROVE_FLAGS='-j4'
real1m1.577s
user7m32.579s
sys 2m17.767s

meson:

time meson test
real0m42.178s
user7m8.533s
sys 2m17.711s

FWIW, I just rebased my older patch to cache and copy initdb during the
tests. The %user saved is impressive enough to pursue it again...

time make -j48 -s -Otarget check-world PROVE_FLAGS='-j4'
real0m52.655s
user2m19.504s
sys 1m26.264s

time meson test:

real0m36.370s
user2m14.748s
sys 1m36.741s

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

Hi,

On 2022-09-22 13:05:33 -0700, Nathan Bossart wrote:
> I gave the meson build system a try, and it seems to work nicely.  It
> didn't take long at all to adapt my workflow.
> 
> A few notes from my experience:
> 
> * I'm using an Ubuntu-based distribution, and the version of meson that apt
> installed was not new enough for Postgres.  I ended up cloning meson [0]
> and using the newest tag.  This is no big deal.

I assume this is 20.04 LTS? If so, we're missing it by one version of meson
currently. There's unfortunately a few features that'd be a bit painful to not
have.


> * The installed binaries were unable to locate libraries like libpq.  I
> ended up setting the extra_lib_dirs option to the directory where these
> libraries were installed to fix this.  This one is probably worth
> investigating further.

I think that should be "fixed" in a later commit in the meson tree - any
chance you could try that?

https://github.com/anarazel/postgres/tree/meson


> * meson really doesn't like it when there are things leftover from
> configure/make.  Whenever I switch from make to meson, I have to run 'make
> maintainer-clean'.

Yes. I recommend building out-of-tree with autoconf as well.


> Otherwise, all of my usual build options, ccache, etc. are working just
> like before.  Nice work!

Cool!

Thanks for testing,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-22 Thread Peter Geoghegan

On Thu, Sep 22, 2022 at 1:05 PM Nathan Bossart  wrote:
> Otherwise, all of my usual build options, ccache, etc. are working just
> like before.  Nice work!

+1

Is it generally recommended that individual hackers mostly switch over
to Meson for their day to day work soon? I'm guessing that this
question doesn't really have a clear answer yet, but thought I'd ask,
just in case.

--
Peter Geoghegan

Re: [RFC] building postgres with meson - v13

2022-09-22 Thread Nathan Bossart

I gave the meson build system a try, and it seems to work nicely.  It
didn't take long at all to adapt my workflow.

A few notes from my experience:

* I'm using an Ubuntu-based distribution, and the version of meson that apt
installed was not new enough for Postgres.  I ended up cloning meson [0]
and using the newest tag.  This is no big deal.

* The installed binaries were unable to locate libraries like libpq.  I
ended up setting the extra_lib_dirs option to the directory where these
libraries were installed to fix this.  This one is probably worth
investigating further.

* meson really doesn't like it when there are things leftover from
configure/make.  Whenever I switch from make to meson, I have to run 'make
maintainer-clean'.

Otherwise, all of my usual build options, ccache, etc. are working just
like before.  Nice work!

[0] https://github.com/mesonbuild/meson

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com

Re: [RFC] building postgres with meson - v13

2022-09-22 Thread Alvaro Herrera

On 2022-Sep-22, Tom Lane wrote:

> Ah, right, the joys of maintaining multiple build systems.  I wonder
> if there's any way to avoid that by scraping file lists from one
> group to the other.

Or maybe we could have a file common to both, say OBJS, which both
scrape in their own way.  That could be easier than one scraping the
other.

> We got a little spoiled perhaps by the MSVC scripts managing to do
> that in most cases.

Right ... and it was so annoying in the cases it couldn't.

-- 
Álvaro HerreraBreisgau, Deutschland  —  https://www.EnterpriseDB.com/
"Aprender sin pensar es inútil; pensar sin aprender, peligroso" (Confucio)

Re: [RFC] building postgres with meson - v13

Hi,

On 2022-09-22 04:29:15 -0400, Andrew Dunstan wrote:
> Great. Now I'll start on buildfarm support. Given my current
> commitments, this will take me a while, but I hope to have a working
> client by about the beginning of November.

Great! Let me know if there's something I can do to help.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

Hi,

On 2022-09-22 16:56:57 +0200, Alvaro Herrera wrote:
> On 2022-Sep-22, Tom Lane wrote:
> > Initial reports from the cfbot are mostly promising, but there are a few
> > patches where all the meson builds fail while all the autoconf ones pass,
> > so there's something for you to look at.  So far CF entries 3464, 3733,
> > 3771, 3808 look that way.
> 
> Hmm, but those patches add files, which means they're now outdated: they
> need to add these files to the respective meson.build file.

Yea, I looked through all of these and they all need need a simple addition of
a file to be built or installed.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-22 Thread Tom Lane

Alvaro Herrera  writes:
> On 2022-Sep-22, Tom Lane wrote:
>> Initial reports from the cfbot are mostly promising, but there are a few
>> patches where all the meson builds fail while all the autoconf ones pass,
>> so there's something for you to look at.  So far CF entries 3464, 3733,
>> 3771, 3808 look that way.

> Hmm, but those patches add files, which means they're now outdated: they
> need to add these files to the respective meson.build file.

Ah, right, the joys of maintaining multiple build systems.  I wonder
if there's any way to avoid that by scraping file lists from one
group to the other.  We got a little spoiled perhaps by the MSVC
scripts managing to do that in most cases.

regards, tom lane

Re: [RFC] building postgres with meson - v13

2022-09-22 Thread Alvaro Herrera

On 2022-Sep-22, Tom Lane wrote:

> Initial reports from the cfbot are mostly promising, but there are a few
> patches where all the meson builds fail while all the autoconf ones pass,
> so there's something for you to look at.  So far CF entries 3464, 3733,
> 3771, 3808 look that way.

Hmm, but those patches add files, which means they're now outdated: they
need to add these files to the respective meson.build file.

-- 
Álvaro HerreraBreisgau, Deutschland  —  https://www.EnterpriseDB.com/
"La experiencia nos dice que el hombre peló millones de veces las patatas,
pero era forzoso admitir la posibilidad de que en un caso entre millones,
las patatas pelarían al hombre" (Ijon Tichy)

Re: [RFC] building postgres with meson - v13

2022-09-22 Thread Tom Lane

Andres Freund  writes:
> On 2022-09-21 09:46:30 -0700, Andres Freund wrote:
>> I'm planning to commit this today, unless somebody wants to argue against
>> that.

> And done!

Yay!

Initial reports from the cfbot are mostly promising, but there are a few
patches where all the meson builds fail while all the autoconf ones pass,
so there's something for you to look at.  So far CF entries 3464, 3733,
3771, 3808 look that way.

regards, tom lane

Re: [RFC] building postgres with meson - v13

2022-09-22 Thread Andrew Dunstan



On 2022-09-22 Th 01:57, Andres Freund wrote:
> Hi,
>
> On 2022-09-21 09:46:30 -0700, Andres Freund wrote:
>> I'm planning to commit this today, unless somebody wants to argue against
>> that.
> And done!
>
> Changes:
> - fixed a few typos (thanks Thomas)
> - less duplication in the CI tasks
> - removed an incomplete implementation of the target for abbrevs.txt - do we
>   even want to have that?
> - plenty hand wringing on my part
>
>
> I also rebased my meson git tree, which still has plenty additional test
> platforms (netbsd, openbsd, debian sid, fedora rawhide, centos 8, centos 7,
> opensuse tumbleweed), but without the autoconf versions of those targets.  I
> also added a commit that translates most of the CompilerWarnings task to
> meson.  Still need to add a headerscheck / cpluspluscheck target.
>

Great. Now I'll start on buildfarm support. Given my current
commitments, this will take me a while, but I hope to have a working
client by about the beginning of November.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Re: [RFC] building postgres with meson - v13

2022-09-21 Thread Andres Freund

Hi,

On 2022-09-21 09:46:30 -0700, Andres Freund wrote:
> I'm planning to commit this today, unless somebody wants to argue against
> that.

And done!

Changes:
- fixed a few typos (thanks Thomas)
- less duplication in the CI tasks
- removed an incomplete implementation of the target for abbrevs.txt - do we
  even want to have that?
- plenty hand wringing on my part

I also rebased my meson git tree, which still has plenty additional test
platforms (netbsd, openbsd, debian sid, fedora rawhide, centos 8, centos 7,
opensuse tumbleweed), but without the autoconf versions of those targets.  I
also added a commit that translates most of the CompilerWarnings task to
meson.  Still need to add a headerscheck / cpluspluscheck target.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-21 Thread Andres Freund

Hi,

On 2022-09-21 09:46:30 -0700, Andres Freund wrote:
> After that I am planning to split the "ci" commit so that it converts a few of
> the CI tasks to use meson, without adding all the other platforms I added for
> development. I think that's important to get in soon, given that it'll
> probably take a bit until the buildfarm grows meson coverage and because it
> provides cfbot coverage which seems important for now as well.
> 
> I think we should:
> 
> - convert windows to build with ninja - it builds faster, runs all tests,
>   parallelizes tests. That means that msbuild based builds don't have coverage
>   via CI / cfbot, but we don't currently have the resources to test both.

I was working on that and hit an issue that took me a while to resolve: Once I
tested only the "main" meson commit plus CI the windows task was running out
of memory. There was an outage of the CI provider at the same time, so I first
blamed it on that. But it turns out to be "legitimately" high memory usage
related to debug symbols - the only reason CI didn't show that before was that
it's incidentally fixed as a indirect consequence of using precompiled
headers, in a later commit. Argh.  It can also be fixed by the option required
to use ccache at some point, so I'll do that for now.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-21 Thread Justin Pryzby

On Wed, Sep 21, 2022 at 09:46:30AM -0700, Andres Freund wrote:
> I think we should:
> 
> - convert windows to build with ninja - it builds faster, runs all tests,
>   parallelizes tests. That means that msbuild based builds don't have coverage
>   via CI / cfbot, but we don't currently have the resources to test both.

+1

If multiple Windows (or other) tasks are going to exist, I think they
should have separate "ci-os-only" conditions, like windows-msvc,
windows-ninja, ...  It should be possible to run only one.

Re: [RFC] building postgres with meson - v13

2022-09-21 Thread Andres Freund

Hi,

On 2022-09-21 13:56:37 -0400, Tom Lane wrote:
> Andres Freund  writes:
> > I think we should:
> 
> > - convert windows to build with ninja - it builds faster, runs all tests,
> >   parallelizes tests. That means that msbuild based builds don't have 
> > coverage
> >   via CI / cfbot, but we don't currently have the resources to test both.
> 
> Check.  The sooner we can get rid of the custom MSVC scripts, the better,
> because now we'll be on the hook to maintain *three* build systems.

Agreed. I think the only "major" missing thing is the windows resource file
generation stuff, which is mostly done in one of the "later" commits. Also
need to test a few more of the optional dependencies (ICU, gettext, ...) on
windows (I did test zlib, lz4, zstd). And of course get a bit of wider
exposure than "just me and CI".

> > I'm less clear on whether we should convert macos / freebsd to meson at this
> > point?
> 
> We certainly could debug/polish the meson stuff just on linux and windows
> for now, but is there a reason to wait on the others?

No - freebsd and macos have worked in CI for a long time. I was wondering
whether we want more coverage for autoconf in CI, but thinking about it
futher, it's more important to have the meson coverage.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-21 Thread Tom Lane

Andres Freund  writes:
> I think we should:

> - convert windows to build with ninja - it builds faster, runs all tests,
>   parallelizes tests. That means that msbuild based builds don't have coverage
>   via CI / cfbot, but we don't currently have the resources to test both.

Check.  The sooner we can get rid of the custom MSVC scripts, the better,
because now we'll be on the hook to maintain *three* build systems.

> - add a linux build using meson, we currently can afford building both with
>   autoconf and meson for linux

Right.

> I'm less clear on whether we should convert macos / freebsd to meson at this
> point?

We certainly could debug/polish the meson stuff just on linux and windows
for now, but is there a reason to wait on the others?

regards, tom lane

Re: [RFC] building postgres with meson - v13

2022-09-20 Thread Andres Freund

Hi,

On 2022-09-21 09:52:48 +0700, John Naylor wrote:
> On Wed, Sep 21, 2022 at 7:11 AM Andres Freund  wrote:
> >
> > Hi,
> >
> > On 2022-09-19 19:16:30 -0700, Andres Freund wrote:
> > > To have some initial "translation" for other developers I've started a
> wiki
> > > page with a translation table. Still very WIP:
> > > https://wiki.postgresql.org/wiki/Meson
> > >
> > > For now, a bit of polishing aside, I'm just planning to add a minimal
> > > explanation of what's happening, and a reference to this thread.
> >
> > I added installation instructions for meson for a bunch of platforms, but
> 
> Small typo: The homebrew section is still labeled with "find MacPorts
> libraries".

Thanks, fixed. I wrote these blindly, so there's probably more wrong than this
- although Thomas was helpful enough to provide some information / testing.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-20 Thread John Naylor

On Wed, Sep 21, 2022 at 7:11 AM Andres Freund  wrote:
>
> Hi,
>
> On 2022-09-19 19:16:30 -0700, Andres Freund wrote:
> > To have some initial "translation" for other developers I've started a
wiki
> > page with a translation table. Still very WIP:
> > https://wiki.postgresql.org/wiki/Meson
> >
> > For now, a bit of polishing aside, I'm just planning to add a minimal
> > explanation of what's happening, and a reference to this thread.
>
> I added installation instructions for meson for a bunch of platforms, but

Small typo: The homebrew section is still labeled with "find MacPorts
libraries".

--
John Naylor
EDB: http://www.enterprisedb.com

Re: [RFC] building postgres with meson - v13

2022-09-20 Thread Andres Freund

Hi,

On 2022-09-19 19:16:30 -0700, Andres Freund wrote:
> To have some initial "translation" for other developers I've started a wiki
> page with a translation table. Still very WIP:
> https://wiki.postgresql.org/wiki/Meson
> 
> For now, a bit of polishing aside, I'm just planning to add a minimal
> explanation of what's happening, and a reference to this thread.

I added installation instructions for meson for a bunch of platforms, but
failed to figure out how to do so in a rhel9 container. I don't have a rhel
subscription, and apparently the repos with developer tools now require a
subscription. Great way to make it easy for projects to test anything on RHEL.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-19 Thread Andres Freund

Hi,

On 2022-09-19 05:25:59 -0400, Peter Eisentraut wrote:
> IMO, the following commits are ready to be pushed now:

Slowly working through them.

To have some initial "translation" for other developers I've started a wiki
page with a translation table. Still very WIP:
https://wiki.postgresql.org/wiki/Meson

For now, a bit of polishing aside, I'm just planning to add a minimal
explanation of what's happening, and a reference to this thread.

> b7d7fe009731 Remove DLLTOOL, DLLWRAP from configure / Makefile.global.in
> 979f26889544 Don't hardcode tmp_check/ as test directory for tap tests
> 9fc657fbb7e2 Split TESTDIR into TESTLOGDIR and TESTDATADIR
> 6de8f1de0ffa meson: prereq: Extend gendef.pl in preparation for meson
> 5a9731dcc2e6 meson: prereq: port: Include c.h instead of postgres.h in 
> *p{read,write}*.c

Done

> 7054861f0fef meson: prereq: Add src/tools/gen_export.pl

This one I'm planning to merge with the "main" commit, given there's no other 
user.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-19 Thread Peter Eisentraut


On 19.09.22 02:29, Andres Freund wrote:

Hi,

On September 18, 2022 5:24:06 PM PDT, Peter Eisentraut 
 wrote:

On 15.09.22 04:26, Andres Freund wrote:

Attached is v13 of the meson patchset. The biggest changes are:


Did something about warning flags change from the previous patch set?  I see 
it's building with -Wextra now, which combined with -Werror causes the build to 
fail for me.  I have never encountered that with any of the previous patch sets.


In older versions of the patch the default warning level was set to include 
Wextra, and I had added my local flags to suppress uninteresting warnings. 
Comparing the warning flags I reduced the warning level and removed the 
suppressing flags - but changing default options only affects new build trees. 
To change existing ones do meson configure -Dwarning_level=1


Ok that was the reason.  It works now.

IMO, the following commits are ready to be pushed now:

b7d7fe009731 Remove DLLTOOL, DLLWRAP from configure / Makefile.global.in
979f26889544 Don't hardcode tmp_check/ as test directory for tap tests
9fc657fbb7e2 Split TESTDIR into TESTLOGDIR and TESTDATADIR
6de8f1de0ffa meson: prereq: Extend gendef.pl in preparation for meson
7054861f0fef meson: prereq: Add src/tools/gen_export.pl
1aa586f2921c meson: prereq: Refactor PG_TEST_EXTRA logic in autoconf build
5a9731dcc2e6 meson: prereq: port: Include c.h instead of postgres.h in 
*p{read,write}*.c
1939bdcfbfea meson: Add meson based buildsystem

Re: [RFC] building postgres with meson - v13

2022-09-18 Thread Andres Freund

Hi, 

On September 18, 2022 5:24:06 PM PDT, Peter Eisentraut 
 wrote:
>On 15.09.22 04:26, Andres Freund wrote:
>> Attached is v13 of the meson patchset. The biggest changes are:
>
>Did something about warning flags change from the previous patch set?  I see 
>it's building with -Wextra now, which combined with -Werror causes the build 
>to fail for me.  I have never encountered that with any of the previous patch 
>sets.

In older versions of the patch the default warning level was set to include 
Wextra, and I had added my local flags to suppress uninteresting warnings. 
Comparing the warning flags I reduced the warning level and removed the 
suppressing flags - but changing default options only affects new build trees. 
To change existing ones do meson configure -Dwarning_level=1
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [RFC] building postgres with meson - v13

2022-09-18 Thread Peter Eisentraut


On 15.09.22 04:26, Andres Freund wrote:

Attached is v13 of the meson patchset. The biggest changes are:


Did something about warning flags change from the previous patch set?  I 
see it's building with -Wextra now, which combined with -Werror causes 
the build to fail for me.  I have never encountered that with any of the 
previous patch sets.

Re: [RFC] building postgres with meson - v13

2022-09-17 Thread Andres Freund

Hi,

On 2022-09-16 16:22:35 -0400, Tom Lane wrote:
> Andres Freund  writes:
> > On 2022-09-16 09:14:20 +1200, Thomas Munro wrote:
> >> On Thu, Sep 15, 2022 at 2:26 PM Andres Freund  wrote:
> >>> - noticed that libpgport.a had and needed a dependency on errcodes.h - 
> >>> that
> >>> seemed wrong. The dependency is due to src/port/*p{read,write}v?.c 
> >>> including
> >>> postgres.h - which seems wrong. So I added a patch changing them to 
> >>> include
> >>> c.h.
> 
> >> Oops.  +1
> 
> > Looks like this has been the case since
> > 0d56acfbaa799553c0c6ea350fd6e68d81025994 in 14. Any opinions on whether we
> > should backpatch the "fix"?
> 
> +1, those files have no business including all of postgres.h

Done.

I've been wondering whether we should protect against this kind of issue on
the buildsystem level. Whenever building frontend code, add something like
-DBUILDING_FRONTEND, and error out if postgres.h is included without going
through postgres_fe.h.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-16 Thread Tom Lane

Andres Freund  writes:
> On 2022-09-16 09:14:20 +1200, Thomas Munro wrote:
>> On Thu, Sep 15, 2022 at 2:26 PM Andres Freund  wrote:
>>> - noticed that libpgport.a had and needed a dependency on errcodes.h - that
>>> seemed wrong. The dependency is due to src/port/*p{read,write}v?.c including
>>> postgres.h - which seems wrong. So I added a patch changing them to include
>>> c.h.

>> Oops.  +1

> Looks like this has been the case since
> 0d56acfbaa799553c0c6ea350fd6e68d81025994 in 14. Any opinions on whether we
> should backpatch the "fix"?

+1, those files have no business including all of postgres.h

regards, tom lane

Re: [RFC] building postgres with meson - v13

2022-09-16 Thread Andres Freund

Hi,

On 2022-09-16 09:14:20 +1200, Thomas Munro wrote:
> On Thu, Sep 15, 2022 at 2:26 PM Andres Freund  wrote:
> > - noticed that libpgport.a had and needed a dependency on errcodes.h - that
> >   seemed wrong. The dependency is due to src/port/*p{read,write}v?.c 
> > including
> >   postgres.h - which seems wrong. So I added a patch changing them to 
> > include
> >   c.h.
> 
> Oops.  +1

Looks like this has been the case since
0d56acfbaa799553c0c6ea350fd6e68d81025994 in 14. Any opinions on whether we
should backpatch the "fix"?

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-15 Thread Andres Freund

Hi,

On 2022-09-16 09:14:20 +1200, Thomas Munro wrote:
> GCC 12 produces a bunch of warnings by default with meson, and that
> turned out to be because the default optimisation level is -O3.
> That's a change from the make build, which uses -O2.  Should we set a
> default of 2, or is there some meson-way-of-doing-things reason why
> not?

We can change the defaults - the only downside is that there's a convenience
setting 'buildtype' (debug, debugoptimized, release, minsize, custom, plain)
that changes multiple settings (optimization level, amount of debugging
information) and that doesn't work as nicely if you change the default
compiler optimization setting.

They made a similar discovery as us, deriving the defaults of settings based
on other settings quickly can become confusing. I think they're looking at how
to make that UI a bit nicer.

I'd prefer to defer fine-tuning the default settings till a bunch of this has
gone in, but I won't insist on that course.

Their default warning flags passed to compilers trigger a bunch of warnings in
our build (irrespective of -O*), so I lowered the warning level. But I think
their set of settings likely is sensible, an we should just disable a bunch of
warnings we don't care about. But I haven't done that for now, to keep the set
of warning flags the same between meson and autoconf.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-15 Thread Thomas Munro

On Thu, Sep 15, 2022 at 2:26 PM Andres Freund  wrote:
> - noticed that libpgport.a had and needed a dependency on errcodes.h - that
>   seemed wrong. The dependency is due to src/port/*p{read,write}v?.c including
>   postgres.h - which seems wrong. So I added a patch changing them to include
>   c.h.

Oops.  +1

GCC 12 produces a bunch of warnings by default with meson, and that
turned out to be because the default optimisation level is -O3.
That's a change from the make build, which uses -O2.  Should we set a
default of 2, or is there some meson-way-of-doing-things reason why
not?

Re: [RFC] building postgres with meson - v13

2022-09-14 Thread Andres Freund

Hi,

On 2022-09-15 01:10:16 -0400, Tom Lane wrote:
> Andres Freund  writes:
> > I'm inclined to build the static lib on windows as long as we do it on other
> > platforms.
> 
> Maybe I spent too much time working for Red Hat, but I'm kind of
> unhappy that we build static libraries at all.

Yea, I have been wondering about that too.

Oddly enough, given our current behaviour, the strongest case for static
libraries IMO is on windows, due to the lack of a) rpath b) a general library
search path.

Peter IIRC added the static libraries to the meson port just to keep the set
of installed files the same, which makes sense.

> They are maintenance hazards and therefore security hazards by definition,
> because if you find a problem in $package_x you will have to find and
> rebuild every other package that has statically-embedded code from
> $package_x.  So Red Hat has, or least had, a policy against packages
> exporting such libraries.

It obviously is a bad idea for widely used system packages. I think there are
a few situations, e.g. a downloadable self-contained and relocatable
application, where shared libraries provide less of a benefit.

> I realize that there are people for whom other considerations outweigh
> that, but I don't think that we should install static libraries by
> default.  Long ago it was pretty common for configure scripts to
> offer --enable-shared and --enable-static options ... should we
> resurrect that?

It'd be easy enough. I don't really have an opinion on whether it's worth
having the options. I think most packaging systems have ways of not including
files even if $software installs them.

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v13

2022-09-14 Thread Tom Lane

Andres Freund  writes:
> I'm inclined to build the static lib on windows as long as we do it on other
> platforms.

Maybe I spent too much time working for Red Hat, but I'm kind of
unhappy that we build static libraries at all.  They are maintenance
hazards and therefore security hazards by definition, because if
you find a problem in $package_x you will have to find and rebuild
every other package that has statically-embedded code from $package_x.
So Red Hat has, or least had, a policy against packages exporting
such libraries.

I realize that there are people for whom other considerations outweigh
that, but I don't think that we should install static libraries by
default.  Long ago it was pretty common for configure scripts to
offer --enable-shared and --enable-static options ... should we
resurrect that?

regards, tom lane

Re: [RFC] building postgres with meson - v12

2022-09-14 Thread Peter Eisentraut


On 08.09.22 09:42, Alvaro Herrera wrote:

On 2022-Sep-07, Peter Eisentraut wrote:


A possible drawback is that the intermediate postgres-full.xml file is

10MB, but I guess we're past the point where we are worrying about that

kind of thing.


I think we are, but maybe mark it .PRECIOUS?  IIUC that would prevent it
from being removed if there's a problem in the other recipes.


I don't think .PRECIOUS is the right tool here.  There are existing uses 
of .SECONDARY in doc/src/sgml/Makefile; I integrated my patch there.

Re: [RFC] building postgres with meson - v12

2022-09-14 Thread Peter Eisentraut


On 07.09.22 09:53, Peter Eisentraut wrote:

On 07.09.22 09:19, Peter Eisentraut wrote:
This could also be combined with the idea of the postgres.sgml.valid 
thing you have in the meson patch set.


I'll finish this up and produce a proper patch.


Something like this.

This does make the rules more straightforward and avoids repeated 
xmllint calls.  I suppose this also helps writing the meson rules in a 
simpler way.


committed this

Re: [RFC] building postgres with meson - v12

2022-09-12 Thread Andres Freund

Hi,

On 2022-09-07 07:00:17 +0200, Peter Eisentraut wrote:
> On 31.08.22 20:11, Andres Freund wrote:
> > > src/port/win32ver.rc.in: This is redundant with src/port/win32ver.rc.
> > > (Note that the latter is also used as an input file for text
> > > substitution.  So having another file named *.in next to it would be
> > > super confusing.)
> > Yea, this stuff isn't great. I think the better solution, both for meson and
> > for configure, would be to move to do all the substitution to the C
> > preprocessor.
>
> Yeah, I think if we can get rid of the evil date-based versioning, then
> this could be done like

> -win32ver.o: win32ver.rc
> -   $(WINDRES) -i $< -o $@ --include-dir=$(top_builddir)/src/include 
> --include-dir=$(srcdir)
> +win32ver.o: $(top_srcdir)/src/port/win32ver.rc
> +   $(WINDRES) -i $< -o $@ --include-dir=$(top_builddir)/src/include 
> --include-dir=$(srcdir) -D FILEDESC=$(PGFILEDESC) -D VFT_APP=$(PGFTYPE) 
> -D_ICO_=$(PGICOSTR) -D_INTERNAL_NAME_=$(if 
> $(shlib),s;_INTERNAL_NAME_;"$(basename $(shlib))";,d) -D_ORIGINAL_NAME_=$(if 
> $(shlib),s;_ORIGINAL_NAME_;"$(shlib)";,d)

It tried this and while it works for some places, it doesn't work for all. It
looks like windres uses broken quoting when internally invoking cpp. It
escapes e.g. whitespaces, but it doesn't escape at least < and >. Which
doesn't work well with descriptions like

PGFILEDESC  = "cyrillic <-> mic text conversions"

resulting in this:

strace --string-limit=2000 -f -e execve \
x86_64-w64-mingw32-windres -DPGFILEDESC="cyrillic <-> mic text conversions" 
-DPGFTYPE=VFT_DLL -DPGNAME=cyrillic_and_mic -DPGFILEENDING=dll 
-I../../../../../../src/include -I/home/andres/src/postgresql/src/include  
-I/home/andres/src/postgresql/src/include/port/win32  
"-I/home/andres/src/postgresql/src/include/port/win32" 
-DWIN32_STACK_RLIMIT=4194304 -i 
/home/andres/src/postgresql/src/port/win32ver.rc -o win32ver.o
...
[pid 1788987] execve("/bin/sh", ["sh", "-c", "x86_64-w64-mingw32-gcc -E -xc 
-DRC_INVOKED -DPGFILEDESC=cyrillic\\ <->\\ mic\\ text\\ conversions 
-DPGFTYPE=VFT_DLL -DPGNAME=cyrillic_and_mic -DPGFILEENDING=dll 
-I../../../../../../src/include -I/home/andres/src/postgresql/src/include 
-I/home/andres/src/postgresql/src/include/port/win32 
-I/home/andres/src/postgresql/src/include/port/win32 
-DWIN32_STACK_RLIMIT=4194304 
/home/andres/src/postgresql/src/port/win32ver.rc"], 0x7ffd47edc790 /* 67 vars 
*/) = 0
sh: 1: cannot open -: No such file
[pid 1788987] +++ exited with 2 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1788987, si_uid=1000, 
si_status=2, si_utime=0, si_stime=0} ---
x86_64-w64-mingw32-windres: preprocessing failed.

given this shoddy quoting, I think it's probably not wise to go down this
path?

We could invoke the preprocessor ourselves, but that requires feeding the
compiler via stdin (otherwise it'll just warn "linker input file unused
because linking not done") and defining -DRC_INVOKED (otherwise there'll be
syntax errors). That feels like too much magic?

Greetings,

Andres Freund

Re: [RFC] building postgres with meson - v12

2022-09-09 Thread Andres Freund

Hi,

On 2022-08-31 11:11:54 -0700, Andres Freund wrote:
> > If the above are addressed, I think this will be just about at the
> > point where the above patches can be committed.
>
> Woo!

There was a lot less progress over the last ~week than I had hoped. The reason
is that I was trying to figure out the reason for the occasional failures of
ecpg tests getting compiled when building on windows in CI, with msbuild.

I went into many layers of rabbitholes while investigating. Wasting an absurd
amount of time.

The problem:

Occasionally ecpg test files would fail to compile, exiting with -1073741819:
C:\BuildTools\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(241,5):
error MSB8066: Custom build for
'C:\cirrus\build\meson-private\custom_target.rule' exited with code
-1073741819.
[c:\cirrus\build\src\interfaces\ecpg\test\sql\3701597@@twophase.c@cus.vcxproj]

-1073741819 is 0xc005, which in turn is STATUS_ACCESS_VIOLATION, i.e. a
segfault. This happens in roughly 1/3 of the builds, but with "streaks" of not
happening and more frequently happening.

However, despite our CI images having a JIT debugger configured (~coredump
handler), no crash report was triggered. The problem never occurs in my
windows VM.

At first I thought that might be because it's an assertion failure or such,
which only causes a dump when a bunch of magic is done (see main.c). But
despite adding all the necessary magic to ecpg.exe, no dump.

Unfortunately, adding debug output reduces the frequency of the issue
substantially.

Eventually I figured out that it's not actually ecpg.exe that is crashing. It
is meson's python wrapper around built binaries as part of the build (for
setting PATH, working directory, without running into cmd.exe issues). A
modified meson wrapper showed that ecpg.exe completes successfully.

The only thing the meson wrapper does after running the command is to call
sys.exit(returncode), and I had printed out the returncode, which is 0.

I looked through a lot of the python code, to see why no crashdump and no
details are forthcoming. There weren't any relevant
SetErrorMode(SEM_NOGPFAULTERRORBOX) calls. I tried to set PYTHONFAULTHANDLER,
but still no stack trace.

Next I suspected that cmd.exe might be crashing and causing the
problem. Modified meson to add 'echo %ERRORLEVEL%' to the msbuild
custombuild. Which indeed shows the STATUS_ACCESS_VIOLATION returncode after
running python. So it's not cmd.exe.

The problem even persisted when replacing meson's sys.exit() with os._exit(),
which indeed just calls _exit().

I tried to reproduce the problem using a python with debugging enabled. The
problem doesn't occur despite quite a few runs.

I found scattered other reports of this problem happening on windows. Went
down a few more rabbitholes. Too boring to repeat here.

At this point I finally figured out that the reason the crash reports don't
happen is that everythin started by cirrus-ci on windows has an errormode of
SEM_FAILCRITICALERRORS | SEM_NOGPFAULTERRORBOX | SEM_NOOPENFILEERRORBOX.

A good bit later I figured out that while cirrus-ci isn't intentionally
setting that, golang does so *unconditionally* on windows:
https://github.com/golang/go/blob/54182ff54a687272dd7632c3a963e036ce03cb7c/src/runtime/signal_windows.go#L14
https://github.com/golang/go/blob/54182ff54a687272dd7632c3a963e036ce03cb7c/src/runtime/os_windows.go#L553
Argh. I should have checked what the error mode is earlier, but this is just
very sneaky.

So I modified meson to change the errormode and tried to reproduce the issue
again, to finally get a stackdump. And tried again. And again. Without a
single relevant failure (I saw tests fail in ways that are discussed on the
list, but that's irrelevant here).

I've run this through enough attempts by now that I'm quite confident that the
problem does not occur when the errormode does not include
SEM_NOOPENFILEERRORBOX. I'll want a few more runs to be certain, but...

Given that the problem appears to happen after _exit() is called, and only
when SEM_NOOPENFILEERRORBOX is not set, it seems likely to be an OS / C
runtime bug. Presumably it's related to something that python does first, but
I don't see how anything could justify crashing only if SEM_NOOPENFILEERRORBOX
is set (rather than the opposite).

I have no idea how to debug this further, given that the problem is quite rare
(can't attach a debugger and wait), only happens when crashdumps are prevented
from happening (so no idea where it crashes) and is made less common by debug
printfs.

So for now the best way forward I can see is to change the error mode for CI
runs. Which is likely a good idea anyway, so we can see crashdumps for
binaries other than postgres.exe (which does SetErrorMode() internally). I
managed to do so by setting CIRRUS_SHELL to a python wrapper around cmd.exe
that does SetErrorMode(). I'm sure there's easier ways, but I couldn't figure
out any.

I'd like to reclaim my time. But I'm afraid nobody will

Re: [RFC] building postgres with meson - v11

2022-09-08 Thread samay sharma

On Tue, Sep 6, 2022 at 9:46 PM Peter Eisentraut <
peter.eisentr...@enterprisedb.com> wrote:

> On 02.09.22 01:12, samay sharma wrote:
> > So, I was thinking of the following structure:
> > - Supported Platforms
> > - Getting the Source
> > - Building with make and autoconf
> >-- Short version
> >-- Requirements
> >-- Installation Procedure and it's subsections
> > - Building with Meson
> >-- Short version
> >-- Requirements
> >-- Installation Procedure and it's subsections
> > - Post-installation Setup
> > - Platform specific notes
>
> I like that.
>

Attached is a docs-only patch with that structure. We need to update the
platform specific notes section to add meson specific nuances. Also, in
terms of supported platforms, if there are platforms which work with make
but not with meson, we have to add that too.

Regards,
Samay

>
> > As a follow up patch, we could also try to fit the Windows part into
> > this model. We could add a Building with visual C++ or Microsoft windows
> > SDK section. It doesn't have a short version but follows the remaining
> > template of requirements and installation procedure subsections
> > (Building, Cleaning and Installing and Running Regression tests) well.
>
> We were thinking about removing the old Windows build system for PG 16.
> Let's see how that goes.  Otherwise, yes, that would be good as well.
>


v1-0001-Add-meson-docs-to-existing-build-from-source.patch
Description: Binary data

Re: [RFC] building postgres with meson - v12

2022-09-08 Thread Andres Freund

On 2022-09-08 14:10:45 +0700, John Naylor wrote:
> Okay, done that way.

Thanks!

Re: [RFC] building postgres with meson - v12

2022-09-08 Thread Alvaro Herrera

On 2022-Sep-07, Peter Eisentraut wrote:

> A possible drawback is that the intermediate postgres-full.xml file is
> >10MB, but I guess we're past the point where we are worrying about that
> kind of thing.

I think we are, but maybe mark it .PRECIOUS?  IIUC that would prevent it
from being removed if there's a problem in the other recipes.

-- 
Álvaro HerreraBreisgau, Deutschland  —  https://www.EnterpriseDB.com/
"¿Cómo puedes confiar en algo que pagas y que no ves,
y no confiar en algo que te dan y te lo muestran?" (Germán Poo)

Re: [RFC] building postgres with meson - v11

2022-09-08 Thread samay sharma

Hi,

On Tue, Sep 6, 2022 at 9:48 PM Peter Eisentraut <
peter.eisentr...@enterprisedb.com> wrote:

> On 02.09.22 19:16, samay sharma wrote:
> > Another thing I'd like input on. A common question I've heard from
> > people who've tried out the docs is How do we do the equivalent of X in
> > make with meson. As meson will be new for a bunch of people who are
> > fluent with make, I won't be surprised if this is a common ask. To
> > address that, I was planning to add a page to specify the key things one
> > needs to keep in mind while "migrating" from make to meson and having a
> > translation table of commonly used commands.
> >
> > I was planning to add it in the meson section, but if we go ahead with
> > the structure proposed above, it doesn't fit it into one as cleanly.
> > Maybe, it still goes in the meson section? Thoughts?
>
> This could go into the wiki.
>
> For example, we have
> , which was added
> during the CVS->Git transition.
>

That's a good idea. I'll add a page to the wiki about this topic and share
it on the list for review.


>
> This avoids that we make the PostgreSQL documentation a substitute
> manual for a third-party product.
>

Regards,
Samay

Re: [RFC] building postgres with meson - v12

2022-09-08 Thread John Naylor

On Wed, Sep 7, 2022 at 3:36 PM Alvaro Herrera  wrote:
>
> On 2022-Sep-06, John Naylor wrote:
>
> > Note that the indentation hasn't changed. My thought there: perltidy
> > will be run again next year, at which time it will be part of a listed
> > whitespace-only commit. Any objections, since that could confuse
> > someone before then?
>
> I think a good plan is to commit the fix without tidy, then commit the
> tidy separately, then add the latter commit to .git-blame-ignore-revs.
> That avoids leaving the code untidy for a year.

Okay, done that way. I also made sure we got the same info for error
reporting. It's not identical, but arguably better, going from:

Bareword found where operator expected at (eval 4480) line 3, near "'btree' xxx"
(Missing operator before xxx?)
../../../src/include/catalog/pg_amop.dat: error parsing line 20:

to:

Bareword found where operator expected at (eval 12) line 20, near "'btree' xxx"
(Missing operator before xxx?)
error parsing ../../../src/include/catalog/pg_amop.dat

-- 
John Naylor
EDB: http://www.enterprisedb.com

Re: [RFC] building postgres with meson - v12

2022-09-07 Thread Alvaro Herrera

On 2022-Sep-06, John Naylor wrote:

> Note that the indentation hasn't changed. My thought there: perltidy
> will be run again next year, at which time it will be part of a listed
> whitespace-only commit. Any objections, since that could confuse
> someone before then?

I think a good plan is to commit the fix without tidy, then commit the
tidy separately, then add the latter commit to .git-blame-ignore-revs.
That avoids leaving the code untidy for a year.

-- 
Álvaro HerreraBreisgau, Deutschland  —  https://www.EnterpriseDB.com/
 Are you not unsure you want to delete Firefox?
   [Not unsure] [Not not unsure][Cancel]
   http://smylers.hates-software.com/2008/01/03/566e45b2.html

Re: [RFC] building postgres with meson - v12

2022-09-07 Thread Peter Eisentraut


On 07.09.22 09:19, Peter Eisentraut wrote:
This could also be combined with the idea of the postgres.sgml.valid 
thing you have in the meson patch set.


I'll finish this up and produce a proper patch.


Something like this.

This does make the rules more straightforward and avoids repeated 
xmllint calls.  I suppose this also helps writing the meson rules in a 
simpler way.


A possible drawback is that the intermediate postgres-full.xml file is 
>10MB, but I guess we're past the point where we are worrying about 
that kind of thing.


I don't know if there is any performance difference between xsltproc 
reading one big file versus many smaller files.From 6aa880a6c27de3bce412f24ed3de0c3926e7be04 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut 
Date: Wed, 7 Sep 2022 09:50:27 +0200
Subject: [PATCH] Run xmllint validation only once

---
 doc/src/sgml/Makefile | 44 +++
 1 file changed, 24 insertions(+), 20 deletions(-)

diff --git a/doc/src/sgml/Makefile b/doc/src/sgml/Makefile
index 4ae7ca2be7..b739a20b6b 100644
--- a/doc/src/sgml/Makefile
+++ b/doc/src/sgml/Makefile
@@ -61,15 +61,22 @@ ALLSGML := $(wildcard $(srcdir)/*.sgml 
$(srcdir)/ref/*.sgml) $(GENERATED_SGML)
 ALL_IMAGES := $(wildcard $(srcdir)/images/*.svg)
 
 
+# Run validation only once, common to all subsequent targets.  While
+# we're at it, also resolve all entities (that is, copy all included
+# files into one big file).  This helps tools that don't understand
+# vpath builds (such as dbtoepub).
+postgres-full.xml: postgres.sgml $(ALLSGML)
+   $(XMLLINT) $(XMLINCLUDE) --output $@ --noent --valid $<
+
+
 ##
 ## Man pages
 ##
 
 man distprep-man: man-stamp
 
-man-stamp: stylesheet-man.xsl postgres.sgml $(ALLSGML)
-   $(XMLLINT) $(XMLINCLUDE) --noout --valid $(word 2,$^)
-   $(XSLTPROC) $(XMLINCLUDE) $(XSLTPROCFLAGS) $(XSLTPROC_MAN_FLAGS) 
$(wordlist 1,2,$^)
+man-stamp: stylesheet-man.xsl postgres-full.xml
+   $(XSLTPROC) $(XSLTPROCFLAGS) $(XSLTPROC_MAN_FLAGS) $^
touch $@
 
 
@@ -117,7 +124,7 @@ INSTALL.html: %.html : stylesheet-text.xsl %.xml
$(XMLLINT) --noout --valid $*.xml
$(XSLTPROC) $(XSLTPROCFLAGS) $(XSLTPROC_HTML_FLAGS) $^ >$@
 
-INSTALL.xml: standalone-profile.xsl standalone-install.xml postgres.sgml 
$(ALLSGML)
+INSTALL.xml: standalone-profile.xsl standalone-install.xml postgres-full.xml
$(XSLTPROC) $(XMLINCLUDE) $(XSLTPROCFLAGS) --xinclude $(wordlist 
1,2,$^) >$@
 
 
@@ -131,8 +138,7 @@ endif
 
 html: html-stamp
 
-html-stamp: stylesheet.xsl postgres.sgml $(ALLSGML) $(ALL_IMAGES)
-   $(XMLLINT) $(XMLINCLUDE) --noout --valid $(word 2,$^)
+html-stamp: stylesheet.xsl postgres-full.xml $(ALL_IMAGES)
$(XSLTPROC) $(XMLINCLUDE) $(XSLTPROCFLAGS) $(XSLTPROC_HTML_FLAGS) 
$(wordlist 1,2,$^)
cp $(ALL_IMAGES) html/
cp $(srcdir)/stylesheet.css html/
@@ -140,16 +146,14 @@ html-stamp: stylesheet.xsl postgres.sgml $(ALLSGML) 
$(ALL_IMAGES)
 
 htmlhelp: htmlhelp-stamp
 
-htmlhelp-stamp: stylesheet-hh.xsl postgres.sgml $(ALLSGML) $(ALL_IMAGES)
-   $(XMLLINT) $(XMLINCLUDE) --noout --valid $(word 2,$^)
+htmlhelp-stamp: stylesheet-hh.xsl postgres-full.xml $(ALL_IMAGES)
$(XSLTPROC) $(XMLINCLUDE) $(XSLTPROCFLAGS) $(wordlist 1,2,$^)
cp $(ALL_IMAGES) htmlhelp/
cp $(srcdir)/stylesheet.css htmlhelp/
touch $@
 
 # single-page HTML
-postgres.html: stylesheet-html-nochunk.xsl postgres.sgml $(ALLSGML) 
$(ALL_IMAGES)
-   $(XMLLINT) $(XMLINCLUDE) --noout --valid $(word 2,$^)
+postgres.html: stylesheet-html-nochunk.xsl postgres-full.xml $(ALL_IMAGES)
$(XSLTPROC) $(XMLINCLUDE) $(XSLTPROCFLAGS) $(XSLTPROC_HTML_FLAGS) -o $@ 
$(wordlist 1,2,$^)
 
 # single-page text
@@ -166,13 +170,11 @@ postgres.pdf:
 
 XSLTPROC_FO_FLAGS += --stringparam img.src.path '$(srcdir)/'
 
-%-A4.fo: stylesheet-fo.xsl %.sgml $(ALLSGML)
-   $(XMLLINT) $(XMLINCLUDE) --noout --valid $(word 2,$^)
-   $(XSLTPROC) $(XMLINCLUDE) $(XSLTPROCFLAGS) $(XSLTPROC_FO_FLAGS) 
--stringparam paper.type A4 -o $@ $(wordlist 1,2,$^)
+%-A4.fo: stylesheet-fo.xsl %-full.xml
+   $(XSLTPROC) $(XMLINCLUDE) $(XSLTPROCFLAGS) $(XSLTPROC_FO_FLAGS) 
--stringparam paper.type A4 -o $@ $^
 
-%-US.fo: stylesheet-fo.xsl %.sgml $(ALLSGML)
-   $(XMLLINT) $(XMLINCLUDE) --noout --valid $(word 2,$^)
-   $(XSLTPROC) $(XMLINCLUDE) $(XSLTPROCFLAGS) $(XSLTPROC_FO_FLAGS) 
--stringparam paper.type USletter -o $@ $(wordlist 1,2,$^)
+%-US.fo: stylesheet-fo.xsl %-full.xml
+   $(XSLTPROC) $(XMLINCLUDE) $(XSLTPROCFLAGS) $(XSLTPROC_FO_FLAGS) 
--stringparam paper.type USletter -o $@ $^
 
 %.pdf: %.fo $(ALL_IMAGES)
$(FOP) -fo $< -pdf $@
@@ -183,8 +185,11 @@ XSLTPROC_FO_FLAGS += --stringparam img.src.path 
'$(srcdir)/'
 ##
 
 epub: postgres.epub
-postgres.epub: postgres.sgml $(ALLSGML) $(ALL_IMAGES)
-   $(XMLLINT) --noout --valid $<
+postgres.epub: postgres-full.xml $(ALL_IMAGES)
+ifeq ($(vpath_build),yes)
+   $(MKDIR_P) images
+   cp $(ALL_IMAGES)

Re: [RFC] building postgres with meson - v12

2022-09-07 Thread Peter Eisentraut


On 31.08.22 20:11, Andres Freund wrote:

doc/src/sgml/resolv.xsl: I don't understand what this is doing.  Maybe
at least add a comment in the file.

It's only used for building epubs. Perhaps I should extract that into a
separate patch as well? The relevant section is:


#
# epub
#

# This was previously implemented using dbtoepub - but that doesn't seem to
# support running in build != source directory (i.e. VPATH builds already
# weren't supported).
if pandoc.found() and xsltproc.found()
   # XXX: Wasn't able to make pandoc successfully resolve entities
   # XXX: Perhaps we should just make all targets use this, to avoid repeatedly
   # building whole thing? It's comparatively fast though.
   postgres_full_xml = custom_target('postgres-full.xml',
 input: ['resolv.xsl', 'postgres.sgml'],
 output: ['postgres-full.xml'],
 depends: doc_generated + [postgres_sgml_valid],
 command: [xsltproc, '--path', '@OUTDIR@/', xsltproc_flags,
   '-o', '@OUTPUT@', '@INPUT@'],
 build_by_default: false,
   )

A noted, I couldn't make pandoc resolve our entities, so I used resolv.xsl
them, before calling pandoc.

I'll rename it to resolve-entities.xsl and add a comment.


We can have xmllint do this.  The following gets the epub build working 
with vpath:


diff --git a/doc/src/sgml/Makefile b/doc/src/sgml/Makefile
index 4ae7ca2be7..33b72d03db 100644
--- a/doc/src/sgml/Makefile
+++ b/doc/src/sgml/Makefile
@@ -184,8 +184,12 @@ XSLTPROC_FO_FLAGS += --stringparam img.src.path 
'$(srcdir)/'


 epub: postgres.epub
 postgres.epub: postgres.sgml $(ALLSGML) $(ALL_IMAGES)
-   $(XMLLINT) --noout --valid $<
-   $(DBTOEPUB) -o $@ $<
+   $(XMLLINT) $(XMLINCLUDE) --output tmp.sgml --noent --valid $<
+ifeq ($(vpath_build),yes)
+   $(MKDIR_P) images
+   cp $(ALL_IMAGES) images/
+endif
+   $(DBTOEPUB) -o $@ tmp.sgml


This could also be combined with the idea of the postgres.sgml.valid 
thing you have in the meson patch set.


I'll finish this up and produce a proper patch.

Re: [RFC] building postgres with meson - v12


On 02.09.22 18:57, Andres Freund wrote:

Is it worth running ninja -t missingdeps as a test? At the time we run tests
we'll obviously have built and thus collected "real" dependencies, so we would
have the necessary information to determine whether dependencies are missing.
I think it'd be fine to do so only for ninja >= 1.11, rather than falling back
to the llvm python implementation, which is much slower (0.068s vs
3.760s). And also because it's not as obvious how to include the python script.

Alternatively, we could just document that ninja -t missingdeps is worth
running. Perhaps at the top of the toplevel build.meson file?


In the GNU/make world there is a distinction between "check" and 
"maintainer-check" for this kind of thing.


I think here if we put these kinds of things into a different, what's 
the term, "suite", then that would be a clear way to collect them and be 
able to run them all easily.

Re: [RFC] building postgres with meson - v12




On 31.08.22 20:11, Andres Freund wrote:

src/port/win32ver.rc.in: This is redundant with src/port/win32ver.rc.
(Note that the latter is also used as an input file for text
substitution.  So having another file named *.in next to it would be
super confusing.)

Yea, this stuff isn't great. I think the better solution, both for meson and
for configure, would be to move to do all the substitution to the C
preprocessor.


Yeah, I think if we can get rid of the evil date-based versioning, then
this could be done like

diff --git a/src/makefiles/Makefile.win32 b/src/makefiles/Makefile.win32
index 17d6819644..609156382f 100644
--- a/src/makefiles/Makefile.win32
+++ b/src/makefiles/Makefile.win32
@@ -65,21 +65,12 @@ endif
 # win32ver.rc or furnish a rule for generating one.  Set $(PGFILEDESC) to
 # signal win32ver.rc availability to the dll build rule below.
 ifndef PGXS
-win32ver.rc: $(top_srcdir)/src/port/win32ver.rc
-   sed -e 's;FILEDESC;$(PGFILEDESC);' \
-   -e 's;VFT_APP;$(PGFTYPE);' \
-   -e 's;_ICO_;$(PGICOSTR);' \
-   -e 's;\(VERSION.*\),0 *$$;\1,'`date '+%y%j' | sed 's/^0*//'`';' \
-   -e '/_INTERNAL_NAME_/$(if $(shlib),s;_INTERNAL_NAME_;"$(basename 
$(shlib))";,d)' \
-   -e '/_ORIGINAL_NAME_/$(if $(shlib),s;_ORIGINAL_NAME_;"$(shlib)";,d)' \
- $< >$@
-
 # Depend on Makefile.global to force rebuild on re-run of configure.
 win32ver.rc: $(top_builddir)/src/Makefile.global
 endif

-win32ver.o: win32ver.rc
-   $(WINDRES) -i $< -o $@ --include-dir=$(top_builddir)/src/include 
--include-dir=$(srcdir)
+win32ver.o: $(top_srcdir)/src/port/win32ver.rc
+   $(WINDRES) -i $< -o $@ --include-dir=$(top_builddir)/src/include --include-dir=$(srcdir) -D 
FILEDESC=$(PGFILEDESC) -D VFT_APP=$(PGFTYPE) -D_ICO_=$(PGICOSTR) -D_INTERNAL_NAME_=$(if 
$(shlib),s;_INTERNAL_NAME_;"$(basename $(shlib))";,d) -D_ORIGINAL_NAME_=$(if 
$(shlib),s;_ORIGINAL_NAME_;"$(shlib)";,d)


Probably needs some careful checking of the quoting.  But that should be
the right thing in principle.

Re: [RFC] building postgres with meson - v11


On 02.09.22 19:16, samay sharma wrote:
Another thing I'd like input on. A common question I've heard from 
people who've tried out the docs is How do we do the equivalent of X in 
make with meson. As meson will be new for a bunch of people who are 
fluent with make, I won't be surprised if this is a common ask. To 
address that, I was planning to add a page to specify the key things one 
needs to keep in mind while "migrating" from make to meson and having a 
translation table of commonly used commands.


I was planning to add it in the meson section, but if we go ahead with 
the structure proposed above, it doesn't fit it into one as cleanly. 
Maybe, it still goes in the meson section? Thoughts?


This could go into the wiki.

For example, we have 
, which was added 
during the CVS->Git transition.


This avoids that we make the PostgreSQL documentation a substitute 
manual for a third-party product.

Re: [RFC] building postgres with meson - v11