Re[2]: Alter index rename concurrently to

2018-10-03 Thread Andrey Klychkov
> Based on these assertions, here is my proposed patch. It lowers the > lock level for index renaming to ShareUpdateExclusiveLock. Hi, Peter I made small review for your patch: Server source code got from https://github.com/postgres/postgres.git 1. Patch was applied without any errors except a

Re: [RFC] Removing "magic" oids

2018-10-03 Thread Andreas Karlsson
On 09/30/2018 05:48 AM, Andres Freund wrote:> I think we should drop WITH OIDs support. +1 2) We need to be able to manually insert into catalog tables. Both initdb and emergency surgery. My current hack is doing so directly in nodeModifyTable.c but that's beyond ugly. I think we

Re: BUG #15307: Low numerical precision of (Co-) Variance

2018-10-03 Thread Dean Rasheed
On Thu, 27 Sep 2018 at 14:22, Dean Rasheed wrote: > I'll post an updated patch in a while. > I finally got round to looking at this again. Here is an update, based on the review comments. The next question is whether or not to back-patch this. Although this was reported as a bug, my inclination

Re: BUG #15307: Low numerical precision of (Co-) Variance

2018-10-03 Thread Madeleine Thompson
On Wed, Oct 3, 2018 at 9:04 AM Dean Rasheed wrote: > > On Thu, 27 Sep 2018 at 14:22, Dean Rasheed wrote: > > I'll post an updated patch in a while. > > > > I finally got round to looking at this again. Here is an update, based > on the review comments. > > The next question is whether or not to

Re: partition tree inspection functions

2018-10-03 Thread Jesper Pedersen
Hi Michael, On 10/2/18 11:37 PM, Michael Paquier wrote: isleaf is not particularly useful as it can just be fetched with a join on pg_class/relkind. So I think that we had better remove it. Removing isleaf would require extra round trips to the server to get that information. So, I think

Re: BUG #15307: Low numerical precision of (Co-) Variance

2018-10-03 Thread Tom Lane
Dean Rasheed writes: > The next question is whether or not to back-patch this. Although this > was reported as a bug, my inclination is to apply this to HEAD only, > based on the lack of prior complaints. That also matches our decision > for other similar patches, e.g., 7d9a4737c2. +1 for no

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Andres Freund
Hi, On 2018-10-03 11:59:27 -0400, Tom Lane wrote: > I wrote: > > ... However, I did add recent glibc (Fedora 28) > > to the mix, and I was interested to discover that they seem to have > > added a fast-path for format strings that are exactly "%s", just as > > NetBSD did. I wonder if we should

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
Andres Freund writes: > On 2018-10-03 11:59:27 -0400, Tom Lane wrote: >> I experimented with adding an initial check for "format is exactly %s" >> at the top of dopr(), and couldn't get excited about that. Instrumenting >> things showed that the optimization fired in only 1.8% of the calls >>

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
I wrote: > Andres Freund writes: >> - I know it's not new, but is it actually correct to use va_arg(args, int64) >> for ATYPE_LONGLONG? > Well, the problem with just doing s/int64/long long/g is that the > code would then fail on compilers without a "long long" type. > We could ifdef our way

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
I wrote: > ... However, I did add recent glibc (Fedora 28) > to the mix, and I was interested to discover that they seem to have > added a fast-path for format strings that are exactly "%s", just as > NetBSD did. I wonder if we should reconsider our position on doing > that. It'd be a simple

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
Andres Freund writes: > FWIW, it seems that using a local buffer and than pstrdup'ing that in > float8out_internal is a bit faster, and would probably save a bit of > memory on average: > float8out using sprintf via pg_double_to_string, pstrdup: > 15370.774 > float8out using strfromd via

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
Andres Freund writes: > On 2018-10-03 12:07:32 -0400, Tom Lane wrote: >> [ scratches head ... ] How would that work? Seems like it necessarily >> adds a strlen() call to whatever we'd be doing otherwise. palloc isn't >> going to be any faster just from asking it for slightly fewer bytes. >> I

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
Andres Freund writes: > On 2018-10-03 12:54:52 -0400, Tom Lane wrote: >> (1) The need to use sprintf for portability means that we need very >> tight constraints on the precision spec *and* the buffer size *and* >> the format type (%f pretty much destroys certainty about how long the >> output

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
Andres Freund writes: > On 2018-10-03 13:18:35 -0400, Tom Lane wrote: >> Having said that, maybe there's a case for changing the type spec >> in only the va_arg() call, and leaving snprintf's related local >> variables as int64. (Is that what you actually meant?) Then, >> if long long really is

Re: Early WIP/PoC for inlining CTEs

2018-10-03 Thread David Fetter
On Tue, Oct 02, 2018 at 12:03:06PM +0200, Andreas Karlsson wrote: > Hi, > > Here is an updated patch which adds some simple syntax for adding the > optimization barrier. For example: > > WITH x AS MATERIALIZED ( >SELECT 1 > ) > SELECT * FROM x; > > Andreas This is great! Is there any

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Andres Freund
Hi, On 2018-10-03 08:20:14 -0400, Tom Lane wrote: > Andres Freund writes: > >> While there might be value in implementing our own float printing code, > >> I have a pretty hard time getting excited about the cost/benefit ratio > >> of that. I think that what we probably really ought to do here

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
Andres Freund writes: > It seems the general "use strfromd if available" approach is generally > useful, even if we need to serialize the precision. Agreed. > Putting it into an > inline appears to be helpful, avoids some of the otherwise precision > related branches. Do you have any feelings

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Andres Freund
Hi, On 2018-10-03 12:54:52 -0400, Tom Lane wrote: > Andres Freund writes: > > It seems the general "use strfromd if available" approach is generally > > useful, even if we need to serialize the precision. > > Agreed. > > > Putting it into an > > inline appears to be helpful, avoids some of the

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Andres Freund
On 2018-10-03 13:40:03 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-10-03 13:31:09 -0400, Tom Lane wrote: > >> I do not see the point of messing with snprintf.c here. I doubt that > >> strfromd is faster than the existing sprintf call (because the latter > >> can use ".*" instead

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Andres Freund
Hi, On 2018-10-03 13:18:35 -0400, Tom Lane wrote: > I wrote: > > Andres Freund writes: > >> - I know it's not new, but is it actually correct to use va_arg(args, > >> int64) > >> for ATYPE_LONGLONG? > > > Well, the problem with just doing s/int64/long long/g is that the > > code would then

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
Andres Freund writes: > On 2018-10-03 13:31:09 -0400, Tom Lane wrote: >> I do not see the point of messing with snprintf.c here. I doubt that >> strfromd is faster than the existing sprintf call (because the latter >> can use ".*" instead of serializing and deserializing the precision). > I'm

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
Andres Freund writes: > So when using pg's snprintf() to print a single floating point number > with precision, we get nearly a 10% boost. I just tested that using my little standalone testbed, and I failed to replicate the result. I do see that strfromd is slightly faster, but it's just a few

Re: libpq host/hostaddr/conninfo inconsistencies

2018-10-03 Thread Arthur Zakirov
Sorry for late answer. On 9/30/18 10:21 AM, Fabien COELHO wrote: sh> psql "host=127.0.0.2 hostaddr=127.0.0.1" I'm not sure that is is the issue. User defined the host name and psql show it. The issue is that "host" is an ip, "\conninfo" will inform wrongly that you are connected to

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Andres Freund
Hi, On 2018-10-03 12:22:13 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-10-03 12:07:32 -0400, Tom Lane wrote: > >> [ scratches head ... ] How would that work? Seems like it necessarily > >> adds a strlen() call to whatever we'd be doing otherwise. palloc isn't > >> going to be

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Andres Freund
Hi, On 2018-10-03 13:31:09 -0400, Tom Lane wrote: > I do not see the point of messing with snprintf.c here. I doubt that > strfromd is faster than the existing sprintf call (because the latter > can use ".*" instead of serializing and deserializing the precision). I'm confused, the numbers I

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Alvaro Herrera
On 2018-Oct-03, Tom Lane wrote: > Andres Freund writes: > BTW, so far as I can tell on F28, strfromd isn't exposed without > "-D__STDC_WANT_IEC_60559_BFP_EXT__", which seems fairly scary; > what else does that affect? https://en.cppreference.com/w/c/experimental/fpext1 -- Álvaro Herrera

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Andres Freund
Hi, On 2018-10-03 14:01:35 -0400, Tom Lane wrote: > Andres Freund writes: > > So when using pg's snprintf() to print a single floating point number > > with precision, we get nearly a 10% boost. > > I just tested that using my little standalone testbed, and I failed > to replicate the result.

Shouldn't ExecShutdownNode() be called from standard_ExecutorFinish()?

2018-10-03 Thread Andres Freund
Hi, There's a few cases where by the time ExecutorFinish() is called, ExecShutdownNode() has not yet been called. As, among other reasons, ExecShutdownNode() also collects instrumentation, I think that's problematic. In ExplainOnePlan() we call /* run cleanup too */

Re: executor relation handling

2018-10-03 Thread Tom Lane
I wrote: > Amit Langote writes: >> Should this check that we're not in a parallel worker process? > Hmm. I've not seen any failures in the parallel parts of the regular > regression tests, but maybe I'd better do a force_parallel_mode > run before committing. > In general, I'm not on board with

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
I wrote: >> Hm. I guess that'd be a bit better, but I'm not sure it's worth it. How >> about we simply add a static assert that long long isn't bigger than >> int64? > WFM, I'll make it happen. Actually, while writing a comment to go with that assertion, I decided this was dumb. If we're

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
Andres Freund writes: > On 2018-10-03 14:01:35 -0400, Tom Lane wrote: >> BTW, so far as I can tell on F28, strfromd isn't exposed without >> "-D__STDC_WANT_IEC_60559_BFP_EXT__", which seems fairly scary; >> what else does that affect? > So I don't think anything's needed to enable that in pg,

RE: Global shared meta cache

2018-10-03 Thread Ideriha, Takeshi
Hi, Thank you for the previous discussion while ago. I’m afraid I haven't replied to all. To move forward this development I attached a PoC patch. I introduced a guc called shared_catacache_mem to specify how much memory is supposed be allocated on the shared memory area. It defaults to zero,

Re: [HACKERS] Transactions involving multiple postgres foreign servers, take 2

2018-10-03 Thread Chris Travers
The following review has been posted through the commitfest application: make installcheck-world: tested, failed Implements feature: not tested Spec compliant: not tested Documentation:tested, failed I am hoping I am not out of order in writing this before the

Re: pgsql: Improve autovacuum logging for aggressive and anti-wraparound ru

2018-10-03 Thread Masahiko Sawada
On Tue, Oct 2, 2018 at 11:15 PM Tom Lane wrote: > > Michael Paquier writes: > > My brain is rather fried for the rest of the day... But we could just > > be looking at using USE_ASSERT_CHECKING. Thoughts from other are > > welcome. > > I'd go with folding the condition into a plain Assert.

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Andres Freund
On 2018-10-02 17:54:31 -0400, Tom Lane wrote: > Here's a version of this patch rebased over commit 625b38ea0. > > That commit's fix for the possibly-expensive memset means that we need > to reconsider performance numbers for this patch. I re-ran my previous > tests, and it's still looking like

Re: [WIP PATCH] Index scan offset optimisation using visibility map

2018-10-03 Thread Michael Paquier
On Wed, Oct 03, 2018 at 10:54:14AM +0500, Andrey Borodin wrote: > Let's move this to CF 2018-11? Obviously, it is WiP, but it seems that > patch is being discussed, author cares about it. If you are still working on it, which is not something obvious based on the thread activity, then moving it

Re: file cloning in pg_upgrade and CREATE DATABASE

2018-10-03 Thread Michael Paquier
On Tue, Oct 02, 2018 at 11:07:06PM -0300, Alvaro Herrera wrote: > I see. Peter is proposing to have a fourth mode, essentially > --transfer-mode=clone-or-copy. Yes, a mode which depends on what the file system supports. Perhaps "safe" or "fast" could be another name, in the shape of the fastest

DROP DATABASE doesn't force other backends to close FDs

2018-10-03 Thread Andres Freund
Hi, Joerg reported on IRC that after a DROP DATABASE the space of the dropped database wasn't available to the OS until he killed a session that was active from before the drop. I was kinda incredulous at first, the code looks sane... Thomas figured out significant parts of this. dropdb() does

Re: Skylake-S warning

2018-10-03 Thread Andres Freund
Hi, On 2018-10-03 14:29:39 -0700, Daniel Wood wrote: > If running benchmarks or you are a customer which is currently > impacted by GetSnapshotData() on high end multisocket systems be wary > of Skylake-S. > Performance differences of nearly 2X can be seen on select only > pgbench due to

Re: Skylake-S warning

2018-10-03 Thread Daniel Wood
> On October 3, 2018 at 3:55 PM Andres Freund wrote: > In the thread around > https://www.postgresql.org/message-id/20160411214029.ce3fw6zxim5k6...@alap3.anarazel.de > I'd found doing more aggressive padding helped a lot. Unfortunately I > didn't pursue this further :( Interesting. Looks

Re: Add SKIP LOCKED to VACUUM and ANALYZE

2018-10-03 Thread Michael Paquier
On Wed, Oct 03, 2018 at 04:20:42PM +, Bossart, Nathan wrote: > Here is what I have so far for the docs: > > [snip] Thanks, I have committed the patch, after minor tweaks to the docs. Mainly, in the VACUUM page, I have added a mention that VACUUM ANALYZE can block for row sampling. I have

Re: partition tree inspection functions

2018-10-03 Thread Michael Paquier
On Wed, Oct 03, 2018 at 08:12:59AM -0400, Jesper Pedersen wrote: > Removing isleaf would require extra round trips to the server to get > that information. So, I think we should keep it. I don't really get your point about extra round trips with the server, and getting the same level of

RE: Function for listing archive_status directory

2018-10-03 Thread Iwata, Aya
Hi Christoph, I think it is convenient to be able to check the archive_status directory contents information. I reviewed patch. It applies and passes regression test. I checked the code. It refers to the patch which added pg_ls_waldir() and pg_ls_logdir(), so I think it is good. There is

Re: Skylake-S warning

2018-10-03 Thread Andres Freund
On 2018-10-03 17:01:44 -0700, Daniel Wood wrote: > > > On October 3, 2018 at 3:55 PM Andres Freund wrote: > > > In the thread around > > https://www.postgresql.org/message-id/20160411214029.ce3fw6zxim5k6...@alap3.anarazel.de > > I'd found doing more aggressive padding helped a lot.

Re: Skylake-S warning

2018-10-03 Thread Daniel Wood
One other thought. Could we update pgxact->xmin less often? What would be the impact of this lower bound being lower than it would normally be with the existing scheme. Yes, it needs to be moved forward "occasionally". FYI, be careful with padding PGXACT's to a full cache line. With 1024

Re: Add SKIP LOCKED to VACUUM and ANALYZE

2018-10-03 Thread Bossart, Nathan
On 10/3/18, 7:10 PM, "Michael Paquier" wrote: > Thanks, I have committed the patch, after minor tweaks to the docs. Thanks! > Mainly, in the VACUUM page, I have added a mention that VACUUM ANALYZE > can block for row sampling. I have also removed for now the set of > recommendations the patch

Re: [RFC] Removing "magic" oids

2018-10-03 Thread Vik Fearing
On 30/09/18 05:48, Andres Freund wrote: > 5a) The fact that system table oids don't show up in selects by default >makes it more work than necessary to look at catalogs. As a user, this is a big pain point for me. +1 for getting rid of it. -- Vik Fearing

Re: SerializeParamList vs machines with strict alignment

2018-10-03 Thread Amit Kapila
On Wed, Oct 3, 2018 at 10:02 AM Amit Kapila wrote: > > > > Now chipmunk also failed for the same test. > > > > > What I find more interesting is > > > that both of the live Sparc critters are happy --- so despite > > > Thomas' statements upthread, they're coping with unaligned accesses. > > >

Re: Performance improvements for src/port/snprintf.c

2018-10-03 Thread Tom Lane
Andres Freund writes: > On 2018-10-02 17:54:31 -0400, Tom Lane wrote: >> Here's a version of this patch rebased over commit 625b38ea0. > Cool. Let's get that in... Cool, I'll push it shortly. >> While there might be value in implementing our own float printing code, >> I have a pretty hard

Re: Query is over 2x slower with jit=on

2018-10-03 Thread Andres Freund
Hi, On 2018-10-01 00:32:18 -0700, Lukas Fittl wrote: > On Tue, Sep 25, 2018 at 1:17 PM, Andres Freund wrote: > > > > I've pushed the change without that bit - it's just a few additional > > lines if we want to change that. > > > > It seems that since this commit, JIT statistics are now only

Skylake-S warning

2018-10-03 Thread Daniel Wood
If running benchmarks or you are a customer which is currently impacted by GetSnapshotData() on high end multisocket systems be wary of Skylake-S. Performance differences of nearly 2X can be seen on select only pgbench due to nothing else but unlucky choices for max_connections. Scale 1000,

Re: [HACKERS] Secondary index access optimizations

2018-10-03 Thread David Rowley
On 12 September 2018 at 08:32, Konstantin Knizhnik wrote: > Also the patch proposed by you is much simple and does mostly the same. Yes, > it is not covering CHECK constraints, I started to look at this and found a problem in regards to varno during the predicate_implied_by() test. The problem

Re: Shouldn't ExecShutdownNode() be called from standard_ExecutorFinish()?

2018-10-03 Thread Amit Kapila
On Thu, Oct 4, 2018 at 12:33 AM Andres Freund wrote: > > Hi, > > There's a few cases where by the time ExecutorFinish() is called, > ExecShutdownNode() has not yet been called. As, among other reasons, > ExecShutdownNode() also collects instrumentation, I think that's > problematic. > > In

RE: Protect syscache from bloating with negative cache entries

2018-10-03 Thread Ideriha, Takeshi
Hi, thank you for the explanation. >From: Kyotaro HORIGUCHI [mailto:horiguchi.kyot...@lab.ntt.co.jp] >> >> Can I confirm about catcache pruning? >> syscache_memory_target is the max figure per CatCache. >> (Any CatCache has the same max value.) So the total max size of >> catalog caches is

Re: [HACKERS] Transactions involving multiple postgres foreign servers, take 2

2018-10-03 Thread Chris Travers
On Wed, Oct 3, 2018 at 9:56 AM Chris Travers wrote: > > > On Wed, Oct 3, 2018 at 9:41 AM Chris Travers > wrote: > >> >> (errmsg("preparing foreign transactions > (max_prepared_foreign_transactions > 0) requires maX_foreign_xact_resolvers > > 0"))); > Two more critical notes here which I think

Re: Segfault when creating partition with a primary key and sql_drop trigger exists

2018-10-03 Thread Michael Paquier
On Fri, Sep 28, 2018 at 12:17:00PM +0900, Michael Paquier wrote: > I think that Alvaro should definitely look at this patch to be sure, or > I could do it, but I would need to spend way more time on this and check > event trigger interactions. > > Anyway, I was struggling a bit regarding the

Re: pgsql: Improve autovacuum logging for aggressive and anti-wraparound ru

2018-10-03 Thread Michael Paquier
On Wed, Oct 03, 2018 at 04:37:29PM +0900, Masahiko Sawada wrote: > Thank you for the comment. Attached the updated patch. Thanks for the patch. This looks clean to me at quick glance, not tested though.. -- Michael signature.asc Description: PGP signature

Re: [HACKERS] Transactions involving multiple postgres foreign servers, take 2

2018-10-03 Thread Chris Travers
On Wed, Oct 3, 2018 at 9:41 AM Chris Travers wrote: > The following review has been posted through the commitfest application: > make installcheck-world: tested, failed > Implements feature: not tested > Spec compliant: not tested > Documentation:tested, failed >

Re: [HACKERS] Transactions involving multiple postgres foreign servers, take 2

2018-10-03 Thread Chris Travers
On Wed, Oct 3, 2018 at 9:41 AM Chris Travers wrote: > The following review has been posted through the commitfest application: > make installcheck-world: tested, failed > Implements feature: not tested > Spec compliant: not tested > Documentation:tested, failed > > I