Re: Questions/Observations related to Gist vacuum

2019-10-15 Thread Dilip Kumar
On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas wrote: > > On 15/10/2019 09:37, Amit Kapila wrote: > > While reviewing a parallel vacuum patch [1], we noticed a few things > > about $SUBJECT implemented in commit - > > 7df159a620b760e289f1795b13542ed1b3e13b87. > > > > 1. A new memory context

Re: let's make the list of reportable GUCs configurable (was Re: Add %r substitution for psql prompts to show recovery status)

2019-10-15 Thread Dave Cramer
On Sat, 12 Oct 2019 at 05:05, Tom Lane wrote: > Andres Freund writes: > > On 2019-10-11 16:30:17 -0400, Robert Haas wrote: > >> But, if it does need to be changed, it seems like a terrible idea to > >> allow it to be done via SQL. Otherwise, the user can break the driver > >> by using SQL to

Re: pgbench - extend initialization phase control

2019-10-15 Thread btendouan
Here is rebase v3. Hi, Thanks for your new patch. Failed regression test. It's necessary to change the first a in “allowed step characters are” to uppercase A in the regression test of 002_pgbench_no_server.pl. The behavior of "g" is different between v12 and the patche, and backward

Re: Collation versioning

2019-10-15 Thread Thomas Munro
On Wed, Oct 16, 2019 at 5:33 PM Thomas Munro wrote: > On Tue, Oct 15, 2019 at 5:39 PM Thomas Munro wrote: > > Here's a version with a small note added to the documentation. I'm > > planning to commit this tomorrow. > > Done. The buildfarm is telling me that I didn't test this with the full set

Re: Collation versioning

2019-10-15 Thread Thomas Munro
On Tue, Oct 15, 2019 at 5:39 PM Thomas Munro wrote: > Here's a version with a small note added to the documentation. I'm > planning to commit this tomorrow. Done. It's not much, but it's a start. Some things to do: * handle default collation (probably comes with CF entry 2256?) * preserve

Re: Ordering of header file inclusion

2019-10-15 Thread vignesh C
On Wed, Oct 16, 2019 at 8:10 AM Amit Kapila wrote: > > Thanks for working on this. I will look into this in the coming few > days or during next CF. Can you please register it for the next CF > (https://commitfest.postgresql.org/25/)? > Thanks, I have added it to the commitfest. Regards,

RE: Copy data to DSA area

2019-10-15 Thread ideriha.take...@fujitsu.com
Hi, Sorry for waiting. >Thomas Munro wrote: >>What do you think about the following? Even though I know you want to >>start with much simpler kinds of cache, I'm looking ahead to the lofty >>end-goal of having a shared plan cache. No doubt, that involves >>solving many other problems that

Re: Ordering of header file inclusion

2019-10-15 Thread Amit Kapila
On Tue, Oct 15, 2019 at 10:57 PM vignesh C wrote: > > On Wed, Oct 9, 2019 at 11:37 AM Amit Kapila wrote: > > > > On Tue, Oct 8, 2019 at 8:19 PM Tom Lane wrote: > > > > > > Amit Kapila writes: > > > > On Wed, Oct 2, 2019 at 2:57 PM vignesh C wrote: > > > >> I noticed that some of the header

Re: maintenance_work_mem used by Vacuum

2019-10-15 Thread Masahiko Sawada
On Sat, Oct 12, 2019 at 8:45 PM Amit Kapila wrote: > > On Sat, Oct 12, 2019 at 10:49 AM Masahiko Sawada > wrote: > > > > On Fri, Oct 11, 2019 at 5:13 PM Amit Kapila wrote: > > > > > > That's right, but OTOH, if the user specifies gin_pending_list_limit > > > as an option during Create Index

Understanding TupleQueue impact and overheads?

2019-10-15 Thread Tom Mercha
I have been looking at PostgreSQL's Tuple Queue (/include/executor/tqueue.h) which provides functionality for queuing tuples between processes through shm_mq. I am still familiarising myself with the bigger picture and TupTableStores. I can see that a copy (not a reference) of a HeapTuple

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Masahiko Sawada
On Tue, Oct 15, 2019 at 6:33 PM Amit Kapila wrote: > > On Tue, Oct 15, 2019 at 1:26 PM Masahiko Sawada wrote: > > > > On Tue, Oct 15, 2019 at 4:15 PM Masahiko Sawada > > wrote: > > > > > > > > > If we avoid postponing deleting empty pages till the cleanup phase, > > > > > > then we don't have

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread David Rowley
On Wed, 16 Oct 2019 at 11:40, Justin Pryzby wrote: > It didn't occur to me at the time, but that would also allow > creating numerous, partial BRIN indices, each of which would have separate > correlation computed over just their "restricted range", which *might* also > handle your case,

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread David Rowley
On Wed, 16 Oct 2019 at 05:05, Jeremy Finzel wrote: > But perhaps it would be worth exploring if there could be more detailed stats > on physical vs logical correlation, such as when ANALYZE takes its samples, > noting physical locations as well as logical values, and allowing the > correlation

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread Justin Pryzby
This reminds me of an issue I reported several years ago where Btree index scans were chosen over seq scan of a large, INSERT-only table due to very high correlation, but performed poorly. I concluded that use of the the high "large scale" correlation on a large 50+GB table caused the planner to

Re: v12.0 ERROR: trying to store a heap tuple into wrong type of slot

2019-10-15 Thread Justin Pryzby
On Tue, Oct 15, 2019 at 01:50:09PM -0700, Andres Freund wrote: > On 2019-10-13 07:51:06 -0700, Andres Freund wrote: > > On 2019-10-11 16:03:20 -0500, Justin Pryzby wrote: > > > ts=# CLUSTER huawei_m2000_config_enodebcell_enodeb USING > > > huawei_m2000_config_enodebcell_enodeb_coalesce_idx ; > >

Re: v12.0 ERROR: trying to store a heap tuple into wrong type of slot

2019-10-15 Thread Andres Freund
Hi, On 2019-10-13 07:51:06 -0700, Andres Freund wrote: > On 2019-10-11 16:03:20 -0500, Justin Pryzby wrote: > > I'm not sure why we have that index, and my script probably should have > > known > > to choose a better one to cluster on, but still.. > > > > ts=# CLUSTER

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread Michael Lewis
Thanks for closing the loop on the data correlation question. I've been playing with BRIN indexes on a log table of sorts and this thread helped clear up some of the behavior I have been seeing. I am curious, would a partial btree index fit your needs? Perhaps the maintenance overhead is too

v12.0: interrupt reindex CONCURRENTLY: ccold: ERROR: could not find tuple for parent of relation ...

2019-10-15 Thread Justin Pryzby
On a badly-overloaded VM, we hit the previously-reported segfault in progress reporting. This left around some *ccold indices. I tried to drop them but: sentinel=# DROP INDEX child.alarms_null_alarm_id_idx1_ccold; -- child.alarms_null_alarm_time_idx_ccold; -- alarms_null_alarm_id_idx_ccold;

Re: BRIN index which is much faster never chosen by planner

2019-10-15 Thread Jeremy Finzel
Thank you for the thorough and thoughtful reply! Please see below. On Mon, Oct 14, 2019 at 3:48 PM David Rowley wrote: > Another thing which you might want to look at is the correlation > column in the pg_stats view for the rec_insert_time column. Previous > to 7e534adcd, BRIN index were

Re: Questions/Observations related to Gist vacuum

2019-10-15 Thread Heikki Linnakangas
On 15/10/2019 09:37, Amit Kapila wrote: While reviewing a parallel vacuum patch [1], we noticed a few things about $SUBJECT implemented in commit - 7df159a620b760e289f1795b13542ed1b3e13b87. 1. A new memory context GistBulkDeleteResult->page_set_context has been introduced, but it doesn't seem

Re: Fix most -Wundef warnings

2019-10-15 Thread Andrew Gierth
> "Mark" == Mark Dilger writes: >> +#ifdef HSTORE_IS_HSTORE_NEW Mark> Checking the current sources, git history, and various older Mark> commits, I did not find where HSTORE_IS_HSTORE_NEW was ever Mark> defined. In contrib/hstore, it never was. The current version of contrib/hstore

Re: tuplesort test coverage

2019-10-15 Thread Peter Geoghegan
On Sun, Oct 13, 2019 at 3:41 PM Andres Freund wrote: > - cluster for expression indexes (line 935) We've never had coverage of this, but perhaps that can be added now. > - sorts exceeding INT_MAX / 2 memory (line 1337), but that seems hard to > test realistically I don't think that that can

Re: Zedstore - compressed in-core columnar storage

2019-10-15 Thread Ashutosh Sharma
Hi, I got chance to spend some time looking into the recent changes done in the zedstore code, basically the functions for packing datums into the attribute streams and handling attribute leaf pages. I didn't find any issues but there are some minor comments that I found when reviewing. I have

Re: configure fails for perl check on CentOS8

2019-10-15 Thread Kyotaro Horiguchi
Hi. Sorry for the delay. At Thu, 10 Oct 2019 11:51:21 -0400, Tom Lane wrote in > Andrew Dunstan writes: > > On 10/10/19 1:46 AM, Kyotaro Horiguchi wrote: > >> Hello, While I'm moving to CentOS8 environment, I got stuck at > >> ./configure with the following error. > >> configure: error:

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Amit Kapila
On Tue, Oct 15, 2019 at 1:26 PM Masahiko Sawada wrote: > > On Tue, Oct 15, 2019 at 4:15 PM Masahiko Sawada wrote: > > > > > > > If we avoid postponing deleting empty pages till the cleanup phase, > > > > > then we don't have the problem for gist indexes. > > > > > > > > Yes. But considering your

Re: ProcArrayGroupClearXid() compare-exchange style

2019-10-15 Thread Amit Kapila
On Tue, Oct 15, 2019 at 9:23 AM Noah Misch wrote: > > ProcArrayGroupClearXid() has this: > > while (true) > { > nextidx = > pg_atomic_read_u32(>procArrayGroupFirst); > > ... > > if >

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Dilip Kumar
On Tue, Oct 15, 2019 at 12:25 PM Amit Kapila wrote: > > Right, apart from some functions for memory allocation/estimation and > stats copy, we might need something like amcanparallelvacuum, so that > index methods can have the option to not participate in parallel > vacuum due to reasons similar

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Masahiko Sawada
On Tue, Oct 15, 2019 at 4:15 PM Masahiko Sawada wrote: > > On Tue, Oct 15, 2019 at 3:55 PM Amit Kapila wrote: > > > > On Tue, Oct 15, 2019 at 10:34 AM Masahiko Sawada > > wrote: > > > > > > On Mon, Oct 14, 2019 at 6:37 PM Amit Kapila > > > wrote: > > > > > > > > > > > 3. Do we really need to

Re: Columns correlation and adaptive query optimization

2019-10-15 Thread Konstantin Knizhnik
On 15.10.2019 1:20, legrand legrand wrote: Hello Konstantin, What you have proposed regarding join_selectivity and multicolumn statistics is a very good new ! Regarding your auto_explain modification, maybe an "advisor" mode would also be helpfull (with

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Masahiko Sawada
On Tue, Oct 15, 2019 at 3:55 PM Amit Kapila wrote: > > On Tue, Oct 15, 2019 at 10:34 AM Masahiko Sawada > wrote: > > > > On Mon, Oct 14, 2019 at 6:37 PM Amit Kapila wrote: > > > > > > > > 3. Do we really need to give the responsibility of deleting empty > > > pages

Clean up MinGW def file generation

2019-10-15 Thread Peter Eisentraut
I was mystified by this comment in Makefile.shlib: # We need several not-quite-identical variants of .DEF files to build # DLLs for Windows. These are made from the single source file # exports.txt. Since we can't assume that Windows boxes will have # sed, the .DEF files are always built and

Questions/Observations related to Gist vacuum

2019-10-15 Thread Amit Kapila
While reviewing a parallel vacuum patch [1], we noticed a few things about $SUBJECT implemented in commit - 7df159a620b760e289f1795b13542ed1b3e13b87. 1. A new memory context GistBulkDeleteResult->page_set_context has been introduced, but it doesn't seem to be used. 2. Right now, in gistbulkdelete

Re: v12.0: segfault in reindex CONCURRENTLY

2019-10-15 Thread Michael Paquier
On Mon, Oct 14, 2019 at 08:57:16AM +0900, Michael Paquier wrote: > I need to think about that, but shouldn't we have a way to reproduce > that case rather reliably with an isolation test? The patch looks to > good to me, these are also the two places I spotted yesterday after a > quick lookup.