Re: Pluggable storage

2018-06-18 Thread AJG
@Amit Re: Vacuum etc. Chrome V8 just released this blog post around concurrent marking, which may be of interest considering how cpu limited the browser is. Contains benchmark numbers etc in post as well. https://v8project.blogspot.com/2018/06/concurrent-marking.html "This post describes the

Re: Pluggable storage

2018-06-18 Thread Andres Freund
On 2018-06-18 12:43:57 -0700, AJG wrote: > @Amit > > Re: Vacuum etc. > > Chrome V8 just released this blog post around concurrent marking, which may > be of interest considering how cpu limited the browser is. Contains > benchmark numbers etc in post as well. > >

Re: Index Skip Scan

2018-06-18 Thread Andrew Dunstan
On 06/18/2018 11:25 AM, Jesper Pedersen wrote: Hi all, I would like to start a discussion on Index Skip Scan referred to as Loose Index Scan in the wiki [1]. awesome I wasn't planning on making this a patch submission for the July CommitFest due to the reasons mentioned above, but

Re: Removing "Included attributes in B-tree indexes" section from docs

2018-06-18 Thread Andrew Dunstan
On 06/18/2018 01:31 PM, Andres Freund wrote: On 2018-06-18 13:21:43 -0400, Alvaro Herrera wrote: On 2018-Jun-17, Peter Geoghegan wrote: On Sat, Jun 16, 2018 at 8:51 PM, Alvaro Herrera wrote: I don't necessarily object to the proposed change, but I think you should generally wait a bit

Re: libpq compression

2018-06-18 Thread Robbie Harwood
tKonstantin Knizhnik writes: > On 06.06.2018 02:03, Thomas Munro wrote: >> On Wed, Jun 6, 2018 at 2:06 AM, Konstantin Knizhnik >> wrote: >>> Thank you for review. Updated version of the patch fixing all reported >>> problems is attached. >> Small problem on Windows[1]: >> >>

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-18 Thread Claudio Freire
On Mon, Jun 18, 2018 at 4:59 PM Peter Geoghegan wrote: > > Note, guaranteed allowable time of index scans (used for quick deletion) > > will be achieved by storing equal-key index tuples in physical TID order [2] > > with patch [3]. > > I now have greater motivation to develop that patch into a

Re: WAL prefetch

2018-06-18 Thread Robert Haas
On Sat, Jun 16, 2018 at 3:41 PM, Andres Freund wrote: >> The posix_fadvise approach is not perfect, no doubt about that. But it >> works pretty well for bitmap heap scans, and it's about 13249x better >> (rough estimate) than the current solution (no prefetching). > > Sure, but investing in an

Re: Speedup of relation deletes during recovery

2018-06-18 Thread Fujii Masao
On Sat, Jun 16, 2018 at 3:28 AM, Andres Freund wrote: > Hi, > > On 2018-06-15 10:45:04 -0700, Andres Freund wrote: >> > + >> > + srels = palloc(sizeof(SMgrRelation) * ndelrels); >> > for (i = 0; i < ndelrels; i++) >> > - { >> > - SMgrRelation srel = smgropen(delrels[i],

Re: Query Rewrite for Materialized Views (Postgres Extension)

2018-06-18 Thread Nico Williams
On Mon, Jun 18, 2018 at 07:38:13PM +0100, Dent John wrote: > I commented to Corey (privately) that, while my rewrite extension has > gotten me a server that responds quickly to aggregate queries, the > constant need to refresh the supporting MVs means the system’s load > average is constant and

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-18 Thread Peter Geoghegan
On Sun, Jun 17, 2018 at 9:39 PM, Andrey V. Lepikhov wrote: > I have written a code for quick indextuple deletion from an relation by heap > tuple TID. The code relate to "Retail IndexTuple deletion" enhancement of > btree index on postgresql wiki [1]. I knew that somebody would eventually read

Re: Speedup of relation deletes during recovery

2018-06-18 Thread Fujii Masao
On Sat, Jun 16, 2018 at 2:54 AM, Teodor Sigaev wrote: >> We just had a customer hit this issue. I kind of wonder whether this >> shouldn't be backpatched: Currently the execution on the primary is >> O(NBuffers * log(ndrels)) whereas it's O(NBuffers * ndrels) on the >> standby - with a lot higher

Re: Speedup of relation deletes during recovery

2018-06-18 Thread Andres Freund
On 2018-06-19 03:06:54 +0900, Fujii Masao wrote: > On Sat, Jun 16, 2018 at 3:28 AM, Andres Freund wrote: > > Hi, > > > > On 2018-06-15 10:45:04 -0700, Andres Freund wrote: > >> > + > >> > + srels = palloc(sizeof(SMgrRelation) * ndelrels); > >> > for (i = 0; i < ndelrels; i++) > >> > - { >

Re: Transform for pl/perl

2018-06-18 Thread Tom Lane
ilm...@ilmari.org (Dagfinn Ilmari =?utf-8?Q?Manns=C3=A5ker?=) writes: > [ 0001-Fix-excess-enreferencing-in-plperl-jsonb-transform.patch ] I tested this a bit more thoroughly by dint of applying Data::Dumper to the Perl values, and found that we were still getting extra references to sub-objects,

Re: Fix slot's xmin advancement and subxact's lost snapshots in decoding.

2018-06-18 Thread Alvaro Herrera
On 2018-Jun-11, Antonin Houska wrote: > Arseny Sher wrote: > > Please see detailed description of the issues, tests which reproduce > > them and fixes in the attached patch. > > I've played with your tests and gdb for a while, both w/o and with your > patch. I think I can understand both

Re: Query Rewrite for Materialized Views (Postgres Extension)

2018-06-18 Thread Dent John
I commented to Corey (privately) that, while my rewrite extension has gotten me a server that responds quickly to aggregate queries, the constant need to refresh the supporting MVs means the system’s load average is constant and much higher than before. I’m happy with the tradeoff for now, but

Re: WAL prefetch

2018-06-18 Thread Andres Freund
On 2018-06-18 16:44:09 -0400, Robert Haas wrote: > On Sat, Jun 16, 2018 at 3:41 PM, Andres Freund wrote: > >> The posix_fadvise approach is not perfect, no doubt about that. But it > >> works pretty well for bitmap heap scans, and it's about 13249x better > >> (rough estimate) than the current

Re: [HACKERS] Statement-level rollback

2018-06-18 Thread Alvaro Herrera
On 2018-Jun-16, Robert Haas wrote: > I'm not sure that really solves the problem, because changing the GUC > in either direction causes the system to behave differently. I don't > see any particular reason to believe that changing the behavior from A > to B is any more or less likely to break

Re: [HACKERS] Statement-level rollback

2018-06-18 Thread Robert Haas
On Mon, Jun 18, 2018 at 4:51 PM, Alvaro Herrera wrote: > On 2018-Jun-16, Robert Haas wrote: >> I'm not sure that really solves the problem, because changing the GUC >> in either direction causes the system to behave differently. I don't >> see any particular reason to believe that changing the

Re: Index Skip Scan

2018-06-18 Thread Alexander Korotkov
On Mon, Jun 18, 2018 at 11:20 PM Andrew Dunstan wrote: > On 06/18/2018 11:25 AM, Jesper Pedersen wrote: > > I wasn't planning on making this a patch submission for the July > > CommitFest due to the reasons mentioned above, but can do so if people > > thinks it is best. T > > New large features

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-18 Thread Peter Geoghegan
On Mon, Jun 18, 2018 at 1:42 PM, Claudio Freire wrote: > Actually, once btree tids are sorted, you can continue tree descent > all the way to the exact leaf page that contains the tuple to be > deleted. > > Thus, the single-tuple interface ends up being quite OK. Sure, you can > optimize things a

Re: Performance regression with PostgreSQL 11 and partitioning

2018-06-18 Thread Thomas Reiss
Le 18/06/2018 à 10:46, David Rowley a écrit : > On 12 June 2018 at 01:49, Robert Haas wrote: >> On Fri, Jun 8, 2018 at 3:08 PM, Tom Lane wrote: >>> Robert Haas writes: That being said, I don't mind a bit if you want to look for further speedups here, but if you do, keep in mind

Server crashed with TRAP: FailedAssertion("!(parallel_workers > 0)" when partitionwise_aggregate true.

2018-06-18 Thread Rajkumar Raghuwanshi
Hi, Below test case crashed, when set enable_partitionwise_aggregate to true. CREATE TABLE part (c1 INTEGER,c2 INTEGER,c3 CHAR(10)) PARTITION BY RANGE(c1); CREATE TABLE part_p1 PARTITION OF part FOR VALUES FROM (MINVALUE) TO (500); CREATE TABLE part_p2 PARTITION OF part FOR VALUES FROM (500) TO

Re: Concurrency bug in UPDATE of partition-key

2018-06-18 Thread Amit Khandekar
On 18 June 2018 at 17:56, Amit Kapila wrote: > On Mon, Jun 18, 2018 at 11:28 AM, Dilip Kumar wrote: >> Should we also create a test case where we can verify that some >> unnecessary or duplicate triggers are not executed? >> > > I am not sure how much value we will add by having such a test. In

Re: server crashed with TRAP: FailedAssertion("!(!parallel_aware || pathnode->path.parallel_safe)"

2018-06-18 Thread Amit Khandekar
On 16 June 2018 at 10:44, Amit Kapila wrote: > On Thu, Jun 14, 2018 at 10:05 PM, Tom Lane wrote: >> I wrote: >>> This appears to be the fault of commit ab7271677, whose authors I've cc'd: >>> the stanza starting at about allpaths.c:1672 is bullheadedly creating a >>> parallel path whether that's

Re: Server crashed with TRAP: FailedAssertion("!(parallel_workers > 0)" when partitionwise_aggregate true.

2018-06-18 Thread Jeevan Chalke
On Mon, Jun 18, 2018 at 5:02 PM, Rajkumar Raghuwanshi < rajkumar.raghuwan...@enterprisedb.com> wrote: > Hi, > > Below test case crashed, when set enable_partitionwise_aggregate to true. > I will have a look over this. Thanks for reporting. > > CREATE TABLE part (c1 INTEGER,c2 INTEGER,c3

Re: ON CONFLICT DO NOTHING on pg_dump

2018-06-18 Thread Surafel Temesgen
On Sat, Jun 16, 2018 at 11:36 AM, Dilip Kumar wrote: > > @@ -172,6 +172,7 @@ typedef struct _dumpOptions > char*outputSuperuser; > > int sequence_data; /* dump sequence data even in schema-only mode */ > + int do_nothing; > } DumpOptions; > > The new structure member appears out of

Runtime partition pruning for MergeAppend

2018-06-18 Thread David Rowley
Back in the v11 cycle, there was just not quite enough time to get the MergeAppend run-time partition pruning patch in. I've attached v24 of this patch. The only changes done from v23 [1] are to re-base the patch atop of current master. There's was a bit of churn in the partition pruning and

Re: server crashed with TRAP: FailedAssertion("!(!parallel_aware || pathnode->path.parallel_safe)"

2018-06-18 Thread Amit Khandekar
On 16 June 2018 at 19:30, Amit Kapila wrote: > On Sat, Jun 16, 2018 at 10:44 AM, Amit Kapila wrote: >> Yeah, or perhaps disallow creation of any partial paths at the first >> place like in attached. This will save us some work as well. >> > > Attached patch contains test case as well. I have

Re: Possible bug in logical replication.

2018-06-18 Thread Michael Paquier
On Fri, Jun 15, 2018 at 06:27:56PM +0300, Arseny Sher wrote: > I confirm that starting reading WAL since restart_lsn as implemented in > f731cfa fixes this issue, as well as the second issue tushar mentioned > at [1]. Thanks! +/* + * Start reading WAL at restart_lsn, which certainly

Re: Slow planning time for simple query

2018-06-18 Thread Amit Kapila
On Sun, Jun 17, 2018 at 9:22 PM, Andrew Gierth wrote: >> "Tom" == Tom Lane writes: > > Tom> 2. Although _bt_killitems doesn't WAL-log its setting of kill > Tom> bits, those bits could propagate to the standby anyway, as a > Tom> result of a subsequent WAL action on the index page that

Re: AtEOXact_ApplyLauncher() and subtransactions

2018-06-18 Thread Amit Khandekar
On 16 June 2018 at 00:03, Amit Khandekar wrote: > The way I am implementing this can be seen in attached > apply_launcher_subtrans_WIP.patch. (check launcher.c changes). I > haven't started testing it yet though. Attached patch passes some basic testing I did. Will do some more testing, and some

Re: Concurrency bug in UPDATE of partition-key

2018-06-18 Thread Amit Kapila
On Mon, Jun 18, 2018 at 11:28 AM, Dilip Kumar wrote: > On Mon, Jun 18, 2018 at 10:21 AM, Amit Khandekar > wrote: >> Attached is v2 version of the patch. It contains the above >> trigger-related issue fixed. >> >> The updated tuple is passed back using the existing newslot parameter >> of

Re: Server crashed with TRAP: FailedAssertion("!(parallel_workers > 0)" when partitionwise_aggregate true.

2018-06-18 Thread Andres Freund
On 2018-06-18 17:10:12 +0530, Jeevan Chalke wrote: > On Mon, Jun 18, 2018 at 5:02 PM, Rajkumar Raghuwanshi < > rajkumar.raghuwan...@enterprisedb.com> wrote: > > > Hi, > > > > Below test case crashed, when set enable_partitionwise_aggregate to true. > > > > I will have a look over this. > >

Re: [PATCH] Find additional connection service files in pg_service.conf.d directory

2018-06-18 Thread Arthur Zakirov
Hello, On Thu, Mar 01, 2018 at 01:40:10PM -0500, Curt Tilmes wrote: > New patch limits files to ".conf". Isn't it worth to check errno for stat(), opendir() and readdir() calls? I think when some error occured in searchServiceFileDirectory() then the error "definition of service \"%s\" not

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-18 Thread Joe Conway
On 06/18/2018 09:49 AM, Robert Haas wrote: > On Wed, Jun 13, 2018 at 9:20 AM, Joe Conway wrote: >>> Also, if I understand correctly, at unconference session there also >>> were two suggestions about the design other than the suggestion by >>> Alexander: implementing TDE at column level using

Re: Slow planning time for simple query

2018-06-18 Thread Tom Lane
Amit Kapila writes: > On Sun, Jun 17, 2018 at 9:22 PM, Andrew Gierth > wrote: >> That's OK as long as we're ignoring those hints on the standby. > What if we don't ignore those hints on standby for a specific case > like the one in get_actual_variable_range? Yeah, that's the same idea I

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-18 Thread Joe Conway
On 06/18/2018 10:26 AM, Robert Haas wrote: > On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway wrote: >> Not necessarily. Our pages probably have enough predictable bytes to aid >> cryptanalysis, compared to user data in a column which might not be very >> predicable. > > Really? I would guess that

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2018-06-18 Thread Claudio Freire
On Fri, Jun 15, 2018 at 8:47 PM Peter Geoghegan wrote: > > I think it would be helpful if you could talk more about these > > regressions (and the wins). > > I think that the performance regressions are due to the fact that when > you have a huge number of duplicates today, it's useful to be

Index Skip Scan

2018-06-18 Thread Jesper Pedersen
Hi all, I would like to start a discussion on Index Skip Scan referred to as Loose Index Scan in the wiki [1]. My use-case is the simplest form of Index Skip Scan (B-Tree only), namely going from CREATE TABLE t1 (a integer PRIMARY KEY, b integer); CREATE INDEX idx_t1_b ON t1 (b); INSERT

Re: Index Skip Scan

2018-06-18 Thread Alexander Korotkov
Hi! On Mon, Jun 18, 2018 at 6:26 PM Jesper Pedersen wrote: > I would like to start a discussion on Index Skip Scan referred to as > Loose Index Scan in the wiki [1]. Great, I glad to see you working in this! > However, as Robert Haas noted in the thread there are issues with the > patch as is,

Re: Removing "Included attributes in B-tree indexes" section from docs

2018-06-18 Thread Andres Freund
On 2018-06-18 13:21:43 -0400, Alvaro Herrera wrote: > On 2018-Jun-17, Peter Geoghegan wrote: > > > On Sat, Jun 16, 2018 at 8:51 PM, Alvaro Herrera > > wrote: > > > I don't necessarily object to the proposed change, but I think you > > > should generally wait a bit longer for others to react. > >

Re: Query Rewrite for Materialized Views (Postgres Extension)

2018-06-18 Thread Corey Huinker
> > Hope it is useful or interesting for someone! Questions or comments are >> very welcome. >> > > good idea. > > Regards > > Pavel > In a recent PgConf NYC presentation [1] I was talking about the technical hurdles to implementing materialized views that could be kept up to date at all times,

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-18 Thread Alvaro Herrera
On 2018-Jun-18, Ashutosh Bapat wrote: > That's a wrong comparison. Inheritance based partitioning works even > after declarative partitioning feature is added. So, users can > continue using inheritance based partitioning if they don't want to > move to declarative partitioning. That's not true

Re: why partition pruning doesn't work?

2018-06-18 Thread Tom Lane
Amit Langote writes: > [ 0001-Open-partitioned-tables-during-Append-initialization.patch ] I took a look at this. While I'm in agreement with the general idea of holding open the partitioned relations' relcache entries throughout the query, I do not like anything about this patch: * It seems

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2018-06-18 Thread Peter Geoghegan
On Mon, Jun 18, 2018 at 7:57 AM, Claudio Freire wrote: > Way back when I was dabbling in this kind of endeavor, my main idea to > counteract that, and possibly improve performance overall, was a > microvacuum kind of thing that would do some on-demand cleanup to > remove duplicates or make room

Re: Removing "Included attributes in B-tree indexes" section from docs

2018-06-18 Thread Alvaro Herrera
On 2018-Jun-17, Peter Geoghegan wrote: > On Sat, Jun 16, 2018 at 8:51 PM, Alvaro Herrera > wrote: > > I don't necessarily object to the proposed change, but I think you > > should generally wait a bit longer for others to react. > > What wait period do you think is appropriate in this case?

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-18 Thread Tomas Vondra
On 06/18/2018 05:06 PM, Joe Conway wrote: On 06/18/2018 10:52 AM, Tom Lane wrote: Robert Haas writes: On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway wrote: Not necessarily. Our pages probably have enough predictable bytes to aid cryptanalysis, compared to user data in a column which might

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-18 Thread David G. Johnston
On Mon, Jun 18, 2018 at 9:59 AM, Alvaro Herrera wrote: > > alvherre=# select tgname, tgrelid::regclass, tgisinternal from pg_trigger; > tgname │ tgrelid │ tgisinternal > ┼─┼── > trig_p │ parent │ f > trig_p │ child │ t > trig_c │ child │ f > (3 filas) > > So

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-18 Thread Nico Williams
On Mon, Jun 11, 2018 at 06:22:22PM +0900, Masahiko Sawada wrote: > As per discussion at PGCon unconference, I think that firstly we need > to discuss what threats we want to defend database data against. If We call that a threat model. There can be many threat models, of course. > user wants to

Re: Removing "Included attributes in B-tree indexes" section from docs

2018-06-18 Thread Peter Geoghegan
On Mon, Jun 18, 2018 at 10:21 AM, Alvaro Herrera wrote: > One which includes at least half a working day in a different timezone. > You asked mid-afternoon on a Friday in a timezone pretty far west. It was 11 am PST. I'll make a note about this. It won't happen again. -- Peter Geoghegan

Re: Removing "Included attributes in B-tree indexes" section from docs

2018-06-18 Thread Robert Haas
On Mon, Jun 18, 2018 at 1:31 PM, Andres Freund wrote: > I think there's also a question of how much a patch is blocking you / > others. A shorter question period is more understandable if it's step > 3/40, rather than 1/1... Agreed. For non-critical stuff like this it seems like waiting 2 or 3

Re: Making all nbtree entries unique by having heap TIDs participate in comparisons

2018-06-18 Thread Claudio Freire
On Mon, Jun 18, 2018 at 2:03 PM Peter Geoghegan wrote: > > On Mon, Jun 18, 2018 at 7:57 AM, Claudio Freire > wrote: > > Way back when I was dabbling in this kind of endeavor, my main idea to > > counteract that, and possibly improve performance overall, was a > > microvacuum kind of thing that

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-18 Thread Joe Conway
On 06/18/2018 10:52 AM, Tom Lane wrote: > Robert Haas writes: >> On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway wrote: >>> Not necessarily. Our pages probably have enough predictable bytes to aid >>> cryptanalysis, compared to user data in a column which might not be very >>> predicable. > >>

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-18 Thread Tom Lane
Robert Haas writes: > On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway wrote: >> Not necessarily. Our pages probably have enough predictable bytes to aid >> cryptanalysis, compared to user data in a column which might not be very >> predicable. > Really? I would guess that the amount of entropy in

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-18 Thread Robert Haas
On Mon, Jun 18, 2018 at 1:20 AM, Ashutosh Bapat wrote: > That's a wrong comparison. Inheritance based partitioning works even > after declarative partitioning feature is added. So, users can > continue using inheritance based partitioning if they don't want to > move to declarative partitioning.

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-18 Thread Robert Haas
On Wed, Jun 13, 2018 at 9:20 AM, Joe Conway wrote: >> Also, if I understand correctly, at unconference session there also >> were two suggestions about the design other than the suggestion by >> Alexander: implementing TDE at column level using POLICY, and >> implementing TDE at table-space

Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

2018-06-18 Thread Robert Haas
On Mon, Jun 18, 2018 at 10:12 AM, Joe Conway wrote: > Not necessarily. Our pages probably have enough predictable bytes to aid > cryptanalysis, compared to user data in a column which might not be very > predicable. Really? I would guess that the amount of entropy in a page is WAY higher than

Re: ON CONFLICT DO NOTHING on pg_dump

2018-06-18 Thread Nico Williams
On Fri, Jun 15, 2018 at 02:20:21AM +, Ideriha, Takeshi wrote: > >From: Nico Williams [mailto:n...@cryptonector.com] > >On Tue, Jun 12, 2018 at 09:05:23AM +, Ideriha, Takeshi wrote: > >> Only the difference of data can be restored. > > > >But that's additive-only. Only missing rows are

Re: Invisible Indexes

2018-06-18 Thread Andrew Dunstan
On 06/18/2018 05:44 PM, Peter Geoghegan wrote: On Mon, Jun 18, 2018 at 2:36 PM, Andrew Dunstan wrote: This is a MySQL feature, where an index is not considered by the planner. Implementing it should be fairly straightforward, adding a new boolean to pg_index, and options to CREATE INDEX and

Re: Invisible Indexes

2018-06-18 Thread Jaime Casanova
On 18 June 2018 at 16:36, Andrew Dunstan wrote: > > This is a MySQL feature, where an index is not considered by the planner. > Implementing it should be fairly straightforward, adding a new boolean to > pg_index, and options to CREATE INDEX and ALTER INDEX. I guess VISIBLE would > become a new

Re: Invisible Indexes

2018-06-18 Thread Andrew Dunstan
On 06/18/2018 05:46 PM, Jaime Casanova wrote: On 18 June 2018 at 16:36, Andrew Dunstan wrote: This is a MySQL feature, where an index is not considered by the planner. Implementing it should be fairly straightforward, adding a new boolean to pg_index, and options to CREATE INDEX and ALTER

Re: Invisible Indexes

2018-06-18 Thread Tom Lane
Andrew Dunstan writes: > This is a MySQL feature, where an index is not considered by the > planner. Implementing it should be fairly straightforward, adding a new > boolean to pg_index, and options to CREATE INDEX and ALTER INDEX. I > guess VISIBLE would become a new unreserved keyword. >

Re: Invisible Indexes

2018-06-18 Thread Andres Freund
On 2018-06-18 17:50:44 -0400, Andrew Dunstan wrote: > > > On 06/18/2018 05:46 PM, Jaime Casanova wrote: > > On 18 June 2018 at 16:36, Andrew Dunstan > > wrote: > > > This is a MySQL feature, where an index is not considered by the planner. > > > Implementing it should be fairly

Re: Invisible Indexes

2018-06-18 Thread Andres Freund
Hi, On 2018-06-18 17:57:04 -0400, Tom Lane wrote: > Andrew Dunstan writes: > > This is a MySQL feature, where an index is not considered by the > > planner. Implementing it should be fairly straightforward, adding a new > > boolean to pg_index, and options to CREATE INDEX and ALTER INDEX. I >

Re: Invisible Indexes

2018-06-18 Thread Andres Freund
On 2018-06-18 18:05:11 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2018-06-18 17:57:04 -0400, Tom Lane wrote: > >> I think the actually desirable way to handle this sort of thing is through > >> an "index advisor" sort of plugin, which can hide a given index from the > >> planner

Re: Invisible Indexes

2018-06-18 Thread Tom Lane
Peter Geoghegan writes: > On Mon, Jun 18, 2018 at 2:57 PM, Tom Lane wrote: >> I think the actually desirable way to handle this sort of thing is through >> an "index advisor" sort of plugin, which can hide a given index from the >> planner without any globally visible side-effects. > The

Re: Invisible Indexes

2018-06-18 Thread Peter Geoghegan
On Mon, Jun 18, 2018 at 3:12 PM, Tom Lane wrote: > Perhaps there are use-cases where you want globally visible effects, > but the primary use-case Andrew cited (i.e. EXPLAIN experimentation) > would not want that. > > Anyway, if we do it with a GUC, the user can control the scope of > the

Re: Invisible Indexes

2018-06-18 Thread Tom Lane
Andrew Dunstan writes: > On 06/18/2018 06:12 PM, Tom Lane wrote: >> Anyway, if we do it with a GUC, the user can control the scope of >> the effects. > Yeah, but Peter makes the case that people want it for global > experimentation. "We think we can safely drop this humungous index that >

Invisible Indexes

2018-06-18 Thread Andrew Dunstan
This is a MySQL feature, where an index is not considered by the planner. Implementing it should be fairly straightforward, adding a new boolean to pg_index, and options to CREATE INDEX and ALTER INDEX. I guess VISIBLE would become a new unreserved keyword. The most obvious use case is to

Re: Transform for pl/perl

2018-06-18 Thread Tom Lane
I wrote: > The remaining unresolved issue in this thread is Ilmari's suggestion > that we should convert integers to Perl IV (or UV?) if they fit, rather > than always convert to NV as now. Oh ... after re-reading the thread I realized there was one other point that we'd all forgotten about,

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-18 Thread Peter Geoghegan
On Mon, Jun 18, 2018 at 2:54 PM, Peter Geoghegan wrote: > On Sun, Jun 17, 2018 at 9:39 PM, Andrey V. Lepikhov > wrote: >> Patch '0001-retail-indextuple-deletion' introduce new function >> amtargetdelete() in access method interface. Patch >> '0002-quick-vacuum-strategy' implements this function

Re: Invisible Indexes

2018-06-18 Thread Tom Lane
Andres Freund writes: > On 2018-06-18 17:57:04 -0400, Tom Lane wrote: >> I think the actually desirable way to handle this sort of thing is through >> an "index advisor" sort of plugin, which can hide a given index from the >> planner without any globally visible side-effects. > Although I'm a

Re: Invisible Indexes

2018-06-18 Thread Peter Geoghegan
On Mon, Jun 18, 2018 at 2:57 PM, Tom Lane wrote: > Admittedly, this isn't great in a production environment, but neither > would be disabling the index in the way you suggest. > > I think the actually desirable way to handle this sort of thing is through > an "index advisor" sort of plugin, which

Re: Invisible Indexes

2018-06-18 Thread Julien Rouhaud
On Tue, Jun 19, 2018 at 12:05 AM, Tom Lane wrote: > > Well, what I was thinking about was that this functionality already > exists (I think) in one or more "index advisor" plugins. It's possible > that they've all bit-rotted for lack of support, which would not speak > highly of the demand for

Re: Invisible Indexes

2018-06-18 Thread Andrew Dunstan
On 06/18/2018 06:12 PM, Tom Lane wrote: Peter Geoghegan writes: On Mon, Jun 18, 2018 at 2:57 PM, Tom Lane wrote: I think the actually desirable way to handle this sort of thing is through an "index advisor" sort of plugin, which can hide a given index from the planner without any globally

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-18 Thread Robert Treat
On Mon, Jun 18, 2018 at 12:59 PM, Alvaro Herrera wrote: > On 2018-Jun-18, Ashutosh Bapat wrote: > >> That's a wrong comparison. Inheritance based partitioning works even >> after declarative partitioning feature is added. So, users can >> continue using inheritance based partitioning if they

Re: Invisible Indexes

2018-06-18 Thread Robert Treat
On Mon, Jun 18, 2018 at 6:11 PM, Andres Freund wrote: > On 2018-06-18 18:05:11 -0400, Tom Lane wrote: >> Andres Freund writes: >> > On 2018-06-18 17:57:04 -0400, Tom Lane wrote: >> >> I think the actually desirable way to handle this sort of thing is through >> >> an "index advisor" sort of

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-18 Thread Peter Geoghegan
On Mon, Jun 18, 2018 at 4:05 PM, Peter Geoghegan wrote: > Finally, doing things this way would let you delete multiple > duplicates in one shot, as I described in an earlier e-mail. Only a > single descent of the tree is needed to delete quite a few index > tuples, provided that they all happen

Re: pg_config.h.win32 missing a set of flags from pg_config.h.in added in v11 development

2018-06-18 Thread Michael Paquier
On Mon, Jun 18, 2018 at 10:57:34AM +0900, Michael Paquier wrote: > As there is visibly a consensus for HEAD, for now I propose to just > process with the previous patch on only the master branch then. This > will address the open item. Any objections to that? As there is a consensus at least on

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-18 Thread Amit Langote
Hi. On 2018/06/19 1:59, Alvaro Herrera wrote: > What does worry me a little bit now, reading this discussion, is whether > we've made the triggers in partitions visible enough. We'll have this > problem once we implement BEFORE ROW triggers as proposed, and I think > we already have this problem

Re: Index Skip Scan

2018-06-18 Thread Michael Paquier
On Tue, Jun 19, 2018 at 12:06:59AM +0300, Alexander Korotkov wrote: > Assuming this, should we have possibility to register patch to > September CF from now? There cannot be two commit fests marked as open at the same time as Magnus mentions here:

Re: [PATCH] Find additional connection service files in pg_service.conf.d directory

2018-06-18 Thread Michael Paquier
On Mon, Jun 18, 2018 at 07:17:14PM +0300, Arthur Zakirov wrote: > Isn't it worth to check errno for stat(), opendir() and readdir() > calls? Checking for such errors is mandatory. There are some cases where there are filters based on errno like ENOENT, but if something unexpected is happening

Re: Supporting tls-server-end-point as SCRAM channel binding for OpenSSL 1.0.0 and 1.0.1

2018-06-18 Thread Michael Paquier
On Sat, Jun 09, 2018 at 09:28:17AM +0900, Michael Paquier wrote: > I am still not completely sure what is the correct course of action > here. Heikki and Peter and not much in favor of adding more complexity > here as OpenSSL has a long history of having a non-linear history across > platforms.

Re: Partitioning with temp tables is broken

2018-06-18 Thread Amit Langote
Hello. On 2018/06/18 15:02, Michael Paquier wrote: > On Mon, Jun 18, 2018 at 01:27:51PM +0900, Amit Langote wrote: >> On 2018/06/17 22:11, Michael Paquier wrote: >> Which checks do you think are missing other than those added by the >> proposed patch? > > I was just wondering how this reacted if

Adding tests for inheritance trees with temporary tables

2018-06-18 Thread Michael Paquier
Hi all, While look at the handling of temporary relations with partition trees, I have noticed that there are no tests for inheritance trees with temp tables. This has been mentioned here: https://www.postgresql.org/message-id/20180618060200.gg3...@paquier.xyz Attached is a patch to close the

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-18 Thread Peter Geoghegan
On Mon, Jun 18, 2018 at 4:05 PM, Peter Geoghegan wrote: > IOW, the approach you've taken in bttargetdelete() isn't quite correct > because you imagine that it's okay to occasionally "lose" the index > tuple that you originally found, and just move on. That needs to be > 100% reliable, or else

Re: pg_config.h.win32 missing a set of flags from pg_config.h.in added in v11 development

2018-06-18 Thread Michael Paquier
On Tue, Jun 19, 2018 at 09:11:20AM +0900, Michael Paquier wrote: > On Mon, Jun 18, 2018 at 10:57:34AM +0900, Michael Paquier wrote: > > As there is visibly a consensus for HEAD, for now I propose to just > > process with the previous patch on only the master branch then. This > > will address the

Re: Libpq support to connect to standby server as priority

2018-06-18 Thread Haribabu Kommi
On Wed, Jan 24, 2018 at 9:01 AM Jing Wang wrote: > Hi All, > > Recently I put a proposal to support 'prefer-read' parameter in > target_session_attrs in libpq. Now I updated the patch with adding content > in the sgml and regression test case. > > Some people may have noticed there is already

Re: server crashed with TRAP: FailedAssertion("!(!parallel_aware || pathnode->path.parallel_safe)"

2018-06-18 Thread Amit Kapila
On Mon, Jun 18, 2018 at 3:09 PM, Amit Khandekar wrote: > On 16 June 2018 at 19:30, Amit Kapila wrote: >> On Sat, Jun 16, 2018 at 10:44 AM, Amit Kapila >> wrote: >>> Yeah, or perhaps disallow creation of any partial paths at the first >>> place like in attached. This will save us some work as

RE: [bug fix] ECPG: freeing memory for pgtypes crashes on Windows

2018-06-18 Thread Tsunakawa, Takayuki
> On Tue, Jun 12, 2018 at 1:09 PM, Tsunakawa, Takayuki > wrote: > > My colleague is now preparing a patch for that, which adds a function > ECPGFreeSQLDA() in libecpg.so. That thread is here: > > > https://www.postgresql.org/message-id/25C1C6B2E7BE044889E4FE8643A58BA9 > 63A42097@G01JPEXMBKW03 >

Re: Partitioning with temp tables is broken

2018-06-18 Thread Michael Paquier
On Mon, Jun 18, 2018 at 01:27:51PM +0900, Amit Langote wrote: > On 2018/06/17 22:11, Michael Paquier wrote: > Which checks do you think are missing other than those added by the > proposed patch? I was just wondering how this reacted if trying to attach a foreign table to a partition tree which

Re: Performance regression with PostgreSQL 11 and partitioning

2018-06-18 Thread David Rowley
On 12 June 2018 at 01:49, Robert Haas wrote: > On Fri, Jun 8, 2018 at 3:08 PM, Tom Lane wrote: >> Robert Haas writes: >>> That being said, I don't mind a bit if you want to look for further >>> speedups here, but if you do, keep in mind that a lot of queries won't >>> even use partition-wise

Re: pg_config.h.win32 missing a set of flags from pg_config.h.in added in v11 development

2018-06-18 Thread Michael Paquier
On Tue, Jun 19, 2018 at 12:01:39AM -0400, Andrew Dunstan wrote: >> In my own environment I have a copy of ssleay32.dll and libeay32.dll in >> c:\Windows\System32 so as that I don't get any complaints when running >> regression tests. Is your environment using something specific? > > I have

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-18 Thread Ashutosh Bapat
On Mon, Jun 18, 2018 at 10:29 PM, Alvaro Herrera wrote: > On 2018-Jun-18, Ashutosh Bapat wrote: > >> That's a wrong comparison. Inheritance based partitioning works even >> after declarative partitioning feature is added. So, users can >> continue using inheritance based partitioning if they

Re: Slow planning time for simple query

2018-06-18 Thread Amit Kapila
On Mon, Jun 18, 2018 at 7:50 PM, Tom Lane wrote: > Amit Kapila writes: >> On Sun, Jun 17, 2018 at 9:22 PM, Andrew Gierth >> wrote: >>> That's OK as long as we're ignoring those hints on the standby. > >> What if we don't ignore those hints on standby for a specific case >> like the one in

Re: Pluggable storage

2018-06-18 Thread Amit Kapila
On Tue, Jun 19, 2018 at 1:13 AM, AJG wrote: > @Amit > > Re: Vacuum etc. > > Chrome V8 just released this blog post around concurrent marking, which may > be of interest considering how cpu limited the browser is. Contains > benchmark numbers etc in post as well. > >

Re: pg_config.h.win32 missing a set of flags from pg_config.h.in added in v11 development

2018-06-18 Thread Andrew Dunstan
On 06/18/2018 11:13 PM, Michael Paquier wrote: On Tue, Jun 19, 2018 at 09:11:20AM +0900, Michael Paquier wrote: On Mon, Jun 18, 2018 at 10:57:34AM +0900, Michael Paquier wrote: As there is visibly a consensus for HEAD, for now I propose to just process with the previous patch on only the

Re: Partitioning with temp tables is broken

2018-06-18 Thread Michael Paquier
On Tue, Jun 19, 2018 at 10:56:49AM +0900, Amit Langote wrote: > On 2018/06/18 15:02, Michael Paquier wrote: >> Those tests should be upper-case I think to keep consistency with the >> surrounding code. > > As you may have seen in the changed code, the guard in MergeAttributes > really just checks

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-18 Thread Ashutosh Bapat
On Tue, Jun 19, 2018 at 3:51 AM, Robert Treat wrote: > > So +1 for thinking we do need to worry about it. I'm not exactly sure > how we want to expose that info; with \d+ we list various "Partition > X:" sections, perhaps adding one for "Partition triggers:" would be > enough, although I am

Excessive CPU usage in StandbyReleaseLocks()

2018-06-18 Thread Thomas Munro
Hello hackers, Andres Freund diagnosed a case of $SUBJECT in a customer's 9.6 system. I've written a minimal reproducer and a prototype patch to address the root cause. The problem is that StandbyReleaseLocks() does a linear search of all known AccessExclusiveLocks when a transaction ends.

  1   2   >