Re: Tab completion for ALTER INDEX|TABLE ALTER COLUMN SET STATISTICS

2018-12-20 Thread Michael Paquier
On Fri, Dec 21, 2018 at 02:51:51PM +0900, Tatsuro Yamada wrote: > Attached file is a WIP patch. Before sorting out the exotic part of the feature, why not first fixing the simple cases where we know that tab completion is broken and can be improved? This is what I meant in this email:

RE: Timeout parameters

2018-12-20 Thread Nagaura, Ryohei
Hi, Fabien. The next CF will start so I want to restart the discussion. > About "socket_timeout" > If you face the following situation, this parameter will be needed. If you feel that this situation can't happen or the use case is too limited, please point out so. > > I think that there is

Speeding up creating UPDATE/DELETE generic plan for partitioned table into a lot

2018-12-20 Thread Kato, Sho
Hi, I want to speed up the creation of UPDATE/DELETE generic plan for tables partitioned into a lot. Currently, creating a generic plan of UPDATE/DELTE for such table, planner creates a plan to scan all partitions. So it takes a very long time. I tried with a table partitioned into 8192, it

Re: Remove Deprecated Exclusive Backup Mode

2018-12-20 Thread David Steele
On 12/21/18 2:01 AM, Michael Paquier wrote: On Thu, Dec 20, 2018 at 12:29:48PM +0200, David Steele wrote: Cannot move patch to the same commitfest, and multiple future commitfests exist! I am not sure what it means either. What if you just mark the existing entry as returned with feedback,

Re: Change pgarch_readyXlog() to return .history files first

2018-12-20 Thread David Steele
On 12/21/18 6:49 AM, Kyotaro HORIGUCHI wrote: "else if (!historyFound || ishistory)" strcpy(xlog, newxlog); The caller prepares sufficient memory for basename, and we no longer copy ".ready" into newxlog. Douldn't we work directly on xlog instead of allocating newxlog? I

Re: Add timeline to partial WAL segments

2018-12-20 Thread David Steele
On 12/20/18 10:56 PM, Robert Haas wrote: On Fri, Dec 14, 2018 at 6:05 PM David Steele wrote: The question in my mind: is it safe to back-patch? I cannot imagine it being a good idea to stick a behavioral change like this into a minor release. Yeah, it lets people get out from under this

Re: Tab completion for ALTER INDEX|TABLE ALTER COLUMN SET STATISTICS

2018-12-20 Thread Tatsuro Yamada
On 2018/12/20 10:47, Tatsuro Yamada wrote: On 2018/12/20 10:38, Michael Paquier wrote: On Thu, Dec 20, 2018 at 10:05:30AM +0900, Tatsuro Yamada wrote: Alright, I'll create new patches including these:    - No completion after "ALTER TABLE/INDEX SET STATISTICS" instead of schema names    -

Re: Change pgarch_readyXlog() to return .history files first

2018-12-20 Thread Kyotaro HORIGUCHI
At Fri, 21 Dec 2018 14:17:25 +0900, Michael Paquier wrote in <20181221051724.gg1...@paquier.xyz> > On Fri, Dec 21, 2018 at 01:49:18PM +0900, Kyotaro HORIGUCHI wrote: > > FWIW it seems to me a bug that making an inconsistent set of > > files in archive directory. > > Okay, point taken! FWIW, I

Re: Change pgarch_readyXlog() to return .history files first

2018-12-20 Thread Michael Paquier
On Fri, Dec 21, 2018 at 01:49:18PM +0900, Kyotaro HORIGUCHI wrote: > FWIW it seems to me a bug that making an inconsistent set of > files in archive directory. Okay, point taken! FWIW, I have no actual objections in not back-patching that. > At Fri, 21 Dec 2018 12:19:21 +0900, Michael Paquier

Re: Change pgarch_readyXlog() to return .history files first

2018-12-20 Thread Kyotaro HORIGUCHI
Hello. FWIW it seems to me a bug that making an inconsistent set of files in archive directory. At Fri, 21 Dec 2018 12:19:21 +0900, Michael Paquier wrote in <20181221031921.ge1...@paquier.xyz> > On Thu, Dec 20, 2018 at 01:57:30PM +0200, David Steele wrote: > > Good point. The new patch uses

Clean up some elog messages and comments for do_pg_stop_backup and do_pg_start_backup

2018-12-20 Thread Michael Paquier
Hi all, Alvaro has cleaned up a couple of error messages recently so as they do not include the function name in what gets translated as per 68f6f2b7. While looking in the code for similar patterns, I have been reminded that pg_stop_backup() is included in some messages when waiting for segments

Re: Tid scan improvements

2018-12-20 Thread Tom Lane
Edmund Horner writes: > For the forward scan, I seem to recall, from your merge join example, > that it's useful to set the pathkeys even when there are no > query_pathkeys. We just have to unconditionally set them so that the > larger plan can make use of them. No. Look at indxpath.c: it does

Re: Improve selectivity estimate for range queries

2018-12-20 Thread Kyotaro HORIGUCHI
Hello. At Thu, 20 Dec 2018 17:21:29 +0900, "Yuzuko Hosoya" wrote in <008701d4983d$02e731c0$08b59540$@lab.ntt.co.jp> > In my environment, the selectivity for id > 0 was 0.1, > and the selectivity for id < 1 was 0.1. Then, the > value of rqlist->hibound and

Re: [HACKERS] Macros bundling RELKIND_* conditions

2018-12-20 Thread Kyotaro HORIGUCHI
Mmm. My mail on this topic seems to have sent to nowhere.. At Fri, 21 Dec 2018 07:50:04 +0530, Ashutosh Bapat wrote in > On Wed, Dec 19, 2018 at 11:37 PM Alvaro Herrera > wrote: > > > On 2017-Aug-02, Tom Lane wrote: > > > > > I think Peter's got the error and the detail backwards. It

Re: Tid scan improvements

2018-12-20 Thread Edmund Horner
On Fri, 21 Dec 2018 at 13:25, David Rowley wrote: > On Fri, 21 Dec 2018 at 13:09, Edmund Horner wrote: > > On Fri, 21 Dec 2018 at 11:21, Tom Lane wrote: > > > I'm having a hard time wrapping my mind around why you'd bother with > > > backwards TID scans. The amount of code needed versus the

Re: Change pgarch_readyXlog() to return .history files first

2018-12-20 Thread Michael Paquier
On Thu, Dec 20, 2018 at 01:57:30PM +0200, David Steele wrote: > Good point. The new patch uses IsTLHistoryFileName(). OK, I have been reviewing the patch and the logic is correct, still I could not resist reducing the number of inner if's for readability. I also did not like the high-jacking of

Re: [HACKERS] Macros bundling RELKIND_* conditions

2018-12-20 Thread Ashutosh Bapat
On Wed, Dec 19, 2018 at 11:37 PM Alvaro Herrera wrote: > On 2017-Aug-02, Tom Lane wrote: > > > I think Peter's got the error and the detail backwards. It should be > > more like > > > > ERROR: "someview" cannot have constraints > > DETAIL: "someview" is a view. > > > > If we do it like that, we

RE: [Proposal] Add accumulated statistics

2018-12-20 Thread Yotsunaga, Naoki
On Wed, Nov 21, 2018 at 9:27 PM, Bruce Momjian wrote: Hi, thank you for the information. I understood that sampling is effective for investigation of waiting events. By the way, you can see the number of wait events with "LWLOCK_STATS", right? Is this function implemented because it is

Re: A few new options for vacuumdb

2018-12-20 Thread Michael Paquier
On Thu, Dec 20, 2018 at 04:48:11PM +, Bossart, Nathan wrote: > The --skip-locked option in vacuumdb is part of 0002, so I don't think > there's much precedent here. It looks like I was not looking at the master branch here ;) > We do currently fall back to the > unparenthesized syntax for

Re: lock level for DETACH PARTITION looks sketchy

2018-12-20 Thread Amit Langote
On 2018/12/21 6:07, Alvaro Herrera wrote: > On 2018-Dec-20, Robert Haas wrote: > >> On Thu, Dec 20, 2018 at 3:58 PM Alvaro Herrera >> wrote: >>> I think what prompted the lock to be AccessShareLock for the child rel >>> in the first place is the fact that ATExecDropInherit() (ALTER TABLE NO >>>

Re: Tid scan improvements

2018-12-20 Thread David Rowley
On Fri, 21 Dec 2018 at 13:09, Edmund Horner wrote: > On Fri, 21 Dec 2018 at 11:21, Tom Lane wrote: > > I'm having a hard time wrapping my mind around why you'd bother with > > backwards TID scans. The amount of code needed versus the amount of > > usefulness seems like a pretty bad cost/benefit

Re: Remove Deprecated Exclusive Backup Mode

2018-12-20 Thread Michael Paquier
On Thu, Dec 20, 2018 at 12:29:48PM +0200, David Steele wrote: > Cannot move patch to the same commitfest, and multiple future commitfests > exist! I am not sure what it means either. What if you just mark the existing entry as returned with feedback, and create a new one ahead? -- Michael

Re: gist microvacuum doesn't appear to care about hot standby?

2018-12-20 Thread Alexander Korotkov
On Thu, Dec 20, 2018 at 1:41 AM Alexander Korotkov wrote: > Please, find attached two patches I'm going to commit: for master and > for backpatching. So, pushed. -- Alexander Korotkov Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Tid scan improvements

2018-12-20 Thread Andres Freund
Hi, On 2018-12-20 18:06:41 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2018-12-20 17:21:07 -0500, Tom Lane wrote: > >> I'm having a hard time wrapping my mind around why you'd bother with > >> backwards TID scans. > > > I've not followed this thread, but wouldn't that be quite useful

Re: Tid scan improvements

2018-12-20 Thread Tom Lane
Andres Freund writes: > On 2018-12-20 17:21:07 -0500, Tom Lane wrote: >> I'm having a hard time wrapping my mind around why you'd bother with >> backwards TID scans. > I've not followed this thread, but wouldn't that be quite useful to be > able to move old tuples to free space earlier in the

Re: Why are we PageInit'ing buffers in RelationAddExtraBlocks()?

2018-12-20 Thread Andres Freund
Hi, On 2018-12-19 16:56:36 -0800, Andres Freund wrote: > On 2018-12-19 19:39:33 -0500, Tom Lane wrote: > > Robert Haas writes: > > > On Wed, Dec 19, 2018 at 5:37 PM Andres Freund wrote: > > >> What's gained by the logic of emitting that warning in VACUUM after a > > >> crash? I don't really see

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-20 Thread Tom Lane
John Naylor writes: > On 12/18/18, Tom Lane wrote: >> I'd be kind of inclined to convert all uses of ScanKeyword to the new way, >> if only for consistency's sake. On the other hand, I'm not the one >> volunteering to do the work. > That's reasonable, as long as the design is nailed down

Re: Tid scan improvements

2018-12-20 Thread Andres Freund
Hi, On 2018-12-20 17:21:07 -0500, Tom Lane wrote: > Edmund Horner writes: > > [ tid scan patches ] > > I'm having a hard time wrapping my mind around why you'd bother with > backwards TID scans. The amount of code needed versus the amount of > usefulness seems like a pretty bad cost/benefit

Re: Tid scan improvements

2018-12-20 Thread Tom Lane
Edmund Horner writes: > [ tid scan patches ] I'm having a hard time wrapping my mind around why you'd bother with backwards TID scans. The amount of code needed versus the amount of usefulness seems like a pretty bad cost/benefit ratio, IMO. I can see that there might be value in knowing that

monitoring CREATE INDEX [CONCURRENTLY]

2018-12-20 Thread Alvaro Herrera
Monitoring progress of CREATE INDEX [CONCURRENTLY] is sure to be welcome, so here's a proposal. There are three distinct interesting cases. One is straight CREATE INDEX of a standalone table; then we have CREATE INDEX CONCURRENTLY; finally, CREATE INDEX on a partitioned table. Note that there's

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-12-20 Thread Robert Haas
On Thu, Dec 20, 2018 at 4:11 PM Alvaro Herrera wrote: > Oh, so maybe this case is already handled by plan invalidation -- I > mean, if we run DDL, the stored plan is thrown away and a new one > recomputed. IOW this was already a solved problem and I didn't need to > spend effort on it. /me slaps

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-12-20 Thread Alvaro Herrera
On 2018-Dec-20, Robert Haas wrote: > I didn't handle that. If partition pruning relies on nothing changing > between planning and execution, isn't that broken regardless of any of > this? It's true that with the simple query protocol we'll hold locks > continuously from planning into execution,

Re: lock level for DETACH PARTITION looks sketchy

2018-12-20 Thread Alvaro Herrera
On 2018-Dec-20, Robert Haas wrote: > On Thu, Dec 20, 2018 at 3:58 PM Alvaro Herrera > wrote: > > I think what prompted the lock to be AccessShareLock for the child rel > > in the first place is the fact that ATExecDropInherit() (ALTER TABLE NO > > INHERIT) uses AccessShare for the *parent*

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-12-20 Thread Robert Haas
On Thu, Dec 20, 2018 at 3:58 PM Alvaro Herrera wrote: > > Introduce the concept of a partition directory. > > > > Teach the optimizer and executor to use it, so that a single planning > > cycle or query execution gets the same PartitionDesc for the same table > > every time it looks it up. This

Re: lock level for DETACH PARTITION looks sketchy

2018-12-20 Thread Robert Haas
On Thu, Dec 20, 2018 at 3:58 PM Alvaro Herrera wrote: > I think what prompted the lock to be AccessShareLock for the child rel > in the first place is the fact that ATExecDropInherit() (ALTER TABLE NO > INHERIT) uses AccessShare for the *parent* relation. Seems like apples and oranges, and also

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-12-20 Thread Alvaro Herrera
Thanks for this work! I like the name "partition directory". On 2018-Dec-20, Robert Haas wrote: > 0002 introduces the concept of a partition directory. The idea is > that the planner will create a partition directory, and so will the > executor, and all calls which occur in those places to >

Re: lock level for DETACH PARTITION looks sketchy

2018-12-20 Thread Alvaro Herrera
I think what prompted the lock to be AccessShareLock for the child rel in the first place is the fact that ATExecDropInherit() (ALTER TABLE NO INHERIT) uses AccessShare for the *parent* relation. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support,

Re: Add timeline to partial WAL segments

2018-12-20 Thread Robert Haas
On Fri, Dec 14, 2018 at 6:05 PM David Steele wrote: > The question in my mind: is it safe to back-patch? I cannot imagine it being a good idea to stick a behavioral change like this into a minor release. Yeah, it lets people get out from under this problem a lot sooner, but it potentially

Re: ATTACH/DETACH PARTITION CONCURRENTLY

2018-12-20 Thread Robert Haas
On Tue, Dec 18, 2018 at 8:04 PM Michael Paquier wrote: > On Tue, Dec 18, 2018 at 01:41:06PM -0500, Robert Haas wrote: > > OK. I'll post what I have by the end of the week. > > Thanks, Robert. OK, so I got slightly delayed here by utterly destroying my laptop, but I've mostly reconstructed what

Re: Switching to 64-bit Bitmapsets

2018-12-20 Thread David Rowley
On Fri, 21 Dec 2018 at 06:26, Tom Lane wrote: > Pushed with some fiddling with the comment. Great. Thanks! > I wasn't excited about the test case you offered --- on HEAD, it pretty > much all devolves to file access operations (probably, checking the > current length of all the child

Re: lock level for DETACH PARTITION looks sketchy

2018-12-20 Thread Alvaro Herrera
On 2018-Dec-20, Robert Haas wrote: > One problem about which I thought is the partition check constraint. > When we attach, we need to validate that the check constraint holds of > every row in the partition, which means that we need to keep new rows > from being added while we're attaching. But

Performance issue in foreign-key-aware join estimation

2018-12-20 Thread Tom Lane
In connection with David Rowley's proposal to change bitmapset.c to use 64-bit words, I dug out an old test case I had for a complex-to-plan query (attached). Andres Freund posted this to the lists perhaps ten years ago, though I can't locate the original posting right now. I was distressed to

Re: lock level for DETACH PARTITION looks sketchy

2018-12-20 Thread Robert Haas
On Thu, Dec 20, 2018 at 9:29 AM Alvaro Herrera wrote: > I can patch that one too, but it's separate -- it goes back to pg10 I > think (the other to pg11) -- and let's think about the lock mode for a > bit: as far as I can see, ShareUpdateExclusive is enough; the difference > with AccessExclusive

Re: Switching to 64-bit Bitmapsets

2018-12-20 Thread Tom Lane
David Rowley writes: > On Thu, 20 Dec 2018 at 17:50, Tom Lane wrote: >> Hm, are you thinking of making BITS_PER_BITMAPWORD match sizeof(Pointer) >> or something like that? That seems like a good compromise from here. > Yeah, something along those lines. I've implemented that in the attached.

Re: A case for UPDATE DISTINCT attribute

2018-12-20 Thread Alexey Bashtanov
Hello Gajus, I have observed that the following pattern is repeating in our data management programs: UPDATE   event SET   fuid = ${fuid},   venue_id = ${venueId},   url = ${url} WHERE   id = ${id} AND   fuid IS != ${fuid} AND   venue_id IS != ${venueId} AND   url IS DISTINCT FROM ${url};

Re: A few new options for vacuumdb

2018-12-20 Thread Bossart, Nathan
Hi Michael, Thanks for taking a look. On 12/19/18, 8:05 PM, "Michael Paquier" wrote: > On Wed, Dec 19, 2018 at 08:50:10PM +, Bossart, Nathan wrote: >> If an option is specified for a server version that is not supported, >> the option is silently ignored. For example, SKIP_LOCKED was only

Re: insensitive collations

2018-12-20 Thread Daniel Verite
Tom Lane wrote: > I don't really find it "natural" for equality to consider obviously > distinct values to be equal. According to https://www.merriam-webster.com/dictionary/natural "natural" has no less than 15 meanings. The first in the list is "based on an inherent sense of right and

Re: lock level for DETACH PARTITION looks sketchy

2018-12-20 Thread Alvaro Herrera
On 2018-Dec-19, Robert Haas wrote: > On Wed, Dec 19, 2018 at 2:44 PM Alvaro Herrera > wrote: > > Oh, I remember eyeing that suspiciously, but was too distracted on > > making the other thing work to realize it was actually wrong :-( > > I agree that it's wrong. > > OK, cool. If you're going

Re: Index Skip Scan

2018-12-20 Thread Dmitry Dolgov
> On Wed, Nov 21, 2018 at 9:56 PM Dmitry Dolgov <9erthali...@gmail.com> wrote: > > > On Wed, Nov 21, 2018 at 4:38 PM Alexander Kuzmenkov > > wrote: > > > > On 11/18/18 02:27, Dmitry Dolgov wrote: > > > > > > [0001-Index-skip-scan-v4.patch] > > > > I ran a couple of tests on this, please see the

START/END line number for COPY FROM

2018-12-20 Thread Surafel Temesgen
Hi, Currently we can skip header line on COPY FROM but having the ability to skip and stop copying at any line can use to divide long copy operation and enable to copy a subset of the file and skipping footer. Attach is a patch for it Regards Surafel diff --git a/doc/src/sgml/ref/copy.sgml

Re: Add timeline to partial WAL segments

2018-12-20 Thread David Steele
On 12/15/18 1:56 AM, Michael Paquier wrote: On Fri, Dec 14, 2018 at 06:05:18PM -0500, David Steele wrote: On 12/14/18 3:26 PM, Robert Haas wrote: The new TLI is the only thing that is guaranteed to be unique with each new promotion, and I would guess that it is therefore the right thing to use

Re: Change pgarch_readyXlog() to return .history files first

2018-12-20 Thread David Steele
On 12/15/18 2:10 AM, Michael Paquier wrote: On Fri, Dec 14, 2018 at 08:43:20AM -0500, David Steele wrote: On 12/13/18 7:15 PM, Michael Paquier wrote: Or you could just use IsTLHistoryFileName here? We'd have to truncate .ready off the string to make that work, which seems easy enough. Is

Re: Tid scan improvements

2018-12-20 Thread David Rowley
Review of v5: 0001: looks good. 0002: 1. I don't think you need palloc0() here. palloc() looks like it would be fine. if (tidRangeArray->ranges == NULL) tidRangeArray->ranges = (TidRange *) palloc0(tidRangeArray->numAllocated * sizeof(TidRange)); if that wasn't the case, then you'll need to

Re: Using POPCNT and other advanced bit manipulation instructions

2018-12-20 Thread Jose Luis Tallon
On 20/12/18 6:53, David Rowley wrote: Back in 2016 [1] there was some discussion about using the POPCNT instruction to improve the performance of counting the number of bits set in a word. Improving this helps various cases, such as bms_num_members and also things like counting the allvisible

[patch] de.po REINDEX error

2018-12-20 Thread Christoph Berg
de.po's error message when you try to "REINDEX DATABASE otherdb" has been the wrong way round since 2011: diff --git a/src/backend/po/de.po b/src/backend/po/de.po index ca52df6731..6aa4354d82 100644 --- a/src/backend/po/de.po +++ b/src/backend/po/de.po @@ -7540,7 +7540,7 @@ msgstr "Tabelle »%s«

Re: Remove Deprecated Exclusive Backup Mode

2018-12-20 Thread David Steele
On 12/20/18 1:35 AM, Michael Paquier wrote: On Wed, Dec 19, 2018 at 06:38:00PM +0200, David Steele wrote: I'll push this to the first post-PG12 CF when one appears. Can we just go ahead and create a 2019-07 CF now? I know we generally discuss this at PGCon, but we can always rename it, right?

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-12-20 Thread Kyotaro HORIGUCHI
Hello. At Fri, 30 Nov 2018 18:27:05 +0100, Dmitry Dolgov <9erthali...@gmail.com> wrote in > > On Wed, Nov 14, 2018 at 4:48 AM Kyotaro HORIGUCHI > > wrote: > > > > 0004 was shot by e9edc1ba0b. Rebased to the current HEAD. > > Successfully built and passeed all regression/recovery tests > >

Improve selectivity estimate for range queries

2018-12-20 Thread Yuzuko Hosoya
Hi, I found the problem about selectivity estimate for range queries on master as follows. This is my test case: postgres=# CREATE TABLE test(id int, val text); CREATE TABLE postgres=# INSERT INTO test SELECT i, 'testval' FROM generate_series(0,2)i; INSERT 0 3 postgres=# VACUUM analyze

RE: Localization tools for Postgres

2018-12-20 Thread Ideriha, Takeshi
Hi cc’ed pgsql-translators I personally uses ‘poedit’ when working around po files. Takeshi Ideriha Fujitsu Limited From: Дмитрий Воронин [mailto:carriingfat...@yandex.ru] Sent: Thursday, December 20, 2018 12:54 AM To: pgsql-hack...@postgresql.org Subject: Localization

Re: Using POPCNT and other advanced bit manipulation instructions

2018-12-20 Thread David Rowley
On Thu, 20 Dec 2018 at 20:17, Gavin Flower wrote: > Looking at the normalized standard deviations, the patched results have > a higher than 5% chance of being better simply by chance. I suspect > that you have made an improvement, but the statistics are not convincing. Yeah, I'd hoped that I