Re: [WIP] CREATE SUBSCRIPTION with FOR TABLES clause (table filter)

2018-12-26 Thread Evgeniy Efimkin
Hello! In latest patch i removed `FOR ALL TABLES` clause and `alltables` parameter, now it's look more simple. Add new system view pg_user_subscirption to allow non-superuser use pg_dump and select addition column from pg_subscrption Changes docs. Thanks! Ефимкин Евгений diff --git

RE: Speeding up creating UPDATE/DELETE generic plan for partitioned table into a lot

2018-12-26 Thread Kato, Sho
Hi, Amit Thank you for your reply. > What do you mean by "since the partitions to access are partial"? I mean planner create scan nodes based on the parameters specified for EXECUTE and backend keep them in CachedPlan. If CachedPlan does not have a scan node for accessing partition, planning

RE: Cache relation sizes?

2018-12-26 Thread Jamison, Kirk
Hello, I also find this proposed feature to be beneficial for performance, especially when we want to extend or truncate large tables. As mentioned by David, currently there is a query latency spike when we make generic plan for partitioned table with many partitions. I tried to apply Thomas'

Re: random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Thomas Munro
On Thu, Dec 27, 2018 at 3:55 PM Peter Geoghegan wrote: > On Wed, Dec 26, 2018 at 6:39 PM Tom Lane wrote: > > The point here is not to be cryptographically strong at every single > > place where the backend might want a random number; I think we're > > all agreed that we don't need that. To me,

Re: removal of dangling temp tables

2018-12-26 Thread Michael Paquier
On Wed, Dec 26, 2018 at 08:51:56PM -0300, Alvaro Herrera wrote: > Having been victim of ABI incompatibility myself, I loathe giving too > much priority to other issues that can be resolved in other ways, so I > don't necessarily support your view on bugs. > That said, I think in this case it

Re: random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Peter Geoghegan
On Wed, Dec 26, 2018 at 6:39 PM Tom Lane wrote: > The point here is not to be cryptographically strong at every single > place where the backend might want a random number; I think we're > all agreed that we don't need that. To me, the point is to ensure that > the user-accessible random

Re: [PATCH] kNN for btree

2018-12-26 Thread Alexander Korotkov
Hi! On Fri, Nov 30, 2018 at 3:02 PM Nikita Glukhov wrote: > On 29.11.2018 18:24, Dmitry Dolgov wrote: > >> On Wed, Sep 26, 2018 at 5:41 PM Nikita Glukhov > >> wrote: > >> > >> Attached 3rd version of the patches rebased onto the current master. > >> > >> Changes from the previous version: > >>

Re: pgsql: Fix failure to check for open() or fsync() failures.

2018-12-26 Thread Michael Paquier
On Wed, Dec 26, 2018 at 08:35:22PM -0500, Tom Lane wrote: > Oh, I see. Yeah, if we're ignoring errors anyway, the stat calls > seem redundant. For this one, I think that we could simplify as attached (this causes open() to fail additionally because of the sync flags, but that's not really worth

Re: random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Michael Paquier
On Wed, Dec 26, 2018 at 08:46:25PM -0500, Tom Lane wrote: > One thing I was wondering is if we should try to enforce a policy > like this by putting, say, > > #define random() pg_random() > > into port.h or so. That would have the advantages of not having to touch > any existing calls and not

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2018-12-26 Thread Michael Paquier
On Wed, Dec 26, 2018 at 03:19:06PM -0300, Alvaro Herrera wrote: > As for REINDEX, I think it's valuable to move tablespace together with > the reindexing. You can already do it with the CREATE INDEX > CONCURRENTLY recipe we recommend, of course; but REINDEX CONCURRENTLY is > not going to provide

[information] Winter vacation

2018-12-26 Thread Nagaura, Ryohei
Hi all. -Information- Members of Fujitsu Japan may not be able to reply in the term below. TERM: 29th December 2018 ~ 6th January 2019 NOTE: Members of Fujitsu Japan are those whose mail domain is "@jp.fujitsu.com" Best regards, - Ryohei Nagaura

Re: random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Tom Lane
Michael Paquier writes: > On Wed, Dec 26, 2018 at 01:45:00PM -0500, Tom Lane wrote: >> A quick grep says that there's a dozen or so callers, so this patch >> certainly isn't the only offender ... but should we make an effort >> to convert them all to use, say, pg_erand48()? I think all the >>

Re: pgsql: Fix failure to check for open() or fsync() failures.

2018-12-26 Thread Michael Paquier
On Wed, Dec 26, 2018 at 05:55:36PM -0500, Tom Lane wrote: > It appears to me that the code is intentionally not worrying about > fsync failure, so it seems wrong for it to FATAL out if it's unable > to open the file to fsync it. And it surely shouldn't do so if the > file isn't there. My point

RE: removal of dangling temp tables

2018-12-26 Thread Tsunakawa, Takayuki
From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > Having been victim of ABI incompatibility myself, I loathe giving too much > priority to other issues that can be resolved in other ways, so I don't > necessarily support your view on bugs. > That said, I think in this case it shouldn't be a

Re: Shared Memory: How to use SYSV rather than MMAP ?

2018-12-26 Thread Thomas Munro
On Thu, Dec 27, 2018 at 8:59 AM Thomas Munro wrote: > ... So far I can't reproduce the effect here, probably due to > lack of cores and probably also various changes that have been made > (but not the main ones described in that report, apparently). I decided to blow today's coffee money on a

Re: removal of dangling temp tables

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-26, Tsunakawa, Takayuki wrote: > From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > > The more aggressive action is to backpatch 943576bddcb5 ("Make autovacuum > > more aggressive to remove orphaned temp tables") which is currently only > > in pg11. We would put the new PGPROC

RE: removal of dangling temp tables

2018-12-26 Thread Tsunakawa, Takayuki
From: Alvaro Herrera [mailto:alvhe...@2ndquadrant.com] > The more aggressive action is to backpatch 943576bddcb5 ("Make autovacuum > more aggressive to remove orphaned temp tables") which is currently only > in pg11. We would put the new PGPROC member at the end of the struct, to > avoid ABI

Re: pgsql: Fix failure to check for open() or fsync() failures.

2018-12-26 Thread Tom Lane
Michael Paquier writes: > On Wed, Dec 26, 2018 at 09:08:23PM +, Tom Lane wrote: >> Fix failure to check for open() or fsync() failures. >> >> While it seems OK to not be concerned about fsync() failure for a >> pre-existing signal file, it's not OK to not even check for open() >> failure.

Re: Move regression.diffs of pg_upgrade test suite

2018-12-26 Thread Noah Misch
On Wed, Dec 26, 2018 at 05:02:37PM -0500, Tom Lane wrote: > Andrew Dunstan writes: > > On 12/23/18 10:44 PM, Noah Misch wrote: > >> A disadvantage of any change here is that it degrades buildfarm reports, > >> which > >> recover slowly as owners upgrade to a fixed buildfarm release. This will

Re: pgsql: Fix failure to check for open() or fsync() failures.

2018-12-26 Thread Michael Paquier
On Wed, Dec 26, 2018 at 09:08:23PM +, Tom Lane wrote: > Fix failure to check for open() or fsync() failures. > > While it seems OK to not be concerned about fsync() failure for a > pre-existing signal file, it's not OK to not even check for open() > failure. This at least causes complaints

Re: Move regression.diffs of pg_upgrade test suite

2018-12-26 Thread Tom Lane
Andrew Dunstan writes: > On 12/23/18 10:44 PM, Noah Misch wrote: >> A disadvantage of any change here is that it degrades buildfarm reports, >> which >> recover slowly as owners upgrade to a fixed buildfarm release. This will be >> similar to the introduction of --outputdir=output_iso. On

Re: Small doc tweak for array/string functions

2018-12-26 Thread Tom Lane
Ian Barwick writes: > On these pages: > - https://www.postgresql.org/docs/current/functions-array.html > - https://www.postgresql.org/docs/current/functions-string.html > we point out via "See also" the existence of aggregate array and string > functions, but I think it would be useful to

Re: random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Peter Geoghegan
On Wed, Dec 26, 2018 at 1:21 PM Tom Lane wrote: > One thing we'd have to think about if we want to take this seriously > is whether a process-wide PRNG state is really adequate; if you're > trying to make a particular call site be deterministic, you'd likely > wish it weren't interfered with by

Re: Speeding up text_position_next with multibyte encodings

2018-12-26 Thread John Naylor
On 12/22/18, Heikki Linnakangas wrote: > On 14/12/2018 20:20, John Naylor wrote: > I'm afraid that script doesn't work as a performance test. The > position() function is immutable, so the result gets cached in the plan > cache. All you're measuring is the speed to get the constant from the >

Re: random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Tom Lane
Peter Geoghegan writes: > On Wed, Dec 26, 2018 at 12:19 PM Tom Lane wrote: >> Replacing random() might actually make that easier not harder, since >> we'd have more control over what happens when. > That does seem useful. I'm in favor. But why does the function to seed > the internal PRNG have

Re: Feature: temporary materialized views

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-26, Mitar wrote: > OptTemp seems to have a misleading warning in some cases when it is > not used on tables though: > > "GLOBAL is deprecated in temporary table creation" > > Should we change this language to something else? "GLOBAL is > deprecated in temporary object creation"?

Re: random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Peter Geoghegan
On Wed, Dec 26, 2018 at 12:19 PM Tom Lane wrote: > Replacing random() might actually make that easier not harder, since > we'd have more control over what happens when. That does seem useful. I'm in favor. But why does the function to seed the internal PRNG have to be loadable? Can't it just be

Re: random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Tom Lane
Peter Geoghegan writes: > I'm beginning to think that the technique that I came up with to make > "getting tired" deterministic ought to be supporting as a debugging > option if we're to do away with internal use of the generic/seedable > backend PRNG. I have no objection to providing such a

Re: Shared Memory: How to use SYSV rather than MMAP ?

2018-12-26 Thread Thomas Munro
On Thu, Dec 27, 2018 at 6:48 AM Robert Haas wrote: > On Wed, Dec 26, 2018 at 11:43 AM Tom Lane wrote: > > Thomas Munro writes: > > > Since it's not fixing a bug, we wouldn't back-patch that into existing > > > releases. But I agree that we should do something like this for > > > PostgreSQL 12,

Re: random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Tom Lane
Peter Geoghegan writes: > On Wed, Dec 26, 2018 at 10:45 AM Tom Lane wrote: >> I wonder whether we should establish a project policy to avoid use >> of random() for internal purposes, ie try to get to a point where >> drandom() is the only caller in the backend. A quick grep says >> that there's

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-26 Thread Tom Lane
Andres Freund writes: > On 2018-12-26 10:45:11 -0500, Robert Haas wrote: >> I'm not sure that I understand quite what you have in mind for a >> serialized non-perfect hashtable. Are you thinking that we'd just >> construct a simplehash and serialize it? > I was basically thinking that we'd have

Re: random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Peter Geoghegan
On Wed, Dec 26, 2018 at 10:45 AM Tom Lane wrote: > I wonder whether we should establish a project policy to avoid use > of random() for internal purposes, ie try to get to a point where > drandom() is the only caller in the backend. A quick grep says > that there's a dozen or so callers, so this

Re: removal of dangling temp tables

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-16, Michael Paquier wrote: > On Sat, Dec 15, 2018 at 09:51:31AM -0500, Tom Lane wrote: > > Alvaro Herrera writes: > >> Oh, we already have it! Sorry, I overlooked it. With that, it seems > >> the patch is fairly simple ... I wonder about the locking implications > >> in autovacuum,

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-26 Thread Tom Lane
Andres Freund writes: > My bet is, and has been for quite a while, that we'll have to go for a > hand-written recursive descent type parser. I will state right up front that that will happen over my dead body. It's impossible to write correct RD parsers by hand for any but the most trivial,

Re: Shared Memory: How to use SYSV rather than MMAP ?

2018-12-26 Thread Tom Lane
Robert Haas writes: > On Wed, Dec 26, 2018 at 11:43 AM Tom Lane wrote: >> I'm -1 on making this user configurable via a GUC; that adds documentation >> and compatibility burdens that we don't need, for something of no value >> to 99.99% of users. > ... > You may be right that this is of no value

random() (was Re: New GUC to sample log queries)

2018-12-26 Thread Tom Lane
Alvaro Herrera writes: > Thanks! I pushed this with two changes -- one was to reword the docs a > bit more, and the other was to compute in_sample only if it's going to > be used (when exceeded is true). I hope this won't upset any compilers ... > I wonder if there's any sensible way to verify

Re: Offline enabling/disabling of data checksums

2018-12-26 Thread Fabien COELHO
Hallo Michael, It adds an (now mandatory) --action parameter that takes either verify, enable or disable as argument. I'd rather have explicit switches for verify, enable & disable, and verify would be the default if none is provided. This is basically meant as a stop-gap measure in case

Re: Shared Memory: How to use SYSV rather than MMAP ?

2018-12-26 Thread Andres Freund
On December 26, 2018 6:48:31 PM GMT+01:00, Robert Haas wrote: >I disagree. I think there is a growing body of evidence that >b0fc0df9364d2d2d17c0162cf3b8b59f6cb09f67 killed performance on many >types of non-Linux systems. This is the first report I recall about >AIX, but there have been

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-26, Robert Haas wrote: > On Mon, Dec 24, 2018 at 6:08 AM Alexey Kondratov > wrote: > > I would like to propose a change, which allow CLUSTER, VACUUM FULL and > > REINDEX to modify relation tablespace on the fly. > > ALTER TABLE already has a lot of logic that is oriented towards

Re: Allow CLUSTER, VACUUM FULL and REINDEX to change tablespace on the fly

2018-12-26 Thread Robert Haas
On Mon, Dec 24, 2018 at 6:08 AM Alexey Kondratov wrote: > I would like to propose a change, which allow CLUSTER, VACUUM FULL and > REINDEX to modify relation tablespace on the fly. ALTER TABLE already has a lot of logic that is oriented towards being able to do multiple things at the same time.

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-26 Thread Andres Freund
Hi, On 2018-12-26 10:45:11 -0500, Robert Haas wrote: > I'm not sure that I understand quite what you have in mind for a > serialized non-perfect hashtable. Are you thinking that we'd just > construct a simplehash and serialize it? I was basically thinking that we'd have the perl script

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-26 Thread Andres Freund
Hi, On 2018-12-26 11:50:18 -0500, Robert Haas wrote: > On Wed, Dec 26, 2018 at 11:22 AM Tom Lane wrote: > > I think there's a lot of goalpost-moving going on here. The original > > idea was to trim the physical size of the data structure, as stated > > in the thread subject, and just reap

Re: "repliation" as database name

2018-12-26 Thread Tom Lane
Kyotaro HORIGUCHI writes: > I found that in the documentation thanks to a notification > off-list. And after some reconfirmation, what I want to fix is > only a few lines of comment in pg_hba.conf.sample. > -# database name, or a comma-separated list thereof. The "all" > -# keyword does not

Re: Offline enabling/disabling of data checksums

2018-12-26 Thread Robert Haas
On Fri, Dec 21, 2018 at 6:28 PM Michael Paquier wrote: > 2) Which kind of interface do we want to use? When I did my own > flavor of pg_checksums, I used an --action switch able to use the > following values: > - enable > - disable > - verify > The switch cannot be specified twice (perhaps we

Re: Shared Memory: How to use SYSV rather than MMAP ?

2018-12-26 Thread Robert Haas
On Wed, Dec 26, 2018 at 11:43 AM Tom Lane wrote: > Thomas Munro writes: > > Since it's not fixing a bug, we wouldn't back-patch that into existing > > releases. But I agree that we should do something like this for > > PostgreSQL 12, and I think we should make it user configurable. > > I'm -1

Re: Feature: temporary materialized views

2018-12-26 Thread Pavel Stehule
st 26. 12. 2018 v 18:20 odesílatel Mitar napsal: > Hi! > > On Wed, Dec 26, 2018 at 9:00 AM Alvaro Herrera > wrote: > > I think MVs that are dropped at session end are a sensible feature. > > Thanks. > > > I probably wouldn't go as far as allowing ON COMMIT actions, though > > I agree. I do not

Re: Feature: temporary materialized views

2018-12-26 Thread Mitar
Hi! On Wed, Dec 26, 2018 at 9:00 AM Alvaro Herrera wrote: > I think MVs that are dropped at session end are a sensible feature. Thanks. > I probably wouldn't go as far as allowing ON COMMIT actions, though I agree. I do not see much usefulness for it. The only use case I can think of would be

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-26 Thread Tom Lane
Robert Haas writes: > I'm kinda surprised that you haven't seen ScanKeywordLookup() in > there, but I agree with you that the size of the main parser tables is > a real issue, and that there's no easy solution. At various times > there has been discussion of using some other parser generator, and

Re: Feature: triggers on materialized views

2018-12-26 Thread Mitar
Hi! On Wed, Dec 26, 2018 at 4:38 AM Alvaro Herrera wrote: > Sounds like you could do this by fixing concurrent refresh to also work > when the MV is WITH NO DATA. Yes, I do not think this would be too hard to fix. I could do this nevertheless. > > Ah, yes. I could just do TRUNCATE and INSERT,

Re: Feature: temporary materialized views

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-25, Mitar wrote: > Sometimes materialized views are used to cache a complex query on > which a client works. But after client disconnects, the materialized > view could be deleted. Regular VIEWs and TABLEs both have support for > temporary versions which get automatically dropped at

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-26 Thread Robert Haas
On Wed, Dec 26, 2018 at 11:22 AM Tom Lane wrote: > I think there's a lot of goalpost-moving going on here. The original > idea was to trim the physical size of the data structure, as stated > in the thread subject, and just reap whatever cache benefits we got > along the way from that. I am

Re: Shared Memory: How to use SYSV rather than MMAP ?

2018-12-26 Thread Tom Lane
Thomas Munro writes: > Since it's not fixing a bug, we wouldn't back-patch that into existing > releases. But I agree that we should do something like this for > PostgreSQL 12, and I think we should make it user configurable. I'm -1 on making this user configurable via a GUC; that adds

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-26 Thread Tom Lane
Robert Haas writes: > On Wed, Dec 19, 2018 at 8:01 PM Andres Freund wrote: >> The last time I looked into perfect hash functions, it wasn't easy to >> find a generator that competed with a decent normal hashtable (in >> particular gperf's are very unconvincing). The added tooling is a >> concern

Re: reducing the footprint of ScanKeyword (was Re: Large writable variables)

2018-12-26 Thread Robert Haas
On Wed, Dec 19, 2018 at 8:01 PM Andres Freund wrote: > The last time I looked into perfect hash functions, it wasn't easy to > find a generator that competed with a decent normal hashtable (in > particular gperf's are very unconvincing). The added tooling is a > concern imo. OTOH, we're

Re: Change pgarch_readyXlog() to return .history files first

2018-12-26 Thread David Steele
On 12/24/18 1:31 PM, Michael Paquier wrote: On Sat, Dec 22, 2018 at 08:55:14AM +0900, Michael Paquier wrote: Thanks for the lookups. I can see that the patch applies without conflicts down to 9.4, and based on the opinions gathered on this thread back-patching this stuff is the consensus,

Re: [Patch] pg_rewind: options to use restore_command from recovery.conf or command line

2018-12-26 Thread Alexey Kondratov
Greetings, - Reusing the GUC parser is something I would avoid as well.  Not worth the complexity. Yes, I don't like it either. I will try to make guc-file.l frontend safe. Any success with that? I looked into it and found that currently guc-file.c is built as part of guc.c, so it seems

Re: Alternative to \copy in psql modelled after \g

2018-12-26 Thread Fabien COELHO
Hello, Is this just kind of a bug fix? Beforehand the documentation says "\g fn" sends to file, but that was not happening with COPY, and now it does as it says? Yes. [...] It does not add "unless the query is a COPY", so it seems right to make that just work, and call it a bug fix. Does

Re: Feature: triggers on materialized views

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-25, Mitar wrote: > On Tue, Dec 25, 2018 at 7:05 PM Alvaro Herrera > wrote: > > But then I'm not clear *why* you would like to do a non-concurrent > > refresh. > > I mostly wanted to support if for two reasons: > > - completeness: maybe we cannot imagine the use case yet, but

Re: chained transactions

2018-12-26 Thread Alvaro Herrera
On 2018-Dec-26, Fabien COELHO wrote: > > > Copying & comparing nodes are updated. Should making, outing and reading > > > nodes also be updated? > > > > TransactionStmt isn't covered by the node serialization functions, so I > > didn't see anything to update. What did you have in mind? > >

Re: Alternative to \copy in psql modelled after \g

2018-12-26 Thread Daniel Verite
Fabien COELHO wrote: > Is this just kind of a bug fix? Beforehand the documentation says "\g fn" > sends to file, but that was not happening with COPY, and now it does as it > says? Yes. The doc says about \g: Sends the current query buffer to the server for execution. If an

Re: CF app feature request

2018-12-26 Thread Magnus Hagander
On Sun, Dec 23, 2018 at 3:59 PM Alvaro Herrera wrote: > On 2018-Dec-23, Magnus Hagander wrote: > > > On Wed, Nov 21, 2018 at 12:52 AM Michael Paquier > > wrote: > > > > > On Tue, Nov 20, 2018 at 03:30:38PM -0300, Alvaro Herrera wrote: > > > > On 2018-Nov-20, Tom Lane wrote: > > > > Certainly

Re: chained transactions

2018-12-26 Thread Fabien COELHO
Updated patch attached. The previous (v2) patch apparently didn't apply anymore. Second patch applies cleanly, compiles, "make check" ok. Also about testing, I'd do less rounds, 4 quite seems enough. -- Fabien.

Re: chained transactions

2018-12-26 Thread Fabien COELHO
Updated patch attached. The previous (v2) patch apparently didn't apply anymore. Second patch applies cleanly, compiles, "make check" ok. As I do not know much about the SPI stuff, some of the comments below may be very stupid. I'm wary of changing the SPI_commit and SPI_rollback

Re: Feature: triggers on materialized views

2018-12-26 Thread Mitar
Hi! I did a bit of benchmarking. It seems my version with UPDATE takes even slightly less time (~5%). Mitar On Mon, Dec 24, 2018 at 6:17 PM Mitar wrote: > > Hi! > > I made another version of the patch. This one does UPDATEs for changed > row instead of DELETE/INSERT. > > All existing