Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-08-21 Thread Michael Paquier
On Thu, Aug 22, 2019 at 12:36:10AM +0900, Masahiko Sawada wrote: > I will update the patch and register to the next Commit Fest tomorrow > if nobody is interested in. Thanks, Sawada-san. -- Michael signature.asc Description: PGP signature

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Dilip Kumar
On Thu, Aug 22, 2019 at 10:24 AM Andres Freund wrote: > > Hi, > > On 2019-08-22 10:19:04 +0530, Dilip Kumar wrote: > > On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote: > > > > We can not know the complete size of

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Dilip Kumar
On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote: > > Hi, > > On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote: > > We can not know the complete size of the record even by reading the > > header because we have a payload that is variable part and payload > > length are stored in the payload

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Andres Freund
Hi, On 2019-08-22 10:19:04 +0530, Dilip Kumar wrote: > On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote: > > > > Hi, > > > > On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote: > > > We can not know the complete size of the record even by reading the > > > header because we have a payload that

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Andres Freund
Hi, On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote: > We can not know the complete size of the record even by reading the > header because we have a payload that is variable part and payload > length are stored in the payload header which again can be at random > offset. Wait, but that's just

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Dilip Kumar
On Wed, Aug 21, 2019 at 9:04 PM Robert Haas wrote: > > On Wed, Aug 21, 2019 at 3:55 AM Dilip Kumar wrote: > > I have already attempted that part and I feel it is not making code > > any simpler than what we have today. For packing, it's fine because I > > can process all the member once and

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-21 Thread Noah Misch
On Wed, Aug 21, 2019 at 04:32:38PM +0900, Kyotaro Horiguchi wrote: > At Mon, 19 Aug 2019 18:59:59 +0900 (Tokyo Standard Time), Kyotaro Horiguchi > wrote in > <20190819.185959.118543656.horikyota@gmail.com> > > At Sat, 17 Aug 2019 20:52:30 -0700, Noah Misch wrote in > >

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-21 Thread Michael Paquier
On Wed, Aug 21, 2019 at 12:25:22PM -0400, Stephen Frost wrote: > That'd be the kind of thing that I was really hoping we could provide a > common library for. Indeed. There could be many use cases for that. Most of the parsing logic is in guc-file.l. There is little dependency to elog() and

Re: Grouping isolationtester tests in the schedule

2019-08-21 Thread Michael Paquier
On Wed, Aug 07, 2019 at 03:17:02PM +0300, Heikki Linnakangas wrote: > On 07/08/2019 14:42, Thomas Munro wrote: >> I think I'd put nowait and skip locked under a separate category "FOR >> UPDATE" or "row locking" or something, but maybe that's just me... can >> you call that stuff DML? > > Yeah, I

Re: [proposal] de-TOAST'ing using a iterator

2019-08-21 Thread John Naylor
On Thu, Aug 22, 2019 at 12:10 AM Binguo Bao wrote: > [v9 patch] Thanks, looks good. I'm setting it to ready for committer. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Cleanup isolation specs from unused steps

2019-08-21 Thread Michael Paquier
On Wed, Aug 21, 2019 at 11:07:19AM -0700, Melanie Plageman wrote: > So, I think I completely misunderstood the purpose of 'dry-run'. If no > one is using it, having a check for unused steps in dry-run may not be > useful. Okay. After sleeping on it and seeing how this thread evolves, it looks

Re: Why overhead of SPI is so large?

2019-08-21 Thread Kyotaro Horiguchi
Hello. At Wed, 21 Aug 2019 19:41:08 +0300, Konstantin Knizhnik wrote in > Hi, hackers. > > One of our customers complains about slow execution of PL/pgSQL > functions comparing with Oracle. > So he wants to compile PL/pgSQL functions (most likely just-in-time > compilation). > Certainly

Re: Remove page-read callback from XLogReaderState.

2019-08-21 Thread Kyotaro Horiguchi
Thank you for the suggestion, Heikki. At Mon, 29 Jul 2019 22:39:57 +0300, Heikki Linnakangas wrote in > On 12/07/2019 10:10, Kyotaro Horiguchi wrote: > >> Just FYI, to me this doesn't clearly enough look like an improvement, > >> for a change of this size. > > Thanks for the opiniton. I kinda

RE: Why overhead of SPI is so large?

2019-08-21 Thread Tsunakawa, Takayuki
From: Konstantin Knizhnik [mailto:k.knizh...@postgrespro.ru] > PL/pgSQL: 29044.361 ms > C/SPI: 22785.597 ms > > The fact that difference between PL/pgSQL and function implemented in C > using SPI is not so large was expected by me. This PL/pgSQL overhead is not so significant

Does TupleQueueReaderNext() really need to copy its result?

2019-08-21 Thread Thomas Munro
Hi, A comment in tqueue.c says that the bytes return by shm_mq_receive() "had better be sufficiently aligned", before assigning the pointer to htup.t_data. Then it copies htup and returns the copy (and it did so in the earlier version that had all the remapping stuff, too, but sometimes it

XPRS

2019-08-21 Thread Thomas Munro
Hello, After rereading some old papers recently, I wanted to share some thoughts about XPRS and modern PostgreSQL. XPRS stood for "eXtended Postgres on RAID and Sprite", and was a research project done nearly three decades ago at Berkeley by the POSTGRES group working with operating system

Re: Adding a test for speculative insert abort case

2019-08-21 Thread Melanie Plageman
On Wed, Aug 7, 2019 at 1:47 PM Melanie Plageman wrote: > > > On Wed, Jul 24, 2019 at 11:48 AM Andres Freund wrote: > >> > diff --git a/src/test/isolation/specs/insert-conflict-specconflict.spec >> b/src/test/isolation/specs/insert-conflict-specconflict.spec >> > index 3a70484fc2..7f29fb9d02

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Robert Haas
On Wed, Aug 14, 2019 at 2:57 AM Andres Freund wrote: > - My reading of the current xact.c integration is that it's not workable > as is. Undo is executed outside of a valid transaction state, > exceptions aren't properly undone, logic would need to be duplicated > to a significant degree,

Re: Change ereport level for QueuePartitionConstraintValidation

2019-08-21 Thread Tom Lane
Sergei Kornilov writes: > I noticed appveyor build on windows is not happy: >> perl buildsetup.pl >> Could not determine contrib module type for alter_table >> at buildsetup.pl line 38. > But I have no idea why. I can't check on windows. Possible I miss some change > while adding new module to

Re: "ago" times on buildfarm status page

2019-08-21 Thread Tom Lane
Andrew Dunstan writes: > What I have done quickly is to store a measure of the clock skew. We > already calculated it but we didn't store it. Initial indications are > that only a few have significant skew. Oh, I didn't know that the server had the ability to measure that. (Yes, I agree that

Re: configure still looking for crypt()?

2019-08-21 Thread Peter Eisentraut
On 2019-08-20 16:07, Tom Lane wrote: > Peter Eisentraut writes: >> I noticed that configure is still looking for crypt() and crypt.h. >> Isn't that long obsolete? >> If so, I suggest to remove it with the attached patch. > > +1 done -- Peter Eisentraut http://www.2ndQuadrant.com/

Re: Change ereport level for QueuePartitionConstraintValidation

2019-08-21 Thread Sergei Kornilov
Hello I noticed appveyor build on windows is not happy: > perl buildsetup.pl > Could not determine contrib module type for alter_table > at buildsetup.pl line 38. But I have no idea why. I can't check on windows. Possible I miss some change while adding new module to tree. Will check. Please

Re: pg_upgrade fails with non-standard ACL

2019-08-21 Thread Bruce Momjian
On Tue, Aug 20, 2019 at 04:38:18PM +0300, Anastasia Lubennikova wrote: > > Solving this in pg_upgrade does seem like it's probably the better > > approach rather than trying to do it in pg_dump. Unfortunately, that > > likely means that all we can do is have pg_upgrade point out to the user > >

Re: "ago" times on buildfarm status page

2019-08-21 Thread Andrew Dunstan
On 8/21/19 11:07 AM, Andrew Dunstan wrote: > On 8/21/19 9:55 AM, Tom Lane wrote: >> The real problem with that column though is that it relies on run start >> times that are self-reported by the buildfarm clients, and some of them >> have system clocks that are many hours off reality. What

Removing pg_pltemplate and creating "trustable" extensions

2019-08-21 Thread Tom Lane
We've repeatedly kicked around the idea of getting rid of the pg_pltemplate catalog in favor of keeping that information directly in the languages' extension files [1][2][3][4]. The primary abstract argument for that is that it removes a way in which our in-tree PLs are special compared to

Re: Cleanup isolation specs from unused steps

2019-08-21 Thread Alvaro Herrera
On 2019-Aug-21, Melanie Plageman wrote: > In Greenplum, we mainly add new tests to a separate isolation > framework (called isolation2) which uses a completely different > syntax. It doesn't use isolationtester at all. So, I haven't had a use > case to add long options to isolationtester yet :)

Re: Cleanup isolation specs from unused steps

2019-08-21 Thread Melanie Plageman
On Tue, Aug 20, 2019 at 6:34 PM Michael Paquier wrote: > On Tue, Aug 20, 2019 at 09:54:56AM -0400, Alvaro Herrera wrote: > > On 2019-Aug-20, Tom Lane wrote: > >> If you can warn in both cases, that'd be OK perhaps. But Alvaro's > >> description of the intended use of dry-run makes it sound like

Re: Cleanup isolation specs from unused steps

2019-08-21 Thread Melanie Plageman
On Mon, Aug 19, 2019 at 7:01 PM Michael Paquier wrote: > > It is rather a pain to pass down custom options to isolationtester. > For example, I have tested the updated version attached after > hijacking -n into isolation_start_test(). Ugly hack, but for testing > that's enough. Do you make use

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Andres Freund
On August 21, 2019 8:36:34 AM PDT, Robert Haas wrote: > We treat LWLockAcquire() as a no-fail operation in many >places; in my opinion, that elog(ERROR) that we have for too many >LWLocks should be changed to elog(PANIC) precisely because we do treat >LWLockAcquire() as no-fail in lots of

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-08-21 Thread Anastasia Lubennikova
20.08.2019 4:04, Peter Geoghegan wrote: On Fri, Aug 16, 2019 at 8:56 AM Anastasia Lubennikova wrote: It seems that now all replace operations are crash-safe. The new patch passes all regression tests, so I think it's ready for review again. I'm looking at it now. I'm going to spend a

Re: [proposal] de-TOAST'ing using a iterator

2019-08-21 Thread Binguo Bao
John Naylor 于2019年8月19日周一 下午12:55写道: > init_toast_buffer(): > > + * Note the constrain buf->position <= buf->limit may be broken > + * at initialization. Make sure that the constrain is satisfied > + * when consume chars. > > s/constrain/constraint/ (2 times) > s/consume/consuming/ > > Also,

Why overhead of SPI is so large?

2019-08-21 Thread Konstantin Knizhnik
Hi, hackers. One of our customers complains about slow execution of PL/pgSQL functions comparing with Oracle. So he wants to compile PL/pgSQL functions (most likely just-in-time compilation). Certainly interpreter adds quite large overhead comparing with native code (~10 times) but most of

Re: [PATCH] Stop ALTER SYSTEM from making bad assumptions

2019-08-21 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > + > + External tools may also > + modify postgresql.auto.conf. It is not > + recommended to do this while the server is running, since a > + concurrent ALTER SYSTEM command could overwrite > + such changes. Such tools

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Robert Haas
On Wed, Aug 21, 2019 at 3:55 AM Dilip Kumar wrote: > I have already attempted that part and I feel it is not making code > any simpler than what we have today. For packing, it's fine because I > can process all the member once and directly pack it into one memory > chunk and I can insert that to

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-08-21 Thread Masahiko Sawada
On Thu, Aug 22, 2019 at 12:19 AM Alvaro Herrera wrote: > > Can I interest someone into updating this patch? We now have (I think) > an agreed design, and I think the development work needed should be > straightforward. We also already have the popcount stuff, so that's a > few lines to be

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Robert Haas
On Wed, Aug 21, 2019 at 6:38 AM Amit Kapila wrote: > > FWIW, although I also thought of doing what you are describing here, I > > think Andres's proposal is probably preferable, because it's simpler. > > There's not really any reason why we can't take the buffer locks from > > within the critical

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-08-21 Thread Alvaro Herrera
Can I interest someone into updating this patch? We now have (I think) an agreed design, and I think the development work needed should be straightforward. We also already have the popcount stuff, so that's a few lines to be removed from the patch ... -- Álvaro Herrera

Re: "ago" times on buildfarm status page

2019-08-21 Thread Andrew Dunstan
On 8/21/19 3:40 AM, Peter Eisentraut wrote: > I find the time displays like > > 01:03 ago > > on the buildfarm status page unhelpful. > > First, I can never tell whether this is hours-minutes or minutes-seconds > -- there is probably a less ambiguous format available. This is

Re: "ago" times on buildfarm status page

2019-08-21 Thread Andrew Dunstan
On 8/21/19 9:55 AM, Tom Lane wrote: > > The real problem with that column though is that it relies on run start > times that are self-reported by the buildfarm clients, and some of them > have system clocks that are many hours off reality. What *I'd* like to > see is for the column to contain

Re: Optimization of vacuum for logical replication

2019-08-21 Thread Konstantin Knizhnik
On 21.08.2019 14:45, Bernd Helmle wrote: Am Mittwoch, den 21.08.2019, 13:26 +0300 schrieb Konstantin Knizhnik: Yes, it is possible to have physical replica withotu replication slot. But it is not safe, because there is always a risk that lag between master and replica becomes larger than

Re: "ago" times on buildfarm status page

2019-08-21 Thread Andrew Dunstan
On 8/21/19 8:32 AM, Dagfinn Ilmari Mannsåker wrote: > Magnus Hagander writes: > >> On Wed, Aug 21, 2019 at 9:40 AM Peter Eisentraut < >> peter.eisentr...@2ndquadrant.com> wrote: >> >>> I find the time displays like >>> >>> 01:03 ago >>> >>> on the buildfarm status page unhelpful. >>> >>>

Re: [HACKERS] WAL logging problem in 9.4.3?

2019-08-21 Thread Kyotaro Horiguchi
Hello. New version is attached. At Mon, 19 Aug 2019 18:59:59 +0900 (Tokyo Standard Time), Kyotaro Horiguchi wrote in <20190819.185959.118543656.horikyota@gmail.com> > Thank you for taking time. > > At Sat, 17 Aug 2019 20:52:30 -0700, Noah Misch wrote in >

Re: "ago" times on buildfarm status page

2019-08-21 Thread Tom Lane
Magnus Hagander writes: > On Wed, Aug 21, 2019 at 9:40 AM Peter Eisentraut < > peter.eisentr...@2ndquadrant.com> wrote: >> I find the time displays like >> 01:03 ago >> on the buildfarm status page unhelpful. >> >> I notice that the page source actually includes absolute times that are >> then

Re: Remove one last occurrence of "replication slave" in comments

2019-08-21 Thread Dagfinn Ilmari Mannsåker
Peter Eisentraut writes: > On 2019-06-19 19:04, Dagfinn Ilmari Mannsåker wrote: >> There were some more master/slave references in the plpgsql foreign key >> tests, which the attached chages to base/leaf instead. > > base/leaf doesn't sound like a good pair. I committed it with root/leaf >

WIP/PoC for parallel backup

2019-08-21 Thread Asif Rehman
Hi Hackers, I have been looking into adding parallel backup feature in pg_basebackup. Currently pg_basebackup sends BASE_BACKUP command for taking full backup, server scans the PGDATA and sends the files to pg_basebackup. In general, server takes the following steps on BASE_BACKUP command: - do

when the IndexScan reset to the next ScanKey for in operator

2019-08-21 Thread Alex
given the following example: postgres=# create table t2 as select generate_series(1, 10) as a, generate_series(1, 10) as b; SELECT 10 postgres=# create index t2_idx on t2(a); CREATE INDEX postgres=# set enable_seqscan = 0; SET postgres=# select * from t2 where a in (1, 10); a

Re: "ago" times on buildfarm status page

2019-08-21 Thread Dagfinn Ilmari Mannsåker
Magnus Hagander writes: > On Wed, Aug 21, 2019 at 9:40 AM Peter Eisentraut < > peter.eisentr...@2ndquadrant.com> wrote: > >> I find the time displays like >> >> 01:03 ago >> >> on the buildfarm status page unhelpful. >> >> First, I can never tell whether this is hours-minutes or

Re: Optimization of vacuum for logical replication

2019-08-21 Thread Bernd Helmle
Am Mittwoch, den 21.08.2019, 13:26 +0300 schrieb Konstantin Knizhnik: > Yes, it is possible to have physical replica withotu replication > slot. > But it is not safe, because there is always a risk that lag between > master and replica becomes larger than size of WAL kept at master. Sure, but

Re: Optimization of vacuum for logical replication

2019-08-21 Thread Sergei Kornilov
Hello > Also I can't believe that  DBA which explicitly sets wal_level is set to > logical will use streaming replication without associated replication slot. I am. > Yes, it is possible to have physical replica withotu replication slot. > But it is not safe, because there is always a risk that

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Amit Kapila
On Tue, Aug 20, 2019 at 8:10 PM Robert Haas wrote: > > On Tue, Aug 20, 2019 at 2:42 AM Amit Kapila wrote: > > > Well, my main point, which so far has largely been ignored, was that we > > > may not acquire page locks when we still need to search for victim > > > buffers later. If we don't need

Re: Optimization of vacuum for logical replication

2019-08-21 Thread Konstantin Knizhnik
On 21.08.2019 12:34, Bernd Helmle wrote: Am Mittwoch, den 21.08.2019, 12:20 +0300 schrieb Konstantin Knizhnik: I wonder if we can check that 1. wal_revel is "logical" 2. There are no physical replication slots 3. WAL archiving is disables Not sure i get that correctly, i can still have a

Re: Remove one last occurrence of "replication slave" in comments

2019-08-21 Thread Peter Eisentraut
On 2019-06-19 19:04, Dagfinn Ilmari Mannsåker wrote: > There were some more master/slave references in the plpgsql foreign key > tests, which the attached chages to base/leaf instead. base/leaf doesn't sound like a good pair. I committed it with root/leaf instead. -- Peter Eisentraut

Re: Optimization of vacuum for logical replication

2019-08-21 Thread Bernd Helmle
Am Mittwoch, den 21.08.2019, 12:20 +0300 schrieb Konstantin Knizhnik: > I wonder if we can check that > > 1. wal_revel is "logical" > 2. There are no physical replication slots > 3. WAL archiving is disables Not sure i get that correctly, i can still have a physical standby without replication

Optimization of vacuum for logical replication

2019-08-21 Thread Konstantin Knizhnik
Hi, hackers. Right now if replication level is rgeater or equal than "replica", vacuum  of relation copies all its data to WAL:     /*      * We need to log the copied data in WAL iff WAL archiving/streaming is      * enabled AND it's a WAL-logged rel.      */     use_wal = XLogIsNeeded() &&

Re: Global temporary tables

2019-08-21 Thread Konstantin Knizhnik
On 20.08.2019 20:01, Pavel Stehule wrote: Another solution is wait on ZHeap storage and replica can to have own UNDO log. I thought about implementation of special table access method for temporary tables. +1 Unfortunately implementing special table access method for temporary

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Dilip Kumar
On Tue, Aug 20, 2019 at 7:57 PM Robert Haas wrote: > > On Mon, Aug 19, 2019 at 2:04 AM Dilip Kumar wrote: > > Currently, In UnpackedUndoRecord we store all members directly which > > are set by the caller. We store pointers to some header which are > > allocated internally by the undo layer and

Re: "ago" times on buildfarm status page

2019-08-21 Thread Magnus Hagander
On Wed, Aug 21, 2019 at 9:40 AM Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > I find the time displays like > > 01:03 ago > > on the buildfarm status page unhelpful. > > First, I can never tell whether this is hours-minutes or minutes-seconds > -- there is probably a less

"ago" times on buildfarm status page

2019-08-21 Thread Peter Eisentraut
I find the time displays like 01:03 ago on the buildfarm status page unhelpful. First, I can never tell whether this is hours-minutes or minutes-seconds -- there is probably a less ambiguous format available. But more importantly, the page doesn't say when it was generated, so a relative

Re: Fix typos and inconsistencies for HEAD (take 11)

2019-08-21 Thread Michael Paquier
On Tue, Aug 20, 2019 at 04:47:41PM -0400, Alvaro Herrera wrote: > Yeah, Alexander proposed change is correct. I just pushed it. Thanks, Alvaro. -- Michael signature.asc Description: PGP signature

Re: ICU for global collation

2019-08-21 Thread Peter Eisentraut
On 2019-08-21 08:56, Andrey Borodin wrote: > postgres=# create database a template template0 collation_provider icu > lc_collate 'en_US.utf8'; > CREATE DATABASE > postgres=# \c a > 2019-08-21 11:43:40.379 +05 [41509] FATAL: collations with different collate > and ctype values are not supported

Re: some SCRAM read_any_attr() confusion

2019-08-21 Thread Peter Eisentraut
On 2019-08-17 14:57, Michael Paquier wrote: > On Sat, Aug 17, 2019 at 10:11:27AM +0200, Peter Eisentraut wrote: >> I was a bit confused by some of the comments around the SCRAM function >> read_any_attr(), used to skip over extensions. >> >> The comment "Returns NULL if there is attribute.",

Re: Add "password_protocol" connection parameter to libpq

2019-08-21 Thread Michael Paquier
On Tue, Aug 20, 2019 at 07:09:25PM -0700, Jeff Davis wrote: > OK, new patch attached. Seems like everyone is in agreement that we > need a channel_binding param. + +A setting of require means that the connection must +employ channel binding; and that the client will not

Re: ICU for global collation

2019-08-21 Thread Andrey Borodin
Hi! > 20 авг. 2019 г., в 19:21, Peter Eisentraut > написал(а): > > Here is an initial patch to add the option to use ICU as the global > collation provider, a long-requested feature. > > To activate, use something like > >initdb --collation-provider=icu --locale=... > > A trick here is