Re: [PATCH] Speedup truncates of relation forks

2019-09-16 Thread Michael Paquier
On Tue, Sep 17, 2019 at 01:44:12AM +, Jamison, Kirk wrote: > On Friday, September 13, 2019 10:06 PM (GMT+9), Fujii Masao wrote: >> On Fri, Sep 13, 2019 at 9:51 PM Alvaro Herrera >> wrote: As committed, the smgrdounlinkfork case is actually dead code; it's never called from anywhere.

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Michael Paquier
On Mon, Sep 16, 2019 at 03:26:10PM +0900, Tattsu Yama wrote: > I should have explained the API changes more. I added cmdtype as a given > parameter for all functions (See below). > Therefore, I suppose that my patch is similar to the following fix as you > mentioned on -hackers. Yes, that's an

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-09-16 Thread Michael Paquier
On Tue, Sep 17, 2019 at 09:23:45AM +0530, Amit Kapila wrote: > We always return a single tuple/record from this function, so do we > really need to return SETOF record or just returning record is > sufficient? Right (with the doc update). > If you want to use the same size array, then you might

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-09-16 Thread Amit Kapila
On Tue, Sep 17, 2019 at 6:14 AM Michael Paquier wrote: > > On Mon, Sep 16, 2019 at 11:11:06AM -0300, Alvaro Herrera wrote: > > On 2019-Sep-16, Michael Paquier wrote: > > Thanks, fixed. > > Amit, what do you think? Does the patch match with what you have in > mind? > * CREATE FUNCTION

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Tatsuro Yamada
Hi Alvaro! Is this fix strictly necessary for pg12, or is this something that we can leave for pg13? Not only me but many DBA needs this progress report feature on PG12, therefore I'm trying to fix the problem. If you send other patch to fix the problem, and it is more elegant than mine, I

Re: Fwd: Extending range type operators to cope with elements

2019-09-16 Thread David Fetter
On Sun, Sep 15, 2019 at 04:30:52PM +0200, Esteban Zimanyi wrote: > > So yes, I've had a need for those operators in the past. What I don't > know is whether adding these functions will be worth the catalog clutter. > > The operators are tested and running within MobilityDB. It concerns lines >

Re: refactoring - share str2*int64 functions

2019-09-16 Thread Michael Paquier
On Tue, Sep 17, 2019 at 11:29:13AM +0900, Michael Paquier wrote: > The code paths of the patch calling pg_strtouint64_check to parse > completion tags (spi.c and pg_stat_statements.c) should complain in > those cases as the format of the tags for SELECT and COPY is fixed. > > In order to unify

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Tatsuro Yamada
Hi Alvaro! Is this fix strictly necessary for pg12, or is this something that we can leave for pg13? Not only me but many DBA needs this progress report feature on PG12, therefore I'm trying to fix the problem. If you send other patch to fix the problem, and it is more elegant than mine, I

Re: refactoring - share str2*int64 functions

2019-09-16 Thread Michael Paquier
On Mon, Sep 16, 2019 at 10:08:19AM -0700, Andres Freund wrote: > On 2019-09-14 15:02:36 +0900, Michael Paquier wrote: >> - Switching OID to use pg_strtoint32 causes a failure with initdb. > > Needs to be debugged too. Although I suspect this might just be that you > need to use unsigned variant.

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Alvaro Herrera
On 2019-Sep-17, Tatsuro Yamada wrote: > On 2019/09/16 23:12, Alvaro Herrera wrote: > > Is this fix strictly necessary for pg12, or is this something that we > > can leave for pg13? > > Not only me but many DBA needs this progress report feature on PG12, > therefore I'm trying to fix the

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Tatsuro Yamada
Hi Alvaro, On 2019/09/16 23:12, Alvaro Herrera wrote: On 2019-Sep-16, Tattsu Yama wrote: I should have explained the API changes more. I added cmdtype as a given parameter for all functions (See below). Therefore, I suppose that my patch is similar to the following fix as you mentioned on

RE: [PATCH] Speedup truncates of relation forks

2019-09-16 Thread Jamison, Kirk
On Friday, September 13, 2019 10:06 PM (GMT+9), Fujii Masao wrote: > On Fri, Sep 13, 2019 at 9:51 PM Alvaro Herrera > wrote: > > > > On 2019-Sep-13, Fujii Masao wrote: > > > > > On Mon, Sep 9, 2019 at 3:52 PM Jamison, Kirk > wrote: > > > > > > > Please add a preliminary patch that removes the

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-09-16 Thread Michael Paquier
On Mon, Sep 16, 2019 at 11:11:06AM -0300, Alvaro Herrera wrote: > On 2019-Sep-16, Michael Paquier wrote: >> On Mon, Sep 16, 2019 at 11:46:16AM +0530, Amit Kapila wrote: >> Okay, using two separate columns leads to the attached. Any thoughts? >> This also includes a fix for cases with

Re: SQL/JSON: JSON_TABLE

2019-09-16 Thread Nikita Glukhov
On 23.07.2019 16:58, Pavel Stehule wrote: I got warning ar crs libpgcommon.a base64.o config_info.o controldata_utils.o d2s.o exec.o f2s.o file_perm.o ip.o keywords.o kwlookup.o link-canary.o md5.o pg_lzcompress.o pgfnames.o psprintf.o relpath.o rmtree.o saslprep.o scram-common.o string.o

Nondeterministic collations vs. text_pattern_ops

2019-09-16 Thread Tom Lane
Whilst poking at the leakproofness-of-texteq issue, I realized that there's an independent problem caused by the nondeterminism patch. To wit, that the text_pattern_ops btree opclass uses texteq as its equality operator, even though that operator is no longer guaranteed to be bitwise equality.

Re: Define jsonpath functions as stable

2019-09-16 Thread Chapman Flack
On 09/16/19 17:10, Tom Lane wrote: > I was initially troubled > by the fact that XML Schema regexps are implicitly anchored, ie must > match the whole string; that's a huge difference from POSIX. However, > 19075-6 says that jsonpath like_regex works the same as the LIKE_REGEX > predicate in

Re: Define jsonpath functions as stable

2019-09-16 Thread Jonathan S. Katz
On 9/16/19 5:10 PM, Tom Lane wrote: > "Jonathan S. Katz" writes: >> On 9/16/19 11:20 AM, Tom Lane wrote: >>> I think we could possibly get away with not having any special marker >>> on regexes, but just explaining in the documentation that "features >>> so-and-so are not implemented". Writing

Re: POC: Cleaning up orphaned files using undo logs

2019-09-16 Thread Thomas Munro
On Tue, Sep 17, 2019 at 3:09 AM Kuntal Ghosh wrote: > On Mon, Sep 16, 2019 at 11:23 AM Thomas Munro wrote: > > Agreed. I added a line to break out of that loop if !block->in_use. > > > I think we should skip the block if !block->in_use. Because, the undo > buffer can be registered in a

Re: [PATCH][PROPOSAL] Add enum releation option type

2019-09-16 Thread Alvaro Herrera
On 2019-Sep-13, Nikolay Shaplov wrote: > I've played with it around, and did some googling, but without much success. > If we are moving this way (an this way seems to be good one), I need you > help, > because this thing is beyond my C knowledge, I will not manage it myself. So I kinda did it

Re: [proposal] de-TOAST'ing using a iterator

2019-09-16 Thread Alvaro Herrera
On 2019-Sep-10, Binguo Bao wrote: > +/* > + * Support for de-TOASTing toasted value iteratively. "need" is a pointer > + * between the beginning and end of iterator's ToastBuffer. The marco > + * de-TOAST all bytes before "need" into iterator's ToastBuffer. > + */ > +#define

Re: Define jsonpath functions as stable

2019-09-16 Thread Tom Lane
"Jonathan S. Katz" writes: > On 9/16/19 11:20 AM, Tom Lane wrote: >> I think we could possibly get away with not having any special marker >> on regexes, but just explaining in the documentation that "features >> so-and-so are not implemented". Writing that text would require closer >> analysis

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-16 Thread Tomas Vondra
On Mon, Sep 16, 2019 at 10:29:18PM +0300, Konstantin Knizhnik wrote: On 16.09.2019 19:54, Alexey Kondratov wrote: On 30.08.2019 18:59, Konstantin Knizhnik wrote: I think that instead of defining savepoints it is simpler and more efficient to use BeginInternalSubTransaction +

Re: block-level incremental backup

2019-09-16 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Sep 16, 2019 at 1:10 PM Stephen Frost wrote: > > I disagree with this on a couple of levels. The first is pretty simple- > > we don't have all of the information. The user may have some reason to > > believe that timestamp-based

Re: Bug in GiST paring heap comparator

2019-09-16 Thread Alexander Korotkov
On Mon, Sep 16, 2019 at 3:47 PM Nikita Glukhov wrote: > On 13.09.2019 20:17, Alexander Korotkov wrote: > > On Fri, Sep 13, 2019 at 5:23 PM Nikita Glukhov > wrote: > > I have moved handling of NULL ordering keys from opclasses to the common > SP-GiST code, but really I don't like how it is

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-16 Thread Konstantin Knizhnik
On 16.09.2019 19:54, Alexey Kondratov wrote: On 30.08.2019 18:59, Konstantin Knizhnik wrote: I think that instead of defining savepoints it is simpler and more efficient to use BeginInternalSubTransaction + ReleaseCurrentSubTransaction/RollbackAndReleaseCurrentSubTransaction as it is

Re: block-level incremental backup

2019-09-16 Thread Robert Haas
On Mon, Sep 16, 2019 at 1:10 PM Stephen Frost wrote: > I disagree with this on a couple of levels. The first is pretty simple- > we don't have all of the information. The user may have some reason to > believe that timestamp-based is a bad idea, for example, and therefore > having an option to

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-09-16 Thread Peter Geoghegan
On Mon, Sep 16, 2019 at 8:48 AM Anastasia Lubennikova wrote: > Attached is v14 based on v12 (v13 changes are not merged). > > In this version, I fixed the bug you mentioned and also fixed nbtinsert, > so that it doesn't save newposting in xlog record anymore. Cool. > I tested patch with

Re: range test for hash index?

2019-09-16 Thread Paul A Jungwirth
On Mon, Sep 16, 2019 at 5:28 AM Amit Kapila wrote: > I don't see this function on the master branch. Is this function name > correct? Are you looking at some different branch? Sorry about that! You're right, I was on my multirange branch. But I see the same thing on latest master (but calling

Re: block-level incremental backup

2019-09-16 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Sep 16, 2019 at 9:30 AM Robert Haas wrote: > > > Isn't some operations where at the end we directly call heap_sync > > > without writing WAL will have a similar problem as well? > > > > Maybe. Can you give an example? > >

Re: Define jsonpath functions as stable

2019-09-16 Thread Jonathan S. Katz
On 9/16/19 11:20 AM, Tom Lane wrote: > "Jonathan S. Katz" writes: >> It sounds like the easiest path to completion without potentially adding >> futures headaches pushing back the release too far would be that, e.g. >> these examples: > >> $.** ? (@ like_regex "O(w|v)" pg flag "i") >>

Re: refactoring - share str2*int64 functions

2019-09-16 Thread Fabien COELHO
Bonjour Michaël, Otherwise, this batch of changes looks ok to me. Thanks. About v15: applies cleanly, compiles, "make check" ok. While re-reading the patch, there are bit of repetitions on pg_strtou?int* comments. I'm wondering whether it would make sense to write a global comments

Re: Commit fest 2019-09

2019-09-16 Thread Alvaro Herrera
The third week for this commitfest starts. The numbers now: statusstring │ week1 │ week2 │ week3 ┼───┼───┼─── Needs review │ 165 │ 138 │ 116 Waiting on Author │30 │44 │51 Ready for Committer│11 │ 5 │

Re: block-level incremental backup

2019-09-16 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Sep 16, 2019 at 10:38 AM Stephen Frost wrote: > > In a number of cases, trying to make sure that on a failover or copy of > > the backup the next 'incremental' is really an 'incremental' is > > dangerous. A better strategy to

Re: refactoring - share str2*int64 functions

2019-09-16 Thread Andres Freund
Hi, On 2019-09-14 15:02:36 +0900, Michael Paquier wrote: > On Fri, Sep 13, 2019 at 06:38:31PM -0700, Andres Freund wrote: > > On 2019-09-10 12:05:25 +0900, Michael Paquier wrote: > >> On Mon, Sep 09, 2019 at 05:27:04AM -0700, Andres Freund wrote: > >> Attached is an updated patch? How does it

Re: POC: Cleaning up orphaned files using undo logs

2019-09-16 Thread Robert Haas
On Mon, Sep 16, 2019 at 11:09 AM Kuntal Ghosh wrote: > Okay. In that case, we need to rethink the cases for multi-inserts and > non-inlace updates both of which currently inserts multiple undo > record corresponding to a single WAL record. For multi-inserts, it can > be solved easily by moving

Re: block-level incremental backup

2019-09-16 Thread Robert Haas
On Mon, Sep 16, 2019 at 10:38 AM Stephen Frost wrote: > In a number of cases, trying to make sure that on a failover or copy of > the backup the next 'incremental' is really an 'incremental' is > dangerous. A better strategy to address this, and the other issues > realized on this thread

Re: block-level incremental backup

2019-09-16 Thread Robert Haas
On Mon, Sep 16, 2019 at 9:30 AM Robert Haas wrote: > > Isn't some operations where at the end we directly call heap_sync > > without writing WAL will have a similar problem as well? > > Maybe. Can you give an example? Looking through the code, I found two cases where we do this. One is a bulk

Re: [HACKERS] [WIP] Effective storage of duplicates in B-tree index.

2019-09-16 Thread Anastasia Lubennikova
13.09.2019 4:04, Peter Geoghegan wrote: On Wed, Sep 11, 2019 at 2:04 PM Peter Geoghegan wrote: I think that the new WAL record has to be created once per posting list that is generated, not once per page that is deduplicated -- that's the only way that I can see that avoids a huge increase in

Re: Define jsonpath functions as stable

2019-09-16 Thread Tom Lane
"Jonathan S. Katz" writes: > It sounds like the easiest path to completion without potentially adding > futures headaches pushing back the release too far would be that, e.g. > these examples: > $.** ? (@ like_regex "O(w|v)" pg flag "i") > $.** ? (@ like_regex "O(w|v)" pg) > If it's

Re: Default JIT setting in V12

2019-09-16 Thread Jeff Janes
On Wed, Sep 4, 2019 at 11:24 AM Andres Freund wrote: > Hi, > > On 2019-09-04 07:51:16 -0700, Andres Freund wrote: > > Or better, something slightly more complete, like the attached (which > affects both code-gen time optimizations (which are more like peephole > ones), and both function/global

Re: POC: Cleaning up orphaned files using undo logs

2019-09-16 Thread Kuntal Ghosh
Hello Thomas, On Mon, Sep 16, 2019 at 11:23 AM Thomas Munro wrote: > > > 1. In UndoLogAllocateInRecovery, when we find the current log number > > from the list of registered blocks, we don't check whether the > > block->in_use flag is true or not. In XLogResetInsertion, we just > > reset in_use

Re: Define jsonpath functions as stable

2019-09-16 Thread Jonathan S. Katz
Hi, On 7/29/19 8:33 PM, Chapman Flack wrote: > On 07/29/19 18:27, Alexander Korotkov wrote: > >> What do you think about renaming existing operator from like_regex to >> pg_like_regex? Or introducing special flag indicating that PostgreSQL >> regex engine is used ('p' for instance)? > >

Re: block-level incremental backup

2019-09-16 Thread Stephen Frost
Greetings, * Robert Haas (robertmh...@gmail.com) wrote: > On Mon, Sep 16, 2019 at 4:31 AM Amit Kapila wrote: > > Can we think of using creation time for file? Basically, if the file > > creation time is later than backup-labels "START TIME:", then include > > that file entirely. I think one

Re: Extending range type operators to cope with elements

2019-09-16 Thread Esteban Zimanyi
> > > So yes, I've had a need for those operators in the past. What I don't know > is whether adding these functions will be worth the catalog clutter. > The operators are tested and running within MobilityDB. It concerns lines 231-657 for the C code in file

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-09-16 Thread Alvaro Herrera
On 2019-Sep-16, Michael Paquier wrote: > On Mon, Sep 16, 2019 at 11:46:16AM +0530, Amit Kapila wrote: > > I don't see much use of separating information for infomask and infomask2. > > Okay, using two separate columns leads to the attached. Any thoughts? > This also includes a fix for cases

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Alvaro Herrera
On 2019-Sep-16, Tattsu Yama wrote: > I should have explained the API changes more. I added cmdtype as a given > parameter for all functions (See below). > Therefore, I suppose that my patch is similar to the following fix as you > mentioned on -hackers. Is this fix strictly necessary for pg12,

Re: ecpglib major version changed

2019-09-16 Thread Tom Lane
Peter Eisentraut writes: > The ecpglib major version (SO_MAJOR_VERSION) was changed in > bd7c95f0c1a38becffceb3ea7234d57167f6d4bf (Add DECLARE STATEMENT support > to ECPG.), but I don't see anything in that patch that would warrant > that. I think we should undo that change. Ouch. Yeah, that's

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-09-16 Thread Michael Paquier
On Mon, Sep 16, 2019 at 11:46:16AM +0530, Amit Kapila wrote: > I don't see much use of separating information for infomask and infomask2. Okay, using two separate columns leads to the attached. Any thoughts? This also includes a fix for cases with IS_LOCKED_ONLY and UPGRADED. -- Michael diff

Re: block-level incremental backup

2019-09-16 Thread Robert Haas
On Mon, Sep 16, 2019 at 4:31 AM Amit Kapila wrote: > This seems to be a blocking problem for the LSN based design. Well, only the simplest version of it, I think. > Can we think of using creation time for file? Basically, if the file > creation time is later than backup-labels "START TIME:",

Re: [PATCH] Improve performance of NOTIFY over many databases (v2)

2019-09-16 Thread Tom Lane
Martijn van Oosterhout writes: > On Mon, 16 Sep 2019 at 00:14, Tom Lane wrote: >> ... I also think we >> can simplify the handling of other-database listeners by including >> them in the set signaled by SignalBackends, but only if they're >> several pages behind. So that leads me to the

Re: (Re)building index using itself or another index of the same table

2019-09-16 Thread Arseny Sher
Tomas Vondra writes: > On Thu, Sep 12, 2019 at 11:08:28AM -0400, Tom Lane wrote: >>I have exactly no faith that this fixes things enough to make such >>cases supportable. And I have no interest in opening that can of >>worms anyway. I'd rather put in some code to reject database >>accesses

Re: Bug in GiST paring heap comparator

2019-09-16 Thread Nikita Glukhov
On 13.09.2019 20:17, Alexander Korotkov wrote: On Fri, Sep 13, 2019 at 5:23 PM Nikita Glukhov wrote: I have moved handling of NULL ordering keys from opclasses to the common SP-GiST code, but really I don't like how it is implemented now. Maybe it's worth to move handling of NULL order-by

Re: range test for hash index?

2019-09-16 Thread Amit Kapila
On Mon, Sep 16, 2019 at 7:23 AM Paul A Jungwirth wrote: > > On Sat, Sep 14, 2019 at 5:13 AM Amit Kapila wrote: > > In general, the hash_range is covered by some of the existing test, > > but I don't which test. See the code coverage report here: > >

ecpglib major version changed

2019-09-16 Thread Peter Eisentraut
The ecpglib major version (SO_MAJOR_VERSION) was changed in bd7c95f0c1a38becffceb3ea7234d57167f6d4bf (Add DECLARE STATEMENT support to ECPG.), but I don't see anything in that patch that would warrant that. I think we should undo that change. -- Peter Eisentraut

Re: A problem presentaion about ECPG, DECLARE STATEMENT

2019-09-16 Thread Peter Eisentraut
On 2019-09-11 18:04, Michael Meskes wrote: Is it acceptable for PG12? >>> In general absolutely. >> >> It seems far too late to be considering any major rewrite for v12; >> even assuming that there was consensus on the rewrite being an >> improvement, which I bet there won't be. > > Oops, I

Re: [PATCH] Improve performance of NOTIFY over many databases (v2)

2019-09-16 Thread Martijn van Oosterhout
Hoi Tom, On Mon, 16 Sep 2019 at 00:14, Tom Lane wrote: > > I spent some more time thinking about this, and I'm still not too > satisfied with this patch's approach. It seems to me the key insights > we're trying to make use of are: > > 1. We don't really need to keep the global tail pointer

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2019-09-16 Thread Tomas Vondra
On Sun, Sep 15, 2019 at 09:33:33PM -0400, James Coleman wrote: On Sat, Jul 20, 2019 at 11:25 AM Tomas Vondra wrote: >> ... >> >> I think this may be a thinko, as this plan demonstrates - but I'm not >> sure about it. I wonder if this might be penalizing some other types of >> plans

Re: refactoring - share str2*int64 functions

2019-09-16 Thread Michael Paquier
On Sat, Sep 14, 2019 at 10:24:10AM +0200, Fabien COELHO wrote: > It should rather call pg_strtoint16? And possibly switch the "short int" > declaration to int16? Sure, but you get into other problems if using the 16-bit version for some other fields, which is why it seems to me that we should add

Re: block-level incremental backup

2019-09-16 Thread Amit Kapila
On Mon, Sep 16, 2019 at 7:22 AM Robert Haas wrote: > > On Thu, Sep 12, 2019 at 9:13 AM Jeevan Chalke > wrote: > > I had a look over this issue and observed that when the new database is > > created, the catalog files are copied as-is into the new directory > > corresponding to a newly created

Re: [HACKERS] CLUSTER command progress monitor

2019-09-16 Thread Tattsu Yama
Hi Michael, >> > Attached file is WIP patch.In my patch, I added "command id" to all >> APIs of >> > progress reporting to isolate commands. Therefore, it doesn't allow to >> > cascade updating system views. And my patch is on WIP so it needs >> clean-up >> > and test. >> > I share it anyway. :)