Re: [HACKERS] Custom compression methods (mac+lz4.h)

On Sun, Mar 21, 2021 at 9:10 AM Dilip Kumar wrote: > > On Sun, Mar 21, 2021 at 7:03 AM Tom Lane wrote: > > > > BTW, I tried doing "make installcheck" after having adjusted > > default_toast_compression to be "lz4". The compression test > > itself fails because it's expecting the other setting;

Re: pspg pager is finished

ne 21. 3. 2021 v 0:48 odesílatel Thomas Munro napsal: > On Sat, Mar 20, 2021 at 4:35 PM Pavel Stehule > wrote: > > I finished work on pspg. > > tmunro@x1:~/projects/postgresql$ pspg --stream --graph > pspg: unrecognized option '--graph' > Try pspg --help > > Hmm, seems to be not finished :-) >

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

Hi Zhihing, I think @Andreas ment to mark it as a todo to cleanup later. On Sun, 21 Mar 2021 at 4:49 AM Zhihong Yu wrote: > Hi, > In v11-0004-fk_arrays_elems_edits.patch : > > + riinfo->fk_reftypes[i] == > FKCONSTR_REF_EACH_ELEMENT ? OID_ARRAY_ELEMCONTAINED_OP : >

Re: [HACKERS] Custom compression methods (mac+lz4.h)

On Sun, Mar 21, 2021 at 7:03 AM Tom Lane wrote: > > BTW, I tried doing "make installcheck" after having adjusted > default_toast_compression to be "lz4". The compression test > itself fails because it's expecting the other setting; that > ought to be made more robust. Yeah, we need to set the

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

Hi, In v11-0004-fk_arrays_elems_edits.patch : + riinfo->fk_reftypes[i] == FKCONSTR_REF_EACH_ELEMENT ? OID_ARRAY_ELEMCONTAINED_OP : riinfo->pf_eq_oprs[i], // XXX Is XXX placeholder for some comment you will fill in later ? Cheers On Sat, Mar 20, 2021 at 11:42 AM Mark

Re: [HACKERS] Custom compression methods (mac+lz4.h)

BTW, I tried doing "make installcheck" after having adjusted default_toast_compression to be "lz4". The compression test itself fails because it's expecting the other setting; that ought to be made more robust. Also, I see some diffs in the indirect_toast test, which seems perhaps worthy of

Re: popcount

On Sat, Mar 20, 2021 at 09:01:25PM +0100, Peter Eisentraut wrote: > On 18.03.21 13:51, John Naylor wrote: > > Hi David, > > > > Just a nitpick: > > > > +SET bytea_output TO hex; > > > > Since we don't see the string in the output, I don't immediately see the > > reason to change the output

Re: [HACKERS] Custom compression methods (mac+lz4.h)

Justin Pryzby writes: > On Sat, Mar 20, 2021 at 05:37:07PM -0400, Tom Lane wrote: >> Digging around, it looks like the "-I/opt/local/include" bit came >> from LZ4_CFLAGS, which we then stuck into CFLAGS, but it needed >> to be put in CPPFLAGS in order to make this test work. > If it's the same

Re: pspg pager is finished

On Sat, Mar 20, 2021 at 4:35 PM Pavel Stehule wrote: > I finished work on pspg. tmunro@x1:~/projects/postgresql$ pspg --stream --graph pspg: unrecognized option '--graph' Try pspg --help Hmm, seems to be not finished :-)

Re: proposal - psql - use pager for \watch command

On Sun, Mar 21, 2021 at 11:44 AM Thomas Munro wrote: > [review] Oh, just BTW, to save confusion for others who might try this: It seems there is something wrong with pspg --stream on macOS, at least when using MacPorts. I assumed it might be just pspg 3.1.5 being too old (that's what MacPorts

Re: [HACKERS] Custom compression methods

On 3/20/21 4:21 PM, Justin Pryzby wrote: > On Sat, Mar 20, 2021 at 04:13:47PM +0100, Tomas Vondra wrote: >> +++ b/src/backend/access/brin/brin_tuple.c >> @@ -213,10 +213,20 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, >> BrinMemTuple *tuple, >>

Re: Using COPY FREEZE in pgbench

Hi Fabien, > Hello Tatsuo-san, > >>> I have looked in the code of PQprotocolVersion. The only case in which >>> it returns 0 is there's no connection. Yes, you are right. Once the >>> connection has been successfuly established, there's no chance it >>> fails. So I agree with you. >> >> Attached

Re: [HACKERS] Custom compression methods

Rebased on HEAD. 0005 forgot to update compression_1.out. Included changes to ./configure.ac and some other patches, but not Tomas's, since it'll make CFBOT get mad as soon as that's pushed. -- Justin >From bf1336d284792b29e30b42af2ec820bb0c256916 Mon Sep 17 00:00:00 2001 From: Justin Pryzby

Re: [HACKERS] Custom compression methods (mac+lz4.h)

On Sat, Mar 20, 2021 at 05:37:07PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > On Fri, Mar 19, 2021 at 02:07:31PM -0400, Robert Haas wrote: > >> On Fri, Mar 19, 2021 at 1:44 PM Justin Pryzby wrote: > >>> configure: WARNING: lz4.h: accepted by the compiler, rejected by the > >>>

Re: proposal - psql - use pager for \watch command

On Thu, Mar 4, 2021 at 11:28 PM Pavel Stehule wrote: > čt 4. 3. 2021 v 7:37 odesílatel Pavel Stehule > napsal: >> Here is a little bit updated patch - detection of end of any child process >> cannot be used on WIN32. Yeah, it's OK for me if this feature only works on Unix until the right

Re: [HACKERS] Custom compression methods (mac+lz4.h)

Justin Pryzby writes: > On Fri, Mar 19, 2021 at 02:07:31PM -0400, Robert Haas wrote: >> On Fri, Mar 19, 2021 at 1:44 PM Justin Pryzby wrote: >>> configure: WARNING: lz4.h: accepted by the compiler, rejected by the >>> preprocessor! >>> configure: WARNING: lz4.h: proceeding with the compiler's

Re: Replication slot stats misgivings

Hi, On 2021-03-20 10:28:06 +0530, Amit Kapila wrote: > On Sat, Mar 20, 2021 at 9:25 AM Amit Kapila wrote: > > This idea is worth exploring to address the complaints but what do we > > do when we detect that the stats are from the different slot? It has > > mixed of stats from the old and new

Re: Replication slot stats misgivings

Hi, On 2021-03-20 09:25:40 +0530, Amit Kapila wrote: > On Sat, Mar 20, 2021 at 12:22 AM Andres Freund wrote: > > > > And then more generally about the feature: > > - If a slot was used to stream out a large amount of changes (say an > > initial data load), but then replication is interrupted

Re: [HACKERS] Custom compression methods (mac+lz4.h)

On Fri, Mar 19, 2021 at 02:07:31PM -0400, Robert Haas wrote: > On Fri, Mar 19, 2021 at 1:44 PM Justin Pryzby wrote: > > Working with one of Andrey's patches on another thread, he reported offlist > > getting this message, resolved by this patch. Do you see this warning > > during > >

Re: [HACKERS] Custom compression methods (./configure)

Justin Pryzby writes: > On Fri, Mar 19, 2021 at 05:35:58PM -0300, Alvaro Herrera wrote: >> I find this behavior confusing; I'd rather have configure error out if >> it can't find the package support I requested, than continuing with a >> set of configure options different from what I gave. >

Re: [HACKERS] Custom compression methods (./configure)

On 2021-Mar-20, Justin Pryzby wrote: > On Fri, Mar 19, 2021 at 05:35:58PM -0300, Alvaro Herrera wrote: > > Hmm, if I use configure --with-lz4, I get this: > > > > checking whether to build with LZ4 support... yes > > checking for liblz4... no > > configure: error: Package

Re: [HACKERS] Custom compression methods (./configure)

On Fri, Mar 19, 2021 at 05:35:58PM -0300, Alvaro Herrera wrote: > Hmm, if I use configure --with-lz4, I get this: > > checking whether to build with LZ4 support... yes > checking for liblz4... no > configure: error: Package requirements (liblz4) were not met: > > No

Re: [HACKERS] Custom compression methods

Andrew Dunstan writes: > On 3/20/21 3:03 PM, Tom Lane wrote: >> I fixed up some issues in 0008/0009 (mostly cosmetic, except that >> you forgot a server version check in dumpToastCompression) and >> pushed that, so we can see if it makes crake happy. > It's still produced a significant amount

Re: [HACKERS] Custom compression methods

I wrote: > I fixed up some issues in 0008/0009 (mostly cosmetic, except that > you forgot a server version check in dumpToastCompression) and > pushed that, so we can see if it makes crake happy. crake was still unhappy with that:

Re: [HACKERS] Custom compression methods

On 3/20/21 3:03 PM, Tom Lane wrote: > I wrote: >> Yeah, _doSetFixedOutputState is the wrong place: that runs on the >> pg_restore side of the fence, and would not have access to the >> necessary info in a separated dump/restore run. >> It might be necessary to explicitly pass the state through

Re: popcount

On 18.03.21 13:51, John Naylor wrote: Hi David, Just a nitpick: +SET bytea_output TO hex; Since we don't see the string in the output, I don't immediately see the reason to change the output format here? Aside from that, this patch works as expected, and is ready for committer. I have

Re: [HACKERS] Custom compression methods

I wrote: > Yeah, _doSetFixedOutputState is the wrong place: that runs on the > pg_restore side of the fence, and would not have access to the > necessary info in a separated dump/restore run. > It might be necessary to explicitly pass the state through in a TOC item, > as we do for things like

Re: [HACKERS] Custom compression methods

On Sat, Mar 20, 2021 at 01:36:15PM -0400, Tom Lane wrote: > Justin Pryzby writes: > > On Sat, Mar 20, 2021 at 04:37:53PM +0530, Dilip Kumar wrote: > >> - And, 0008 and 0009, I think my > >> 0001-Fixup-dump-toast-compression-method.patch[1] is doing this in a > >> much simpler way, please have a

Re: [PATCH] proposal for regexp_count, regexp_instr, regexp_substr and regexp_replace

Hi, This is a new version of the patch that now implements all the XQUERY regexp functions as described in the standard, minus the differences of PostgerSQL regular expression explain in [1]. The standard SQL describe functions like_regex(), occurrences_regex(), position_regex(),

Re: pspg pager is finished

On Sat, Mar 20, 2021 at 9:05 AM Pavel Stehule wrote: > Hi > > I finished work on pspg. > > https://github.com/okbob/pspg > > Now it has special features like rows or block selection by mouse, and > export related data to file or to clipboard in csv or tsv or insert > formats. Some basic features

Re: [HACKERS] Custom compression methods

Justin Pryzby writes: > On Sat, Mar 20, 2021 at 04:37:53PM +0530, Dilip Kumar wrote: >> - And, 0008 and 0009, I think my >> 0001-Fixup-dump-toast-compression-method.patch[1] is doing this in a >> much simpler way, please have a look and let me know if you think that >> has any problems and we

Re: pg_upgrade failing for 200+ million Large Objects

On 3/20/21 11:23 AM, Tom Lane wrote: Jan Wieck writes: All that aside, the entire approach doesn't scale. Yeah, agreed. When we gave large objects individual ownership and ACL info, it was argued that pg_dump could afford to treat each one as a separate TOC entry because "you wouldn't have

Re: pg_upgrade failing for 200+ million Large Objects

Bruce Momjian writes: > On Sat, Mar 20, 2021 at 11:23:19AM -0400, Tom Lane wrote: >> Of course, that just reduces the memory consumption on the client >> side; it does nothing for the locks. Can we get away with releasing the >> lock immediately after doing an ALTER OWNER or GRANT/REVOKE on a

Re: pg_upgrade failing for 200+ million Large Objects

On Sat, Mar 20, 2021 at 11:23:19AM -0400, Tom Lane wrote: > I wonder if pg_dump could improve matters cheaply by aggregating the > large objects by owner and ACL contents. That is, do > > select distinct lomowner, lomacl from pg_largeobject_metadata; > > and make just *one* BLOB TOC entry for

Re: pspg pager is finished

so 20. 3. 2021 v 17:01 odesílatel Nikolay Samokhvalov napsal: > On Fri, Mar 19, 2021 at 20:35 Pavel Stehule > wrote: > >> Hi >> >> I finished work on pspg. >> >> https://github.com/okbob/pspg >> > > Thank you, Pavel. I use it always when possible, and highly recommend it > to others. > >>

Re: pspg pager is finished

On Fri, Mar 19, 2021 at 20:35 Pavel Stehule wrote: > Hi > > I finished work on pspg. > > https://github.com/okbob/pspg > Thank you, Pavel. I use it always when possible, and highly recommend it to others. > > Nik

Re: [HACKERS] Custom compression methods

On Sat, Mar 20, 2021 at 04:37:53PM +0530, Dilip Kumar wrote: > - 0006 is fine but not sure what is the advantage over what we have today? The advantage is that it's dozens of lines shorter, and automatically includes a HINT. SET default_toast_compression = 'I do not exist compression';

Re: Logical Replication vs. 2PC

On Sat, Mar 20, 2021 at 4:02 PM Dilip Kumar wrote: > > On Sat, Mar 20, 2021 at 7:50 AM Amit Kapila wrote: > > > > On Fri, Mar 19, 2021 at 9:22 PM Markus Wanner > > wrote: > > > So, I think you are using xid of publisher and origin_id of > > subscription to achieve uniqueness because both will

Re: pg_upgrade failing for 200+ million Large Objects

Jan Wieck writes: > On 3/8/21 11:58 AM, Tom Lane wrote: >> So it seems like the path of least resistance is >> (a) make pg_upgrade use --single-transaction when calling pg_restore >> (b) document (better) how to get around too-many-locks failures. > That would first require to fix how pg_upgrade

Re: [HACKERS] Custom compression methods

On Sat, Mar 20, 2021 at 04:13:47PM +0100, Tomas Vondra wrote: > +++ b/src/backend/access/brin/brin_tuple.c > @@ -213,10 +213,20 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, > BrinMemTuple *tuple, > (atttype->typstorage == TYPSTORAGE_EXTENDED || >

Re: pg_upgrade failing for 200+ million Large Objects

On 3/20/21 12:39 AM, Jan Wieck wrote: > On 3/8/21 11:58 AM, Tom Lane wrote: >> The answer up to now has been "raise max_locks_per_transaction enough >> so you don't see the failure".  Having now consumed a little more >> caffeine, I remember that that works in pg_upgrade scenarios too, >> since

Re: Logical Replication vs. 2PC

On Sat, Mar 20, 2021 at 2:57 PM Markus Wanner wrote: > > On 20.03.21 03:17, Amit Kapila wrote: > > Are you saying that users might use the same GID which we have > > constructed internally (say by combining origin and xid: originid_xid) > > and then there will be conflict while replaying such

Re: [HACKERS] Custom compression methods

On 3/20/21 11:45 AM, Tomas Vondra wrote: > > > On 3/20/21 11:18 AM, Dilip Kumar wrote: >> On Sat, Mar 20, 2021 at 3:05 PM Tomas Vondra >> wrote: >>> >>> Hi, >>> >>> I think this bit in brin_tuple.c is wrong: >>> >>> ... >>> Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, >>>

Re: Using COPY FREEZE in pgbench

Hello Tatsuo-san, I have looked in the code of PQprotocolVersion. The only case in which it returns 0 is there's no connection. Yes, you are right. Once the connection has been successfuly established, there's no chance it fails. So I agree with you. Attached v3 patch addresses this. The

Re: simplifying foreign key/RI checks

On Mon, Mar 8, 2021 at 11:41 PM Amit Langote wrote: > On Thu, Mar 4, 2021 at 5:15 AM Tom Lane wrote: > > I guess I'm disturbed by the idea > > that we'd totally replace the implementation technology for only one > > variant of foreign key checks. That means that there'll be a lot > > of minor

Re: Log message for GSS connection is missing once connection authorization is successful.

On Sat, Mar 20, 2021 at 05:37:47PM +0900, Michael Paquier wrote: > It seems to me that this would make the tests faster, that the test > would not need to wait for the logging collector and that the code > could just use slurp_file($node->logfile) to get the data it wants to > check for a given

Re: [HACKERS] Custom compression methods

On Sat, Mar 20, 2021 at 1:14 PM Justin Pryzby wrote: > > See attached. I have looked into your patches - 0001 to 0005 and 0007 look fine to me so maybe you can merge them all and create a fixup patch. Thanks for fixing this, these were some silly mistakes I made in my patch. - 0006 is fine but

Re: pspg pager is finished

so 20. 3. 2021 v 7:51 odesílatel Vik Fearing napsal: > On 3/20/21 4:34 AM, Pavel Stehule wrote: > > Hi > > > > I finished work on pspg. > > > > https://github.com/okbob/pspg > > > > Now it has special features like rows or block selection by mouse, and > > export related data to file or to

Re: [HACKERS] Custom compression methods

On 3/20/21 11:18 AM, Dilip Kumar wrote: > On Sat, Mar 20, 2021 at 3:05 PM Tomas Vondra > wrote: >> >> Hi, >> >> I think this bit in brin_tuple.c is wrong: >> >> ... >> Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, >> keyno); >>

Re: [PATCH] Hooks at XactCommand level

On Fri, Mar 19, 2021 at 11:02:29PM +0100, Gilles Darold wrote: > Le 12/03/2021 à 06:55, Julien Rouhaud a écrit : > > > > I don't think we need to pass any information at least for the rollback > at statement level extension. All information needed are accessible and > actually at

Re: Logical Replication vs. 2PC

On Sat, Mar 20, 2021 at 7:50 AM Amit Kapila wrote: > > On Fri, Mar 19, 2021 at 9:22 PM Markus Wanner > wrote: > So, I think you are using xid of publisher and origin_id of > subscription to achieve uniqueness because both will be accessible in > prepare and commit prepared. Right? If so, I

Re: [HACKERS] Custom compression methods

On Sat, Mar 20, 2021 at 3:05 PM Tomas Vondra wrote: > > Hi, > > I think this bit in brin_tuple.c is wrong: > > ... > Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, > keyno); > Datum cvalue = toast_compress_datum(value, >

Re: Columns correlation and adaptive query optimization

On 19.03.2021 20:32, Zhihong Yu wrote: Hi, In AddMultiColumnStatisticsForQual(), +   /* Loop until we considered all vars */ +   while (vars != NULL) ... +       /* Contruct list of unique vars */ +       foreach (cell, vars) What if some cell / node, gets into the else block: +            

Re: [HACKERS] Custom compression methods

Hi, I think this bit in brin_tuple.c is wrong: ... Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, keyno); Datum cvalue = toast_compress_datum(value, att->attcompression); The

Re: Logical Replication vs. 2PC

On 20.03.21 03:17, Amit Kapila wrote: Are you saying that users might use the same GID which we have constructed internally (say by combining origin and xid: originid_xid) and then there will be conflict while replaying such transactions? No, I was pondering about a user doing (in short

Re: [HACKERS] Custom compression methods

On 2021-03-19 15:44:34 -0400, Robert Haas wrote: > I committed the core patch (0003) with a bit more editing. Let's see > what the buildfarm thinks. Congrats Dilip, Robert, All. The slow toast compression has been a significant issue for a long time.

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

It would be really convenient if user-visible serialisations of the query id had something that identifies the computation method. maybe prefix 'N' for internal, 'S' for pg_stat_statements etc. This would immediately show in logs at what point the id calculator was changed On Fri, Mar 19, 2021

Re: shared memory stats: high level design decisions: consistency, dropping

> But now we could instead schedule stats to be removed at commit time. That's not trivial of course, as we'd need to handle cases where the commit fails after the commit record, but before processing the dropped stats. We likely can not remove them at commit time, but only after the oldest open

Re: [HACKERS] Custom compression methods

On Fri, Mar 19, 2021 at 04:38:03PM -0400, Robert Haas wrote: > On Fri, Mar 19, 2021 at 4:20 PM Tom Lane wrote: > > Nope ... crake's displeased with your assumption that it's OK to > > clutter dumps with COMPRESSION clauses. As am I: that is going to > > be utterly fatal for cross-version

Re: Log message for GSS connection is missing once connection authorization is successful.

On Wed, Dec 02, 2020 at 02:44:31PM -0500, Stephen Frost wrote: > And committed. This has been committed as of dc11f31a, that changed the configuration of the node in the kerberos test to use logging_collector. Wouldn't it be simpler to not use the logging collector here and use a logic similar

Re: [HACKERS] Custom compression methods

On Sat, Mar 20, 2021 at 1:22 PM Dilip Kumar wrote: > > On Sat, Mar 20, 2021 at 8:11 AM Robert Haas wrote: > > > > On Fri, Mar 19, 2021 at 8:25 PM Tom Lane wrote: > > > Extrapolating from the way we've dealt with similar issues > > > in the past, I think the structure of pg_dump's output ought

Re: [HACKERS] Custom compression methods

On Sat, Mar 20, 2021 at 8:11 AM Robert Haas wrote: > > On Fri, Mar 19, 2021 at 8:25 PM Tom Lane wrote: > > Extrapolating from the way we've dealt with similar issues > > in the past, I think the structure of pg_dump's output ought to be: > > > > 1. SET default_toast_compression = 'source

Re: [HACKERS] Custom compression methods

See attached. One issue is that the pg_dump tests are no longer exercising the COMPRESSION clause. I don't know how to improve on that, since lz4 may not be available. ..unless we changed attcompression='\0' to mean (for varlena) "the default compression". Rather than "resolving" to the

replication cleanup code incorrect way to use of HTAB HASH_REMOVE ?

Hi, I found some dubious looking HTAB cleanup code for replication streams (see file:worker.c, function:stream_cleanup_files). viz. -- static void stream_cleanup_files(Oid subid, TransactionId xid) { charpath[MAXPGPATH]; StreamXidHash *ent; /* Remove the xid entry

Re: pspg pager is finished

On 3/20/21 4:34 AM, Pavel Stehule wrote: > Hi > > I finished work on pspg. > > https://github.com/okbob/pspg > > Now it has special features like rows or block selection by mouse, and > export related data to file or to clipboard in csv or tsv or insert > formats. Some basic features like

Re: a verbose option for autovacuum

On Sat, Mar 20, 2021 at 01:06:51PM +0900, Masahiko Sawada wrote: > It's not bad but it seems redundant a bit to me. We pass the idx in > spite of passing also Irel[idx] and &(vacrelstats->indstats[idx]). I > think your first idea that is done in v4 patch (saving index names at > the beginning of

Re: pspg pager is finished

so 20. 3. 2021 v 4:45 odesílatel Julien Rouhaud napsal: > On Sat, Mar 20, 2021 at 04:34:30AM +0100, Pavel Stehule wrote: > > Hi > > > > I finished work on pspg. > > > > https://github.com/okbob/pspg > > > > Now it has special features like rows or block selection by mouse, and > > export related

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

On Fri, Mar 19, 2021 at 12:53:18PM -0400, Bruce Momjian wrote: > > Well, given we don't really want to support multiple query id types > being generated or displayed, the "error out" above should fix it. > > Let's do this --- tell extensions to error out if the query id is > already set, either