Re: Fallback table AM for relkinds without storage

2021-02-21 Thread Michael Paquier
On Sun, Feb 21, 2021 at 09:43:59AM -0600, Justin Pryzby wrote: > If you apply this patch, will you want to actually revert those > earlier changes? That's not in the plan. > Also (related), this still crashes if methods are omitted from the > initializer, > like: > > // .slot_callbacks = no_sto

Table AM and DDLs

2021-02-21 Thread Mats Kindahl
Hi all, I am quite new to PostgreSQL so forgive me if my understanding of the code below is wrong and please clarify what I have misunderstood. I started to experiment with the table access method interface to see if it can be used for some ideas I have. For the experiment, I am using a simple i

Re: pg_collation_actual_version() ERROR: cache lookup failed for collation 123

2021-02-21 Thread Michael Paquier
On Mon, Feb 22, 2021 at 06:34:22PM +1300, Thomas Munro wrote: > On Thu, Feb 18, 2021 at 8:15 PM Michael Paquier wrote: >> Could you just add a test with pg_collation_current_version(0)? > > Done. > >> + pg_strncasecmp("POSIX.", collcollate, 6) != 0) >> >> I didn't know that "POSIX." was po

RE: Parallel INSERT (INTO ... SELECT ...)

2021-02-21 Thread houzj.f...@fujitsu.com
> Posting a new version of the patches, with the following updates: > - Moved the update of glob->relationOIDs (i.e. addition of partition OIDs that > plan depends on, resulting from parallel-safety checks) from within > max_parallel_hazard() to set_plan_references(). > - Added an extra test for pa

Re: Improve new hash partition bound check error messages

2021-02-21 Thread Peter Eisentraut
On 15.02.21 17:45, Peter Eisentraut wrote: On 2021-02-03 15:52, Peter Eisentraut wrote: On 2021-02-02 13:26, Heikki Linnakangas wrote: How about this? CREATE TABLE fail_part PARTITION OF hash_parted FOR VALUES WITH (MODULUS 25, REMAINDER 3); ERROR:  every hash partition modulus must be a facto

[PATCH] Feature improvement for TRUNCATE tab completion.

2021-02-21 Thread miyake_kouta
Hi. I created a patch which improves psql's TRUNCATE tab completion. Current tab completion can complement only a table name to be truncated. This patch enables psql to complement other keywords related to TRUNCATE. Regards. Kota Miyakediff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/ta

Re: Catalog version access

2021-02-21 Thread Vik Fearing
On 2/22/21 3:24 AM, Andres Freund wrote: > Imagine trying to run regular tests of HEAD, where the tests require a > large database to be loaded. Re-loading the data for every [few] commits > is prohibitively time consuming, and even just running pg_upgrade is > painful. So you'd like to re-use a "t

Re: New IndexAM API controlling index vacuum strategies

2021-02-21 Thread Masahiko Sawada
On Wed, Feb 10, 2021 at 4:12 PM Peter Geoghegan wrote: > > On Tue, Feb 9, 2021 at 6:14 PM Masahiko Sawada wrote: > > Thanks. I think that's very good if we resolve this recycling stuff > > first then try the new approach to skip index vacuum in more cases. > > That way, even if the vacuum strateg

Re: a misbehavior of partition row movement (?)

2021-02-21 Thread Amit Langote
On Fri, Feb 19, 2021 at 5:04 PM Masahiko Sawada wrote: > On Mon, Feb 15, 2021 at 10:37 PM Amit Langote wrote: > > Regarding the patch, I would have liked if it only prevented the > > update when the foreign key that would be violated by the component > > delete references the update query's main

Re: [PATCH] postgres_fdw connection caching - cause remote sessions linger till the local session exit

2021-02-21 Thread Bharath Rupireddy
On Thu, Feb 4, 2021 at 9:36 AM Bharath Rupireddy wrote: > > On Wed, Feb 3, 2021 at 4:22 PM Fujii Masao > wrote: > > Maybe my explanation in the previous email was unclear. What I think is; If > > the server-level option is explicitly specified, its setting is used > > whatever GUC is. On the o

Re: libpq compression

2021-02-21 Thread Craig Ringer
On Thu, 11 Feb 2021, 21:09 Daniil Zakhlystov, wrote:: > > 3. Chunked compression allows to compress only well compressible messages > and save the CPU cycles by not compressing the others > 4. Chunked compression introduces some traffic overhead compared to the > permanent (1.2810G vs 1.2761G TX

Re: pg_collation_actual_version() ERROR: cache lookup failed for collation 123

2021-02-21 Thread Thomas Munro
On Thu, Feb 18, 2021 at 8:15 PM Michael Paquier wrote: > Could you just add a test with pg_collation_current_version(0)? Done. > + pg_strncasecmp("POSIX.", collcollate, 6) != 0) > > I didn't know that "POSIX." was possible. Yeah, that isn't valid on my (quite current) GNU or FreeBSD syste

Re: Use pgstat_progress_update_multi_param instead of single param update

2021-02-21 Thread Michael Paquier
On Sun, Feb 21, 2021 at 04:43:23PM +0530, Bharath Rupireddy wrote: > While we are at it, I wanted to use a single line statement instead of > if else, just like we do it in do_analyze_rel as below. > > pgstat_progress_update_param(PROGRESS_ANALYZE_PHASE, > inh

Re: [PATCH]: Allow errors in parameter values to be reported during the BIND phase itself..

2021-02-21 Thread Justin Pryzby
On Mon, Jan 04, 2021 at 11:09:39AM -0600, Justin Pryzby wrote: > For example: > > $ python3.5 -c "import pg; db=pg.DB(); q = db.query(\"SET > log_parameter_max_length_on_error=-1;\"); db.prepare('p', 'SELECT > \$1::smallint'); db.query_prepared('p',6);" > 2021-01-03 02:21:04.547 CST [20157]

Re: Improvements and additions to COPY progress reporting

2021-02-21 Thread Bharath Rupireddy
On Mon, Feb 22, 2021 at 12:40 AM Matthias van de Meent wrote: > > On Sat, 20 Feb 2021 at 07:09, Bharath Rupireddy > wrote: > > > > For COPY TO the name "source_type" column and for COPY FROM the name > > "destination_type" makes sense. To have a combined column name for > > both, how about namin

RE: libpq debug log

2021-02-21 Thread tsunakawa.ta...@fujitsu.com
From: Iwata, Aya/岩田 彩 > I update patch to v18. It has been fixed in response to Tsunakawa san's > review. (52) + of tracing. If (flags contains PQTRACE_SUPPRESS_TIMESTAMPS), () can be removed? (53) + int inLogging; /* next byte of logging */ I u

Re: repeated decoding of prepared transactions

2021-02-21 Thread Andres Freund
Hi, On 2021-02-19 15:53:32 +0100, Markus Wanner wrote: > However, more generally speaking, I suspect you are overthinking this. All > of the complexity arises because of the assumption that an output plugin > receiving and confirming a PREPARE may not be able to persist that first > phase of trans

Re: repeated decoding of prepared transactions

2021-02-21 Thread Andres Freund
Hi, On 2021-02-22 08:22:35 +0530, Amit Kapila wrote: > On Mon, Feb 22, 2021 at 3:56 AM Andres Freund wrote: > > > > On 2021-02-21 11:32:29 +0530, Amit Kapila wrote: > > > Here, I am assuming you are asking to disable 2PC both via > > > apply-worker and tablesync worker till the initial sync (aka

Re: Finding cause of test fails on the cfbot site

2021-02-21 Thread Andres Freund
Hi, On 2021-02-17 15:18:02 -0500, Andrew Dunstan wrote: > yeah. The cfbot runs check-world which makes it difficult for it to know > which log files to show when there's an error. That's a major part of > the reason the buildfarm runs a much finer grained set of steps. I really think we need a be

RE: Determine parallel-safety of partition relations for Inserts

2021-02-21 Thread houzj.f...@fujitsu.com
Hi, Attaching v7 patches with the changes: * rebase the code on the greg's latest parallel insert patch. Please consider it for further review. Best regards, houzj v7_0004-reloption-parallel_dml-test-and-doc.patch Description: v7_0004-reloption-parallel_dml-test-and-doc.patch v7_000

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-21 Thread Amit Langote
On Fri, Feb 19, 2021 at 7:38 PM Amit Kapila wrote: > On Thu, Feb 18, 2021 at 11:05 AM Amit Langote wrote: > > > > > > It also occurred to me that we can avoid pointless adding of > > > > partitions if the final plan won't use parallelism. For that, the > > > > patch adds checking glob->parallelM

Re: Parallel INSERT (INTO ... SELECT ...)

2021-02-21 Thread Greg Nancarrow
On Fri, Feb 19, 2021 at 9:38 PM Amit Kapila wrote: > > On Thu, Feb 18, 2021 at 11:05 AM Amit Langote wrote: > > > > > > It also occurred to me that we can avoid pointless adding of > > > > partitions if the final plan won't use parallelism. For that, the > > > > patch adds checking glob->paralle

Re: should INSERT SELECT use a BulkInsertState?

2021-02-21 Thread Justin Pryzby
On Mon, Feb 22, 2021 at 02:25:22AM +, houzj.f...@fujitsu.com wrote: > > > Yes, you can see that I've copied the checks from copy. > > > Like copy, some checks are done once, in ExecInitModifyTable, outside > > > of the ExecModifyTable "loop". > > > > > > This squishes some commits together. > >

Re: repeated decoding of prepared transactions

2021-02-21 Thread Amit Kapila
On Mon, Feb 22, 2021 at 3:56 AM Andres Freund wrote: > > On 2021-02-21 11:32:29 +0530, Amit Kapila wrote: > > Here, I am assuming you are asking to disable 2PC both via > > apply-worker and tablesync worker till the initial sync (aka all > > tables are in SUBREL_STATE_READY state) phase is complet

Re: Finding cause of test fails on the cfbot site

2021-02-21 Thread Thomas Munro
On Sat, Feb 20, 2021 at 3:54 AM Andrew Dunstan wrote: > here's a very small and simple (and possibly naive) POC patch that > demonstrates this and seems to do the right thing. As a small variation that might be more parallelism-friendly, would it be better to touch a file with a known name in an

Re: Catalog version access

2021-02-21 Thread Andres Freund
Hi, On 2021-02-21 20:53:52 -0500, Tom Lane wrote: > Andres Freund writes: > >> If we're going to bother with providing a way > >> to get this info, we should make it possible to ask the running server. > > > In Vik's case there is no running server to ask, IIUC. > > Hm. If you're about to init

RE: Parallel INSERT (INTO ... SELECT ...)

2021-02-21 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > It is quite possible what you are saying is correct but I feel that is > not this patch's fault. So, won't it better to discuss this in a > separate thread? > > Good use case but again, I think this can be done as a separate patch. Agreed. I think even the current patch offer

Re: Catalog version access

2021-02-21 Thread Tom Lane
Andres Freund writes: > On 2021-02-21 19:54:01 -0500, Tom Lane wrote: >> FWIW, I think asking pg_config about this is a guaranteed way of having >> version-skew-like bugs. > Could you elaborate a bit? How do you know that the pg_config you found has anything to do with the server you're connecte

Re: Catalog version access

2021-02-21 Thread Andres Freund
Hi, On 2021-02-21 19:54:01 -0500, Tom Lane wrote: > Vik Fearing writes: > > On 2/22/21 12:48 AM, Andres Freund wrote: > >> Seems roughly reasonable. Although I wonder if we rather should make it > >> something more generic than just catversion? E.g. a wal page magic bump > >> will also require a

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-02-21 Thread Justin Pryzby
On Tue, Feb 16, 2021 at 11:15:51PM +1300, David Rowley wrote: > To summarise here, the planner performance gets a fair bit worse with > the patched code. With master, summing the average planning time over > each of the queries resulted in a total planning time of 765.7 ms. > After patching, that w

Re: Catalog version access

2021-02-21 Thread Tom Lane
Vik Fearing writes: > On 2/22/21 12:48 AM, Andres Freund wrote: >> Seems roughly reasonable. Although I wonder if we rather should make it >> something more generic than just catversion? E.g. a wal page magic bump >> will also require a dump/restore or pg_upgrade, but won't be detected by >> just

Re: Catalog version access

2021-02-21 Thread Euler Taveira
On Sun, Feb 21, 2021, at 8:15 PM, Vik Fearing wrote: > and a second patch that adds a read-only guc to get at it on the SQL > level using SHOW catalog_version; or similar. I need that because I > also do a dump of pg_settings and I would like for it to appear there. The catalog version number is a

Re: Is a connection max lifetime useful in a connection pool?

2021-02-21 Thread Julien Rouhaud
On Mon, Feb 22, 2021 at 7:52 AM Andres Freund wrote: > > On 2021-02-21 19:05:03 +0100, Daniele Varrazzo wrote: > > One of the HikariCP configuration parameters is "maxLifetime", whose > > description is: "This property controls the maximum lifetime of a > > connection in the pool. [...] **We stron

Re: Catalog version access

2021-02-21 Thread Vik Fearing
On 2/22/21 12:48 AM, Andres Freund wrote: > Hi, > > On 2021-02-22 00:15:20 +0100, Vik Fearing wrote: >> I do some very regular testing on HEAD and my scripts need to know if >> the catalog version has changed to determine if it needs to pg_restore >> or if a basebackup is okay. In order to get it

Re: Is a connection max lifetime useful in a connection pool?

2021-02-21 Thread Andres Freund
Hi, On 2021-02-21 19:05:03 +0100, Daniele Varrazzo wrote: > One of the HikariCP configuration parameters is "maxLifetime", whose > description is: "This property controls the maximum lifetime of a > connection in the pool. [...] **We strongly recommend setting this > value, and it should be severa

Re: Catalog version access

2021-02-21 Thread Andres Freund
Hi, On 2021-02-22 00:15:20 +0100, Vik Fearing wrote: > I do some very regular testing on HEAD and my scripts need to know if > the catalog version has changed to determine if it needs to pg_restore > or if a basebackup is okay. In order to get it, I have to do this: > > > # Get the catalog vers

Re: Finding cause of test fails on the cfbot site

2021-02-21 Thread Thomas Munro
On Sat, Feb 20, 2021 at 11:33 AM Thomas Munro wrote: > On Sat, Feb 20, 2021 at 10:31 AM Peter Smith wrote: > > Here is another related question about the cfbot error reporting - > > > > The main cfbot "status page" [1] still shows a couple of fails for the > > 32/2914 (for freebsd & linux). But l

Re: Extensions not dumped when --schema is used

2021-02-21 Thread David Fetter
On Thu, Feb 18, 2021 at 11:13:06AM +0100, Guillaume Lelarge wrote: > Le mar. 26 janv. 2021 à 13:42, Guillaume Lelarge a > écrit : > > > Le mar. 26 janv. 2021 à 13:41, Guillaume Lelarge > > a écrit : > > > >> Le mar. 26 janv. 2021 à 05:10, Julien Rouhaud a > >> écrit : > >> > >>> On Mon, Jan 25,

Catalog version access

2021-02-21 Thread Vik Fearing
Hello. I do some very regular testing on HEAD and my scripts need to know if the catalog version has changed to determine if it needs to pg_restore or if a basebackup is okay. In order to get it, I have to do this: # Get the catalog version (there is no better way to do this) tmp=$(mktemp --dir

Re: repeated decoding of prepared transactions

2021-02-21 Thread Andres Freund
Hi, On 2021-02-21 11:32:29 +0530, Amit Kapila wrote: > Here, I am assuming you are asking to disable 2PC both via > apply-worker and tablesync worker till the initial sync (aka all > tables are in SUBREL_STATE_READY state) phase is complete. If we do > that and what if commit prepared happened aft

Re: [HACKERS] GSoC 2017: Foreign Key Arrays

2021-02-21 Thread Justin Pryzby
On Tue, Feb 16, 2021 at 12:07:10PM +0200, Mark Rofail wrote: ... There's some errors in the latest patch: http://cfbot.cputube.org/mark-rofail.html gram.y:16933:20: error: invalid operands to binary expression ('List' (aka 'struct List') and 'void *') Assert(**reftypes != NULL); Did yo

Re: [PATCH] Present all committed transaction to the output plugin

2021-02-21 Thread Tomas Vondra
On 2/21/21 11:05 AM, Markus Wanner wrote: On 21.02.21 03:04, Andres Freund wrote: Cost-wise, yes - a 2pc prepare/commit is expensive enough that comparatively the replay cost is unlikely to be relevant. Good.  I attached an updated patch eliminating only the filtering for empty two-phase t

Re: Improvements and additions to COPY progress reporting

2021-02-21 Thread Matthias van de Meent
On Sat, 20 Feb 2021 at 07:09, Bharath Rupireddy wrote: > > For COPY TO the name "source_type" column and for COPY FROM the name > "destination_type" makes sense. To have a combined column name for > both, how about naming that column as "io_type"? Thank you, that's way better! PFA what I believe

Re: Is a connection max lifetime useful in a connection pool?

2021-02-21 Thread Daniele Varrazzo
On Sun, 21 Feb 2021 at 19:12, Pavel Stehule wrote: > I have very strong experience - it is very useful. On Sun, 21 Feb 2021 at 19:26, Stephen Frost wrote: > Short answer- yes. Sounds good. Thank you very much for your insight! -- Daniele

Re: How to customize postgres for sharing read-only tables in multiple data-dirs between servers

2021-02-21 Thread Justin Pryzby
On Tue, Feb 16, 2021 at 01:49:02PM +, Guttman, Maoz wrote: > Hi, > > Problem statement: > I have to develop a solution in which a single source populates a table. Once > the table is populated, it is considered as read-only and then we run many > read-only queries on it. > Such read-only tab

Re: Bizarre behavior of \w in a regular expression bracket construct

2021-02-21 Thread Joel Jacobson
On Sun, Feb 21, 2021, at 18:39, Tom Lane wrote: > Alvaro Herrera writes: > > This one I didn't understand: > >> ^([\W])$ | pg | > > I think Joel just forgot to mark that as ERROR. Yes, my mistake, sorry about that, (I manually edited the query result and replaced empty-field with "ERROR")

Re: Is a connection max lifetime useful in a connection pool?

2021-02-21 Thread Stephen Frost
Greetings, * Daniele Varrazzo (daniele.varra...@gmail.com) wrote: > I am designing and implementing a connection pool for psycopg3 [1][2]. > Some of the inspiration is coming from HikariCP [3], a Java connection > pool. > > One of the HikariCP configuration parameters is "maxLifetime", whose > de

Re: Is a connection max lifetime useful in a connection pool?

2021-02-21 Thread Pavel Stehule
Hi ne 21. 2. 2021 v 19:05 odesílatel Daniele Varrazzo < daniele.varra...@gmail.com> napsal: > Hello, > > I am designing and implementing a connection pool for psycopg3 [1][2]. > Some of the inspiration is coming from HikariCP [3], a Java connection > pool. > > One of the HikariCP configuration pa

Is a connection max lifetime useful in a connection pool?

2021-02-21 Thread Daniele Varrazzo
Hello, I am designing and implementing a connection pool for psycopg3 [1][2]. Some of the inspiration is coming from HikariCP [3], a Java connection pool. One of the HikariCP configuration parameters is "maxLifetime", whose description is: "This property controls the maximum lifetime of a connect

Re: Bizarre behavior of \w in a regular expression bracket construct

2021-02-21 Thread Tom Lane
Alvaro Herrera writes: > It looks like the interpretation of these other engines is that [\d-a] > is the set of \d, the literal character "-", and the literal character > "a". In other words, the - preceded by \d or \w (or any other character > class, I guess?) loses its special meaning of identi

Re: Bizarre behavior of \w in a regular expression bracket construct

2021-02-21 Thread Alvaro Herrera
On 2021-Feb-21, Joel Jacobson wrote: >regex| engine |deduced_ranges > ++--- > ^([a-z])$ | pg | [a-z] > ^([a-z])$ | pl | [a-z] > ^([a-z])$ | v8 | [a-z] > ^([\d-a])$ | pg | > ^([\d-a])$ | pl | [-0-9a] > ^([\d-a])$

Re: Fallback table AM for relkinds without storage

2021-02-21 Thread Justin Pryzby
On Mon, Feb 15, 2021 at 04:21:38PM +0900, Michael Paquier wrote: > On Tue, Feb 09, 2021 at 04:27:34PM +0900, Michael Paquier wrote: > > Putting sanity checks within all the table_* functions of tableam.h > > would not be a good idea, as nothing prevents the call of what's > > stored in rel->rd_tabl

Re: GROUP BY DISTINCT

2021-02-21 Thread Vik Fearing
On 2/21/21 3:06 PM, e...@xs4all.nl wrote: >> On 2021.02.21. 13:52 Vik Fearing wrote: >> >> Attached is a patch to implement this for PostgreSQL. >> [] > > The changed line that gets stuffed into sql_features is missing a terminal > value (to fill the 'comments' column). > This line: > '+T434

Re: GROUP BY DISTINCT

2021-02-21 Thread er
> On 2021.02.21. 13:52 Vik Fearing wrote: > > Attached is a patch to implement this for PostgreSQL. > [] The changed line that gets stuffed into sql_features is missing a terminal value (to fill the 'comments' column). This line: '+T434 GROUP BY DISTINCT YES' (A tab at

Re: Extend more usecase for planning time partition pruning and init partition pruning.

2021-02-21 Thread Andy Fan
On Fri, Feb 19, 2021 at 6:03 PM Andy Fan wrote: > > > On Mon, Feb 8, 2021 at 3:43 PM Andy Fan wrote: > >> >> >> On Mon, Jan 25, 2021 at 10:21 AM Andy Fan >> wrote: >> >>> >>> >>> On Sun, Jan 24, 2021 at 6:34 PM Andy Fan >>> wrote: >>> Hi: I recently found a use case like this.

GROUP BY DISTINCT

2021-02-21 Thread Vik Fearing
When combining multiple grouping items, such as rollups and cubes, the resulting flattened grouping sets can contain duplicate items. The standard provides for this by allowing GROUP BY DISTINCT to deduplicate them prior to doing the actual work. For example: GROUP BY ROLLUP (a,b), ROLLUP (a,c)

Re: [HACKERS] Custom compression methods

2021-02-21 Thread Dilip Kumar
On Sat, Feb 20, 2021 at 11:04 AM Dilip Kumar wrote: > > On Sat, Feb 20, 2021 at 2:51 AM Robert Haas wrote: > > > > On Fri, Feb 19, 2021 at 11:12 AM Dilip Kumar wrote: > > I think that these performance tests aren't really exercising the > > expanded-record stuff, just the ExecEvalRow changes. We

Re: Use pgstat_progress_update_multi_param instead of single param update

2021-02-21 Thread Bharath Rupireddy
On Sun, Feb 21, 2021 at 4:18 PM Michael Paquier wrote: > > On Sun, Feb 21, 2021 at 11:30:21AM +0530, Bharath Rupireddy wrote: > > Attached is a patch that replaces some subsequent multiple > > update_param calls with a single update_multi_param. > > Looks mostly fine to me. > > -if (OidIsValid

Re: Use pgstat_progress_update_multi_param instead of single param update

2021-02-21 Thread Michael Paquier
On Sun, Feb 21, 2021 at 11:30:21AM +0530, Bharath Rupireddy wrote: > Attached is a patch that replaces some subsequent multiple > update_param calls with a single update_multi_param. Looks mostly fine to me. -if (OidIsValid(indexOid)) -pgstat_progress_update_param(PROGRESS_CLUSTER_COM

Re: Hybrid Hash/Nested Loop joins and caching results from subplans

2021-02-21 Thread Andy Fan
On Tue, Feb 16, 2021 at 6:16 PM David Rowley wrote: > On Wed, 3 Feb 2021 at 19:51, David Rowley wrote: > > I've attached a spreadsheet with the results of each of the tests. > > > > The attached file v13_costing_hacks.patch.txt is the quick and dirty > > patch I put together to run test 5. > > I

Re: [PATCH] Present all committed transaction to the output plugin

2021-02-21 Thread Markus Wanner
On 21.02.21 03:04, Andres Freund wrote: Cost-wise, yes - a 2pc prepare/commit is expensive enough that comparatively the replay cost is unlikely to be relevant. Good. I attached an updated patch eliminating only the filtering for empty two-phase transactions. Behaviourally I'm still not co