Re: Rethinking opclass member checks and dependency strength

2020-07-28 Thread Hamid Akhtar
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:not tested Looks good to me. CORRECTION: In my previous review I had mistakenl

Re: [PATCH] Tab completion for VACUUM of partitioned tables

2020-07-28 Thread Michael Paquier
On Wed, Jul 29, 2020 at 01:27:07PM +0900, Masahiko Sawada wrote: > Good catch. The patch looks good to me. While this patch is logically correct. I think that we should try to not increase more the number of queries used to scan pg_class depending on a list of relkinds. For example, did you noti

Re: proposal: unescape_text function

2020-07-28 Thread Pavel Stehule
Hi > > Hi Pavel, > > Since the idea originated from unescaping unicode string literals i.e. >select unescape('Odpov\u011Bdn\u00E1 osoba'); > > Shouldn't the built-in function support the above syntax as well? > good idea. The prefixes u (4 digits) and U (8 digits) are supported Regards

Re: Improving connection scalability: GetSnapshotData()

2020-07-28 Thread Thomas Munro
On Fri, Jul 24, 2020 at 1:11 PM Andres Freund wrote: > On 2020-07-15 21:33:06 -0400, Alvaro Herrera wrote: > > On 2020-Jul-15, Andres Freund wrote: > > > It could make sense to split the conversion of > > > VariableCacheData->latestCompletedXid to FullTransactionId out from 0001 > > > into is own

Re: Doc patch: mention indexes in pg_inherits docs

2020-07-28 Thread Michael Paquier
On Tue, Jul 28, 2020 at 12:21:29PM +0100, Dagfinn Ilmari Mannsåker wrote: > When partitioned index support was added in veresion 11, the pg_inherits > docs missed the memo and still only say it describes table inheritance. > The attached patch adds mentions of indexes too, and notes that they can >

Re: printing oid with %d

2020-07-28 Thread Michael Paquier
On Tue, Jul 28, 2020 at 10:35:54AM -0400, Tom Lane wrote: > Yeah, given that it should be an unreachable case, there's likely > no need to back-patch. Thanks. Fixed on HEAD then. -- Michael signature.asc Description: PGP signature

Re: Move syncscan.c?

2020-07-28 Thread Thomas Munro
On Wed, Jun 24, 2020 at 6:28 AM Andres Freund wrote: > On 2020-06-23 13:30:39 +1200, Thomas Munro wrote: > > I suppose it's remotely possible that someone might invent > > physical-order index scans, and once you have those you might sync > > scans of those too, and then even table would be too sp

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-28 Thread Ashutosh Sharma
Hi Beena, Thanks for the review. 1. We would be marking buffer dirty and writing wal even if we have > not done any changes( ex if we pass invalid/dead tids). Maybe we can > handle this better? > Yeah, we can skip this when nothing has changed. Will take care of it in the next version of patch.

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-28 Thread Ashutosh Sharma
> > I think we should let VACUUM do that. > Sometimes VACUUM will not get to these pages, because they are marked All > Frozen. > An possibly some tuples will get stale on this page again > Hmm, okay, will have a look into this. Thanks. > > > > Are there any caveats with concurrent VACUUM? (I do

Re: [PATCH] Tab completion for VACUUM of partitioned tables

2020-07-28 Thread Masahiko Sawada
On Wed, 29 Jul 2020 at 02:04, Justin Pryzby wrote: > > Could maybe backpatch to v10. > > diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c > index 272f799c24..06ef658afb 100644 > --- a/src/bin/psql/tab-complete.c > +++ b/src/bin/psql/tab-complete.c > @@ -578,14 +578,23 @@ stat

Re: Rethinking opclass member checks and dependency strength

2020-07-28 Thread Hamid Akhtar
On Tue, Jul 28, 2020 at 8:43 PM Tom Lane wrote: > Hamid Akhtar writes: > > I've gone through the patch and applied on the master branch, other than > a few hunks, and whether as suggested upthread, the default case for > "switch (op->number)" should throw an error or not, I found that bloom > re

Re: Threading in BGWorkers (!)

2020-07-28 Thread James Sewell
> We need some kind of concrete plan for what is a > usable amount of functionality and what has to be done to get it. > This is exactly the type of discussion I'm after. I'll start. A usable amount of functionality would be the ability to start threads from: - a background worker These cas

Re: Threading in BGWorkers (!)

2020-07-28 Thread Tom Lane
Thomas Munro writes: > On Thu, Jul 2, 2020 at 6:39 PM James Sewell wrote: >> The patch replaces sigprocmask with pthread_sigmask. They have identical >> APIs ("[pthread_sigmask] shall be equivalent to sigprocmask(), without the >> restriction that the call be made in a single-threaded process"[

Re: Threading in BGWorkers (!)

2020-07-28 Thread Thomas Munro
On Thu, Jul 2, 2020 at 6:39 PM James Sewell wrote: > The patch replaces sigprocmask with pthread_sigmask. They have identical APIs > ("[pthread_sigmask] shall be equivalent to sigprocmask(), without the > restriction that the call be made in a single-threaded process"[1]) -#define PG_SETMASK(ma

Re: shared tempfile was not removed on statement_timeout

2020-07-28 Thread Justin Pryzby
On Mon, Jul 27, 2020 at 05:39:02AM -0500, Justin Pryzby wrote: > On Mon, Jul 27, 2020 at 08:00:46PM +1200, Thomas Munro wrote: > > Why can't tuplesort_end do it? > > Because then I think the parallel workers remove their own files, with tests > failing like: > > +ERROR: could not open temporary

Re: HashAgg's batching counter starts at 0, but Hash's starts at 1.

2020-07-28 Thread David Rowley
On Mon, 27 Jul 2020 at 14:54, Justin Pryzby wrote: > > On Mon, Jul 27, 2020 at 10:48:45AM +1200, David Rowley wrote: > > On Wed, 1 Jul 2020 at 18:46, Jeff Davis wrote: > > > > > > On Tue, Jun 30, 2020, 7:04 PM David Rowley wrote: > > >> > > >> Does anyone have any objections to that being change

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Peter Geoghegan
On Tue, Jul 28, 2020 at 2:46 PM Peter Geoghegan wrote: > The fact remains that this function (originally known as > IndexBuildHeapScan(), now heapam_index_build_range_scan()) did not > care about whether or not the index is unique for about 3 years > (excluding the tupleIsAlive stuff, which was al

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Peter Geoghegan
On Tue, Jul 28, 2020 at 1:26 PM Peter Geoghegan wrote: > On Tue, Jul 28, 2020 at 1:04 PM Tom Lane wrote: > > No, I don't think so. It was designed for the case of unique key X > > being inserted immediately after a deletion of the same key. The > > deleted tuple is presumably not yet vacuumed-a

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-28 Thread MBeena Emerson
Hello Ashutosh, On Fri, 24 Jul 2020 at 14:35, Ashutosh Sharma wrote: > > Hi All, > > Attached is the patch that adds heap_force_kill(regclass, tid[]) and > heap_force_freeze(regclass, tid[]) functions which Robert mentioned in the > first email in this thread. The patch basically adds an extens

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Peter Geoghegan
On Tue, Jul 28, 2020 at 1:04 PM Tom Lane wrote: > No, I don't think so. It was designed for the case of unique key X > being inserted immediately after a deletion of the same key. The > deleted tuple is presumably not yet vacuumed-away, so the new tuple > should have a different TID. In no case

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Tom Lane
Peter Geoghegan writes: > I also find it suspicious that I can no longer produce the assertion > failure once I force all non-unique system catalog indexes (such as > Justin's repro index, pg_class_tblspc_relfilenode_index) to go through > the HEAPTUPLE_INSERT_IN_PROGRESS + > !TransactionIdIsCurre

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Peter Geoghegan
On Tue, Jul 28, 2020 at 12:00 PM Peter Geoghegan wrote: > I still don't know what's going on here, but I find it suspicious that > some relevant tuples go through the HEAPTUPLE_INSERT_IN_PROGRESS + > !TransactionIdIsCurrentTransactionId() path of > heapam_index_build_range_scan(). After all, if th

Re: display offset along with block number in vacuum errors

2020-07-28 Thread Justin Pryzby
On Wed, Jul 29, 2020 at 12:35:17AM +0530, Mahendra Singh Thalor wrote: > Apart from these, I fixed comments given by Sawada and Michael in the > latest patch. Attaching v2 patch for review. Thanks. lazy_check_needs_freeze iterates over blocks and this patch changes it to update vacrelstats. I th

Re: [BUG] Error in BRIN summarization

2020-07-28 Thread Peter Geoghegan
On Mon, Jul 27, 2020 at 10:25 AM Alvaro Herrera wrote: > (I was also considering whether it needs to be a loop to reobtain root > tuples, in case a concurrent transaction can create a new item while > we're checking that item; but I don't think that can really happen for > one individual tuple.)

Re: display offset along with block number in vacuum errors

2020-07-28 Thread Mahendra Singh Thalor
Thanks Justin, Sawada and Michael for reviewing. On Mon, 27 Jul 2020 at 16:43, Justin Pryzby wrote: > > On Fri, Jul 24, 2020 at 11:18:43PM +0530, Mahendra Singh Thalor wrote: > > Hi hackers, > > We discussed in another email thread[1], that it will be helpful if we can > > display offset along wi

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Peter Geoghegan
On Tue, Jul 28, 2020 at 10:37 AM Peter Geoghegan wrote: > It's starting to look more like that. I can reproduce the bug by > running the REINDEX in a tight loop while "make installcheck" runs. It > looks as if the two tuples passed to comparetup_index_btree() are > separate tuples that each point

Re: [PATCH] Initial progress reporting for COPY command

2020-07-28 Thread Pavel Stehule
út 28. 7. 2020 v 20:25 odesílatel Josef Šimánek napsal: > Thanks for the info. I am waiting for review. Is there any summary of > requested changes needed? > Maybe it is just noise - you wrote so you will resend a patch to different thread > >> I tried to reattach the thread there. I'll prepare

Re: [PATCH] Initial progress reporting for COPY command

2020-07-28 Thread Josef Šimánek
Thanks for the info. I am waiting for review. Is there any summary of requested changes needed? Dne út 28. 7. 2020 19:00 uživatel Fujii Masao napsal: > > > On 2020/07/02 21:51, Daniel Gustafsson wrote: > >> On 2 Jul 2020, at 14:42, Josef Šimánek wrote: > >> čt 2. 7. 2020 v 14:25 odesílatel Dani

psql: add \si, \sm, \st and \sr functions to show CREATE commands for indexes, matviews, triggers and tables

2020-07-28 Thread a . pervushina
Hi, I've attached a patch that implements \si, \sm, \st and \sr functions that show the CREATE command for indexes, matviews, triggers and tables. The functions are implemented similarly to the existing sf/sv functions with some modifications. For triggers, I've decided to change input forma

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Peter Geoghegan
On Tue, Jul 28, 2020 at 10:06 AM Peter Geoghegan wrote: > I wrote the assertion that fails here with the bug that I fixed in > commit 4974d7f87e62a58e80c6524e49677cb25cc10e12 in mind specifically. > That was a bug that involved a scan that returned duplicate tuples due > to a problem in heapam_ind

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Peter Geoghegan
On Tue, Jul 28, 2020 at 8:47 AM Peter Geoghegan wrote: > This is very likely to be related to incremental sort because it's a > use-after-free issue, which is the kind of thing that could only > really happen inside tuplesort.c. This is clear because some of the > variables have the tell-tale 0x7f

[PATCH] Tab completion for VACUUM of partitioned tables

2020-07-28 Thread Justin Pryzby
Could maybe backpatch to v10. diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index 272f799c24..06ef658afb 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -578,14 +578,23 @@ static const SchemaQuery Query_for_list_of_vacuumables = { .cat

Re: [PATCH] Initial progress reporting for COPY command

2020-07-28 Thread Fujii Masao
On 2020/07/02 21:51, Daniel Gustafsson wrote: On 2 Jul 2020, at 14:42, Josef Šimánek wrote: čt 2. 7. 2020 v 14:25 odesílatel Daniel Gustafsson mailto:dan...@yesql.se>> napsal: The automated patchtester for the Commitfest gets confused when there are two versions of the same changeset attac

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Peter Geoghegan
On Tue, Jul 28, 2020 at 8:40 AM James Coleman wrote: > Is that assumption largely based on the incremental sort patch > refactoring tuplesort.c a bit? I haven't looked at it much at all, but > I'm wondering if the issue could also be related to the btree > duplicates changes in 13 given that we're

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Justin Pryzby
On Tue, Jul 28, 2020 at 11:40:14AM -0400, James Coleman wrote: > > This appears to be an issue with d2d8a229b (Incremental Sort), so I will add > > at: https://wiki.postgresql.org/wiki/PostgreSQL_13_Open_Items > > Is that assumption largely based on the incremental sort patch > refactoring tupleso

Re: Rethinking opclass member checks and dependency strength

2020-07-28 Thread Tom Lane
Hamid Akhtar writes: > I've gone through the patch and applied on the master branch, other than a > few hunks, and whether as suggested upthread, the default case for "switch > (op->number)" should throw an error or not, I found that bloom regression is > crashing. > - > test bloom

Re: 13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread James Coleman
On Tue, Jul 28, 2020 at 11:10 AM Justin Pryzby wrote: > > An internal server aborted last night while running a maintenance script. I > reproduced this easily running the crashing command in a loop, and verified > this is a live issue on REL_13_STABLE (dc6f2fb43). > REINDEX INDEX pg_class_tblspc_

13dev failed assert: comparetup_index_btree(): ItemPointer values should never be equal

2020-07-28 Thread Justin Pryzby
An internal server aborted last night while running a maintenance script. I reproduced this easily running the crashing command in a loop, and verified this is a live issue on REL_13_STABLE (dc6f2fb43). REINDEX INDEX pg_class_tblspc_relfilenode_index It looks like this crashed once before, and I

Re: Rethinking opclass member checks and dependency strength

2020-07-28 Thread Hamid Akhtar
The following review has been posted through the commitfest application: make installcheck-world: tested, failed Implements feature: tested, failed Spec compliant: not tested Documentation:not tested I've gone through the patch and applied on the master branch, other t

Re: printing oid with %d

2020-07-28 Thread Tom Lane
Michael Paquier writes: > Good catch. Yep, Oids are unsigned. We don't backpatch such things > usually, do we? Particularly, this one should not be triggerable > normally because no code paths should call JsonEncodeDateTime() with > an unsupported type Oid. Yeah, given that it should be an unr

Re: proposal: unescape_text function

2020-07-28 Thread Asif Rehman
On Thu, Jul 2, 2020 at 10:10 PM Pavel Stehule wrote: > > > čt 2. 7. 2020 v 17:27 odesílatel Daniel Gustafsson > napsal: > >> > On 23 Jun 2020, at 11:51, Pavel Stehule >> wrote: >> >> > I changed the name to more accurately "unicode_unescape". Patch is >> assigned >> >> You've made this function

Re: Should we remove a fallback promotion? take 2

2020-07-28 Thread Hamid Akhtar
There have been no real objections on this patch and it seems to work. So, IMHO, the only thing that needs to be done is perhaps update the patch so that it applies clearly on the master branch. On Mon, Jul 27, 2020 at 9:31 PM Fujii Masao wrote: > > > On 2020/07/27 17:53, Hamid Akhtar wrote: > >

Doc patch: mention indexes in pg_inherits docs

2020-07-28 Thread Dagfinn Ilmari Mannsåker
Hi Hackers, When partitioned index support was added in veresion 11, the pg_inherits docs missed the memo and still only say it describes table inheritance. The attached patch adds mentions of indexes too, and notes that they can not participate in multiple inheritance. I don't know what the poli

Re: Parallel worker hangs while handling errors.

2020-07-28 Thread Bharath Rupireddy
On Tue, Jul 28, 2020 at 11:05 AM vignesh C wrote: > > Thanks for your comments Bharath. > > On Mon, Jul 27, 2020 at 10:13 AM Bharath Rupireddy > wrote: > > 1. Do we need "worker" as a function argument in > > update_parallel_worker_sigmask(BackgroundWorker *worker, ? Since > > MyBgworkerEntry

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-07-28 Thread Julien Rouhaud
On Tue, Jul 28, 2020 at 10:07 AM torikoshia wrote: > > Thanks for updating! > I tested the patch setting log_statement = 'all', but %Q in > log_line_prefix > was always 0 even when pg_stat_statements.queryid and > pg_stat_activity.queryid are not 0. > > Is this an intentional behavior? > >[...] T

Re: Parallel bitmap index scan

2020-07-28 Thread Dilip Kumar
On Mon, Jul 27, 2020 at 3:48 AM Thomas Munro wrote: > > On Mon, Jul 27, 2020 at 1:58 AM Dilip Kumar wrote: > > On Sun, Jul 26, 2020 at 6:42 PM Dilip Kumar wrote: > > > > > > I would like to propose a patch for enabling the parallelism for the > > > bitmap index scan path. > > Wor

Re: recovering from "found xmin ... from before relfrozenxid ..."

2020-07-28 Thread Andrey M. Borodin
> 27 июля 2020 г., в 09:36, Ashutosh Sharma написал(а): > > > Also, should we try to fix VM along the way? > > I think we should let VACUUM do that. Sometimes VACUUM will not get to these pages, because they are marked All Frozen. An possibly some tuples will get stale on this page again >

Re: Feature improvement: can we add queryId for pg_catalog.pg_stat_activity view?

2020-07-28 Thread torikoshia
On 2020-07-14 20:24, Julien Rouhaud wrote: On Tue, Jul 14, 2020 at 07:11:02PM +0900, Atsushi Torikoshi wrote: Hi, v9 patch fails to apply to HEAD, could you check and rebase it? Thanks for the notice, v10 attached! And here are minor typos. 79 +* utility statements. Note that we

Re: printing oid with %d

2020-07-28 Thread Michael Paquier
On Mon, Jul 27, 2020 at 08:55:23PM -0500, Justin Pryzby wrote: > +JsonEncodeDateTime(char *buf, Datum value, Oid typid) > ... > + elog(ERROR, "unknown jsonb value datetime type oid > %d", typid); > > I think this should be %u. Good catch. Yep, Oids are unsigned. We don't

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-28 Thread Alexey Kondratov
On 2020-07-28 03:33, Andrey Lepikhov wrote: 27.07.2020 21:34, Alexey Kondratov пишет: Hi Andrey, On 2020-07-23 09:23, Andrey V. Lepikhov wrote: On 7/16/20 2:14 PM, Amit Langote wrote: Amit Langote EnterpriseDB: http://www.enterprisedb.com Version 5 of the patch. With changes caused by Amit