Re: [HACKERS] A weird bit in pg_upgrade/exec.c

a.akent...@postgrespro.ru wrote: > This function has two calls: > check_bin_dir(_cluster); > check_bin_dir(_cluster); > > I'd like to substitute these last two lines with this: > get_bin_version(cluster); Odd indeed. One would think that if a cluster variable is passed as parameter, the global

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Alvaro Herrera wrote: > I think your argument is sensible for some uses (where people run manual > VACUUM after loading data) but not others (where people just use manual > VACUUM in place of autovacuuming -- because they don't trust autovac, or > the schedule isn't convenient, or wh

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Peter Geoghegan wrote: > Andres Freund wrote: > > Staring at the vacuumlazy hunk I think I might have found a related bug: > > heap_update_tuple() just copies the old xmax to the new tuple's xmax if > > a multixact and still running. It does so without verifying liveliness >

Re: [HACKERS] ucs_wcwidth vintage

Thomas Munro wrote: > Hi hackers, > > src/backend/utils/mb/wchar.c contains a ~16 year old wcwidth > implementation that originally arrived in commit df4cba68, but the > upstream code[1] apparently continued evolving and there have been > more Unicode revisions since. It probably doesn't matter

Re: [HACKERS] ucs_wcwidth vintage

Thomas Munro wrote: > Hi hackers, > > src/backend/utils/mb/wchar.c contains a ~16 year old wcwidth > implementation that originally arrived in commit df4cba68, but the > upstream code[1] apparently continued evolving and there have been > more Unicode revisions since. I think we should update it

Re: [HACKERS] Proposal: Local indexes for partitioned table

> Robert Haas writes: > > We could do that, but the motivation for the current system was to > > avoid leaking memory in a long-lived context. Yeah, my approach here is to use a CATCH block that deletes the memory context just created, thus avoiding a long-lived leak.

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tom Lane wrote: > Alvaro Herrera <alvhe...@alvh.no-ip.org> writes: > > Tom Lane wrote: > >> Do we still need the complication in brinsummarize to discriminate > >> against the last partial range? Now that the lock consideration > >> is gone, I thin

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tom Lane wrote: > Alvaro Herrera <alvhe...@alvh.no-ip.org> writes: > > > Yeah, I think this approach results in better code. The attached patch > > implements that, and it passes the test for me (incl. calling > > brin_summarize_new_values concurrently with vacuu

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Alvaro Herrera wrote: > Maybe a solution is to call RelationGetNumberOfBlocks() after the > placeholder tuple has been inserted, for the case where we would be > scanning past end of relation; passing InvalidBlockNumber as stop point > would indicate to do things that wa

Re: [HACKERS] dropping partitioned tables without CASCADE

Ashutosh Bapat wrote: > On Fri, Nov 3, 2017 at 1:42 PM, Alvaro Herrera <alvhe...@alvh.no-ip.org> > wrote: > > I think adding "is partitioned" at end of line isn't good; looks like a > > phrase but isn't translatable. Maybe add keyword PARTITIONED instead?

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tom Lane wrote: > Alvaro Herrera <alvhe...@alvh.no-ip.org> writes: > > Rather than remove the capability, I'd be inclined to make > > brin_summarize_new_values summarize the final partial range, and have > > VACUUM not do it. Would that be too inconsistent? > >

Re: [HACKERS] dropping partitioned tables without CASCADE

Amit Langote wrote: > On 2017/09/06 19:14, Amit Langote wrote: > > On 2017/09/06 18:46, Rushabh Lathia wrote: > >> Okay, I have marked this as ready for committer. > > > > Thanks Ashutosh and Rushabh for rebasing and improving the patch. Looks > > good to me too. > > Patch needed to be rebased

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tom Lane wrote: > Alvaro Herrera <alvhe...@alvh.no-ip.org> writes: > > Tom Lane wrote: > >> If VACUUM and brin_summarize_new_values both ignore the partial > >> range, then what else would request this? Can't we just decree > >> that we

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tom Lane wrote: > Alvaro Herrera <alvhe...@alvh.no-ip.org> writes: > > 2. when summarization is requested on the partial range at the end of a > > table, we acquire extension lock on the rel, then compute relation size > > and run summarization with the lock held. Th

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

ices >From a760bfd44afeff8d1629599411ac7ce87acc7d26 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera <alvhe...@alvh.no-ip.org> Date: Thu, 2 Nov 2017 18:35:10 +0100 Subject: [PATCH] Fix summarization concurrent with relation extension --- src/backend/acce

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tom Lane wrote: > Alvaro Herrera <alvhe...@alvh.no-ip.org> writes: > >> Hmm, I'm pretty sure we stress-tested brin in pretty much the same way. > >> But I see this misbehavior too. Looking ... > > > Turns out that this is related to concurrent growth of t

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Stephen Frost wrote: > * Tom Lane (t...@sss.pgh.pa.us) wrote: > > Andres Freund writes: > > > Do we care about people upgrading to unreleased versions? We could do > > > nothing, document it in the release notes, or ??? > > > > Do nothing. > > Agreed. Not much we can do

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Andres Freund wrote: > I spent some time discussing this with Robert today (with both of us > alternating between feeling the other and ourselves as stupid), and the > conclusion I think is that the problem is on the pruning, rather than > the freezing side. Thanks both for spending some more

Re: [HACKERS] Mapping MERGE onto CTEs (Re: MERGE SQL Statement for PG11)

Nico Williams wrote: > As an aside, I'd like to be able to control which CTEs are view-like and > which are table-like. In SQLite3, for example, they are all view-like, > and the optimizer will act accordingly, whereas in PG they are all > table-like, and thus optimizer barriers. There was a

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Alvaro Herrera wrote: > Tomas Vondra wrote: > > > FWIW I can reproduce this on 9.5, and I don't even need to run the > > UPDATE part. That is, INSERT + VACUUM running concurrently is enough to > > produce broken BRIN indexes :-( > > Hmm, I'm pretty sure we str

[HACKERS] strange relcache.c debug message

While messing with BRIN bugs, I noticed this debug message in the server log: 2017-11-01 12:33:24.042 CET [361429] DEBUG: rehashing catalog cache id 14 for pg_opclass; 17 tups, 8 buckets en carácter 194 notice that at the end it says "at character 194". I suppose that's because of some

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tomas Vondra wrote: > FWIW I can reproduce this on 9.5, and I don't even need to run the > UPDATE part. That is, INSERT + VACUUM running concurrently is enough to > produce broken BRIN indexes :-( Hmm, I'm pretty sure we stress-tested brin in pretty much the same way. But I see this misbehavior

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tom Lane wrote: > I really don't understand how any of this "let's release the buffer > lock and then take it back later" logic is supposed to work reliably. So summarize_range first inserts the placeholder tuple, which is there purposefully for other processes to update concurrently; then,

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tom Lane wrote: > So in a few more runs this morning using Alvaro's simplified test case, > I have seen the following behaviors not previously reported: > 1. Crashes in PageIndexTupleOverwrite, which has the same "invalid index > offnum: %u" error report as PageIndexTupleDeleteNoCompact. I note

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tom Lane wrote: > So: I put the blame on the fact that summarize_range() thinks that > the tuple offset it has for the placeholder tuple is guaranteed to > hold good, even across possibly-long intervals where it's holding > no lock on the containing buffer. Yeah, I think this is a pretty

Re: [HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Thanks everyone for the analysis downthread. Here's a test case that provokes the crash faster. Initialize with create table brin_test (a text); create index on brin_test using brin (a) with (pages_per_range = 1); Then in one psql, run this: select brin_summarize_new_values('brin_test_a_idx')

Re: [HACKERS] SIGSEGV in BRIN autosummarize

Alvaro Herrera wrote: > Alvaro Herrera wrote: > > > Before pushing, I'll give a look to the regular autovacuum path to see > > if it needs a similar fix. > > Reading that one, my conclusion is that it doesn't have the same problem > because the strings are allocate

Re: [HACKERS] Remove secondary checkpoint

Robert Haas wrote: > On Tue, Oct 24, 2017 at 7:25 PM, Andres Freund wrote: > > I think it does the contrary. The current mechanism is, in my opinion, > > downright dangerous: > > https://www.postgresql.org/message-id/20160201235854.go8...@awork2.anarazel.de > > A sort of

Re: [HACKERS] pow support for pgbench

Michael Paquier wrote: > Attaching patches directly to a thread is a better practice as if > github goes away, any Postgres developers can still have an access to > any code you publish using the public archives on postgresql.org. Also, by posting to pgsql-hackers indicating intention to

Re: [HACKERS] pow support for pgbench

Michael Paquier wrote: > Please add this patch to the upcoming commit fest if you would like to > get some feedback: > https://commitfest.postgresql.org/15/ > > I am adding as well Fabien in CC who worked in getting the internal > function infrastructure in the shape it is now (waaay better)

[HACKERS] Re: PANIC: invalid index offnum: 186 when processing BRIN indexes in VACUUM

Tomas Vondra wrote: > FWIW I can reproduce this on REL_10_STABLE, but not on REL9_6_STABLE. So > it seems to be due to something that changed in the last release. I've been trying to reproduce it for half an hour with no success (I think my laptop is just too slow). I bet this is related to the

Re: [HACKERS] Proposal: Local indexes for partitioned table

Alvaro Herrera wrote: > I'm now working on the ability to build unique indexes (and unique > constraints) on top of this. So I think there's not a lot of additional code required to support unique indexes with the restrictions mentioned; proof-of-concept (with several holes still) at

Re: [HACKERS] WIP: BRIN bloom indexes

Tomas Vondra wrote: > Not sure "a number of in-core opclasses" is a good reason to (not) add > new ones. Also, we already have two built-in BRIN opclasses (minmax and > inclusion). > > In general, "BRIN bloom" can be packed as a contrib module (at least I > believe so). That's not the case for

Re: [HACKERS] MERGE SQL Statement for PG11

Simon Riggs wrote: > Earlier thoughts on how this could/could not be done were sometimes > imprecise or inaccurate, so I have gone through the command per > SQL:2011 spec and produced a definitive spec in the form of an SGML > ref page. This is what I intend to deliver for PG11. Nice work. I

Re: [HACKERS] Burst in WAL size when UUID is used as PK while full_page_writes are enabled

Amit Kapila wrote: > You might want to give a try with the hash index if you are planning > to use PG10 and your queries involve equality operations. So, btree indexes on monotonically increasing sequences don't write tons of full page writes because typically the same page is touched many times

Re: [HACKERS] taking stdbool.h into use

Michael Paquier wrote: > On Thu, Oct 26, 2017 at 10:51 AM, Alvaro Herrera > <alvhe...@alvh.no-ip.org> wrote: > > I gave this a quick run, to see if my compiler would complain for things > > like this: > > > >boolisprimary = flags & INDEX_CREATE_I

Re: [HACKERS] taking stdbool.h into use

Peter Eisentraut wrote: > Here is an updated patch set. This is just a rebase of the previous > set, no substantial changes. Based on the discussion so far, I'm > proposing that 0001 through 0007 could be ready to commit after review, > whereas the remaining two need more work at some later

Re: [HACKERS] How to determine that a TransactionId is really aborted?

Eric Ridge wrote: > > Again, you'll probably need to put this low level requirement into > > context if you want sound advice from this list. > > I'm just thinking out lout here, but the context is likely something > along the lines of externally storing all transaction ids, and > periodically

Re: [HACKERS] Remove secondary checkpoint

Tsunakawa, Takayuki wrote: > (Although unrelated to this, I've also been wondering why PostgreSQL > flushes WAL to disk when writing a page in the shared buffer, because > PostgreSQL doesn't use WAL for undo.) The reason is that if the system crashes after writing the data page to disk, but

Re: [HACKERS] SIGSEGV in BRIN autosummarize

Alvaro Herrera wrote: > Before pushing, I'll give a look to the regular autovacuum path to see > if it needs a similar fix. Reading that one, my conclusion is that it doesn't have the same problem because the strings are allocated in AutovacuumMemCxt which is not reset by error re

Re: [HACKERS] SIGSEGV in BRIN autosummarize

PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 2543c9ee25245f63653bf342a0240eaa8a1dcd6f Mon Sep 17 00:00:00 2001 From: Alvaro Herrera <alvhe...@alvh.no-ip.org> Date: Mon, 23 Oct 2017 18:55:12 +0200 Subject: [PATCH] Fix autovacuum work ite

Re: [HACKERS] Proposal: Local indexes for partitioned table

Robert Haas wrote: > On Mon, Oct 23, 2017 at 11:12 AM, Alvaro Herrera > <alvhe...@alvh.no-ip.org> wrote: > > I started with Maksim's submitted code, and developed according to the > > ideas discussed in this thread. Attached is a very WIP patch series for > > thi

Re: [HACKERS] Proposal: Local indexes for partitioned table

bool update_pgindex, - bool remove_old_dependencies, + uint16 constr_flags, bool allow_system_table_mods,

Re: [HACKERS] [PATCH] Tests for reloptions

Oh, one more thing: be careful when editing parallel_schedule. There are constraints on the number of entries in each group; you had added a 20th entry after the comment that the group can only have 19. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7

Re: [HACKERS] [PATCH] Tests for reloptions

Nikolay Shaplov wrote: > В письме от 3 октября 2017 11:48:43 пользователь Michael Paquier написал: > I've been thinking a lot, and rereading the patch. When I reread it I've been > thinking that I would like to add more tests to it now... ;-) > > If the only purpose of tests is to get better

Re: [HACKERS] show "aggressive" or not in autovacuum logs

Kyotaro HORIGUCHI wrote: > How about the followings? > > "automatic [agressive ]vacuum of table \"%s..." > "[aggressive ]vacuuming \"%s..." That form of log message seems acceptable to me (first one is missing a 'g'). In any case, please do not construct the sentence with %s expanding the

Re: [HACKERS] SIGSEGV in BRIN autosummarize

Justin Pryzby wrote: > On Wed, Oct 18, 2017 at 06:54:09PM +0200, Alvaro Herrera wrote: > > And the previous code crashes in 45 minutes? That's solid enough for > > me; I'll clean up the patch and push in the next few days. I think what > > you have now should be sufficie

Re: [HACKERS] [COMMITTERS] pgsql: Implement table partitioning.

Robert Haas wrote: > On Wed, Oct 18, 2017 at 11:27 AM, Alvaro Herrera > <alvhe...@alvh.no-ip.org> wrote: > > Maybe there are combinations of different persistence values that can be > > allowed to differ (an unlogged partition is probably OK with a permanent > &

Re: [HACKERS] SIGSEGV in BRIN autosummarize

Justin Pryzby wrote: > No crashes in ~28hr. It occurs to me that it's a weaker test due to not > preserving most compilation options. And the previous code crashes in 45 minutes? That's solid enough for me; I'll clean up the patch and push in the next few days. I think what you have now

Re: [HACKERS] [COMMITTERS] pgsql: Implement table partitioning.

This check is odd (tablecmds.c ATExecAttachPartition line 13861): /* Temp parent cannot have a partition that is itself not a temp */ if (rel->rd_rel->relpersistence == RELPERSISTENCE_TEMP && attachrel->rd_rel->relpersistence != RELPERSISTENCE_TEMP) ereport(ERROR,

Re: [HACKERS] alter table doc fix

Amit Langote wrote: > Hi. > > Noticed that a alter table sub-command's name in Description (where it's > OWNER) differs from that in synopsis (where it's OWNER TO). Attached > patch to make them match, if the difference is unintentional. I agree -- pushed. This paragraph The actions

Re: [HACKERS] v10 telease note for pg_basebackup refers to old --xlog-method argument

Pushed. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] [COMMITTERS] pgsql: Implement table partitioning.

Amit Langote wrote: > On 2017/10/18 1:52, Alvaro Herrera wrote: > > Alvaro Herrera wrote: > >> Robert Haas wrote: > >>> Implement table partitioning. > >> > >> Is it intentional that you can use ALTER TABLE OWNER TO on the parent > >> table

Re: [HACKERS] [COMMITTERS] pgsql: Implement table partitioning.

Alvaro Herrera wrote: > Robert Haas wrote: > > Implement table partitioning. > > Is it intentional that you can use ALTER TABLE OWNER TO on the parent > table, and that this does not recurse to modify the partitions' owners? > This doesn't seem to be mentioned in comments

Re: [HACKERS] [COMMITTERS] pgsql: Implement table partitioning.

Robert Haas wrote: > Implement table partitioning. Is it intentional that you can use ALTER TABLE OWNER TO on the parent table, and that this does not recurse to modify the partitions' owners? This doesn't seem to be mentioned in comments nor documentation, so it seems an oversight to me.

Re: [HACKERS] SIGSEGV in BRIN autosummarize

Justin Pryzby wrote: > I'm happy to try the patch, but in case it makes any difference, we have few > DBs/schemas: I don't expect that it does. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via

Re: [HACKERS] SIGSEGV in BRIN autosummarize

p; Services >From 0a53aaf589dfdbd2f25ae2ee36323d77c2910a60 Mon Sep 17 00:00:00 2001 From: Alvaro Herrera <alvhe...@alvh.no-ip.org> Date: Tue, 17 Oct 2017 12:58:38 +0200 Subject: [PATCH] Fix autovacuum workitems --- src/backend/postmaster/autovacuum.c | 10 -- 1 file changed, 8 in

Re: [HACKERS] [COMMITTERS] pgsql: Fix traversal of half-frozen update chains

Robert Haas wrote: > I haven't really followed this thread in depth, but I'm very nervous > about the idea that we should ever be examining the raw-xmin of a > tuple that has been marked HEAP_XMIN_FROZEN for anything other than > forensic purposes. Yeah, me too. If you see another way to fix

Re: [HACKERS] SIGSEGV in BRIN autosummarize

Justin Pryzby wrote: > On Sun, Oct 15, 2017 at 02:44:58PM +0200, Tomas Vondra wrote: > > Thanks, but I'm not sure that'll help, at this point. We already know > > what happened (corrupted memory), we don't know "how". And core files > > are mostly just "snapshots" so are not very useful in

Re: [HACKERS] ERROR: MultiXactId 3268957 has not been created yet -- apparent wraparound after missused pg_resetxlogs

RADIX Alain - externe wrote: > I'm facing a problem with a PostgreSQL 9.6.2 reporting this error when > selecting a table > ERROR: MultiXactId 3268957 has not been created yet -- apparent wraparound > > The root cause is not a Postgres bug but a buggy person that used > pg_resetxlogs

[HACKERS] relkind check in DefineIndex

gt;From 1a7420321f7f48bbc87dfdd38ba10fa57c6513da Mon Sep 17 00:00:00 2001 From: Alvaro Herrera <alvhe...@alvh.no-ip.org> Date: Fri, 13 Oct 2017 17:56:44 +0200 Subject: [PATCH] reword kind check using switch --- src/backend/commands/indexcmds.c | 19 +-- 1 file changed, 9 insertions(+),

Re: [HACKERS] [COMMITTERS] pgsql: Implement table partitioning.

Robert Haas wrote: > Implement table partitioning. > Currently, tables can be range-partitioned or list-partitioned. List > partitioning is limited to a single column, but range partitioning can > involve multiple columns. A partitioning "column" can be an > expression. I find the "partition

Re: [HACKERS] [COMMITTERS] pgsql: Fix traversal of half-frozen update chains

Peter Geoghegan wrote: > On Fri, Oct 6, 2017 at 8:29 AM, Alvaro Herrera <alvhe...@alvh.no-ip.org> > wrote: > > /* > > * When a tuple is frozen, the original Xmin is lost, but we know it's a > > * committed transaction. So unless the Xmax is

[HACKERS] Re: Extended statistics is not working on Vars hidden under a RelabelType

David Rowley wrote: > The reason Tomas coded it the way it was coded is due to the fact that > there's already code that works exactly the same way in > clauselist_selectivity(). Personally, I don't particularly like that > code, but I'd rather not invent a new way to do the same thing. I pushed

[HACKERS] Re: Extended statistics is not working on Vars hidden under a RelabelType

Tomas Vondra wrote: > On 10/10/2017 05:03 AM, David Rowley wrote: > > Basically, $subject is causing us not to properly find matching > > extended stats in this case. > > > > The attached patch fixes it. > > > > The following test cases is an example of the misbehaviour. Note > > rows=1 vs

Re: [HACKERS] Discussion on missing optimizations

Andres Freund wrote: > On 2017-10-08 11:28:09 -0400, Tom Lane wrote: > > Adam Brusselback writes: > > > On another note: > > >> turning ORs into UNIONs > > > > > This is another one which would be incredibly useful for me. I've had > > > to do this manually for

Re: [HACKERS] replace GrantObjectType with ObjectType

Michael Paquier wrote: > On Thu, Oct 12, 2017 at 7:55 AM, Peter Eisentraut > wrote: > > It seems to me that having ACL_OBJECT_* symbols alongside OBJECT_* > > symbols is not useful and leads to duplication. Digging around in the > > past suggests that we used to

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Andres Freund wrote: > On 2017-10-11 10:53:56 +0200, Alvaro Herrera wrote: > > I wonder if it'd be a good idea to nag external users about pq_sendint > > usage (is a #warning possible?). > > Think we'd need some separate infrastructure, i.e. for gcc ending up > with _

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

Andres Freund wrote: > Hi, > > On 2017-10-03 13:58:37 -0400, Robert Haas wrote: > > On Tue, Oct 3, 2017 at 12:23 PM, Andres Freund wrote: > > > Makes sense? > > > > Yes. > > Here's an updated version of this patchset. Maybe it'd be a good idea to push 0001 with some user

Re: [HACKERS] pg_regress help output

Tom Lane wrote: > Joe Conway writes: > > I have been annoyed at least twice now by the lack of pg_regress command > > line help output for the "--bindir=" option. In passing I noted > > that there was no output for "--help" or "--version" options either. > > > Any objections

Re: [HACKERS] On markers of changed data

Greg Stark wrote: > The general shape of what I would like to see is some log which lists > where each checkpoint starts and ends and what blocks are modified > since the previous checkpoint. Then to generate an incremental backup > from any point in time to the current you union all the block

Re: [HACKERS] Proposal: Local indexes for partitioned table

Robert Haas wrote: > On Fri, Oct 6, 2017 at 12:37 PM, Alvaro Herrera <alvhe...@alvh.no-ip.org> > wrote: > > 2. create one index for each existing partition. These would be > >identical to what would happen if you created the index directly on >

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Wood, Dan wrote: > I’m unclear on what is being repro’d in 9.6. Are you getting the > duplicate rows problem or just the reindex problem? Are you testing > with asserts enabled(I’m not)? I was seeing just the reindex problem. I don't see any more dups. But I've tried to reproduce it afresh

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Peter Geoghegan wrote: > On Sat, Oct 7, 2017 at 1:31 AM, Alvaro Herrera <alvhe...@alvh.no-ip.org> > wrote: > >> As you must have seen, Alvaro said he has a variant of Dan's original > >> script that demonstrates that a problem remains, at least on 9.6+, > >&

Re: [HACKERS] On markers of changed data

Michael Paquier wrote: > That’s actually what pg_rman is doing for what it calls incremental > backups (perhaps that would be differential backup in PG > terminology?), and the performance is bad as you can imagine. We could > have a dedicated LSN map to do such things with 4 bytes per page. I am

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Peter Geoghegan wrote: > On Fri, Oct 6, 2017 at 2:09 PM, Wong, Yi Wen wrote: > > Yesterday, I've been spending time with pg_visibility on the pages when I > > reproduce the issue in 9.6. > > None of the all-frozen or all-visible bits are necessarily set in > > problematic

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Peter Geoghegan wrote: > On Fri, Oct 6, 2017 at 6:18 AM, Alvaro Herrera <alvhe...@alvh.no-ip.org> > wrote: > > Wood, Dan wrote: > >> Yes, I’ve been testing 9.6. I’ll try Alvaro’s patch today. > >> > >> I would prefer to focus on either latest 9X or 11

Re: [HACKERS] Proposal: Local indexes for partitioned table

Hello I've been thinking about this issue too. I think your patch is not ambitious enough. Here's my brain dump on the issue, to revive discussion. As you propose, IMO this new feature would use the standard index creation syntax: CREATE [UNIQUE] INDEX someindex ON parted_table (a, b);

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

By the way, I still wonder if there's any way for a new tuple to get inserted in the place where a HOT redirect would be pointing to, and have it be marked as Frozen, where the old redirect contains a non-invalid Xmax. I tried to think of a way for that to happen, but couldn't think of anything.

Re: [HACKERS] On markers of changed data

Michael Paquier wrote: > The only sane method for Postgres is really to scan the > page header LSNs, and of course you already know that. I hope the idea is not to have to scan every single page in the database, because that would be too slow. It should be possible to build this so that a

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Michael Paquier wrote: > On Fri, Oct 6, 2017 at 10:18 PM, Alvaro Herrera <alvhe...@alvh.no-ip.org> > wrote: > > Wood, Dan wrote: > >> Yes, I’ve been testing 9.6. I’ll try Alvaro’s patch today. > >> > >> I would prefer to focus on either latest

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

amp; Services >From 46ca12d56402d33a78ea0e13367d1e0e25a474dd Mon Sep 17 00:00:00 2001 From: Alvaro Herrera <alvhe...@alvh.no-ip.org> Date: Fri, 6 Oct 2017 14:11:34 +0200 Subject: [PATCH] Fix bugs --- src/backend/access/heap/heapam.c| 52 + src/backend/access/heap/pruneheap.c |

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Michael Paquier wrote: > On Fri, Oct 6, 2017 at 1:24 AM, Alvaro Herrera <alvhe...@alvh.no-ip.org> > wrote: > + /* > +* If the xmax of the old tuple is identical to the xmin of the new one, > +* it's a match. > +*/ > + if (xmax == xmin) > +

Re: [HACKERS] [PATCH] Assert that the correct locks are held when calling PageGetLSN()

Michael Paquier wrote: > On Tue, Oct 3, 2017 at 12:54 AM, Alvaro Herrera <alvhe...@alvh.no-ip.org> > wrote: > > I'd argue about this in the same direction I argued about > > BufferGetPage() needing an LSN check that's applied separately: if it's > > too easy for a d

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

I think this is the patch for 9.3. I ran the test a few hundred times (with some additional changes such as randomly having an update inside a savepoint that's randomly aborted, randomly aborting the transaction, randomly skipping the for key share lock, randomly sleeping at a few points; and

Re: [HACKERS] Partition-wise join for join between (declaratively) partitioned tables

Robert Haas wrote: > Regarding nomenclature and my previous griping about wisdom, I was > wondering about just calling this a "partition join" like you have in > the regression test. So the GUC would be enable_partition_join, you'd > have generate_partition_join_paths(), etc. Basically just

Re: [HACKERS] Logging idle checkpoints

Kyotaro HORIGUCHI wrote: > # This reminded me of a concern. I'd like to count vacuums that > # are required but skipped by lock-failure, or killed by other > # backend. We clearly need to improve the stats and logs related to vacuuming work executed, both by autovacuum and manually invoked. One

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

hink Alvaro should now memorialize > the idea that its value is actually critical in some place > (htup_details.h?). I'd love to be certain that we preserve the original value in all cases. > On Wed, Oct 4, 2017 at 6:46 AM, Alvaro Herrera <alvhe...@alvh.no-ip.org> > wrote: > >

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Wood, Dan wrote: > Whatever you do make sure to also test 250 clients running lock.sql. Even > with the communities fix plus YiWen’s fix I can still get duplicate rows. > What works for “in-block” hot chains may not work when spanning blocks. Good idea. You can achieve a similar effect by

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Wong, Yi Wen wrote: > My interpretation of README.HOT is the check is just to ensure the chain is > continuous; in which case the condition should be: > > > if (TransactionIdIsValid(priorXmax) && > > !TransactionIdEquals(priorXmax, > >

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Alvaro Herrera wrote: > Peter Geoghegan wrote: > > > I thought that we no longer store FrozenTransactionId (xid 2) as our > > "raw" xmin while freezing, and yet that's what we see here. > > I'm doing this in 9.3. I can't tell if the new tuple freezing st

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Wood, Dan wrote: > One minor side note… Is it weird for xmin/xmax to go backwards in a hot row > chain? > > lp | t_ctid | lp_off | t_infomask | t_infomask2 | t_xmin | t_xmax > ++++-++ > 1 | (0,1) | 8152 | 2818 |

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Wood, Dan wrote: > There is a tangled web of issues here. With the community fix we get a > corrupted page(invalid redirect ptr from indexed item). The cause of that is: > pruneheap.c: > > /* >* Check the tuple XMIN against prior XMAX, if any >

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

Peter Geoghegan wrote: > I thought that we no longer store FrozenTransactionId (xid 2) as our > "raw" xmin while freezing, and yet that's what we see here. I'm doing this in 9.3. I can't tell if the new tuple freezing stuff broke things more thoroughly, but it is already broken in earlier

Re: [HACKERS] datetime.h defines like PM conflict with external libraries

Andrew Dunstan wrote: > On 10/03/2017 04:43 PM, Tom Lane wrote: > > I like the new-header-file idea because it will result in minimal code > > churn and thus minimal back-patching hazards. > > > > I do *not* like "PG_PM". For our own purposes that adds no uniqueness > > at all. If we're to

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

So here's my attempt at an explanation for what is going on. At one point, we have this: select lp, lp_flags, t_xmin, t_xmax, t_ctid, to_hex(t_infomask) as infomask, to_hex(t_infomask2) as infomask2 from heap_page_items(get_raw_page('t', 0)); lp | lp_flags | t_xmin | t_xmax | t_ctid | infomask

Re: [HACKERS] list of credits for release notes

Michael Paquier wrote: > Looking at this list, the first name is followed by the family name, > so there are inconsistencies with some Japanese names: > - Fujii Masao should be Masao Fujii. > - KaiGai Kohei should be Kohei Kaigai. We've already been here:

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

Tom Lane wrote: > Alvaro Herrera <alvhe...@alvh.no-ip.org> writes: > > Tom Lane wrote: > >> I don't especially like the Asserts inside spinlocks, either. > > > I didn't change these. It doesn't look to me that these asserts are > > worth very much as pr

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

Fixed the pstrdup problem by replacing with strlcpy() to stack-allocated variables (rather than palloc + memcpy as proposed in Michael's patch). About the other problems: Tom Lane wrote: > I took a quick look through walreceiver.c, and there are a couple of > obvious problems of the same ilk in

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

Andreas Seltenreich wrote: > Michael Paquier writes: > > > I am attaching a patch that addresses the bugs for the spin lock sections. > > [2. text/x-diff; walreceiver-spin-calls.patch] > > I haven't seen a spinlock PANIC since testing with the patch applied. > They occured five times with the

  1   2   3   4   5   6   7   8   9   10   >