Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-09 Thread Peter Geoghegan
d, which is how much work we need to > do to not further worsen the corruption. You're right. Just trying to put the risk in context, and to understand the extent of the concern that you have. -- Peter Geoghegan

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-09 Thread Peter Geoghegan
e actual root (it might be some other HOT chain root following TID recycling by VACUUM)? Assuming that's what you meant: I would have thought that the xmin/xmax matching within heap_get_root_tuples() makes the sanity checking fairly reliable in practice. -- Peter Geoghegan -- Sent via pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-09 Thread Peter Geoghegan
ly. This explanation clears things up, though. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-09 Thread Peter Geoghegan
HOT chains are sane, which is how the enhanced amcheck notices the bug here in practice. (Before this bug was discovered, I would have expected amcheck to catch problems like it slightly later, during the Bloom filter probe for that HOT chain...but, in fact, it never gets there with corruption from this bu

Re: [HACKERS] pageinspect option to forgo buffer locking?

2017-11-09 Thread Peter Geoghegan
n get corrupted > contents that way, but we already allow to pass arbitrary stuff to > heap_page_items(). Since pinning wouldn't be changed, there's no danger > of the page being moved out from under us. +1. I've done things like this before myself. -- Peter Geoghegan -- Sent via pgsql-hac

Re: [HACKERS] Small improvement to compactify_tuples

2017-11-08 Thread Peter Geoghegan
ps://www.postgresql.org/message-id/CAH2-Wz=mv4dmoapficrsyntv2kinxeotbwuy5r7fxxoc-oe...@mail.gmail.com -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Small improvement to compactify_tuples

2017-11-08 Thread Peter Geoghegan
e4e3d0... IIRC, qsort_ssup() was about 20% faster at the time, while qsort_tuple() was 5% - 10% faster. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-07 Thread Peter Geoghegan
On Tue, Nov 7, 2017 at 3:29 PM, Nico Williams <n...@cryptonector.com> wrote: > On Thu, Nov 02, 2017 at 03:25:48PM -0700, Peter Geoghegan wrote: >> Nico Williams <n...@cryptonector.com> wrote: >> >A MERGE mapped to a DML like this: > > I needed to spend mor

Re: [HACKERS] Small improvement to compactify_tuples

2017-11-07 Thread Peter Geoghegan
erms of index bloat (though less so in terms of raw TPS). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Small improvement to compactify_tuples

2017-11-07 Thread Peter Geoghegan
On Tue, Nov 7, 2017 at 2:36 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > Peter Geoghegan <p...@bowt.ie> writes: >> My point is only that it's worth considering that this factor affects >> how representative your sympathetic case is. It's not clear how many >> Page

Re: [HACKERS] Small improvement to compactify_tuples

2017-11-07 Thread Peter Geoghegan
, how important that subset of calls is, and so on. Maybe it doesn't matter at all. [1] https://postgr.es/m/cah2-wzmyry7mnjf0gw5wtk3cszh3gqfhhoxvsyuno5pk8cu...@mail.gmail.com -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] Parallel Hash take II

2017-11-07 Thread Peter Geoghegan
pfree(accessor->write_chunk); > + accessor->write_chunk = (SharedTuplestoreChunk *) > + palloc0(accessor->write_pages * BLCKSZ); > > Are we guaranteed to be in a long-lived memory context here? I imagine

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-06 Thread Peter Geoghegan
at the predicate touches, but of course that isn't at all practical. I should stop trying to make a watertight case against this, even though I still think that's possible. For now, instead, I'll just say that this is *extremely* complicated, and still has unresolved questions about semantics. --

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-06 Thread Peter Geoghegan
it would be important to point out that MERGE with Approach1 isn't special, but ON CONFLICT DO UPDATE is special. We'd also say that higher isolation levels will not have duplicate violations. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes t

Re: [HACKERS] Display number of heap accesses for index scans

2017-11-05 Thread Peter Geoghegan
zation#Making_all_items_in_the_index_unique_by_treating_heap_TID_as_an_implicit_last_attribute -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Small improvement to compactify_tuples

2017-11-04 Thread Peter Geoghegan
tially presored inputs is what I like to call "banana skin effects": https://postgr.es/m/cah2-wzku2xk2dpz7n8-a1mvuuttuvhqkfna+eutwnwctgyc...@mail.gmail.com This may have nothing at all to do with your results; I'm just pointing it out as a possibility. -- Peter Geoghegan -- Sent v

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-03 Thread Peter Geoghegan
mp;& !locker_remains)) xmax_new_tuple = InvalidTransactionId; else xmax_new_tuple = HeapTupleHeaderGetRawXmax(oldtup.t_data); My naive guess is that we have to create a new MultiXactId here in at least some cases, just like FreezeMultiXactId() sometimes does. -- Peter Geoghegan -

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-03 Thread Peter Geoghegan
ething here? Can you be more specific about what you mean here? I think that I understand where you're going with this, but I'm not sure. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-03 Thread Peter Geoghegan
andard? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-11-02 Thread Peter Geoghegan
t have to opt in to BufFile's double buffering and segmentation schemes just to get shared file clean-up, if for some reason you want direct file handles. Is that something that you really think is possible? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-02 Thread Peter Geoghegan
, your example doesn't actually have a source (just a target), so it isn't actually like MERGE. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-02 Thread Peter Geoghegan
Yes, that certainly will make an easier patch for MERGE. Indeed, it will. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-02 Thread Peter Geoghegan
agree that what I propose for MERGE will probably cause confusion; just look into Oracle's MERGE implementation for examples of this. We ought to go out of our way to make it clear that MERGE doesn't provide these guarantees. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-h

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-02 Thread Peter Geoghegan
isn't particularly technically challenging IMV. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-02 Thread Peter Geoghegan
oesn't seem that compelling. I mention it now because It's worth acknowledging that ON CONFLICT could be pushed a bit further in this direction. Of course, this still falls far short of making ON CONFLICT entirely like MERGE. [1] https://commitfest.postgresql.org/15/1241/ -- Peter Geoghegan -- Se

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-02 Thread Peter Geoghegan
t those two cases differently. It was buggy even on its own terms. The FrozenTransactionId test used an xmin from HeapTupleHeaderGetXmin(), not HeapTupleHeaderGetRawXmin(). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscr

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-02 Thread Peter Geoghegan
not it is safe to interrogate clog for a given heap tuple using a tool like amcheck. And, it wasn't obvious that you couldn't have a codepath that failed to account for pre-cutoff non-frozen tuples -- codepaths that call TransactionIdDidCommit() despite it actually being unsafe. If I'm not mis

Re: [HACKERS] MERGE SQL Statement for PG11

2017-11-01 Thread Peter Geoghegan
t was as painful a project as it was. Further generalizing that seems fraught with difficulties. It seems logically impossible to generalize it in a way where you don't end up with two behaviors masquerading as one. [1] https://wiki.postgresql.org/wiki/UPSERT#Theoretical_lock_starvation_hazards [2] https

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-10-31 Thread Peter Geoghegan
just use a BufFileSet directly as a member of SharedSort. FWIW, I agree with you that nobody uses temp_tablespaces this way these days. This seems like a discussion for your hash join patch, though. I'm happy to buy into that. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-10-31 Thread Peter Geoghegan
matters -- perhaps you should leave it there in order to keep the code simple, as you'll be keeping the leader tape in local memory, too. (But it still won't fly to continue to clobber it, of course -- you still need to find a dedicated place for BufFileSet in shared memory.) That's all I have right no

Re: [HACKERS] MERGE SQL Statement for PG11

2017-10-31 Thread Peter Geoghegan
r system. Both Robert and Stephen seem to be almost sold on what I suggest, so I think that I've probably already explained my position quite well. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] MERGE SQL Statement for PG11

2017-10-30 Thread Peter Geoghegan
ust not constructively engaging with me at this point. We're going around in circles. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] MERGE SQL Statement for PG11

2017-10-30 Thread Peter Geoghegan
(at a minimum) define those semantics ahead of time, since you're going to commit us to them in the long term if you continue down this path. It is most emphatically *not* just a "small matter of programming". -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] MERGE SQL Statement for PG11

2017-10-29 Thread Peter Geoghegan
doing so will simplify the implementation, but I can guarantee you that it won't. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] MERGE SQL Statement for PG11

2017-10-28 Thread Peter Geoghegan
CONFLICT guarantees, and ultimately arriving at something that is comparable to other implementations over many releases might be okay if anyone had the slightest idea of what that would look like. You haven't even _described the semantics_, which you could do by addressing the specific points th

Re: [HACKERS] MERGE SQL Statement for PG11

2017-10-28 Thread Peter Geoghegan
ted having inserted such a row, that still counts as a CONFLICT with READ COMMITTED. [1] https://wiki.postgresql.org/wiki/UPSERT#Goals_for_implementation [2] https://www.postgresql.org/docs/devel/static/transaction-iso.html#xact-read-committed -- Peter Geoghegan -- Sent via pgsql-ha

Re: [HACKERS] MERGE SQL Statement for PG11

2017-10-28 Thread Peter Geoghegan
else to resolve the problems. Someone else must square the circle of mixing ON CONFLICT semantics with fully generalized MERGE semantics. But who? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] MERGE SQL Statement for PG11

2017-10-27 Thread Peter Geoghegan
exes, but only in passing, saying something about unique violations, and that unique violations *cannot* be suppressed in MERGE, even though that's possible with other DML statements (with something called IGNORE_DUP_KEY). What other systems *do* have this restriction? I've never seen one that

Re: [HACKERS] MERGE SQL Statement for PG11

2017-10-27 Thread Peter Geoghegan
On Fri, Oct 27, 2017 at 6:24 AM, Robert Haas <robertmh...@gmail.com> wrote: > I think one of the reasons why Peter Geoghegan decided to pursue > INSERT .. ON CONFLICT UPDATE was that, because it is non-standard SQL > syntax, he felt free to mandate a non-standard SQL requ

Re: [HACKERS] MERGE SQL Statement for PG11

2017-10-27 Thread Peter Geoghegan
d/CAM3SWZRP0c3g6+aJ=yydgyactzg0xa8-1_fcvo5xm7hrel3...@mail.gmail.com -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unique index violation after pg_upgrade to PG10

2017-10-24 Thread Peter Geoghegan
----+----+-++++---+ > 27 | 0 |0 | 0 ||| || >|||| | This looks like an LP_DEAD item. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unique index violation after pg_upgrade to PG10

2017-10-24 Thread Peter Geoghegan
they could cause corruption like this if you weren't careful. (In general, I wouldn't recommend using LVM snapshots as any kind of backup solution.) -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] unique index violation after pg_upgrade to PG10

2017-10-24 Thread Peter Geoghegan
petergeoghegan/amcheck Unlike the version in Postgres 10, this enhanced version (V1.2) has "heapallindexed" verification, which is really what you want here. If you install it, and run it against the unique index in question (with "heapallindexed" verification), that could help. It mi

Re: [HACKERS] How to determine that a TransactionId is really aborted?

2017-10-22 Thread Peter Geoghegan
his low level requirement into context if you want sound advice from this list. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] How to determine that a TransactionId is really aborted?

2017-10-22 Thread Peter Geoghegan
On Sun, Oct 22, 2017 at 12:23 PM, Eric Ridge <eeb...@gmail.com> wrote: > Can anyone confirm or deny that this is correct? I feel like it is correct, > but I'm no expert. What are you going to use the code for? I think that that context is likely to matter here. -- Peter Geoghega

Re: [HACKERS] A design for amcheck heapam verification

2017-10-20 Thread Peter Geoghegan
On Thu, Oct 5, 2017 at 7:00 PM, Peter Geoghegan <p...@bowt.ie> wrote: > v3 of the patch series, attached, does it that way -- it adds a > bloom_create(). The new bloom_create() function still allocates its > own memory, but does so while using a FLEXIBLE_ARRAY_MEMBER. A > s

Re: [HACKERS] [COMMITTERS] pgsql: Fix traversal of half-frozen update chains

2017-10-19 Thread Peter Geoghegan
On Thu, Oct 19, 2017 at 9:03 AM, Peter Geoghegan <p...@bowt.ie> wrote: >> /me studies the problem for a while. >> >> What's bothering me about this is: how is cutoff_xid managing to be a >> new enough transaction ID for this to happen in the first place? The

Re: [HACKERS] [COMMITTERS] pgsql: Fix traversal of half-frozen update chains

2017-10-19 Thread Peter Geoghegan
hain). Now that the problem is fixed by a5736bf7, this test case will prune and get an LP_REDIRECT ItemId (not an LP_DEAD one), as we're no longer confused about the continuity of HOT chains within heap_prune_chain(). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-h

Re: [HACKERS] [COMMITTERS] pgsql: Fix traversal of half-frozen update chains

2017-10-18 Thread Peter Geoghegan
eap-only tuples, as well as HOT root tuples and non-HOT tuples. But, as I said, I'm still playing catch-up on MultiXacts, and I too feel like I might still be missing important details. [1] https://postgr.es/m/20171017100200.ruszi2c6qqwetce5@alvherre.pgsql -- Peter Geoghegan -- Sent via pgsql

[HACKERS] Re: heap/SLRU verification, relfrozenxid cut-off, and freeze-the-dead bug (Was: amcheck (B-Tree integrity checking tool))

2017-10-18 Thread Peter Geoghegan
ly while holding a super-exclusive lock on the buffer. I can probably find a way to ensure this only needs to happen in a rare slow path, when it looks like the invariant might be violated but we need to make sure (I'm already following this pattern in a couple of places). Realistically, there will be some amou

Re: [HACKERS] [COMMITTERS] pgsql: Fix traversal of half-frozen update chains

2017-10-17 Thread Peter Geoghegan
earn about MultiXacts in the past few months. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-10-16 Thread Peter Geoghegan
On Sat, Oct 14, 2017 at 2:47 PM, Peter Geoghegan <p...@bowt.ie> wrote: > I am working on an experimental version of pg_filedump, customized to > output XML that can be interpreted by an open source hex editor. The > XML makes the hex editor produce color coded, commented > tags/a

[HACKERS] Re: heap/SLRU verification, relfrozenxid cut-off, and freeze-the-dead bug (Was: amcheck (B-Tree integrity checking tool))

2017-10-16 Thread Peter Geoghegan
ostgres can have. I may never be able to get general buy-in here; building broad consensus like that is a lot harder than writing some code for a contrib module. Making the checking code the *authoritative* record of how invariants are *expected* to work is a major goal of the project, though. -- Pe

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-10-14 Thread Peter Geoghegan
pens, the "work backwards from the storage format" perspective feels very natural to me. I do think that I understand where you're coming from too, though. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-10-13 Thread Peter Geoghegan
play the information in what I > believed to be the clearest and most accurate way. pg_filedump doesn't display HEAP_XMIN_FROZEN, either. (Nor does it ever display any of the other composite t_infomask/t_infomask2 values.) -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Pluggable storage

2017-10-13 Thread Peter Geoghegan
erent >> way) and so on. > > Fully agreed. If we implement that interface, where does that leave EvalPlanQual()? Do those semantics have to be preserved? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Fix traversal of half-frozen update chains

2017-10-13 Thread Peter Geoghegan
ugh. > And maybe still busted when > things wrap around multiple times and raw-xmins are reused? I'm more concerned about the situation that will remain (or has now been created) on 9.3, where we don't have raw xmin, and so must use very forgiving FrozenTransactionId matching within

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2017-10-12 Thread Peter Geoghegan
? Where does it end? I think that we should prominently document that HEAP_XMIN_COMMITTED |HEAP_XMIN_ABORTED == HEAP_XMIN_FROZEN, rather than trying to hide complexity that we have no business hiding in a tool like pageinspect. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] Pluggable storage

2017-10-11 Thread Peter Geoghegan
the MySQL model. It's not just the MySQL model, FWIW. SQL-on-Hadoop systems like Impala, certain NoSQL systems, and AFAIK any database system that claims to have pluggable storage all do it this way. That is, core transaction management functions (e.g. MVCC snapshot acquisition) is outsour

Re: [HACKERS] On markers of changed data

2017-10-09 Thread Peter Geoghegan
a. > > Yes, we've seen environments like that also. I'm pretty sure that those cases are cases where there are many more FPIs than might be expected, due to a lack of locality. (UUID PKs can make the size of WAL balloon, for example.) -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] heap/SLRU verification, relfrozenxid cut-off, and freeze-the-dead bug (Was: amcheck (B-Tree integrity checking tool))

2017-10-09 Thread Peter Geoghegan
e "premature" truncation takes place. This may be related to the recent HOT chain/freezing bugs, and our (only partial [3]) fixes may have fixed that, too -- I just don't know. [1] https://commitfest.postgresql.org/15/1263/ [2] https://postgr.es/m/CAMkU=1y-tnp_7jdfg_ubxqdb8esmo280acdgdw

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-08 Thread Peter Geoghegan
Perhaps you can be more specific? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Prepared statements assume text type in PG10

2017-10-07 Thread Peter Geoghegan
On Sat, Oct 7, 2017 at 4:27 PM, Peter Geoghegan <p...@bowt.ie> wrote: > I suspect commit d8d32d9 is involved here, though I haven't verified that. Weirdly, there is a git hash collision here, so you'll have to put in d8d32d9a (8 characters -- the default of 7 for a short git hash isn'

Re: [HACKERS] Prepared statements assume text type in PG10

2017-10-07 Thread Peter Geoghegan
s > jack=# execute ps('Hello, there'); >?column? > -- > Hello, there > (1 row) > > Time: 0.437 ms > > I looked through the git log and couldn't find any commits referencing this. > Is this an intended behavior change? I suspect commit d8d32d9 is involved here, though I

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-07 Thread Peter Geoghegan
t/stress-test. I tried to do so on the master branch git tip. This reinforces the theory that there is some timing sensitivity, because the remaining race condition is very narrow. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-06 Thread Peter Geoghegan
g further, which I assume he'll post as soon as he can. There doesn't seem to be much point in not waiting for that. [1] https://postgr.es/m/20171005162402.jahqflf3mekileqm@alvherre.pgsql -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-06 Thread Peter Geoghegan
On Fri, Oct 6, 2017 at 11:34 AM, Peter Geoghegan <p...@bowt.ie> wrote: >> I don't know if it's really the freeze map at fault or something else. > > Ideally, it would be possible to effectively disable the new freeze > map stuff in a minimal way, for testing purposes

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-06 Thread Peter Geoghegan
retically) impossible to reproduce the problem on versions < 9.6, though not on 9.6+. Applying Occam's razor, the problem doesn't seem particularly likely to be in the freeze map stuff, which isn't actually all that closely related. [1] https://postgr.es/m/20171005162402.jahqflf3mekileqm@

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-06 Thread Peter Geoghegan
uch earlier commits)? * NB: It is not enough to set hint bits to indicate something is * committed/invalid -- they might not be set on a standby, or after crash * recovery. We really need to remove old xids. */ We WAL-log setting hint bits during freezing now, iff tuple xmin is before the Xid cutoff and

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-06 Thread Peter Geoghegan
eappear. What problem persists? The original one (or, at least, the original symptom of pruning HOT chains incorrectly)? If that's what you mean, I wouldn't be so quick to assume that it's the freeze map. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.or

Re: [HACKERS] A design for amcheck heapam verification

2017-10-05 Thread Peter Geoghegan
On Fri, Sep 29, 2017 at 10:54 AM, Peter Geoghegan <p...@bowt.ie> wrote: >> Something that allocates new memory as the patch's bloom_init() >> function does I'd tend to call 'make' or 'create' or 'new' or >> something, rather than 'init'. > > I tend to agree. I'l

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-05 Thread Peter Geoghegan
ansaction. It is particularly > noticeably in our current test case because we use a min freeze age of > 0, with many concurrrent modifying the same page. What this says to me > is that VACUUM FREEZE is mildly dangerous when there's lots of high > concurrent HOT UPDATE activity. I'

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-04 Thread Peter Geoghegan
we only need to check for FrozenTransactionId on 9.3, why not just do that on all versions, and never bother with HeapTupleHeaderGetRawXmin()? ("Sheer paranoia" is a valid answer; I just want us to be clear on the reasoning.) Obviously any race would have a ridiculously tiny window, but it's not obvious

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-04 Thread Peter Geoghegan
r, which must be well out of range for ItemIds on the page. Could be worth a comment. (I guess that heap_prune_chain() wouldn't need to be changed if we decide to add such comments, because the speculative tuple ItemId is going to be skipped over due to being ItemIdIsUsed() before we even get th

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-04 Thread Peter Geoghegan
| > 5 || 0 |0 || > 6 || 0 |0 || > 7 | (0,7) | 8112 |1 | 11010 | 32771 > (7 rows) That's also what I see. This is a good thing, I think; that's how we ought to prune. -- Peter Geoghega

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-03 Thread Peter Geoghegan
king for the day, but I will point out that src/backend/access/heap/README.HOT says a fair amount about this, under "Abort Cases". -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-03 Thread Peter Geoghegan
, I may be missing something, but why is priorXmax updated > even for dead tuples? For example just doing that is also taking care > of the problem: I'll study what you suggest here some more tomorrow. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-03 Thread Peter Geoghegan
||| | 3 |0 | ||| | 4 |0 | ||| | 5 |0 | ||| | 6 |0 | ||| | 7 |1 | 1846001 | 0 | (0,7) | 2b02 | 8003 (7 rows) -- Peter G

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-10-03 Thread Peter Geoghegan
ved for forensic analysis, following commit 37484ad2? I guess what you mean is that this is what you see having modified the code to actually store FrozenTransactionId as xmin once more, in an effort to fix this? -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To

Re: [HACKERS] 64-bit queryId?

2017-10-02 Thread Peter Geoghegan
ause there was no 32-bit unsigned int type that we could have used (except possibly Oid, but that's weird). You see the same pattern in a couple of other places. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-10-02 Thread Peter Geoghegan
ea of how things are canonicalized or sanitized. It's too late for that now. [1] https://www.postgresql.org/message-id/CAH2-WzmVtRyNg2gT=u=ktEC-jM3aKq4bYzJ0u2=osxe+o3k...@mail.gmail.com [2] https://www.postgresql.org/message-id/f6c0fca7-e277-3f46-c0c1-adc001bff...@2ndquadrant.com -- Peter Geoghegan -- Se

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-10-01 Thread Peter Geoghegan
to make such significant changes. The general community opinion > also seems to be in favor of letting things be. It does seem too late. It's disappointing that we did not do better here. This problem was entirely avoidable. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hack

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-09-30 Thread Peter Geoghegan
me of the "tupkeep" stuff, 0002-Don-t-freeze-recently-dead-HOT-tuples, which you posted on September 6th. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-30 Thread Peter Geoghegan
eral change in the inherited root collator that doesn't really affect English speakers. There is no practical question about how you're supposed to sort English text. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://

Re: [HACKERS] Re: CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-30 Thread Peter Geoghegan
assed a string in BCP 47 format. I'm surprised that this issue was not resolved earlier in the week. I presumed that all of this was obvious to Peter E., but I seem to have been wrong about that. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make cha

Re: [HACKERS] 64-bit queryId?

2017-09-30 Thread Peter Geoghegan
On Sat, Sep 30, 2017 at 8:39 AM, Peter Geoghegan <p...@bowt.ie> wrote: > Isn't that already true in the case of queryId? I've never heard any > complaints about collisions. Most people don't change > pg_stat_statements.max, so the probability of a collision is more like &g

Re: [HACKERS] 64-bit queryId?

2017-09-30 Thread Peter Geoghegan
on. I have heard complaints about a number of different things in pg_stat_statements, like the fact that we don't always manage to replace constants with '?'/'$n' characters in all cases. I heard about that quite a few times during my time at Heroku. But never this. -- Peter Geoghegan -- Sent

Re: [HACKERS] CREATE COLLATION does not sanitize ICU's BCP 47 language tags. Should it?

2017-09-29 Thread Peter Geoghegan
d of shipping v10? This remains an open item. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

2017-09-29 Thread Peter Geoghegan
rly for me to comment anything in this area. I need > to study this more closely. As an initial goal I was just focused on > understanding the current implementation of the patch and incorporate > the BufFileSet APIs. Fair enough. -- Peter Geoghegan -- Sent via pgsql-hackers mai

Re: [HACKERS] pgbench stuck with 100% cpu usage

2017-09-29 Thread Peter Geoghegan
rted separately. Yeah, I use pgbench for stuff like that all the time. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A design for amcheck heapam verification

2017-09-29 Thread Peter Geoghegan
ize of the set), so it's a fairly common requirement. And, we have a convenient way to get a second independent uint32 hash function now (murmurhash32()). -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postg

Re: [HACKERS] A design for amcheck heapam verification

2017-09-29 Thread Peter Geoghegan
does I'd tend to call 'make' or 'create' or 'new' or > something, rather than 'init'. I tend to agree. I'll adopt that style in the next version. I just didn't want the caller to have to manage the memory themselves. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hacke

Re: [HACKERS] The case for removing replacement selection sort

2017-09-29 Thread Peter Geoghegan
e evidence we have supports the > contention that it is no longer effective. I'll go commit this. Thanks. [1] https://lwn.net/Articles/353411/ -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] The case for removing replacement selection sort

2017-09-28 Thread Peter Geoghegan
On Thu, Sep 28, 2017 at 3:18 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Jul 14, 2017 at 6:20 PM, Peter Geoghegan <p...@bowt.ie> wrote: >> With the additional enhancements made to Postgres 10, I doubt that >> there are any remaining cases where it wins. >

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-09-28 Thread Peter Geoghegan
On Thu, Sep 28, 2017 at 3:20 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Sep 28, 2017 at 5:47 PM, Peter Geoghegan <p...@bowt.ie> wrote: >> In the end, commit 6bfa88a fixed that old recovery bug by making sure >> the recovery routine heap_xlog_lock() d

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-09-28 Thread Peter Geoghegan
On Thu, Sep 28, 2017 at 2:47 PM, Peter Geoghegan <p...@bowt.ie> wrote: > In the end, commit 6bfa88a fixed that old recovery bug by making sure > the recovery routine heap_xlog_lock() did the right thing. In both > cases (Feb 2014 and today), the index wasn't really corrupt -- it

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-09-28 Thread Peter Geoghegan
ed that old recovery bug by making sure the recovery routine heap_xlog_lock() did the right thing. In both cases (Feb 2014 and today), the index wasn't really corrupt -- it just pointed to the root of a HOT chain when it should point to some child tuple (or maybe a successor HOT chain). -- Pete

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-09-28 Thread Peter Geoghegan
ostgresql.org/message-id/CAM3SWZTMQiCi5PV5OWHb+bYkUcnCk=o67w0csswpvv7xfuc...@mail.gmail.com -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] A design for amcheck heapam verification

2017-09-27 Thread Peter Geoghegan
to keep things simple. I'm not aware of any use case that calls for the user to use a custom hash function. That said, I could believe that someone would want to use their own hash value for each bloom_add_element(), when they have one close at hand anyway -- much like addHyperLogLog(). Again, th

Re: [HACKERS] The case for removing replacement selection sort

2017-09-27 Thread Peter Geoghegan
work with a special heap comparator that compares run number, but only when replacement selection is in use. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

  1   2   3   4   5   6   7   8   9   10   >