Re: [HACKERS] pgbench stuck with 100% cpu usage

laptop's battery started draining out much more quickly. Having seen the problem, it seems very obvious though. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] pgbench stuck with 100% cpu usage

performance, but testing crash recovery and I use it very often. This particular test was run to catch another bug which will be reported separately. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

[HACKERS] pgbench stuck with 100% cpu usage

t sure what was the rationale for that decision, may be to deal with transient failures? The commit that introduced this code is 12788ae49e1933f463bc. So I am copying Heikki. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Tr

Re: [HACKERS] Parallel worker error

rallelWorkerMain call > SetCurrentRoleId using information passed via the FixedParallelState > (not sure of the precise details here). > > Seems like a reasonable approach to me. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Parallel worker error

oncurrent role drop problem someday, the single session problem looks more pressing. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

[HACKERS] Parallel worker error

stand the differences in "role" and "session authorization", but it still looks like a bug to me. May be SerializeGUCState() should check if SetRoleIsActive is true and only then save the role information? Thanks, Pavan -- Pavan Deolasee http://www.2ndQ

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Jul 28, 2017 at 5:57 AM, Peter Geoghegan <p...@bowt.ie> wrote: > Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > I'll be happy if someone wants to continue hacking the patch further and > > get it in a committable shape. I can stay actively involved. But T

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Jul 26, 2017 at 6:26 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Apr 18, 2017 at 4:25 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > I'll include the fix in the next set of patches. > > I haven't see a new set of patches. Are you i

Re: [HACKERS] logical replication and PANIC during shutdown checkpoint in publisher

e_prune_opt(). Do we need to worry about wal_log_hints or checksums producing WAL because of hint bit updates? While I haven't read the thread, I am assuming if HOT pruning can happen, surely hint bits can get set too. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Assertion failure in REL9_5_STABLE

On Wed, Apr 19, 2017 at 11:19 AM, Andrew Gierth <and...@tao11.riddles.org.uk > wrote: > >>>>> "Pavan" == Pavan Deolasee <pavan.deola...@gmail.com> writes: > > Pavan> I am attaching a patch that throws a similar ERROR during > Pavan> plannin

Re: [HACKERS] Assertion failure in REL9_5_STABLE

On Wed, Apr 19, 2017 at 11:19 AM, Andrew Gierth <and...@tao11.riddles.org.uk > wrote: > >>>>> "Pavan" == Pavan Deolasee <pavan.deola...@gmail.com> writes: > > Pavan> I am attaching a patch that throws a similar ERROR during > Pavan> plannin

[HACKERS] Assertion failure in REL9_5_STABLE

frame #19: 0x00010f3eb066 postgres`PostmasterMain(argc=3, argv=0x7fb151403740) + 5414 at postmaster.c:1306 frame #20: 0x00010f32bddf postgres`main(argc=3, argv=0x7fb151403740) + 751 at main.c:228 frame #21: 0x7fff9c0b1255 libdyld.dylib`start + 1 frame

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

ge, thus failing to make to the storage. Can you please test with the attached patch and confirm it works? I was able to reproduce the exact same assertion on my end and the patch seems to fix it. But an additional check won't harm. I'll include the fix in the next set of pat

Re: [HACKERS] PANIC in pg_commit_ts slru after crashes

again increment ShmemVariableCache->nextXid, when there are no prepared transactions in the system. I wonder if we should do as in attached patch. It's not entirely clear to me why only CommitTS fails and not other slrus. One possible reason could be that CommitTS is started before t

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 12, 2017 at 10:42 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Apr 11, 2017 at 1:20 PM, Pavan Deolasee > > > 5. Added code to set a CLEAR pointer to a WARM pointer when we know that > the > > entire chain is WARM. This should address the wor

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

e insert (if everything matches) or set a PREWARM flag on the old pointer, and insert the new tuple with POSTWARM flag. Searching for old index pointer will be non-starter for non-unique indexes, unless they are also sorted by TID, something that Claudio's patch does. What I am not sur

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 12, 2017 at 9:23 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Tue, Apr 11, 2017 at 10:50 PM, Pavan Deolasee > > > > > And I fixed them as quickly as humanly possible. > > > > Yes, you have responded to them quickly, but I didn't get

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

is doing additional work, without providing any benefit, may be you can still find regression. I am willing to fix them as long as they are fixable and we are comfortable with the additional code complexity. IMHO certain trade-offs are good, but I understand that not everybody will agree with my

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

nough time reviewing other patches. I know its critical, and I'll try to improve on that. Congratulations to all whose work got accepted and many thanks to all reviewers/committers/CF managers. I know how difficult and thankless that work is. Thanks, Pavan -- Pavan Deolasee ht

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

ce is between spending considerable amount of time trying to handle every case vs doing it incrementally and start delivering to majority of the users, yet keeping the patch at a manageable level. Even if we were to provide table level option, my preference would be keep it ON by default. Thanks

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Apr 6, 2017 at 1:06 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Apr 5, 2017 at 2:32 PM, Pavan Deolasee <pavan.deola...@gmail.com> > wrote: > > > The other way is to pass old tuple values along with the new tuple > values to > > amwarmin

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 5, 2017 at 8:42 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Apr 4, 2017 at 10:21 PM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > On Thu, Mar 30, 2017 at 7:55 PM, Robert Haas <robertmh...@gmail.com> > wrote: > >> bu

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

and then access TOAST to handle that. Would that work? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

ly the code that needs to be written. > > > > Yes, but once it is written it will take years before those bits can be > > used on most installations. > > Actually, the 2 bits from old-style VACUUM FULL bits could be reused if > one of the WARM bits woul

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

the same tuple is rechecked often. But if a tuple is rechecked only once then may be other overheads will kick-in, thus reducing the regression significantly. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 11:16 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Fri, Mar 31, 2017 at 10:24 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > Having worked on it for some time now, I can say that WARM uses pretty > much > > the

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

les and I know no known bugs, but if we don't want to risk system tables, we might want to turn it off (just prior to release may be). Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 7:27 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Mar 30, 2017 at 6:37 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > I think we can fix that by comparing compressed values. I know you had > > raised concerns, but Rob

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

multiple clients? Ok. Might become hard to control HOT behaviour though. Or will need to do mix of WARM/HOT updates. Will see if this is something easily doable by setting high FF etc. > For > your information, I am also trying to setup some tests along with one > of my colleague a

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 4:42 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > > > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila <amit.kapil...@gmail.com> > > wrote:

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila <amit.kapil...@gmail.com> > wrote: > > On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee > > <pavan.deola...@gmail.com> wrote: >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 8:34 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Mon, Mar 27, 2017 at 10:25 PM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > On Tue, Mar 28, 2017 at 1:59 AM, Robert Haas <robertmh...@gmail.com> > wrote: > >> On Th

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

DatumGetPointer(untoasted_values[i]))" check should prevent that. But I could be reading those macros wrong. They are probably heavily uncommented and it's not clear what each of those VARATT_* macro do. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

ven that it confused you, I think I should rewrite those comments. Will do. > 3. > + * HCWC_WARM_UPDATED_TUPLE - a tuple with HEAP_WARM_UPDATED is found > somewhere > + *in the chain. Note that when a tuple is WARM > + *updated, both old and new versions are marked > + * with th

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

r if toasted heap values are the only remaining problem in this area. So heap_tuple_attr_equals() should first detoast the heap values and then do the comparison. I already have a test case that fails for this reason, so let me try this approach. Thanks, Pavan -- Pavan Deolasee http://www

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

ed normally ok 6 - psql exited normally ok 7 - psql exited normally ok 8 - psql exited normally ok 9 - psql exited normally ok 10 - Fine match ok All tests successful. Files=2, Tests=12, 22 wallclock secs ( 0.03 usr 0.00 sys + 7.94 cusr 2.41 csys = 10.38 CPU) Result: PASS make: Leaving directory '/

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 4:45 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Sat, Mar 25, 2017 at 1:24 PM, Amit Kapila <amit.kapil...@gmail.com> > wrote: > > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > > <pavan.deola...@gmail.com> wrote: >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 7:49 AM, Bruce Momjian <br...@momjian.us> wrote: > On Mon, Mar 27, 2017 at 04:29:56PM -0400, Robert Haas wrote: > > On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee > > <pavan.deola...@gmail.com> wrote: > > > It's quite hard to say th

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 1:59 AM, Robert Haas <robertmh...@gmail.com> wrote: > On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > It's quite hard to say that until we see many more benchmarks. As author > of > > the patch, I

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

one) tests currently using that in Postgres. I've run these tests on OSX, will try on some linux platform too. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

[HACKERS] Stale comments in vacuumlazy.c

I happened to notice a stale comment at the very beginning of vacuumlazy.c. ISTM we forgot to fix that when we introduced FSM. With FSM, vacuum no longer needed to track per-page free space info. I propose attached fix. Thanks, Pavan -- Pavan Deolasee http://www

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

heap binary values (I think). To decide whether to do WARM update or not in heap_update we only rely on binary comparison. Could it happen that for two different binary heap values, we still compute the same index attribute? Even when expression indexes are not supported? Thanks, Pavan > -- > Peter Geo

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

ll return true for both the hash entries. That's a bummer as far as supporting WARM for hash indexes is concerned, unless we find a way to avoid duplicate index entries. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 24, 2017 at 4:04 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Fri, Mar 24, 2017 at 12:25 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > > > > > On Thu, Mar 23, 2017 at 7:53 PM, Amit Kapila <amit.kapil...@gmail.com> &

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 7:53 PM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Thu, Mar 23, 2017 at 3:44 PM, Pavan Deolasee > > > > > Yes, this is a very fair point. The way I proposed to address this > upthread > > is by introducing a set of threshold/scale

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 11:44 PM, Mithun Cy <mithun...@enterprisedb.com> wrote: > Hi Pavan, > On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB > &g

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 4:08 PM, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > > On Thu, Mar 23, 2017 at 3:02 PM, Amit Kapila <amit.kapil...@gmail.com> > wrote: > >> >> >> That sounds like you are dodging the actual problem. I mean you can

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

s than master, especially when there are more than 1 indexes on the table, and sometimes those checks will go waste. I am ok if we want to provide table-specific knob to disable WARM, but not sure if others would like that idea. Thanks, Pavan -- Pavan Deolasee http://ww

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Thanks Amit. v19 addresses some of the comments below. On Thu, Mar 23, 2017 at 10:28 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila <amit.kapil...@gmail.com> > wrote: > > On Tue, Mar 21, 2017 at 6:47 PM, Pavan D

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 22, 2017 at 4:53 PM, Mithun Cy <mithun...@enterprisedb.com> wrote: > On Wed, Mar 22, 2017 at 3:44 PM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > > > This looks quite weird to me. Obviously these numbers are completely > > non-comparable

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

lame it on AWS instance completely, but the pattern in your tests looks very similar where the number slowly and steadily keeps going up. If you do complete retest but run v18/v19 first and then run master, may be we'll see a complete opposite picture? Thanks, Pavan -- Pavan Deolasee

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 22, 2017 at 8:43 AM, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > > BTW may I request another test with the attached patch? In this patch, we > check if the PageIsFull() even before deciding which attributes to check > for modification. If the

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

uple is smaller than the tuple used in previous UPDATE and we have just enough space to do HOT update this time, but I can think that's too narrow). Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services 0001_

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 21, 2017 at 10:34 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Tue, Mar 21, 2017 at 12:49 PM, Bruce Momjian <br...@momjian.us> wrote: > > On Tue, Mar 21, 2017 at 09:25:49AM -0400, Robert Haas wrote: > >> On Tue, Mar 21, 2017 at 8:41 AM, Pavan

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

> case, Hmm. I am kinda surprised you say that because I never thought it was a narrow test case that we are targeting here. But may be I'm wrong. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

at the second idea would require, is that it can > easily mask bugs. Agree. That's probably one reason why Alvaro wrote the patch to start with. I'll give the first of those two options a try. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

HOT nor WARM is possible. In this case since there is just one index and as soon as we check the second column we know neither HOT nor WARM is possible, we will return early. It might complicate the API a lot, but I can give it a shot if that's what is needed to make progress. Any other ideas?

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 14, 2017 at 7:17 AM, Alvaro Herrera wrote: > > @@ -234,6 +236,21 @@ index_beginscan(Relation heapRelation, > > scan->heapRelation = heapRelation; > > scan->xs_snapshot = snapshot; > > > > + /* > > + * If the index supports recheck, make

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 15, 2017 at 12:46 AM, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Pavan Deolasee wrote: > > On Tue, Mar 14, 2017 at 7:17 AM, Alvaro Herrera < > alvhe...@2ndquadrant.com> > > wrote: > > > > I have already commented a

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 20, 2017 at 8:11 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Sun, Mar 19, 2017 at 3:05 AM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > On Thu, Mar 16, 2017 at 12:53 PM, Robert Haas <robertmh...@gmail.com> > wrote: > > >>

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 16, 2017 at 12:53 PM, Robert Haas <robertmh...@gmail.com> wrote: > On Wed, Mar 15, 2017 at 3:44 PM, Pavan Deolasee > <pavan.deola...@gmail.com> wrote: > > I couldn't find a better way without a lot of complex infrastructure. > Even > > though we now ha

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 14, 2017 at 7:16 PM, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Pavan Deolasee wrote: > > On Tue, Mar 14, 2017 at 7:17 AM, Alvaro Herrera < > alvhe...@2ndquadrant.com> > > wrote: > > > > I have already commented about the executor invo

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 14, 2017 at 7:19 PM, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Pavan Deolasee wrote: > > > BTW I wanted to share some more numbers from a recent performance test. I > > thought it's important because the latest patch has fully functional > chain

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

instead invent something similar to IndexRecheck(), but instead of running ExecQual(), this new routine will compare the index values by the given HeapTuple against given IndexTuple. ISTM that for this to work we'll need to modify all callers of index_getnext() and teach them to invoke the AM specific recheck method if xs_tuple_recheck flag is set to true by index_getnext(). Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Skip all-visible pages during second HeapScan of CIC

tched CIC (or any other operation which may not like concurrent VM set) is in progress. We could use what Stephen suggested upthread to find that state. But right now it's hard to think because there is nothing on either side so we don't know what gets impacted by aggressive VM set and how. Thanks, P

Re: [HACKERS] Skip all-visible pages during second HeapScan of CIC

On Wed, Mar 8, 2017 at 7:33 AM, Robert Haas wrote: > On Tue, Mar 7, 2017 at 4:26 PM, Stephen Frost wrote: > > Right, that's what I thought he was getting at and my general thinking > > was that we would need a way to discover if a CIC is ongoing on the

Re: [HACKERS] Cleanup: avoid direct use of ip_posid/ip_blkid

On Thu, Mar 2, 2017 at 9:55 PM, Peter Eisentraut < peter.eisentr...@2ndquadrant.com> wrote: > On 2/22/17 08:38, Pavan Deolasee wrote: > > One reason why these macros are not always used is because they > > typically do assert-validation to ensure ip_posid has a valid value

[HACKERS] Skip all-visible pages during second HeapScan of CIC

which conflicts with CIC on the relation lock, I don't see any risk of incorrectly skipping pages that the second scan should have scanned. Comments? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Serv

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

ned the idea of overloading ip_posid bits a few times now and haven't heard any objection so far. Well, that could either mean that nobody has read those emails seriously or there is general acceptance to that idea.. I am assuming latter :-)) Thanks, Pavan -- Pavan Deolasee http://www.2nd

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

m the page. Similarly, we will convert all WARM chains to HOT chains and then check for all-visibility of the page. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

them all-visible. That's why when a tuple is WARM updated, we carry that information in the subsequent versions even when later updates are HOT updates. The chain conversion algorithm will handle this by clearing those bits and thus allowing index-only scans again. Thanks, Pavan -- Pavan Deol

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

ch will be the case typically, but the algorithm can deal with the case where there are multiple tuples but with matching index keys. Hope this helps. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

tch gives you no benefit. It might be worthwhile to see if patch causes any regression in these scenarios, though I think it will be minimal or zero. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

every* index and half updates insert new index entry *only* in affected index. That itself does a good bit for performance. So to answer your question: yes, joining two HOT chains via WARM is much cheaper because it results in creating new index entries just for affected indexes. Thanks, Pavan --

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Feb 23, 2017 at 11:30 PM, Bruce Momjian <br...@momjian.us> wrote: > On Wed, Feb 1, 2017 at 10:46:45AM +0530, Pavan Deolasee wrote: > > > contains a WARM tuple. Alternate ideas/suggestions and review of > the > > design > > > are welcome!

[HACKERS] Cleanup: avoid direct use of ip_posid/ip_blkid

with --enable-tap-tests passes. Comments/suggestions? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services remove_ip_posid_blkid_ref_v3.patch Description: Binary data -- Sent via pgsql-hackers mailing

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

tests with data validation and haven't found any new issues with the most recent patches. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

index-only scan on a table which is being concurrently updated. So not sure even that will catch every possible case. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [COMMITTERS] pgsql: Release note updates.

committed-good transactions. It's all theory and I haven't had time to try this out. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] [COMMITTERS] pgsql: Release note updates.

ewval" may not return any result, "SELECT * FROM tab WHERE key = oldval" may actually return the updated (and wrong) tuple. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

de the push, and barring negative reports from the buildfarm, > it will be in today's releases. > Thank you for taking care of it. Buildfarm is looking green until now. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

On Mon, Feb 6, 2017 at 8:01 AM, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > >> > I like this approach. I'll run the patch on a build with > CACHE_CLOBBER_ALWAYS, but I'm pretty sure it will be ok. > While it looks certain that the fix will miss this point releas

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

omething. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

least some sort of band-aid fix which helps the situation, yet have minimal side-effects, is warranted. I believe proposed retry patch does exactly that. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

On Mon, Feb 6, 2017 at 8:15 AM, Amit Kapila <amit.kapil...@gmail.com> wrote: > On Mon, Feb 6, 2017 at 8:01 AM, Pavan Deolasee <pavan.deola...@gmail.com> > wrote: > > > > > > On Mon, Feb 6, 2017 at 1:44 AM, Tom Lane <t...@sss.pgh.pa.us> wrote: > >>

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

gger machine, with many users (on a decent AWS machine, I would find every third CIC to be corrupt). Moreover the setup is not doing anything extraordinary. Just concurrent updates which change between HOT to non-HOT and a CIC. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

OBBER_ALWAYS, but I'm pretty sure it will be ok. I also like the fact that retry logic is now limited to only when index set changes, which fits in the situation we're dealing with. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

like these? Or do you think it will be hard to come up with a proper fix for the issue and it will need some serious refactoring? Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

DEX CONCURRENTLY so that at least the first phase which add the index entry to the catalog happens with a strong lock. This will lead to momentary blocking of write queries, but protect us from the race. I'm assuming this is the only code that can cause the problem, and I haven't checked other

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

that it has a stable set of indexes. This also makes it safe 4750 * (deadlock-free) for us to take locks on the relation's indexes. I've attached a revised patch based on these thoughts. It looks a bit more invasive than the earlier one-liner, but looks better to me. Thanks, Pavan -- Pavan Deolase

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

and replica identity) > + */ > > + relation->rd_indexattr = NULL; > + > > I think setting directly to NULL will leak the memory. > Good catch, thanks. Will fix. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

On Mon, Jan 30, 2017 at 7:20 PM, Pavan Deolasee <pavan.deola...@gmail.com> wrote: > > Based on my investigation so far and the evidence that I've collected, > what probably happens is that after a relcache invalidation arrives at the > new transaction, it recomputes the rd_i

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

of simple_heap_foo plus CatalogIndexInsert > >> in the places where this optimization had been applied. > > > This looks reasonable enough to me. > > Done. > > Thanks for taking care of this. Shame that I missed this because I'd specifically noted the special casing

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

nism in place to clear those bits. So may be we add something to do that. I'd some other ideas (and a patch too) to reuse bits from t_ctid.ip_pos given that offset numbers can be represented in just 13 bits, even with the maximum block size. Can look at that if it comes to finding more bits. Thanks, Pavan

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Jan 31, 2017 at 7:37 PM, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Pavan Deolasee wrote: > > > > > Sounds good. Should I submit that as a separate patch on current master? > > Yes, please. > > Attached. Two new APIs added. -

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Jan 31, 2017 at 7:21 PM, Alvaro Herrera <alvhe...@2ndquadrant.com> wrote: > Pavan Deolasee wrote: > > On Thu, Jan 26, 2017 at 2:38 AM, Alvaro Herrera < > alvhe...@2ndquadrant.com> > > wrote: > > > > The simple_heap_update + CatalogUpdateIndexe

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

ch_buffer() and heap_hot_search() should return > a tri-valued enum instead of boolean; that idea looks reasonable in > theory but callers have to do more work afterwards, so maybe not. > > Ok. I'll try to rearrange it a bit. May be we just have one API after all? There are only a very few callers of these APIs. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

RelationPutHeapTuple(relation, buffer, heaptup, > false, > > + _offnum); > > becomes > > root_offnum = RelationPutHeapTuple(relation, buffer, heaptup, > false, > InvalidOffsetNumber); &g

[HACKERS] Index corruption with CREATE INDEX CONCURRENTLY

[xid=4670908]LOG: heap_update (cic_tab_accounts), hot_attrs ((b 9)) Other session, which I included for comparison, sees the new index and recomputes its rd_indexattr in time and hence that update works as expected. Thanks, Pavan -- Pavan Deolasee http://www.2ndQuadrant.com/

Re: [HACKERS] Failure in commit_ts tap tests

On Sat, Jan 21, 2017 at 9:39 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > Pavan Deolasee <pavan.deola...@gmail.com> writes: > > On Sat, Jan 21, 2017 at 9:09 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > >> Hm, but what of the "null" value? Also, I get >

  1   2   3   4   5   6   7   8   >