Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

> On 28 Jul 2017, at 16:46, Robert Haas wrote: > > On Fri, Jul 28, 2017 at 12:39 AM, Pavan Deolasee > wrote: >> I see your point. But I would like to think this way: does the technology >> significantly help many common use cases, that are currently not addressed >> by HOT? It probably won't hel

Re: In-place index updates and HOT (Was: [HACKERS] Patch: Write Amplification Reduction Method (WARM))

Claudio Freire wrote: README.HOT says that that cost is not worth the benefit of preventing a new index write, but I think that it ought to take into account that not all index writes are equal. There is an appreciable difference between inserting a new tuple, and updating one in-place. We can r

Re: In-place index updates and HOT (Was: [HACKERS] Patch: Write Amplification Reduction Method (WARM))

On Fri, Jul 28, 2017 at 8:32 PM, Peter Geoghegan wrote: > README.HOT says that that cost is not worth the benefit of > preventing a new index write, but I think that it ought to take into > account that not all index writes are equal. There is an appreciable > difference between inserting a new tu

In-place index updates and HOT (Was: [HACKERS] Patch: Write Amplification Reduction Method (WARM))

Pavan Deolasee wrote: One good thing is that the patch is ready and fully functional. So that allows those who are keen to run real performance tests and see the actual impact of the patch. Very true. I see your point. But I would like to think this way: does the technology significantly hel

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Jul 28, 2017 at 12:39 AM, Pavan Deolasee wrote: > I see your point. But I would like to think this way: does the technology > significantly help many common use cases, that are currently not addressed > by HOT? It probably won't help all workloads, that's given. Also, we don't > have any c

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Jul 28, 2017 at 5:57 AM, Peter Geoghegan wrote: > Pavan Deolasee wrote: > > I'll be happy if someone wants to continue hacking the patch further and > > get it in a committable shape. I can stay actively involved. But TBH the > > amount of time I can invest is far as compared to what I c

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Pavan Deolasee wrote: > I'll be happy if someone wants to continue hacking the patch further and > get it in a committable shape. I can stay actively involved. But TBH the > amount of time I can invest is far as compared to what I could during the > last cycle. That's disappointing. I personally

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Jul 26, 2017 at 6:26 PM, Robert Haas wrote: > On Tue, Apr 18, 2017 at 4:25 AM, Pavan Deolasee > wrote: > > I'll include the fix in the next set of patches. > > I haven't see a new set of patches. Are you intending to continue > working on this? > > Looks like I'll be short on bandwidth

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 18, 2017 at 4:25 AM, Pavan Deolasee wrote: > I'll include the fix in the next set of patches. I haven't see a new set of patches. Are you intending to continue working on this? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Apr 14, 2017 at 9:21 PM, Jaime Casanova wrote: > > > Hi Pavan, > > I run a test on current warm patchset, i used pgbench with a scale of > 20 and a fillfactor of 90 and then start the pgbench run with 6 > clients in parallel i also run sqlsmith on it. > > And i got a core dump after somet

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On 5 April 2017 at 13:32, Pavan Deolasee wrote: > > Ok. I've extensively updated the README to match the current state of > affairs. Updated patch set attached. Hi Pavan, I run a test on current warm patchset, i used pgbench with a scale of 20 and a fillfactor of 90 and then start the pgbench ru

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 12, 2017 at 10:42 PM, Robert Haas wrote: > On Tue, Apr 11, 2017 at 1:20 PM, Pavan Deolasee > > > 5. Added code to set a CLEAR pointer to a WARM pointer when we know that > the > > entire chain is WARM. This should address the workload Dilip ran and > found > > regression (I don't th

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Apr 13, 2017 at 2:04 AM, Peter Geoghegan wrote: > On Wed, Apr 12, 2017 at 10:12 AM, Robert Haas > wrote: > >> I may have missed something, but there is no intention to ignore known > >> regressions/reviews. Of course, I don't think that every regression > will be > >> solvable, like if y

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 12, 2017 at 10:12 AM, Robert Haas wrote: >> I may have missed something, but there is no intention to ignore known >> regressions/reviews. Of course, I don't think that every regression will be >> solvable, like if you run a CPU-bound workload, setting it up in a way such >> that you r

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 11, 2017 at 1:20 PM, Pavan Deolasee wrote: > I don't know why you say that regressions are not addressed. Here are a few > things I did to address the regressions/reviews/concerns, apart from fixing > all the bugs discovered, but please let me know if there are things I've not > addres

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 12, 2017 at 9:23 AM, Amit Kapila wrote: > On Tue, Apr 11, 2017 at 10:50 PM, Pavan Deolasee > > > > > And I fixed them as quickly as humanly possible. > > > > Yes, you have responded to them quickly, but I didn't get a chance to > re-verify all of those. However, I think the main poin

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 11, 2017 at 10:50 PM, Pavan Deolasee wrote: > > > On Tue, Apr 11, 2017 at 7:10 PM, Robert Haas wrote: >> >> >> >> Yes, and as Andres says, you don't help with those, and then you're >> upset when your own patch doesn't get attention. > > > I am not upset, I was obviously a bit disappo

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Apr 10, 2017 at 04:34:50PM -0700, Andres Freund wrote: > Hi, > > > On 2017-04-08 23:36:13 +0530, Pavan Deolasee wrote: > > On Wed, Apr 5, 2017 at 11:57 PM, Andres Freund wrote: > > > > > On 2017-04-05 09:36:47 -0400, Robert Haas wrote: > > > > By the way, the "Converting WARM chains bac

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 11, 2017 at 7:10 PM, Robert Haas wrote: > > > Yes, and as Andres says, you don't help with those, and then you're > upset when your own patch doesn't get attention. I am not upset, I was obviously a bit disappointed which I think is a very natural emotion after spending weeks on it.

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, Apr 8, 2017 at 2:06 PM, Pavan Deolasee wrote: > Thank you all for the reviews, feedback, tests, criticism. And apologies > for keep pushing it till the last minute even though it was clear to me > quite some time back the patch is not going to make it. But if I'd given up, > it would have

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Hi, On 2017-04-08 23:36:13 +0530, Pavan Deolasee wrote: > On Wed, Apr 5, 2017 at 11:57 PM, Andres Freund wrote: > > > On 2017-04-05 09:36:47 -0400, Robert Haas wrote: > > > By the way, the "Converting WARM chains back to HOT chains" section of > > > README.WARM seems to be out of date. Any cha

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, Apr 8, 2017 at 11:36:13PM +0530, Pavan Deolasee wrote: > Thank you all for the  reviews, feedback, tests, criticism. And apologies for > keep pushing it till the last minute even though it was clear to me quite some > time back the patch is not going to make it. But if I'd given up, it wou

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 5, 2017 at 11:57 PM, Andres Freund wrote: > On 2017-04-05 09:36:47 -0400, Robert Haas wrote: > > By the way, the "Converting WARM chains back to HOT chains" section of > > README.WARM seems to be out of date. Any chance you could update that > > to reflect the current state and think

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Apr 6, 2017 at 12:20 AM, Peter Geoghegan wrote: > On Wed, Apr 5, 2017 at 11:27 AM, Andres Freund wrote: > > I propose we move this patch to the next CF. > > I agree. I think it's too late to be working out fine details around > TOAST like this. This is a patch that touches the storage fo

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Apr 6, 2017 at 1:06 AM, Robert Haas wrote: > On Wed, Apr 5, 2017 at 2:32 PM, Pavan Deolasee > wrote: > > > The other way is to pass old tuple values along with the new tuple > values to > > amwarminsert, build index tuples and then do a comparison. For duplicate > > index tuples, skip WA

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 5, 2017 at 2:32 PM, Pavan Deolasee wrote: >> The only other idea that I have for a really clean solution here is to >> support this only for index types that are amcanreturn, and actually >> compare the value stored in the index tuple with the one stored in the >> heap tuple, ensuring

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 5, 2017 at 11:27 AM, Andres Freund wrote: > I propose we move this patch to the next CF. I agree. I think it's too late to be working out fine details around TOAST like this. This is a patch that touches the storage format in a fairly fundamental way. The idea of turning WARM on or o

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On 2017-04-05 09:36:47 -0400, Robert Haas wrote: > By the way, the "Converting WARM chains back to HOT chains" section of > README.WARM seems to be out of date. Any chance you could update that > to reflect the current state and thinking of the patch? I propose we move this patch to the next CF.

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 4, 2017 at 11:43 PM, Pavan Deolasee wrote: > Well, better than causing a deadlock ;-) Yep. > Lets see if we want to go down the path of blocking WARM when tuples have > toasted attributes. I submitted a patch yesterday, but having slept over it, > I think I made mistakes there. It mi

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Apr 5, 2017 at 8:42 AM, Robert Haas wrote: > On Tue, Apr 4, 2017 at 10:21 PM, Pavan Deolasee > wrote: > > On Thu, Mar 30, 2017 at 7:55 PM, Robert Haas > wrote: > >> but > >> try to access the TOAST table would be fatal; that probably would have > >> deadlock hazards among other problem

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Apr 4, 2017 at 10:21 PM, Pavan Deolasee wrote: > On Thu, Mar 30, 2017 at 7:55 PM, Robert Haas wrote: >> but >> try to access the TOAST table would be fatal; that probably would have >> deadlock hazards among other problems. > > Hmm. I think you're right. We could make a copy of the heap

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 7:55 PM, Robert Haas wrote: > but > try to access the TOAST table would be fatal; that probably would have > deadlock hazards among other problems. Hmm. I think you're right. We could make a copy of the heap tuple, drop the lock and then access TOAST to handle that. Wou

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 11:17 PM, Bruce Momjian wrote: > On Tue, Mar 21, 2017 at 04:04:58PM -0400, Bruce Momjian wrote: > > On Tue, Mar 21, 2017 at 04:56:16PM -0300, Alvaro Herrera wrote: > > > Bruce Momjian wrote: > > > > On Tue, Mar 21, 2017 at 04:43:58PM -0300, Alvaro Herrera wrote: > > > > >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 12:31 PM, Dilip Kumar wrote: > On Thu, Mar 30, 2017 at 5:27 PM, Amit Kapila > wrote: > > I am not sure if we can consider it as completely synthetic because we > > might see some similar cases for json datatypes. Can we once try to > > see the impact when the same test r

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 11:54 PM, Pavan Deolasee wrote: > > On Fri, Mar 31, 2017 at 11:16 PM, Robert Haas wrote: >> >> Now, I understand you to be suggesting a flag at >> table-creation time that would, maybe, be immutable after that, but >> even then - are we going to run completely unmodified

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 4:13 AM, Pavan Deolasee wrote: > > > On Thu, Mar 30, 2017 at 3:29 PM, Dilip Kumar > wrote: > >> On Wed, Mar 29, 2017 at 11:51 AM, Pavan Deolasee >> wrote: >> > Thanks. I think your patch of tracking interesting attributes seems ok >> too >> > after the performance issue

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 11:16 PM, Robert Haas wrote: > On Fri, Mar 31, 2017 at 10:24 AM, Pavan Deolasee > wrote: > > Having worked on it for some time now, I can say that WARM uses pretty > much > > the same infrastructure that HOT uses for cleanup/pruning tuples from the > > heap. So the risk o

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 10:24 AM, Pavan Deolasee wrote: > Having worked on it for some time now, I can say that WARM uses pretty much > the same infrastructure that HOT uses for cleanup/pruning tuples from the > heap. So the risk of having a bug which can eat your data from the heap is > very low.

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 10:49 AM, Petr Jelinek wrote: > While reading this thread I am thinking if we could just not do WARM on > TOAST and compressed values if we know there might be regressions there. > I mean I've seen the problem WARM tries to solve mostly on timestamp or > boolean values and

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 6:47 PM, Robert Haas wrote: > > > 2. WARM is a non-optional feature which touches the on-disk format. > There is nothing more dangerous than that. If hash indexes have bugs, > people can avoid those bugs by not using them; there are good reasons > to suppose that hash inde

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 31, 2017 at 7:53 AM, Simon Riggs wrote: > So Andres says defer this, but Robert says "more review", which is > more than just deferral. > > We have some risky things in this release such as Hash Indexes, > function changes. I perfectly understand that perception of risk is > affected s

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On 30 March 2017 at 16:50, Robert Haas wrote: > On Thu, Mar 30, 2017 at 11:41 AM, Andres Freund wrote: >> On 2017-03-30 16:43:41 +0530, Pavan Deolasee wrote: >>> Looks like OID conflict to me.. Please try rebased set. >> >> Pavan, Alvaro, everyone: I know you guys are working very hard on this, >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 5:27 PM, Amit Kapila wrote: > I am not sure if we can consider it as completely synthetic because we > might see some similar cases for json datatypes. Can we once try to > see the impact when the same test runs from multiple clients? For > your information, I am also try

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 21, 2017 at 04:04:58PM -0400, Bruce Momjian wrote: > On Tue, Mar 21, 2017 at 04:56:16PM -0300, Alvaro Herrera wrote: > > Bruce Momjian wrote: > > > On Tue, Mar 21, 2017 at 04:43:58PM -0300, Alvaro Herrera wrote: > > > > Bruce Momjian wrote: > > > > > > > > > I don't think it makes sens

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 11:41 AM, Andres Freund wrote: > On 2017-03-30 16:43:41 +0530, Pavan Deolasee wrote: >> Looks like OID conflict to me.. Please try rebased set. > > Pavan, Alvaro, everyone: I know you guys are working very hard on this, > but I think at this point it's too late to commit th

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Hi, On 2017-03-30 16:43:41 +0530, Pavan Deolasee wrote: > Looks like OID conflict to me.. Please try rebased set. Pavan, Alvaro, everyone: I know you guys are working very hard on this, but I think at this point it's too late to commit this for v10. This is patch that's affecting the on-disk for

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On 30/03/17 16:04, Pavan Deolasee wrote: > > > On Thu, Mar 30, 2017 at 7:27 PM, Robert Haas > wrote: > > On Thu, Mar 30, 2017 at 6:37 AM, Pavan Deolasee > mailto:pavan.deola...@gmail.com>> wrote: > > I think we can fix that by comparing compressed value

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 10:08 AM, Amit Kapila wrote: > I think we should not consider doing compression and decompression as > free at this point in code, because we hold a buffer lock during > recheck. Buffer locks are meant for short-term locks (it is even > mentioned in storage/buffer/README),

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 5:55 PM, Pavan Deolasee wrote: > > On Thu, Mar 30, 2017 at 5:27 PM, Amit Kapila > wrote: >> >> >> >> How have you verified that? Have you checked that in >> heap_prepare_insert it has called toast_insert_or_update() and then >> returned a tuple different from what the inp

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 7:27 PM, Robert Haas wrote: > On Thu, Mar 30, 2017 at 6:37 AM, Pavan Deolasee > wrote: > > I think we can fix that by comparing compressed values. I know you had > > raised concerns, but Robert confirmed that (IIUC) it's not a problem > today. > > I'm not sure that's an

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 6:37 AM, Pavan Deolasee wrote: > I think we can fix that by comparing compressed values. I know you had > raised concerns, but Robert confirmed that (IIUC) it's not a problem today. I'm not sure that's an entirely fair interpretation of what I said. My point was that, whi

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 5:27 PM, Amit Kapila wrote: > > > How have you verified that? Have you checked that in > heap_prepare_insert it has called toast_insert_or_update() and then > returned a tuple different from what the input tup is? Basically, I > am easily able to see it and even the reas

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 30, 2017 at 4:07 PM, Pavan Deolasee wrote: > > On Wed, Mar 29, 2017 at 4:42 PM, Amit Kapila > wrote: >> >> On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee >> wrote: >> > >> > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila >> > wrote: >> >> >> >> On Wed, Mar 29, 2017 at 11:52 AM, Amit

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 4:42 PM, Amit Kapila wrote: > On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee > wrote: > > > > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila > > wrote: > >> > >> On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila > >> wrote: > > > > Then during recheck, we pass already compr

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 11:51 AM, Pavan Deolasee wrote: > Thanks. I think your patch of tracking interesting attributes seems ok too > after the performance issue was addressed. Even though we can still improve > that further, at least Mithun confirmed that there is no significant > regression any

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Pavan Deolasee wrote: > On Wed, Mar 29, 2017 at 3:42 AM, Alvaro Herrera > wrote: > > > I pushed 0002 after some makeup, since it's just cosmetic and not > > controversial. > > Thanks. I think your patch of tracking interesting attributes seems ok too > after the performance issue was addressed.

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 7:12 AM, Amit Kapila wrote: > No as I agreed above, it won't double-compress, but still looks > slightly risky to rely on different set of values passed to > index_form_tuple and then compare them. It assumes that the compressor is completely deterministic, which I'm fairl

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 1:10 PM, Pavan Deolasee wrote: > > On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila > wrote: >> >> On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila >> wrote: > > Then during recheck, we pass already compressed values to > index_form_tuple(). But my point is, the following code

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 12:02 PM, Amit Kapila wrote: > On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila > wrote: > > On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee > > wrote: > >> > >> On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila > >> wrote: > >>> > >>> For such an heap insert, we will pass >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 10:31 PM, Pavan Deolasee wrote: > > On Tue, Mar 28, 2017 at 4:05 PM, Amit Kapila > wrote: >> >> As asked previously, can you explain me on what basis are you >> considering it robust? The comments on top of datumIsEqual() clearly >> indicates the danger of using it for to

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 29, 2017 at 11:52 AM, Amit Kapila wrote: > On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee > wrote: >> >> On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila >> wrote: >>> >>> For such an heap insert, we will pass >>> the actual value of column to index_form_tuple during index insert. >>

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 10:35 PM, Pavan Deolasee wrote: > > On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila > wrote: >> >> >> >> For such an heap insert, we will pass >> the actual value of column to index_form_tuple during index insert. >> However during recheck when we fetch the value of c2 from

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

I pushed 0002 after some makeup, since it's just cosmetic and not controversial. Here's 0003 rebased on top of it. (Also, I took out the gin and gist changes: it would be wrong to change that unconditionally, because the 0x pattern appears in indexes that would be pg_upgraded. We need a diff

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 8:34 PM, Robert Haas wrote: > On Mon, Mar 27, 2017 at 10:25 PM, Pavan Deolasee > wrote: > > On Tue, Mar 28, 2017 at 1:59 AM, Robert Haas > wrote: > >> On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee > >> wrote: > >> > It's quite hard to say that until we see many more b

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 7:04 PM, Amit Kapila wrote: > > > For such an heap insert, we will pass > the actual value of column to index_form_tuple during index insert. > However during recheck when we fetch the value of c2 from heap tuple > and pass it index tuple, the value is already in compres

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 4:07 PM, Amit Kapila wrote: > > Noted few cosmetic issues in 0005_warm_updates_v21: > > 1. > pruneheap.c(939): warning C4098: 'heap_get_root_tuples' : 'void' > function returning a value > Thanks. Will fix. > > 2. > + * HCWC_WARM_UPDATED_TUPLE - a tuple with HEAP_WARM_

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 4:05 PM, Amit Kapila wrote: > > > As asked previously, can you explain me on what basis are you > considering it robust? The comments on top of datumIsEqual() clearly > indicates the danger of using it for toasted values (Also, it will > probably not give the answer you w

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Hi Pavan, On 3/28/17 11:04 AM, Robert Haas wrote: On Mon, Mar 27, 2017 at 10:25 PM, Pavan Deolasee wrote: On Tue, Mar 28, 2017 at 1:59 AM, Robert Haas wrote: On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee wrote: It's quite hard to say that until we see many more benchmarks. As author of t

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 10:25 PM, Pavan Deolasee wrote: > On Tue, Mar 28, 2017 at 1:59 AM, Robert Haas wrote: >> On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee >> wrote: >> > It's quite hard to say that until we see many more benchmarks. As author >> > of >> > the patch, I might have got repeti

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 4:05 PM, Amit Kapila wrote: > On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee > wrote: >> On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee >> wrote: >>> >>> >>> >>> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >>> wrote: I was worried for the case if

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee wrote: > > > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> >>> >>> >>> I was worried for the case if the index is created non-default >>> collation, will the datumIs

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 2:19 PM, Pavan Deolasee wrote: > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> >>> >>> >>> I was worried for the case if the index is created non-default >>> collation, will the datumIsEqua

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 1:32 AM, Alvaro Herrera wrote: > Is the WARM tap test suite supposed to work when applied without all the > other patches? I just tried applied that one and running "make check -C > src/test/modules", and it seems to hang after giving "ok 5" for > t/002_warm_stress.pl. (

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 4:45 PM, Amit Kapila wrote: > On Sat, Mar 25, 2017 at 1:24 PM, Amit Kapila > wrote: > > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > > wrote: > >> > >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila > >> wrote: > >>> > > > >> While looking at this problem, it occurr

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 08:04:34AM +0530, Pavan Deolasee wrote: > And I've answered it so many times by now :-)  LOL > Just to add more to what I just said in another email, note that HOT/WARM > chains are created when a new root line pointer is created in the heap (a line > pointer that has an i

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 7:49 AM, Bruce Momjian wrote: > On Mon, Mar 27, 2017 at 04:29:56PM -0400, Robert Haas wrote: > > On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee > > wrote: > > > It's quite hard to say that until we see many more benchmarks. As > author of > > > the patch, I might have go

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 1:59 AM, Robert Haas wrote: > On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee > wrote: > > It's quite hard to say that until we see many more benchmarks. As author > of > > the patch, I might have got repetitive with my benchmarks. But I've seen > > over 50% improvement i

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Mon, Mar 27, 2017 at 04:29:56PM -0400, Robert Haas wrote: > On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee > wrote: > > It's quite hard to say that until we see many more benchmarks. As author of > > the patch, I might have got repetitive with my benchmarks. But I've seen > > over 50% improvem

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Tue, Mar 28, 2017 at 1:32 AM, Alvaro Herrera wrote: > Is the WARM tap test suite supposed to work when applied without all the > other patches? I just tried applied that one and running "make check -C > src/test/modules", and it seems to hang after giving "ok 5" for > t/002_warm_stress.pl. (

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 2:47 PM, Pavan Deolasee wrote: > It's quite hard to say that until we see many more benchmarks. As author of > the patch, I might have got repetitive with my benchmarks. But I've seen > over 50% improvement in TPS even without chain conversion (6 indexes on a 12 > column ta

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Is the WARM tap test suite supposed to work when applied without all the other patches? I just tried applied that one and running "make check -C src/test/modules", and it seems to hang after giving "ok 5" for t/002_warm_stress.pl. (I had to add a Makefile too, attached.) -- Álvaro Herrera

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, Mar 25, 2017 at 1:24 PM, Amit Kapila wrote: > On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee > wrote: >> >> On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila >> wrote: >>> > >> While looking at this problem, it occurred to me that the assumptions made >> for hash indexes are also wrong :-(

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, Mar 25, 2017 at 11:24 PM, Pavan Deolasee wrote: > > On Sat, 25 Mar 2017 at 11:03 PM, Peter Geoghegan wrote: >> >> On Sat, Mar 25, 2017 at 12:54 AM, Amit Kapila >> wrote: >> > I am not sure how do you want to binary compare two datums, if you are >> > thinking datumIsEqual(), that won't w

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, 25 Mar 2017 at 11:03 PM, Peter Geoghegan wrote: > On Sat, Mar 25, 2017 at 12:54 AM, Amit Kapila > wrote: > > I am not sure how do you want to binary compare two datums, if you are > > thinking datumIsEqual(), that won't work. I think you need to use > > datatype specific compare functio

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Sat, Mar 25, 2017 at 12:54 AM, Amit Kapila wrote: > I am not sure how do you want to binary compare two datums, if you are > thinking datumIsEqual(), that won't work. I think you need to use > datatype specific compare function something like what we do in > _bt_compare(). How will that inter

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 24, 2017 at 11:49 PM, Pavan Deolasee wrote: > > > On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila > wrote: >> >> >> >> I was worried for the case if the index is created non-default >> collation, will the datumIsEqual() suffice the need. Now again >> thinking about it, I think it will b

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 24, 2017 at 6:46 PM, Amit Kapila wrote: > > > I was worried for the case if the index is created non-default > collation, will the datumIsEqual() suffice the need. Now again > thinking about it, I think it will because in the index tuple we are > storing the value as in heap tuple.

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 3:54 PM, Pavan Deolasee wrote: > Thanks Amit. v19 addresses some of the comments below. > > On Thu, Mar 23, 2017 at 10:28 AM, Amit Kapila > wrote: >> >> On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila >> wrote: >> > On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee >> > wrote

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 24, 2017 at 4:04 PM, Amit Kapila wrote: > On Fri, Mar 24, 2017 at 12:25 AM, Pavan Deolasee > wrote: > > > > > > On Thu, Mar 23, 2017 at 7:53 PM, Amit Kapila > > > > The general sense I've got > > here is that we're ok to push some work in background if it helps the > > real-time que

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Fri, Mar 24, 2017 at 12:25 AM, Pavan Deolasee wrote: > > > On Thu, Mar 23, 2017 at 7:53 PM, Amit Kapila > wrote: >> >> > >> >> I am not sure on what basis user can set such parameters, it will be >> quite difficult to tune those parameters. I think the point is >> whatever threshold we keep,

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 7:53 PM, Amit Kapila wrote: > On Thu, Mar 23, 2017 at 3:44 PM, Pavan Deolasee > > > > > Yes, this is a very fair point. The way I proposed to address this > upthread > > is by introducing a set of threshold/scale GUCs specific to WARM. So > users > > can control when to in

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 11:44 PM, Mithun Cy wrote: > Hi Pavan, > On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee > wrote: > > Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB > > RAM, attached SSD) and results are shown below. But I think it is > important > > to get ind

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Hi Pavan, On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee wrote: > Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB > RAM, attached SSD) and results are shown below. But I think it is important > to get independent validation from your side too, just to ensure I am not > m

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 3:44 PM, Pavan Deolasee wrote: > On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila > wrote: >> > >> 3. >> + /* >> + * HASH indexes compute a hash value of the key and store that in the >> + * index. So >> we must first obtain the hash of the value obtained from the >> + * heap

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 4:08 PM, Pavan Deolasee wrote: > > > On Thu, Mar 23, 2017 at 3:02 PM, Amit Kapila > wrote: > >> >> >> That sounds like you are dodging the actual problem. I mean you can >> put that same PageIsFull() check in master code as well and then you >> will most probably again s

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 3:02 PM, Amit Kapila wrote: > > > That sounds like you are dodging the actual problem. I mean you can > put that same PageIsFull() check in master code as well and then you > will most probably again see the same regression. Well I don't see it that way. There was a spe

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

Thanks Amit. v19 addresses some of the comments below. On Thu, Mar 23, 2017 at 10:28 AM, Amit Kapila wrote: > On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila > wrote: > > On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee > > wrote: > >> > >>> > >> > >> Please find attached rebased patches. > >> > >

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee wrote: > > Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB > RAM, attached SSD) and results are shown below. But I think it is important > to get independent validation from your side too, just to ensure I am not > making an

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Wed, Mar 22, 2017 at 4:06 PM, Amit Kapila wrote: > On Tue, Mar 21, 2017 at 6:47 PM, Pavan Deolasee > wrote: >> >>> >> >> Please find attached rebased patches. >> > > Few comments on 0005_warm_updates_v18.patch: > Few more comments on 0005_warm_updates_v18.patch: 1. @@ -234,6 +241,25 @@ index_

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

On Thu, Mar 23, 2017 at 12:19 AM, Pavan Deolasee wrote: > > > On Wed, Mar 22, 2017 at 4:53 PM, Mithun Cy > wrote: > Ok, no problem. I did some tests on AWS i2.xlarge instance (4 vCPU, 30GB > RAM, attached SSD) and results are shown below. But I think it is important > to get independent validatio

  1   2   3   >