Re: POC: Cleaning up orphaned files using undo logs

2019-09-15 Thread Thomas Munro
On Mon, Sep 16, 2019 at 5:27 AM Kuntal Ghosh wrote: > While testing zheap over undo apis, we've found the following > issues/scenarios that might need some fixes/discussions: Thanks! > 1. In UndoLogAllocateInRecovery, when we find the current log number > from the list of registered blocks, we

Re: POC: Cleaning up orphaned files using undo logs

2019-09-15 Thread Kuntal Ghosh
Hello Thomas, While testing zheap over undo apis, we've found the following issues/scenarios that might need some fixes/discussions: 1. In UndoLogAllocateInRecovery, when we find the current log number from the list of registered blocks, we don't check whether the block->in_use flag is true or

Re: POC: Cleaning up orphaned files using undo logs

2019-09-11 Thread Alvaro Herrera
On 2019-Sep-06, vignesh C wrote: > Hi Thomas, > > While testing one of the recovery scenarios I found one issue: > FailedAssertion("!(logno == context->recovery_logno) I marked this patch Waiting on Author. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development,

Re: POC: Cleaning up orphaned files using undo logs

2019-09-05 Thread vignesh C
Hi Thomas, While testing one of the recovery scenarios I found one issue: FailedAssertion("!(logno == context->recovery_logno) The details of the same is mentioned below: The context's try_location was not updated in UndoLogAllocateInRecovery, in PrepareUndoInsert the try_location was updated

Re: POC: Cleaning up orphaned files using undo logs

2019-09-01 Thread Thomas Munro
On Fri, Aug 30, 2019 at 8:27 PM Kuntal Ghosh wrote: > I'm getting the following assert failure while performing the recovery > with the same. > "TRAP: FailedAssertion("slot->meta.status == UNDO_LOG_STATUS_FULL", > File: "undolog.c", Line: 997)" > > I found that we don't emit an WAL record when we

Re: POC: Cleaning up orphaned files using undo logs

2019-08-30 Thread Kuntal Ghosh
Hello Thomas, I was doing some testing for the scenario where the undo written by a transaction overflows to multiple undo logs. For that I've modified the following macro: #define UndoLogMaxSize (1024 * 1024) /* 1MB undo log size */ (I should have used the provided pg_force_switch_undo though..)

Re: POC: Cleaning up orphaned files using undo logs

2019-08-24 Thread Robert Haas
On Wed, Aug 21, 2019 at 4:34 PM Robert Haas wrote: > ReleaseResourcesAndProcessUndo() is only supposed to be called after > AbortTransaction(), and the additional steps it performs -- > AtCleanup_Portals() and AtEOXact_Snapshot() or alternatively > AtSubCleanup_Portals -- are taken from

Re: POC: Cleaning up orphaned files using undo logs

2019-08-24 Thread Robert Haas
On Fri, Aug 23, 2019 at 2:04 AM Thomas Munro wrote: > 2. Strict self-update-only: We could update it as part of > transaction cleanup. That is, when you commit or abort, probably at > some time when your transaction is still advertised as running, you go > and update your own transaction

Re: POC: Cleaning up orphaned files using undo logs

2019-08-23 Thread Thomas Munro
On Wed, Aug 21, 2019 at 4:44 AM Andres Freund wrote: > On 2019-08-20 21:02:18 +1200, Thomas Munro wrote: > > 1. Anyone is allowed to try to read or write data at any UndoRecPtr > > that has been allocated, through the buffer pool (though you'd usually > > want to check it with

Re: POC: Cleaning up orphaned files using undo logs

2019-08-22 Thread Dilip Kumar
On Thu, Aug 22, 2019 at 9:55 PM Andres Freund wrote: > > Hi > > On August 22, 2019 9:14:10 AM PDT, Dilip Kumar wrote: > > But, those requests will > >ultimately be used for collecting the record by the bulk fetch. So if > >we are planning to change the bulk fetch to read forward then maybe we >

Re: POC: Cleaning up orphaned files using undo logs

2019-08-22 Thread Dilip Kumar
On Thu, Aug 22, 2019 at 7:34 PM Robert Haas wrote: > > On Thu, Aug 22, 2019 at 1:34 AM Dilip Kumar wrote: > > Yeah, we can handle the bulk fetch as you suggested and it will make > > it a lot easier. But, currently while registering the undo request > > (especially during the first pass) we

Re: POC: Cleaning up orphaned files using undo logs

2019-08-22 Thread Robert Haas
On Thu, Aug 22, 2019 at 12:54 AM Andres Freund wrote: > But why? It makes a *lot* more sense to have it in the beginning. I > don't think bulk-fetch really requires it to be in the end - we can > still process records forward on a page-by-page basis. There are two separate needs here: to be able

Re: POC: Cleaning up orphaned files using undo logs

2019-08-22 Thread Andres Freund
Hi On August 22, 2019 9:14:10 AM PDT, Dilip Kumar wrote: > But, those requests will >ultimately be used for collecting the record by the bulk fetch. So if >we are planning to change the bulk fetch to read forward then maybe we >don't need the valid last undo record pointer because that we will

Re: POC: Cleaning up orphaned files using undo logs

2019-08-22 Thread Dilip Kumar
On Thu, Aug 22, 2019 at 9:21 PM Dilip Kumar wrote: > > On Thu, Aug 22, 2019 at 7:34 PM Robert Haas wrote: > > > > On Thu, Aug 22, 2019 at 1:34 AM Dilip Kumar wrote: > > > Yeah, we can handle the bulk fetch as you suggested and it will make > > > it a lot easier. But, currently while

Re: POC: Cleaning up orphaned files using undo logs

2019-08-22 Thread Robert Haas
On Thu, Aug 22, 2019 at 1:34 AM Dilip Kumar wrote: > Yeah, we can handle the bulk fetch as you suggested and it will make > it a lot easier. But, currently while registering the undo request > (especially during the first pass) we need to compute the from_urecptr > and the to_urecptr. And, for

Re: POC: Cleaning up orphaned files using undo logs

2019-08-22 Thread Amit Kapila
On Thu, Aug 22, 2019 at 11:04 AM Dilip Kumar wrote: > > On Thu, Aug 22, 2019 at 10:24 AM Andres Freund wrote: > > > > Hi, > > > > On 2019-08-22 10:19:04 +0530, Dilip Kumar wrote: > > > On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote: > > > > > > > > Hi, > > > > > > > > On 2019-08-22

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Dilip Kumar
On Thu, Aug 22, 2019 at 10:24 AM Andres Freund wrote: > > Hi, > > On 2019-08-22 10:19:04 +0530, Dilip Kumar wrote: > > On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote: > > > > > > Hi, > > > > > > On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote: > > > > We can not know the complete size of

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Dilip Kumar
On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote: > > Hi, > > On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote: > > We can not know the complete size of the record even by reading the > > header because we have a payload that is variable part and payload > > length are stored in the payload

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Andres Freund
Hi, On 2019-08-22 10:19:04 +0530, Dilip Kumar wrote: > On Thu, Aug 22, 2019 at 9:58 AM Andres Freund wrote: > > > > Hi, > > > > On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote: > > > We can not know the complete size of the record even by reading the > > > header because we have a payload that

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Andres Freund
Hi, On 2019-08-22 09:51:22 +0530, Dilip Kumar wrote: > We can not know the complete size of the record even by reading the > header because we have a payload that is variable part and payload > length are stored in the payload header which again can be at random > offset. Wait, but that's just

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Dilip Kumar
On Wed, Aug 21, 2019 at 9:04 PM Robert Haas wrote: > > On Wed, Aug 21, 2019 at 3:55 AM Dilip Kumar wrote: > > I have already attempted that part and I feel it is not making code > > any simpler than what we have today. For packing, it's fine because I > > can process all the member once and

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Robert Haas
On Wed, Aug 14, 2019 at 2:57 AM Andres Freund wrote: > - My reading of the current xact.c integration is that it's not workable > as is. Undo is executed outside of a valid transaction state, > exceptions aren't properly undone, logic would need to be duplicated > to a significant degree,

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Andres Freund
On August 21, 2019 8:36:34 AM PDT, Robert Haas wrote: > We treat LWLockAcquire() as a no-fail operation in many >places; in my opinion, that elog(ERROR) that we have for too many >LWLocks should be changed to elog(PANIC) precisely because we do treat >LWLockAcquire() as no-fail in lots of

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Robert Haas
On Wed, Aug 21, 2019 at 3:55 AM Dilip Kumar wrote: > I have already attempted that part and I feel it is not making code > any simpler than what we have today. For packing, it's fine because I > can process all the member once and directly pack it into one memory > chunk and I can insert that to

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Robert Haas
On Wed, Aug 21, 2019 at 6:38 AM Amit Kapila wrote: > > FWIW, although I also thought of doing what you are describing here, I > > think Andres's proposal is probably preferable, because it's simpler. > > There's not really any reason why we can't take the buffer locks from > > within the critical

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Amit Kapila
On Tue, Aug 20, 2019 at 8:10 PM Robert Haas wrote: > > On Tue, Aug 20, 2019 at 2:42 AM Amit Kapila wrote: > > > Well, my main point, which so far has largely been ignored, was that we > > > may not acquire page locks when we still need to search for victim > > > buffers later. If we don't need

Re: POC: Cleaning up orphaned files using undo logs

2019-08-21 Thread Dilip Kumar
On Tue, Aug 20, 2019 at 7:57 PM Robert Haas wrote: > > On Mon, Aug 19, 2019 at 2:04 AM Dilip Kumar wrote: > > Currently, In UnpackedUndoRecord we store all members directly which > > are set by the caller. We store pointers to some header which are > > allocated internally by the undo layer and

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Robert Haas
On Tue, Aug 13, 2019 at 8:11 AM Robert Haas wrote: > > We can probably check the fxid queue and error queue to get that > > value. However, I am not sure if that is sufficient because incase we > > perform the request in the foreground, it won't be present in queues. > > Oh, I forgot about that

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Robert Haas
On Mon, Aug 19, 2019 at 2:04 AM Dilip Kumar wrote: > Currently, In UnpackedUndoRecord we store all members directly which > are set by the caller. We store pointers to some header which are > allocated internally by the undo layer and the caller need not worry > about setting them. So now you

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Robert Haas
On Tue, Aug 20, 2019 at 5:02 AM Thomas Munro wrote: > 3. UndoLogDiscard() uses DiscardBuffer() to invalidate any currently > unpinned buffers, and marks as BM_DISCARDED any that happen to be > pinned right now, so they can't be immediately invalidated. Such > buffers are never written back and

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Andres Freund
On 2019-08-20 09:44:23 -0700, Andres Freund wrote: > On 2019-08-20 21:02:18 +1200, Thomas Munro wrote: > > Aside from code changes based on review (and I have more to come of > > those), the attached experimental patchset (also at > > https://github.com/EnterpriseDB/zheap/tree/undo) has a new

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Andres Freund
Hi, On 2019-08-20 17:11:38 +0530, Dilip Kumar wrote: > On Wed, Aug 14, 2019 at 10:35 PM Andres Freund wrote: > > On 2019-08-14 14:48:07 +0530, Dilip Kumar wrote: > > > On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > > > I don't think we can normally pin the undo buffers properly at

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Andres Freund
Hi, On 2019-08-20 21:02:18 +1200, Thomas Munro wrote: > Aside from code changes based on review (and I have more to come of > those), the attached experimental patchset (also at > https://github.com/EnterpriseDB/zheap/tree/undo) has a new protocol > that, I hope, allows for better concurrency,

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Andres Freund
Hi, On 2019-08-20 09:08:29 -0400, Robert Haas wrote: > On Sat, Aug 17, 2019 at 1:28 PM Andres Freund wrote: > > The primary one in the context here is that if we do *not* have to lock > > the buffers all ahead of time, we can simplify the interface. We > > certainly can't lock the buffers over

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Robert Haas
On Mon, Aug 19, 2019 at 8:22 AM Amit Kapila wrote: > One point to remember in this regard is that we do need to modify the > LSN in undo pages after writing WAL, so all the undo pages need to be > locked by that time or we again need to take the lock on them. Uh, but a big part of the point of

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Robert Haas
On Tue, Aug 20, 2019 at 2:42 AM Amit Kapila wrote: > > Well, my main point, which so far has largely been ignored, was that we > > may not acquire page locks when we still need to search for victim > > buffers later. If we don't need to lock the pages up-front, but only do > > so once we're

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Robert Haas
On Mon, Aug 19, 2019 at 5:16 PM Andres Freund wrote: > Well, my main point, which so far has largely been ignored, was that we > may not acquire page locks when we still need to search for victim > buffers later. If we don't need to lock the pages up-front, but only do > so once we're actually

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Robert Haas
On Sat, Aug 17, 2019 at 1:28 PM Andres Freund wrote: > The primary one in the context here is that if we do *not* have to lock > the buffers all ahead of time, we can simplify the interface. We > certainly can't lock the buffers over IO (due to buffer reclaim) as > we're doing right now, so we'd

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Dilip Kumar
On Wed, Aug 14, 2019 at 10:35 PM Andres Freund wrote: > > Hi, > > On 2019-08-14 14:48:07 +0530, Dilip Kumar wrote: > > On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > I don't think we can normally pin the undo buffers properly at that > stage. Without knowing the correct contents of the

Re: POC: Cleaning up orphaned files using undo logs

2019-08-20 Thread Amit Kapila
On Tue, Aug 20, 2019 at 2:46 AM Andres Freund wrote: > > On 2019-08-19 17:52:24 +0530, Amit Kapila wrote: > > On Sat, Aug 17, 2019 at 10:58 PM Andres Freund wrote: > > > > > > > Well, I don't understand why you're on about this. We've discussed it > > > > a number of times but I'm still

Re: POC: Cleaning up orphaned files using undo logs

2019-08-19 Thread Andres Freund
On 2019-08-19 17:52:24 +0530, Amit Kapila wrote: > On Sat, Aug 17, 2019 at 10:58 PM Andres Freund wrote: > > > > Hi, > > > > On 2019-08-17 12:05:21 -0400, Robert Haas wrote: > > > On Wed, Aug 14, 2019 at 12:39 PM Andres Freund wrote: > > > > > > Again, I think it's not ok to just assume you can

Re: POC: Cleaning up orphaned files using undo logs

2019-08-19 Thread Amit Kapila
On Sat, Aug 17, 2019 at 10:58 PM Andres Freund wrote: > > Hi, > > On 2019-08-17 12:05:21 -0400, Robert Haas wrote: > > On Wed, Aug 14, 2019 at 12:39 PM Andres Freund wrote: > > > > > Again, I think it's not ok to just assume you can lock an essentially > > > > > unbounded number of buffers. This

Re: POC: Cleaning up orphaned files using undo logs

2019-08-19 Thread Dilip Kumar
On Wed, Aug 14, 2019 at 10:35 PM Andres Freund wrote: > > > > > - When reading an undo record, the whole stage of UnpackUndoData() > > > reading data into a the UndoPackContext is omitted, reading directly > > > into the UnpackedUndoRecord. That removes one further copy of the > > >

Re: POC: Cleaning up orphaned files using undo logs

2019-08-18 Thread Dilip Kumar
On Sat, Aug 17, 2019 at 9:35 PM Robert Haas wrote: > > On Wed, Aug 14, 2019 at 12:39 PM Andres Freund wrote: > > > > Again, I think it's not ok to just assume you can lock an essentially > > > > unbounded number of buffers. This seems almost guaranteed to result in > > > > deadlocks. And there's

Re: POC: Cleaning up orphaned files using undo logs

2019-08-17 Thread Andres Freund
Hi, On 2019-08-17 12:05:21 -0400, Robert Haas wrote: > On Wed, Aug 14, 2019 at 12:39 PM Andres Freund wrote: > > > > Again, I think it's not ok to just assume you can lock an essentially > > > > unbounded number of buffers. This seems almost guaranteed to result in > > > > deadlocks. And there's

Re: POC: Cleaning up orphaned files using undo logs

2019-08-17 Thread Robert Haas
On Wed, Aug 14, 2019 at 12:39 PM Andres Freund wrote: > > > Again, I think it's not ok to just assume you can lock an essentially > > > unbounded number of buffers. This seems almost guaranteed to result in > > > deadlocks. And there's limits on how many lwlocks one can hold etc. > > > > I think

Re: POC: Cleaning up orphaned files using undo logs

2019-08-16 Thread Dilip Kumar
On Fri, Aug 16, 2019 at 10:56 AM Andres Freund wrote: > > Hi, > > On 2019-08-16 09:44:25 +0530, Dilip Kumar wrote: > > On Wed, Aug 14, 2019 at 2:48 PM Dilip Kumar wrote: > > > > > > On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > > > > > > I think that batch reading should just copy

Re: POC: Cleaning up orphaned files using undo logs

2019-08-15 Thread Andres Freund
Hi, On 2019-08-16 09:44:25 +0530, Dilip Kumar wrote: > On Wed, Aug 14, 2019 at 2:48 PM Dilip Kumar wrote: > > > > On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > > > > I think that batch reading should just copy the underlying data into a > > > char* buffer. Only the records that

Re: POC: Cleaning up orphaned files using undo logs

2019-08-15 Thread Thomas Munro
Hi Kuntal, On Thu, Jul 25, 2019 at 5:40 PM Kuntal Ghosh wrote: > Here are some review comments on 0003-Add-undo-log-manager.patch. I've > tried to avoid duplicate comments as much as possible. Thanks! Replies inline. I'll be posting a new patch set shortly with these and other fixes. > 1. In

Re: POC: Cleaning up orphaned files using undo logs

2019-08-15 Thread Thomas Munro
Hi Amit, I've combined three of your messages into one below, and responded inline. New patch set to follow shortly, with the fixes listed below (and others from other reviewers). On Wed, Jul 24, 2019 at 9:15 PM Amit Kapila wrote: > 0003-Add-undo-log-manager.patch > 1. >

Re: POC: Cleaning up orphaned files using undo logs

2019-08-15 Thread Dilip Kumar
On Wed, Aug 14, 2019 at 2:48 PM Dilip Kumar wrote: > > On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > > I think that batch reading should just copy the underlying data into a > > char* buffer. Only the records that currently are being used by > > higher layers should get exploded

Re: POC: Cleaning up orphaned files using undo logs

2019-08-15 Thread Dilip Kumar
On Wed, Aug 14, 2019 at 10:35 PM Andres Freund wrote: > > Hi, > > On 2019-08-14 14:48:07 +0530, Dilip Kumar wrote: > > On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > > > - I think there's two fairly fundamental, and related, problems with > > > the sequence outlined above: > > > > > >

Re: POC: Cleaning up orphaned files using undo logs

2019-08-14 Thread Andres Freund
Hi, On 2019-08-14 14:48:07 +0530, Dilip Kumar wrote: > On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > > - I think there's two fairly fundamental, and related, problems with > > the sequence outlined above: > > > > - We can't search for target buffers to store undo data, while

Re: POC: Cleaning up orphaned files using undo logs

2019-08-14 Thread Andres Freund
Hi, On 2019-08-13 17:05:27 +0530, Dilip Kumar wrote: > On Mon, Aug 5, 2019 at 11:59 PM Andres Freund wrote: > > (as I was out of context due to dealing with bugs, I've switched to > > lookign at the current zheap/undoprocessing branch. > > > > On 2019-07-30 01:02:20 -0700, Andres Freund wrote: >

Re: POC: Cleaning up orphaned files using undo logs

2019-08-14 Thread Andres Freund
Hi, On 2019-08-13 13:53:59 +0530, Dilip Kumar wrote: > On Tue, Jul 30, 2019 at 1:32 PM Andres Freund wrote: > > > + /* Loop until we have fetched all the buffers in which we need to > > > write. */ > > > + while (size > 0) > > > + { > > > + bufidx =

Re: POC: Cleaning up orphaned files using undo logs

2019-08-14 Thread Dilip Kumar
On Wed, Aug 14, 2019 at 12:27 PM Andres Freund wrote: > > Hi, > > - I think there's two fairly fundamental, and related, problems with > the sequence outlined above: > > - We can't search for target buffers to store undo data, while holding > the "table" page content locked. > > The

Re: POC: Cleaning up orphaned files using undo logs

2019-08-14 Thread Andres Freund
Hi, On 2019-08-06 14:18:42 -0700, Andres Freund wrote: > Here's the last section of my low-leve review. Plan to write a higher > level summary afterwards, now that I have a better picture of the code. General comments: - For a feature of this complexity, there's very little architectural

Re: POC: Cleaning up orphaned files using undo logs

2019-08-13 Thread Robert Haas
On Tue, Aug 13, 2019 at 6:28 AM Amit Kapila wrote: > So, for top-level transactions rollback, we can directly refer from > UndoRequest *, the start and end locations. But, what should we do > for sub-transactions (rollback to savepoint)? One related point is > that we also need information

Re: POC: Cleaning up orphaned files using undo logs

2019-08-13 Thread Dilip Kumar
On Tue, Aug 6, 2019 at 1:26 PM Andres Freund wrote: > > Hi, > > On 2019-08-05 11:29:34 -0700, Andres Freund wrote: > > Need to do something else for a bit. More later. > > > > + /* > > + * Compute the header size of the undo record. > > + */ > > +Size > > +UndoRecordHeaderSize(uint16 uur_info)

Re: POC: Cleaning up orphaned files using undo logs

2019-08-13 Thread Dilip Kumar
On Mon, Aug 5, 2019 at 11:59 PM Andres Freund wrote: > > Hi, > > (as I was out of context due to dealing with bugs, I've switched to > lookign at the current zheap/undoprocessing branch. > > On 2019-07-30 01:02:20 -0700, Andres Freund wrote: > > +/* > > + * Insert a previously-prepared undo

Re: POC: Cleaning up orphaned files using undo logs

2019-08-13 Thread Amit Kapila
On Thu, Aug 8, 2019 at 7:01 PM Andres Freund wrote: > On 2019-08-07 14:50:17 +0530, Amit Kapila wrote: > > On Tue, Aug 6, 2019 at 1:26 PM Andres Freund wrote: > > > On 2019-08-05 11:29:34 -0700, Andres Freund wrote: > > > > > typedef struct TwoPhaseFileHeader > > > > { > > > > @@ -927,6

Re: POC: Cleaning up orphaned files using undo logs

2019-08-13 Thread Amit Kapila
On Fri, Aug 9, 2019 at 1:57 AM Robert Haas wrote: > > On Thu, Aug 8, 2019 at 9:31 AM Andres Freund wrote: > > I know that Robert is working on a patch that revises the undo request > > layer somewhat, it's possible that this is best discussed afterwards. > > Here's what I have at the moment.

Re: POC: Cleaning up orphaned files using undo logs

2019-08-13 Thread Dilip Kumar
On Tue, Jul 30, 2019 at 1:32 PM Andres Freund wrote: > > Hi, > > Amit, short note: The patches aren't attached in patch order. Obviously > a miniscule thing, but still nicer if that's not the case. > > Dilip, this also contains the start of a review for the undo record > interface further down.

Re: POC: Cleaning up orphaned files using undo logs

2019-08-12 Thread Dilip Kumar
On Tue, Jul 30, 2019 at 12:21 PM Thomas Munro wrote: > > Hi Dilip, > > > commit 2f3c127b9e8bc7d27cf7adebff0a355684dfb94e > > Author: Dilip Kumar > > Date: Thu May 2 11:28:13 2019 +0530 > > > >Provide interfaces to store and fetch undo records. > > +#include "commands/tablecmds.h" >

Re: POC: Cleaning up orphaned files using undo logs

2019-08-12 Thread Dilip Kumar
On Wed, Jul 24, 2019 at 11:28 AM Rushabh Lathia wrote: > > Hi, > > I have stated review of > 0008-Provide-interfaces-to-store-and-fetch-undo-records.patch, here are few > quick comments. > > 1) README.undointerface should provide more information like API details or > the sequence in which API

Re: POC: Cleaning up orphaned files using undo logs

2019-08-12 Thread Dilip Kumar
On Fri, Jul 19, 2019 at 2:28 PM Amit Kapila wrote: > > On Thu, Jul 11, 2019 at 9:17 AM Dilip Kumar wrote: > > > > On Thu, Jul 11, 2019 at 12:38 AM Robert Haas wrote: > > > > > > I don't like the fact that undoaccess.c has a new global, > > > undo_compression_info. I haven't read the code

Re: POC: Cleaning up orphaned files using undo logs

2019-08-12 Thread Dilip Kumar
On Thu, Jul 18, 2019 at 4:58 PM Amit Kapila wrote: > > On Tue, Jul 16, 2019 at 2:20 PM Dilip Kumar wrote: > > > > Few comments on the new patch: > > 1. > Additionally, > +there is a mechanism for multi-insert, wherein multiple records are prepared > +and inserted at a time. > > Which mechanism

Re: POC: Cleaning up orphaned files using undo logs

2019-08-09 Thread Robert Haas
On Wed, Aug 7, 2019 at 6:57 AM Heikki Linnakangas wrote: > Yeah, that's also a problem with complicated WAL record types. Hopefully > the complex cases are an exception, not the norm. A complex case is > unlikely to fit any pre-defined set of fields anyway. (We could look at > how e.g. protobuf

Re: POC: Cleaning up orphaned files using undo logs

2019-08-09 Thread Amit Kapila
On Fri, Jul 26, 2019 at 9:57 PM Amit Khandekar wrote: > > On Fri, 26 Jul 2019 at 12:25, Amit Kapila wrote: > > I agree with all your other comments. > > Thanks for addressing the comments. Below is the continuation of my > comments from

Re: POC: Cleaning up orphaned files using undo logs

2019-08-09 Thread Amit Kapila
On Tue, Jul 23, 2019 at 8:12 PM Amit Khandekar wrote: > > > > Some further review comments for undoworker.c : > > > +/* Sets the worker's lingering status. */ > +static void > +UndoWorkerIsLingering(bool sleep) > The function name sounds like "is the worker lingering ?". Can

Re: POC: Cleaning up orphaned files using undo logs

2019-08-09 Thread Amit Kapila
On Mon, Jul 22, 2019 at 8:39 PM Amit Khandekar wrote: > > On Mon, 22 Jul 2019 at 14:21, Amit Kapila wrote: > > I have started review of > 0014-Allow-execution-and-discard-of-undo-by-background-wo.patch. Below > are some quick comments to start with: > > +++ b/src/backend/access/undo/undoworker.c

Re: POC: Cleaning up orphaned files using undo logs

2019-08-08 Thread Amit Kapila
On Fri, Aug 9, 2019 at 1:57 AM Robert Haas wrote: > > On Thu, Aug 8, 2019 at 9:31 AM Andres Freund wrote: > > I know that Robert is working on a patch that revises the undo request > > layer somewhat, it's possible that this is best discussed afterwards. > > Here's what I have at the moment.

Re: POC: Cleaning up orphaned files using undo logs

2019-08-08 Thread Robert Haas
On Thu, Aug 8, 2019 at 9:31 AM Andres Freund wrote: > I know that Robert is working on a patch that revises the undo request > layer somewhat, it's possible that this is best discussed afterwards. Here's what I have at the moment. This is not by any means a complete replacement for Amit's undo

Re: POC: Cleaning up orphaned files using undo logs

2019-08-08 Thread Andres Freund
Hi, On 2019-08-07 14:50:17 +0530, Amit Kapila wrote: > On Tue, Aug 6, 2019 at 1:26 PM Andres Freund wrote: > > On 2019-08-05 11:29:34 -0700, Andres Freund wrote: > > > +/* > > > + * Binary heap comparison function to compare the time at which an error > > > + * occurred for transactions. > > > +

Re: POC: Cleaning up orphaned files using undo logs

2019-08-08 Thread Amit Kapila
On Wed, Aug 7, 2019 at 5:06 PM Thomas Munro wrote: > > On Thu, Aug 1, 2019 at 1:22 AM Amit Kapila wrote: > > On Wed, Jul 31, 2019 at 10:13 AM Amit Kapila > > wrote: > > > On Tue, Jul 30, 2019 at 5:26 PM Thomas Munro > > > wrote: > > > > but > > > > here's a small thing: I managed to reach

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Thomas Munro
On Thu, Aug 1, 2019 at 1:22 AM Amit Kapila wrote: > On Wed, Jul 31, 2019 at 10:13 AM Amit Kapila wrote: > > On Tue, Jul 30, 2019 at 5:26 PM Thomas Munro wrote: > > > but > > > here's a small thing: I managed to reach an LWLock self-deadlock in > > > the undo worker launcher: > > > > > > > I

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Heikki Linnakangas
On 07/08/2019 13:52, Dilip Kumar wrote: I have one more problem related to compression of the command id field. Basically, the problem is that we don't set the command id in the WAL and we will always store FirstCommandId in the undo[1]. So suppose there were 2 operations under a different

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Heikki Linnakangas
On 05/08/2019 16:24, Robert Haas wrote: On Sun, Aug 4, 2019 at 5:16 AM Heikki Linnakangas wrote: I feel that the level of abstraction is not quite right. There are a bunch of fields, like uur_block, uur_offset, uur_tuple, that are probably useful for some UNDO resource managers (zheap I

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Dilip Kumar
On Tue, Jul 30, 2019 at 1:54 PM Dilip Kumar wrote: > > On Tue, Jul 30, 2019 at 12:21 PM Thomas Munro wrote: > > > > One data structure that could perhaps hold this would be > > UndoLogTableEntry (the per-backend cache, indexed by undo log number, > > with pretty fast lookups; used for things

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Amit Kapila
On Tue, Aug 6, 2019 at 1:26 PM Andres Freund wrote: > On 2019-08-05 11:29:34 -0700, Andres Freund wrote: > I am responding to some of the points where I need some more inputs or some discussion is required. Some of the things need more thoughts which I will respond later and some are quite

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Dilip Kumar
On Tue, Aug 6, 2019 at 1:26 PM Andres Freund wrote: > > Hi, > > On 2019-08-05 11:29:34 -0700, Andres Freund wrote: > > Need to do something else for a bit. More later. > > > + * false, otherwise. > > + */ > > +static bool > > +UndoAlreadyApplied(FullTransactionId full_xid, UndoRecPtr

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Dilip Kumar
On Tue, Aug 6, 2019 at 1:26 PM Andres Freund wrote: > > Hi, > > On 2019-08-05 11:29:34 -0700, Andres Freund wrote: > > Need to do something else for a bit. More later. > > Here we go. > Thanks for the review. I will work on them. Currently, I need suggestions on some of the review comments. > >

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Kuntal Ghosh
Hello Andres, On Tue, Aug 6, 2019 at 1:26 PM Andres Freund wrote: > > +/* Each worker queue is a binary heap. */ > > +typedef struct > > +{ > > + binaryheap *bh; > > + union > > + { > > + UndoXidQueue *xid_elems; > > + UndoSizeQueue *size_elems; > > +

Re: POC: Cleaning up orphaned files using undo logs

2019-08-07 Thread Thomas Munro
Hi, I'll be responding to a bunch of long review emails in this thread point by point separately, but just picking out a couple of points here that jumped out at me: On Wed, Aug 7, 2019 at 9:18 AM Andres Freund wrote: > > + { > > + /* > > +

Re: POC: Cleaning up orphaned files using undo logs

2019-08-06 Thread Andres Freund
Hi, On 2019-08-06 00:56:26 -0700, Andres Freund wrote: > Out of energy. Here's the last section of my low-leve review. Plan to write a higher level summary afterwards, now that I have a better picture of the code. > +static void > +UndoDiscardOneLog(UndoLogSlot *slot, TransactionId xmin, bool

Re: POC: Cleaning up orphaned files using undo logs

2019-08-06 Thread Andres Freund
Hi, On 2019-08-05 11:29:34 -0700, Andres Freund wrote: > Need to do something else for a bit. More later. Here we go. > + /* > + * Compute the header size of the undo record. > + */ > +Size > +UndoRecordHeaderSize(uint16 uur_info) > +{ > + Sizesize; > + > + /* Add fixed

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Amit Kapila
On Mon, Aug 5, 2019 at 6:29 PM Robert Haas wrote: > > On Mon, Aug 5, 2019 at 6:16 AM Amit Kapila wrote: > > For zheap, we collect all the records of a page, apply them together > > and then write the entire page in WAL. The progress of transaction is > > updated at either transaction end

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Robert Haas
On Mon, Aug 5, 2019 at 12:42 PM Andres Freund wrote: > A good move in the right direction, imo. I spent some more time thinking about this and talking to Thomas about it and I'd like to propose a somewhat more aggressive restructuring proposal, with the aim of getting a cleaner separation

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Andres Freund
Hi, (as I was out of context due to dealing with bugs, I've switched to lookign at the current zheap/undoprocessing branch. On 2019-07-30 01:02:20 -0700, Andres Freund wrote: > +/* > + * Insert a previously-prepared undo records. > + * > + * This function will write the actual undo record into

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Andres Freund
Hi, On 2019-08-05 11:25:10 -0400, Robert Haas wrote: > The obvious thing to do seems to be to have UndoLogControl objects own > SmgrRelations. That would be something of a novelty, since it looks > like currently only a Relation ever owns an SMgrRelation, but the smgr > infrastructure seems to

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Robert Haas
On Tue, Jul 30, 2019 at 4:02 AM Andres Freund wrote: > I'm a bit worried about expanding the use of > ReadBufferWithoutRelcache(). Not so much because of the relcache itself, > but because it requires doing separate smgropen() calls. While not > crazily expensive, it's also not free. Especially

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Robert Haas
On Sun, Aug 4, 2019 at 5:16 AM Heikki Linnakangas wrote: > I feel that the level of abstraction is not quite right. There are a > bunch of fields, like uur_block, uur_offset, uur_tuple, that are > probably useful for some UNDO resource managers (zheap I presume), but > seem kind of arbitrary. How

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Robert Haas
On Mon, Aug 5, 2019 at 6:16 AM Amit Kapila wrote: > For zheap, we collect all the records of a page, apply them together > and then write the entire page in WAL. The progress of transaction is > updated at either transaction end (rollback complete) or after > processing some threshold of undo

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Amit Kapila
On Mon, Aug 5, 2019 at 12:09 PM Heikki Linnakangas wrote: > > On 05/08/2019 07:23, Thomas Munro wrote: > > On Mon, Aug 5, 2019 at 3:54 PM Amit Kapila wrote: > >> On Sun, Aug 4, 2019 at 2:46 PM Heikki Linnakangas wrote: > >>> Could we leave out the UNDO and discard worker processes for now? >

Re: POC: Cleaning up orphaned files using undo logs

2019-08-05 Thread Heikki Linnakangas
On 05/08/2019 07:23, Thomas Munro wrote: On Mon, Aug 5, 2019 at 3:54 PM Amit Kapila wrote: On Sun, Aug 4, 2019 at 2:46 PM Heikki Linnakangas wrote: Could we leave out the UNDO and discard worker processes for now? Execute all UNDO actions immediately at rollback, and after crash recovery.

Re: POC: Cleaning up orphaned files using undo logs

2019-08-04 Thread Thomas Munro
On Mon, Aug 5, 2019 at 3:54 PM Amit Kapila wrote: > On Sun, Aug 4, 2019 at 2:46 PM Heikki Linnakangas wrote: > > I had a look at the UNDO patches at > > https://www.postgresql.org/message-id/CAA4eK1KKAFBCJuPnFtgdc89djv4xO%3DZkAdXvKQinqN4hWiRbvA%40mail.gmail.com, > > and at the patch to use the

Re: POC: Cleaning up orphaned files using undo logs

2019-08-04 Thread Amit Kapila
On Sun, Aug 4, 2019 at 2:46 PM Heikki Linnakangas wrote: > > I had a look at the UNDO patches at > https://www.postgresql.org/message-id/CAA4eK1KKAFBCJuPnFtgdc89djv4xO%3DZkAdXvKQinqN4hWiRbvA%40mail.gmail.com, > and at the patch to use the UNDO logs to clean up orphaned files, from >

Re: POC: Cleaning up orphaned files using undo logs

2019-08-04 Thread Heikki Linnakangas
I had a look at the UNDO patches at https://www.postgresql.org/message-id/CAA4eK1KKAFBCJuPnFtgdc89djv4xO%3DZkAdXvKQinqN4hWiRbvA%40mail.gmail.com, and at the patch to use the UNDO logs to clean up orphaned files, from undo-2019-05-10.tgz earlier in this thread. Are these the latest ones to

Re: POC: Cleaning up orphaned files using undo logs

2019-08-02 Thread Amit Kapila
On Mon, Jul 22, 2019 at 3:51 PM Dilip Kumar wrote: > > On Mon, Jul 22, 2019 at 2:21 PM Amit Kapila wrote: > > > > I have reviewed 0012-Infrastructure-to-execute-pending-undo-actions, > Please find my comment so far. .. > 4. > +void > +undoaction_redo(XLogReaderState *record) > +{ > + uint8 info

Re: POC: Cleaning up orphaned files using undo logs

2019-07-31 Thread Amit Kapila
On Wed, Jul 31, 2019 at 10:13 AM Amit Kapila wrote: > > On Tue, Jul 30, 2019 at 5:26 PM Thomas Munro wrote: > > > > Hi Amit > > > > I've been testing some undo worker workloads (more on that soon), > > > > One small point, there is one small bug in the error queues which is > that the element

  1   2   3   >