[HACKERS] SLRU
Hey I was reading the multi transaction log manager, multixact.c. I didn't get what SLRU does. I want the goal of this module, and why we use it. I'm kind of newbie, be patient with me ;) Regards Soroosh
Re: [HACKERS] SLRU
On Mon, Jun 17, 2013 at 1:22 PM, Soroosh Sardari soroosh.sard...@gmail.comwrote: Hey I was reading the multi transaction log manager, multixact.c. I didn't get what SLRU does. I want the goal of this module, and why we use it. I'm kind of newbie, be patient with me ;) Did you look at src/backend/access/transam/slru.c ? The first para in that file is quite explanatory: * We use a simple least-recently-used scheme to manage a pool of page * buffers. Under ordinary circumstances we expect that write * traffic will occur mostly to the latest page (and to the just-prior * page, soon after a page transition). Read traffic will probably touch * a larger span of pages, but in any case a fairly small number of page * buffers should be sufficient. So, we just search the buffers using plain * linear search; there's no need for a hashtable or anything fancy. * The management algorithm is straight LRU except that we will never swap * out the latest page (since we know it's going to be hit again eventually). Thanks, Pavan -- Pavan Deolasee http://www.linkedin.com/in/pavandeolasee
Re: [HACKERS] SLRU limits
Is this a TODO? --- Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 09.06.2011 15:50, Robert Haas wrote: And I would guess that there's a lot more interest in raising BLCKSZ than lowering it. It might not be a bad idea to adopt the fix you propose anyway, but it doesn't seem urgent. I guess we could fix pg_subtrans by not allowing BLCKSZ 8k. That leaves the problem with pg_serial. Kevin has already worked around, but I'm not very happy with that workaround. If we don't want to change it wholesale, one option would be to support different lengths of filenames in slru.c for different slrus. At a quick glance, it seems pretty easy. That would allow keeping clog unchanged - that's the one that's most likely to have unforeseen consequences if changed. pg_subtrans and pg_serial are more ephemeral, they don't need to be retained over shutdown, so they seem less likely to cause trouble. That seems like the best option to me. I agree with Robert that this is completely not urgent. If you want to fool with it for 9.2, fine, but let's not destabilize 9.1 for it. (BTW, while I've not looked at the SLRU code in several years, I'm quite unconvinced that this is only a matter of filename lengths.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SLRU limits
Bruce Momjian br...@momjian.us wrote: Is this a TODO? Tom Lane wrote: Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: If we don't want to change it wholesale, one option would be to support different lengths of filenames in slru.c for different slrus. At a quick glance, it seems pretty easy. That would allow keeping clog unchanged - that's the one that's most likely to have unforeseen consequences if changed. pg_subtrans and pg_serial are more ephemeral, they don't need to be retained over shutdown, so they seem less likely to cause trouble. That seems like the best option to me. I agree with Robert that this is completely not urgent. If you want to fool with it for 9.2, fine, but let's not destabilize 9.1 for it. (BTW, while I've not looked at the SLRU code in several years, I'm quite unconvinced that this is only a matter of filename lengths.) I think it should be. I agree that the workaround is kinda ugly, and this is really the only clean solution I see. I don't think it's so ugly that it takes precedence over trying to bring some of Robert's LW locking improvements into the SSI area, so this one is likely not to get into 9.2, in spite of appearing to be a smaller change. So a TODO entry seems like the right way to deal with it. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] SLRU limits
While reviewing the SLRU code in predicate.c again, I remembered this old thread: http://archives.postgresql.org/pgsql-hackers/2010-12/msg02374.php SLRU has a limit of 64k segment files, because the files are named using four hex digits like 00CE. Kevin's math shows that that's just enough to store 2^32 four-byte integers, which wasn't enough for predicate.c, which needs to store uint64s. Kevin worked around that by simply limiting the max range of open xids to fit the SLRU limit, ie. 2^31. However, that math was based on 8k block size, and the situation is worse for smaller block sizes. If you set BLCKSZ to 2048 or less, pg_subtrans can only hold 1 billion transactions. With 1024 block size, only half a billion. It's awfully late in the release cycle, but how about we add another digit to the filenames used by SLRU, to up the limit? At a quick glance, I don't see any protection against wrapping around page numbers in subtrans.c, so that ought to be fixed somehow. And it would make the SLRU code in predicate.c simpler (I note that the warning logic at least is wrong as it is - it doesn't take XID wrap-around into account). -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SLRU limits
On Thu, Jun 9, 2011 at 7:46 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: While reviewing the SLRU code in predicate.c again, I remembered this old thread: http://archives.postgresql.org/pgsql-hackers/2010-12/msg02374.php SLRU has a limit of 64k segment files, because the files are named using four hex digits like 00CE. Kevin's math shows that that's just enough to store 2^32 four-byte integers, which wasn't enough for predicate.c, which needs to store uint64s. Kevin worked around that by simply limiting the max range of open xids to fit the SLRU limit, ie. 2^31. However, that math was based on 8k block size, and the situation is worse for smaller block sizes. If you set BLCKSZ to 2048 or less, pg_subtrans can only hold 1 billion transactions. With 1024 block size, only half a billion. I'm pretty unexcited about this. It's not terribly sane to keep a transaction open for half a billion XIDs anyway, because of VACUUM. And I would guess that there's a lot more interest in raising BLCKSZ than lowering it. It might not be a bad idea to adopt the fix you propose anyway, but it doesn't seem urgent. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SLRU limits
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: It's awfully late in the release cycle, but how about we add another digit to the filenames used by SLRU, to up the limit? It's way too late for that kind of thing, unless you are saying that SSI in and of itself is going to cause a release slip. (Which I'm getting the uncomfortable feeling may be true anyway.) That is not a one-line kind of fix --- it is likely to have a lot of unforeseen consequences. And who builds with smaller-than-default BLCKSZ anyway? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SLRU limits
On 09.06.2011 15:50, Robert Haas wrote: On Thu, Jun 9, 2011 at 7:46 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: While reviewing the SLRU code in predicate.c again, I remembered this old thread: http://archives.postgresql.org/pgsql-hackers/2010-12/msg02374.php SLRU has a limit of 64k segment files, because the files are named using four hex digits like 00CE. Kevin's math shows that that's just enough to store 2^32 four-byte integers, which wasn't enough for predicate.c, which needs to store uint64s. Kevin worked around that by simply limiting the max range of open xids to fit the SLRU limit, ie. 2^31. However, that math was based on 8k block size, and the situation is worse for smaller block sizes. If you set BLCKSZ to 2048 or less, pg_subtrans can only hold 1 billion transactions. With 1024 block size, only half a billion. I'm pretty unexcited about this. It's not terribly sane to keep a transaction open for half a billion XIDs anyway, because of VACUUM. I agree, but if you or your application is insane enough to do that anyway, it's not appropriate response for the system to give warnings about apparent XID wrap-around, and in the worst case mix up subtransactions belonging to different transactions, possibly leading to data loss. I have not actually tested that, but I don't see any safeguards to stop it from happening. Of course, another alternative is to add such safeguards. And I would guess that there's a lot more interest in raising BLCKSZ than lowering it. It might not be a bad idea to adopt the fix you propose anyway, but it doesn't seem urgent. I guess we could fix pg_subtrans by not allowing BLCKSZ 8k. That leaves the problem with pg_serial. Kevin has already worked around, but I'm not very happy with that workaround. If we don't want to change it wholesale, one option would be to support different lengths of filenames in slru.c for different slrus. At a quick glance, it seems pretty easy. That would allow keeping clog unchanged - that's the one that's most likely to have unforeseen consequences if changed. pg_subtrans and pg_serial are more ephemeral, they don't need to be retained over shutdown, so they seem less likely to cause trouble. That seems like the best option to me. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SLRU limits
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 09.06.2011 15:50, Robert Haas wrote: And I would guess that there's a lot more interest in raising BLCKSZ than lowering it. It might not be a bad idea to adopt the fix you propose anyway, but it doesn't seem urgent. I guess we could fix pg_subtrans by not allowing BLCKSZ 8k. That leaves the problem with pg_serial. Kevin has already worked around, but I'm not very happy with that workaround. If we don't want to change it wholesale, one option would be to support different lengths of filenames in slru.c for different slrus. At a quick glance, it seems pretty easy. That would allow keeping clog unchanged - that's the one that's most likely to have unforeseen consequences if changed. pg_subtrans and pg_serial are more ephemeral, they don't need to be retained over shutdown, so they seem less likely to cause trouble. That seems like the best option to me. I agree with Robert that this is completely not urgent. If you want to fool with it for 9.2, fine, but let's not destabilize 9.1 for it. (BTW, while I've not looked at the SLRU code in several years, I'm quite unconvinced that this is only a matter of filename lengths.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SLRU limits
On 09.06.2011 19:24, Tom Lane wrote: (BTW, while I've not looked at the SLRU code in several years, I'm quite unconvinced that this is only a matter of filename lengths.) I don't see anything but the filename length needing adjustment. In fact, when I hacked predicate.c to ignore the issue and use too high page numbers, I ended up with files like 100AB in the directory. So slru.c goes merrily above the limit, it just won't recognize them when it's time to truncate the slru because of the unexpected length, and will not clean them up. (perhaps not worth risking it in 9.1 anyway, though..) -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SLRU API tweak
Excerpts from Kevin Grittner's message of mié dic 29 20:46:55 -0300 2010: Attached is a small patch to avoid putting an opaque structure into the slru.h file and using it in an external function call where external callers must always specify NULL. Thanks, committed. -- Álvaro Herrera alvhe...@commandprompt.com The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] SLRU API tweak
Attached is a small patch to avoid putting an opaque structure into the slru.h file and using it in an external function call where external callers must always specify NULL. -Kevin *** a/src/backend/access/transam/clog.c --- b/src/backend/access/transam/clog.c *** *** 445,451 BootStrapCLOG(void) slotno = ZeroCLOGPage(0, false); /* Make sure it's written out */ ! SimpleLruWritePage(ClogCtl, slotno, NULL); Assert(!ClogCtl-shared-page_dirty[slotno]); LWLockRelease(CLogControlLock); --- 445,451 slotno = ZeroCLOGPage(0, false); /* Make sure it's written out */ ! SimpleLruWritePage(ClogCtl, slotno); Assert(!ClogCtl-shared-page_dirty[slotno]); LWLockRelease(CLogControlLock); *** *** 698,704 clog_redo(XLogRecPtr lsn, XLogRecord *record) LWLockAcquire(CLogControlLock, LW_EXCLUSIVE); slotno = ZeroCLOGPage(pageno, false); ! SimpleLruWritePage(ClogCtl, slotno, NULL); Assert(!ClogCtl-shared-page_dirty[slotno]); LWLockRelease(CLogControlLock); --- 698,704 LWLockAcquire(CLogControlLock, LW_EXCLUSIVE); slotno = ZeroCLOGPage(pageno, false); ! SimpleLruWritePage(ClogCtl, slotno); Assert(!ClogCtl-shared-page_dirty[slotno]); LWLockRelease(CLogControlLock); *** a/src/backend/access/transam/multixact.c --- b/src/backend/access/transam/multixact.c *** *** 1454,1460 BootStrapMultiXact(void) slotno = ZeroMultiXactOffsetPage(0, false); /* Make sure it's written out */ ! SimpleLruWritePage(MultiXactOffsetCtl, slotno, NULL); Assert(!MultiXactOffsetCtl-shared-page_dirty[slotno]); LWLockRelease(MultiXactOffsetControlLock); --- 1454,1460 slotno = ZeroMultiXactOffsetPage(0, false); /* Make sure it's written out */ ! SimpleLruWritePage(MultiXactOffsetCtl, slotno); Assert(!MultiXactOffsetCtl-shared-page_dirty[slotno]); LWLockRelease(MultiXactOffsetControlLock); *** *** 1465,1471 BootStrapMultiXact(void) slotno = ZeroMultiXactMemberPage(0, false); /* Make sure it's written out */ ! SimpleLruWritePage(MultiXactMemberCtl, slotno, NULL); Assert(!MultiXactMemberCtl-shared-page_dirty[slotno]); LWLockRelease(MultiXactMemberControlLock); --- 1465,1471 slotno = ZeroMultiXactMemberPage(0, false); /* Make sure it's written out */ ! SimpleLruWritePage(MultiXactMemberCtl, slotno); Assert(!MultiXactMemberCtl-shared-page_dirty[slotno]); LWLockRelease(MultiXactMemberControlLock); *** *** 1986,1992 multixact_redo(XLogRecPtr lsn, XLogRecord *record) LWLockAcquire(MultiXactOffsetControlLock, LW_EXCLUSIVE); slotno = ZeroMultiXactOffsetPage(pageno, false); ! SimpleLruWritePage(MultiXactOffsetCtl, slotno, NULL); Assert(!MultiXactOffsetCtl-shared-page_dirty[slotno]); LWLockRelease(MultiXactOffsetControlLock); --- 1986,1992 LWLockAcquire(MultiXactOffsetControlLock, LW_EXCLUSIVE); slotno = ZeroMultiXactOffsetPage(pageno, false); ! SimpleLruWritePage(MultiXactOffsetCtl, slotno); Assert(!MultiXactOffsetCtl-shared-page_dirty[slotno]); LWLockRelease(MultiXactOffsetControlLock); *** *** 2001,2007 multixact_redo(XLogRecPtr lsn, XLogRecord *record) LWLockAcquire(MultiXactMemberControlLock, LW_EXCLUSIVE); slotno = ZeroMultiXactMemberPage(pageno, false); ! SimpleLruWritePage(MultiXactMemberCtl, slotno, NULL); Assert(!MultiXactMemberCtl-shared-page_dirty[slotno]); LWLockRelease(MultiXactMemberControlLock); --- 2001,2007 LWLockAcquire(MultiXactMemberControlLock, LW_EXCLUSIVE); slotno = ZeroMultiXactMemberPage(pageno, false); ! SimpleLruWritePage(MultiXactMemberCtl, slotno); Assert(!MultiXactMemberCtl-shared-page_dirty[slotno]); LWLockRelease(MultiXactMemberControlLock); *** a/src/backend/access/transam/slru.c --- b/src/backend/access/transam/slru.c *** *** 78,83 typedef struct SlruFlushData --- 78,85 int segno[MAX_FLUSH_BUFFERS]; /* their log seg#s */ } SlruFlushData; + typedef struct SlruFlushData *SlruFlush; + /* * Macro to mark a buffer slot most recently used. Note multiple evaluation * of arguments! *** *** 123,128 static int slru_errno; --- 125,131 static void SimpleLruZeroLSNs(SlruCtl ctl, int slotno); static void SimpleLruWaitIO(SlruCtl ctl, int slotno); + static
[HACKERS] SLRU overview
Is there an overview of SLRU anywhere? I've looked over the code enough to know that it'll save me a day or two if I can get an overview of correct usage, rather than to reverse engineer it from source code. There is no README with useful information, and searches of the Wiki and the docs have come up dry. Barring that, any tips or warnings welcome. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
