Re: [HACKERS] calculating an aspect of shared buffer state from a background worker
On Mon, Mar 10, 2014 at 2:09 PM, Robert Berry berrydigi...@gmail.com wrote: Is there a way to get access to the StrategyControl pointer in the context of a background worker? StrategyControl is inherent to freelist.c and has no external declaration so you could not have it even if you the BGWORKER_SHMEM_ACCESS flag. In order to calculate that, an idea could be to go through the array of BufferDescriptors and then get the information necessary. Locks are necessary when doing that if you want to get a consistent picture of the buffers. Perhaps more experienced people have better ideas though... Regards, -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] calculating an aspect of shared buffer state from a background worker
Robert Berry berrydigi...@gmail.com writes: I'm looking at doing a calculation to determine the number of free buffers available. A n example ratio that is based on some data structures in freelist.c as follows: (StrategyControl-lastFreeBuffer - StrategyControl-firstFreeBuffer) / (double) NBuffers Is there a way to get access to the StrategyControl pointer in the context of a background worker? The BufferStrategyControl struct is in shared memory, so you can certainly get at it. One way would be to modify freelist.c to export its static pointer variable. Alternatively, you could call ShmemInitStruct an extra time to look up the struct for yourself, and then save it in your own static variable. Having said that, though, I'm pretty dubious of the premise. I trust you realize that the above calculation is entirely wrong; firstFreeBuffer and lastFreeBuffer are list head and tail pointers, and have no numerical relation to the list length. The only way to determine the list length accurately would be to chase down the whole list, which you'd have to hold the BufFreelistLock while doing, which'd be disastrous for performance if the list was long. (If you're okay with modifying the backend code you could dodge this by teaching freelist.c to maintain a counter, I guess.) An even bigger issue is that it's not clear that the length of the free list is actually a useful number to have; in steady-state usage it frequently is always zero. Buffers only get put back on the freelist if they're invalidated, eg by dropping the relation they belonged to. Normal usage tends to allocate buffers by reclaiming ones whose usage_count has reached zero in the clock sweep algorithm. So a better picture of the availability of buffers would require scanning the buffer pool to see how many there are of each usage_count level. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] calculating an aspect of shared buffer state from a background worker
Thank you both for the thoughtful and helpful responses. The utility of the length of the free list is somewhat dubious. I imagine it could be useful to answer the question of is there a chance that increasing shared buffers would be useless? in an optimization context. Agreed it's not useful in most steady state scenarios. I saw the approach in the pg_buffercache contrib module and am looking for lockless alternatives for at least estimating the size of free buffers. I'm a relatively inexperienced so I'd be curious to know whether there is a danger beyond an inconsistent result in traversing / sampling the BufferDescriptors without a lock? Also I got the impression that there is a ring approach to freeing buffers, and that assuming the descriptors are allocated in sequential addresses, taking the difference in the first and last could be used to get a rough estimate accounting for sizes or other shenanigans? Thank you again, the clues to look at buffer descriptors and ShmemInitStruct are very helpful. Best Regards, Robert On Mon, Mar 10, 2014 at 7:33 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Berry berrydigi...@gmail.com writes: I'm looking at doing a calculation to determine the number of free buffers available. A n example ratio that is based on some data structures in freelist.c as follows: (StrategyControl-lastFreeBuffer - StrategyControl-firstFreeBuffer) / (double) NBuffers Is there a way to get access to the StrategyControl pointer in the context of a background worker? The BufferStrategyControl struct is in shared memory, so you can certainly get at it. One way would be to modify freelist.c to export its static pointer variable. Alternatively, you could call ShmemInitStruct an extra time to look up the struct for yourself, and then save it in your own static variable. Having said that, though, I'm pretty dubious of the premise. I trust you realize that the above calculation is entirely wrong; firstFreeBuffer and lastFreeBuffer are list head and tail pointers, and have no numerical relation to the list length. The only way to determine the list length accurately would be to chase down the whole list, which you'd have to hold the BufFreelistLock while doing, which'd be disastrous for performance if the list was long. (If you're okay with modifying the backend code you could dodge this by teaching freelist.c to maintain a counter, I guess.) An even bigger issue is that it's not clear that the length of the free list is actually a useful number to have; in steady-state usage it frequently is always zero. Buffers only get put back on the freelist if they're invalidated, eg by dropping the relation they belonged to. Normal usage tends to allocate buffers by reclaiming ones whose usage_count has reached zero in the clock sweep algorithm. So a better picture of the availability of buffers would require scanning the buffer pool to see how many there are of each usage_count level. regards, tom lane