Re: [HACKERS] Postgresql Caching

2006-10-17 Thread Anon Mous
Hi I've loved reading all of your thoughts and comments. Yet, I'm left with the question: Can we can brainstorm a caching solution that is workable... I've seen some posts talking about some of the challenges. 1.) Only good for static data As it was proposed that is largely true. This

Re: [HACKERS] Postgresql Caching

2006-10-17 Thread Jeff Davis
On Tue, 2006-10-17 at 16:51 -0700, Anon Mous wrote: Hi I've loved reading all of your thoughts and comments. Yet, I'm left with the question: Can we can brainstorm a caching solution that is workable... I think you're making this a little complicated. A lot of these problems can

Re: [HACKERS] Postgresql Caching

2006-10-16 Thread Shane Ambler
Harvell F wrote: Getting back to the original posting, as I remember it, the question was about seldom changed information. In that case, and assuming a repetitive query as above, a simple query results cache that is keyed on the passed SQL statement string and that simply returns the

Re: [HACKERS] Postgresql Caching

2006-10-16 Thread mark
On Sun, Oct 15, 2006 at 06:33:36PM -0700, Jeremy Drake wrote: 2) When updating a PostgreSQL record, I updated the memcache record to the new value. If another process comes along in parallel before I commit, that is still looking at an older view, cross-referencing

Re: [HACKERS] Postgresql Caching

2006-10-16 Thread mark
On Mon, Oct 16, 2006 at 05:59:05PM +0930, Shane Ambler wrote: Registering each cache entry by the tables included in the query and invalidating the cache during on a committed update or insert transaction to any of the tables would, transparently, solve the consistency problem. That was

Re: [HACKERS] Postgresql Caching

2006-10-16 Thread Markus Schaber
Hi, Shane, Shane Ambler wrote: CREATE TABLESPACE myramcache LOCATION MEMORY(2GB); It's already possible to do this, just create the TABLESPACE in a ramdisk / tmpfs or whatever is available for your OS. HTH, Markus -- Markus Schaber | Logical TrackingTracing International AG Dipl. Inf. |

Re: [HACKERS] Postgresql Caching

2006-10-16 Thread Merlin Moncure
On 10/15/06, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Using memcache, I've had problems with consistency brought right to the front. Both of these have failed me: 1) When updating a PostgreSQL record, I invalidate the memcache record. If another process comes along in parallel

Re: [HACKERS] Postgresql Caching

2006-10-16 Thread Harvell F
On 16 Oct 2006, at 4:29, Shane Ambler wrote: Harvell F wrote: Getting back to the original posting, as I remember it, the question was about seldom changed information. In that case, and assuming a repetitive query as above, a simple query results cache that is keyed on the passed

Re: [HACKERS] Postgresql Caching

2006-10-16 Thread Neil Conway
On Mon, 2006-10-16 at 13:59 +0200, Markus Schaber wrote: It's already possible to do this, just create the TABLESPACE in a ramdisk / tmpfs or whatever is available for your OS. This is not an ideal solution: if the machine reboots, the content of the tablespace will disappear, requiring manual

Re: [HACKERS] Postgresql Caching

2006-10-16 Thread mark
On Mon, Oct 16, 2006 at 12:40:44PM -0400, Neil Conway wrote: On Mon, 2006-10-16 at 13:59 +0200, Markus Schaber wrote: It's already possible to do this, just create the TABLESPACE in a ramdisk / tmpfs or whatever is available for your OS. This is not an ideal solution: if the machine reboots,

Re: [HACKERS] Postgresql Caching

2006-10-16 Thread Jeremy Drake
On Mon, 16 Oct 2006, [EMAIL PROTECTED] wrote: On Sun, Oct 15, 2006 at 06:33:36PM -0700, Jeremy Drake wrote: 2) When updating a PostgreSQL record, I updated the memcache record to the new value. If another process comes along in parallel before I commit, that is still

[HACKERS] Postgresql Caching

2006-10-15 Thread Anon Mous
Hi I may have a workable idea on a way to add caching to Postgres without disturbing the MVCC functionality. Caching, as I've been reading can provide an amazing and sometimes almost unbelievable performance boost to a database based application, especially for data that is rarely modified.

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread mark
On Sun, Oct 15, 2006 at 03:41:25AM -0700, Anon Mous wrote: However, the problem is surmountable and has been figured out by Oracle, although I don't know how they did it: http://www.oracle.com/technology/products/ias/joc/index.html I'm pretty sure this is application-side caching. The

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Merlin Moncure
On 10/15/06, Anon Mous [EMAIL PROTECTED] wrote: Would it be possible to combine a special memcache implementation of memcache with a Postgresql interface wrapper? have you seen http://people.freebsd.org/~seanc/pgmemcache/ merlin ---(end of

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread mark
On Sun, Oct 15, 2006 at 08:31:36PM +0530, Merlin Moncure wrote: On 10/15/06, Anon Mous [EMAIL PROTECTED] wrote: Would it be possible to combine a special memcache implementation of memcache with a Postgresql interface wrapper? have you seen http://people.freebsd.org/~seanc/pgmemcache/

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Alvaro Herrera
[EMAIL PROTECTED] wrote: Using memcache, I've had problems with consistency brought right to the front. Both of these have failed me: 1) When updating a PostgreSQL record, I invalidate the memcache record. If another process comes along in parallel before I commit, notices

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Andrew Dunstan
Alvaro Herrera wrote: [EMAIL PROTECTED] wrote: Using memcache, I've had problems with consistency brought right to the front. Both of these have failed me: 1) When updating a PostgreSQL record, I invalidate the memcache record. If another process comes along in parallel before I

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Shane Ambler
Merlin Moncure wrote: On 10/15/06, Anon Mous [EMAIL PROTECTED] wrote: Would it be possible to combine a special memcache implementation of memcache with a Postgresql interface wrapper? have you seen http://people.freebsd.org/~seanc/pgmemcache/ merlin Now you got me thinkin ;-P Just

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Lexington Luthor
[EMAIL PROTECTED] wrote: Using memcache, I've had problems with consistency brought right to the front. Both of these have failed me: 1) When updating a PostgreSQL record, I invalidate the memcache record. If another process comes along in parallel before I commit, notices

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread mark
On Sun, Oct 15, 2006 at 06:43:43PM +0100, Lexington Luthor wrote: [EMAIL PROTECTED] wrote: Using memcache, I've had problems with consistency brought right to the front. Both of these have failed me: 1) When updating a PostgreSQL record, I invalidate the memcache record. If another

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread mark
On Mon, Oct 16, 2006 at 03:08:39AM +0930, Shane Ambler wrote: Just throwing some ideas around - What if we could do something like CREATE TABLESPACE myramcache LOCATION MEMORY(2GB); CREATE TABLE mycache ( id as integer, data as varchar(50)) USING TABLESPACE myramcache; INSERT INTO mycache

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread mark
On Sun, Oct 15, 2006 at 06:43:43PM +0100, Lexington Luthor wrote: With a bit of careful planning (and a few SELECT FOR UPDATE queries to prevent deadlock), having perfect consistency and correct caching is possible. I didn't respond directly to this claim of yours. SELECT FOR UPDATE is only

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Shane Ambler
[EMAIL PROTECTED] wrote: On Mon, Oct 16, 2006 at 03:08:39AM +0930, Shane Ambler wrote: You could setup a table in memory to contain known popular data, you could also use this to create a temporary table in memory to speed up multiple intermediate calculations without touching disks. I'm

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Tom Lane
Shane Ambler [EMAIL PROTECTED] writes: [EMAIL PROTECTED] wrote: None of this avoids the cost of query planning, or query execution. No but you can avoid costly disk access and still have the postgres level of integrity and integration that memcached doesn't offer. If you're just trying to

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread mark
On Mon, Oct 16, 2006 at 05:14:59AM +0930, Shane Ambler wrote: [EMAIL PROTECTED] wrote: On Mon, Oct 16, 2006 at 03:08:39AM +0930, Shane Ambler wrote: You could setup a table in memory to contain known popular data, you could also use this to create a temporary table in memory to speed up

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Shane Ambler
[EMAIL PROTECTED] wrote: As a thought experiment, I'm not seeing the benefit. I think if you could prove a benefit, then any proof you provided could be used to improve the already existing caching layers, and would apply equally to read-only or read-write pages. For example, why not be able to

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Josh Berkus
Mark, Using memcache, I've had problems with consistency brought right to the front. Both of these have failed me: 1) When updating a PostgreSQL record, I invalidate the memcache record. If another process comes along in parallel before I commit, notices that the memcache

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread mark
On Sun, Oct 15, 2006 at 02:39:36PM -0700, Josh Berkus wrote: Using memcache, I've had problems with consistency brought right to the front. Both of these have failed me: 1) When updating a PostgreSQL record, I invalidate the memcache record. If another process comes along in

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread mark
On Mon, Oct 16, 2006 at 07:00:20AM +0930, Shane Ambler wrote: [EMAIL PROTECTED] wrote: As a thought experiment, I'm not seeing the benefit. I think if you could prove a benefit, then any proof you provided could be used to improve the already existing caching layers, and would apply equally

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Jeremy Drake
On Sun, 15 Oct 2006, [EMAIL PROTECTED] wrote: On Sun, Oct 15, 2006 at 08:31:36PM +0530, Merlin Moncure wrote: On 10/15/06, Anon Mous [EMAIL PROTECTED] wrote: Would it be possible to combine a special memcache implementation of memcache with a Postgresql interface wrapper? have you seen

Re: [HACKERS] Postgresql Caching

2006-10-15 Thread Harvell F
On 15 Oct 2006, at 19:55, [EMAIL PROTECTED] wrote: On Mon, Oct 16, 2006 at 07:00:20AM +0930, Shane Ambler wrote: [EMAIL PROTECTED] wrote: As a thought experiment, I'm not seeing the benefit. I think if you could prove a benefit, then any proof you provided could be used to improve the