[PERFORM] Re: 回复: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Stefan Keller
Hi Wales 2012/2/27 Wales Wang wormw...@yahoo.com wrote: There are many approach for PostgreSQL in-memory. The quick and easy way is making slave pgsql run on persistent RAM filesystem, the slave is part of master/slave replication cluster. The fstab and script make RAM file system persistent

[PERFORM] Re: [PERFORM] Re: 回复: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Claudio Freire
On Tue, Feb 28, 2012 at 5:30 AM, Stefan Keller sfkel...@gmail.com wrote: But what I'm finally after is a solution, where records don't get pushed back to disk a.s.a.p. but rather got hold in memory as long as possible assuming that there is enough memory. fsync = off ? -- Sent via

[PERFORM] Re: [PERFORM] Re: [PERFORM] Re: 回复: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Tomas Vondra
On 28 Únor 2012, 14:08, Claudio Freire wrote: On Tue, Feb 28, 2012 at 5:30 AM, Stefan Keller sfkel...@gmail.com wrote: But what I'm finally after is a solution, where records don't get pushed back to disk a.s.a.p. but rather got hold in memory as long as possible assuming that there is enough

[PERFORM] Re: [PERFORM] Re: [PERFORM] Re: 回复: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Claudio Freire
On Tue, Feb 28, 2012 at 10:38 AM, Tomas Vondra t...@fuzzy.cz wrote: On 28 Únor 2012, 14:08, Claudio Freire wrote: On Tue, Feb 28, 2012 at 5:30 AM, Stefan Keller sfkel...@gmail.com wrote: But what I'm finally after is a solution, where records don't get pushed back to disk a.s.a.p. but rather

[PERFORM] Re: [PERFORM] Re: [PERFORM] Re: [PERFORM] Re: 回复: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Tomas Vondra
On 28 Únor 2012, 14:52, Claudio Freire wrote: On Tue, Feb 28, 2012 at 10:38 AM, Tomas Vondra t...@fuzzy.cz wrote: On 28 Únor 2012, 14:08, Claudio Freire wrote: On Tue, Feb 28, 2012 at 5:30 AM, Stefan Keller sfkel...@gmail.com wrote: But what I'm finally after is a solution, where records

[PERFORM] Re: [PERFORM] Re: [PERFORM] Re: [PERFORM] Re: 回复: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Claudio Freire
On Tue, Feb 28, 2012 at 11:15 AM, Tomas Vondra t...@fuzzy.cz wrote: I haven't investigated why exactly the data are not cached initially, but none of the options that I can think of could be fixed by setting fsync=off. That's something that influences writes (not read-only database) and I

[PERFORM] Re: 回复: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Jeff Janes
On Tue, Feb 28, 2012 at 12:30 AM, Stefan Keller sfkel...@gmail.com wrote: Thank you for the tipp. Making slave pgsql run on persistent RAM filesystem is surely at least a possibility which I'll try out. But what I'm finally after is a solution, where records don't get pushed back to disk

[PERFORM] Re: [PERFORM] Re: [PERFORM] Re: [PERFORM] Re: [PERFORM] Re: 回复: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Tomas Vondra
On 28 Únor 2012, 15:24, Claudio Freire wrote: On Tue, Feb 28, 2012 at 11:15 AM, Tomas Vondra t...@fuzzy.cz wrote: I haven't investigated why exactly the data are not cached initially, but none of the options that I can think of could be fixed by setting fsync=off. That's something that

[PERFORM] Re: [PERFORM] Re: [PERFORM] Re: [PERFORM] Re: [PERFORM] Re: 回复: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Claudio Freire
On Tue, Feb 28, 2012 at 1:05 PM, Tomas Vondra t...@fuzzy.cz wrote: On 28 Únor 2012, 15:24, Claudio Freire wrote: It speeds a lot more than the initial load of data. Assuming the database is read-only, but not the filesystem (ie: it's not a slave, in which case all this is moot, as you said,

Re: [PERFORM] Very long deletion time on a 200 GB database

2012-02-28 Thread David Kerr
On 02/27/2012 12:08 AM, Reuven M. Lerner wrote: Hi, everyone. I wanted to thank you again for your help on the huge delete problem that I was experiencing. After a lot of trial and error, we finally came to the conclusion that deleting this much data in the time frame that they need, on

[PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Jon Nelson
I happened to be looking in the PostgreSQL logs (8.4.10, x86_64, ScientificLinux 6.1) and noticed that an app was doing some sorting (group by, order by, index creation) that ended up on disk rather than staying in memory. So I enabled trace_sort and restarted the app. What followed confused me.

Re: [PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Tom Lane
Jon Nelson jnelson+pg...@jamponi.net writes: The config shows 128MB for work_mem and 2GB for maintenance_work_mem. Why does PostgreSQL /sometimes/ use the globally-configured values and sometimes use the values that come from the connection? You sure those log entries are all from the same

Re: [PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Jon Nelson
On Tue, Feb 28, 2012 at 1:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jon Nelson jnelson+pg...@jamponi.net writes: The config shows 128MB for work_mem and 2GB for maintenance_work_mem. Why does PostgreSQL /sometimes/ use the globally-configured values and sometimes use the values that come from

Re: [PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Tom Lane
Jon Nelson jnelson+pg...@jamponi.net writes: On Tue, Feb 28, 2012 at 1:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jon Nelson jnelson+pg...@jamponi.net writes: Why does PostgreSQL /sometimes/ use the globally-configured values and sometimes use the values that come from the connection? You

Re: [PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Jon Nelson
On Tue, Feb 28, 2012 at 2:51 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jon Nelson jnelson+pg...@jamponi.net writes: On Tue, Feb 28, 2012 at 1:28 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jon Nelson jnelson+pg...@jamponi.net writes: Why does PostgreSQL /sometimes/ use the globally-configured values

Re: [PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Kääriäinen Anssi
Quoting Jon Nelson: The config shows 128MB for work_mem and 2GB for maintenance_work_mem. Why does PostgreSQL /sometimes/ use the globally-configured values and sometimes use the values that come from the connection? Am I wrong in misunderstanding what 'session' variables are? I thought that

Re: [PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Tom Lane
Jon Nelson jnelson+pg...@jamponi.net writes: On Tue, Feb 28, 2012 at 2:51 PM, Tom Lane t...@sss.pgh.pa.us wrote: Hmph ... does seem a bit weird.  Can you turn on log_statements and identify which operations aren't using the session values? I had log_min_duration_statement = 1000. That's not

Re: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Claudio Freire
On Tue, Feb 28, 2012 at 5:48 PM, Stefan Keller sfkel...@gmail.com wrote: P.S. And yes, the database is aka 'read-only' and truncated and re-populated from scratch every night. fsync is off so I don't care about ACID. After the indexes on name, hstore and geometry are generated I do a VACUUM

Re: [PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Claudio Freire
On Tue, Feb 28, 2012 at 6:54 PM, Tom Lane t...@sss.pgh.pa.us wrote: ... which is the ORM library (SQLAlchemy) doing a reflection of the table(s) involved. Oh, there's an ORM involved?  I'll bet a nickel it's doing something surprising, like not issuing your SET until much later than you

Re: [PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Jon Nelson
On Tue, Feb 28, 2012 at 4:46 PM, Claudio Freire klaussfre...@gmail.com wrote: On Tue, Feb 28, 2012 at 6:54 PM, Tom Lane t...@sss.pgh.pa.us wrote: ... which is the ORM library (SQLAlchemy) doing a reflection of the table(s) involved. Oh, there's an ORM involved?  I'll bet a nickel it's doing

Re: [PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Claudio Freire
On Tue, Feb 28, 2012 at 8:00 PM, Jon Nelson jnelson+pg...@jamponi.net wrote: I cranked the logging /all/ the way up and isolated the server. I suspect that your theory is correct. Another option, depending on your SQLA version, when connections are sent back to the pool, I seem to remember they

Re: [PERFORM] PG as in-memory db? How to warm up and re-populate buffers? How to read in all tuples into memory?

2012-02-28 Thread Stefan Keller
2012/2/28 Claudio Freire klaussfre...@gmail.com: On Tue, Feb 28, 2012 at 5:48 PM, Stefan Keller sfkel...@gmail.com wrote: P.S. And yes, the database is aka 'read-only' and truncated and re-populated from scratch every night. fsync is off so I don't care about ACID. After the indexes on name,

Re: [PERFORM] problems with set_config, work_mem, maintenance_work_mem, and sorting

2012-02-28 Thread Ants Aasma
On Feb 29, 2012 1:44 AM, Claudio Freire klaussfre...@gmail.com wrote: Another option, depending on your SQLA version, when connections are sent back to the pool, I seem to remember they were reset. That would also reset the work_mem, you'd still see the same pid on PG logs, but it's not the