Re: [PERFORM] Fast insert, but slow join and updates for table with 4 billion rows

2016-10-25 Thread Lars Aksel Opsahl
really fun to work with Postgres/Postgis open source software hold a very high quality. Thanks. Lars Fra: pgsql-performance-ow...@postgresql.org <pgsql-performance-ow...@postgresql.org> på vegne av Tom Lane <t...@sss.pgh.pa.us> Sendt: 24. okt

Re: [PERFORM] Fast insert, but slow join and updates for table with 4 billion rows

2016-10-24 Thread Lars Aksel Opsahl
? Thanks. Lars EXPLAIN analyze SELECT count(o.*) FROM met_vaer_wisline.nora_bc25_observation o WHERE o.epoch = 128844; -[ RECORD 1

[PERFORM] Fast insert, but slow join and updates for table with 4 billion rows

2016-10-24 Thread Lars Aksel Opsahl
unt(*) from met_vaer_wisline.new_data; count --- 5 (1 row) SELECT count(*) from met_vaer_wisline.nora_bc25_observation ; count 4263866304 Thanks . Lars

Re: [PERFORM] synchronous_commit off

2011-08-02 Thread lars hofhansl
No: The commit has the same guarantees as a synchronous commit w.r.t. data consistency. The commit can only fail (as a whole) due to hardware problems or postgres backend crashes. And yes: The client commit returns, but the server can fail later and not persist the transaction and it will be

Re: [PERFORM] Which Join is better

2011-08-02 Thread lars hofhansl
Unless you use the explicit join syntax: select p.* from A p join B q on (p.id = q.id) and also set  join_collapse_limit= 1 The order of the joins is determined by the planner. Also explain is your friend :) From: Adarsh Sharma adarsh.sha...@orkash.com To:

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-16 Thread lars
to their calls. regards, tom lane If you have a patch in mind I'm happy to test it on my setup and report back. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [PERFORM] Statistics and Multi-Column indexes

2011-07-15 Thread lars
tenants with relatively little data and a few with a lot of data. So the number of tenants is known ahead of time and might be 1000's. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-15 Thread lars
, but checkout that one expensive semop! 2s!! -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-13 Thread lars
to recompile with a patch applied, etc. Thanks. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-13 Thread lars
question -- what happens to the timings if your SELECTs are done with synchronous_commit = off? Just tried that... In that case the WAL is still written (as seen via iostat), but not synchronously by the transaction (as seen by strace). -- Lars -- Sent via pgsql-performance mailing list

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-13 Thread lars
synchronous_commit = on; = update test set created_by = '001' where tenant = '001'; UPDATE 3712 Time: 384.702 ms lars= select count(*) from test where tenant = '001' and created_date = '2011-6-30'; count --- 3712 (1 row) Time: 36.571 ms = select count(*) from

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread lars
On 07/12/2011 08:13 AM, Ivan Voras wrote: On 12/07/2011 02:09, lars wrote: Oh, and iowait hovers around 20% when SELECTs are slow: avg-cpu: %user %nice %system %iowait %steal %idle 1.54 0.00 0.98 18.49 0.07 78.92 When SELECTs are fast it looks like this: avg-cpu: %user %nice %system %iowait

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread lars
) for that IO to complete is making a fully cached database far less useful. I just artificially created this scenario. ... Just dropped the table to test something so I can't get the plan right now. Will send an update as soon as I get it setup again. Thanks again. -- Lars -- Sent via pgsql

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread lars
On 07/12/2011 01:04 PM, lars wrote: On 07/12/2011 12:08 PM, Kevin Grittner wrote: larslhofha...@yahoo.com wrote: select count(*) from test where tenant = $1 and created_date = $2 Ah, that might be a major clue -- prepared statements. What sort of a plan do you get for that as a prepared

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread lars
On 07/12/2011 02:51 PM, Kevin Grittner wrote: I ran x a bunch of times to get a baseline, then y once, then x a bunch more times. The results were a bit surprising: cir= \timing Timing is on. cir= execute x('001','2011-6-30'); count --- 3456 (1 row) Time: 9.823 ms cir=

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread lars
\377\377\0\0D..., 66, 0, NULL, 0) = 66 No writing to the WAL. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-12 Thread Lars
shared_buffers is big enough to hold the entire database, and there is plenty of extra space. (verified with PG_buffercache) So i don't think that is the reason. Tom Lane t...@sss.pgh.pa.us schrieb: Jeff Janes jeff.ja...@gmail.com writes: On 7/12/11, lars lhofha...@yahoo.com wrote

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-11 Thread lars
On 07/11/2011 10:33 AM, Kevin Grittner wrote: lars hofhansllhofha...@yahoo.com wrote: Yep, I am not seeing the SELECTs slow down (measurably) during checkpoints (i.e. when dirty pages are flushed to disk), but only during writing of the WAL files. How about if you do a whole slew

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-11 Thread lars
On 07/11/2011 02:43 PM, Merlin Moncure wrote: On Mon, Jul 11, 2011 at 2:16 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: larslhofha...@yahoo.com wrote: Stopping the UPDATEs, waiting for any CHECKPOINTs to finish, and then running the SELECTs indeed shows a similar slowdown.

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-11 Thread lars
On 07/11/2011 04:02 PM, lars wrote: On 07/11/2011 02:43 PM, Merlin Moncure wrote: On Mon, Jul 11, 2011 at 2:16 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: larslhofha...@yahoo.com wrote: Stopping the UPDATEs, waiting for any CHECKPOINTs to finish, and then running the SELECTs

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-11 Thread lars
) UPDATEs at all when the database resides in the cache completely. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

[PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-10 Thread lars
this behavior. Thanks. -- Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-10 Thread lars
insights, though :) Thanks. -- Lars On 07/07/2011 04:56 PM, lars wrote: I am doing some research that will hopefully lead to replacing a big Oracle installation with a set PostgreSQL servers. The current Oracle installations consists of multiple of RAC clusters with 8 RAC nodes each. Each RAC

[PERFORM] Statistics and Multi-Column indexes

2011-07-10 Thread lars
only setting n_distinct, i.e. set it low if the inner column is not selective within the context of a tenant and high otherwise. For various reasons #5 is also not an option. And of course the same set of questions comes up with joins. Thanks. -- Lars -- Sent via pgsql-performance mailing

Re: [PERFORM] UPDATEDs slowing SELECTs in a fully cached database

2011-07-10 Thread lars hofhansl
own drive, and then increased the effect I was seeing. I'll do more research and try to provide more useful details. Thanks for the pg_catalog link, I'll have a look at it. -- Lars - Original Message From: Craig Ringer cr...@postnewspapers.com.au To: pgsql-performance@postgresql.org

Re: [PERFORM] very long updates very small tables

2011-04-04 Thread Lars Feistner
On 03/30/2011 06:54 PM, Kevin Grittner wrote: Lars Feistnerfeist...@uni-heidelberg.de wrote: On 03/29/2011 09:28 PM, Kevin Grittner wrote: Lars Feistnerfeist...@uni-heidelberg.de wrote: The log tells me that certain update statements take sometimes about 3-10 minutes. But we are talking

Re: [PERFORM] very long updates very small tables

2011-03-30 Thread Lars Feistner
Hello Kevin, On 03/29/2011 09:28 PM, Kevin Grittner wrote: Lars Feistnerfeist...@uni-heidelberg.de wrote: The log tells me that certain update statements take sometimes about 3-10 minutes. But we are talking about updates on tables with 1000 to 1 rows and updates that are supposed

[PERFORM] very long updates very small tables

2011-03-29 Thread Lars Feistner
= 't', $2 = '31' LOG: duration: 124654.000 ms execute unnamed: UPDATE extjs_recentlist SET visible=$1 WHERE recentlist_id=$2 DETAIL: parameters: $1 = 't', $2 = '1362' LOG: process 3844 still waiting for ShareLock on transaction 74839 after 8000.000 ms Thanx in advance. Lars

Re: [PERFORM] Migrating to Postgresql and new hardware

2011-01-19 Thread Lars
, but I really wont compare apples in Canada to oranges in Japan. :-) Hehe /Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Migrating to Postgresql and new hardware

2011-01-19 Thread Lars
redundancy in case of disc failure. How do you handle this with fusionIO? Two mirrored cards? /Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Migrating to Postgresql and new hardware

2011-01-19 Thread Lars
data that is stored in a special shared database. /Lars -Ursprungligt meddelande- Från: mark [mailto:dvlh...@gmail.com] Skickat: den 19 januari 2011 05:10 Till: Lars Kopia: pgsql-performance@postgresql.org Ämne: RE: [PERFORM] Migrating to Postgresql and new hardware Comments in line, take

[PERFORM] Migrating to Postgresql and new hardware

2011-01-18 Thread Lars
the two alternatives would compare in performance running Postgresql? How would the hardware usage of Postgresql compare to MySqls? Regards /Lars -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [PERFORM] High update activity, PostgreSQL vs BigDBMS

2007-01-02 Thread Lars Heidieker
differences between ordered and meta data only journaling should be very small enyway - -- Viele Grüße, Lars Heidieker [EMAIL PROTECTED] http://paradoxon.info - Mystische Erklärungen. Die mystischen Erklärungen gelten für tief; die Wahrheit ist, dass sie noch