Re: [PERFORM] General key issues when comparing performance between PostgreSQL and oracle

2013-07-16 Thread Samuel Gendler
On Tue, Jul 16, 2013 at 9:51 AM, Brian Fehrle wrote: > > Are there any known differences between the database systems in terms of > query planners or general operations (sorting, unions) that are notable > different between the systems that would make postgres slow down when > executing the exact

Re: [PERFORM] Thousands databases or schemas

2012-11-08 Thread Samuel Gendler
On Thu, Nov 8, 2012 at 1:36 AM, Denis wrote: > > P.S. > Not to start a holywar, but FYI: in a similar project where we used MySQL > now we have about 6000 DBs and everything works like a charm. > You seem to have answered your own question here. If my recollection of a previous discussion about

Re: [PERFORM] Invalid memory alloc request size

2012-10-31 Thread Samuel Gendler
This was answered on the list last time you asked it. You are exceeding a maximum buffer size. There was an implication that it was related to converting a string from one encoding to another that could maybe be alleviated by using the same encoding in both client and server, but a more reliable so

Re: [PERFORM] have: seq scan - want: index scan

2012-10-17 Thread Samuel Gendler
On Tue, Oct 16, 2012 at 4:45 PM, Chris Ruprecht wrote: > Hi guys, > > PG = 9.1.5 > OS = winDOS 2008R8 > > I have a table that currently has 207 million rows. > there is a timestamp field that contains data. > more data gets copied from another database into this database. > How do I make this do

Re: [PERFORM] Scaling 10 million records in PostgreSQL table

2012-10-09 Thread Samuel Gendler
On Mon, Oct 8, 2012 at 1:25 PM, Navaneethan R wrote: > >After created the index for WHERE clause "WHERE dealer_id = 270001"..It > is performing better.I have more dealer ids Should I do it for each > dealer_id? > > All you've really done is confuse the issue. Please read the wiki page on how

Re: [PERFORM] Scaling 10 million records in PostgreSQL table

2012-10-08 Thread Samuel Gendler
On Mon, Oct 8, 2012 at 1:27 PM, Craig Ringer wrote: > > If you already have appropriate indexes and have used `explain analyze` to > verify that the query isn't doing anything slow and expensive, it's > possible the easiest way to improve performance is to set up async > replication or log shippi

Re: [PERFORM] Inaccurate Explain Cost

2012-09-26 Thread Samuel Gendler
On Wed, Sep 26, 2012 at 1:21 PM, hubert depesz lubaczewski < dep...@depesz.com> wrote: > On Wed, Sep 26, 2012 at 02:38:09PM -0400, Robert Sosinski wrote: > > The first query shows a cost of 190,169.55 and runs in 199,806.951 ms. > > When I disable nested loop, I get a cost of 2,535,992.34 which ru

Re: [PERFORM] Is drop/restore trigger transactional?

2012-08-07 Thread Samuel Gendler
On Tue, Aug 7, 2012 at 2:39 PM, Craig James wrote: > > Obviously this is a very expensive trigger, but one that we can drop > in a very specific circumstance. But we NEVER want to drop it for > everyone. It seems like a very reasonable use-case to me. > > Sounds like you should try doing the wo

Re: [PERFORM] very very slow inserts into very large table

2012-07-16 Thread Samuel Gendler
On Mon, Jul 16, 2012 at 7:06 AM, Mark Thornton wrote: > >> Every insert updates four indexes, so at least 3 of those will be in > random order. The indexes don't fit in memory, so all those updates will > involve reading most of the relevant b-tree pages from disk (or at least > the leaf level).

Re: [PERFORM] The need for clustered indexes to boost TPC-V performance

2012-07-05 Thread Samuel Gendler
On Thu, Jul 5, 2012 at 1:37 PM, Reza Taheri wrote: > > I provided more config details in an earlier email. > > ** ** > > > I hate to disagree, but unless I didn't get a message sent to the list, you haven't provided any details about your postgresql config or otherwise adhered to the guidelin

Re: [PERFORM] The need for clustered indexes to boost TPC-V performance

2012-07-05 Thread Samuel Gendler
On Thu, Jul 5, 2012 at 12:13 PM, Reza Taheri wrote: > Hi Robert, > Yes, the same concept. Oracle's IOT feature is used often with TPC > benchmarks. > > Reza, it would be very helpful if you were to provide the list with a lot more information about your current software and hardware configuration

Re: [PERFORM] Expected performance of querying 5k records from 4 million records?

2012-06-18 Thread Samuel Gendler
On Mon, Jun 18, 2012 at 9:39 AM, Anish Kejariwal wrote: > > So, it looks like clustering the index appropriately fixes things! Also, > I'll recreate the index switching the order to (dataset_id, stat_id,data_id) > > Just keep in mind that clustering is a one-time operation. Inserts and updates

Re: [PERFORM] Parallel Scaling of a pgplsql problem

2012-04-25 Thread Samuel Gendler
On Wed, Apr 25, 2012 at 11:52 AM, Venki Ramachandran < venki_ramachand...@yahoo.com> wrote: > Hi all: > Can someone please guide me as to how to solve this problem? If this is > the wrong forum, please let me know which one to post this one in. I am new > to Postgres (about 3 months into it) > > I

Re: [PERFORM] TCP Overhead on Local Loopback

2012-04-02 Thread Samuel Gendler
On Sun, Apr 1, 2012 at 6:11 PM, Andrew Dunstan wrote: > > > On 04/01/2012 08:29 PM, Claudio Freire wrote: > >> On Sun, Apr 1, 2012 at 8:54 PM, Andrew Dunstan >> wrote: >> >>> You could try using Unix domain socket and see if the performance improves. A relevant link: >>> >>> He said Wi

Re: [PERFORM] Very long deletion time on a 200 GB database

2012-02-27 Thread Samuel Gendler
On Mon, Feb 27, 2012 at 6:59 AM, Reuven M. Lerner wrote: > > So for now, we'll just try to DELETE faster than we INSERT, and combined > with autovacuum, I'm hoping that this crisis will be averted. That said, > the current state of affairs with these machines is pretty fragile, and I > think that

Re: [PERFORM] Very long deletion time on a 200 GB database

2012-02-24 Thread Samuel Gendler
On Thu, Feb 23, 2012 at 10:39 PM, Reuven M. Lerner wrote: > Hi, everyone. > > So it turns out that we're not using 25 GB of virtual memory. (That's > what I had been shown yesterday, and it was a bit surprising, to say the > least...) > > A few statistics that I managed to get from the Windows de

Re: [PERFORM] Indexes and Primary Keys on Rapidly Growing Tables

2012-02-21 Thread Samuel Gendler
On Tue, Feb 21, 2012 at 9:59 AM, Alessandro Gagliardi wrote: > I was thinking about that (as per your presentation last week) but my > problem is that when I'm building up a series of inserts, if one of them > fails (very likely in this case due to a unique_violation) I have to > rollback the enti

Re: [PERFORM] Duplicate deletion optimizations

2012-01-06 Thread Samuel Gendler
On Fri, Jan 6, 2012 at 12:22 PM, Marc Eberhard wrote: > Hi Samuel! > > On 6 January 2012 20:02, Samuel Gendler wrote: > > Have you considered doing the insert by doing a bulk insert into a temp > > table and then pulling rows that don't exist across to the final

Re: [PERFORM] Duplicate deletion optimizations

2012-01-06 Thread Samuel Gendler
On Fri, Jan 6, 2012 at 6:35 AM, wrote: > Hello, > > I've a table with approximately 50 million rows with a schema like this: > >id bigint NOT NULL DEFAULT nextval('stats_5mn'::regclass)**, >t_value integer NOT NULL DEFAULT 0, >t_record integer NOT NULL DEFAULT 0, >output_id intege

Re: [PERFORM] Performance question 83 GB Table 150 million rows, distinct select

2011-11-16 Thread Samuel Gendler
On Wed, Nov 16, 2011 at 3:32 PM, Scott Marlowe wrote: > > If the OP's considering partitioning, they should really consider > upgrading to 9.1 which has much better performance of things like > aggregates against partition tables. > > Could you elaborate on this a bit, or point me at some docs? I

Re: [PERFORM] Error while vacuuming

2011-11-07 Thread Samuel Gendler
On Mon, Nov 7, 2011 at 10:33 PM, Bhakti Ghatkar wrote: > Tom, > > Currently we are using version 9.0.1. > > Which version shall we update to? 9.05 or 9.1 ? > 9.0.5 should be compatible with your installed db and contain any bug fixes that have been released. Which isn't to say that you shouldn'

Re: [PERFORM] backups blocking everything

2011-10-28 Thread Samuel Gendler
On Fri, Oct 28, 2011 at 2:20 PM, Merlin Moncure wrote: > > hrm -- it doesn't look like you are i/o bound -- postgres is > definitely the bottleneck. taking a dump off of production is > throwing something else out of whack which is affecting your other > processes. > > band aid solutions might b

Re: [PERFORM] backups blocking everything

2011-10-27 Thread Samuel Gendler
On Thu, Oct 27, 2011 at 2:15 PM, Samuel Gendler wrote: > > > There are definitely no bloated tables. The large tables are all > insert-only, and old data is aggregated up and then removed by dropping > whole partitions. There should be no bloat whatsoever. The OLTP side of >

Re: [PERFORM] backups blocking everything

2011-10-27 Thread Samuel Gendler
On Thu, Oct 27, 2011 at 1:45 PM, Nicholson, Brad (Toronto, ON, CA) < bnichol...@hp.com> wrote: > >From: pgsql-performance-ow...@postgresql.org [mailto: > pgsql-performance-ow...@postgresql.org] On Behalf Of Samuel Gendler > >Sent: Thursday, October 27, 2011 12:47 PM >

[PERFORM] backups blocking everything

2011-10-27 Thread Samuel Gendler
I've got a large mixed-used database, with the data warehouse side of things consisting of several tables at hundreds of millions of rows, plus a number of tables with tens of millions. There is partitioning, but as the volume of data has risen, individual partitions have gotten quite large. Hard

Re: [PERFORM] Adding more memory = hugh cpu load

2011-10-10 Thread Samuel Gendler
On Mon, Oct 10, 2011 at 1:52 PM, Kevin Grittner wrote: > alexandre - aldeia digital wrote: > > > I came to the list to see if anyone else has experienced the same > > problem > > A high load average or low idle CPU isn't a problem, it's a > potentially useful bit of information in diagnosing a p

Re: [PERFORM] Performance Anomaly with "col in (A,B)" vs. "col = A OR col = B" ver. 9.0.3

2011-09-26 Thread Samuel Gendler
On Mon, Sep 26, 2011 at 2:11 PM, Timothy Garnett wrote: > > Though maybe in a lot of common use situations people only supply values > that are known present so maybe this would make things worse more often then > better (maybe limit 1 or better EXISTS would be a hint the value is not > known pres

Re: [PERFORM] raid array seek performance

2011-09-13 Thread Samuel Gendler
On Tue, Sep 13, 2011 at 12:13 PM, Samuel Gendler wrote: > I'm just beginning the process of benchmarking and tuning a new server. > Something I really haven't done before. I'm using Greg's book as a guide. > I started with bonnie++ (1.96) and immediately got

[PERFORM] raid array seek performance

2011-09-13 Thread Samuel Gendler
I'm just beginning the process of benchmarking and tuning a new server. Something I really haven't done before. I'm using Greg's book as a guide. I started with bonnie++ (1.96) and immediately got anomalous results (I think). Hardware is as follows: 2x quad core xeon 5504 2.0Ghz, 2x4MB cache 1

Re: [PERFORM] Need to tune for Heavy Write

2011-08-04 Thread Samuel Gendler
On Wed, Aug 3, 2011 at 9:56 PM, Adarsh Sharma wrote: > Dear all, > > From the last few days, I researched a lot on Postgresql Performance Tuning > due to slow speed of my server. > My application selects data from mysql database about 10 rows , process > it & insert into postgres 2 tables by m

Re: [PERFORM] cpu comparison

2011-07-18 Thread Samuel Gendler
I'm just top posting this because this whole thread needs a reset before it goes any farther. Start with a real description of these hosts - Number and types of disks, filesystem configs, processors, memory, OS, etc. If your db is small enough to fit into RAM, please show us the db config you are

Re: [PERFORM] Statistics and Multi-Column indexes

2011-07-10 Thread Samuel Gendler
On Sun, Jul 10, 2011 at 2:16 PM, lars wrote: > I know this has been discussed various times... > > We are maintaining a large multi tenant database where *all* tables have a > tenant-id and all indexes and PKs lead with the tenant-id. > Statistics and counts for the all other columns are only rea

Re: [PERFORM] query total time im milliseconds

2011-07-10 Thread Samuel Gendler
On Sun, Jul 10, 2011 at 4:41 AM, Radhya sahal wrote: > Dear all , > could any one help me? > when i use pgadmin to exceute a query it shows the total time for query .. > such as > (select * form table_name.)query total time is for example 100 ms > i want to know the command that can reten

Re: [PERFORM] Query in 9.0.2 not using index in 9.0.0 works fine

2011-07-06 Thread Samuel Gendler
On Wed, Jul 6, 2011 at 5:50 AM, Matthias Howell wrote: > > > However, in the end, it was user brain damage. > > ** ** > > It does use the doc id index for the subquery, but for some reason, the > primary key on sentences - the sentenceid - was not set. So in fact, there > is no index. > >

Re: [PERFORM] Query in 9.0.2 not using index in 9.0.0 works fine

2011-07-06 Thread Samuel Gendler
On Tue, Jul 5, 2011 at 1:50 PM, Matthias Howell wrote: > I've just copied a database from one linux machine to another. > > "Fast" machine is CentOS 5.5, running postgres 9.0.0 64 bit > > ** ** > > "Slow" machine is Red Hat 5.5 running postgres 9.0.2 64 bit. > > ** ** > > Here's the q

Re: [PERFORM] near identical queries have vastly different plans

2011-07-01 Thread Samuel Gendler
On Fri, Jul 1, 2011 at 3:46 PM, Tom Lane wrote: > Samuel Gendler writes: > > I've got 2 nearly identical queries that perform incredibly differently. > > The reason the slow query sucks is that the planner is estimating at > most one "s" row will match that co

Re: [PERFORM] near identical queries have vastly different plans

2011-06-30 Thread Samuel Gendler
On Thu, Jun 30, 2011 at 1:53 AM, Samuel Gendler wrote: > If I could figure out either a query structure or an index structure which > will force the fast query plan, I'd be much happier. So that is what I am > looking for - an explanation of how I might convince the planner to al

Re: [PERFORM] near identical queries have vastly different plans

2011-06-30 Thread Samuel Gendler
On Thu, Jun 30, 2011 at 1:53 AM, Samuel Gendler wrote: > If I could figure out either a query structure or an index structure which > will force the fast query plan, I'd be much happier. So that is what I am > looking for - an explanation of how I might convince the planner to al

[PERFORM] near identical queries have vastly different plans

2011-06-30 Thread Samuel Gendler
Here's the setup: I'm cross joining two dimensions before left outer joining to a fact table so that I can throw a default value into the resultset wherever a value is missing from the fact table. I have a time dimension and another dimension. I want the cross join to only cross a subset of rows

Re: [PERFORM] generating a large XML document

2011-06-20 Thread Samuel Gendler
On Sun, Jun 19, 2011 at 11:36 PM, Julius Tuskenis wrote: > Hello, > > I'm sorry to write again, but as I received no answer I wonder if there is > a better mailing list to address concerning this question? Or is there > nothing to be done about the speed of xmlagg ?. Please let me as no answer >

Re: [PERFORM] Large rows number, and large objects

2011-06-19 Thread Samuel Gendler
On Sat, Jun 18, 2011 at 9:06 PM, Jose Ildefonso Camargo Tolosa < ildefonso.cama...@gmail.com> wrote: > Greetings, > > I have been thinking a lot about pgsql performance when it is dealing > with tables with lots of rows on one table (several millions, maybe > thousands of millions). Say, the Larg

Re: [PERFORM] poor performance when recreating constraints on large tables

2011-06-09 Thread Samuel Gendler
On Wed, Jun 8, 2011 at 10:57 PM, Greg Smith wrote: > Samuel Gendler wrote: > >> Sure, but if it is a query that is slow enough for a time estimate to be >> useful, odds are good that stats that are that far out of whack would >> actually be interesting to whoever is look

Re: [PERFORM] poor performance when recreating constraints on large tables

2011-06-08 Thread Samuel Gendler
On Wed, Jun 8, 2011 at 12:53 PM, Kevin Grittner wrote: > Samuel Gendler wrote: > > > The planner knows how many rows are expected for each step of the > > query plan, so it would be theoretically possible to compute how > > far along it is in processing a query

Re: [PERFORM] poor performance when recreating constraints on large tables

2011-06-08 Thread Samuel Gendler
On Wed, Jun 8, 2011 at 12:28 PM, Robert Haas wrote: > On Mon, Jun 6, 2011 at 6:10 PM, Mike Broers wrote: > > Thanks for the suggestion, maintenance_work_mem is set to the default of > > 16MB on the host that was taking over an hour as well as on the host that > > was taking less than 10 minutes.

Re: [PERFORM] Oracle v. Postgres 9.0 query performance

2011-06-08 Thread Samuel Gendler
On Wed, Jun 8, 2011 at 12:03 PM, Tony Capobianco wrote: > My current setting is 22G. According to some documentation, I want to > set effective_cache_size to my OS disk cache + shared_buffers. In this > case, I have 4 quad-core processors with 512K cache (8G) and my > shared_buffers is 7680M.

Re: [PERFORM] picking a filesystem

2011-05-31 Thread Samuel Gendler
On Tue, May 31, 2011 at 8:35 AM, Robert Haas wrote: > > So if you're running a RHEL5.4 or RHEL5.5 system, are you basically > stuck with ext3? I'm not sure if I'm remembering correctly, but ISTM > that you've been uncomfortable with BOTH ext4 and XFS prior to RHEL6; > but OK with both beginning

Re: [PERFORM] The shared buffers challenge

2011-05-26 Thread Samuel Gendler
On Thu, May 26, 2011 at 4:10 PM, Greg Smith wrote: > > As for figuring out how this impacts more complicated cases, I hear > somebody wrote a book or something that went into pages and pages of detail > about all this. You might want to check it out. > > I was just going to suggest that there wa

Re: [PERFORM] SORT performance - slow?

2011-05-19 Thread Samuel Gendler
Plus the entire explain analyze output into the form at http://explain.depesz.com/ and you'll get a nicely human readable output which shows both the inclusive and exclusive time spent on each step of the query. It also highlights any steps which show inaccurate statistics. It will also give you

Re: [PERFORM] setting configuration values inside a stored proc

2011-05-13 Thread Samuel Gendler
On Fri, May 13, 2011 at 1:28 AM, Andres Freund wrote: > Hi, > > On Friday, May 13, 2011 01:10:19 AM Samuel Gendler wrote: > > I would suggest doing ALTER FUNCTION blub(blarg) SET work_mem = '512MB'; > > Ah! That's perfect and very convenient. Thanks. --sam

[PERFORM] setting configuration values inside a stored proc

2011-05-12 Thread Samuel Gendler
I've got a stored proc that constructs some aggregation queries as strings and then executes them. I'd like to be able to increase work_mem before running those queries. If I set a new value for work_mem within the stored proc prior to executing my query string, will that actually have an impact

Re: [PERFORM] stored proc and inserting hundreds of thousands of rows

2011-04-30 Thread Samuel Gendler
On Sat, Apr 30, 2011 at 5:12 PM, Jeff Janes wrote: > > > gist indices are designed to make this type of thing fast, by using > techniques to rule out most of those comparisons without actually > performing them. I don't know enough about the > guts of either your distance function or the gist in

Re: [PERFORM] reducing random_page_cost from 4 to 2 to force index scan

2011-04-28 Thread Samuel Gendler
On Wed, Apr 27, 2011 at 5:19 PM, Sok Ann Yap wrote: > On Thu, Apr 28, 2011 at 7:23 AM, Kevin Grittner > > > I understand the need to tune PostgreSQL properly for my use case. > What I am curious about is, for the data set I have, under what > circumstances (hardware/workload/cache status/etc) wou

Re: [PERFORM] index usage on queries on inherited tables

2011-04-27 Thread Samuel Gendler
On Wed, Apr 27, 2011 at 2:11 PM, Joseph Shraibman wrote: > On 04/27/2011 04:32 PM, Robert Haas wrote: > > In the first case, PostgreSQL evidently thinks that using the indexes > > will be slower than just ignoring them. You could find out whether > > it's right by trying it with enable_seqscan=of

Re: [PERFORM] Is there a way to selective dump of records in Postgres 9.0.3?

2011-04-18 Thread Samuel Gendler
On Mon, Apr 18, 2011 at 8:11 AM, Nikolas Everett wrote: > This probably isn't the right place to ask that question but you may as > well try `pg_dump -t PATTERN`. Man pg_dump for more information on how to > form that pattern. > > > On Mon, Apr 18, 2011 at 11:05 AM, Sethu Prasad > wrote: > >> H

Re: [PERFORM] COPY with high # of clients, partitioned table locking issues?

2011-03-30 Thread Samuel Gendler
On Wed, Mar 30, 2011 at 5:48 PM, Bob Lunney wrote: > John, > > Sorry to hear you're struggling with such underpowered hardware. ;-) A > little more information would be helpful, though: > > 1. What version of PG are you running? > 2. What are the constraints on the child tables? > 3. How man

Re: [PERFORM] multiple table scan performance

2011-03-29 Thread Samuel Gendler
On Tue, Mar 29, 2011 at 5:05 PM, Marti Raudsepp wrote: > On Wed, Mar 30, 2011 at 01:16, Samuel Gendler > wrote: > > You can trick Postgres (8.3.x and newer) into doing it in parallel > anyway: open 3 separate database connections and issue each of these > 'INSERT

[PERFORM] multiple table scan performance

2011-03-29 Thread Samuel Gendler
I've got some functionality that necessarily must scan a relatively large table. Even worse, the total workload is actually 3 similar, but different queries, each of which requires a table scan. They all have a resultset that has the same structure, and all get inserted into a temp table. Is the

Re: [PERFORM] Reason of Slowness of query

2011-03-23 Thread Samuel Gendler
On Tue, Mar 22, 2011 at 11:28 PM, Adarsh Sharma wrote: > * > *I perform a join query on it as : > > * explain analyze select distinct(p.crawled_page_id) from page_content p , > clause2 c where p.crawled_page_id != c.source_id ; > > *What it takes more than 1 hour to complete. As I issue the expla

Re: [PERFORM] Anyone tried Flashcache with PostgreSQL?

2011-03-02 Thread Samuel Gendler
On Wed, Mar 2, 2011 at 7:29 AM, Robert Haas wrote: > On Mon, Feb 28, 2011 at 2:09 PM, Josh Berkus wrote: > > Does anyone have the hardware to test FlashCache with PostgreSQL? > > > > http://perspectives.mvdirona.com/2010/04/29/FacebookFlashcache.aspx > > > > I'd be interested to hear how it perf

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Samuel Gendler
Neat. That was my 'you learn something every day' moment. Thanks. On Thu, Feb 3, 2011 at 9:06 PM, David Wilson wrote: > > > On Thu, Feb 3, 2011 at 6:05 PM, Robert Haas wrote: > >> >> If you want to randomly pick 10,000 rows out of all the rows that are >> going to be inserted in the table with

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-01 Thread Samuel Gendler
On Tue, Feb 1, 2011 at 7:40 PM, Mladen Gogala wrote: > Samuel Gendler wrote: > >> >> Don't listen to him. He's got an oracle bias. >> > And bad sinuses, too. > > Slashdot already announced that NoSQL is actually going to dominate the >>

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-01 Thread Samuel Gendler
On Tue, Feb 1, 2011 at 3:44 PM, Mladen Gogala wrote: > On 2/1/2011 6:03 PM, Andrew Dunstan wrote: > >> Whether or not it's bad application design, it's ubiquitous, and we >> should make it work as best we can, IMNSHO. This often generates >> complaints about Postgres, and if we really plan for wor

Re: [PERFORM] Wrong docs on wal_buffers?

2011-01-07 Thread Samuel Gendler
On Fri, Jan 7, 2011 at 7:07 AM, Tom Lane wrote: > Samuel Gendler writes: > > Does it not seem that this insistence on shipping a default config that > > works out of the box on every system incurs a dramatic penalty when it > comes > > to getting a useful postgres config

Re: [PERFORM] Wrong docs on wal_buffers?

2011-01-07 Thread Samuel Gendler
On Thu, Jan 6, 2011 at 8:37 PM, Greg Smith wrote: > Josh Berkus wrote: > >> We talked about bumping it to 512kB or 1MB for 9.1. Did that get in? >> Do I need to write that patch? >> >> > > If it defaulted to 3% of shared_buffers, min 64K & max 16MB for the auto > setting, it would for the most p

Re: [PERFORM] Query Performance SQL Server vs. Postgresql

2010-11-22 Thread Samuel Gendler
On Sun, Nov 21, 2010 at 10:21 PM, Humair Mohammed wrote: > > Correct, the optimizer did not take the settings with the pg_ctl reload > command. I did a pg_ctl restart and work_mem now displays the updated value. > I had to bump up all the way to 2047 MB to get the response below (with > work_mem a

Re: [PERFORM] Simple (hopefully) throughput question?

2010-11-05 Thread Samuel Gendler
On Thu, Nov 4, 2010 at 8:07 AM, Vitalii Tymchyshyn wrote: > 04.11.10 16:31, Nick Matheson написав(ла): > > Heikki- >> >>> >>> Try COPY, ie. "COPY bulk_performance.counts TO STDOUT BINARY". >>> >>> Thanks for the suggestion. A preliminary test shows an improvement >> closer to our expected 35 MB

Re: [PERFORM] Simple (hopefully) throughput question?

2010-11-05 Thread Samuel Gendler
On Fri, Nov 5, 2010 at 12:23 PM, Samuel Gendler wrote: > On Thu, Nov 4, 2010 at 8:07 AM, Vitalii Tymchyshyn wrote: > >> 04.11.10 16:31, Nick Matheson написав(ла): >> >> Heikki- >>> >>>> >>>> Try COPY, ie. "COPY bulk_performance.cou

Re: [PERFORM] Index scan is not working, why??

2010-10-21 Thread Samuel Gendler
please provide non-default config options on this host plus the same from a host which is using an index scan, please. Also, postgresql version, OS, and all of the other stuff that is asked for in this document: http://wiki.postgresql.org/wiki/SlowQueryQuestions. It is impossible to say why the q

Re: [PERFORM] No hash join across partitioned tables?

2010-10-17 Thread Samuel Gendler
On Sat, Oct 16, 2010 at 8:29 AM, Alvaro Herrera wrote: > Excerpts from Samuel Gendler's message of sáb oct 16 02:35:46 -0300 2010: > > > An issue with automatically analyzing the entire hierarchy is 'abstract' > > table definitions. I've got a set of tables for storing the same data at > > differ

Re: [PERFORM] No hash join across partitioned tables?

2010-10-15 Thread Samuel Gendler
On Fri, Oct 15, 2010 at 10:22 PM, Tom Lane wrote: > Alvaro Herrera writes: > > If we analyze the parent, do we also update the children stats, or is it > > just that we keep two stats for the parent, one with children and one > > without, both being updated when the parent is analyzed? > > The l

Re: [PERFORM] oracle to psql migration - slow query in postgres

2010-10-15 Thread Samuel Gendler
On Thu, Oct 14, 2010 at 8:59 PM, Mladen Gogala wrote: > If working with partitioning, be very aware that PostgreSQL optimizer has > certain problems with partitions, especially with group functions. If you > want speed, everything must be prefixed with partitioning column: indexes, > expressions,

Re: [PERFORM] Query much faster with enable_seqscan=0

2010-10-12 Thread Samuel Gendler
On Tue, Sep 21, 2010 at 4:30 PM, Tom Lane wrote: > Ogden writes: > > SELECT tr.id, tr.sid > > FROM > > test_registration tr, > > INNER JOIN test_registration_result r on (tr.id = > r.test_registration_id) > > WHERE. > > > tr.test_administration_id=

[PERFORM] bulk load performance question

2010-10-12 Thread Samuel Gendler
I've got a few tables that periodically get entirely refreshed via a COPY. I don't really have a good mechanism for detecting only rows which have changed so even though the differences are small, a full copy is easiest. However, the data includes a primary key column, so I can't simply load into

Re: [PERFORM] Slow count(*) again...

2010-10-12 Thread Samuel Gendler
On Tue, Oct 12, 2010 at 9:02 AM, Scott Carey wrote: > > > However, for large reporting queries and sequential scans, XFS will win in > the long run if you use the online defragmenter. Otherwise, your sequential > scans won't be all that sequential on any file system over time if your > tables aren

Re: [PERFORM] Slow count(*) again...

2010-10-11 Thread Samuel Gendler
On Mon, Oct 11, 2010 at 9:06 PM, Scott Carey wrote: > I can't speak to documentation, but it is something that helps as your I/O > subsystem gets more powerful, and how much it helps depends more on your > hardware, which may have adaptive read ahead on its own, and your file > system which may be

Re: [PERFORM] Slow count(*) again...

2010-10-11 Thread Samuel Gendler
On Mon, Oct 11, 2010 at 7:19 PM, Greg Smith wrote: > > > This is a problem for the operating system to solve, and such solutions out > there are already good enough that PostgreSQL has little reason to try and > innovate in this area. I routinely see seq scan throughput double on Linux > just by

Re: [PERFORM] Slow count(*) again...

2010-10-11 Thread Samuel Gendler
On Mon, Oct 11, 2010 at 12:54 PM, Neil Whelchel wrote: > > 2. You need a slice of the data which requires another scan to the table to > get, and using the same WHERE clause as above. This seems like a total > waste, > because we just did that with the exception of actually fetching the data. > >

Re: [PERFORM] Slow count(*) again...

2010-10-09 Thread Samuel Gendler
On Sat, Oct 9, 2010 at 7:44 PM, Mladen Gogala wrote: > The architects of Postgres database would be well advised to operate under > the assumption that every production database has a competent DBA keeping an > eye on the database. > I'd actually go so far as to say that they have already made t

Re: [PERFORM] How does PG know if data is in memory?

2010-10-01 Thread Samuel Gendler
2010/10/1 Fabrício dos Anjos Silva >Craig, > >I agree with you. Not completely, but I do. > >I'm just stuck in a situation where I can't figure out what values to > use for the parameters. I can't even think of a way on how to test and > discover that. >I followed Josh Berkus' GUC

Re: [PERFORM] How does PG know if data is in memory?

2010-09-29 Thread Samuel Gendler
2010/9/29 Fabrício dos Anjos Silva > > >When setting seq_page_cost and random_page_cost, do I have to consider > the probability that data will be in memory? Or does seq_page_cost mean > "sequential access on disk" and random_page_cost mean "random access on > disk"? > > The reason seq_page_c

Re: [PERFORM] Is disableing nested_loops a bad idea ?

2010-09-16 Thread Samuel Gendler
Without knowing more about your queries and table structure, it is hard to say if there is a better solution. But one thing you should probably consider doing is just finding the queries where disabling nested loops is verifiably effective and then just disabling nested loops on that connection be

Re: [PERFORM] in-memory sorting

2010-08-19 Thread Samuel Gendler
On Thu, Aug 19, 2010 at 12:06 AM, Samuel Gendler wrote: > Incidentally, if I set values on the connection before querying, is there > an easy way to get things back to default values or will my code need to > know the prior value and explicitly set it back? Something like > >

Re: [PERFORM] in-memory sorting

2010-08-19 Thread Samuel Gendler
27;s data, and all the queries are slow instead of > one. Keep an eye on how high work_mem affects your kernel cache. > > On Thu, Aug 19, 2010 at 12:38 AM, Samuel Gendler > wrote: > > Yeah, although with 48GB of available memory and not that much > concurrency, > > I'

Re: [PERFORM] yet another q

2010-08-18 Thread Samuel Gendler
wizard 2010-08-17 cpu_tuple_cost = 0.0030 # sam cpu_index_tuple_cost = 0.0010 # sam cpu_operator_cost = 0.0005 # sam random_page_cost = 2.0 # sam On Wed, Aug 18, 2010 at 11:50 PM, Samuel Gendler wrote: > On Wed, Aug 18, 2010 at 11:14

Re: [PERFORM] yet another q

2010-08-18 Thread Samuel Gendler
On Wed, Aug 18, 2010 at 11:14 PM, Samuel Gendler wrote: > Please forgive the barrage of questions. I'm just learning how to tune > things in postgres and I've still got a bit of learning curve to get over, > apparently. I have done a lot of reading, though, I swear. > &g

Re: [PERFORM] in-memory sorting

2010-08-18 Thread Samuel Gendler
8, 2010 at 11:45 PM, Samuel Gendler > wrote: > > Answered my own question. Cranking work_mem up to 350MB revealed that > > the in-memory sort requires more memory than the disk sort. > > Note that unless you run VERY few client connections, it's usually > better to

[PERFORM] yet another q

2010-08-18 Thread Samuel Gendler
Please forgive the barrage of questions. I'm just learning how to tune things in postgres and I've still got a bit of learning curve to get over, apparently. I have done a lot of reading, though, I swear. I've got two identical queries except for a change of one condition which cuts the number o

Re: [PERFORM] in-memory sorting

2010-08-18 Thread Samuel Gendler
Answered my own question. Cranking work_mem up to 350MB revealed that the in-memory sort requires more memory than the disk sort. On Wed, Aug 18, 2010 at 10:23 PM, Samuel Gendler wrote: > I've got this explain: http://explain.depesz.com/s/Xh9 > > And these settings: > default_

[PERFORM] in-memory sorting

2010-08-18 Thread Samuel Gendler
I've got this explain: http://explain.depesz.com/s/Xh9 And these settings: default_statistics_target = 50 # pgtune wizard 2010-08-17 maintenance_work_mem = 1GB # pgtune wizard 2010-08-17 constraint_exclusion = on # pgtune wizard 2010-08-17 checkpoint_completion_target = 0.9 # pgtune wizard 2010-08

Re: [PERFORM] write barrier question

2010-08-18 Thread Samuel Gendler
On Wed, Aug 18, 2010 at 1:25 PM, Yeb Havinga wrote: > Samuel Gendler wrote: >> >> When running pgbench on a db which fits easily into RAM (10% of RAM = >> -s 380), I see transaction counts a little less than 5K.  When I go to >> 90% of RAM (-s 3420), transaction rate dr

Re: [PERFORM] write barrier question

2010-08-18 Thread Samuel Gendler
I am. I was giving mean numbers On Wed, Aug 18, 2010 at 12:56 PM, Craig James wrote: > On 8/18/10 12:24 PM, Samuel Gendler wrote: >> >> With barriers off, I saw a transaction rate of about 1200.  With >> barriers on, it was closer to 1050.  The test had a concurrency o

[PERFORM] write barrier question

2010-08-18 Thread Samuel Gendler
I'm just starting the process of trying to tune new hardware, which is 2x quad core xeon, 48GB RAM, 8x300GB SAS 15K drives in RAID 1+0, 2x72GB 15K SAS drives in RAID 1 for WAL and system. It is a PERC 6/i card with BBU. Write-back cache is enabled. The system volume is ext3. The large data partit

Re: [PERFORM] Pooling in Core WAS: Need help in performance tuning.

2010-07-09 Thread Samuel Gendler
Sent from my iPhone On Jul 9, 2010, at 18:25, Josh Berkus wrote: So while adding (b) to core alone would be very useful for some users, ironically it's generally for the more advanced users which are not the ones we're trying to help on this thread. It would seem from evidence presente

Re: [PERFORM] performance on new linux box

2010-07-09 Thread Samuel Gendler
On Fri, Jul 9, 2010 at 2:08 AM, Russell Smith wrote: > On 09/07/10 02:31, Ryan Wexler wrote: > > > The only other difference between the boxes is the postgresql version.  The > new one has 8.4-2 from the yum install instructions on the site: > http://yum.pgrpms.org/reporpms/repoview/pgdg-centos.ht

Re: [PERFORM] Need help in performance tuning.

2010-07-08 Thread Samuel Gendler
On Thu, Jul 8, 2010 at 8:11 PM, Craig Ringer wrote: > If you're not using a connection pool, start using one. > > Do you really need 100 *active* working query threads at one time? Because > if you do, you're going to need a scary-big disk subsystem and a lot of > processors. I see this issue and

[PERFORM] ideal storage configuration

2010-06-29 Thread Samuel Gendler
I've been reading this list for a couple of weeks, so I've got some sense of what you folks are likely to recommend, but I'm curious what is considered an ideal storage solution if building a database system from scratch. I just got an exploratory call from my boss, asking what my preference would

[PERFORM] indexes in partitioned tables - again

2010-04-06 Thread Samuel Gendler
I know this problem crops up all the time and I have read what I could find, but I'm still not finding an answer to my problem. This is all postgres 8.3. Yes, I've enabled constraint_exclusion. Yes, there are indexes on the partitions, not just on the parent. I've got a table with 1 month partit