Re: [PERFORM] How to access data of SQL server database from PostgreSQL

2017-08-01 Thread Joshua D. Drake
On 07/31/2017 09:25 PM, Daulat Ram wrote: Hi team, I need to connect to MS-SQL server 2008/2012 from PostgreSQL 9.5 in Windows7 environment to fetch the tables of SQL server. Please help on this. https://github.com/tds-fdw/tds_fdw JD Regards, Daulat --

Re: [PERFORM] Very poor read performance, query independent

2017-07-11 Thread Joshua D. Drake
On 07/11/2017 04:15 PM, Merlin Moncure wrote: On Mon, Jul 10, 2017 at 9:03 AM, Charles Nadeau wrote: I’m running PostgreSQL 9.6.3 on Ubuntu 16.10 (kernel 4.4.0-85-generic). Hardware is: *2x Intel Xeon E5550 *72GB RAM *Hardware RAID10 (4 x 146GB SAS 10k) P410i controller with 1GB FBWC (80% re

Re: [PERFORM] [ADMIN] Monitoring tool for Postgres Database

2017-05-26 Thread Joshua D. Drake
On 05/25/2017 07:15 PM, Scott Mead wrote: Thanks ravi We use Zabbix. JD -- Command Prompt, Inc. http://the.postgres.company/ +1-503-667-4564 PostgreSQL Centered full stack support, consulting and development. Everyone

Re: [PERFORM] Questionaire: Common WAL write rates on busy servers.

2017-04-27 Thread Joshua D. Drake
On 04/27/2017 09:34 AM, Andres Freund wrote: On 2017-04-27 09:31:34 -0700, Joshua D. Drake wrote: On 04/27/2017 08:59 AM, Andres Freund wrote: I would agree it isn't yet a widespread issue. I'm not yet sure about that actually. I suspect a large percentage of people with such

Re: [PERFORM] Questionaire: Common WAL write rates on busy servers.

2017-04-27 Thread Joshua D. Drake
On 04/27/2017 08:59 AM, Andres Freund wrote: Ok, based on the, few, answers I've got so far, my experience is indeed skewed. A number of the PG users I interacted with over the last couple years had WAL write ranges somewhere in the range of 500MB/s to 2.2GB/s (max I'veseen). At that point WA

Re: [PERFORM] How can I find the source of postgresql per-connection memory leaks?

2017-01-12 Thread Joshua D. Drake
On 01/12/2017 09:08 AM, Eric Jensen wrote: I'm using postgresql 9.5.4 on amazon RDS with ~1300 persistent connections from rails 4.2 with "prepared_statements: false". Over the enter image description here PostgreSQL on RDS is a closed product. My recommendation would be to contact Amazon supp

Re: [PERFORM] Isolation of tx logs on VMware

2016-12-13 Thread Joshua D. Drake
On 12/13/2016 12:16 PM, ProPAAS DBA wrote: Hi All; I'm not a VMware expert, however I thought VMware would allow the creation of multiple disk volumes and attach them via separate mount points. Is this not true? If it is an option can someone point me to a how to... Yes it is possible to do

Re: [PERFORM] Big Memory Boxes and pgtune

2016-10-28 Thread Joshua D. Drake
On 10/28/2016 08:44 AM, Warner, Gary, Jr wrote: I've recently been blessed to move one of my databases onto a huge IBM P8 computer. Its a power PC architecture with 20 8-way cores (so postgres SHOULD believe there are 160 cores available) and 1 TB of RAM. I've always done my postgres tuning

Re: [PERFORM] Indexes for hashes

2016-06-15 Thread Joshua D. Drake
On 06/15/2016 07:20 AM, Ivan Voras wrote: Hi, Just for testing... is there a fast (i.e. written in C) crc32 or a similar small hash function for PostgreSQL? https://www.postgresql.org/docs/9.5/static/pgcrypto.html We also have a builtin md5(). JD -- Command Prompt, Inc. http

Re: [PERFORM] checkpoints, proper config

2015-12-10 Thread Joshua D. Drake
On 12/10/2015 12:58 PM, Tory M Blue wrote: synchronous is commented out, is it on by default? Yes it is on by default. This is a slony slave node, so I'm not too worried about this particular host losing it's data, thus fsync is off, thanks again sir Tory -- Command Prompt, Inc. - htt

Re: [PERFORM] checkpoints, proper config

2015-12-10 Thread Joshua D. Drake
On 12/10/2015 10:35 AM, Tory M Blue wrote: Thiis valid regardless of the workload? Yes. Seems that I would be storing a ton of data and writing it once an hour, so would have potential perf hits on the hour. I guess I'm not too up to date on the checkpoint configuration. No, that isn't

Re: [PERFORM] checkpoints, proper config

2015-12-10 Thread Joshua D. Drake
On 12/10/2015 01:12 AM, Tory M Blue wrote: checkpoint_timeout = 5min checkpoint_completion_target = 0.9 The above is your problem. Make checkpoint_timeout = 1h . Also, considering turning synchronous_commit off. JD -- Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564 P

Re: [PERFORM] Hmmm... why does CPU-intensive pl/pgsql code parallelise so badly when queries parallelise fine? Anyone else seen this?

2015-07-08 Thread Joshua D. Drake
On 07/08/2015 10:48 AM, Craig James wrote: I admit that I haven't read this whole thread but: Using Apache Fast-CGI, you are going to fork a process for each instance of the function being executed and that in turn will use all CPUs up to the max available resource. With P

Re: [PERFORM] Hmmm... why does CPU-intensive pl/pgsql code parallelise so badly when queries parallelise fine? Anyone else seen this?

2015-07-07 Thread Joshua D. Drake
On 07/07/2015 08:05 PM, Craig James wrote: No ideas, but I ran into the same thing. I have a set of C/C++ functions that put some chemistry calculations into Postgres as extensions (things like, "calculate the molecular weight of this molecule"). As SQL functions, the whole thing bogged down,

Re: [PERFORM] New server: SSD/RAID recommendations?

2015-07-06 Thread Joshua D. Drake
On 07/06/2015 09:56 AM, Steve Crawford wrote: On 07/02/2015 07:01 AM, Wes Vaske (wvaske) wrote: For what it's worth, in my most recent iteration I decided to go with the Intel Enterprise NVMe drives and no RAID. My reasoning was thus: 1. Modern SSDs are so fast that even if you had an infini

Re: [PERFORM] Do work_mem and shared buffers have 1g or 2g limit on 64 bit linux?

2015-06-13 Thread Joshua D. Drake
On 06/13/2015 10:27 AM, Kaijiang Chen wrote: Hi, I am using postgresql 9.2.10 on centos 6.2, 64 bit version. The server has 512 GB mem. The jobs are mainly OLAP like. So I need larger work_mem and shared buffers. From the source code, there is a constant MaxAllocSize==1GB. So, I wonder whether

Re: [PERFORM] Are there tuning parameters that don't take effect immediately?

2015-06-12 Thread Joshua D. Drake
On 06/12/2015 01:37 PM, Michael Nolan wrote: Last night I was doing some tuning on a database The longest query I was running was taking around 160 seconds. I didn't see much change in the running time for that query, even after restarting PG. Today, with roughly the same system load (possibl

Re: [PERFORM] How to reduce writing on disk ? (90 gb on pgsql_tmp)

2015-06-03 Thread Joshua D. Drake
On 06/03/2015 03:16 PM, Tomas Vondra wrote: What is more important, though, is the amount of memory. OP reported the query writes ~95GB of temp files (and dies because of full disk, so there may be more). The on-disk format is usually more compact than the in-memory representation - for example

Re: [PERFORM] PostgreSQL disk fragmentation causes performance problems on Windows

2015-04-29 Thread Joshua D. Drake
On 04/29/2015 01:08 AM, Andres Freund wrote: Which OS and filesystem is this done on? Because many halfway modern systems, like e.g ext4 and xfs, implement this in the background as 'delayed allocation'. Oh, it's in the subject. Stupid me, sorry for that. I'd consider testing how much better

Re: [PERFORM] [ADMIN] Hardware Configuration and other Stuff

2015-03-18 Thread Joshua D. Drake
chitecture of a rather complex solution and a migration. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, @cmdpromptinc Now I get it: your serv

Re: [PERFORM] MusicBrainz postgres performance issues

2015-03-15 Thread Joshua D. Drake
On 03/15/2015 09:43 AM, Scott Marlowe wrote: * Consider installing perf (linux-utils-$something) and doing a systemwide profile. 3.2 isn't the greatest kernel around, efficiency wise. At some point you might want to upgrade to something newer. I've seen remarkable differences around this.

Re: [PERFORM] MusicBrainz postgres performance issues

2015-03-15 Thread Joshua D. Drake
On 03/15/2015 05:08 AM, Robert Kaye wrote: On Mar 15, 2015, at 12:41 PM, Andreas Kretschmer wrote: just a wild guess: raid-controller BBU faulty We don’t have a BBU in this server, but at least we have redundant power supplies. In any case, how would a fault batter possibly cause this?

Re: [PERFORM] Turn off Hyperthreading! WAS: 60 core performance with 9.3

2014-08-21 Thread Joshua D. Drake
On 08/21/2014 04:29 PM, Josh Berkus wrote: On 08/21/2014 04:08 PM, Steve Crawford wrote: On 08/21/2014 03:51 PM, Josh Berkus wrote: On 08/21/2014 02:26 PM, Scott Marlowe wrote: I'm running almost the exact same setup in production as a spare. It has 4 of those CPUs, 256G RAM, and is currentl

Re: [PERFORM] pg_dump vs pg_basebackup

2014-03-25 Thread Joshua D. Drake
On 03/25/2014 08:21 AM, Magnus Hagander wrote: I would say that's the one thing that rsync is *not*. pg_basebackup takes care of a lot of things under the hood. rsync is a lot more complicated, in particular in failure scenarios, since you have to manually deal with pg_start/stop_backup(). Ther

Re: [PERFORM] pg_dump vs pg_basebackup

2014-03-25 Thread Joshua D. Drake
On 03/25/2014 08:18 AM, Ilya Kosmodemiansky wrote: Joshua, that is really good point: an alternative is to use pg_basebackup through ssh tunnel with compression, but rsync is much simpler. Or rsync over ssh. The advantage is that you can create backups that don't have to be restored, just s

Re: [PERFORM] pg_dump vs pg_basebackup

2014-03-25 Thread Joshua D. Drake
On 03/25/2014 05:05 AM, Claudio Freire wrote: On Tue, Mar 25, 2014 at 4:39 AM, David Johnston wrote: Hai, Can anyone tell me the difference and performance between pgdump and pg_basebackup if I want to backup a large database. Honestly, Neither is particularly good at backing up large da

Re: [PERFORM] One huge db vs many small dbs

2013-12-05 Thread Joshua D. Drake
One of the many questions we have is about performance of the db if we work with only one (using a ClientID to separete de clients info) or thousands of separate dbs. The management of the dbs is not a huge concert as we have an automated tool. If you are planning on using persisted connectio

Re: [PERFORM] How clustering for scale out works in PostgreSQL

2013-08-29 Thread Joshua D. Drake
On 08/29/2013 09:42 AM, bsreejithin wrote: The performance test that was conducted was for 1 Hour. There are 6 transactions. 2 DB inserts and 4 SELECTs. Every 2 minutes there will be 4 SELECTs. And every 3 minutes there will be 2 DB inserts. This shouldn't be a problem with proper hardware a

Re: [PERFORM] How clustering for scale out works in PostgreSQL

2013-08-29 Thread Joshua D. Drake
On 08/29/2013 07:59 AM, Richard Huxton wrote: On 29/08/13 13:14, bsreejithin wrote: I am *expecting 1000+ hits to my PostgreSQL DB* and I doubt my standalone DB will be able to handle it. We are going to need a little more detail here. In a normal environment 1000+ "hits" isn't that much,

Re: [PERFORM] Performance autovaccum

2013-07-09 Thread Joshua D. Drake
On 07/09/2013 03:14 PM, Josh Berkus wrote: On 07/08/2013 09:14 AM, Jeison Bedoya wrote: Hi, i have a postgresql 9.2.2, but i don´t use autovaccum but i want to begin to use it. some recommendation about the optimal configuration? or some link to explain it. Initial configuration: autovacuu

Re: [PERFORM] Reliability with RAID 10 SSD and Streaming Replication

2013-05-22 Thread Joshua D. Drake
On 05/22/2013 07:17 PM, Merlin Moncure wrote: > According the the data sheet it is power safe. > > http://investors.micron.com/releasedetail.cfm?ReleaseID=732650 > http://www.micron.com/products/solid-state-storage/client-ssd/m500-ssd Wow, that seems like a pretty good deal then assuming i

Re: [PERFORM] Reliability with RAID 10 SSD and Streaming Replication

2013-05-22 Thread Joshua D. Drake
On 05/22/2013 04:37 PM, Merlin Moncure wrote: On Wed, May 22, 2013 at 5:42 PM, Joshua D. Drake wrote: I am curious how the 710 or S3700 stacks up against the new M500 from Crucial? I know Intel is kind of the goto for these things but the m500 is power off protected and rated at: Endurance

Re: [PERFORM] Reliability with RAID 10 SSD and Streaming Replication

2013-05-22 Thread Joshua D. Drake
n the poke but it sure seems like a very reasonable drive for the price: http://www.newegg.com/Product/Product.aspx?Item=20-148-695&ParentOnly=1&IsVirtualParent=1 Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 509-416-6579 PostgreSQL Support,

Re: [PERFORM] Reliability with RAID 10 SSD and Streaming Replication

2013-05-22 Thread Joshua D. Drake
On 05/22/2013 11:06 AM, Greg Smith wrote: I have some moderately fast SSD based transactional systems that are still using traditional drives with battery-backed cache for the sequential writes of the WAL volume, where the data volume is on Intel 710 disks. WAL writes really burn through flash

Re: [PERFORM] Setup of four 15k SAS disk with LSI raid controller

2013-03-13 Thread Joshua D. Drake
On 03/13/2013 11:45 AM, Vasilis Ventirozos wrote: Its better to split WAL segments and data just because these two have different io requirements and because its easier to measure and tune things if you have them on different disks. Generally speaking you are correct but we are talking about R

Re: [PERFORM] Optimize SELECT * from table WHERE foreign_key_id IN (key1,key2,key3,key4...)

2013-03-05 Thread Joshua D. Drake
On 03/05/2013 03:51 PM, Niels Kristian Schjødt wrote: 3ms isn't slow Sorry, it's 3323ms! Can I do anything to optimize that query or maybe the index or something? your index is already used Okay this leaves me with - "get better hardware" or? What does explain analyze say versus just

Re: [PERFORM] Request for help with slow query

2012-10-29 Thread Joshua D. Drake
at is your default_statistics_target? Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, Postgres-XC @cmdpromptinc - 509-416-6579 -- Sent via pgsql-performance mailing list (pgsql-p

Re: [PERFORM] auto-vacuum vs. full table update

2012-04-26 Thread Joshua D. Drake
On 04/26/2012 12:49 PM, Craig James wrote: An update to our system means I'm going to be rewriting every row of some large tables (20 million rows by 15 columns). In a situation like this, can auto-vacuum take care of it, or should I plan on vacuum-full/reindex to clean up? If you rewrite t

Re: [PERFORM] Performance

2011-04-13 Thread Joshua D. Drake
On 04/13/2011 05:03 PM, Tom Lane wrote: That 4:1 ratio is based on some rather extensive experimentation that I did back in 2000. In the interim, disk transfer rates have improved quite a lot more than disk seek times have, and the CPU cost to process a page's worth of data has also improved com

Re: [PERFORM] Linux: more cores = less concurrency.

2011-04-11 Thread Joshua D. Drake
On Mon, 11 Apr 2011 13:09:15 -0500, "Kevin Grittner" wrote: > Glyn Astill wrote: > >> The new server uses 4 x 8 core Xeon X7550 CPUs at 2Ghz > > Which has hyperthreading. > >> our current servers are 2 x 4 core Xeon E5320 CPUs at 2Ghz. > > Which doesn't have hyperthreading. > > PostgreS

Re: [PERFORM] Performance Test for PostgreSQL9

2011-03-02 Thread Joshua D. Drake
On Thu, 2011-03-03 at 13:16 +0800, Selva manickaraja wrote: > Thanks for the enlightenment. I will then look into other tools that > help > with performance testing. Is pgbench really useful? We need to produce > the > reports and statistics to our management as we are planning to migrate > one > s

Re: [PERFORM] Performance Test for PostgreSQL9

2011-03-02 Thread Joshua D. Drake
can't solve that problem you really should contract with someone to help you with your performance tests because you are not going to be able to adequately tune PostgreSQL for a proper test. That said, the reason you can't find make is that you don't have the proper de

Re: [PERFORM] Write-heavy pg_stats_collector on mostly idle server

2011-02-07 Thread Joshua D. Drake
On Mon, 2011-02-07 at 14:58 -0800, Josh Berkus wrote: > >> Anyone seen anything like this before? > >> > > > > it is the expected behavior, IIRC > > OK. It just seems kind of pathological for stats file writing to be 10X > the volume of data writing. I see why it's happening, but I think it's >

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Joshua D. Drake
On Thu, 2011-02-03 at 18:12 -0800, Conor Walsh wrote: > > I can't remember > > anyone ever complaining "ANALYZE took too long to run". I only > > remember complaints of the form "I had to remember to manually run it > > and I wish it had just happened by itself". > > Robert, > > This sounds like

Re: [HACKERS] [PERFORM] Slow count(*) again...

2011-02-03 Thread Joshua D. Drake
On Thu, 2011-02-03 at 18:33 -0500, Mladen Gogala wrote: > > > > > Exactly what we don't want. > > > Who is "we"? The majority of long term hackers. -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Cust

Re: [PERFORM] SELECT INTO large FKyed table is slow

2010-11-30 Thread Joshua D. Drake
On Sun, 2010-11-28 at 12:46 +0100, Mario Splivalo wrote: > The database for monitoring certain drone statuses is quite simple: > > This is the slow part: > INSERT INTO drones_history (sample_id, drone_id, drone_log_notice, > drone_temperature, drone_pressure) > SELECT * FROM tmpUpdate; > > For

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Joshua D. Drake
On Thu, 2010-10-28 at 12:59 -0700, Ben wrote: > On Oct 28, 2010, at 12:44 PM, Joshua D. Drake wrote: > > > > My tests show you are incorrect: > > > > > > part_test=# explain analyze select * from foo join bar using (i) where > > i=9; > >

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Joshua D. Drake
On Thu, 2010-10-28 at 12:25 -0700, Ben wrote: > i think we are talking about two different things here: the constraints on > the table, and the where-clause constraints in a query which may or may not > trigger constraint exclusion. i understand that table constraints have to be > constants --

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Joshua D. Drake
(as an example) > > >> is my intuition completely off on this? > > > > You may actually want to look into expression indexes, not clustered > > ones. Take a look at the docs: http://www.postgresql.org/docs/8.4/interactive/indexes-expressional.html It "could&quo

Re: [PERFORM] partitioning question 1

2010-10-28 Thread Joshua D. Drake
. Not sure about the operator question honestly. > is my intuition completely off on this? You may actually want to look into expression indexes, not clustered ones. Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ -

Re: [PERFORM] Postgres insert performance and storage requirement compared to Oracle

2010-10-25 Thread Joshua D. Drake
ta for that right now. This is interesting, are you using libpq or libpqXX? Joshua D. Drake > > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdp

Re: [PERFORM] Postgres insert performance and storage requirement compared to Oracle

2010-10-25 Thread Joshua D. Drake
point of time. Did you test? > > 2. What are the average storage requirements of postgres compared to > Oracle? I > inserted upto 1 million records. The storage requirement of postgreSQL > is almost > double than that of Oracle. What was your table structure? Jos

Re: [PERFORM] BBU Cache vs. spindles

2010-10-20 Thread Joshua D. Drake
On Wed, 2010-10-20 at 22:13 -0400, Bruce Momjian wrote: > Ben Chobot wrote: > > On Oct 7, 2010, at 4:38 PM, Steve Crawford wrote: > > > > > I'm weighing options for a new server. In addition to PostgreSQL, this > > > machine will handle some modest Samba and Rsync load. > > > > > > I will have en

Re: [PERFORM] XFS vs Ext3, and schedulers, for WAL

2010-10-11 Thread Joshua D. Drake
On Mon, 2010-10-11 at 10:50 -0700, Josh Berkus wrote: > > There's a number of blog tests floating around comparing XFS and Ext3, > > and the various Linux schedulers, for PGDATA or for an all-in-one mount. > > > > However, the WAL has a rather particular write pattern, and it's > > reasonable to a

Re: [PERFORM] Issue for partitioning with extra check constriants

2010-10-04 Thread Joshua D. Drake
On Mon, 2010-10-04 at 11:34 -0700, Josh Berkus wrote: > > And your point is? The design center for the current setup is maybe 5 > > or 10 partitions. We didn't intend it to be used for more partitions > > than you might have spindles to spread the data across. > > Where did that come from? Yea

Re: [PERFORM] Query much faster with enable_seqscan=0

2010-09-21 Thread Joshua D. Drake
m_page_cost back to 4 and try increasing cpu_tuple_cost instead. Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://ident

Re: [PERFORM] Held idle connections vs use of a Pooler

2010-09-14 Thread Joshua D. Drake
in-house apps/scripts/..etc started to > experience show stopping problems. Use pgbouncer. It is what Skype uses. Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, En

Re: [PERFORM] Where does data in pg_stat_user_tables come from?

2010-09-13 Thread Joshua D. Drake
On Mon, 2010-09-13 at 16:47 -0700, Josh Berkus wrote: > On 9/13/10 4:41 PM, Joshua D. Drake wrote: > > On Mon, 2010-09-13 at 16:06 -0700, Josh Berkus wrote: > >> All, > >> > >> I've been looking at pg_stat_user_tables (in 8.3, because of a project I > &

Re: [PERFORM] Where does data in pg_stat_user_tables come from?

2010-09-13 Thread Joshua D. Drake
ms and it was a bottleneck. (IIRC) Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt -- Sent via pgsq

Re: [PERFORM] Testing Sandforce SSD

2010-07-24 Thread Joshua D. Drake
he toy. I can get 4 SATA-II with RAID Controller, with battery backed cache, for the same price or less :P Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering

Re: [PERFORM] Strange explain on partitioned tables

2010-07-23 Thread Joshua D. Drake
this case... It is variable based on workload and as I recall has to do with the planning time. As the number of children increases, so does the planning time. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Tra

Re: [PERFORM] Pooling in Core WAS: Need help in performance tuning.

2010-07-23 Thread Joshua D. Drake
aster control. Yeah but that won't happen. Also I think we may have a libevent dependency that we have to work out. > > You probably can't get much leaner than pgbouncer. Oh don't get me wrong. I love pgbouncer. It is my recommended pooler but even it has limitations (such as

Re: [PERFORM] Pooling in Core WAS: Need help in performance tuning.

2010-07-22 Thread Joshua D. Drake
On Thu, 2010-07-22 at 14:36 -0400, Robert Haas wrote: > On Mon, Jul 12, 2010 at 6:58 AM, Craig Ringer > wrote: > > So rather than asking "should core have a connection pool" perhaps > > what's needed is to ask "what can an in-core pool do that an external > > pool cannot do?" > > Avoid sending e

Re: [PERFORM] Identical query slower on 8.4 vs 8.3

2010-07-15 Thread Joshua D. Drake
3. You didn't run ANALYZE on one of the machines Sincerely, Joshua D. Drake > > Thanks! > Patrick > -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineerin

Re: [PERFORM] partition queries hitting all partitions even though check key is specified

2010-07-12 Thread Joshua D. Drake
>>> I cant figure out why we're scanning all of our partitions. > > I don't think extract() is immutable, which would pretty much invalidate > your check constraints as far as CE is concerned. Correct. Joshua D. Drake -- PostgreSQL.org Major Contributor Command

Re: [PERFORM] performance on new linux box

2010-07-08 Thread Joshua D. Drake
IOS. It will tell you. Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org

Re: [PERFORM] Occasional giant spikes in CPU load

2010-06-24 Thread Joshua D. Drake
attached output from "top > -b", which shows what is happening during one of the CPU spikes. checkpoint causing IO Wait. What does sar say about these times? Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.65

Re: [PERFORM] PostgreSQL as a local in-memory cache

2010-06-24 Thread Joshua D. Drake
On Thu, 2010-06-24 at 21:14 +0200, Pavel Stehule wrote: > 2010/6/24 Josh Berkus : > > > >> And I'm also planning to implement unlogged tables, which have the > >> same contents for all sessions but are not WAL-logged (and are > >> truncated on startup). > > this is similar MySQL's memory tables. P

Re: [PERFORM] slow index lookup

2010-06-22 Thread Joshua D. Drake
On Tue, 2010-06-22 at 18:00 -0700, Anj Adu wrote: > i have several partitions like this (similar size ...similar data > distribution)..these partitions are only "inserted"..never updated. > Why would I need to vacuum.. > An explain analyze is what is in order for further diagnosis. JD > I can

Re: [PERFORM] Very high effective_cache_size == worse performance?

2010-04-20 Thread Joshua D. Drake
> > incidentally, both tests were starting from a blank database. > > Is this expected? Without a more complete picture of the configuration, this post doesn't mean a whole lot. Further, effective_cash_size is not likely to effect a bulk load at all. Joshua D. Drake > &g

Re: [PERFORM] Occasional giant spikes in CPU load

2010-04-07 Thread Joshua D. Drake
On Wed, 2010-04-07 at 14:45 -0700, Craig James wrote: > On 4/7/10 2:40 PM, Joshua D. Drake wrote: > > On Wed, 2010-04-07 at 14:37 -0700, Craig James wrote: > >> Most of the time Postgres runs nicely, but two or three times a day we get > >> a huge spike in the CPU l

Re: [PERFORM] Occasional giant spikes in CPU load

2010-04-07 Thread Joshua D. Drake
processes, but I think it missed the bulk of the spike. What does iostat 5 say during the jump? Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering Respect i

Re: [PERFORM] disk space usage unexpected

2010-02-15 Thread Joshua D. Drake
enode.1,filenode.2,...filenode.43) created in the database subdirectory > within PGDATA/base. > > This caused the disk space quickly used up. Is this expected? Yes. Especially if autovacuum is not keeping up with the number of updates. Joshua D. Drake > > Thanks for any in

Re: [PERFORM] Dell PERC H700/H800

2010-02-11 Thread Joshua D. Drake
dds_only/ > > As one of the comments points out, that kind of makes them no longer SATA > or SAS compatible, and they shouldn't be allowed to use those acronyms any > more. That's interesting. I know that IBM at least on some of their models have done the same. Glad I u

Re: [PERFORM] New server to improve performance on our large and busy DB - advice?

2010-01-14 Thread Joshua D. Drake
3 series which I believe is 8.3.9. With the imminent release of 8.5 (6 months), it may be time to move to 8.4.2 instead. Joshua D. Drake > > Thanks, > > Carlo > > The details of our use: > > . The DB hosts is a data warehouse and a knowledgebase (KB) tracking th

Re: [PERFORM] 8.4.1 ubuntu karmic slow createdb

2009-12-11 Thread Joshua D. Drake
, whether intentionally or not. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander

Re: [PERFORM] 8.4.1 ubuntu karmic slow createdb

2009-12-11 Thread Joshua D. Drake
't work then try a mini xfs. Do not turn fsync off. That is bad advice. I would not suggest ext4 at this point for database operations. Use ext3. It is backward compatible. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.66

Re: [PERFORM] truncate in transaction blocks read access

2009-11-30 Thread Joshua D. Drake
t, then start a transaction that drops the old table and > renames the new one. > > Any thoughts? Use partitioning so you can roll off data. http://www.postgresql.org/docs/8.3/interactive/ddl-partitioning.html Joshua D. Drake > > Thanks, > Craig > > -- Postgr

Re: [PERFORM] database size growing continously

2009-10-29 Thread Joshua D. Drake
S. If you overran your max_fsm_pages you are going to have indexes that are not properly cleaned up, even after a vacuum full. You will need to cluster or reindex. Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting,

Re: [PERFORM] dump time increase by 1h with new kernel

2009-10-08 Thread Joshua D. Drake
On Thu, 2009-10-08 at 10:44 -0700, Justin T Pryzby wrote: > Hi Everyone Did your scheduler change between the kernel versions? > Not sure if sar can provide other data included by vmstat: IO merged > in/out, {,soft}irq ticks? > > Thanks, > Justin > -- PostgreSQL.org Major Contributor Command

Re: [PERFORM] partition queries hitting all partitions even though check key is specified

2009-09-02 Thread Joshua D. Drake
ains constants. A parameterized query will not be optimized, since the planner cannot know which partitions the parameter value might select at run time. For the same reason, "stable" functions such as CURRENT_DATE must be avoided. http://www.postgresql.org/docs/8.3/st

Re: [PERFORM] Bottleneck?

2009-08-06 Thread Joshua D. Drake
On Thu, 2009-08-06 at 12:38 -0400, Ray Stell wrote: > On Thu, Aug 06, 2009 at 09:12:22AM -0700, Joshua D. Drake wrote: > > Why aren't you using systemtap again? > > 1. significant solaris responsibilites There is your problem right there ;) > 2. significant linux res

Re: [PERFORM] Bottleneck?

2009-08-06 Thread Joshua D. Drake
ng systemtap again? As I recall it uses the same interface as dtrace. The front end is just different. Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, servi

Re: [PERFORM] Postgres replication: dump/restore, PITR, Slony,...?

2009-06-11 Thread Joshua D. Drake
? :) > > Buchardo? :) A new desert, Buchardo CMO: Two shots of brandy One shot of rum Vanilla Ice cream Cherries Blend to perfection. Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprom

Re: [PERFORM] Postgres replication: dump/restore, PITR, Slony,...?

2009-06-11 Thread Joshua D. Drake
> > Then there are Slony-I, Buchardo, Mamoth Replicator from CMO, simple > replication in Postgres 8.4 and other projects... CMO? :) Joshua D. Drake > > Suggestions? > Thanks, > > -- Shaul > -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Dev

Re: [PERFORM] Hosted servers with good DB disk performance?

2009-05-27 Thread Joshua D. Drake
On Tue, 2009-05-26 at 19:52 -0600, Scott Marlowe wrote: > On Tue, May 26, 2009 at 7:41 PM, Scott Carey wrote: > > > > On 5/26/09 6:17 PM, "Greg Smith" wrote: > > > >> On Tue, 26 May 2009, Joshua D. Drake wrote: > >> > >>> CMD doe

Re: [PERFORM] Hosted servers with good DB disk performance?

2009-05-26 Thread Joshua D. Drake
t folk. CMD doesn't rent hardware you would have to provide that, Rack Space does. Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1

Re: [PERFORM] raid10 hard disk choice

2009-05-21 Thread Joshua D. Drake
ume to WAL), and more total performance on paper. I > would also, if you can afford it and they fit, get two small sata > drives, mount raid 1 and put the o/s on those. +1 on that. Joshua D. Drake > > merlin > -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, De

Re: [PERFORM] Any better plan for this query?..

2009-05-12 Thread Joshua D. Drake
On Tue, 2009-05-12 at 20:34 -0400, Aidan Van Dyk wrote: > * Joshua D. Drake [090512 19:27]: > > > Apache solved this problem back when it was still called NSCA HTTPD. Why > > aren't we preforking again? > > Of course, preforking and connection pooling are totally

Re: [PERFORM] Any better plan for this query?..

2009-05-12 Thread Joshua D. Drake
as problems too under heavy load (due to libevent issues). It also doesn't support all of our auth methods. Apache solved this problem back when it was still called NSCA HTTPD. Why aren't we preforking again? Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org

Re: [PERFORM] Any better plan for this query?..

2009-05-12 Thread Joshua D. Drake
onnect and reconnect yes. If they keep the connections live then no. Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent via pgsql-p

Re: [PERFORM] PostgreSQL with PostGIS on embedded hardware

2009-05-08 Thread Joshua D. Drake
On Fri, 2009-05-08 at 18:06 +0200, Paolo Rizzi wrote: > Hi all, > recently I came across a question from a customer of mine, asking me if > it would feasible to run PostgreSQL along with PostGIS on embedded hardware. > They didn't give me complete information, but it should be some kind of > indu

Re: [PERFORM] performance for high-volume log insertion

2009-04-22 Thread Joshua D. Drake
en if you do a single BEGIN; with 1000 inserts you are still getting a round trip for every insert until you commit. Based on 20ms round trip time, you are talking 20seconds additional overhead. Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Suppor

Re: [PERFORM] Using IOZone to simulate DB access patterns

2009-04-10 Thread Joshua D. Drake
e instances. The interesting thing here is the -s 1000m and -r8k. Those options are basically use a 1000 meg file (like our data files) with 8k chunks (like our pages). Based on your partitioning scheme, what is the break out? Can you reasonably expect all partitions to be used eq

Re: [PERFORM] Performance of archive logging in a PITR restore

2009-03-16 Thread Joshua D. Drake
ce to > restore. Archive logs are restored in a serialized manner so they will be slower to restore in general. Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Comp

Re: [PERFORM] explanation of some configs

2009-02-06 Thread Joshua D. Drake
checkpoint segment, dirty data is written to disk? A checkpoint occurs. Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent vi

Re: [PERFORM] suggestions for postgresql setup on Dell 2950 , PERC6i controller

2009-02-05 Thread Joshua D. Drake
On Thu, 2009-02-05 at 12:40 +, Matt Burke wrote: > Arjen van der Meijden wrote: > > Are there any reasonable choices for bigger (3+ shelf) direct-connected > RAID10 arrays, or are hideously expensive SANs the only option? I've > checked out the latest Areca controllers, but the manual availab

Re: [PERFORM] Using multiple cores for index creation?

2009-01-29 Thread Joshua D. Drake
and break that appart. Reading the TOC is pretty self evident. Once you get down to index creation you can create multiple files each with a group of indexes to create. Then call pg_restore multiple times in a script against the individual TOC and you will use all cores. Joshua D. Drake P.S.

Re: [PERFORM] [PERFORMANCE] Buying hardware

2009-01-26 Thread Joshua D. Drake
5 hours (versus 15). IMO, that is a *little* more than somewhat. Maybe, a bit? ;) Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Company, serving since 1997 -- Sent

Re: [PERFORM] SSD performance

2009-01-23 Thread Joshua D. Drake
There is no correct in right. Correctness is the result of friction caused by the mingling of the two. Sincerely, Joshua D. Drake -- PostgreSQL - XMPP: jdr...@jabber.postgresql.org Consulting, Development, Support, Training 503-667-4564 - http://www.commandprompt.com/ The PostgreSQL Co

  1   2   3   4   5   6   >