Re: [PERFORM] Runtime dependency from size of a bytea field

2010-10-07 Thread Sander, Ingo (NSN - DE/Munich)
Hi, The difference to my test is that we use the ODBC interface in our C program. Could it be that the difference in the runtimes is caused by the ODBC? BR Ingo -Original Message- From: ext Merlin Moncure [mailto:mmonc...@gmail.com] Sent: Thursday, October 07, 2010 7:17 PM To: Sander

[PERFORM] BBU Cache vs. spindles

2010-10-07 Thread Steve Crawford
I'm weighing options for a new server. In addition to PostgreSQL, this machine will handle some modest Samba and Rsync load. I will have enough RAM so the virtually all disk-read activity will be cached. The average PostgreSQL read activity will be modest - a mix of single-record and fairly la

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Aaron Turner
On Thu, Oct 7, 2010 at 2:47 PM, Greg Smith wrote: > Aaron Turner wrote: >> >> Are newer PG versions more memory efficient? >> > > Moving from PostgreSQL 8.1 to 8.3 or later should make everything you do > happen 2X to 3X faster, before even taking into account that you can tune > the later version

Re: [PERFORM] Odd behaviour with redundant CREATE statement

2010-10-07 Thread Dave Crooke
Thanks folks, that makes sense. We're now being more precise with our DDL :-) Cheers Dave On Thu, Oct 7, 2010 at 3:40 PM, Robert Haas wrote: > On Mon, Sep 27, 2010 at 3:27 PM, Gurjeet Singh > wrote: > > On Mon, Sep 27, 2010 at 8:50 PM, Dave Crooke wrote: > >> > >> Our Java application manages

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Greg Smith
Aaron Turner wrote: Are newer PG versions more memory efficient? Moving from PostgreSQL 8.1 to 8.3 or later should make everything you do happen 2X to 3X faster, before even taking into account that you can tune the later versions better too. See http://suckit.blog.hu/2009/09/29/postgres

Re: [PERFORM] Odd behaviour with redundant CREATE statement

2010-10-07 Thread Robert Haas
On Mon, Sep 27, 2010 at 3:27 PM, Gurjeet Singh wrote: > On Mon, Sep 27, 2010 at 8:50 PM, Dave Crooke wrote: >> >> Our Java application manages its own schema. Some of this is from >> Hibernate, but some is hand-crafted JDBC. >> >> By way of an upgrade path, we have a few places where we have adde

Re: [PERFORM] [HACKERS] MIT benchmarks pgsql multicore (up to 48)performance

2010-10-07 Thread Robert Haas
On Thu, Oct 7, 2010 at 1:21 PM, Kevin Grittner wrote: > Robert Haas wrote: > >> perhaps it would be possible by, say, increasing the number of >> lock partitions by 8x.  It would be nice to segregate these issues >> though, because using pread/pwrite is probably a lot less work >> than rewriting

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Aaron Turner
On Thu, Oct 7, 2010 at 12:02 PM, Stephen Frost wrote: > * Aaron Turner (synfina...@gmail.com) wrote: >> Basically, each connection is taking about 100MB resident > > Errr..  Given that your shared buffers are around 100M, I think you're > confusing what you see in top with reality.  The shared buf

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Stephen Frost
* Aaron Turner (synfina...@gmail.com) wrote: > Basically, each connection is taking about 100MB resident Errr.. Given that your shared buffers are around 100M, I think you're confusing what you see in top with reality. The shared buffers are visible in every process, but it's all the same actual

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Stephen Frost
* Aaron Turner (synfina...@gmail.com) wrote: > The graphing front end CGI is all SELECT. There's 12k tables today, > and new tables are created each month. That's a heck of alot of tables.. Probably more than you really need. Not sure if reducing that number would help query times though. > T

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Stephen Frost
* Dan Harris (f...@drivefaster.net) wrote: > On 10/7/10 11:47 AM, Aaron Turner wrote: >> Basically, each connection is taking about 100MB resident. As we need >> to increase the number of threads to be able to query all the devices >> in the 5 minute window, we're running out of memory. > I think

Re: [PERFORM] large dataset with write vs read clients

2010-10-07 Thread Dan Harris
On 10/7/10 11:47 AM, Aaron Turner wrote: Basically, each connection is taking about 100MB resident. As we need to increase the number of threads to be able to query all the devices in the 5 minute window, we're running out of memory. I think the first thing to do is look into using a connecti

Re: [PERFORM] [HACKERS] MIT benchmarks pgsql multicore (up to 48)performance

2010-10-07 Thread Kevin Grittner
Stephen Frost wrote: > Kevin Grittner (kevin.gritt...@wicourts.gov) wrote: >> Robert Haas wrote: >>> perhaps it would be possible by, say, increasing the number of >>> lock partitions by 8x. >> changing this 4 to a 7?: >> >> #define LOG2_NUM_LOCK_PARTITIONS 4 > I'm pretty sure we were t

Re: [PERFORM] [HACKERS] MIT benchmarks pgsql multicore (up to 48)performance

2010-10-07 Thread Stephen Frost
* Kevin Grittner (kevin.gritt...@wicourts.gov) wrote: > Robert Haas wrote: > > perhaps it would be possible by, say, increasing the number of > > lock partitions by 8x. It would be nice to segregate these issues > > though, because using pread/pwrite is probably a lot less work > > than rewriting

[PERFORM] large dataset with write vs read clients

2010-10-07 Thread Aaron Turner
currently PG 8.1.3. See attached for my postgresql.conf. Server is freebsd 6.2 w/ a fast 3TB storage array and only 2GB of ram. We're running RTG which is a like mrtg, cricket, etc. basically queries network devices via SNMP, throws stats into the DB for making pretty bandwidth graphs. We've

Re: [PERFORM] [HACKERS] MIT benchmarks pgsql multicore (up to 48)performance

2010-10-07 Thread Kevin Grittner
Robert Haas wrote: > perhaps it would be possible by, say, increasing the number of > lock partitions by 8x. It would be nice to segregate these issues > though, because using pread/pwrite is probably a lot less work > than rewriting our lock manager. You mean easier than changing this 4 to a

Re: [PERFORM] Runtime dependency from size of a bytea field

2010-10-07 Thread Merlin Moncure
On Thu, Oct 7, 2010 at 10:49 AM, Merlin Moncure wrote: > On Thu, Oct 7, 2010 at 12:11 AM, Sander, Ingo (NSN - DE/Munich) > wrote: >> As written before I have rerun the test a) without compression and b) >> with enlarged BLOCK_SIZE. Result was the same. > > Using libpqtypes (source code follows af

Re: [PERFORM] Runtime dependency from size of a bytea field

2010-10-07 Thread Merlin Moncure
On Thu, Oct 7, 2010 at 12:11 AM, Sander, Ingo (NSN - DE/Munich) wrote: > As written before I have rerun the test a) without compression and b) > with enlarged BLOCK_SIZE. Result was the same. Using libpqtypes (source code follows after sig), stock postgres, stock table, I was not able to confirm

Re: [PERFORM] [HACKERS] MIT benchmarks pgsql multicore (up to 48)performance

2010-10-07 Thread Ivan Voras
On 10/07/10 02:39, Robert Haas wrote: On Wed, Oct 6, 2010 at 6:31 PM, Ivan Voras wrote: On 10/04/10 20:49, Josh Berkus wrote: The other major bottleneck they ran into was a kernel one: reading from the heap file requires a couple lseek operations, and Linux acquires a mutex on the inode to do

Re: [PERFORM] Optimizing query

2010-10-07 Thread Srikanth K
Sorry to forget to give the postgres version as 8.1 On Thu, Oct 7, 2010 at 2:12 PM, Srikanth K wrote: > Hi Can u Please let me know how can i optimize this query better. As i am > attaching u the Query, Schema and Explain Analyze Report. > > Plerase help me in optimizing this query. > > select >

Re: [PERFORM] [HACKERS] MIT benchmarks pgsql multicore (up to 48)performance

2010-10-07 Thread Robert Haas
On Wed, Oct 6, 2010 at 10:07 PM, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: >> It's good to be you. > > They're HP BL465 G7's w/ 2x 12-core AMD processors and 48G of RAM. > Unfortunately, they currently only have local storage, but it seems > unlikely that would be an issu

Re: [PERFORM] [HACKERS] MIT benchmarks pgsql multicore (up to 48)performance

2010-10-07 Thread Ivan Voras
On 7 October 2010 03:25, Tom Lane wrote: > Ivan Voras writes: >> On 10/04/10 20:49, Josh Berkus wrote: The other major bottleneck they ran into was a kernel one: reading from the heap file requires a couple lseek operations, and Linux acquires a mutex on the inode to do that. > >>

Re: [PERFORM] On Scalability

2010-10-07 Thread Vincenzo Romano
Hi all. I laready posted this a couple of months ago on -hackers: http://archives.postgresql.org/pgsql-hackers/2010-07/msg01519.php I've also been directed to ask here for better and deeper details. What came out is that the management of both inheritance hierarchy and partial indexes doesn't scal