Re: [PERFORM] Kernel Resources and max_connections

2005-05-03 Thread Mark Kirkwood
Chris Hebrard wrote: I set the values in etc/sysctl.conf: # $FreeBSD: src/etc/sysctl.conf,v 1.1.2.3 2002/04/15 00:44:13 dougb Exp $ # # This file is read when going to multi-user and its contents piped thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. # # Added by IMP

Re: [PERFORM] Kernel Resources and max_connections

2005-05-03 Thread Jim C. Nasby
On Wed, May 04, 2005 at 01:46:34PM +1200, Mark Kirkwood wrote: > (This not-very-clear distinction between what is sysctl'abe and what is > a kernel tunable is a bit of a downer). I think this is documented somewhere, though I can't think of where right now. Also, note that some sysctl's can only

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-05-03 Thread Mischa Sandberg
Quoting Josh Berkus : > Mischa, > > > Okay, although given the track record of page-based sampling for > > n-distinct, it's a bit like looking for your keys under the > streetlight, > > rather than in the alley where you dropped them :-) > > Bad analogy, but funny. Bad analogy? Page-

[PERFORM] Kernel Resources Solved

2005-05-03 Thread Chris Hebrard
Problem sovled by setting: kern.ipc.semmni: 280 kern.ipc.semmns: 300 Chris. Mark Kirkwood wrote: Chris Hebrard wrote: kern.ipc.shmmax and kern.ipc.shmmin will not stay to what I set them to. What am I doing wrong or not doing at all? These need to go in /etc/sysctl.conf. You might need to set sh

Re: [PERFORM] Kernel Resources and max_connections

2005-05-03 Thread Chris Hebrard
Mark Kirkwood wrote: Chris Hebrard wrote: kern.ipc.shmmax and kern.ipc.shmmin will not stay to what I set them to. What am I doing wrong or not doing at all? These need to go in /etc/sysctl.conf. You might need to set shmall as well. (This not-very-clear distinction between what is sysctl'abe and

Re: [PERFORM] Kernel Resources and max_connections

2005-05-03 Thread Mark Kirkwood
Chris Hebrard wrote: kern.ipc.shmmax and kern.ipc.shmmin will not stay to what I set them to. What am I doing wrong or not doing at all? These need to go in /etc/sysctl.conf. You might need to set shmall as well. (This not-very-clear distinction between what is sysctl'abe and what is a kernel tuna

[PERFORM] Kernel Resources and max_connections

2005-05-03 Thread Chris Hebrard
Hi, I have postgres 8.0.2 installed on FreeBSD FreeBSD 4.11-RELEASE with 2GB of RAM. When trying to set max_connections=256 I get the following error message: FATAL: could not create semaphores: No space left on device DETAIL: Failed system call was semget(5432017, 17, 03600). HINT: This error

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-05-03 Thread Josh Berkus
John, > But doesn't an index only sample one column at a time, whereas with > page-based sampling, you can sample all of the columns at once. Hmmm. Yeah, we're not currently doing that though. Another good idea ... -- --Josh Josh Berkus Aglio Database Solutions San Francisco --

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-05-03 Thread John A Meinel
Josh Berkus wrote: Mischa, Okay, although given the track record of page-based sampling for n-distinct, it's a bit like looking for your keys under the streetlight, rather than in the alley where you dropped them :-) Bad analogy, but funny. The issue with page-based vs. pure random sampling is th

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-05-03 Thread Josh Berkus
Mischa, > Okay, although given the track record of page-based sampling for > n-distinct, it's a bit like looking for your keys under the streetlight, > rather than in the alley where you dropped them :-) Bad analogy, but funny. The issue with page-based vs. pure random sampling is that to do, fo

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-05-03 Thread Mischa Sandberg
Quoting Markus Schaber <[EMAIL PROTECTED]>: > Hi, Josh, > > Josh Berkus wrote: > > > Yes, actually. We need 3 different estimation methods: > > 1 for tables where we can sample a large % of pages (say, >= 0.1) > > 1 for tables where we sample a small % of pages but are "easily > estimated" > >

Re: [PERFORM] batch inserts are "slow"

2005-05-03 Thread Dave Cramer
Kris is correct, This code was not added or even submitted to CVS. The purpose of this was to work out the bugs with people who are actually using copy. The api is a separate issue however. There's no reason that copy can't support more than one api. Dave Kris Jurka wrote: On Tue, 3 May 2005, J

Re: [PERFORM] batch inserts are "slow"

2005-05-03 Thread Christopher Petrilli
On 5/3/05, Tim Terlegård <[EMAIL PROTECTED]> wrote: > > Just as on Oracle you would use SQL*Loader for this application, you > > should use the COPY syntax for PostgreSQL. You will find it a lot > > faster. I have used it by building the input files and executing > > 'psql' with a COPY command, a

Re: [PERFORM] COPY vs INSERT

2005-05-03 Thread Mischa Sandberg
> Steven Rosenstein <[EMAIL PROTECTED]> writes: > > My question is, are there any advantages, drawbacks, or outright > > restrictions to using multiple simultaneous COPY commands to load > data into > > the same table? Do you mean, multiple COPY commands (connections) being putline'd from the same

Re: [PERFORM] batch inserts are "slow"

2005-05-03 Thread Steve Wampler
Tim Terlegård wrote: >> >>Just as on Oracle you would use SQL*Loader for this application, you >>should use the COPY syntax for PostgreSQL. You will find it a lot >>faster. I have used it by building the input files and executing >>'psql' with a COPY command, and also by using it with a subproces

[PERFORM] Bad choice of query plan from PG 7.3.6 to PG 7.3.9 part 1b

2005-05-03 Thread Jona
Please refer to part 1a for questions and part 2 for more queries and query plans. Why won't this list accept my questions and sample data in one mail??? /Jona Query 1: EXPLAIN ANALYZE SELECT DI

Re: [PERFORM] batch inserts are "slow"

2005-05-03 Thread Tim Terlegård
> > I'm converting an application to be using postgresql instead of oracle. > > There seems to be only one issue left, batch inserts in postgresql seem > > significant slower than in oracle. I have about 200 batch jobs, each > > consisting of about 14 000 inserts. Each job takes 1.3 seconds in > >

[PERFORM] Bad choice of query plan from PG 7.3.6 to PG 7.3.9 part 2

2005-05-03 Thread Jona
Please refer to part 1 for question and query 1 Cheers Jona --- Query 2: EXPLAIN ANALYZE SELECT DISTINCT CatType_Tbl.id, CatType_Tbl.url, Category_Tbl.name, Min(SubCatType_Tbl.id)

[PERFORM] Testing list access

2005-05-03 Thread Jona
Testing list access ---(end of broadcast)--- TIP 9: the planner will ignore your desire to choose an index scan if your joining column's datatypes do not match

Re: [PERFORM]

2005-05-03 Thread Tom Lane
Steven Rosenstein <[EMAIL PROTECTED]> writes: > My question is, are there any advantages, drawbacks, or outright > restrictions to using multiple simultaneous COPY commands to load data into > the same table? It will work; not sure about whether there is any performance benefit. I vaguely recall s

Re: [PERFORM] Foreign key constraints compile faster in 7.4

2005-05-03 Thread Tom Lane
Ashish Arte <[EMAIL PROTECTED]> writes: > We recently upgraded to Postgres 7.4 from 7.3.9 and noticed that the > foreign key constraints compile noticeably faster. In 7.3 the > constraints would typically take more than an hour to run on our > production data. Now they take a minute or two. > Can

Re: [PERFORM] batch inserts are "slow"

2005-05-03 Thread Kris Jurka
On Tue, 3 May 2005, Josh Berkus wrote: > > There are several hacks floating around that add COPY capabilities to > > the pgjdbc driver. As they all are rather simple hacks, they have not > > been included in the cvs yet, but they tend to work fine. > > FWIW, Dave Cramer just added beta COPY cap

[PERFORM]

2005-05-03 Thread Steven Rosenstein
In our application we have tables that we regularly load with 5-10 million records daily. We *were* using INSERT (I know... Still kicking ourselves for *that* design decision), and we now converting over to COPY. For the sake of robustness, we are planning on breaking the entire load into ch

Re: [PERFORM] batch inserts are "slow"

2005-05-03 Thread Josh Berkus
People, > There are several hacks floating around that add COPY capabilities to > the pgjdbc driver. As they all are rather simple hacks, they have not > been included in the cvs yet, but they tend to work fine. FWIW, Dave Cramer just added beta COPY capability to JDBC. Contact him on the JDBC

[PERFORM] Foreign key constraints compile faster in 7.4

2005-05-03 Thread Ashish Arte
Hello Everybody, We recently upgraded to Postgres 7.4 from 7.3.9 and noticed that the foreign key constraints compile noticeably faster. In 7.3 the constraints would typically take more than an hour to run on our production data. Now they take a minute or two. Can anybody explain such a major per

Re: [PERFORM] batch inserts are "slow"

2005-05-03 Thread Markus Schaber
Hi, all, David Parker wrote: > We ran into the need to use COPY, but our application is also in Java. > We wrote a JNI bridge to a C++ routine that uses the libpq library to do > the COPY. The coding is a little bit weird, but not too complicated - > the biggest pain in the neck is probably gettin

Re: [HACKERS] [PERFORM] Bad n_distinct estimation; hacks suggested?

2005-05-03 Thread Markus Schaber
Hi, Josh, Josh Berkus wrote: > Yes, actually. We need 3 different estimation methods: > 1 for tables where we can sample a large % of pages (say, >= 0.1) > 1 for tables where we sample a small % of pages but are "easily estimated" > 1 for tables which are not easily estimated by we can't afford