Re: [PERFORM] trouble with a join on OS X

2007-02-05 Thread Shane Ambler
Kirk Wythers wrote: The 4G (32bit) limit may be where you hit the out of memory errors (or is postgres get around that with it's caching?). Any idea if postgres on OS X can truely access more that 4 gigs if the 64 bit version is built? I have tried building the 64 bit version of some other

Re: [PERFORM] trouble with a join on OS X

2007-02-04 Thread Kirk Wythers
On Feb 3, 2007, at 9:59 AM, Shane Ambler wrote: just so you can look into it for your own curiosity ;-) - Mac OS X uses the startup disk for VM storage. You can find the files in - / var/vm You will find the swapfiles there, the size of the swapfiles progressively get larger -

[PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
I am trying to do fairly simple joins on climate databases that should return ~ 7 million rows of data. However, I'm getting an error message on a OS X (10.4 tiger server) machine that seems to imply that I am running out of memory. The errors are: psql(15811) malloc: ***

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Steinar H. Gunderson
On Fri, Feb 02, 2007 at 07:52:48AM -0600, Kirk Wythers wrote: psql(15811) malloc: *** vm_allocate(size=8421376) failed (error code=3) psql(15811) malloc: *** error: can't allocate region psql(15811) malloc: *** set a breakpoint in szone_error to debug It sounds like you are out of memory. Have

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Gábriel Ákos
Richard Huxton wrote: Kirk Wythers wrote: I am trying to do fairly simple joins on climate databases that should return ~ 7 million rows of data. However, I'm getting an error message on a OS X (10.4 tiger server) machine that seems to imply that I am running out of memory. The errors are:

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
=?ISO-8859-1?Q?G=E1briel_=C1kos?= [EMAIL PROTECTED] writes: Richard Huxton wrote: Kirk Wythers wrote: I am trying to do fairly simple joins on climate databases that should return ~ 7 million rows of data. If you look at the message carefully, it looks like (for me) that the client is

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
Thanks for the reply Steiner, On Feb 2, 2007, at 8:41 AM, Steinar H. Gunderson wrote: On Fri, Feb 02, 2007 at 07:52:48AM -0600, Kirk Wythers wrote: psql(15811) malloc: *** vm_allocate(size=8421376) failed (error code=3) psql(15811) malloc: *** error: can't allocate region psql(15811)

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
On Feb 2, 2007, at 9:46 AM, Tom Lane wrote: =?ISO-8859-1?Q?G=E1briel_=C1kos?= [EMAIL PROTECTED] writes: Richard Huxton wrote: Kirk Wythers wrote: I am trying to do fairly simple joins on climate databases that should return ~ 7 million rows of data. If you look at the message

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Steinar H. Gunderson
On Fri, Feb 02, 2007 at 10:05:29AM -0600, Kirk Wythers wrote: Thanks Tom... Any suggestions as to how much to raise ulimit -d? And how to raise ulimit -d? Try multiplying it by 100 for a start: ulimit -d 614400 /* Steinar */ -- Homepage: http://www.sesse.net/

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
Steinar H. Gunderson [EMAIL PROTECTED] writes: On Fri, Feb 02, 2007 at 10:05:29AM -0600, Kirk Wythers wrote: Thanks Tom... Any suggestions as to how much to raise ulimit -d? And how to raise ulimit -d? Try multiplying it by 100 for a start: ulimit -d 614400 Or just ulimit -d unlimited

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
Tom, I tried ulimit -d 614400, but the query ended with the same error. I thought then that the message: psql(21522) malloc: *** vm_allocate(size=8421376) failed (error code=3) psql(21522) malloc: *** error: can't allocate region psql(21522) malloc: *** set a breakpoint in szone_error to

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
Kirk Wythers [EMAIL PROTECTED] writes: However, setting ulimit to unlimited does not seem to solve the issue. Output from ulimit -a is: Possibly a silly question, but you are running the client code under the shell session that you adjusted ulimit for, yes? regards,

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
At this point there are no silly questions. But I am running the query under the shell session that I adjusted. I did discover that ulimit -d only changes the shell session that you issue the command in. So I changed ulimit -d to unlimited, connected to the db with psql db_name, then ran

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
On Feb 2, 2007, at 10:11 AM, Tom Lane wrote: Steinar H. Gunderson [EMAIL PROTECTED] writes: On Fri, Feb 02, 2007 at 10:05:29AM -0600, Kirk Wythers wrote: Thanks Tom... Any suggestions as to how much to raise ulimit -d? And how to raise ulimit -d? Try multiplying it by 100 for a start:

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
Kirk Wythers [EMAIL PROTECTED] writes: However, setting ulimit to unlimited does not seem to solve the issue. After some experimentation I'm left wondering exactly what ulimit's -d option is for on OS X, because it sure doesn't seem to be limiting process data size. (I should have been

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Luke Lonergan
Tom, On 2/2/07 2:18 PM, Tom Lane [EMAIL PROTECTED] wrote: as of 8.2 there's a psql variable FETCH_COUNT that can be set to make it happen behind the scenes.) FETCH_COUNT is a godsend and works beautifully for exactly this purpose. Now he's got to worry about how to page through 8GB of

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Joshua D. Drake
Luke Lonergan wrote: Tom, On 2/2/07 2:18 PM, Tom Lane [EMAIL PROTECTED] wrote: as of 8.2 there's a psql variable FETCH_COUNT that can be set to make it happen behind the scenes.) FETCH_COUNT is a godsend and works beautifully for exactly this purpose. Now he's got to worry about how

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Luke Lonergan
\o /tmp/really_big_cursor_return ;) Tough crowd :-D - Luke ---(end of broadcast)--- TIP 2: Don't 'kill -9' the postmaster

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
On Feb 2, 2007, at 7:53 PM, Luke Lonergan wrote: Tough crowd :-D No kidding ;-)

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Joshua D. Drake
Luke Lonergan wrote: \o /tmp/really_big_cursor_return ;) Tough crowd :-D Yeah well Andrew probably would have said use sed and pipe it through awk to get the data you want. Joshua D. Drake - Luke ---(end of broadcast)--- TIP 2:

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
On Feb 2, 2007, at 7:39 PM, Luke Lonergan wrote: Tom, On 2/2/07 2:18 PM, Tom Lane [EMAIL PROTECTED] wrote: as of 8.2 there's a psql variable FETCH_COUNT that can be set to make it happen behind the scenes.) FETCH_COUNT is a godsend and works beautifully for exactly this purpose. Now

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Geoffrey
Joshua D. Drake wrote: Luke Lonergan wrote: \o /tmp/really_big_cursor_return ;) Tough crowd :-D Yeah well Andrew probably would have said use sed and pipe it through awk to get the data you want. Chances are, if you're using awk, you shouldn't need sed. :) -- Until later, Geoffrey Those

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Joshua D. Drake
Geoffrey wrote: Joshua D. Drake wrote: Luke Lonergan wrote: \o /tmp/really_big_cursor_return ;) Tough crowd :-D Yeah well Andrew probably would have said use sed and pipe it through awk to get the data you want. Chances are, if you're using awk, you shouldn't need sed. :) Chances

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Tom Lane
Kirk Wythers [EMAIL PROTECTED] writes: On Feb 2, 2007, at 7:39 PM, Luke Lonergan wrote: Now he's got to worry about how to page through 8GB of results in something less than geological time with the space bar ;-) I actually have no intention of paging through the results, but rather need

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Kirk Wythers
On Feb 2, 2007, at 8:32 PM, Tom Lane wrote: Kirk Wythers [EMAIL PROTECTED] writes: On Feb 2, 2007, at 7:39 PM, Luke Lonergan wrote: Now he's got to worry about how to page through 8GB of results in something less than geological time with the space bar ;-) I actually have no intention of

Re: [PERFORM] trouble with a join on OS X

2007-02-02 Thread Joshua D. Drake
Kirk Wythers wrote: On Feb 2, 2007, at 8:32 PM, Tom Lane wrote: Kirk Wythers [EMAIL PROTECTED] writes: On Feb 2, 2007, at 7:39 PM, Luke Lonergan wrote: Now he's got to worry about how to page through 8GB of results in something less than geological time with the space bar ;-) I actually