Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-29 Thread Shaun Thomas
On 10/27/2012 10:49 PM, Віталій Тимчишин wrote: It can be that some query(s) use a lot of work mem, either because of high work_mem setting or because of planner error. In this case the moment query runs it will need memory that will later be returned and become free. Usually this can be seen

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-27 Thread Віталій Тимчишин
Sorry for late response, but may be you are still strugling. It can be that some query(s) use a lot of work mem, either because of high work_mem setting or because of planner error. In this case the moment query runs it will need memory that will later be returned and become free. Usually this

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-23 Thread Shaun Thomas
On 10/22/2012 01:44 PM, Claudio Freire wrote: I think, unless it gives you trouble with the page cache, numactl --prefer=+0 should work nicely for postgres overall. Failing that, numactl --interleave=all would, IMO, be better than the system default. Thanks, I'll consider that. FWIW, our

[PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Shaun Thomas
Hey everyone! This is pretty embarrassing, but I've never seen this before. This is our system's current memory allocation from 'free -m': total used free buffers cached Mem: 72485 58473 14012 3 34020 -/+ buffers/cache:

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Claudio Freire
On Mon, Oct 22, 2012 at 2:35 PM, Shaun Thomas stho...@optionshouse.com wrote: So, I've got 14GB of RAM that the OS is just refusing to use for disk or page cache. Does anyone know what might cause that? Maybe there's just nothing to put inside? How big is your database? How much of it gets

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Shaun Thomas
On 10/22/2012 12:44 PM, Claudio Freire wrote: Maybe there's just nothing to put inside? How big is your database? How much of it gets accessed? Trust me, there's plenty. We have a DB that's 6x larger than RAM that's currently experiencing 6000TPS, and according to iostat, anywhere from

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Frank Lanitz
On Mon, 22 Oct 2012 12:35:32 -0500 Shaun Thomas stho...@optionshouse.com wrote: Hey everyone! This is pretty embarrassing, but I've never seen this before. This is our system's current memory allocation from 'free -m': total used free buffers cached

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Claudio Freire
On Mon, Oct 22, 2012 at 2:49 PM, Shaun Thomas stho...@optionshouse.com wrote: Maybe there's just nothing to put inside? How big is your database? How much of it gets accessed? Trust me, there's plenty. We have a DB that's 6x larger than RAM that's currently experiencing 6000TPS, and

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Marcus Larsson
On Mon, Oct 22, 2012 at 12:49:49PM -0500, Shaun Thomas wrote: Trust me, there's plenty. We have a DB that's 6x larger than RAM that's currently experiencing 6000TPS, and according to iostat, anywhere from 20-60% disk utilization that's mostly reads. Could it be related to zone_reclaim_mode?

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Shaun Thomas
On 10/22/2012 12:53 PM, Claudio Freire wrote: Did you check the kernel's zone_reclaim_mode ? It's currently set to 0, which as I'm led to believe, is the setting I want there. But here's something interesting: numactl --hardware available: 2 nodes (0-1) node 0 cpus: 0 2 4 6 8 10 12 14 16

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Claudio Freire
On Mon, Oct 22, 2012 at 3:01 PM, Shaun Thomas stho...@optionshouse.com wrote: Did you check the kernel's zone_reclaim_mode ? It's currently set to 0, which as I'm led to believe, is the setting I want there. Yep But here's something interesting: numactl --hardware available: 2 nodes

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Franklin, Dan (FEN)
...@postgresql.org [mailto:pgsql-performance-ow...@postgresql.org] On Behalf Of Claudio Freire Sent: Monday, October 22, 2012 2:14 PM To: stho...@optionshouse.com Cc: pgsql-performance@postgresql.org Subject: Re: [PERFORM] Tons of free RAM. Can't make it go away. On Mon, Oct 22, 2012 at 3:01 PM, Shaun Thomas stho

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Shaun Thomas
On 10/22/2012 01:20 PM, Franklin, Dan (FEN) wrote: http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-archite cture/ Yeah, I remember reading that a while back. While interesting, it doesn't really apply to PG, in that unlike MySQL, we don't allocate any large memory segments

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Shaun Thomas
On 10/22/2012 01:14 PM, Claudio Freire wrote: You may want to try setting the numa policy before launching postgres: numactl --interleave=all pg_ctl start I thought about that. I'd try it on one of our stage nodes, but both of them show an even memory split. I'm not sure why our prod node

Re: [PERFORM] Tons of free RAM. Can't make it go away.

2012-10-22 Thread Claudio Freire
On Mon, Oct 22, 2012 at 3:24 PM, Shaun Thomas stho...@optionshouse.com wrote: http://blog.jcole.us/2010/09/28/mysql-swap-insanity-and-the-numa-archite cture/ Yeah, I remember reading that a while back. While interesting, it doesn't really apply to PG, in that unlike MySQL, we don't allocate