Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-10 Thread Noah Misch
I'm okay with any of the proposed designs or with dropping the idea. Closing the loop on a few facts: On Sat, Mar 07, 2015 at 04:34:41PM -0600, Jim Nasby wrote: > If we go that route, does it still make sense to explicitly use > repalloc_huge? It will just cut over to that at some point (128M?) a

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-09 Thread Andres Freund
On 2015-03-09 17:12:22 -0500, Jim Nasby wrote: > That will be significantly more code than a simple repalloc, but as long as > people are OK with that I can go that route. I still would like to see some actual evidence of need for change before we invest more time/code here. Greetings, Andres Fr

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-09 Thread Jim Nasby
On 3/9/15 12:28 PM, Alvaro Herrera wrote: Robert Haas wrote: On Sat, Mar 7, 2015 at 5:49 PM, Andres Freund wrote: On 2015-03-05 15:28:12 -0600, Jim Nasby wrote: I was thinking the simpler route of just repalloc'ing... the memcpy would suck, but much less so than the extra index pass. 64M gets

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-09 Thread Alvaro Herrera
Robert Haas wrote: > On Sat, Mar 7, 2015 at 5:49 PM, Andres Freund wrote: > > On 2015-03-05 15:28:12 -0600, Jim Nasby wrote: > >> I was thinking the simpler route of just repalloc'ing... the memcpy would > >> suck, but much less so than the extra index pass. 64M gets us 11M tuples, > >> which prob

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-09 Thread Robert Haas
On Sat, Mar 7, 2015 at 5:49 PM, Andres Freund wrote: > On 2015-03-05 15:28:12 -0600, Jim Nasby wrote: >> I was thinking the simpler route of just repalloc'ing... the memcpy would >> suck, but much less so than the extra index pass. 64M gets us 11M tuples, >> which probably isn't very common. > > T

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-07 Thread Jim Nasby
On 3/7/15 4:49 PM, Andres Freund wrote: On 2015-03-05 15:28:12 -0600, Jim Nasby wrote: I was thinking the simpler route of just repalloc'ing... the memcpy would suck, but much less so than the extra index pass. 64M gets us 11M tuples, which probably isn't very common. That has the chance of co

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-07 Thread Andres Freund
On 2015-03-05 15:28:12 -0600, Jim Nasby wrote: > I was thinking the simpler route of just repalloc'ing... the memcpy would > suck, but much less so than the extra index pass. 64M gets us 11M tuples, > which probably isn't very common. That has the chance of considerably increasing the peak memory

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-07 Thread Jim Nasby
On 3/7/15 12:48 AM, Noah Misch wrote: On Sat, Mar 07, 2015 at 12:46:42AM -0500, Tom Lane wrote: Noah Misch writes: On Thu, Mar 05, 2015 at 03:28:12PM -0600, Jim Nasby wrote: I was thinking the simpler route of just repalloc'ing... the memcpy would suck, but much less so than the extra index p

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-06 Thread Noah Misch
On Sat, Mar 07, 2015 at 12:46:42AM -0500, Tom Lane wrote: > Noah Misch writes: > > On Thu, Mar 05, 2015 at 03:28:12PM -0600, Jim Nasby wrote: > >> I was thinking the simpler route of just repalloc'ing... the memcpy would > >> suck, but much less so than the extra index pass. 64M gets us 11M tuples

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-06 Thread Tom Lane
Noah Misch writes: > On Thu, Mar 05, 2015 at 03:28:12PM -0600, Jim Nasby wrote: >> I was thinking the simpler route of just repalloc'ing... the memcpy would >> suck, but much less so than the extra index pass. 64M gets us 11M tuples, >> which probably isn't very common. > +1. Start far below 64

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-06 Thread Noah Misch
On Thu, Mar 05, 2015 at 03:28:12PM -0600, Jim Nasby wrote: > On 3/4/15 9:10 AM, Robert Haas wrote: > >On Wed, Feb 25, 2015 at 5:06 PM, Jim Nasby wrote: > >>Could the large allocation[2] for the dead tuple array in lazy_space_alloc > >>cause problems with linux OOM? [1] and some other things I've r

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-05 Thread Jim Nasby
On 3/4/15 9:10 AM, Robert Haas wrote: On Wed, Feb 25, 2015 at 5:06 PM, Jim Nasby wrote: Could the large allocation[2] for the dead tuple array in lazy_space_alloc cause problems with linux OOM? [1] and some other things I've read indicate that a large mmap will count towards total system memory

Re: [HACKERS] Question about lazy_space_alloc() / linux over-commit

2015-03-04 Thread Robert Haas
On Wed, Feb 25, 2015 at 5:06 PM, Jim Nasby wrote: > Could the large allocation[2] for the dead tuple array in lazy_space_alloc > cause problems with linux OOM? [1] and some other things I've read indicate > that a large mmap will count towards total system memory, including > producing a failure i