On Mon, Aug 9, 2010 at 7:17 PM, Tom Lane wrote:
> Robert Haas writes:
>> On Mon, Aug 9, 2010 at 4:18 PM, Tom Lane wrote:
>>> Particular implementations might cope with such cases in useful ways, or
>>> then again they might not.
>> That doesn't seem like a big problem to me. I was assuming we'd
On Mon, Aug 9, 2010 at 9:44 PM, Robert Haas wrote:
> That doesn't seem like a big problem to me. I was assuming we'd need
> to remap when the size changed.
I had thought about this in the past too, just for supporting run-time
changes to shared_buffers. I always assumed we would just allocate
sh
Robert Haas writes:
> On Mon, Aug 9, 2010 at 4:18 PM, Tom Lane wrote:
>> Particular implementations might cope with such cases in useful ways, or
>> then again they might not.
> That doesn't seem like a big problem to me. I was assuming we'd need
> to remap when the size changed.
Well, as long
On Mon, Aug 09, 2010 at 02:16:47PM -0400, Robert Haas wrote:
> I believe I went back and reread
> the old threads on this topic and it seems like the sticking point as
> far as POSIX shm goes it that it lacks a readable equivalent of
> shm_nattch. I think it was proposed to use a small syv shm and
On Mon, Aug 9, 2010 at 4:18 PM, Tom Lane wrote:
> Robert Haas writes:
>> On Mon, Aug 9, 2010 at 3:20 PM, Tom Lane wrote:
>>> It's not portable. That's exactly what we were looking into back when.
>
>> Uggh, that sucks. Can you provide any more details?
>
> You don't really have to go further t
Robert Haas wrote:
> I don't think it's going to be too easy to provide, short of (as
> Tom says) moving to the MySQL model of many threads working in a
> single process.
Well, it's a bit misleading to refer to it as the MySQL model. It's
used by Microsoft SQL Server, MySQL, Informix, and Syb
Robert Haas writes:
> On Mon, Aug 9, 2010 at 3:20 PM, Tom Lane wrote:
>> It's not portable. That's exactly what we were looking into back when.
> Uggh, that sucks. Can you provide any more details?
You don't really have to go further than consulting the relevant
standards, eg SUS says at
http
On Mon, Aug 9, 2010 at 3:20 PM, Tom Lane wrote:
> Robert Haas writes:
>> On Mon, Aug 9, 2010 at 11:41 AM, Tom Lane wrote:
>>> ... and on some platforms, it'll be flat out impossible. We looked at
>>> this years ago and concluded that changing the size of the shmem segment
>>> after postmaster s
Hi,
On 08/09/2010 09:14 PM, Tom Lane wrote:
As far as SLRU is concerned, the already-agreed-to plan is to get rid of
the separate arenas for SLRU and merge those things into the main shared
buffers arena.
I didn't know about that plan. Sounds good. (I'm personally thinking
this is trying to s
Robert Haas writes:
> On Mon, Aug 9, 2010 at 11:41 AM, Tom Lane wrote:
>> ... and on some platforms, it'll be flat out impossible. We looked at
>> this years ago and concluded that changing the size of the shmem segment
>> after postmaster start was impractical from a portability standpoint.
>>
Markus Wanner writes:
> However, I'd like to get back to the original intent of the posted
> patch. Which is about dynamically allocating memory *within a fixed size
> pool*.
> That's something SRLU or shared_buffers do to some extent, but with lots
> of limitations. And without the ability to
Hi,
On 08/09/2010 08:45 PM, Robert Haas wrote:
Yeah, I think that's a real concern. I think we need to distinguish
memory needs from memory wants. Ideally, we'd like our entire
database to be cached in RAM. But that may or may not be feasible, so
we page what we can into shared_buffers and pa
On Mon, Aug 9, 2010 at 3:00 PM, Bruce Momjian wrote:
>> That would be one way to tackle the problem, but there are
>> difficulties. If we just created new shared memory segments at need,
>> we might end up with a lot of shared memory segments. I suspect that
>> would get complicated and present
On 08/09/2010 09:00 PM, Bruce Momjian wrote:
You could allocate shared memory in chunks and then pass that out to
requestors, the same way sbrk() does it.
sbrk() is described [1] as a "low-level memory allocator", which "is
typically only used by the high-level malloc memory allocator
impleme
On 08/09/2010 08:50 PM, Bruce Momjian wrote:
You effectively have to add infrastructure to add/remove shared memory
segments to match memory requests. It is another step, but it is the
same behavior.
That's of no use without a dynamic allocator, I think. Or else it is a
vague description of a
Robert Haas wrote:
> On Mon, Aug 9, 2010 at 2:50 PM, Bruce Momjian wrote:
> > Robert Haas wrote:
> >> On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian wrote:
> >> >> > Let me be more concrete. ?Suppose you are using threads, and you want
> >> >> > to
> >> >> > increase your shared memory from 20MB
On Mon, Aug 9, 2010 at 2:50 PM, Bruce Momjian wrote:
> Robert Haas wrote:
>> On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian wrote:
>> >> > Let me be more concrete. ?Suppose you are using threads, and you want to
>> >> > increase your shared memory from 20MB to 30MB. ?How do you do that? ?If
>> >>
On 08/09/2010 08:49 PM, Bruce Momjian wrote:
Markus Wanner wrote:
That's what my patch allows you to do, yes. Currently you are bound to
pre-allocate shared memory at startup. Or how would you allocate small
chunks from shared memory at the moment?
We don't --- we allocate it all at startup.
Robert Haas wrote:
> On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian wrote:
> >> > Let me be more concrete. ?Suppose you are using threads, and you want to
> >> > increase your shared memory from 20MB to 30MB. ?How do you do that? ?If
> >> > you want it contiguous, you have to use realloc, which mig
Markus Wanner wrote:
> On 08/09/2010 08:33 PM, Bruce Momjian wrote:
> > Robert Haas wrote:
> >> You probably wouldn't do either of those things. You'd just allocate
> >> small chunks here and there for whatever you need them for.
> >
> > Well, then we do that with shared memory then --- my point i
On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian wrote:
>> > Let me be more concrete. ?Suppose you are using threads, and you want to
>> > increase your shared memory from 20MB to 30MB. ?How do you do that? ?If
>> > you want it contiguous, you have to use realloc, which might move the
>> > pointer. ?
On Mon, Aug 9, 2010 at 2:28 PM, Markus Wanner wrote:
> Another issue to be discussed would be the limits of sharing free memory
> between subsystems. Maybe we even reach the conclusion that we absolutely
> *want* fixed maximum sizes for every single subsystem so as to be able to
> guarantee a cert
On 08/09/2010 08:33 PM, Bruce Momjian wrote:
Robert Haas wrote:
You probably wouldn't do either of those things. You'd just allocate
small chunks here and there for whatever you need them for.
Well, then we do that with shared memory then --- my point is that it is
the same problem with threa
Markus Wanner wrote:
> Hi,
>
> On 08/09/2010 06:31 PM, Bruce Momjian wrote:
> > Let me be more concrete. Suppose you are using threads, and you want to
> > increase your shared memory from 20MB to 30MB. How do you do that?
>
> There's absolutely no need to pre-allocate 20 MB in advance in a
>
Robert Haas wrote:
> On Mon, Aug 9, 2010 at 12:31 PM, Bruce Momjian wrote:
> > Bruce Momjian wrote:
> >> > With our process-based design, the default is private memory (i.e. not
> >> > shared). If you need shared memory, you must specify a certain amount in
> >> > advance. That chunk of shared mem
Markus Wanner wrote:
> Hi,
>
> On 08/09/2010 06:10 PM, Bruce Momjian wrote:
> > My point is that you can treat malloc the same as "add shared memory",
> > to some extent, with the same limiations.
>
> Once one of the SLRU buffers is full, it cannot currently allocate from
> another SLRU buffer's
Hi,
On 08/09/2010 06:31 PM, Bruce Momjian wrote:
Let me be more concrete. Suppose you are using threads, and you want to
increase your shared memory from 20MB to 30MB. How do you do that?
There's absolutely no need to pre-allocate 20 MB in advance in a
threaded environment. You just allocat
On Mon, Aug 9, 2010 at 12:31 PM, Bruce Momjian wrote:
> Bruce Momjian wrote:
>> > With our process-based design, the default is private memory (i.e. not
>> > shared). If you need shared memory, you must specify a certain amount in
>> > advance. That chunk of shared memory then is reserved and can'
On Mon, Aug 9, 2010 at 12:03 PM, Bruce Momjian wrote:
> Robert Haas wrote:
>> On Mon, Aug 9, 2010 at 11:02 AM, Bruce Momjian wrote:
>> > I am not sure threads would greatly help us. ?The major problem is that
>> > all of our our structures are currently contiguous in memory for quick
>> > access.
On Mon, Aug 9, 2010 at 11:41 AM, Tom Lane wrote:
> Robert Haas writes:
>> So imagine that thread-or-process A allocates allocates a new chunk of
>> memory and then writes a pointer to the new chunk in a previously
>> allocated section of memory. Thread-or-process B then follows the
>> pointer.
Hi,
On 08/09/2010 06:10 PM, Bruce Momjian wrote:
My point is that you can treat malloc the same as "add shared memory",
to some extent, with the same limiations.
Once one of the SLRU buffers is full, it cannot currently allocate from
another SLRU buffer's unused memory area. That memory there
Hi,
On 08/09/2010 05:41 PM, Tom Lane wrote:
... and on some platforms, it'll be flat out impossible. We looked at
this years ago and concluded that changing the size of the shmem segment
after postmaster start was impractical from a portability standpoint.
I have not seen anything to change tha
On Mon, 2010-08-09 at 11:41 -0400, Tom Lane wrote:
> Robert Haas writes:
> > So imagine that thread-or-process A allocates allocates a new chunk of
> > memory and then writes a pointer to the new chunk in a previously
> > allocated section of memory. Thread-or-process B then follows the
> > point
Bruce Momjian wrote:
> > With our process-based design, the default is private memory (i.e. not
> > shared). If you need shared memory, you must specify a certain amount in
> > advance. That chunk of shared memory then is reserved and can't ever be
> > used by another subsystem. Even if you bare
Markus Wanner wrote:
> Hi,
>
> On 08/09/2010 05:02 PM, Bruce Momjian wrote:
> > [ Sorry to be jumping into this thread late.]
>
> No problem at all.
>
> > I am not sure threads would greatly help us.
>
> Note that I'm absolutely, certainly not advocating the use of threads
> for Postgres.
>
>
Robert Haas wrote:
> On Mon, Aug 9, 2010 at 11:02 AM, Bruce Momjian wrote:
> > I am not sure threads would greatly help us. ?The major problem is that
> > all of our our structures are currently contiguous in memory for quick
> > access. ?I don't see how threading would help with that. ?We could u
Robert Haas writes:
> So imagine that thread-or-process A allocates allocates a new chunk of
> memory and then writes a pointer to the new chunk in a previously
> allocated section of memory. Thread-or-process B then follows the
> pointer. In a threaded model, this is guaranteed to be safe. In
Hi,
On 08/09/2010 05:02 PM, Bruce Momjian wrote:
[ Sorry to be jumping into this thread late.]
No problem at all.
I am not sure threads would greatly help us.
Note that I'm absolutely, certainly not advocating the use of threads
for Postgres.
The major problem is that
all of our our st
On Mon, Aug 9, 2010 at 11:02 AM, Bruce Momjian wrote:
> I am not sure threads would greatly help us. The major problem is that
> all of our our structures are currently contiguous in memory for quick
> access. I don't see how threading would help with that. We could use
> realloc(), but we can
Markus Wanner wrote:
> Hi,
>
> On 07/26/2010 07:16 PM, Robert Haas wrote:
> > Of course, there are other parts of the system (a whole bunch of them)
> > that used shared memory also, and perhaps some of those could be
> > modified to use the dynamic allocator as well. But they're getting by
> > w
On Mon, Jul 26, 2010 at 3:16 PM, Kevin Grittner
wrote:
> Robert Haas wrote:
>
>> I actually think that memory management is one of the weakest
>> elements of our current architecture
>
> I'm actually pretty impressed by the memory contexts in PostgreSQL.
> Apparently I'm not alone in that, either
Robert Haas wrote:
> I actually think that memory management is one of the weakest
> elements of our current architecture
I'm actually pretty impressed by the memory contexts in PostgreSQL.
Apparently I'm not alone in that, either; a paper by Hellerstein,
Stonebraker, and Hamilton[1] has this
On Mon, Jul 26, 2010 at 1:50 PM, Markus Wanner wrote:
> Note however, that a thread based design doesn't have this problem *at all*.
> Memory generally is shared (between threads) and you can dynamically
> allocate more or less (until Linux' OOM killer hits you.. yet another
> story). The OS reuse
Hi,
On 07/26/2010 07:16 PM, Robert Haas wrote:
Of course, there are other parts of the system (a whole bunch of them)
that used shared memory also, and perhaps some of those could be
modified to use the dynamic allocator as well. But they're getting by
without it now, so maybe they don't really
On Mon, Jul 26, 2010 at 12:51 PM, Markus Wanner wrote:
>> Dynamically allocating out of a 2MB
>> segment gives up most of that flexibility.
>
> Absolutely, that's why I'd like to see other modules that use the dynamic
> allocator. The more the better.
Right, I agree. The problem is that I don't
Hi,
On 07/26/2010 06:33 PM, Robert Haas wrote:
It would be nice to think, for example, that this could be used as
infrastructure for parallel query to stream results back from worker
processes to the backend connected to the user. If you're using 16
processors to concurrently scan 16 partitions
Hi,
On 07/26/2010 04:31 PM, Alvaro Herrera wrote:
Excerpts from Robert Haas's message of lun jul 26 08:52:46 -0400 2010:
Here's another idea. Instead of making imessages use an SLRU, how
about having it steal pages from shared_buffers? This would require
segmenting messages into small enough
On Mon, Jul 26, 2010 at 10:31 AM, Alvaro Herrera
wrote:
> Excerpts from Robert Haas's message of lun jul 26 08:52:46 -0400 2010:
>> Here's another idea. Instead of making imessages use an SLRU, how
>> about having it steal pages from shared_buffers? This would require
>> segmenting messages into
Excerpts from Robert Haas's message of lun jul 26 08:52:46 -0400 2010:
> Here's another idea. Instead of making imessages use an SLRU, how
> about having it steal pages from shared_buffers? This would require
> segmenting messages into small enough chunks that they'd fit, but the
> nice part is
On Thu, Jul 22, 2010 at 3:09 PM, Markus Wanner wrote:
>> FWIW I don't think you should be thinking in "replacing imessages with
>> SLRU". I rather think you should be thinking in how can you implement
>> the imessages API on top of SLRU.
>
> Well, I'm rather comparing SLRU with the dynamic alloca
Hi,
On 07/22/2010 08:31 PM, Alvaro Herrera wrote:
FWIW I don't think you should be thinking in "replacing imessages with
SLRU". I rather think you should be thinking in how can you implement
the imessages API on top of SLRU.
Well, I'm rather comparing SLRU with the dynamic allocator. So far I
Excerpts from Markus Wanner's message of jue jul 22 08:49:29 -0400 2010:
> Of course, as mentioned in the bgworker patch, this could be done
> differently. Using solely shared memory, or maybe SLRU to store change
> sets. However, I certainly like the abstraction and guarantees such a
> message
Greg,
On 07/22/2010 03:59 PM, Greg Smith wrote:
There's a fairly good mapping of what techniques are patented and which
were only mentioned in research in the Sun dynamic memory patent at
http://www.freepatentsonline.com/7328316.html ; that mentions an earlier
paper by the author of the techniqu
Markus Wanner wrote:
On 07/20/2010 09:05 PM, Alvaro Herrera wrote:
Hmm, deriving code from a paper published by IBM sounds like bad news --
who knows what patents they hold on the techniques there?
Yeah, that might be an issue. Note, however, that the lock-based
variant differs substantially
Hi,
On 07/22/2010 01:04 PM, Robert Haas wrote:
Well, shared_buffers has to be allocated as one contiguous slab
because we index into it that way. So I don't really see how
dynamically allocating memory could help. What you'd need is a
different system for assigning buffer tags, so that a parti
On Thu, Jul 22, 2010 at 3:01 AM, Markus Wanner wrote:
> On 07/22/2010 12:11 AM, Robert Haas wrote:
>>
>> I'm not sure why merging the SLRU pools with shared_buffers would
>> benefit from dynamically allocated shared memory.
>
> Well, I'm not sure how you'd merge SLRU pools with shared_buffers. IMO
Hi,
On 07/22/2010 12:11 AM, Robert Haas wrote:
I'm not sure why merging the SLRU pools with shared_buffers would
benefit from dynamically allocated shared memory.
Well, I'm not sure how you'd merge SLRU pools with shared_buffers. IMO
that inherently leads to the problem of allocating memory d
On Wed, Jul 21, 2010 at 2:53 PM, Markus Wanner wrote:
>> Consider also the contrary situation,
>> where the imessages stuff is not in use (even for a short period of
>> time, like a few minutes). Then we'd really rather not still have
>> memory carved out for it.
>
> Huh? That's exactly what dyna
Hi,
first of all, thanks for your feedback, I enjoy the discussion.
On 07/21/2010 07:25 PM, Robert Haas wrote:
Given what you're trying to do, it does sound like you're going to
need some kind of an algorithm for space management; but you'll be
managing space within the SLRU rather than within
On Wed, Jul 21, 2010 at 4:33 AM, Markus Wanner wrote:
> Okay, so I just need to grok the SLRU stuff. Thanks for clarifying.
>
> Note that I sort of /want/ to mess with shared memory. It's what I know how
> to deal with. It's how threaded programs work as well. Ya know, locks,
> conditional variabl
On 07/21/2010 01:52 AM, Robert Haas wrote:
On Tue, Jul 20, 2010 at 5:46 PM, Alvaro Herrera
wrote:
I guess what Robert is saying is that you don't need shmem to pass
messages around. The new LISTEN implementation was just an example.
imessages aren't supposed to use it directly. Rather, the i
On Tue, Jul 20, 2010 at 5:46 PM, Alvaro Herrera
wrote:
> Excerpts from Markus Wanner's message of mar jul 20 14:54:42 -0400 2010:
>
>> > With respect to imessages specifically, what is the motivation for
>> > using shared memory rather than something like an SLRU? The new
>> > LISTEN implementati
Excerpts from Markus Wanner's message of mar jul 20 14:54:42 -0400 2010:
> > With respect to imessages specifically, what is the motivation for
> > using shared memory rather than something like an SLRU? The new
> > LISTEN implementation uses an SLRU and handles variable-size messages,
> > so it
Hi,
On 07/20/2010 09:05 PM, Alvaro Herrera wrote:
Hmm, deriving code from a paper published by IBM sounds like bad news --
who knows what patents they hold on the techniques there?
Yeah, that might be an issue. Note, however, that the lock-based variant
differs substantially from what's been
Excerpts from Markus Wanner's message of mar jul 20 14:36:55 -0400 2010:
> > I'm also unconvinced that spinlocks are the best locking primitive here.
> > Why not lwlocks?
>
> It's derived from a completely lock-free algorithm, as proposed by Maged
> M. Michael in: Scalable Lock-Free Dynamic Memo
Hi,
On 07/20/2010 08:23 PM, Robert Haas wrote:
Well, you can't really fix that problem with this infrastructure,
No, but it would allow you to better use the existing amount of shared
memory. Possibly avoiding the problem is certain scenarios.
The failure might
manifest itself in a totally
Hello Alvaro,
thank you for looking through this code.
On 07/20/2010 07:50 PM, Alvaro Herrera wrote:
Interesting, thanks.
I gave it a skim and found that it badly needs a lot more code comments.
Hm.. yeah, the dynshmem stuff could probably need more comments. (The
bgworker stuff is probably
On Tue, Jul 20, 2010 at 1:50 PM, Alvaro Herrera
wrote:
> I'm not sure what kind of resistance you'll see to the idea of a
> dynamically allocatable shmem area. Maybe we could use this in other
> areas such as allocating space for heavyweight lock objects. Right now
> the memory usage for them co
Excerpts from Markus Wanner's message of vie jul 02 19:44:46 -0400 2010:
> Having written a very primitive kind of a dynamic memory allocator for
> imessages [1], I've always wanted a better alternative. So I've
> investigated a bit, refactored step-by-step, and finally came up with
> the attac
Hi,
for quite some time, I've been under the impression, that there's still
one disadvantage left from using processes instead of threads: we can
only use statically sized chunks of shared memory. Every component that
wants to use shared memory needs to pre-allocate whatever it thinks is
suffi
70 matches
Mail list logo
