2015-09-17 22:13 GMT+02:00 Robert Haas <robertmh...@gmail.com>:
> On Thu, Sep 17, 2015 at 11:16 AM, Pavel Stehule <pavel.steh...@gmail.com>
> > Is there some risk if we take too much large DSM segment? And what is max
> > size of DSM segment? When we use shm_mq, we don't need to solve limits.
> I can't really think you are going to have a problem. How much data
> do you really intend to send back? Surely it's going to be <100kB.
> If you think it's not a problem to have a running query stop and send
> a gigabyte of data someplace anytime somebody asks, well, I don't
> think I agree.
I afraid so <100kB is too optimistic. I know so GoodData environment is a
exception - it uses query generator, but I found few plans >1MB . It is
unusual probably due long identifiers too - used 63bytes long hash - and
some queries and models are strange. It does translation between analytic
multi dimensional business oriented query language and SQL. Back to topic.
With high probability we can calculate <10MB.
> >> Also, if there are any bugs in the way the shm_mq is being used,
> >> they're likely to be quite rare and hard to find, because the vast
> >> majority of messages will probably be short enough to be sent in a
> >> single chunk, so whatever bugs may exist when the processes play
> >> ping-ping are unlikely to occur in practice except in unusual cases
> >> where the message being returned is very long.
> > This is true for any functionality based on shm_mq - parallel seq scan,
> Parallel sequential scan is likely to put a lot more data through a
> shm_mq than you would for this.
> >> Second, using a shm_mq manipulates the state of the process latch. I
> >> don't think you can make the assumption that it's safe to reset the
> >> process latch at any and every place where we check for interrupts.
> >> For example, suppose the process is already using a shm_mq and the
> >> CHECK_FOR_INTERRUPTS() call inside that code then discovers that
> >> somebody has activated this mechanism and you now go try to send and
> >> receive from a new shm_mq. But even if that and every other
> >> CHECK_FOR_INTERRUPTS() in the code can tolerate a process latch reset
> >> today, it's a new coding rule that could easily trip people up in the
> >> future.
> > It is valid, and probably most important. But if we introduce own
> > we will play with process latch too (although we can use LWlocks)
> With the design I proposed, there is zero need to touch the process
> latch, which is good, because I'm pretty sure that is going to be a
> problem. I don't think there is any need to use LWLocks here either.
> When you get a request for data, you can just publish a DSM segment
> with the data and that's it. Why do you need anything more? You
> could set the requestor's latch if it's convenient; that wouldn't be a
> problem. But the process supplying the data can't end up in a
> different state than it was before supplying that data, or stuff WILL
sure - but the same behave have to have shm_mq - if not, then only one can
be used for communication between process - that is little bit limiting.
> >> Using a shm_mq is appropriate when the amount of data that needs to be
> >> transmitted might be very large - too large to just allocate a buffer
> >> for the whole thing - or when the amount of data can't be predicted
> >> before memory is allocated. But there is obviously no rule that a
> >> shm_mq should be used any time we have "data exchange between
> >> processes"; we have lots of shared-memory based IPC already and have
> >> for many years, and shm_mq is newer than the vast majority of that
> >> code.
> > I am little bit disappointed - I hoped so shm_mq can be used as generic
> > interprocess mechanism - that will ensure all corner cases for work with
> > shared memory. I understand to shm_mq is new, and nobody used it, but
> > risk is better than invent wheels again and again.
> shm_mq is useful, but if you insist on using a complicated tool when a
> simple one is plenty sufficient, you may not get the results you're
> hoping for.
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company