Re: [HACKERS] Explicit psqlrc

[Bruce Momjian]

Simon Riggs wrote:
> On Tue, 2010-07-20 at 09:05 -0400, Robert Haas wrote:
> > On Tue, Jul 20, 2010 at 8:21 AM, Simon Riggs  wrote:
> > > On Tue, 2010-07-20 at 07:49 -0400, Robert Haas wrote:
> > >> A further point is that it's very difficult to
> > >> keep track of progress if the CF page reflects a whole bunch of
> > >> supposedly "Waiting on Author" patches that are really quite
> > >> thoroughly dead.
> > >
> > > True, but the point under discussion is what to do if no reply is
> > > received from an author. That is something entirely different from a
> > > patch hitting a brick wall.
> > >
> > > We gain nothing by moving early on author-delay situations, so I suggest
> > > we don't.
> > 
> > No, we gain something quite specific and tangible, namely, the
> > expectation that patch authors will stay on top of their patches if
> > they want them reviewed by the community.  If that expectation doesn't
> > seem important to you, feel free to try running a CommitFest without
> > it.  If you can make it work, I'll happily sign on.
> 
> I don't think so. We can assume people wrote a patch because they want
> it included in Postgres. Bumping them doesn't help them or us, since
> there is always an issue other than wish-to-complete. Not everybody is
> able to commit time in the way we do and we should respect that better.
> 
> Authors frequently have to wait a long time for a review; why should
> reviewers not be as patient as authors must be?
> 
> We should be giving authors as much leeway as possible, or they may not
> come back.

By marking patches as 'returned with feedback' long before the end of
the commit-fest, we show feedback of how close we are to completing the
commit-fest.  If we keep patches in limbo status, it is unclear how
close we are to CF completion.  And, of course, the author can
reactivate the patch just by replying.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] MERGE Specification

[Heikki Linnakangas]

On 10/08/10 06:03, Boxuan Zhai wrote:

I have put everything in one patch, against the latest git repository. The
program is tested on my machine.


Thanks! I get a few compiler warnings:

analyze.c: In function ‘transformMergeStmt’:
analyze.c:2476: warning: unused variable ‘lastaction’
gram.y: In function ‘base_yyparse’:
gram.y:7437: warning: assignment from incompatible pointer type
gram.y:7441: warning: assignment from incompatible pointer type
trigger.c: In function ‘ExecBSMergeTriggers’:
trigger.c:2360: warning: assignment from incompatible pointer type
trigger.c: In function ‘ExecASMergeTriggers’:
trigger.c:2411: warning: assignment from incompatible pointer type
planner.c: In function ‘merge_action_planner’:
planner.c:681: warning: assignment from incompatible pointer type
var.c: In function ‘push_up_merge_action_vars’:
var.c:738: warning: passing argument 1 of 
‘push_up_merge_action_vars_walker’ from incompatible pointer type
var.c:96: note: expected ‘struct Node *’ but argument is of type ‘struct 
List *’
var.c:740: warning: passing argument 1 of 
‘push_up_merge_action_vars_walker’ from incompatible pointer type
var.c:96: note: expected ‘struct Node *’ but argument is of type ‘struct 
List *’


The merge.sgml file should be in doc/src/sgml/ref, not doc/src/sgml. 
After moving it there, I get a few errors from compiling the docs:


openjade:ref/merge.sgml:128:55:X: reference to non-existent ID 
"SQL-SELECT-TITLE"
openjade:ref/merge.sgml:129:55:X: reference to non-existent ID 
"SQL-VALUES-TITLE"
openjade:ref/merge.sgml:185:42:X: reference to non-existent ID 
"SQL-INSERT-TITLE"
openjade:ref/merge.sgml:170:42:X: reference to non-existent ID 
"SQL-UPDATE-TITLE"


Those can be fixed by simply removing the endterm attributes from those 
lines, they're not needed.


--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Explicit psqlrc

[Bruce Momjian]

Kevin Grittner wrote:
> > We should be giving authors as much leeway as possible, or they
> > may not come back.
>  
> One phenomenon I've noticed is that sometimes a patch is submitted
> because an end user has solved their own problem for themselves, but
> wishes to share the solution with the community.  They're not always
> motivated to go to the lengths required to polish it up to the
> standard required for inclusion in core.  In such cases, unless
> someone with the time to do so finds it interesting enough to pick
> up, it is just going to drop.  I hope such authors feel comfortable
> submitting their next effort, as it might be something which
> interests a larger audience than the previous effort.  We should do
> what we can to ensure that they understand the dynamics of that.

This brings up the larger issue of whether incomplete/unapplied patches
are recorded on the TODO list or just ignored.  We never really came up
with a plan for that.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TODO 9.0 done items removed

[Bruce Momjian]

Andrew Dunstan wrote:
> 
> 
> On 08/09/2010 09:36 PM, Tom Lane wrote:
> > "David E. Wheeler"  writes:
> >> On Aug 9, 2010, at 5:45 PM, Bruce Momjian wrote:
> >>> I figured it out;  done:
> >>> http://wiki.postgresql.org/wiki/TodoDone90
> >> Jeepers. That's a long list!
> > Uh, there seems to be quite a lot there that is *not* done in 9.0.
> > In fact, none of the first page's worth are done ... I stopped
> > reading at that point.
> >
> > 
> 
> 
> yeah. It seems complete nonsense to me too. For example, this item is BS 
> AFAIK:
> 
> 
>   PL/Perl
> 
> Incomplete item
> Allow data
> to be passed in native language formats, rather than only text

Thanks, removed.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TODO 9.0 done items removed

[Bruce Momjian]

Tom Lane wrote:
> "David E. Wheeler"  writes:
> > On Aug 9, 2010, at 5:45 PM, Bruce Momjian wrote:
> >> I figured it out;  done:
> >> http://wiki.postgresql.org/wiki/TodoDone90
> 
> > Jeepers. That's a long list!
> 
> Uh, there seems to be quite a lot there that is *not* done in 9.0.
> In fact, none of the first page's worth are done ... I stopped
> reading at that point.

Oh, I thought the wiki page was only a snapshot of the TODO list as of
9.0.  I now see it is just completed items.  I have updated the 9.0 TODO
page accordingly:

http://wiki.postgresql.org/wiki/TodoDone90

Now I see why the page is useful.  :-)

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] host name support in pg_hba.conf

[Peter Eisentraut]

On mån, 2010-08-09 at 15:25 -0400, Tom Lane wrote:
> Peter Eisentraut  writes:
> > Here is a patch for host name support in pg_hba.conf.
> 
> My recollection is that the previous discussions got stuck on the cost
> of doing DNS lookups for every connect; and the obvious solution of
> trying to cache the names was shot down on the basis of not knowing when
> to flush the cache.  Have you decided that people who want this feature
> will just have to pay that cost?

Either that or install nscd.

While the speed of establishing a connection is critical to some setups,
there are also plenty of setups where the connection setup speed is
pretty much unimportant because of connection pooling.

> If so, I think the documentation needs
> to be a bit more explicit about names being more expensive than IP
> addresses in pg_hba.conf.

Fair enough.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] host name support in pg_hba.conf

[Peter Eisentraut]

On mån, 2010-08-09 at 23:01 +0200, Kristian Larsson wrote:
> In which order are things evaluated? What if I only include IP
> addresses in my pg_hba, will it still cost me a DNS lookup or
> two?

No, if you don't use this feature, it won't cost you anything.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] host name support in pg_hba.conf

[Peter Eisentraut]

On mån, 2010-08-09 at 13:56 -0500, Kevin Grittner wrote:
> Peter Eisentraut  wrote:
>  
> > The client's IP address (known from the kernel)
>  
> Some machines have several IP addresses; how is that handled?

A connection comes in over exactly one address.
 
> > is reverse looked up, which results in a host name.
>  
> Some IP addresses have several host names, including in reverse
> lookup; how is that handled?

This is not possible, or at least the C library APIs don't expose it.
Compare the getnameinfo() and getaddrinfo() man pages, for example.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] host name support in pg_hba.conf

[Peter Eisentraut]

On mån, 2010-08-09 at 15:05 -0400, Robert Haas wrote:
> -1 from me, on this part.  I think we should be trying to eliminate
> any dependencies we have on how localhost resolves, and we certainly
> should not add more.

Maybe this is something that distributors could add if they have more
knowledge about the "localhost" policies on their hosts.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] grouping sets - updated patch

[Pavel Stehule]

2010/8/10 Joshua Tolley :
> On Mon, Aug 09, 2010 at 10:59:26PM +0200, Pavel Stehule wrote:
>> Hello
>>
>> I fixed an issues with empty sets. It just work, but there are some ugly 
>> hacks.
>>
>> It's really needs own planner node - now grouping functions are not
>> supported by ORDER BY clause.
>
> I haven't made it through the last version much, but I'll poke through this
> instead. I have a few days of family business coming up, and might be
> unrespondive during that time.
>

ok,

Pavel
> --
> Joshua Tolley / eggyknap
> End Point Corporation
> http://www.endpoint.com
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.9 (GNU/Linux)
>
> iEYEARECAAYFAkxgxlgACgkQRiRfCGf1UMM3NwCgkJ3EEWIj6MLiDcU2SHT/hH7a
> 4BsAn2hTqqzsLYLFZbflIJK/x/WMsZ2d
> =dIod
> -END PGP SIGNATURE-
>
>

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] grouping sets - updated patch

[Joshua Tolley]

On Mon, Aug 09, 2010 at 10:59:26PM +0200, Pavel Stehule wrote:
> Hello
> 
> I fixed an issues with empty sets. It just work, but there are some ugly 
> hacks.
> 
> It's really needs own planner node - now grouping functions are not
> supported by ORDER BY clause.

I haven't made it through the last version much, but I'll poke through this
instead. I have a few days of family business coming up, and might be
unrespondive during that time.

--
Joshua Tolley / eggyknap
End Point Corporation
http://www.endpoint.com


signature.asc
Description: Digital signature

Re: [HACKERS] Universal B-tree

[Daniel Oliveira]

There is a way to acess a index inside a c function without using a sql
statement ?

Best regards,

Daniel Oliveira

2010/8/9 Daniel Oliveira 

> For research purpose, I think that expression index is a good idea. I just
> want to do a proof of concept.
>
> The other issue is that my algorithm break a z-order interval into several
> intervals that represents the query box. How should I create it without
> creating any overhead?
>
> Best regards,
>
> daniel
>
> 2010/8/9 Greg Stark 
>
> On Mon, Aug 9, 2010 at 5:31 PM, Daniel Oliveira
>>  wrote:
>> > I don't need to change B-tree estructure. I just need integrate my
>> encode
>> > function that transforms multiple keys into one key by bit-interleaving
>> and
>> > to acess elements given several intervals (range search).
>>
>> You could build a "expression" index on the function which will build
>> a regular btree. Then any range clause on a similar expression in the
>> query would be able to use the index. Postgres is capable of detecting
>> multiple range clauses that can use one or more indexes and combine
>> the results.  It might not be perfect for your use case, being a
>> general purpose tool.
>>
>> I'm not sure how feasible it would be to implement a special purpose
>> case that matches your needs. Perhaps a special index type and index
>> access method similar to gist "tsquery" data type. But that's a lot
>> more work and a lot more C coding.
>>
>> --
>> greg
>>
>
>

[HACKERS] 1-byte id for SharedInvalidationMessages

[Robert Haas]

On Fri, Aug 6, 2010 at 2:43 PM, Tom Lane  wrote:
>> Do you think it's worth worrying about the reduction in the number of
>> possible SI message types?
>
> IIRC the number of message types is the number of catalog caches plus
> half a dozen or so.  We're a long way from exhausting even a 1-byte
> ID field; and we could play more games if we had to, since there would
> be a padding byte free in the message types that refer to a catalog
> cache.  IOW, 1-byte id doesn't bother me.

I took a look at what is required to implement $SUBJECT tonight and it
appears to be remarkably straightforward.  I suppose this qualifies as
a reason to increment WAL_PAGE_MAGIC, since SharedInvalidationMessages
are now xlogged; and I added an Assert() to
AddCatCacheInvalidationMessage to detect overruns of the id field, but
other than that it seems to be just a matter of s/int16/int8/ in a
handful of places.

For those following along at home:

http://archives.postgresql.org/pgsql-hackers/2010-07/msg00355.php [the
patch for which this is a prerequisite]
http://archives.postgresql.org/pgsql-hackers/2010-08/msg00366.php [why
it needs it]
http://archives.postgresql.org/pgsql-hackers/2010-08/msg00425.php [how
this helps]

Thoughts?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company


sinvalmsg-v1.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TODO 9.0 done items removed

[Andrew Dunstan]

  
  


On 08/09/2010 09:36 PM, Tom Lane wrote:

  "David E. Wheeler"  writes:

  
On Aug 9, 2010, at 5:45 PM, Bruce Momjian wrote:


  I figured it out;  done:
http://wiki.postgresql.org/wiki/TodoDone90


  
  

  
Jeepers. That's a long list!

  
  
Uh, there seems to be quite a lot there that is *not* done in 9.0.
In fact, none of the first page's worth are done ... I stopped
reading at that point.

			




yeah. It seems complete nonsense to me too. For example, this item
is BS AFAIK:

PL/Perl 
  Allow data to
  be passed in native language formats, rather than only text
  


cheers

andrew

  

Re: [HACKERS] TODO 9.0 done items removed

[Tom Lane]

"David E. Wheeler"  writes:
> On Aug 9, 2010, at 5:45 PM, Bruce Momjian wrote:
>> I figured it out;  done:
>> http://wiki.postgresql.org/wiki/TodoDone90

> Jeepers. That's a long list!

Uh, there seems to be quite a lot there that is *not* done in 9.0.
In fact, none of the first page's worth are done ... I stopped
reading at that point.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Robert Haas]

On Mon, Aug 9, 2010 at 7:17 PM, Tom Lane  wrote:
> Robert Haas  writes:
>> On Mon, Aug 9, 2010 at 4:18 PM, Tom Lane  wrote:
>>> Particular implementations might cope with such cases in useful ways, or
>>> then again they might not.
>> That doesn't seem like a big problem to me.  I was assuming we'd need
>> to remap when the size changed.
> Well, as long as you can do that, sure.  I'm concerned about what
> happens if/when remapping fails (not at all unlikely in 32-bit address
> spaces in particular).  You mentioned that that would probably have to
> be a PANIC condition, which I think I agree with; and that idea pretty
> much kills any argument that this would be a good way to improve server
> uptime.

In some cases, you might be able to get by with FATAL.  Still, it's
easier to imagine using this in cases for things like resizing
shared_buffers (where the alternative is to restart the server anyway)
than it is to use it for routine memory allocation.

> Another issue is that if you're doing dynamic remapping you almost
> certainly can't assume that the segment will appear at the same
> addresses in every backend.  We could live with that for shared buffers
> without too much pain, but not so much for most other shared
> datastructures.

Hmm.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TODO 9.0 done items removed

[David E. Wheeler]

On Aug 9, 2010, at 5:45 PM, Bruce Momjian wrote:

> I figured it out;  done:
> 
>   http://wiki.postgresql.org/wiki/TodoDone90

Jeepers. That's a long list!

David


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Greg Stark]

On Mon, Aug 9, 2010 at 9:44 PM, Robert Haas  wrote:
> That doesn't seem like a big problem to me.  I was assuming we'd need
> to remap when the size changed.

I had thought about this in the past too, just for supporting run-time
changes to shared_buffers. I always assumed we would just allocate
shared memory in chunks and create separate mappings for each chunk.



-- 
greg

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] TODO 9.0 done items removed

[Bruce Momjian]

> Alvaro Herrera wrote:
> > Excerpts from Bruce Momjian's message of sb jul 17 21:21:52 -0400 
> > 2010:
> > > Since we branched 9.1 before we released Postgres 9.0, I had to remove
> > > the 9.0 TODO items before 9.0 was released, or people might have marked
> > > items as "done" when they were done only in 9.1.
> >
> > Should we create a TodoDone90 page like this one?
> > http://wiki.postgresql.org/wiki/TodoDone84
> 
> Sure.  How is that done?  :-O

I figured it out;  done:

http://wiki.postgresql.org/wiki/TodoDone90

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] RecordTransactionCommit() and SharedInvalidationMessages

[Robert Haas]

It appears to me that RecordTransactionCommit() only needs to WAL-log
shared invalidation messages when wal_level is hot_standby, but I
don't see a guard to prevent it from doing it in all cases.  Am I
missing something?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist - 0.8

[David Fetter]

On Mon, Aug 09, 2010 at 06:35:47PM -0300, Euler Taveira de Oliveira wrote:
> Alexander Korotkov escreveu:
> > Such approach can give benefit when we need to filter by
> > similarity. For example, in pg_trgm "%" is used for similarity
> > filtering, but similarity threshold is global for session. That's
> > why we can't create complex queries which contain similarity
> > filtering with different threshold.
> > 
> What do you mean by complex queries? You can always use the SET
> command. Sadly it doesn't work when you have different thresholds
> within distinct subqueries.  (In pg_similarity I use this approach
> to set the function's thresholds). What I am investigating is a way
> to build an index with some user-defined parameters. (We already
> have some infra-structure in reloptions for that but it needs some
> work to support my idea). I have some half-baked patch that I'm
> planning to submit to some of the CFs. Unfortunately, I don't have
> time for it ATM.

Do you have enough of it to send out as WIP?

Cheers,
David.
-- 
David Fetter  http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david.fet...@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] security label support, part.2

[Robert Haas]

2010/8/9  :
>> 2. Some of this code refers to "local" security labels.  I'm not sure
>> what's "local" about them - aren't they just security labels?  On a
>> related note, I don't like the trivial wrappers you have here, with
>> DeleteSecurityLabel around DeleteLocalSecLabel, SetSecurityLabel
>> around SetLocalSecLabel, etc.  Just collapse these into a single set
>> of functions.
>>
> In the feature, I plan to assign security labels on the shared database
> objects such as pg_database. The "local" is a contradistinction
> towards these "shared" objects.

Oh, I see.  I don't think that's entirely clear: and in any event it
seems a bit premature, since we're not at that point yet.  Let's just
get rid of this stuff for now as I suggested.

>> 5. Why do we think that the relabel hook needs to be passed the number
>> of expected parents?
>>
> We need to prevent relabeling on inherited relations/columns from
> the multiple origin, like ALTER RENAME TO.
> It requires to pass the expected parents into the provider, or
> to check it in the caller.

Please explain further.  I don't understand.

>> 6. What are we doing about the assignment of initial security labels?
>> I had initially thought that perhaps new objects would just start out
>> unlabeled, and the user would be responsible for labeling them as
>> needed.  But maybe we can do better.  Perhaps we should extend the
>> security provider hook API with a function that gets called when a
>> labellable object gets created, and let each loaded security provider
>> return any label it would like attached.  Even if we don't do that
>> now, esp_relabel_hook_entry needs to be renamed to something more
>> generic; we will certainly want to add more fields to that structure
>> later.
>>
> Starting with "unlabeled" is wrong, because it does not distinguish
> an object created by security sensitive users and insensitive users,
> for example. It is similar to relation's relowner is InvalidOid.
>
> I plan the security provider hook on the creation time works two things.
> 1. It checks user's privilege to create this object.
> 2. It returns security labels to be assigned.
>
> Then, the caller assigns these returned labels on the new object,
> if one or more valid labels are returned.

OK, let's give that a try and see how it looks.  I don't think that's
in this version of the patch, right?

>> 7. I think we need to write and include in the fine documentation some
>> "big picture" documentation about enhanced security providers.  Of
>> course, we have to decide what we want to say.  But the SECURITY LABEL
>> documentation is just kind of hanging out there in space right now; it
>> needs to connect to a broad introduction to the subject.
>>
> OK, I'll try to describe with appropriate granularity.
> Do we need an independent section in addition to the introduction of
> SECURITY LABEL syntax?

I think so.  I suggest a new chapter called "Enhanced Security
Providers" just after "Database Roles and Privileges".

> BTW, I'll go on the area where internet unconnectable during next
> two days. Perhaps, my reply will run late.

No problem.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Tom Lane]

Robert Haas  writes:
> On Mon, Aug 9, 2010 at 4:18 PM, Tom Lane  wrote:
>> Particular implementations might cope with such cases in useful ways, or
>> then again they might not.

> That doesn't seem like a big problem to me.  I was assuming we'd need
> to remap when the size changed.

Well, as long as you can do that, sure.  I'm concerned about what
happens if/when remapping fails (not at all unlikely in 32-bit address
spaces in particular).  You mentioned that that would probably have to
be a PANIC condition, which I think I agree with; and that idea pretty
much kills any argument that this would be a good way to improve server
uptime.

Another issue is that if you're doing dynamic remapping you almost
certainly can't assume that the segment will appear at the same
addresses in every backend.  We could live with that for shared buffers
without too much pain, but not so much for most other shared
datastructures.

> Also, I was assuming that we were
> going to use shms, not files.

It looked to me like the spec for mmap was the same either way.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

[Simon Riggs]

On Tue, 2010-08-10 at 07:43 +0900, Itagaki Takahiro wrote:
> 2010/8/10 Simon Riggs :
> > Any SQL statement that reads a block can do HOT pruning, if the block is
> > otherwise unlocked.
> 
> We use the term "HOT" for two features:
>   1. HOT updates: Avoiding index updates when keys are not modified.
>   2. HOT pruning: Removing tuple bodies, that works even for indexed tuples.
> 
> 2 is the point of the case, but ambiguous "HOT" might confuse some people.

Good point. We should say "HOT updates" and "pruning" as separate
techniques. Itagaki invented the pruning technique and deserves much
credit for that.

-- 
 Simon Riggs   www.2ndQuadrant.com
 PostgreSQL Development, 24x7 Support, Training and Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SHOW TABLES

[Kevin Grittner]

Bruce Momjian  wrote:
> Kevin Grittner wrote:
 
>> I can't picture anything which could be done with views which
>> would allow me to issue one statement and see everything of
>> interest about a table (etc.).  You know: tablespace, owner,
>> permissions, columns, primary key, foreign keys, check
>> constraints, exclusion constraints, ancestor tables, child
>> tables, and whatever interesting features I missed or we later
>> add.  Other products allow that to be generated server-side, so
>> that it is available to any and all clients.  I think we should
>> join the crowd in this respect.
> 
> Consider if the server-side description comes to the client
> unformatted, then that format is going to changes as Postgres adds
> features, and that might really make the output useless except for
> raw display purposes.
 
Yeah, sending it back as unformatted text would be horrible.  See
this post:
 
http://archives.postgresql.org/pgsql-hackers/2010-07/msg00738.php
 
If you follow the link there, you will see examples of another
product returning up to 11 result sets and a few INFO level messages
in response to a single "sp_help objectname".  If it's technically
feasible for PostgreSQL to do something like that, it would
absolutely rock.  Result sets have enough structure to them to be
able to write code dealing with such behavior with relative
confidence.
 
For more naive ramblings by me on the topic, see:
 
http://archives.postgresql.org/pgsql-hackers/2010-07/msg00752.php
http://archives.postgresql.org/pgsql-hackers/2010-07/msg00892.php
http://archives.postgresql.org/pgsql-hackers/2010-07/msg00909.php
 
Nobody has yet come out and said that these ideas *aren't*
technically feasible, so I continue to live with my dreams intact. 
So far
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

[Itagaki Takahiro]

2010/8/10 Simon Riggs :
> Any SQL statement that reads a block can do HOT pruning, if the block is
> otherwise unlocked.

We use the term "HOT" for two features:
  1. HOT updates: Avoiding index updates when keys are not modified.
  2. HOT pruning: Removing tuple bodies, that works even for indexed tuples.

2 is the point of the case, but ambiguous "HOT" might confuse some people.

-- 
Itagaki Takahiro

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SHOW TABLES

[Bruce Momjian]

Kevin Grittner wrote:
> Stefan Kaltenbrunner  wrote:
> > On 07/18/2010 08:58 PM, Andres Freund wrote:
>  
> >> I am quite a bit surprised about all this discussion. I have a
> >> very hard time we will find anything people agree about and can
> >> remember well enough to be usefull for both manual and automatic
> >> processing.
> >>
> >> I agree that the internal pg_* tables are not exactly easy to
> >> query. And that the information_schema. ones arent complete
> >> enough and have enough concept mismatch to be confusing. But why
> >> all this?
> > 
> > exactly my thoughts - but as I said earlier maybe this is actually
> > an opportunity to look at newsysviews again?
>  
> I can't picture anything which could be done with views which would
> allow me to issue one statement and see everything of interest about
> a table (etc.).  You know: tablespace, owner, permissions, columns,
> primary key, foreign keys, check constraints, exclusion constraints,
> ancestor tables, child tables, and whatever interesting features I
> missed or we later add.  Other products allow that to be generated
> server-side, so that it is available to any and all clients.  I
> think we should join the crowd in this respect.

Consider if the server-side description comes to the client unformatted,
then that format is going to changes as Postgres adds features, and that
might really make the output useless except for raw display purposes.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] new client release

[Andrew Dunstan]

There is a new release of the buildfarm client. There are no new 
features, but some cleanup and bug fixes.


The code can be downloaded from: 



Here are the release notes / change log:

. Use git clean to remove build products left by vpath builds, following a 
suggestion from Robert Haas.

. provide for "default" entry in force_every setting

. Accomodate oddities in at least some versions of File::Copy.

. If the web transaction fails, put back the old timestamps so the next run 
works on the same basis.

. provide enough log to avoid server failure if we get an SCM error (bug found 
following report from Greg Sabino Mullane)

. work with older versions of git (bug found following report from Greg Sabino 
Mullane)


cheers

andrew




--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] security label support, part.2

[kaigai]

Thanks for your reviewing.

On Mon, 9 Aug 2010 16:02:12 -0400
Robert Haas  wrote:

> 2010/7/26 KaiGai Kohei :
> > The attached patches are revised ones, as follows.
> 
> I think this is pretty good, and I'm generally in favor of committing
> it.  Some concerns:
> 
> 1. Since nobody has violently objected to the comment.c refactoring
> patch I recently proposed, I'm hopeful that can go in.  And if that's
> the case, then I'd prefer to see that committed first, and then rework
> this to use that code.  That would eliminate some code here, and it
> would also make it much easier to support labels on other types of
> objects.
> 
It seems to me fair enough. This parse_object.c can also provide
a facility to resolve the name of object to be labeled.

> 2. Some of this code refers to "local" security labels.  I'm not sure
> what's "local" about them - aren't they just security labels?  On a
> related note, I don't like the trivial wrappers you have here, with
> DeleteSecurityLabel around DeleteLocalSecLabel, SetSecurityLabel
> around SetLocalSecLabel, etc.  Just collapse these into a single set
> of functions.
> 
In the feature, I plan to assign security labels on the shared database
objects such as pg_database. The "local" is a contradistinction
towards these "shared" objects.

> 3. Is it really appropriate for ExecRelationSecLabel() to have an
> "Exec" prefix?  I don't think so.
> 
I don't have any preferences about 

> 4. Please get rid of the nkeys++ stuff in DeleteLocalSecLabel() and
> just use fixed offsets as we do everywhere else.
> 
OK, I'll fix it.

> 5. Why do we think that the relabel hook needs to be passed the number
> of expected parents?
> 
We need to prevent relabeling on inherited relations/columns from
the multiple origin, like ALTER RENAME TO.
It requires to pass the expected parents into the provider, or
to check it in the caller.

> 6. What are we doing about the assignment of initial security labels?
> I had initially thought that perhaps new objects would just start out
> unlabeled, and the user would be responsible for labeling them as
> needed.  But maybe we can do better.  Perhaps we should extend the
> security provider hook API with a function that gets called when a
> labellable object gets created, and let each loaded security provider
> return any label it would like attached.  Even if we don't do that
> now, esp_relabel_hook_entry needs to be renamed to something more
> generic; we will certainly want to add more fields to that structure
> later.
> 
Starting with "unlabeled" is wrong, because it does not distinguish
an object created by security sensitive users and insensitive users,
for example. It is similar to relation's relowner is InvalidOid.

I plan the security provider hook on the creation time works two things.
1. It checks user's privilege to create this object.
2. It returns security labels to be assigned.

Then, the caller assigns these returned labels on the new object,
if one or more valid labels are returned.

> 7. I think we need to write and include in the fine documentation some
> "big picture" documentation about enhanced security providers.  Of
> course, we have to decide what we want to say.  But the SECURITY LABEL
> documentation is just kind of hanging out there in space right now; it
> needs to connect to a broad introduction to the subject.
> 
OK, I'll try to describe with appropriate granularity.
Do we need an independent section in addition to the introduction of
SECURITY LABEL syntax?

> 8. Generally, the English in both the comments and documentation needs
> work.  However, we can address that problem when we're closer to
> commit.
> 
OK


BTW, I'll go on the area where internet unconnectable during next
two days. Perhaps, my reply will run late.

Thanks,
--
KaiGai Kohei 

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SHOW TABLES

[Robert Haas]

On Mon, Aug 9, 2010 at 5:42 PM, Bruce Momjian  wrote:
> Sorry for the late reply.  If we are going to end up recreating SQL, we
> might as well just keep the backslash mess we have, or tell them to use
> SQL for the complex queries.  My point was that we might find that what
> we cook up is as complex as what we have now.

I think that would require malice aforethought.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SHOW TABLES

[Simon Riggs]

On Mon, 2010-08-09 at 17:42 -0400, Bruce Momjian wrote:
> Robert Haas wrote:
> > On Sat, Jul 17, 2010 at 11:14 PM, Bruce Momjian  wrote:
> > > Robert Haas wrote:
> > >> I'd like to be able to list comments on objects of a particular type.
> > >> And, yeah, I'd like to be able to list all the aggregates that take a
> > >> numeric argument, or all the functions that take, say, an argument of
> > >> type internal. ?Right now, this is an ENORMOUS pain in the neck. ?I
> > >> usually end up running psql -c '' | ?grep |
> > >> awk ... or something like that. ?I have no idea what Windows users do.
> > >> ?I'm sure it's possible to write a query to do it, but it's not
> > >> anything approaching easy. ?All of this talk about backslash commands
> > >> being powerful rings totally hollow for me. ?For ordinary, day to day
> > >> tasks like listing all my tables, or looking at the details of a
> > >> particular table, they're great. ?I use them all the time and would
> > >> still use them even if some other syntax were available. ?But there is
> > >> no reasonable way to pass options to them, and that to me is a pretty
> > >> major drawback.
> > >
> > > I am concerned that implementing a command syntax to show complex output
> > > like above effectively means re-implementing a subset of SQL, and that
> > > subset will never be as flexible.
> > 
> > That's a reasonable concern, but I don't have a better idea.  Do you?
> 
> Sorry for the late reply.  If we are going to end up recreating SQL, we
> might as well just keep the backslash mess we have, or tell them to use
> SQL for the complex queries.  My point was that we might find that what
> we cook up is as complex as what we have now.

My proposal is that SHOW TABLES returns exactly the same output as \d
yet works the same from every interface.

I have no intention of designing or writing what Robert proposes above
and he is welcome to do that, but its clearly a different requirement.

SHOW TABLES is a simple command with a simple purpose: helping newbies
by putting obvious commands in their way that do useful things. The
simple goal I've outlined for SHOW TABLES has nothing whatsoever to do
with inventing what appears to be a new flexible catalog metadata
language that is hard to implement, non-standard and poorly understood
except by only a few people.

Currently, \d runs this SQL

SELECT n.nspname as "Schema",
  c.relname as "Name",
  CASE c.relkind WHEN 'r' THEN 'table' WHEN 'v' THEN 'view' WHEN 'i'
THEN 'index' WHEN 'S' THEN 'sequence' WHEN 's' THEN 'special' END as
"Type",
  pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
FROM pg_catalog.pg_class c
 LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
WHERE c.relkind IN ('r','v','S','')
  AND n.nspname <> 'pg_catalog'
  AND n.nspname <> 'information_schema'
  AND n.nspname !~ '^pg_toast'
  AND pg_catalog.pg_table_is_visible(c.oid)
ORDER BY 1,2;

I would make this into a server view, and then make SHOW TABLES a
synonym for SELECT * FROM pg_show_backslash_d. I'd probably work out a
better name for the view also. It's mostly just refactoring.

-- 
 Simon Riggs   www.2ndQuadrant.com
 PostgreSQL Development, 24x7 Support, Training and Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: to_string, to_array functions

[Pavel Stehule]

Hello

2010/8/9 Tom Lane :
> Brendan Jurd  writes:
>>> I have attached v4 of the patch against HEAD, and also an incremental
>>> patch showing just my changes against v3.
>>>
>>> I'll mark this as ready for committer.
>
> Looking at this, I want to question the implode/explode naming.  I think
> those names are too cute by half, not particularly mnemonic, not visibly
> related to the similar existing functions, and not friendly to any
> future extension in the same area.
>
> My first thought is that we should go back to the string_to_array and
> array_to_string names.  The key reason not to use those names was the
> conflict with the old functions if you didn't specify a third argument,
> but where is the advantage of not specifying the third argument?  It
> would be a lot simpler for people to understand if we just said "the
> two-argument forms work like this, while the three-argument forms work
> like that".  This is especially reasonable because the difference in
> behavior is about nulls in the array, which is exactly what the third
> argument exists to specify.
>

The name isn't important - I believe so you or Robert can choose the
best name. Important is default behave. On an start is idea, so
functions that lost some information isn't optimal - and  it is
array_to_string problem - because this function quietly skip NULL
fields, if there are. So it was a motivation to write these functions.

Regards

Pavel Stehule

> [ Sorry for not complaining about this before, but I was on vacation
> when the previous naming discussion went on. ]
>
>                        regards, tom lane
>

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SHOW TABLES

[Bruce Momjian]

Robert Haas wrote:
> On Sat, Jul 17, 2010 at 11:14 PM, Bruce Momjian  wrote:
> > Robert Haas wrote:
> >> I'd like to be able to list comments on objects of a particular type.
> >> And, yeah, I'd like to be able to list all the aggregates that take a
> >> numeric argument, or all the functions that take, say, an argument of
> >> type internal. ?Right now, this is an ENORMOUS pain in the neck. ?I
> >> usually end up running psql -c '' | ?grep |
> >> awk ... or something like that. ?I have no idea what Windows users do.
> >> ?I'm sure it's possible to write a query to do it, but it's not
> >> anything approaching easy. ?All of this talk about backslash commands
> >> being powerful rings totally hollow for me. ?For ordinary, day to day
> >> tasks like listing all my tables, or looking at the details of a
> >> particular table, they're great. ?I use them all the time and would
> >> still use them even if some other syntax were available. ?But there is
> >> no reasonable way to pass options to them, and that to me is a pretty
> >> major drawback.
> >
> > I am concerned that implementing a command syntax to show complex output
> > like above effectively means re-implementing a subset of SQL, and that
> > subset will never be as flexible.
> 
> That's a reasonable concern, but I don't have a better idea.  Do you?

Sorry for the late reply.  If we are going to end up recreating SQL, we
might as well just keep the backslash mess we have, or tell them to use
SQL for the complex queries.  My point was that we might find that what
we cook up is as complex as what we have now.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: to_string, to_array functions

[Bruce Momjian]

David E. Wheeler wrote:
> On Aug 9, 2010, at 1:10 PM, Robert Haas wrote:
> 
> >> My first thought is that we should go back to the string_to_array and
> >> array_to_string names.  The key reason not to use those names was the
> >> conflict with the old functions if you didn't specify a third argument,
> >> but where is the advantage of not specifying the third argument?  It
> >> would be a lot simpler for people to understand if we just said "the
> >> two-argument forms work like this, while the three-argument forms work
> >> like that".  This is especially reasonable because the difference in
> >> behavior is about nulls in the array, which is exactly what the third
> >> argument exists to specify.
> >> 
> >> [ Sorry for not complaining about this before, but I was on vacation
> >> when the previous naming discussion went on. ]
> > 
> > I can live with that, as long as it's clearly explained in the docs.
> 
> +1

+1

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist - 0.8

[Euler Taveira de Oliveira]

Alexander Korotkov escreveu:
> Such approach
> can give benefit when we need to filter by similarity. For example, in
> pg_trgm "%" is used for similarity filtering, but similarity threshold
> is global for session. That's why we can't create complex queries which
> contain similarity filtering with different threshold.
> 
What do you mean by complex queries? You can always use the SET command. Sadly
it doesn't work when you have different thresholds within distinct subqueries.
 (In pg_similarity I use this approach to set the function's thresholds). What
I am investigating is a way to build an index with some user-defined
parameters. (We already have some infra-structure in reloptions for that but
it needs some work to support my idea). I have some half-baked patch that I'm
planning to submit to some of the CFs. Unfortunately, I don't have time for it
 ATM.


-- 
  Euler Taveira de Oliveira
  http://www.timbira.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Synchronous replication

[Bruce Momjian]

Fujii Masao wrote:
> On Wed, Aug 4, 2010 at 10:38 PM, Heikki Linnakangas
>  wrote:
> > Then you risk running out of disk space. Similar to having an archive
> > command that fails for some reason.
> >
> > That's one reason the registration should not be too automatic - there is
> > serious repercussions if the standby just disappears. If the standby is a
> > synchronous one, the master will stop committing or delay acknowledging
> > commits, depending on the configuration, and the master needs to keep extra
> > WAL around.
> 
> Umm... in addition to registration of each standby, I think we should allow
> users to set the upper limit of the number of WAL files kept in pg_xlog to
> avoid running out of disk space. If it exceeds the upper limit, the master
> disconnects too old standbys from the cluster and removes all the WAL files
> not required for current connected standbys. If you don't want any standby
> to disappear unexpectedly because of the upper limit, you can set it to 0
> (= no limit).
> 
> I'm thinking to make users register and unregister each standbys via SQL
> functions like register_standby() and unregister_standby():
> 
> void register_standby(standby_name text, streaming_start_lsn text)
> void unregister_standby(standby_name text)
> 
> Note that standby_name should be specified in recovery.conf of each
> standby.
> 
> By using them we can easily specify which WAL files are unremovable because
> of new standby when taking the base backup for it as follows:
> 
> SELECT register_standby('foo', pg_start_backup())

I know there has been discussion about how to identify the standby
servers --- how about using the connection application_name in
recovery.conf:

primary_conninfo = 'host=localhost port=5432 application_name=slave1'

The good part is that once recovery.conf goes away because it isn't a
standby anymore, the the application_name is gone.

An even more interesting approach would be to specify the replication
mode in the application_name:

primary_conninfo = 'host=localhost port=5432 application_name=replay'

and imagine being able to view the status of standby servers from
pg_stat_activity.  (Right now standby servers do not appear in
pg_stat_activity.)

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] host name support in pg_hba.conf

[Kristian Larsson]

On Mon, Aug 09, 2010 at 03:25:23PM -0400, Tom Lane wrote:
> Peter Eisentraut  writes:
> > Here is a patch for host name support in pg_hba.conf.
> 
> My recollection is that the previous discussions got stuck on the cost
> of doing DNS lookups for every connect; and the obvious solution of
> trying to cache the names was shot down on the basis of not knowing when
> to flush the cache.  Have you decided that people who want this feature
> will just have to pay that cost?  If so, I think the documentation needs
> to be a bit more explicit about names being more expensive than IP
> addresses in pg_hba.conf.

I am also a bit scared of the startup cost for each connection.
 
In which order are things evaluated? What if I only include IP
addresses in my pg_hba, will it still cost me a DNS lookup or
two? 

Kind regards,
   Kristian. 

-- 
Kristian LarssonKLL-RIPE
+46 704 264511k...@spritelink.net

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Synchronous replication

[Heikki Linnakangas]

I wonder if we can continue to rely on the pg_sleep() loop for sleeping 
in walsender. On those platforms where interrupts don't interrupt sleep, 
sending the signal is not going to promptly wake up walsender. That was 
fine before, but any delay is going to be poison to synchronous 
replication performance.


Thoughts?

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Synchronous replication

[Heikki Linnakangas]

On 01/08/10 15:30, Greg Stark wrote:

On Sun, Aug 1, 2010 at 7:11 AM, Heikki Linnakangas
  wrote:

I don't think any of this quorum stuff makes much sense without explicitly
registering standbys in the master.


This doesn't have to be done manually. The streaming protocol could
include the standby sending its system id to the master. The master
could just keep a list of system ids with the last record they've been
sent and the last they've confirmed receipt, fsync, application,
whatever the protocol covers. If the same system reconnects it just
overwrites the existing data for that system id.


Systemid doesn't work for that. Systemid is assigned at initdb time, so 
all the standbys have the same systemid as the master.


--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Martijn van Oosterhout]

On Mon, Aug 09, 2010 at 02:16:47PM -0400, Robert Haas wrote:
> I believe I went back and reread
> the old threads on this topic and it seems like the sticking point as
> far as POSIX shm goes it that it lacks a readable equivalent of
> shm_nattch.  I think it was proposed to use a small syv shm and then
> do the main shared memory arena with shm_open, but at that point you
> start to wonder you're messing around with at all.

About using a small sysV segment for nattach and allocating the rest
another way: the reason to do it is that "the other way" can be
anything other than sysV. Namely, sysV has pathetic default limits
whereas you can mmap() a few gig anonymously and the kernel won't bat
an eyelid.

Even if "the other way" didn't allow you to resize anything (which is
what people appear to be talking about here) the benefit of being able
to specify useful sizes of shared buffers without having to reconfigure
the kernel makes it (ISTM) worthwhile doing irrespective of anything
else.

Have a nice day,
-- 
Martijn van Oosterhout  http://svana.org/kleptog/
> Patriotism is when love of your own people comes first; nationalism,
> when hate for people other than your own comes first. 
>   - Charles de Gaulle


signature.asc
Description: Digital signature

Re: [HACKERS] Synchronous replication

[Heikki Linnakangas]

On 05/08/10 17:14, Fujii Masao wrote:

I'm thinking to make users register and unregister each standbys via SQL
functions like register_standby() and unregister_standby():


The register/unregister facility should be accessible from the streaming 
replication connection, so that you don't need to connect to any 
particular database in addition to the streaming connection.


--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Robert Haas]

On Mon, Aug 9, 2010 at 4:18 PM, Tom Lane  wrote:
> Robert Haas  writes:
>> On Mon, Aug 9, 2010 at 3:20 PM, Tom Lane  wrote:
>>> It's not portable.  That's exactly what we were looking into back when.
>
>> Uggh, that sucks.  Can you provide any more details?
>
> You don't really have to go further than consulting the relevant
> standards, eg SUS says at
> http://www.opengroup.org/onlinepubs/007908799/xsh/mmap.html
>
>        If the size of the mapped file changes after the call to mmap() as a
>        result of some other operation on the mapped file, the effect of
>        references to portions of the mapped region that correspond to added
>        or removed portions of the file is unspecified.
>
> Particular implementations might cope with such cases in useful ways, or
> then again they might not.

That doesn't seem like a big problem to me.  I was assuming we'd need
to remap when the size changed.  Also, I was assuming that we were
going to use shms, not files.  Take a look at this:

http://www.opengroup.org/onlinepubs/007908799/xsh/shm_open.html -and-
http://www.opengroup.org/onlinepubs/007908799/xsh/ftruncate.html

From the ftruncate page: "If fildes references a shared memory object,
ftruncate() sets the size of the shared memory object to length."

> And even if your platform does, you've set
> an upper limit for the possible segment size in your mmap() call.
>
> Further down the page, SUS also takes pains to point out that you
> probably can't have an unlimited number of mapped regions, so adding
> more mmap'd segments isn't a way out either.

Yeah.  I think any approach that is based on allocating new segments
as needed is pretty much DOA.  I think the point of this would be to
be able to resize things like shared_buffers on the fly - that is, an
explicit administrator action might trigger a resize-and-remap cycle,
but general system activity would not.  The reality is that as
PostgreSQL is used in more and more 24x7 contexts and people put more
and more critical data into it, forced server restarts become more and
more of a problem.  IMHO, we really need to do some creative thinking
about how to crank PGC_POSTMASTER GUCs down to PGC_SIGHUP.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: to_string, to_array functions

[Merlin Moncure]

On Mon, Aug 9, 2010 at 4:34 PM, Tom Lane  wrote:
> Merlin Moncure  writes:
>> Is there any reason why array functions need the type prefix when
>> other type conversion functions don't?  Why didn't we name unnest()
>> array_unnest()?
>
> UNNEST() is in the standard, IIRC, so you'd have to ask the SQL
> committee that.  (And no, they're not exactly being consistent either,
> see array_agg() for example.)
>
> But anyway, my point here is that these functions are close enough to
> the existing string_to_array/array_to_string functions that they should
> be presented as variants of those, not arbitrarily assigned unrelated
> new names.  Whether we'd have chosen different names if we had it to do
> over is academic.

I don't array_agg is the same case, because you're aggregating into an
array, not from one.  all the same, +1 to your names (didn't like
explode much).

merlin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: to_string, to_array functions

[David E. Wheeler]

On Aug 9, 2010, at 1:10 PM, Robert Haas wrote:

>> My first thought is that we should go back to the string_to_array and
>> array_to_string names.  The key reason not to use those names was the
>> conflict with the old functions if you didn't specify a third argument,
>> but where is the advantage of not specifying the third argument?  It
>> would be a lot simpler for people to understand if we just said "the
>> two-argument forms work like this, while the three-argument forms work
>> like that".  This is especially reasonable because the difference in
>> behavior is about nulls in the array, which is exactly what the third
>> argument exists to specify.
>> 
>> [ Sorry for not complaining about this before, but I was on vacation
>> when the previous naming discussion went on. ]
> 
> I can live with that, as long as it's clearly explained in the docs.

+1

David



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: to_string, to_array functions

[Tom Lane]

Merlin Moncure  writes:
> Is there any reason why array functions need the type prefix when
> other type conversion functions don't?  Why didn't we name unnest()
> array_unnest()?

UNNEST() is in the standard, IIRC, so you'd have to ask the SQL
committee that.  (And no, they're not exactly being consistent either,
see array_agg() for example.)

But anyway, my point here is that these functions are close enough to
the existing string_to_array/array_to_string functions that they should
be presented as variants of those, not arbitrarily assigned unrelated
new names.  Whether we'd have chosen different names if we had it to do
over is academic.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Kevin Grittner]

Robert Haas  wrote:
 
> I don't think it's going to be too easy to provide, short of (as
> Tom says) moving to the MySQL model of many threads working in a
> single process.
 
Well, it's a bit misleading to refer to it as the MySQL model.  It's
used by Microsoft SQL Server, MySQL, Informix, and Sybase.  IBM DB2
supports four different process models, and OS threads in a single
process is the default for them on an OS with good threading
support; otherwise they default to one process per connection.
 
Just because MySQL uses a particular technique doesn't
*automatically* mean it's a bad one; it's just not in itself a
confidence-builder.  ;-)
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Hector Beyers wants to stay in touch on LinkedIn

[Hector Beyers]

LinkedIn
Hector Beyers souhaite se connecter à vous sur LinkedIn :
--

Amine,

I'd like to add you to my professional network on LinkedIn.

- Hector Beyers

Accepter l'invitation de Hector Beyers
http://www.linkedin.com/e/cm7uxn-gcns21oh-6h/VWCdriNHrcXMfYpwEG6pcCKmodzjIoHmhgnzwrRR5NC/blk/I76828881_6/6lColZJrmZznQNdhjRQnOpBtn9QfmhBt71BoSd1p65Lr6lOfPpvcjwUe38Udzt9bQJzolFTk7BVbPoVej8Od3oQdj4LrCBxbOYWrSlI/EML_comm_afe/

Voir l'invitation de Hector Beyers
http://www.linkedin.com/e/cm7uxn-gcns21oh-6h/VWCdriNHrcXMfYpwEG6pcCKmodzjIoHmhgnzwrRR5NC/blk/I76828881_6/dBYNe3wUczwSdQALqnpPbOYWrSlI/svi/

--

Quel est l' intérêt de faire partie du réseau de Hector Beyers ?

Vous avez une question ? Le réseau de Hector Beyers a probablement une réponse :
La rubrique Réponses de LinkedIn vous permet de vous informer auprès de Hector 
Beyers et de votre réseau. Obtenez l'information que vous recherchez auprès de 
professionnels expérimentés. 

http://www.linkedin.com/e/cm7uxn-gcns21oh-6h/ash/inv19_ayn/

 
--
(c) 2010, LinkedIn Corporation

Re: [HACKERS] patch: to_string, to_array functions

[Merlin Moncure]

On Mon, Aug 9, 2010 at 4:08 PM, Tom Lane  wrote:
> Brendan Jurd  writes:
>>> I have attached v4 of the patch against HEAD, and also an incremental
>>> patch showing just my changes against v3.
>>>
>>> I'll mark this as ready for committer.
>
> Looking at this, I want to question the implode/explode naming.  I think
> those names are too cute by half, not particularly mnemonic, not visibly
> related to the similar existing functions, and not friendly to any
> future extension in the same area.
>
> My first thought is that we should go back to the string_to_array and
> array_to_string names.  The key reason not to use those names was the
> conflict with the old functions if you didn't specify a third argument,
> but where is the advantage of not specifying the third argument?  It
> would be a lot simpler for people to understand if we just said "the
> two-argument forms work like this, while the three-argument forms work
> like that".  This is especially reasonable because the difference in
> behavior is about nulls in the array, which is exactly what the third
> argument exists to specify.

Is there any reason why array functions need the type prefix when
other type conversion functions don't?  Why didn't we name unnest()
array_unnest()?

merlin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Tom Lane]

Robert Haas  writes:
> On Mon, Aug 9, 2010 at 3:20 PM, Tom Lane  wrote:
>> It's not portable.  That's exactly what we were looking into back when.

> Uggh, that sucks.  Can you provide any more details?

You don't really have to go further than consulting the relevant
standards, eg SUS says at
http://www.opengroup.org/onlinepubs/007908799/xsh/mmap.html

If the size of the mapped file changes after the call to mmap() as a
result of some other operation on the mapped file, the effect of
references to portions of the mapped region that correspond to added
or removed portions of the file is unspecified.

Particular implementations might cope with such cases in useful ways, or
then again they might not.  And even if your platform does, you've set
an upper limit for the possible segment size in your mmap() call.

Further down the page, SUS also takes pains to point out that you
probably can't have an unlimited number of mapped regions, so adding
more mmap'd segments isn't a way out either.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: to_string, to_array functions

[Robert Haas]

On Mon, Aug 9, 2010 at 4:08 PM, Tom Lane  wrote:
> Brendan Jurd  writes:
>>> I have attached v4 of the patch against HEAD, and also an incremental
>>> patch showing just my changes against v3.
>>>
>>> I'll mark this as ready for committer.
>
> Looking at this, I want to question the implode/explode naming.  I think
> those names are too cute by half, not particularly mnemonic, not visibly
> related to the similar existing functions, and not friendly to any
> future extension in the same area.
>
> My first thought is that we should go back to the string_to_array and
> array_to_string names.  The key reason not to use those names was the
> conflict with the old functions if you didn't specify a third argument,
> but where is the advantage of not specifying the third argument?  It
> would be a lot simpler for people to understand if we just said "the
> two-argument forms work like this, while the three-argument forms work
> like that".  This is especially reasonable because the difference in
> behavior is about nulls in the array, which is exactly what the third
> argument exists to specify.
>
> [ Sorry for not complaining about this before, but I was on vacation
> when the previous naming discussion went on. ]

I can live with that, as long as it's clearly explained in the docs.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Robert Haas]

On Mon, Aug 9, 2010 at 3:20 PM, Tom Lane  wrote:
> Robert Haas  writes:
>> On Mon, Aug 9, 2010 at 11:41 AM, Tom Lane  wrote:
>>> ... and on some platforms, it'll be flat out impossible.  We looked at
>>> this years ago and concluded that changing the size of the shmem segment
>>> after postmaster start was impractical from a portability standpoint.
>>> I have not seen anything to change that conclusion.
>
>> I haven't done extensive research into this, but I did take a look at
>> it briefly.  It looked to me like the style of shared memory we're
>> using now (I guess it's System V) has no way to resize a shared memory
>> segment at all, and certainly no way that's portable.  However it also
>> looked as though POSIX shm (shm_open, etc.) can be resized using
>> ftruncate().  Whether this is portable to all the platforms we run on,
>> or whether the behavior of ftruncate() in combination with shm_open()
>> is in the standard, I'm not sure.
>
> It's not portable.  That's exactly what we were looking into back when.

Uggh, that sucks.  Can you provide any more details?

>> I believe I went back and reread
>> the old threads on this topic and it seems like the sticking point as
>> far as POSIX shm goes it that it lacks a readable equivalent of
>> shm_nattch.
>
> Yeah, that was another little problem.  In principle though we only need
> one SysV-style shmem segment to get the required interlock, and there
> could be add-on shmem segments using POSIX or other APIs.  But that
> doesn't get you out from under the portability issue or the memory space
> management issue (it's unlikely you can enlarge a segment without
> remapping it).

Unlikely is probably an understatement.  Still, enlarging a segment
with remapping might be workable for some useful subset of the cases.
But, if enlarging it can't be done portably, then we're pretty much
dead in the water.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: to_string, to_array functions

[Tom Lane]

Brendan Jurd  writes:
>> I have attached v4 of the patch against HEAD, and also an incremental
>> patch showing just my changes against v3.
>> 
>> I'll mark this as ready for committer.

Looking at this, I want to question the implode/explode naming.  I think
those names are too cute by half, not particularly mnemonic, not visibly
related to the similar existing functions, and not friendly to any
future extension in the same area.

My first thought is that we should go back to the string_to_array and
array_to_string names.  The key reason not to use those names was the
conflict with the old functions if you didn't specify a third argument,
but where is the advantage of not specifying the third argument?  It
would be a lot simpler for people to understand if we just said "the
two-argument forms work like this, while the three-argument forms work
like that".  This is especially reasonable because the difference in
behavior is about nulls in the array, which is exactly what the third
argument exists to specify.

[ Sorry for not complaining about this before, but I was on vacation
when the previous naming discussion went on. ]

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] security label support, part.2

[Robert Haas]

2010/7/26 KaiGai Kohei :
> The attached patches are revised ones, as follows.

I think this is pretty good, and I'm generally in favor of committing
it.  Some concerns:

1. Since nobody has violently objected to the comment.c refactoring
patch I recently proposed, I'm hopeful that can go in.  And if that's
the case, then I'd prefer to see that committed first, and then rework
this to use that code.  That would eliminate some code here, and it
would also make it much easier to support labels on other types of
objects.

2. Some of this code refers to "local" security labels.  I'm not sure
what's "local" about them - aren't they just security labels?  On a
related note, I don't like the trivial wrappers you have here, with
DeleteSecurityLabel around DeleteLocalSecLabel, SetSecurityLabel
around SetLocalSecLabel, etc.  Just collapse these into a single set
of functions.

3. Is it really appropriate for ExecRelationSecLabel() to have an
"Exec" prefix?  I don't think so.

4. Please get rid of the nkeys++ stuff in DeleteLocalSecLabel() and
just use fixed offsets as we do everywhere else.

5. Why do we think that the relabel hook needs to be passed the number
of expected parents?

6. What are we doing about the assignment of initial security labels?
I had initially thought that perhaps new objects would just start out
unlabeled, and the user would be responsible for labeling them as
needed.  But maybe we can do better.  Perhaps we should extend the
security provider hook API with a function that gets called when a
labellable object gets created, and let each loaded security provider
return any label it would like attached.  Even if we don't do that
now, esp_relabel_hook_entry needs to be renamed to something more
generic; we will certainly want to add more fields to that structure
later.

7. I think we need to write and include in the fine documentation some
"big picture" documentation about enhanced security providers.  Of
course, we have to decide what we want to say.  But the SECURITY LABEL
documentation is just kind of hanging out there in space right now; it
needs to connect to a broad introduction to the subject.

8. Generally, the English in both the comments and documentation needs
work.  However, we can address that problem when we're closer to
commit.

I am going to mark this Returned with Feedback because I don't believe
it's realistic to get the comment code committed in the next week,
rework this patch, and then get this patch committed also.  However,
I'm feeling pretty good about this effort and I think we're making
good progress toward getting this done.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] host name support in pg_hba.conf

[Stephen Frost]

* Kevin Grittner (kevin.gritt...@wicourts.gov) wrote:
> > The client's IP address (known from the kernel)
>  
> Some machines have several IP addresses; how is that handled?

Sounds like he already described it, or I read it wrong.  The fact that
some machines have several IP addresses hardly matters- whatever IP is
used to connect to PG is what gets the reverse DNS lookup.  That then
returns a host.  That host is then looked up, and as long as *one* of
the IPs associated with that host matches the IP of the connector, it's
good to go.

> > is reverse looked up, which results in a host name.
>  
> Some IP addresses have several host names, including in reverse
> lookup; how is that handled?

Yea...  That's just busted, imnsho.  But then, that's probably
because it breaks Kerberos too. :)

Thanks,

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] dynamically allocating chunks from shared memory

[Markus Wanner]

Hi,

On 08/09/2010 09:14 PM, Tom Lane wrote:

As far as SLRU is concerned, the already-agreed-to plan is to get rid of
the separate arenas for SLRU and merge those things into the main shared
buffers arena.


I didn't know about that plan. Sounds good. (I'm personally thinking 
this is trying to solve the same problem in a more specific fashion).



IIRC, the motivation for designing SLRU the way it is
was to ensure that SLRU uses couldn't be starved for memory due to high
demand for shared buffers.  But that was back when people frequently ran
PG with only a few meg for shared buffers; I think that worry is
obsolete.


Good to know.


So I don't see this patch as offering anything at all that we care about
so far as the core server is concerned.  Maybe there are extensions that
need it badly enough to justify such a feature in core, but SLRU is not
a good argument for it.


Fair enough.

(Patch is already marked as "returned with feedback" on the commitfest 
app, thanks again for additional feedback)


Regards

Markus Wanner

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] host name support in pg_hba.conf

[Tom Lane]

Peter Eisentraut  writes:
> Here is a patch for host name support in pg_hba.conf.

My recollection is that the previous discussions got stuck on the cost
of doing DNS lookups for every connect; and the obvious solution of
trying to cache the names was shot down on the basis of not knowing when
to flush the cache.  Have you decided that people who want this feature
will just have to pay that cost?  If so, I think the documentation needs
to be a bit more explicit about names being more expensive than IP
addresses in pg_hba.conf.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Tom Lane]

Robert Haas  writes:
> On Mon, Aug 9, 2010 at 11:41 AM, Tom Lane  wrote:
>> ... and on some platforms, it'll be flat out impossible.  We looked at
>> this years ago and concluded that changing the size of the shmem segment
>> after postmaster start was impractical from a portability standpoint.
>> I have not seen anything to change that conclusion.

> I haven't done extensive research into this, but I did take a look at
> it briefly.  It looked to me like the style of shared memory we're
> using now (I guess it's System V) has no way to resize a shared memory
> segment at all, and certainly no way that's portable.  However it also
> looked as though POSIX shm (shm_open, etc.) can be resized using
> ftruncate().  Whether this is portable to all the platforms we run on,
> or whether the behavior of ftruncate() in combination with shm_open()
> is in the standard, I'm not sure.

It's not portable.  That's exactly what we were looking into back when.

> I believe I went back and reread
> the old threads on this topic and it seems like the sticking point as
> far as POSIX shm goes it that it lacks a readable equivalent of
> shm_nattch.

Yeah, that was another little problem.  In principle though we only need
one SysV-style shmem segment to get the required interlock, and there
could be add-on shmem segments using POSIX or other APIs.  But that
doesn't get you out from under the portability issue or the memory space
management issue (it's unlikely you can enlarge a segment without
remapping it).

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Tom Lane]

Markus Wanner  writes:
> However, I'd like to get back to the original intent of the posted 
> patch. Which is about dynamically allocating memory *within a fixed size 
> pool*.

> That's something SRLU or shared_buffers do to some extent, but with lots 
> of limitations. And without the ability to move free memory between 
> sub-systems (i.e. between different SLRU buffers).

As far as SLRU is concerned, the already-agreed-to plan is to get rid of
the separate arenas for SLRU and merge those things into the main shared
buffers arena.  IIRC, the motivation for designing SLRU the way it is
was to ensure that SLRU uses couldn't be starved for memory due to high
demand for shared buffers.  But that was back when people frequently ran
PG with only a few meg for shared buffers; I think that worry is
obsolete.

So I don't see this patch as offering anything at all that we care about
so far as the core server is concerned.  Maybe there are extensions that
need it badly enough to justify such a feature in core, but SLRU is not
a good argument for it.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Markus Wanner]

Hi,

On 08/09/2010 08:45 PM, Robert Haas wrote:

Yeah, I think that's a real concern.  I think we need to distinguish
memory needs from memory wants.  Ideally, we'd like our entire
database to be cached in RAM.  But that may or may not be feasible, so
we page what we can into shared_buffers and page out as necessary to
make room for other things.  In contrast, the traditional malloc()
approach doesn't give you much flexibility: if it returns NULL, you
pretty much have to fail whatever operation you were trying to
perform.  For some things, that's OK.  For other things, it's not.


Agreed, it's going to be a difficult compromise and it possibly is very 
hard to find a good one automatically. However, I doubt our current 
approach with hard limits between subsystems is the best compromise.


Regards

Markus Wanner

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Robert Haas]

On Mon, Aug 9, 2010 at 3:00 PM, Bruce Momjian  wrote:
>> That would be one way to tackle the problem, but there are
>> difficulties.  If we just created new shared memory segments at need,
>> we might end up with a lot of shared memory segments.  I suspect that
>> would get complicated and present many management difficulties - which
>> is why I'm so far of the opinion that we should try to architect the
>> system to avoid the need for this functionality.  I don't think it's
>> going to be too easy to provide, short of (as Tom says) moving to the
>> MySQL model of many threads working in a single process.
>
> You could allocate shared memory in chunks and then pass that out to
> requestors, the same way sbrk() does it.

Sure.  But I don't think that gets you very far.  The management of
the chunks is really hard.  I go back to my previous example: you
can't store a pointer that might point to another chunk, because the
chunks won't get mapped into all the address spaces synchronously.
Even if you don't care about doing that (and I bet you do), mapping
and unmapping chunks is a heavyweight operation that requires every
backend to notice that it needs to do something (and, incidentally, if
any of them fail, you pretty much have to PANIC).  I just can't
imagine us building a reliable system this way.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Markus Wanner]

On 08/09/2010 09:00 PM, Bruce Momjian wrote:

You could allocate shared memory in chunks and then pass that out to
requestors, the same way sbrk() does it.


sbrk() is described [1] as a "low-level memory allocator", which "is 
typically only used by the high-level malloc memory allocator 
implemented in the C library".


Think of my patch as the high(er)-level variant ;-)   It's certainly 
doable using processes and shared memory. Yes. My patch shows one way of 
how to go a step into that direction.


Regards

Markus Wanner

[1]: http://www.cs.utah.edu/flux/moss/node39.html

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] host name support in pg_hba.conf

[Robert Haas]

On Mon, Aug 9, 2010 at 2:47 PM, Peter Eisentraut  wrote:
> Here is a patch for host name support in pg_hba.conf.  I have reviewed
> various past threads about this, and there appeared to have been a 50/50
> split of for and against reverse lookup.  I went with the reverse
> lookup, because
>
> 0) I like it.
>
> 1) It is more secure.
>
> 2) It allows extending it to wildcards in the future.
>
> 3) Apache (Allow from) does it that way.
>
> To clarify how it works:  The client's IP address (known from the
> kernel) is reverse looked up, which results in a host name.  That host
> name is compared with the line in pg_hba.conf.  If it matches, a forward
> lookup is performed on the host name to check if any of the resulting IP
> addresses match the client's IP address.  If yes, the line is considered
> to match and the authentication method is selected.

Seems reasonable.

> Anyway, assuming we will go with this, you will also notice that in the
> patch I changed the default pg_hba.conf to match against "localhost"
> instead of numeric addresses.  Initially thought of as a temporary
> change for testing this patch, I think this might actually have some
> permanent value because it saves you from having to change the IPv4 and
> IPv6 lines in tandem most of the times, which is a moderately common
> mistake.  We already rely on localhost being (forward) resolvable for
> the stats collector.

-1 from me, on this part.  I think we should be trying to eliminate
any dependencies we have on how localhost resolves, and we certainly
should not add more.  I have spent way too much time fighting with
problems caused by localhost being present/absent in /etc/hosts; and
more caused by whether it maps to 127.0.0.1 or some IP assigned to one
of the boxes' Ethernet cards.  What's really special is when you have
two parts of the system, one of which will only work with one version
of /etc/hosts and the other of which will only work with a different
version.  I humbly, but fervently, suggest that we try to avoid being
polemical about this.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Markus Wanner]

On 08/09/2010 08:50 PM, Bruce Momjian wrote:

You effectively have to add infrastructure to add/remove shared memory
segments to match memory requests.  It is another step, but it is the
same behavior.


That's of no use without a dynamic allocator, I think. Or else it is a 
vague description of a dynamic allocator.


I'm approaching the problem from another perspective: trying to 
implement a dynamic allocator on top of a fixed size memory pool, first. 
Once we have that, we may start to think about dynamically adding or 
removing underlying segments.


Regards

Markus Wanner

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Bruce Momjian]

Robert Haas wrote:
> On Mon, Aug 9, 2010 at 2:50 PM, Bruce Momjian  wrote:
> > Robert Haas wrote:
> >> On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian  wrote:
> >> >> > Let me be more concrete. ?Suppose you are using threads, and you want 
> >> >> > to
> >> >> > increase your shared memory from 20MB to 30MB. ?How do you do that? 
> >> >> > ?If
> >> >> > you want it contiguous, you have to use realloc, which might move the
> >> >> > pointer. ?If you allocate another 10MB chunk, you then have shared
> >> >> > memory fragments, which is the same as adding another shared memory
> >> >> > segment.
> >> >>
> >> >> You probably wouldn't do either of those things. ?You'd just allocate
> >> >> small chunks here and there for whatever you need them for.
> >> >
> >> > Well, then we do that with shared memory then --- my point is that it is
> >> > the same problem with threads or processes.
> >>
> >> Well, I think your point is wrong, then. ?:-)
> >>
> >> It's not the same at all. ?If you have a bunch of threads in one
> >> address space, "shared" memory is really just process-local. ?You can
> >> grow the total amount of allocated space just by calling malloc().
> >> With our architecture, you can't.
> >
> > You effectively have to add infrastructure to add/remove shared memory
> > segments to match memory requests. ?It is another step, but it is the
> > same behavior.
> 
> That would be one way to tackle the problem, but there are
> difficulties.  If we just created new shared memory segments at need,
> we might end up with a lot of shared memory segments.  I suspect that
> would get complicated and present many management difficulties - which
> is why I'm so far of the opinion that we should try to architect the
> system to avoid the need for this functionality.  I don't think it's
> going to be too easy to provide, short of (as Tom says) moving to the
> MySQL model of many threads working in a single process.

You could allocate shared memory in chunks and then pass that out to
requestors, the same way sbrk() does it.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Robert Haas]

On Mon, Aug 9, 2010 at 2:50 PM, Bruce Momjian  wrote:
> Robert Haas wrote:
>> On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian  wrote:
>> >> > Let me be more concrete. ?Suppose you are using threads, and you want to
>> >> > increase your shared memory from 20MB to 30MB. ?How do you do that? ?If
>> >> > you want it contiguous, you have to use realloc, which might move the
>> >> > pointer. ?If you allocate another 10MB chunk, you then have shared
>> >> > memory fragments, which is the same as adding another shared memory
>> >> > segment.
>> >>
>> >> You probably wouldn't do either of those things. ?You'd just allocate
>> >> small chunks here and there for whatever you need them for.
>> >
>> > Well, then we do that with shared memory then --- my point is that it is
>> > the same problem with threads or processes.
>>
>> Well, I think your point is wrong, then.  :-)
>>
>> It's not the same at all.  If you have a bunch of threads in one
>> address space, "shared" memory is really just process-local.  You can
>> grow the total amount of allocated space just by calling malloc().
>> With our architecture, you can't.
>
> You effectively have to add infrastructure to add/remove shared memory
> segments to match memory requests.  It is another step, but it is the
> same behavior.

That would be one way to tackle the problem, but there are
difficulties.  If we just created new shared memory segments at need,
we might end up with a lot of shared memory segments.  I suspect that
would get complicated and present many management difficulties - which
is why I'm so far of the opinion that we should try to architect the
system to avoid the need for this functionality.  I don't think it's
going to be too easy to provide, short of (as Tom says) moving to the
MySQL model of many threads working in a single process.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Markus Wanner]

On 08/09/2010 08:49 PM, Bruce Momjian wrote:

Markus Wanner wrote:

That's what my patch allows you to do, yes. Currently you are bound to
pre-allocate shared memory at startup. Or how would you allocate small
chunks from shared memory at the moment?


We don't --- we allocate it all at startup.


Exactly. And that's the difference to a thread-based approach. The 
downside of it is that you need to know in advance how much shared 
memory each of the subsystems is going to need. On the upside is the 
certainty, that you already have the memory allocated and cannot run out 
of it. You just have what you have.


(Note that you could do that as well with the thread-based approach, if 
you want. Most other programs I know don't choose that approach, though, 
but instead try to cope with OOM).


Regards

Markus

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] host name support in pg_hba.conf

[Thom Brown]

On 9 August 2010 19:47, Peter Eisentraut  wrote:
> Here is a patch for host name support in pg_hba.conf.  I have reviewed
> various past threads about this, and there appeared to have been a 50/50
> split of for and against reverse lookup.  I went with the reverse
> lookup, because
>
> 0) I like it.
>
> 1) It is more secure.
>
> 2) It allows extending it to wildcards in the future.
>
> 3) Apache (Allow from) does it that way.
>
> To clarify how it works:  The client's IP address (known from the
> kernel) is reverse looked up, which results in a host name.  That host
> name is compared with the line in pg_hba.conf.  If it matches, a forward
> lookup is performed on the host name to check if any of the resulting IP
> addresses match the client's IP address.  If yes, the line is considered
> to match and the authentication method is selected.
>
> Anyway, assuming we will go with this, you will also notice that in the
> patch I changed the default pg_hba.conf to match against "localhost"
> instead of numeric addresses.  Initially thought of as a temporary
> change for testing this patch, I think this might actually have some
> permanent value because it saves you from having to change the IPv4 and
> IPv6 lines in tandem most of the times, which is a moderately common
> mistake.  We already rely on localhost being (forward) resolvable for
> the stats collector.
>
> Something to think about: Maybe we need a quoting mechanism in case
> someone names their hosts "samenet".
>
>
> --

A couple things:

+   matches.  This field can contain either a host name, an IP
+   address range, one of the special key words mentioned below.
+  

s/, one/, or one/

+   If a host name is specified (anything that is not an IP address
+   or a special key word is processed as a potential host name), a
+   reverse DNS lookup is performed on the client's IP address,
+   then a forward DNS lookup on the resulting name to check if it
+   matches the original IP address (that is, at least one of the
+   potentially many IP addresses matches the original one), and
+   the name found in the reverse lookup is compared with the
+   specified host name.

That's one lng sentence!

-- 
Thom Brown
Registered Linux user: #516935

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] host name support in pg_hba.conf

[Kevin Grittner]

Peter Eisentraut  wrote:
 
> The client's IP address (known from the kernel)
 
Some machines have several IP addresses; how is that handled?
 
> is reverse looked up, which results in a host name.
 
Some IP addresses have several host names, including in reverse
lookup; how is that handled?
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Bruce Momjian]

Robert Haas wrote:
> On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian  wrote:
> >> > Let me be more concrete. ?Suppose you are using threads, and you want to
> >> > increase your shared memory from 20MB to 30MB. ?How do you do that? ?If
> >> > you want it contiguous, you have to use realloc, which might move the
> >> > pointer. ?If you allocate another 10MB chunk, you then have shared
> >> > memory fragments, which is the same as adding another shared memory
> >> > segment.
> >>
> >> You probably wouldn't do either of those things. ?You'd just allocate
> >> small chunks here and there for whatever you need them for.
> >
> > Well, then we do that with shared memory then --- my point is that it is
> > the same problem with threads or processes.
> 
> Well, I think your point is wrong, then.  :-)
> 
> It's not the same at all.  If you have a bunch of threads in one
> address space, "shared" memory is really just process-local.  You can
> grow the total amount of allocated space just by calling malloc().
> With our architecture, you can't.

You effectively have to add infrastructure to add/remove shared memory
segments to match memory requests.  It is another step, but it is the
same behavior.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Bruce Momjian]

Markus Wanner wrote:
> On 08/09/2010 08:33 PM, Bruce Momjian wrote:
> > Robert Haas wrote:
> >> You probably wouldn't do either of those things.  You'd just allocate
> >> small chunks here and there for whatever you need them for.
> >
> > Well, then we do that with shared memory then --- my point is that it is
> > the same problem with threads or processes.
> 
> That's what my patch allows you to do, yes. Currently you are bound to 
> pre-allocate shared memory at startup. Or how would you allocate small 
> chunks from shared memory at the moment?

We don't --- we allocate it all at startup.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [JDBC] Trouble with COPY IN

[Kris Jurka]

On Sat, 7 Aug 2010, Kris Jurka wrote:


On Fri, 6 Aug 2010, James William Pye wrote:


I think there's a snag in the patch:

postgres=# COPY data FROM '/Users/jwp/DATA.bcopy' WITH BINARY;
ERROR:  row field count is -1, expected 1
CONTEXT:  COPY data, line 4

Probably a quick/small fix away, I imagine.


Hmm, not quite sure why that is.  That seems to imply that it's not using V3 
protocol, but I thought binary copy could only be used with the V3 protocol. 
In any case, I think this new patch is more bulletproof.




Oh, duh.  It's a server side copy not going through the client at all. 
Here's a hopefully final patch.


Kris Jurka*** a/src/backend/commands/copy.c
--- b/src/backend/commands/copy.c
***
*** 2058,2069  CopyFrom(CopyState cstate)
int16   fld_count;
ListCell   *cur;
  
!   if (!CopyGetInt16(cstate, &fld_count) ||
!   fld_count == -1)
{
done = true;
break;
}
  
if (fld_count != attr_count)
ereport(ERROR,
--- 2058,2088 
int16   fld_count;
ListCell   *cur;
  
!   if (!CopyGetInt16(cstate, &fld_count))
{
done = true;
break;
}
+   
+   if (fld_count == -1)
+   {
+   /*
+* Received EOF marker.  Check to see if this 
is really
+* the EOF and complain if we find any more 
data.
+* In a V3 protocol copy, this also enforces 
that we wait
+* for the protocol end of copy (CopyDone/Fail).
+*/
+   int8 unused;
+   if (CopyGetData(cstate, &unused, 
sizeof(unused), sizeof(unused)))
+   {
+   ereport(ERROR,
+   
(errcode(ERRCODE_BAD_COPY_FILE_FORMAT),
+errmsg("received copy 
data after EOF marker")));
+   }
+ 
+   done = true;
+   break;
+   }
  
if (fld_count != attr_count)
ereport(ERROR,

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Robert Haas]

On Mon, Aug 9, 2010 at 2:33 PM, Bruce Momjian  wrote:
>> > Let me be more concrete. ?Suppose you are using threads, and you want to
>> > increase your shared memory from 20MB to 30MB. ?How do you do that? ?If
>> > you want it contiguous, you have to use realloc, which might move the
>> > pointer. ?If you allocate another 10MB chunk, you then have shared
>> > memory fragments, which is the same as adding another shared memory
>> > segment.
>>
>> You probably wouldn't do either of those things.  You'd just allocate
>> small chunks here and there for whatever you need them for.
>
> Well, then we do that with shared memory then --- my point is that it is
> the same problem with threads or processes.

Well, I think your point is wrong, then.  :-)

It's not the same at all.  If you have a bunch of threads in one
address space, "shared" memory is really just process-local.  You can
grow the total amount of allocated space just by calling malloc().
With our architecture, you can't.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] host name support in pg_hba.conf

[Peter Eisentraut]

Here is a patch for host name support in pg_hba.conf.  I have reviewed
various past threads about this, and there appeared to have been a 50/50
split of for and against reverse lookup.  I went with the reverse
lookup, because

0) I like it.

1) It is more secure.

2) It allows extending it to wildcards in the future.

3) Apache (Allow from) does it that way.

To clarify how it works:  The client's IP address (known from the
kernel) is reverse looked up, which results in a host name.  That host
name is compared with the line in pg_hba.conf.  If it matches, a forward
lookup is performed on the host name to check if any of the resulting IP
addresses match the client's IP address.  If yes, the line is considered
to match and the authentication method is selected.

Anyway, assuming we will go with this, you will also notice that in the
patch I changed the default pg_hba.conf to match against "localhost"
instead of numeric addresses.  Initially thought of as a temporary
change for testing this patch, I think this might actually have some
permanent value because it saves you from having to change the IPv4 and
IPv6 lines in tandem most of the times, which is a moderately common
mistake.  We already rely on localhost being (forward) resolvable for
the stats collector.

Something to think about: Maybe we need a quoting mechanism in case
someone names their hosts "samenet".
diff --git a/doc/src/sgml/client-auth.sgml b/doc/src/sgml/client-auth.sgml
index e8995f9..808c468 100644
--- a/doc/src/sgml/client-auth.sgml
+++ b/doc/src/sgml/client-auth.sgml
@@ -101,9 +101,9 @@
A record can have one of the seven formats
 
 local  database  user  auth-method  auth-options
-host   database  user  CIDR-address  auth-method  auth-options
-hostssldatabase  user  CIDR-address  auth-method  auth-options
-hostnossl  database  user  CIDR-address  auth-method  auth-options
+host   database  user  address  auth-method  auth-options
+hostssldatabase  user  address  auth-method  auth-options
+hostnossl  database  user  address  auth-method  auth-options
 host   database  user  IP-address  IP-mask  auth-method  auth-options
 hostssldatabase  user  IP-address  IP-mask  auth-method  auth-options
 hostnossl  database  user  IP-address  IP-mask  auth-method  auth-options
@@ -218,12 +218,17 @@ hostnossl  database  user
 
 
 
- CIDR-address
+ address
  
   
Specifies the client machine IP address range that this record
-   matches. This field contains an IP address in standard dotted decimal
-   notation and a CIDR mask length. (IP addresses can only be
+   matches.  This field can contain either a host name, an IP
+   address range, one of the special key words mentioned below.
+  
+
+  
+   An IP address is specified in standard dotted decimal
+   notation with a CIDR mask length. (IP addresses can only be
specified numerically, not as domain or host names.)  The mask
length indicates the number of high-order bits of the client
IP address that must match.  Bits to the right of this must
@@ -233,14 +238,7 @@ hostnossl  database  user
   
 
   
-   Instead of a CIDR-address, you can write
-   samehost to match any of the server's own IP
-   addresses, or samenet to match any address in any
-   subnet that the server is directly connected to.
-  
-
-  
-   Typical examples of a CIDR-address are
+   Typical examples of an IP address range specified this way are
172.20.143.89/32 for a single host, or
172.20.143.0/24 for a small network, or
10.6.0.0/16 for a larger one.
@@ -260,6 +258,24 @@ hostnossl  database  user
   
 
   
+   You can also write
+   samehost to match any of the server's own IP
+   addresses, or samenet to match any address in any
+   subnet that the server is directly connected to.
+  
+
+  
+   If a host name is specified (anything that is not an IP address
+   or a special key word is processed as a potential host name), a
+   reverse DNS lookup is performed on the client's IP address,
+   then a forward DNS lookup on the resulting name to check if it
+   matches the original IP address (that is, at least one of the
+   potentially many IP addresses matches the original one), and
+   the name found in the reverse lookup is compared with the
+   specified host name.
+  
+
+  
This field only applies to host,
hostssl, and hostnossl records.
   
@@ -511,12 +527,12 @@ hostnossl  database  user
 # any database user name using Unix-domain sockets (the default for local
 # connections).
 #
-# TYPE  DATABASEUSERCIDR-ADDRESSMETHOD
+# TYPE  DATABASEUSERADDRESS METHOD
 local   all all trust
 
 # The same using local loopback TCP/IP connections.
 #
-# TYPE 

Re: [HACKERS] dynamically allocating chunks from shared memory

[Robert Haas]

On Mon, Aug 9, 2010 at 2:28 PM, Markus Wanner  wrote:
> Another issue to be discussed would be the limits of sharing free memory
> between subsystems. Maybe we even reach the conclusion that we absolutely
> *want* fixed maximum sizes for every single subsystem so as to be able to
> guarantee a certain amount of multi-xact or SLRU entries at any point in
> time (otherwise one memory hungry subsystem could possibly eat it all up
> with another subsystem getting the OOM error when trying to allocate for its
> very first entry).

Yeah, I think that's a real concern.  I think we need to distinguish
memory needs from memory wants.  Ideally, we'd like our entire
database to be cached in RAM.  But that may or may not be feasible, so
we page what we can into shared_buffers and page out as necessary to
make room for other things.  In contrast, the traditional malloc()
approach doesn't give you much flexibility: if it returns NULL, you
pretty much have to fail whatever operation you were trying to
perform.  For some things, that's OK.  For other things, it's not.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Markus Wanner]

On 08/09/2010 08:33 PM, Bruce Momjian wrote:

Robert Haas wrote:

You probably wouldn't do either of those things.  You'd just allocate
small chunks here and there for whatever you need them for.


Well, then we do that with shared memory then --- my point is that it is
the same problem with threads or processes.


That's what my patch allows you to do, yes. Currently you are bound to 
pre-allocate shared memory at startup. Or how would you allocate small 
chunks from shared memory at the moment?


Regards

Markus

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Bruce Momjian]

Markus Wanner wrote:
> Hi,
> 
> On 08/09/2010 06:31 PM, Bruce Momjian wrote:
> > Let me be more concrete.  Suppose you are using threads, and you want to
> > increase your shared memory from 20MB to 30MB.  How do you do that?
> 
> There's absolutely no need to pre-allocate 20 MB in advance in a 
> threaded environment. You just allocate memory in small chunks. For a 
> threaded-model, that memory is shared by default, so the total amount of 
> shared memory can grow and shrink very easily. (And even makes usused 
> memory available to other processes, not just other threads).
> 
> > If
> > you want it contiguous, you have to use realloc, which might move the
> > pointer.  If you allocate another 10MB chunk, you then have shared
> > memory fragments, which is the same as adding another shared memory
> > segment.
> 
> Okay, I think I now understand the requirement of continuity you 
> mentioned earlier already. I agree that with the current approach, we 
> cannot simply use such a dynamic allocator to solve all of our problems.
> 
> Every subsystem would need to be converted to something that allocates 
> shared memory in smaller chunks for such a dynamic allocator to be of 
> any use. Robert already pointed out that this may be troublesome for 
> shared_buffers, which is by far the largest consumer of shared memory. I 
> didn't look into this, yet. And I'd like to hear more about the 
> feasibility of that approach for other subsystems.
> 
> Another issue to be discussed would be the limits of sharing free memory 
> between subsystems. Maybe we even reach the conclusion that we 
> absolutely *want* fixed maximum sizes for every single subsystem so as 
> to be able to guarantee a certain amount of multi-xact or SLRU entries 
> at any point in time (otherwise one memory hungry subsystem could 
> possibly eat it all up with another subsystem getting the OOM error when 
> trying to allocate for its very first entry).

Yep, you would have to use chunks in threads/malloc, and you have to do
the same thing with shared memory.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Bruce Momjian]

Robert Haas wrote:
> On Mon, Aug 9, 2010 at 12:31 PM, Bruce Momjian  wrote:
> > Bruce Momjian wrote:
> >> > With our process-based design, the default is private memory (i.e. not
> >> > shared). If you need shared memory, you must specify a certain amount in
> >> > advance. That chunk of shared memory then is reserved and can't ever be
> >> > used by another subsystem. Even if you barely ever need that much shared
> >> > memory for the subsystem in question.
> >>
> >> Once multiple threads are using the same local memory, you have the same
> >> issues of being unable to resize it because repalloc can change the
> >> pointer location.
> >
> > Let me be more concrete. ?Suppose you are using threads, and you want to
> > increase your shared memory from 20MB to 30MB. ?How do you do that? ?If
> > you want it contiguous, you have to use realloc, which might move the
> > pointer. ?If you allocate another 10MB chunk, you then have shared
> > memory fragments, which is the same as adding another shared memory
> > segment.
> 
> You probably wouldn't do either of those things.  You'd just allocate
> small chunks here and there for whatever you need them for.

Well, then we do that with shared memory then --- my point is that it is
the same problem with threads or processes.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Bruce Momjian]

Markus Wanner wrote:
> Hi,
> 
> On 08/09/2010 06:10 PM, Bruce Momjian wrote:
> > My point is that you can treat malloc the same as "add shared memory",
> > to some extent, with the same limiations.
> 
> Once one of the SLRU buffers is full, it cannot currently allocate from 
> another SLRU buffer's unused memory area. That memory there is plain 
> wasted at that moment. That's my point and the problem the allocator I 
> posted tries to solve.
> 
> I fail to see how malloc could help here. malloc() only allocates 
> process-local memory.

My point is that we have the same limitations with malloc()/threads, as
we have with shared memory.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] knngist - 0.8

[Robert Haas]

On Sun, Aug 8, 2010 at 4:28 PM, Alexander Korotkov  wrote:
> In gist consitent method support only filtering strategies. For such
> strategies consistent method returns true if subtree can contain matching
> node and false otherwise. Knngist introduce also order by strategies. For
> filtering strategies knngist consistent method returns 0 if  subtree can
> contain matching node and -1 otherwise. For order by strategies knngist
> consistent method returns minimal possible distance in subtree. I think we
> can use consistent method with order by strategies not only for ordering but
> also for filtering. If query contain assertion that distance is less than
> some value, than we can call consistent method with order by strategy and
> compare result with query value in order to determine whether scan subtree.

I am not an expert on this code, but after thinking this over, I
believe you are correct and that this is a good point.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Markus Wanner]

Hi,

On 08/09/2010 06:31 PM, Bruce Momjian wrote:

Let me be more concrete.  Suppose you are using threads, and you want to
increase your shared memory from 20MB to 30MB.  How do you do that?


There's absolutely no need to pre-allocate 20 MB in advance in a 
threaded environment. You just allocate memory in small chunks. For a 
threaded-model, that memory is shared by default, so the total amount of 
shared memory can grow and shrink very easily. (And even makes usused 
memory available to other processes, not just other threads).



If
you want it contiguous, you have to use realloc, which might move the
pointer.  If you allocate another 10MB chunk, you then have shared
memory fragments, which is the same as adding another shared memory
segment.


Okay, I think I now understand the requirement of continuity you 
mentioned earlier already. I agree that with the current approach, we 
cannot simply use such a dynamic allocator to solve all of our problems.


Every subsystem would need to be converted to something that allocates 
shared memory in smaller chunks for such a dynamic allocator to be of 
any use. Robert already pointed out that this may be troublesome for 
shared_buffers, which is by far the largest consumer of shared memory. I 
didn't look into this, yet. And I'd like to hear more about the 
feasibility of that approach for other subsystems.


Another issue to be discussed would be the limits of sharing free memory 
between subsystems. Maybe we even reach the conclusion that we 
absolutely *want* fixed maximum sizes for every single subsystem so as 
to be able to guarantee a certain amount of multi-xact or SLRU entries 
at any point in time (otherwise one memory hungry subsystem could 
possibly eat it all up with another subsystem getting the OOM error when 
trying to allocate for its very first entry).


Thanks for bringing this discussion to live again.

Regards

Markus Wanner

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

[Robert Haas]

On Mon, Aug 9, 2010 at 2:23 PM, Heikki Linnakangas
 wrote:
> On 09/08/10 21:21, Robert Haas wrote:
>>
>> On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggs  wrote:
>>>
>>> Any SQL statement that reads a block can do HOT pruning, if the block is
>>> otherwise unlocked.
>>
>> Where does heap_page_prune() get called from in the DELETE path?
>
> heapgetpage()

Ah, OK.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

[Heikki Linnakangas]

On 09/08/10 21:21, Robert Haas wrote:

On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggs  wrote:

Any SQL statement that reads a block can do HOT pruning, if the block is
otherwise unlocked.


Where does heap_page_prune() get called from in the DELETE path?


heapgetpage()

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

[Robert Haas]

On Mon, Aug 9, 2010 at 1:50 PM, Simon Riggs  wrote:
> On Sat, 2010-08-07 at 16:11 -0700, Gordon Shannon wrote:
>
>> So, I guess my real question here is, what happened to the "missing"
>> 100 items?  If it was HOT prune, can anyone summarize what that does?
>
> Itagaki already explained that the second DELETE would have removed the
> 100 dead rows you consider to be missing.
>
> Any SQL statement that reads a block can do HOT pruning, if the block is
> otherwise unlocked.

Where does heap_page_prune() get called from in the DELETE path?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Robert Haas]

On Mon, Aug 9, 2010 at 12:31 PM, Bruce Momjian  wrote:
> Bruce Momjian wrote:
>> > With our process-based design, the default is private memory (i.e. not
>> > shared). If you need shared memory, you must specify a certain amount in
>> > advance. That chunk of shared memory then is reserved and can't ever be
>> > used by another subsystem. Even if you barely ever need that much shared
>> > memory for the subsystem in question.
>>
>> Once multiple threads are using the same local memory, you have the same
>> issues of being unable to resize it because repalloc can change the
>> pointer location.
>
> Let me be more concrete.  Suppose you are using threads, and you want to
> increase your shared memory from 20MB to 30MB.  How do you do that?  If
> you want it contiguous, you have to use realloc, which might move the
> pointer.  If you allocate another 10MB chunk, you then have shared
> memory fragments, which is the same as adding another shared memory
> segment.

You probably wouldn't do either of those things.  You'd just allocate
small chunks here and there for whatever you need them for.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Robert Haas]

On Mon, Aug 9, 2010 at 12:03 PM, Bruce Momjian  wrote:
> Robert Haas wrote:
>> On Mon, Aug 9, 2010 at 11:02 AM, Bruce Momjian  wrote:
>> > I am not sure threads would greatly help us. ?The major problem is that
>> > all of our our structures are currently contiguous in memory for quick
>> > access. ?I don't see how threading would help with that. ?We could use
>> > realloc(), but we can do the same in shared memory if we had a chunk
>> > infrastructure, though concurrent access to that memory would hurt us in
>> > either threads or shared memory.
>> >
>> > Fundamentally, recreating the libc memory allocation routines is not
>> > that hard. ?(Everyone has to detach from the shared memory segment, but
>> > they have to stop using it too, so it doesn't seem that hard.)
>>
>> I actually don't think that's true.  The advantage (and disadvantage)
>> of using threads is that everything runs in one address space.  So you
>> just allocate more memory and everyone immediately sees it.  In a
>> process environment, that's not the case: to expand or shrink the size
>> of the shared memory arena, everyone needs to explicitly change their
>> own mapping.
>
> You can't expand the size of malloc'ed memory --- you have to call
> realloc(), and then you effectively get a new pointer.  Shared memory
> has a similar limitation.  If you allocate shared memory in chunks so
> you don't need to change the location, you are effectively doing another
> malloc(), like you would in a threaded process.

The point isn't what happens when you resize individual chunks; it's
what happens when you need to expand the arena.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Robert Haas]

On Mon, Aug 9, 2010 at 11:41 AM, Tom Lane  wrote:
> Robert Haas  writes:
>> So imagine that thread-or-process A allocates allocates a new chunk of
>> memory and then writes a pointer to the new chunk in a previously
>> allocated section of memory.  Thread-or-process B then follows the
>> pointer.  In a threaded model, this is guaranteed to be safe.  In a
>> process model, it's not: A might have enlarged the shared memory
>> mapping while B has not yet done so.  So I think in our model any sort
>> of change to the shared memory segment is going to require extremely
>> careful gymnastics, and be pretty expensive.
>
> ... and on some platforms, it'll be flat out impossible.  We looked at
> this years ago and concluded that changing the size of the shmem segment
> after postmaster start was impractical from a portability standpoint.
> I have not seen anything to change that conclusion.

I haven't done extensive research into this, but I did take a look at
it briefly.  It looked to me like the style of shared memory we're
using now (I guess it's System V) has no way to resize a shared memory
segment at all, and certainly no way that's portable.  However it also
looked as though POSIX shm (shm_open, etc.) can be resized using
ftruncate().  Whether this is portable to all the platforms we run on,
or whether the behavior of ftruncate() in combination with shm_open()
is in the standard, I'm not sure.  I believe I went back and reread
the old threads on this topic and it seems like the sticking point as
far as POSIX shm goes it that it lacks a readable equivalent of
shm_nattch.  I think it was proposed to use a small syv shm and then
do the main shared memory arena with shm_open, but at that point you
start to wonder you're messing around with at all.

But I can't help but be intrigued by it, even so.  Suppose, for
example, that we kept things that were really fixed-size in shared
memory but moved, say, shared_buffers to a POSIX shm.  Would that
allow you to then make shared_buffers PGC_SIGHUP?  The obvious answer
is "no", because there are a whole bunch of knock-on issues.  Changing
the size of shared_buffers also means changing the number of LWLocks,
changing the number of buffer descriptors, etc.  So maybe it can't be
done.  But I can't stop wondering if there's a way to make it work...

>> I don't care to take a position in the religious war over threads vs.
>> processes, but I do think threads simplify the handling of this
>> particular case.
>
> You meant "I don't think", right?  I agree.  The only way threads would
> simplify this is if we went over to a mysql-style model where there was
> only one process, period, and all backends were threads inside that.
> No shared memory as such, at all.

I think we're saying the same thing in different ways; I agree with
everything in that paragraph that follows the question mark.  By "this
particular case", I meant "shared memory allocation"; it would amount
to just calling malloc() [or palloc()].  But yeah, clearly that only
works in a single-process model.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Markus Wanner]

Hi,

On 08/09/2010 06:10 PM, Bruce Momjian wrote:

My point is that you can treat malloc the same as "add shared memory",
to some extent, with the same limiations.


Once one of the SLRU buffers is full, it cannot currently allocate from 
another SLRU buffer's unused memory area. That memory there is plain 
wasted at that moment. That's my point and the problem the allocator I 
posted tries to solve.


I fail to see how malloc could help here. malloc() only allocates 
process-local memory.


Regards

Markus Wanner

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Markus Wanner]

Hi,

On 08/09/2010 05:41 PM, Tom Lane wrote:

... and on some platforms, it'll be flat out impossible.  We looked at
this years ago and concluded that changing the size of the shmem segment
after postmaster start was impractical from a portability standpoint.
I have not seen anything to change that conclusion.


I haven't tried, but I tend to believe that's true.

However, I'd like to get back to the original intent of the posted 
patch. Which is about dynamically allocating memory *within a fixed size 
pool*.


That's something SRLU or shared_buffers do to some extent, but with lots 
of limitations. And without the ability to move free memory between 
sub-systems (i.e. between different SLRU buffers).



You meant "I don't think", right?  I agree.  The only way threads would
simplify this is if we went over to a mysql-style model where there was
only one process, period, and all backends were threads inside that.
No shared memory as such, at all.


That's how the threaded model normally is used, yes. And with that 
model, allocation of shared memory is very easy. It has none of the 
pre-allocation requirements we are currently facing.


Regards

Markus Wanner

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] is syntax columname(tablename) necessary still?

[Pavel Stehule]

2010/8/9 Robert Haas :
> On Mon, Aug 9, 2010 at 11:24 AM, Pavel Stehule  
> wrote:
>>> They name to be type_func_keywords, perhaps, but not fully reserved.
>>> And they'd still need that treatment anyway.  Even if cube(whatever)
>>> can't mean "extract a column called cube from table whatever", it can
>>> still mean "call a function called cube on a column called whatever".
>>
>> look to gram.y, please.
>>
>> we can use a
>>
>> GROUP BY CUBE(expr, ..)
>> GROUP BY func_name(expr, ..)
>>
>> so these rules are in conflict, because func_name can have a
>> type_func_keywords symbols. So we have to significantly rewrite a
>> rules about func call or CUBE and ROLLUP have to be a reserved words.
>> There isn't any other possibility.
>
> I understand that you have to make CUBE and ROLLUP reserved words.
> But you would still have to do that even if we changed $SUBJECT.

I am not sure if I understand well.

yes - CUBE and ROLLUP have to be reserved keywords - and I don't
calculate with removing a "obsolete" syntax now.

Regards

Pavel



>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise Postgres Company
>

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] is syntax columname(tablename) necessary still?

[Robert Haas]

On Mon, Aug 9, 2010 at 11:24 AM, Pavel Stehule  wrote:
>> They name to be type_func_keywords, perhaps, but not fully reserved.
>> And they'd still need that treatment anyway.  Even if cube(whatever)
>> can't mean "extract a column called cube from table whatever", it can
>> still mean "call a function called cube on a column called whatever".
>
> look to gram.y, please.
>
> we can use a
>
> GROUP BY CUBE(expr, ..)
> GROUP BY func_name(expr, ..)
>
> so these rules are in conflict, because func_name can have a
> type_func_keywords symbols. So we have to significantly rewrite a
> rules about func call or CUBE and ROLLUP have to be a reserved words.
> There isn't any other possibility.

I understand that you have to make CUBE and ROLLUP reserved words.
But you would still have to do that even if we changed $SUBJECT.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Surprising dead_tuple_count from pgstattuple

[Simon Riggs]

On Sat, 2010-08-07 at 16:11 -0700, Gordon Shannon wrote:

> So, I guess my real question here is, what happened to the "missing"
> 100 items?  If it was HOT prune, can anyone summarize what that does?

Itagaki already explained that the second DELETE would have removed the
100 dead rows you consider to be missing.

Any SQL statement that reads a block can do HOT pruning, if the block is
otherwise unlocked.

-- 
 Simon Riggs   www.2ndQuadrant.com
 PostgreSQL Development, 24x7 Support, Training and Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Simon Riggs]

On Mon, 2010-08-09 at 11:41 -0400, Tom Lane wrote:
> Robert Haas  writes:
> > So imagine that thread-or-process A allocates allocates a new chunk of
> > memory and then writes a pointer to the new chunk in a previously
> > allocated section of memory.  Thread-or-process B then follows the
> > pointer.  In a threaded model, this is guaranteed to be safe.  In a
> > process model, it's not: A might have enlarged the shared memory
> > mapping while B has not yet done so.  So I think in our model any sort
> > of change to the shared memory segment is going to require extremely
> > careful gymnastics, and be pretty expensive.
> 
> ... and on some platforms, it'll be flat out impossible.  We looked at
> this years ago and concluded that changing the size of the shmem segment
> after postmaster start was impractical from a portability standpoint.
> I have not seen anything to change that conclusion.

As caches get larger, downtime gets longer. Downtime of more than a few
minutes per year is enough to blow claims of high availability. 

At some point, this project will need to face this particular hurdle. We
may need to balance utility for the majority against portability for the
minority.

We should be laying out an architectural roadmap, not just saying no. We
can make multi-year plans if we wish to.

-- 
 Simon Riggs   www.2ndQuadrant.com
 PostgreSQL Development, 24x7 Support, Training and Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Universal B-tree

[Daniel Oliveira]

For research purpose, I think that expression index is a good idea. I just
want to do a proof of concept.

The other issue is that my algorithm break a z-order interval into several
intervals that represents the query box. How should I create it without
creating any overhead?

Best regards,

daniel

2010/8/9 Greg Stark 

> On Mon, Aug 9, 2010 at 5:31 PM, Daniel Oliveira
>  wrote:
> > I don't need to change B-tree estructure. I just need integrate my encode
> > function that transforms multiple keys into one key by bit-interleaving
> and
> > to acess elements given several intervals (range search).
>
> You could build a "expression" index on the function which will build
> a regular btree. Then any range clause on a similar expression in the
> query would be able to use the index. Postgres is capable of detecting
> multiple range clauses that can use one or more indexes and combine
> the results.  It might not be perfect for your use case, being a
> general purpose tool.
>
> I'm not sure how feasible it would be to implement a special purpose
> case that matches your needs. Perhaps a special index type and index
> access method similar to gist "tsquery" data type. But that's a lot
> more work and a lot more C coding.
>
> --
> greg
>

Re: [HACKERS] knngist - 0.8

[Alexander Korotkov]

In gist consitent method support only filtering strategies. For such
strategies consistent method returns true if subtree can contain matching
node and false otherwise. Knngist introduce also order by strategies. For
filtering strategies knngist consistent method returns 0 if  subtree can
contain matching node and -1 otherwise. For order by strategies knngist
consistent method returns minimal possible distance in subtree. I think we
can use consistent method with order by strategies not only for ordering but
also for filtering. If query contain assertion that distance is less than
some value, than we can call consistent method with order by strategy and
compare result with query value in order to determine whether scan subtree.
Such approach can give benefit when we need to filter by similarity. For
example, in pg_trgm "%" is used for similarity filtering, but similarity
threshold is global for session. That's why we can't create complex queries
which contain similarity filtering with different threshold.


With best regards,
Alexander Korotkov.



On Mon, Aug 2, 2010 at 8:14 PM, Robert Haas  wrote:

> 2010/7/29 Alexander Korotkov :
> > But, in pg_trgm it makes it possible to combine different similarity
> levels
> > in one query. For example:
> > select * from test_trgm order by t <-> 'asdf' < 0.5 or t <-> 'qwer' <
> 0.4;
> > Is there any chance to handle this syntax also?
>
> Maybe I'm missing something, but I don't think that ORDER BY clause
> makes much sense.  OR is going to reduce a true or false value - and
> it's usually not that interesting to order by a column that can only
> take one of two values.
>
> Am I confused?
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise Postgres Company
>

Re: [HACKERS] patch: to_string, to_array functions

[Brendan Jurd]

Apparently, the message I sent (quoted below) didn't make it to
-hackers.  I know that Pavel received the message, as he replied to
it.  I'm calling shenanigans on the mailing list server, but in the
meantime, here are those diffs again.

On 31 July 2010 07:37, Brendan Jurd  wrote:
> Hi Pavel,
>
> I've reviewed your latest patch (which I refer to as v3 to keep
> continuity with previous versions under the "to_array" naming system).
>
> You didn't quite complete the rename of the functions; in-code
> comments and regression tests still referred to the old names.  I
> cleanup that up for you and also reworded some of the in-code comments
> for clarity.
>
> Otherwise the patch looks good and the functions now work exactly as I
> would expect.
>
> I also went ahead and added some more documentation to explain how
> (im|ex)plode differ from their foo_to_bar counterparts, and what kind
> of behaviour you'll get by specifying the arguments as NULL.
>
> I have attached v4 of the patch against HEAD, and also an incremental
> patch showing just my changes against v3.
>
> I'll mark this as ready for committer.
>
> Cheers,
> BJ
>


implode_v4.diff.gz
Description: GNU Zip compressed data


implode_v3-to-4.diff.gz
Description: GNU Zip compressed data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Universal B-tree

[Greg Stark]

On Mon, Aug 9, 2010 at 5:31 PM, Daniel Oliveira
 wrote:
> I don't need to change B-tree estructure. I just need integrate my encode
> function that transforms multiple keys into one key by bit-interleaving and
> to acess elements given several intervals (range search).

You could build a "expression" index on the function which will build
a regular btree. Then any range clause on a similar expression in the
query would be able to use the index. Postgres is capable of detecting
multiple range clauses that can use one or more indexes and combine
the results.  It might not be perfect for your use case, being a
general purpose tool.

I'm not sure how feasible it would be to implement a special purpose
case that matches your needs. Perhaps a special index type and index
access method similar to gist "tsquery" data type. But that's a lot
more work and a lot more C coding.

-- 
greg

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: to_string, to_array functions

[Bruce Momjian]

Pavel Stehule wrote:
> 2010/7/21 Itagaki Takahiro :
> > 2010/7/20 Pavel Stehule :
> >> here is a new version - new these functions are not a strict and
> >> function to_string is marked as stable.
> >
> > We have array_to_string(anyarray, text) and string_to_array(text, text),
> > and you'll introduce to_string(anyarray, text, text) and
> > to_array(text, text, text).
> 
> I have to repeat it, the behave of this functions are little bit
> different. string_to_array and array_to_string are buggy.
> 
> * it isn't support a NULL
> * it doesn't differentiate a empty array and NULL
> * we cannot to change default behave of existing functions
> * array_to_string is badly marked as IMMUTABLE

This email thread linked to from our TODO list explains that arrays
combined with NULLs have many inconsistenciess:

http://archives.postgresql.org/pgsql-bugs/2008-11/msg9.php

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ERROR: argument to pg_get_expr() must come from system catalogs

[Bruce Momjian]

Tom Lane wrote:
> Bruce Momjian  writes:
> > Tom Lane wrote:
> >> Dave Page  writes:
> >>> Thanks. Bruce seemed to think it affected 8.4.4 as well - would that
> >>> be the case, or is it something else?
> >> 
> >> He's mistaken.  The bug is in all the branches, but there have been no
> >> releases with it except 9.0beta3.  I will work on back-patching the
> >> older branches this morning.
> 
> > I was using 8.4.X CVS with pgAdmin 8.4 and was seeing the error.  Is the
> > error in 8.4.X CVS?
> 
> It was in CVS, but it never got into a release.

OK, that explains why I see it and no one else does --- never mind.  ;-)

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ERROR: argument to pg_get_expr() must come from system catalogs

[Tom Lane]

Bruce Momjian  writes:
> Tom Lane wrote:
>> Dave Page  writes:
>>> Thanks. Bruce seemed to think it affected 8.4.4 as well - would that
>>> be the case, or is it something else?
>> 
>> He's mistaken.  The bug is in all the branches, but there have been no
>> releases with it except 9.0beta3.  I will work on back-patching the
>> older branches this morning.

> I was using 8.4.X CVS with pgAdmin 8.4 and was seeing the error.  Is the
> error in 8.4.X CVS?

It was in CVS, but it never got into a release.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] dynamically allocating chunks from shared memory

[Bruce Momjian]

Bruce Momjian wrote:
> > With our process-based design, the default is private memory (i.e. not 
> > shared). If you need shared memory, you must specify a certain amount in 
> > advance. That chunk of shared memory then is reserved and can't ever be 
> > used by another subsystem. Even if you barely ever need that much shared 
> > memory for the subsystem in question.
> 
> Once multiple threads are using the same local memory, you have the same
> issues of being unable to resize it because repalloc can change the
> pointer location.

Let me be more concrete.  Suppose you are using threads, and you want to
increase your shared memory from 20MB to 30MB.  How do you do that?  If
you want it contiguous, you have to use realloc, which might move the
pointer.  If you allocate another 10MB chunk, you then have shared
memory fragments, which is the same as adding another shared memory
segment.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers