[HACKERS] Add a new backend process

[Amir Abdollahi]

Hello,

I want to add a new backend process to postgres, to 
include my own auditing modules.
How can i do that, also how can i 
signal it after!
Sorry if this is very general question!

I 
didn't find any source to learn these things in postgres.

thanks 
in advance





  

Re: [HACKERS] Performance Enhancement/Fix for Array Utility Functions

[Mike Lewis]

>
>
> The existence and naming of ARR_MAX_HEADER_SIZE is somewhat dubious,
> as it is:
>

Thanks you for the feedback.  I cleaned up the patch.


> * Used in exactly one place (not necessarily a reason why it should
> not be reified into a stand-alone definition, though, but
> something to consider)
>

Moved it to one definition


> * The array header refers to the NULL bitmap as well, but the
> interpretation used by the patch does not.
>

I renamed the macros to have NONULL in the name (hopefully it doesn't make
them too long).

I also added a comment.  Not quite sure if it's the appropriate format, but
I didn't feel it warranted 3 lines.

Thanks,
Mike Lewis


detoast-headers-for-array-functions-003.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [v9.1] Add security hook on initialization of instance

[KaiGai Kohei]

(2010/06/15 21:37), Stephen Frost wrote:
> KaiGai,
> 
> * KaiGai Kohei (kai...@ak.jp.nec.com) wrote:
>> In the attached patch, the security hook was moved to ClientAuthentication()
>> from InitPostgres(), for more clarification of the purpose.
>> What I want to do is to assign additional properties to identify the client
>> (such as security label) for each authenticated session.
>>
>> Its purpose is similar to "session" module of PAM in operating system.
>> It allows to assign additional session properties more than user-id.
> 
> That's all fine- but let's work within the confines of the *existing*
> hook that's been discussed to get something working first before we go
> adding hooks in other places.  I think it's important that we put
> together at least a proof of concept that an SELinux module or other
> external auth module can sensible use the DML hook.
> 
Yes, I'd like to introduce the reason and purpose of the hook.

At first, please assume any external security modules should be loaded
as shared preload libraries, because 'local_preload_libraries' setting
can be overwritten using connection string.
So, _PG_init() shall be invoked at the starting-up of the postmaster
daemon process at once, not per connection.

On the other hand, a security feature have to identify the client and
assign an appropriate set of privileges on the session prior to it being
available for users.
E,g. The default PG security assigns a certain database user-id on
the current session, then it will be used for access control decision.
In a similar way, an external security also wants a chance to identify,
authenticate and authorize the client. (SELinux uses getpeercon() to
obtain security label of the peer process, and applies it as privilege
of the current session.)

However, here is no hooks available for the purpose. In this case,
_PG_init() is not called for each connections, because the module is
a shared preload library, so we have to call getpeercon() in another
point.

One idea is, as Robert suggested, that we can invoke getpeercon() at
the first call of SELinux module and store it on the local variable.
It will work well as long as getpeercon() does not cause an error.

Robert pointed out we can always raise ERROR or FATAL using ereport(),
if it is troubled.
However, it seems to me a trouble of getpeercon() is fundamentally
an error within the step of client authentication.
In the case of database user-id, it immediately raises an error, and
close the connection. I think we should follow the manner of similar
features, so I proposed the new security hook at the authentication.

Thanks,

> After that, we can discuss what other hooks are needed.  KaiGai, please,
> before sending in patches of any kind, propose at a high-level what the
> problem is and what the security module needs in general terms.  If you
> have a recommendation, that's fine, but let's talk about it before
> implementing anything.
> 
>   Thanks,
> 
>   Stephen


-- 
KaiGai Kohei 

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] ALTER TABLE...ALTER COLUMN vs inheritance

[Selena Deckelmann]

On Wed, May 26, 2010 at 2:37 PM, Bernd Helmle  wrote:

> Lost it a little from my radar, but here's is a first shot on this issue
> now. The patch creates a new CONSTRAINT_NOTNULL contype and assigns all
> required information for the NOT NULL constraint to it. Currently the
> constraint records the attribute number it belongs to and manages the
> inheritance properties. Passes regression tests with some adjustments to
> pg_constraint output.

Confirmed that this tests fine against commit id
0dca7d2f70872a242d4430c4c3aa01ba8dbd4a8c

I also wrote a little test script and verified that it throws an error
when I try to remove a constraint from the parent table.

Should an explicit test be added for this?

There are some spelling and grammar errors in comments that I can help
fix if you want the help.

-selena

-- 
http://chesnok.com/daily - me

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] debug log in pg_archivecleanup

[Takahiro Itagaki]

Fujii Masao  wrote:

> This is because pg_archivecleanup puts the line break "\n" in the head of
> debug message. Why should we do so?
> 
> ---
>  if (debug)
> fprintf(stderr, "\n%s:  removing \"%s\"", progname, WALFilePath);
> ---

We also need "\n" at line 308.
  L.125: fprintf(stderr, "\n%s:  removing \"%s\"", progname, WALFilePath);
  L.308: fprintf(stderr, "%s:  keep WAL file %s and later", progname, 
exclusiveCleanupFileName);

Note that we don't need a line break at Line 130
because strerror() fills the last %s.
  L.130: fprintf(stderr, "\n%s: ERROR failed to remove \"%s\": %s",

Regards,
---
Takahiro Itagaki
NTT Open Source Software Center



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] to enable O_DIRECT within postgresql

[Tom Lane]

Daniel Ng  writes:
>I am trying to enable the direct IO for the disk-resident
> hash partitions of hashjoin in postgresql.

Why would you think that's a good idea?

> Can anyone advise what's the reason and how to fix this?

Per the open(2) man page:

   The O_DIRECT flag may impose alignment restrictions on the  length  and
   address  of  userspace  buffers  and the file offset of I/Os.  In Linux
   alignment restrictions vary by file system and kernel version and might
   be absent entirely.  However there is currently no file system-indepen-
   dent interface for an application to discover these restrictions for  a
   given  file or file system.

It's unlikely that the code you're hacking makes any attempt to align
the buffers it's using to read/write files.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] hstore ==> and deprecate =>

[David E. Wheeler]

On Jun 15, 2010, at 6:58 PM, Robert Haas wrote:

> Well, the idea is it's like logical-and - give me only those keys that
> appear on both sides...

Yeah, but => doesn't return the keys, -> does. => returns an hstore.

> If there is a critical mass of votes for one of these options, I'm
> fine with whatever.

Put me down for +>.

Best,

David



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] to enable O_DIRECT within postgresql

[Daniel Ng]

Dear all,
   I am trying to enable the direct IO for the disk-resident
hash partitions of hashjoin in postgresql. The basic postgres
environment settings are:
   centos 5.5
   kernel 2.6.18
   ext3 fs
   PostgreSQL 8.4.3

   Previously I added the O_DIRECT flag to the "fileFlags"
parameter of open() within BasicOpenFile() (line 505 in
src/backend/storage/file/fd.c), but strangely I cannot even
start the server, with error:

PANIC:  could not read from control file: Invalid argument
Aborted

   So far what I did is to add the O_DIRECT flag to the
"fileFlags" parameter of PathNameOpenFile() (line 992 & 1007 in
src/backend/storage/file/fd.c), which calls the BasicOpenFIle()
and passes the "fileFlags". This time, I can start the sever,
but when I submit a hashjoin query from the client, it happens

 ERROR:  could not write to hash-join temporary file: Invalid argument

Can anyone advise what's the reason and how to fix this?
Or what's the correct way to enable the direct disk IO within
postgres? I appreciate the suggestions and thanks very much!

Regards
Daniel

Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers

[Robert Haas]

On Tue, Jun 15, 2010 at 8:09 PM, Josh Berkus  wrote:
>
>> I have yet to convince myself of how likely this is to occur.  I tried
>> to reproduce this issue by crashing the database, but I think in 9.0
>> you need an actual operating system crash to cause this problem, and I
>> haven't yet set up an environment in which I can repeatedly crash the
>> OS.  I believe, though, that in 9.1, we're going to want to stream
>> from WAL buffers as proposed in the patch that started out this
>> thread, and then I think this issue can be triggered with just a
>> database crash.
>
> Yes, but it still requires:
>
> a) the master must crash with at least one transaction transmitted to
> the slave an not yet fsync'd

Bt.  Stop right there.  It only requires the master to crash with
at least one *WAL record* written but not transmitted, not one
transaction.  And most WAL record types are not fsync'd immediately.
So in theory I think that, for example, an OS crash in the middle of a
big bulk insert operation should be sufficient to trigger this.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] hstore ==> and deprecate =>

[Robert Haas]

On Tue, Jun 15, 2010 at 9:04 PM, David E. Wheeler  wrote:
> On Jun 15, 2010, at 3:13 PM, Robert Haas wrote:
>
>> On Mon, Jun 14, 2010 at 4:35 PM, Andrew Gierth
>>  wrote:
>>> I'm happy with deprecating the first two => in favour of hstore() if
>>> that is in line with general opinion. The hstore => text[] slice could
>>> be replaced by another operator name; the existing name comes from the
>>> analogy that (hstore -> text[]) returns the list of values, whereas
>>> (hstore => text[]) returns both the keys and values.
>>
>> So, I kind of like Florian Pflug's suggestion upthread of replacing
>> hstore => text by hstore & text[].  I think that's about as mnemonic
>> as we're likely to get, and it gels nicely with the hstore ?& text[]
>> operator, which tests whether all of the named keys are present in the
>> hstore.
>>
>> Does anyone want to bikeshed further before I go do that?
>
> Yeah. It actually doesn't make much sense to me. ?& is all about the keys and 
> their presence, not the values. -> is a much better parallel, it being that 
> it returns the keys in the rhs array. So I think something closer to it would 
> be better.

Well, the idea is it's like logical-and - give me only those keys that
appear on both sides...

> Some suggestions:
>
>  ~>
>  <-
>  #>
>  +>
>
> Ooh, I like +>, as being: give me more than -> does.

If there is a critical mass of votes for one of these options, I'm
fine with whatever.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] hstore ==> and deprecate =>

[David E. Wheeler]

On Jun 15, 2010, at 3:13 PM, Robert Haas wrote:

> On Mon, Jun 14, 2010 at 4:35 PM, Andrew Gierth
>  wrote:
>> I'm happy with deprecating the first two => in favour of hstore() if
>> that is in line with general opinion. The hstore => text[] slice could
>> be replaced by another operator name; the existing name comes from the
>> analogy that (hstore -> text[]) returns the list of values, whereas
>> (hstore => text[]) returns both the keys and values.
> 
> So, I kind of like Florian Pflug's suggestion upthread of replacing
> hstore => text by hstore & text[].  I think that's about as mnemonic
> as we're likely to get, and it gels nicely with the hstore ?& text[]
> operator, which tests whether all of the named keys are present in the
> hstore.
> 
> Does anyone want to bikeshed further before I go do that?

Yeah. It actually doesn't make much sense to me. ?& is all about the keys and 
their presence, not the values. -> is a much better parallel, it being that it 
returns the keys in the rhs array. So I think something closer to it would be 
better. Some suggestions:

  ~>
  <-
  #>
  +>

Ooh, I like +>, as being: give me more than -> does.

Best,

David


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers

[Josh Berkus]

On 6/15/10 5:09 PM, Josh Berkus wrote:
>> > In 9.0, I think we can fix this problem by (1) only streaming WAL that
>> > has been fsync'd and 
> 
> I don't think this is the best solution; it would be a noticeable
> performance penalty on replication. 

Actually, there's an even bigger reason not to mandate waiting for
fsync: what if the user turns fsync off?

One can certainly imagine users choosing to rely on their replication
slaves for crash recovery instead of fsync.

-- 
  -- Josh Berkus
 PostgreSQL Experts Inc.
 http://www.pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers

[Josh Berkus]

> I have yet to convince myself of how likely this is to occur.  I tried
> to reproduce this issue by crashing the database, but I think in 9.0
> you need an actual operating system crash to cause this problem, and I
> haven't yet set up an environment in which I can repeatedly crash the
> OS.  I believe, though, that in 9.1, we're going to want to stream
> from WAL buffers as proposed in the patch that started out this
> thread, and then I think this issue can be triggered with just a
> database crash.

Yes, but it still requires:

a) the master must crash with at least one transaction transmitted to
the slave an not yet fsync'd
b) the slave must not crash as well
c) the master must come back up without the slave ever having been
promoted to master

Note that (a) is fairly improbable to begin with due to both our
batching transactions into bundles for transmission, and network latency
vs. disk latency.

So, is it possible?  Yes.  Will it happen anywhere but the
highest-txn-rate sites one in 10,000 times?  No.

This means that we should look for a solution which does not penalize
the common case in order to close a very improbable hole, if such a
solution exists.

> In 9.0, I think we can fix this problem by (1) only streaming WAL that
> has been fsync'd and 

I don't think this is the best solution; it would be a noticeable
performance penalty on replication.  It also would potentially result in
data loss for the user; if the user fails over to the slave in the
corner case, they can "rescue" the in-flight transaction.  At the least,
this would need to become Yet Another Configuration Option.

>(2) PANIC-ing if the problem occurs anyway.  

The question is, is detecting out-of-order WAL records *sufficient* to
detect a failure?  I'm thinking there are possible sequences where there
would be no out-of-sequence, but the slave would still have a
transaction the master doesn't, which the user wouldn't know until a
page update corrupts their data.

> But
> in 9.1, with sync rep and the performance demands that entails, I
> think that we're going to need to rethink it.

All the more reason to avoid dealing with it now, if we can.

-- 
  -- Josh Berkus
 PostgreSQL Experts Inc.
 http://www.pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [RRR] Reviewfest 2010-06 Plans and Call for Reviewers

[Josh Berkus]

> A few folks from PDXPUG are meeting up this evening to go over some
> patches. We'll be on IRC at #pdxpug from 6pm-8pm PDT if you want to
> join in virtually.

We'll try to organize an SFPUG review for next month, shortly before the
commitfest starts.


-- 
  -- Josh Berkus
 PostgreSQL Experts Inc.
 http://www.pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] 9.0 beta2 pg_upgrade command line parsing

[Bruce Momjian]

Steve Singer wrote:
> 
> When I try running pg_upgrade from beta2 on an AIX server I'm unable to 
> get beyond the command line parsing (no matter what command line 
> arguments I pass in it always complains about the usage).
> 
> 
> When I compile pg_upgrade it complains about:
> 
> option.c: In function 'parseCommandLine':
> option.c:103: warning: comparison is always true due to limited range of 
> data type
> 
> getopt_long  (both our version in ports and the gnu version) is defined 
> to return an int.
> 
> The attach patch allows me to get beyond the options loop under AIX.

Thanks you.  Patch applied.

---


> 
> 
> 
> 
> 
> -- 
> Steve Singer
> Afilias Canada
> Data Services Developer
> 416-673-1142

[ text/x-patch is unsupported, treating like TEXT/PLAIN ]

> diff --git a/contrib/pg_upgrade/option.c b/contrib/pg_upgrade/option.c
> index 83300a9..a4760a7 100644
> --- a/contrib/pg_upgrade/option.c
> +++ b/contrib/pg_upgrade/option.c
> @@ -46,7 +46,7 @@ parseCommandLine(migratorContext *ctx, int argc, char 
> *argv[])
>   {"verbose", no_argument, NULL, 'v'},
>   {NULL, 0, NULL, 0}
>   };
> - charoption; /* Command line option */
> + int option; /* Command line option */
>   int optindex = 0;   /* used by getopt_long */
>   int user_id;
>   

> 
> -- 
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + None of us is going to be here forever. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [RRR] Reviewfest 2010-06 Plans and Call for Reviewers

[Selena Deckelmann]

Hi!

On Mon, Jun 14, 2010 at 9:25 AM, Kevin Grittner
 wrote:

> http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan
>
> calls for a ReviewFest to run from the 15th of June (tomorrow) until
> the start of the first CommitFest for the 9.1 release.  The idea is
> that those with time available to contribute beyond what they can
> usefully contribute to getting 9.0 released can help provide
> feedback on patches submitted so far, to lighten the load of the CF
> proper when it starts.  I have agreed to manage this RF.

I'm helping out too :)

A few folks from PDXPUG are meeting up this evening to go over some
patches. We'll be on IRC at #pdxpug from 6pm-8pm PDT if you want to
join in virtually.

We'll post our results tomorrow!

-selena

-- 
http://chesnok.com/daily - me

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Bruce Momjian]

David E. Wheeler wrote:
> On Jun 15, 2010, at 3:22 PM, Bruce Momjian wrote:
> 
> > I totaly agreed you need funding, and you are very well qualified to do
> > this, and it is a badly needed facility.
> 
> Thanks.
> 
> > The problem I had is that the effort appeared to be "I am creating my
> > own sandbox, fund me" (particularly the FAQ), which is probably not what
> > you wanted to convey.  I understand adjustments are being made and if
> > you can clarify how this is going to relate to the community in terms of
> > input, oversight, and management, it might be something the entire
> > community can get behind, and help fund.
> 
> Agreed. How's this?
> 
>   http://pgxn.org/faq.html

That is something everyone can get behind!  Great.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + None of us is going to be here forever. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Ross J. Reedstrom]

On Tue, Jun 15, 2010 at 03:42:59PM -0700, David E. Wheeler wrote:
> On Jun 15, 2010, at 3:22 PM, Bruce Momjian wrote:
> 
> > I totaly agreed you need funding, and you are very well qualified to do
> > this, and it is a badly needed facility.
> 
> Thanks.
> 
> > The problem I had is that the effort appeared to be "I am creating my
> > own sandbox, fund me" (particularly the FAQ), which is probably not what
> > you wanted to convey.  I understand adjustments are being made and if
> > you can clarify how this is going to relate to the community in terms of
> > input, oversight, and management, it might be something the entire
> > community can get behind, and help fund.
> 
> Agreed. How's this?
> 
>   http://pgxn.org/faq.html
> 
+1 Excellent, actually.

Ross
-- 
Ross Reedstrom, Ph.D. reeds...@rice.edu
Systems Engineer & Admin, Research Scientistphone: 713-348-6166
The Connexions Project  http://cnx.orgfax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E  F888 D3AE 810E 88F0 BEDE

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[David E. Wheeler]

On Jun 15, 2010, at 3:22 PM, Bruce Momjian wrote:

> I totaly agreed you need funding, and you are very well qualified to do
> this, and it is a badly needed facility.

Thanks.

> The problem I had is that the effort appeared to be "I am creating my
> own sandbox, fund me" (particularly the FAQ), which is probably not what
> you wanted to convey.  I understand adjustments are being made and if
> you can clarify how this is going to relate to the community in terms of
> input, oversight, and management, it might be something the entire
> community can get behind, and help fund.

Agreed. How's this?

  http://pgxn.org/faq.html

h/t Josh Berkus.

Best,

David



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[David E. Wheeler]

On Jun 15, 2010, at 3:23 PM, Robert Haas wrote:

> I think this project is a great idea, and I think as a community we
> ought to be behind it 100%.
> 
> However, I do wonder what happened to the original name, which IIRC
> was PGAN.  That seems easier to pronounce, remember, ...

I didn't care for it, personally. "Pee-Gan" sounds weird to my ear. I prefer 
"pee-gee-ex-en." But you can go for "pixin" or "pigskin" if you'd rather. ;-)

My bike shed is chartreuse,

David


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Robert Haas]

On Tue, Jun 15, 2010 at 4:12 PM, Bruce Momjian  wrote:
> This was just posted to announce.  Seems the community now has to
> compete with another extension-based infrastructure if we ever get
> around to developing one of our own.
>
> I personally had no knowledge of this, which is fine, but don't expect
> me to get excited about it, except to consider it a threat to a
> community-lead extension site.

I think this project is a great idea, and I think as a community we
ought to be behind it 100%.

However, I do wonder what happened to the original name, which IIRC
was PGAN.  That seems easier to pronounce, remember, ...

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Bruce Momjian]

David E. Wheeler wrote:
> On Jun 15, 2010, at 2:06 PM, Ross J. Reedstrom wrote:
> 
> > One issue that will come up: this is clearly a more commercial
> > enterprise than Jarkko's CPAN (and the internet is a different place
> > than it was in 1995) You pushed money right to the front with this, so
> > that will lead to certain questions concerning ownership of what
> > arguably should be community resources: the IP of the aggregate index,
> > for example.
> >
> > I'm a big believer in JFDI as well - just be aware that toes will get
> > stepped on, and require some bandages.
> 
> I have bandages. And a machete. ;-P
> 
> Seriously, it's not commercial. I really want to do it and I can't
> really afford to do it in my meager spare time. Those with sore toes
> will be asked to wait and see.

I totaly agreed you need funding, and you are very well qualified to do
this, and it is a badly needed facility.

The problem I had is that the effort appeared to be "I am creating my
own sandbox, fund me" (particularly the FAQ), which is probably not what
you wanted to convey.  I understand adjustments are being made and if
you can clarify how this is going to relate to the community in terms of
input, oversight, and management, it might be something the entire
community can get behind, and help fund.

--
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + None of us is going to be here forever. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] hstore ==> and deprecate =>

[Robert Haas]

On Mon, Jun 14, 2010 at 4:35 PM, Andrew Gierth
 wrote:
> I'm happy with deprecating the first two => in favour of hstore() if
> that is in line with general opinion. The hstore => text[] slice could
> be replaced by another operator name; the existing name comes from the
> analogy that (hstore -> text[]) returns the list of values, whereas
> (hstore => text[]) returns both the keys and values.

So, I kind of like Florian Pflug's suggestion upthread of replacing
hstore => text by hstore & text[].  I think that's about as mnemonic
as we're likely to get, and it gels nicely with the hstore ?& text[]
operator, which tests whether all of the named keys are present in the
hstore.

Does anyone want to bikeshed further before I go do that?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Josh Berkus]

On 6/15/10 2:06 PM, Ross J. Reedstrom wrote:
> You pushed money right to the front with this, so
> that will lead to certain questions concerning ownership of what
> arguably should be community resources: the IP of the aggregate index,
> for example.

Oh, good point.  We clearly need a FAQ item about that.  Or two or
three.  Working on it.

-- 
  -- Josh Berkus
 PostgreSQL Experts Inc.
 http://www.pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Joshua D. Drake]

On Tue, 2010-06-15 at 14:16 -0700, David E. Wheeler wrote:
> On Jun 15, 2010, at 2:06 PM, Ross J. Reedstrom wrote:
> 
> > One issue that will come up: this is clearly a more commercial
> > enterprise than Jarkko's CPAN (and the internet is a different place
> > than it was in 1995) You pushed money right to the front with this, so
> > that will lead to certain questions concerning ownership of what
> > arguably should be community resources: the IP of the aggregate index,
> > for example.
> > 
> > I'm a big believer in JFDI as well - just be aware that toes will get
> > stepped on, and require some bandages.
> 
> I have bandages. And a machete. ;-P
> 
> Seriously, it's not commercial. I really want to do it and I can't really 
> afford to do it in my meager spare time. Those with sore toes will be asked 
> to wait and see.

To help alleviate some of the "commercial" fears, I would suggest an
entry in the FAQ for the project that states the following:

Where the source code will be hosted (for the project itself)
What license it will be released under

Sincerely,

Joshua D. Drake

P.S. If you can monetize this, go for it.

-- 
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Ross J. Reedstrom]

On Tue, Jun 15, 2010 at 01:25:33PM -0700, David E. Wheeler wrote:
> On Jun 15, 2010, at 1:12 PM, Bruce Momjian wrote:
> 
> > This was just posted to announce.  Seems the community now has to
> > compete with another extension-based infrastructure if we ever get
> > around to developing one of our own.
> > 
> > I personally had no knowledge of this, which is fine, but don't expect
> > me to get excited about it, except to consider it a threat to a
> > community-lead extension site.
> 
> This *is* for the community, Bruce. There was extensive discussion of my 
> original proposal back in January:
> 
>   http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg143645.html
> 
> I welcome all contributions from the community. I want it to be a success for 
> PostgreSQL. But note that it doesn't have to be started as a -hackers project 
> (any more than pg_upgrade did). CPAN, for example, was created because Jarkko 
> had an itch. He scratched it. I'm doing the same here.

One issue that will come up: this is clearly a more commercial
enterprise than Jarkko's CPAN (and the internet is a different place
than it was in 1995) You pushed money right to the front with this, so
that will lead to certain questions concerning ownership of what
arguably should be community resources: the IP of the aggregate index,
for example.

I'm a big believer in JFDI as well - just be aware that toes will get
stepped on, and require some bandages.

Ross
-- 
Ross Reedstrom, Ph.D. reeds...@rice.edu
Systems Engineer & Admin, Research Scientistphone: 713-348-6166
The Connexions Project  http://cnx.orgfax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E  F888 D3AE 810E 88F0 BEDE

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] 9.0 beta2 pg_upgrade command line parsing

[Steve Singer]

When I try running pg_upgrade from beta2 on an AIX server I'm unable to 
get beyond the command line parsing (no matter what command line 
arguments I pass in it always complains about the usage).



When I compile pg_upgrade it complains about:

option.c: In function 'parseCommandLine':
option.c:103: warning: comparison is always true due to limited range of 
data type


getopt_long  (both our version in ports and the gnu version) is defined 
to return an int.


The attach patch allows me to get beyond the options loop under AIX.





--
Steve Singer
Afilias Canada
Data Services Developer
416-673-1142
diff --git a/contrib/pg_upgrade/option.c b/contrib/pg_upgrade/option.c
index 83300a9..a4760a7 100644
--- a/contrib/pg_upgrade/option.c
+++ b/contrib/pg_upgrade/option.c
@@ -46,7 +46,7 @@ parseCommandLine(migratorContext *ctx, int argc, char *argv[])
 		{"verbose", no_argument, NULL, 'v'},
 		{NULL, 0, NULL, 0}
 	};
-	char		option;			/* Command line option */
+	int		option;			/* Command line option */
 	int			optindex = 0;	/* used by getopt_long */
 	int			user_id;
 	

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[David E. Wheeler]

On Jun 15, 2010, at 2:06 PM, Ross J. Reedstrom wrote:

> One issue that will come up: this is clearly a more commercial
> enterprise than Jarkko's CPAN (and the internet is a different place
> than it was in 1995) You pushed money right to the front with this, so
> that will lead to certain questions concerning ownership of what
> arguably should be community resources: the IP of the aggregate index,
> for example.
> 
> I'm a big believer in JFDI as well - just be aware that toes will get
> stepped on, and require some bandages.

I have bandages. And a machete. ;-P

Seriously, it's not commercial. I really want to do it and I can't really 
afford to do it in my meager spare time. Those with sore toes will be asked to 
wait and see.

Thanks,

David


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Josh Berkus]

> I, for one am +1 (although I do note a hint of PGX in PGXN :P).

Yeah, we tried to come up with a name which didn't have "PGX" in it, for
obvious reasons.  However, everything we could come up with (here and on
IRC) was very contrived.  And "PGXN" does dovetail nicely with "PGXS",
which it utilizes.

I also hope to fundraise for Dimitri's work; if we could get both of
those projects completed in 2011, it would really change things for the
Postgres world, I think.

-- 
  -- Josh Berkus
 PostgreSQL Experts Inc.
 http://www.pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Josh Berkus]

On 6/15/10 1:12 PM, Bruce Momjian wrote:
> This was just posted to announce.  Seems the community now has to
> compete with another extension-based infrastructure if we ever get
> around to developing one of our own.

It is possible for things to be community and not originate in the Core
Team, Bruce.  Just because you personally are not involved doesn't
automatically make a project "enemy of the community".

-- 
  -- Josh Berkus
 PostgreSQL Experts Inc.
 http://www.pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Dimitri Fontaine]

Bruce Momjian  writes:
> This was just posted to announce.  Seems the community now has to
> compete with another extension-based infrastructure if we ever get
> around to developing one of our own.

The chapter of PGXN as announced here is to build a network of
extensions, the ones I'm working on for core. It's the next step. You
want the ability to easily install, drop, dump and restore extensions,
then you want some sources of them that you can trust and from where
it's easy to fetch and build. That's the "network".

We're also working on the grotty details about how to best work together
at the moment, the separation of work had been agreed on before PgCon
and confirmed in Ottawa.

Regards,
-- 
dim

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Joshua D. Drake]

On Tue, 2010-06-15 at 16:12 -0400, Bruce Momjian wrote:
> This was just posted to announce.  Seems the community now has to
> compete with another extension-based infrastructure if we ever get
> around to developing one of our own.
> 
> I personally had no knowledge of this, which is fine, but don't expect
> me to get excited about it, except to consider it a threat to a
> community-lead extension site.

There was a long discussion about this Bruce. On hackers. David even
sent out an RFC:

http://archives.postgresql.org/pgsql-hackers/2010-01/msg00692.php

I, for one am +1 (although I do note a hint of PGX in PGXN :P).

Sincerely,

Joshua D. Drake

-- 
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[Andrew Dunstan]

Then you weren't paying attention at the devlopers' meeting. It's the 
bottom item on 



cheers

andrew


Bruce Momjian wrote:

This was just posted to announce.  Seems the community now has to
compete with another extension-based infrastructure if we ever get
around to developing one of our own.

I personally had no knowledge of this, which is fine, but don't expect
me to get excited about it, except to consider it a threat to a
community-lead extension site.

---

David E. Wheeler wrote:
  

I'm pleased to announce the launch of the PGXN development project.

  http://www.pgxn.org/

PGXN, the PostgreSQL Extension Network, is modelled on CPAN, the Perl community's archive 
of "all things Perl." PGXN will provide four major pieces of infrastructure to 
the PostgreSQL community:

* An upload and distribution infrastructure for extension developers
* A centralized index and API of distribution metadata
* A website for searching extensions and perusing their documentation
* A command-line client for downloading, testing, and installing extensions

We have started the fundraising phase of the project now. Thanks to founding 
sponsors myYearbook.com and PostgreSQL Experts, Inc., we're already 2/5 of the 
way to our goal. Complete details of the project -- including the 
specification, implementation plan, and  fundraising FAQ -- are on the site.

Best,

David
PostgreSQL Experts, Inc.


---(end of broadcast)---
-To unsubscribe from this list, send an email to:

   pgsql-announce-unsubscr...@postgresql.org



  


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] New PGXN Extension site

[David E. Wheeler]

On Jun 15, 2010, at 1:12 PM, Bruce Momjian wrote:

> This was just posted to announce.  Seems the community now has to
> compete with another extension-based infrastructure if we ever get
> around to developing one of our own.
> 
> I personally had no knowledge of this, which is fine, but don't expect
> me to get excited about it, except to consider it a threat to a
> community-lead extension site.

This *is* for the community, Bruce. There was extensive discussion of my 
original proposal back in January:

  http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg143645.html

I welcome all contributions from the community. I want it to be a success for 
PostgreSQL. But note that it doesn't have to be started as a -hackers project 
(any more than pg_upgrade did). CPAN, for example, was created because Jarkko 
had an itch. He scratched it. I'm doing the same here.

Best,

David
-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] New PGXN Extension site

[Bruce Momjian]

This was just posted to announce.  Seems the community now has to
compete with another extension-based infrastructure if we ever get
around to developing one of our own.

I personally had no knowledge of this, which is fine, but don't expect
me to get excited about it, except to consider it a threat to a
community-lead extension site.

---

David E. Wheeler wrote:
> I'm pleased to announce the launch of the PGXN development project.
> 
>   http://www.pgxn.org/
> 
> PGXN, the PostgreSQL Extension Network, is modelled on CPAN, the Perl 
> community's archive of "all things Perl." PGXN will provide four major pieces 
> of infrastructure to the PostgreSQL community:
> 
> * An upload and distribution infrastructure for extension developers
> * A centralized index and API of distribution metadata
> * A website for searching extensions and perusing their documentation
> * A command-line client for downloading, testing, and installing extensions
> 
> We have started the fundraising phase of the project now. Thanks to founding 
> sponsors myYearbook.com and PostgreSQL Experts, Inc., we're already 2/5 of 
> the way to our goal. Complete details of the project -- including the 
> specification, implementation plan, and  fundraising FAQ -- are on the site.
> 
> Best,
> 
> David
> PostgreSQL Experts, Inc.
> 
> 
> ---(end of broadcast)---
> -To unsubscribe from this list, send an email to:
> 
>pgsql-announce-unsubscr...@postgresql.org

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + None of us is going to be here forever. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers

[Robert Haas]

On Tue, Jun 15, 2010 at 3:57 PM, Josh Berkus  wrote:
>> I wonder if it would be possible to jigger things so that we send the
>> WAL to the standby as soon as it is generated, but somehow arrange
>> things so that the standby knows the last location that the master has
>> fsync'd and never applies beyond that point.
>
> I can't think of any way which would not require major engineering.  And
> you'd be slowing down replication *in general* to deal with a fairly
> unlikely corner case.
>
> I think the panic is the way to go.

I have yet to convince myself of how likely this is to occur.  I tried
to reproduce this issue by crashing the database, but I think in 9.0
you need an actual operating system crash to cause this problem, and I
haven't yet set up an environment in which I can repeatedly crash the
OS.  I believe, though, that in 9.1, we're going to want to stream
from WAL buffers as proposed in the patch that started out this
thread, and then I think this issue can be triggered with just a
database crash.

In 9.0, I think we can fix this problem by (1) only streaming WAL that
has been fsync'd and (2) PANIC-ing if the problem occurs anyway.  But
in 9.1, with sync rep and the performance demands that entails, I
think that we're going to need to rethink it.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers

[Josh Berkus]

> I wonder if it would be possible to jigger things so that we send the
> WAL to the standby as soon as it is generated, but somehow arrange
> things so that the standby knows the last location that the master has
> fsync'd and never applies beyond that point.

I can't think of any way which would not require major engineering.  And
you'd be slowing down replication *in general* to deal with a fairly
unlikely corner case.

I think the panic is the way to go.

-- 
  -- Josh Berkus
 PostgreSQL Experts Inc.
 http://www.pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PERFORM] No hash join across partitioned tables?

[Robert Haas]

On Sun, Jun 13, 2010 at 11:47 PM, Robert Haas  wrote:
> Proposed patch attached.

Hearing no objections, I have committed this patch.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [v9.1] Add security hook on initialization of instance

[Robert Haas]

On Tue, Jun 15, 2010 at 8:37 AM, Stephen Frost  wrote:
> KaiGai,
>
> * KaiGai Kohei (kai...@ak.jp.nec.com) wrote:
>> In the attached patch, the security hook was moved to ClientAuthentication()
>> from InitPostgres(), for more clarification of the purpose.
>> What I want to do is to assign additional properties to identify the client
>> (such as security label) for each authenticated session.
>>
>> Its purpose is similar to "session" module of PAM in operating system.
>> It allows to assign additional session properties more than user-id.
>
> That's all fine- but let's work within the confines of the *existing*
> hook that's been discussed to get something working first before we go
> adding hooks in other places.  I think it's important that we put
> together at least a proof of concept that an SELinux module or other
> external auth module can sensible use the DML hook.

+1.

> After that, we can discuss what other hooks are needed.  KaiGai, please,
> before sending in patches of any kind, propose at a high-level what the
> problem is and what the security module needs in general terms.  If you
> have a recommendation, that's fine, but let's talk about it before
> implementing anything.

+2.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [v9.1] Add security hook on initialization of instance

[Stephen Frost]

KaiGai,

* KaiGai Kohei (kai...@ak.jp.nec.com) wrote:
> In the attached patch, the security hook was moved to ClientAuthentication()
> from InitPostgres(), for more clarification of the purpose.
> What I want to do is to assign additional properties to identify the client
> (such as security label) for each authenticated session.
> 
> Its purpose is similar to "session" module of PAM in operating system.
> It allows to assign additional session properties more than user-id.

That's all fine- but let's work within the confines of the *existing*
hook that's been discussed to get something working first before we go
adding hooks in other places.  I think it's important that we put
together at least a proof of concept that an SELinux module or other
external auth module can sensible use the DML hook.

After that, we can discuss what other hooks are needed.  KaiGai, please,
before sending in patches of any kind, propose at a high-level what the
problem is and what the security module needs in general terms.  If you
have a recommendation, that's fine, but let's talk about it before
implementing anything.

Thanks,

Stephen


signature.asc
Description: Digital signature

Re: [HACKERS] [BUGS] Server crash while trying to read expression using pg_get_expr()

[Florian Pflug]

On Jun 15, 2010, at 9:31 , Heikki Linnakangas wrote:
> You could avoid changing the meaning of fn_expr by putting the check in the 
> parse analysis phase, into transformFuncCall(). That would feel safer at 
> least for back-branches.

For 9.0, wouldn't a cleaner way to accomplish this be a seperate type for 
expressions, say pg_expr, instead of storing them as text? With an input 
function that unconditionally raises and error and no cast to pg_expr, creating 
new instances of that type would be impossible for normal users. The output 
function and casts to text would call pg_get_expr() with zero as the second 
argument.

The internal representation wouldn't change, it's just the type's OID in the 
catalog that'd be different.

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers

[Florian Pflug]

On Jun 15, 2010, at 10:45 , Fujii Masao wrote:
> A transaction commit would need to wait for local fsync and replication
> in a serial manner, in synchronous replication. IOW, walsender cannot
> send the commit record until it's fsync'd in XLogWrite().

Hm, but since 9.0 won't do synchronous replication anyway, the right thing to 
do for 9.0 is still to send only fsync'ed WAL, no? Without synchronous 
replication the overhead seems negligible.

For synchronous replication (and hence for 9.1) I think there are two basic 
options

a) Stream only fsync'ed WAL, like in the asynchronous case. Depending on 
policy, additionally wait for one or more slaves to fsync before reporting 
success.

b) Stream non-fsync'ed WAL. on COMMIT, wait for at last one node (not 
necessarily the master, exact count depends on policy) to fsync before 
reporting success. During recovery of the master, recover up to the latest LSN 
found on any one of the nodes.

Option (b) requires some additional thought, though. Controlled removal of 
slave nodes and concurrent crashes of more than one node are the most difficult 
areas to handle gracefully, it seems.

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers

[Robert Haas]

On Tue, Jun 15, 2010 at 12:46 AM, Fujii Masao  wrote:
> On Mon, Jun 14, 2010 at 10:13 PM, Robert Haas  wrote:
>> On Mon, Jun 14, 2010 at 8:41 AM, Fujii Masao  wrote:
>>> On Mon, Jun 14, 2010 at 8:10 PM, Robert Haas  wrote:
 Maybe.  That sounds like a pretty enormous foot-gun to me, considering
 that we have no way of recovering from the situation where the standby
 gets ahead of the master.
>>>
>>> No, we can do that by reconstructing the standby from the backup.
>>>
>>> And, that situation is not a problem for users including me who prefer to
>>> perform a failover when the master goes down.
>>
>> You don't get to pick - if a backend crashes on the master, it will
>> restart right away and come up, but the slave will now be hosed...
>
> You are concerned about the case where postmaster automatically restarts
> the crash recovery, in particular? Yes, this case is more problematic.
> If the standby is ahead of the master, the standby might find an invalid
> record and run into the infinite retry loop, or keep working without
> noticing the inconsistency between the database and the WAL.
>
> I'm thinking that walreceiver should throw a PANIC when it receives the
> record which is in the LSN older than the last WAL receive location,
> except the beginning of streaming (because the standby always requests
> for streaming from the starting of WAL file at first even if some records
> have already been received in previous time). Thought?

Yeah, that seems like it would be a good safety check.

I wonder if it would be possible to jigger things so that we send the
WAL to the standby as soon as it is generated, but somehow arrange
things so that the standby knows the last location that the master has
fsync'd and never applies beyond that point.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] debug log in pg_archivecleanup

[Fujii Masao]

Hi,

Sometimes the postgres server log message and the pg_archivecleanup debug
message are output in the same line as follows. This is a little hard to
read.

---
LOG:  restored log file "0001006B" from archive
pg_archivecleanup:  keep WAL file 00010068 and later
pg_archivecleanup:  removing
"/dav/head-pgsql/act.arh/00010048"LOG:  restored log
file "0001006C" from archive

pg_archivecleanup:  removing "/dav/head-pgsql/act.arh/00010061"
pg_archivecleanup:  removing "/dav/head-pgsql/act.arh/0001004D"
pg_archivecleanup:  removing "/dav/head-pgsql/act.arh/0001005C"
---

This is because pg_archivecleanup puts the line break "\n" in the head of
debug message. Why should we do so?

---
 if (debug)
fprintf(stderr, "\n%s:  removing \"%s\"", progname, WALFilePath);
---

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

PL/Perl function naming (was: [HACKERS] release notes)

[Tim Bunce]

[Sorry for the delay. I'd stopped checking the pgsql mailing lists.
Thanks to David Wheeler and Josh Berkus for the heads-up.]

On Sun, May 30, 2010 at 06:58:32PM -0400, Andrew Dunstan wrote:
> 
> Tim Bunce wrote:
> >On Mon, May 17, 2010 at 11:34:37AM -0400, Tom Lane wrote:
> >>Andrew Dunstan  writes:
> >>>Why do the release notes say this, under plperl:
> >>>* PL/Perl subroutines are now given names (Tim Bunce)
> >>>  This is for the use of profiling and code coverage tools. DIDN'T
> >>>  THEY HAVE NAMES BEFORE?
> >>>   If whoever put this in the release notes had bothered
> >>>to ask I am sure we would have been happy to explain.
> >>You don't need to complain, just fix it.  The point of the comment is
> >>that more explanation is needed.
> >
> >I think you can just delete it. It's too esoteric to be worth noting.
> >
> >Tim.
> >
> >p.s. It also turned to be insufficiently useful for NYTProf since it
> >doesn't also update some internals to record the 'filename' and line
> >number range of the sub. So PostgreSQL::PLPerl::NYTProf works around
> >that by wrapping mkfuncsrc() to edit the generated perl code to include
> >a subname in the usual "sub $name { ... }" style. I may offer a patch
> >for 9.1 to to make it work that way.
>
> I put this aside to think about it.
> 
> If the "feature" is not any use should we rip it out? We pretty much
> included it because you said it was what you needed for the
> profiler.

Yes, it can go.

*** a/src/pl/plperl/plperl.c
--- b/src/pl/plperl/plperl.c
*** plperl_create_sub(plperl_proc_desc *prod
*** 1319,1328 
(errmsg("didn't get a CODE ref from compiling 
%s",
prodesc->proname)));

-   /* give the subroutine a proper name in the main:: symbol table */
-   CvGV(SvRV(subref)) = (GV *) newSV(0);
-   gv_init(CvGV(SvRV(subref)), PL_defstash, subname, strlen(subname), 
TRUE);
-   
prodesc->reference = subref;

return;

> I'm seriously nervous about adding function names - making functions
> directly callable like that could be a major footgun recipe, since
> now we are free to hide some stuff in the calling mechanism, and can
> (and have in the past) changed that to suit our needs, e.g. when we
> added trigger support via a hidden function argument. That has been
> safe precisely because nobody has had a handle on the subroutine
> which would enable it to be called directly from perl code. There
> have been suggestions in the past of using this for other features,
> so we should not assume that the calling mechanism is fixed in stone.

Agreed. It is a very hard to use footgun though because the oid is
included in the name. It's certainly not something anyone would shoot
themselves with by accident.

[Speaking of calling conventions: I never did get time for some decent
performance optimization. I'd like to get rid of the explicit extra
trigger data argument and corresponding "local $_TD=shift;" prologue.
That could be done much faster in C.]

> Perhaps we could decorate the subs with attributes, or is that not
> doable for anonymous subs?

Perhaps. I'll look into it when I get around to working on the PL/Perl
NYTProf again. For the profiler it would be enough to only enable the
naming when the appropriate perl debugging flag bits are set, so it
wouldn't happen normally.

Tim.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Proposal for 9.1: WAL streaming from WAL buffers

[Fujii Masao]

On Tue, Jun 15, 2010 at 2:16 PM, Heikki Linnakangas
 wrote:
> On 15/06/10 07:47, Fujii Masao wrote:
>>
>> On Tue, Jun 15, 2010 at 12:02 AM, Tom Lane  wrote:
>>>
>>> Fujii Masao  writes:

 Walsender tries to send WAL up to xlogctl->LogwrtResult.Write. OTOH,
 xlogctl->LogwrtResult.Write is updated after XLogWrite() performs fsync.
>>>
>>> Wrong.  LogwrtResult.Write tracks how far we've written out data,
>>> but it is only (known to be) fsync'd as far as LogwrtResult.Flush.
>>
>> Hmm.. I agree that xlogctl->LogwrtResult.Write indicates the byte position
>> we've written. But in the current XLogWrite() code, it's updated after
>> XLogWrite() calls issue_xlog_fsync(). No?
>
> issue_xlog_fsync() is only called if the caller requested a flush by
> advancing WriteRqst.Flush.

True. The scenario that I'm concerned about is:

1. A transaction commit causes XLogFlush() to write *and* fsync WAL up to
   the commit record.
2. XLogFlush() calls XLogWrite(), and xlogctl->LogwrtResult.Write is
   updated to indicate the LSN bigger than or equal to that of the commit
   record after XLogWrite() calls issue_xlog_fsync().
3. Then walsender can send WAL up to the commit record.

A transaction commit would need to wait for local fsync and replication
in a serial manner, in synchronous replication. IOW, walsender cannot
send the commit record until it's fsync'd in XLogWrite().

This scenario will not happen? Am I missing something?

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] InvalidXLogRecPtr in docs

[Fujii Masao]

On Tue, Jun 15, 2010 at 2:41 PM, Heikki Linnakangas
 wrote:
> On 15/06/10 08:23, Fujii Masao wrote:
>>
>> On Thu, Jun 10, 2010 at 11:06 PM, Tom Lane  wrote:
>>>
>>> I'm not sure if it's worth the trouble, or even a particularly smart
>>> idea, to force the output of the status function to be monotonic
>>> regardless of what happens underneath.  I think removing that claim
>>> from the docs altogether is the easiest answer.
>>
>> We should
>>
>> (1) just remove "While streaming replication is in progress this will
>>     increase monotonically." from the description about
>> pg_last_xlog_receive_location()?
>>
>> or
>>
>> (2) add "But if streaming replication is restarted this will back off
>>     to the beginning of current WAL file" into there?
>>
>> I'm for (2) since it's more informative. Thought?
>
> Something like (2) seems better, because even if we remove the note that it
> increases monotonically, people might still assume that.

The attached patch adds the following:

-
But when streaming replication is
restarted this will back off to the replication starting position,
which typically indicates the beginning of the WAL file including the
record in the position which pg_last_xlog_replay_location
points to at the moment.
-

Regards,

-- 
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center


pg_last_xlog_receive_location_doc_v1.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [BUGS] Server crash while trying to read expression using pg_get_expr()

[Heikki Linnakangas]

On 10/06/10 00:24, Tom Lane wrote:

I wrote:

[ thinks for awhile... ]  I wonder whether there is any way of locking
down pg_get_expr so that it throws an error if called with anything
except a suitable field from one of the system catalogs.


I did a bit of research into this idea.  It looks at least somewhat
feasible:

* All PG versions back to 7.4 will pass the calling expression tree via
fcinfo->flinfo->fn_expr.  Lack of that would be a showstopper, because
we can't change the FmgrInfo struct in back branches (ABI break).  So we
can get the arguments and determine whether they are Vars.

* To determine which table a Var actually refers to, we must have access
to the rangetable, and in join cases we also need access to the plan
tree node containing the Var (since we have to drill down to the plan
node's inputs to resolve OUTER and INNER references).  The rangetable is
reachable from the PlanState node, so it's sufficient to make that one
pointer available to functions.  The obvious way to handle this is to
add a field to FmgrInfo, and I would suggest doing it that way as of
9.0.  In the back branches, we could probably hack it without an ABI
break by having fn_expr point to some intermediate node type that
contains both the actual expression tree and the PlanState pointer
(probably, we'd make it point to FuncExprState instead of directly to
the FuncExpr, and then add the field to FuncExprState, which is far less
dangerous than redefining struct FmgrInfo).  Now this depends on the
assumption that no external functions are doing anything with fn_expr
except passing it to the related fmgr.c routines, which we would teach
about this convention.

* Once we've got the above data it's a simple matter of adapting the
Var-interpretation logic used by EXPLAIN in order to find out the table
OID and column number, if any, represented by the Var.  I'd suggest
adding such functions in fmgr.c to extend the API currently offered by
get_fn_expr_argtype() and friends.  It seems possible that other
functions besides pg_get_expr might have use for such a capability.


What I'm suggesting is definitely not a trivial patch, and I would never
consider back-patching it if it weren't a security matter.  But I think
it's doable and we'd be fixing the hole with a determinate amount of
work, whereas trying to verify the validity of pg_get_expr input
directly would be a never-ending source of more security bugs.


Yeah, seems like it would work.

You could avoid changing the meaning of fn_expr by putting the check in 
the parse analysis phase, into transformFuncCall(). That would feel 
safer at least for back-branches.


--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [v9.1] Add security hook on initialization of instance

[KaiGai Kohei]

(2010/06/15 12:47), KaiGai Kohei wrote:
> (2010/06/15 12:28), Tom Lane wrote:
>> KaiGai Kohei   writes:
>> The attached patch tries to add one more security hook on the
>> initialization of PostgreSQL instance (InitPostgres()).
>>
 Yeah, but so what?  Stephen's point is still valid.
>>
>>> On the hook, I'd like to obtain security context of the client process
>>> which connected to the PostgreSQL instance. It is not available at the
>>> _PG_init() phase, because clients don't connect yet.
>>
>> InitPostgres is called by a number of process types that don't *have*
>> clients.  I concur with the other opinions that this hook is badly
>> thought out.
>>
> I intended to skip it when InitPostgres() is called without clients.
> 
> For example, the hook might be better to put on PerformAuthentication()
> for more clarification of the purpose.
> 

In the attached patch, the security hook was moved to ClientAuthentication()
from InitPostgres(), for more clarification of the purpose.
What I want to do is to assign additional properties to identify the client
(such as security label) for each authenticated session.

Its purpose is similar to "session" module of PAM in operating system.
It allows to assign additional session properties more than user-id.

Thanks,
-- 
KaiGai Kohei 


pgsql-v9.1-add-auth-hook.2.patch
Description: application/octect-stream

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers