Re: [HACKERS] [COMMITTERS] pgsql: Use gender-neutral language in documentation

[Garick Hamlin]

On Tue, Sep 22, 2015 at 11:59:21AM -0400, Robert Haas wrote:
> On Tue, Sep 22, 2015 at 11:00 AM, Andrew Dunstan  wrote:
> >> Happily for me, I can continue to write documents in a grammatically
> >> correct way, and no-one will read them and think I'm a grammar-nazi (or
> >> obstinate, or old-fashioned or whatever) because unless they're 
> >> specifically
> >> looking for it no-one will notice that I'm avoiding the contentious usage
> >> altogether. On the other hand, there _will_ be a (perhaps significant)
> >> proportion of people who read your documents and think that you're 
> >> incapable
> >> of writing a grammatically correct sentence.
> >
> > Wow, 1960s feminazis, eh? I originally thought you were just a narrow
> > minded, pedantic and antiquated grammarian. Now I realize that's the least
> > of your troubles. Please take your misogyny elsewhere. I hear the Rabid
> > Puppies have openings.
> 
> I think this discussion could benefit from a little more light and a
> lot less heat.
> 
> Here's my contribution: the use of they does sometimes seek awkward.
> However, it's not remotely new:
> 
> https://stroppyeditor.wordpress.com/2015/04/21/everything-you-ever-wanted-to-know-about-singular-they/
> https://en.wikipedia.org/wiki/Singular_they#Older_usage_by_respected_authors
> http://englishbibles.blogspot.com/2006/09/singular-they-in-english-bibles.html
> 
> And I do think it's generally worthwhile to avoid the use of "he"
> where possible.  Would I have done it exactly the way that Peter did
> it here?  Uh, no.  Is it better than not doing anything?  In my
> opinion, yes.

I agree, I think we should avoid gendered pronouns.  

Also, the modern use of they/their absolutely fits here.  It reflects a
deliberate considered choice of of the writer to be inclusive and correct.

Constructs like 'he or she' exclude people.  Enumerating genders is not 
inclusive.  It leads to exclusion and erasure of people who have an non-binary
identities. 

Garick


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Run pgindent now?

[Garick Hamlin]

On Tue, May 26, 2015 at 04:32:42PM -0300, Alvaro Herrera wrote:
 Robert Haas wrote:
 
  But every time we pgindent, especially with slightly different
  settings, we cause tools like 'git blame' to return less useful
  answers.  And that sucks.
 
 I've wondered a few times whether there's a way to make pgindent commits
 transparent to git blame, i.e. blame their modified lines to whatever
 commits modified them immediately before.

I wonder if it might be a good idea to separate whitespace changes within
a line from line breaks changes.

You could do something like a script (it could be used as a git hook) that
only enforces or warns about line-break style rules (which are easier to get
right, I think), and have a mode of pgindent that only changes whitespace 
within a line and warns about line break style problems.  So an author could
be more or less on the hook to have acceptable line-breaks and fix that on
their end and intra-line spacing could be fixed en mass with little impact.

I can't figure out how painful this would be in practice.

It's probably not worth it

Garick



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] additional json functionality

[Garick Hamlin]

On Wed, Nov 13, 2013 at 04:50:49PM -0800, David E. Wheeler wrote:
 On Nov 13, 2013, at 4:45 PM, Andrew Dunstan and...@dunslane.net wrote:
 
  It should be a pretty-printing function option, perhaps, but not core to 
  the type itself, IMO.
  
  I don't in the least understand how it could be a pretty printing option.
  If we move to a binary rep using the hstore stuff order will be destroyed
  and not stored anywhere, and duplicate keys will be lost. Once that's done,
  how would a pretty print option restore the lost info?
 
 I meant ordering the keys, usually in lexicographic order. I agree that 
 preserving order is untenable.

There is a canonical form.

http://tools.ietf.org/html/draft-staykov-hu-json-canonical-form-00

A Canonical form would be very useful.  Thats a bit trickier than sorting the 
keys and I don't know there is an accepted canonical form for json yet that
can represent all json documents.  (The canonical form is not the pretty form, 
but I think the key ordering should be the same.)

It might be nice to have a more general canonical form if one emerges from 
somewhere that could encode any json.  Since without something like this,
hashing can only be well specified for the 'sensible subset of json' used in
security protocols.

Garick


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] How can I build OSSP UUID support on Windows to avoid duplicate UUIDs?

[Garick Hamlin]

On Thu, Oct 31, 2013 at 09:54:14PM +0900, MauMau wrote:
 From: Robert Haas robertmh...@gmail.com
 ISTM that the biggest problem is that we don't have a random number
 generator which generates enough bits of randomness to implement
 uuid_generate_v3.  I think relatively few people would cry if we
 didn't support uuid_generate_v1(), and the others all look simple
 enough, provided there's somewhere to get lots of random bits.

 On Linux, it seems like we could get those bits from /dev/urandom,
 though I'm not sure how efficient that would be for the case where
 many UUIDs are being generated at once.  But that wouldn't be very
 portable.  It's tempting to think that we'd need a PRNG that generates
 wider values, for which we might find other application also.  But I'm
 not volunteering to be the one to create such a thing.

 OpenSSL provides rand_bytes() which generates random bytes of any length. 
 It uses /dev/urandom or /dev/random on UNIX/Linux and Crypto API of  
 Microsoft on Windows.

What about using a cipher here as the PRNG? It seems like using openssl
rand_bytes() to seed aes in ctr would work ok without starving the system of
entropy when making a lot of uuids.

Garick


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] How can I build OSSP UUID support on Windows to avoid duplicate UUIDs?

[Garick Hamlin]

On Thu, Oct 31, 2013 at 01:59:04PM -0400, Robert Haas wrote:
 On Thu, Oct 31, 2013 at 1:02 PM, Garick Hamlin gham...@isc.upenn.edu wrote:
  On Thu, Oct 31, 2013 at 09:54:14PM +0900, MauMau wrote:
  From: Robert Haas robertmh...@gmail.com
  ISTM that the biggest problem is that we don't have a random number
  generator which generates enough bits of randomness to implement
  uuid_generate_v3.  I think relatively few people would cry if we
  didn't support uuid_generate_v1(), and the others all look simple
  enough, provided there's somewhere to get lots of random bits.
 
  On Linux, it seems like we could get those bits from /dev/urandom,
  though I'm not sure how efficient that would be for the case where
  many UUIDs are being generated at once.  But that wouldn't be very
  portable.  It's tempting to think that we'd need a PRNG that generates
  wider values, for which we might find other application also.  But I'm
  not volunteering to be the one to create such a thing.
 
  OpenSSL provides rand_bytes() which generates random bytes of any length.
  It uses /dev/urandom or /dev/random on UNIX/Linux and Crypto API of
  Microsoft on Windows.
 
  What about using a cipher here as the PRNG? It seems like using openssl
  rand_bytes() to seed aes in ctr would work ok without starving the system of
  entropy when making a lot of uuids.
 
 There are two good reasons for us NOT to rely on OpenSSL:

Right, that makes sense.  openssl is a non-starter here.  In which case that
approach is no easier than any other prng.

I think using /dev/urandom directly would be surprising.  At least it would
have probably have taken me a while to figure out what was depleting the
entropy pool here.

Garick


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Enabling Checksums

[Garick Hamlin]

On Wed, Mar 06, 2013 at 01:34:21PM +0200, Heikki Linnakangas wrote:
 On 06.03.2013 10:41, Simon Riggs wrote:
 On 5 March 2013 18:02, Jeff Davispg...@j-davis.com  wrote:

 Fletcher is probably significantly faster than CRC-16, because I'm just
 doing int32 addition in a tight loop.

 Simon originally chose Fletcher, so perhaps he has more to say.

 IIRC the research showed Fletcher was significantly faster for only a
 small loss in error detection rate.

 It was sufficient to make our error detection  1 million times
 better, possibly more. That seems sufficient to enable early detection
 of problems, since if we missed the first error, a second is very
 likely to be caught (etc). So I am assuming that we're trying to catch
 a pattern of errors early, rather than guarantee we can catch the very
 first error.

 Fletcher's checksum is good in general, I was mainly worried about  
 truncating the Fletcher-64 into two 8-bit values. I can't spot any  
 obvious weakness in it, but if it's indeed faster and as good as a  
 straightforward Fletcher-16, I wonder why that method is not more widely  
 used.

I was wondering about the effectiveness of this resulting truncated hash
function as well.

 Another thought is that perhaps something like CRC32C would be faster to  
 calculate on modern hardware, and could be safely truncated to 16-bits  
 using the same technique you're using to truncate the Fletcher's  
 Checksum. Greg's tests showed that the overhead of CRC calculation is  
 significant in some workloads, so it would be good to spend some time to  
 optimize that. It'd be difficult to change the algorithm in a future  
 release without breaking on-disk compatibility, so let's make sure we  
 pick the best one.

If picking a CRC why not a short optimal one rather than truncate CRC32C?

I've been reading about optimal checksum for small messages for other 
reasons and found this paper quite good.

http://www.ece.cmu.edu/~koopman/roses/dsn04/koopman04_crc_poly_embedded.pdf

I was interested in small messages and small checksums so this paper may not be
as much help here.

Other than CRCs and fletcher sums, Pearson hashing with a 16-bit block might
be worth considering.  Either a pearson hash or a 16-CRC is small enough to
implement with a lookup table rather than a formula.  

I've been wondering what kind of errors we expect?  Single bit flips?  Large
swaths of bytes corrupted?  Are we more worried about collisions (the odds 
total garbage has the same checksum) or the odds we detect a flip of n-bits.
I would think since the message is large and a write to the wrong location
seems about as likely as a bit flip a pearson hash be good.

Any choice seems like it would be a nice improvement of noticing a storage stack
problem.  The difference would be subtle.  Can I estimate the odds of 
undetected corruption that occurred since the condition was first detected
accurately or does the checksum/hash perform poorly?

Garick



 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] COPY .. COMPRESSED

[Garick Hamlin]

On Tue, Jan 15, 2013 at 01:35:57PM +, Greg Stark wrote:
 On Tue, Jan 15, 2013 at 2:33 AM, Stephen Frost sfr...@snowman.net wrote:
  * Peter Eisentraut (pete...@gmx.net) wrote:
  On 1/14/13 11:28 AM, Stephen Frost wrote:
   While there is no option currently for having the server do the
   compression before sending the data over the wire.
 
  OpenSSL?
 
  To be honest, I expected that to come up earlier in this discussion.
  It'd be redundant to use OpenSSL for compression and then ALSO do
  compression on the client side to save into a custom format dump.
 
 For what it's worth there was a security announcement not long ago
 that made OpenSSL disable compression in streams by default. I'm not
 sure if it's relevant to Postgres or not.

It's an interesting question.  It might be.  I thought at first it
wouldn't be relevant, but on reflection it is.

This attack is called the CRIME attack.  This class of attacks stem from reuse
of a dictionary across some sort of confidentiality boundary.  The attacker
looks at the traffic and notices 'how big' the network response is.   This
tells the attacker the compressor has seen already seem the text.  

So imagine, I have a website and I keep session cookies, user names, and 
password crypts in a database.  Imagine the session key is a long 
hexidecimal number.  As an attacker, I could send in the username field 
of the login form guesses of the prefix of somebody else's session key 
I peek at the encrypted traffic from script to the database.  As the 
prefix match length increases the database reply gets shorter.  
Essentially, its a side channel attack that ends up reducing guessing to
a radix search by prefix of all valid session keys.  In this attack, I 
don't have access to the database except through the web form, but I
can see the encrypted database traffic.

It is not a huge vulnerability, but yeah in some use cases if postgresql
used compression it might provide a difficult, but possible route.

Garick


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] proposal - urlencode, urldecode support

[Garick Hamlin]

On Wed, Apr 25, 2012 at 02:41:19PM -0400, Tom Lane wrote:
 Pavel Stehule pavel.steh...@gmail.com writes:
  2012/4/25 Michael Glaesemann g...@seespotcode.net:
  Sounds like a great idea for a PGXN module.
 
  it is one variant - but with support some web technologies - XML,
  JSON, I prefer this in core. Urlcode is one the most used code on
  world now -  implementation is simple - and it can be well integrated
  with decode, encode functions.
 
 Embedding that in encode/decode sounds to me like a pretty horrid idea,
 actually, unless I misunderstand what you are talking about.  URL
 encoding is a text-to-text transformation, no?  If so, it doesn't fit
 into encode/decode, which presume a binary (bytea) decoded form.  People
 would be needing to do entirely bogus text/bytea coercions to use
 such an implementation.

I don't understand the actual proposal here, but urlencoding encodes 
octets as quoted us-ascii.  So, its not really text to text, but
bytes to US-ASCII and US-ASCII to bytes.  AIUI, a unicode character 
has no well specified urlencoding.  A utf-8 encoded unicode character can 
be said to have an urlencoding since we can come up a stream of octets
to urlencode.

Garick

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] JSON for PG 9.2

[Garick Hamlin]

On Fri, Jan 20, 2012 at 09:12:13AM -0800, David E. Wheeler wrote:
 On Jan 19, 2012, at 9:07 PM, Tom Lane wrote:
 
  If his client encoding is UTF8, the value will be letter-perfect JSON
  when it gets to him; and if his client encoding is not UTF8, then he's
  already pretty much decided that he doesn't give a fig about the
  Unicode-centricity of the JSON spec, no?
 
 Don’t entirely agree with this. Some folks are stuck with other encodings and
 cannot change them for one reason or another. That said, they can convert
 JSON from their required encoding into UTF-8 on the client side, so there is
 a workaround.

Perhaps in addition to trying to just 'do the right thing by default',
it makes sense to have a two canonicalization functions?

Say: json_utf8() and json_ascii().

They could give the same output no matter what encoding was set? 

json_utf8 would give nice output where characters were canonicalized to 
native utf8 characters and json_ascii() would output only non-control
ascii characters literally and escape everything else or something
like that?

Garick

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] COUNT(*) and index-only scans

[Garick Hamlin]

On Wed, Oct 12, 2011 at 03:16:54PM +0100, Greg Stark wrote:
 On Wed, Oct 12, 2011 at 2:52 PM, Tom Lane t...@sss.pgh.pa.us wrote:
  I think it's overkill, and possibly unpleasantly unstable as well.
  For the initial attack on this we should just have VACUUM and ANALYZE
  count the number of all-visible blocks and store that in pg_class along
  with the tuple-count statistics.  There's no reason to think that this
  will be any worse than the way we deal with dead tuple counts, for
  instance.
 
 So I have a theory.
 
 Assuming you're in a steady-state situation the amount of all-visible
 blocks will fluctuate from a high just after vacuum to a low just
 before the next vacuum. There are other ways a block can be marked
 all-visible but for the most part I would expect the fraction to go
 steadily down until vacuum comes along and cleans things up.
 
 So if vacuum tracked the fraction of blocks marked all-visible
 *before* it processed them and the fraction it marked all-visible
 after processing we have an upper and lower bound. If we knew how long
 it's been since vacuum we could interpolate between those, or we could
 just take the mean, or we could take the lower bound as a conservative
 estimate.
 
  What I suggest as a first cut for that is: simply derate the visibility 
  fraction as the fraction
  of the table expected to be scanned gets smaller.
 
 I think there's a statistically more rigorous way of accomplishing the
 same thing. If you treat the pages we estimate we're going to read as
 a random sample of the population of pages then your expected value is
 the fraction of the overall population that is all-visible but your
 95th percentile confidence interval will be, uh, a simple formula we
 can compute but I don't recall off-hand.

Incidentally, I had a similar idea at PGCon relating to planning...

My idea was to compute not just the cost but the sensitivity
of the cost an estimate for each plan.   The sensitivity is the 
derivate of the cost.  So, if the total cost was n^2 the sensitivity 
would be 2n.

If you picked a tolerance (like 2 standard deviations) of the 
observed distribution.  You could compare the expected cost and the 
expected 'unlucky cost' for plans.  

(Basically, this is parametric error propagation) 

I know very little about the planner...
I don't know how how often it would lead to picking a better
plan (it might not be worth the cost to compute), but it seemed
like an interesting approach to me.

Garick


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] 9.1beta2 / UNLOGGED + CHECK + INHERITS

[Garick Hamlin]

I wanted to see how much faster unlogged tables might be for an
app I have, so as a quick test I did:

s/CREATE TABLE/CREATE UNLOGGED TABLE/ to get some numbers.
Which lead to a crash.

Here is a trimmed down test case:
$ cat  unlog-test.sql
CREATE UNLOGGED TABLE leases (
mac macaddr NOT NULL,
ip inet NOT NULL,
start_ts timestamp with time zone NOT NULL,
end_ts timestamp with time zone NOT NULL,
id bigint NOT NULL,
truncated integer,
router_ip inet,
CONSTRAINT leases_check CHECK ((start_ts  end_ts))
);
CREATE UNLOGGED TABLE closed_leases (
)
INHERITS (leases);
CREATE UNLOGGED TABLE open_leases (
)
INHERITS (leases);
^D

$ psql91 -U postgres postgres -c '\i unlog-test.sql'
CREATE TABLE
psql91:unlog-test.sql:13: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
psql91:unlog-test.sql:13: connection to server was lost

The CHECK CONSTRAINT on leases is required to cause the crash on 
creating closed_leases.

Garick



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] One Role, Two Passwords

[Garick Hamlin]

On Thu, Jan 20, 2011 at 08:55:42PM -0500, Daniel Farina wrote:
 The problem is that running a reliable,
 centralized LDAP service is not justifiable as compared to role
 mangling on a per-node level, and the role mangling seems has some
 shortcomings that are negotiable with gritted teeth.

Wandering slightly off-topic here...

I often find RADIUS is an easier way to setup a highly available 
(and perhaps too flexible) AuthN/AuthZ service, but I've never used 
PG's RADIUS implementation.   I was curious and I took a look at 
the documentation...

I can't tell if PG supports querying a secondary RADIUS server?  

I don't see how I would do it with the syntax here ... 
http://developer.postgresql.org/pgdocs/postgres/auth-methods.html

Are multiple servers not supported?

Garick

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Streaming base backups

[Garick Hamlin]

On Mon, Jan 10, 2011 at 09:09:28AM -0500, Magnus Hagander wrote:
 On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
 cedric.villemain.deb...@gmail.com wrote:
  2011/1/7 Magnus Hagander mag...@hagander.net:
  On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
  cedric.villemain.deb...@gmail.com wrote:
  2011/1/5 Magnus Hagander mag...@hagander.net:
  On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine dimi...@2ndquadrant.fr 
  wrote:
  Magnus Hagander mag...@hagander.net writes:
  * Stefan mentiond it might be useful to put some
  posix_fadvise(POSIX_FADV_DONTNEED)
    in the process that streams all the files out. Seems useful, as long 
  as that
    doesn't kick them out of the cache *completely*, for other backends 
  as well.
    Do we know if that is the case?
 
  Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
  not already in SHM?
 
  I think that's way more complex than we want to go here.
 
 
  DONTNEED will remove the block from OS buffer everytime.
 
  Then we definitely don't want to use it - because some other backend
  might well want the file. Better leave it up to the standard logic in
  the kernel.
 
  Looking at the patch, it is (very) easy to add the support for that in
  basebackup.c
  That supposed allowing mincore(), so mmap(), and so probably switch
  the fopen() to an open() (or add an open() just for mmap
  requirement...)
 
  Let's go ?
 
 Per above, I still don't think we *should* do this. We don't want to
 kick things out of the cache underneath other backends, and since we
 can't control that. Either way, it shouldn't happen in the beginning,
 and if it does, should be backed with proper benchmarks.

Another option that occurs to me is an option to use direct IO (or another
means as needed) to bypass the cache.  So rather than kicking it out of 
the cache, we attempt just not to pollute the cache by bypassing it for cold
pages and use either normal io for 'hot pages', or use a 'read()' to heat 
the cache afterward.

Garick

 
 I've committed the backend side of this, without that. Still working
 on the client, and on cleaning up Heikki's patch for grammar/parser
 support.
 
 -- 
  Magnus Hagander
  Me: http://www.hagander.net/
  Work: http://www.redpill-linpro.com/
 
 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Streaming base backups

[Garick Hamlin]

On Tue, Jan 11, 2011 at 11:39:20AM -0500, Cédric Villemain wrote:
 2011/1/11 Garick Hamlin gham...@isc.upenn.edu:
  On Mon, Jan 10, 2011 at 09:09:28AM -0500, Magnus Hagander wrote:
  On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
  cedric.villemain.deb...@gmail.com wrote:
   2011/1/7 Magnus Hagander mag...@hagander.net:
   On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
   cedric.villemain.deb...@gmail.com wrote:
   2011/1/5 Magnus Hagander mag...@hagander.net:
   On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine 
   dimi...@2ndquadrant.fr wrote:
   Magnus Hagander mag...@hagander.net writes:
   * Stefan mentiond it might be useful to put some
   posix_fadvise(POSIX_FADV_DONTNEED)
     in the process that streams all the files out. Seems useful, as 
   long as that
     doesn't kick them out of the cache *completely*, for other 
   backends as well.
     Do we know if that is the case?
  
   Maybe have a look at pgfincore to only tag DONTNEED for blocks that 
   are
   not already in SHM?
  
   I think that's way more complex than we want to go here.
  
  
   DONTNEED will remove the block from OS buffer everytime.
  
   Then we definitely don't want to use it - because some other backend
   might well want the file. Better leave it up to the standard logic in
   the kernel.
  
   Looking at the patch, it is (very) easy to add the support for that in
   basebackup.c
   That supposed allowing mincore(), so mmap(), and so probably switch
   the fopen() to an open() (or add an open() just for mmap
   requirement...)
  
   Let's go ?
 
  Per above, I still don't think we *should* do this. We don't want to
  kick things out of the cache underneath other backends, and since we
  can't control that. Either way, it shouldn't happen in the beginning,
  and if it does, should be backed with proper benchmarks.
 
  Another option that occurs to me is an option to use direct IO (or another
  means as needed) to bypass the cache.  So rather than kicking it out of
  the cache, we attempt just not to pollute the cache by bypassing it for cold
  pages and use either normal io for 'hot pages', or use a 'read()' to heat
  the cache afterward.
 
 AFAIR, even Linus is rejecting the idea to use it seriously, except if
 I shuffle in my memory.

Direct IO is generally a pain.

POSIX_FADV_NOREUSE is an alternative (I think).  Realistically I wasn't sure 
which
way(s) actually worked.  My gut was that direct io would likely work right on 
Linux
and Solaris, at least.  If POSIX_FADV_NOREUSE works than maybe that is the 
answer
instead, but I haven't tested either.

Garick


 
 
  Garick
 
 
  I've committed the backend side of this, without that. Still working
  on the client, and on cleaning up Heikki's patch for grammar/parser
  support.
 
  --
   Magnus Hagander
   Me: http://www.hagander.net/
   Work: http://www.redpill-linpro.com/
 
  --
  Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
  To make changes to your subscription:
  http://www.postgresql.org/mailpref/pgsql-hackers
 
 
 
 
 -- 
 Cédric Villemain               2ndQuadrant
 http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Streaming base backups

[Garick Hamlin]

On Tue, Jan 11, 2011 at 12:45:02PM -0500, Tom Lane wrote:
 Florian Pflug f...@phlo.org writes:
  On Jan11, 2011, at 18:09 , Garick Hamlin wrote:
  My gut was that direct io would likely work right on Linux
  and Solaris, at least.
 
  Didn't we discover recently that O_DIRECT fails for ext4 on linux
  if ordered=data, or something like that?
 
 Quite.  Blithe assertions that something like this should work aren't
 worth the electrons they're written on.

Indeed.  I wasn't making such a claim in case that wasn't clear.  I believe,
in fact, there is no single way that will work everywhere.  This isn't
needed for correctness of course, it is merely a tweak for performance as
long as the 'not working case' on platform + filesystem X case degrades to
something close to what would have happened if we didn't try.  I expected
POSIX_FADV_NOREUSE not to work on Linux, but haven't looked at it recently
and not all systems are Linux so I mentioned it.  This was why I thought
direct io might be more realistic.

I did not have a chance to test before I wrote this email so I attempted to 
make my uncertainty clear.  I _know_ it will not work in some environments,
but I thought it was worth looking at if it worked on more than one sane 
common setup, but I can understand if you feel differently about that.

Garick

 
   regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Streaming base backups

[Garick Hamlin]

On Thu, Jan 06, 2011 at 07:47:39PM -0500, Cédric Villemain wrote:
 2011/1/5 Magnus Hagander mag...@hagander.net:
  On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine dimi...@2ndquadrant.fr 
  wrote:
  Magnus Hagander mag...@hagander.net writes:
  * Stefan mentiond it might be useful to put some
  posix_fadvise(POSIX_FADV_DONTNEED)
    in the process that streams all the files out. Seems useful, as long as 
  that
    doesn't kick them out of the cache *completely*, for other backends as 
  well.
    Do we know if that is the case?
 
  Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
  not already in SHM?
 
  I think that's way more complex than we want to go here.
 
 
 DONTNEED will remove the block from OS buffer everytime.
 
 It should not be that hard to implement a snapshot(it needs mincore())
 and to restore previous state. I don't know how basebackup is
 performed exactly...so perhaps I am wrong.
 
 posix_fadvise support is already in postgresql core...we can start by
 just doing a snapshot of the files before starting, or at some point
 in the basebackup, it will need only 256kB per GB of data...

It is actually possible to be more scalable than the simple solution you
outline here (although that solution works pretty well).  

I've written a program that syncronizes the OS cache state using
mmap()/mincore() between two computers.  It haven't actually tested its
impact on performance yet, but I was surprised by how fast it actually runs
and how compact cache maps can be.

If one encodes the data so one remembers the number of zeros between 1s 
one, storage scale by the amount of memory in each size rather than the 
dataset size.  I actually played with doing that, then doing huffman 
encoding of that.  I get around 1.2-1.3 bits / page of _physical memory_ 
on my tests.

I don't have my notes handy, but here are some numbers from memory...

The obvious worst cases are 1 bit per page of _dataset_ or 19 bits per page
of physical memory in the machine.  The latter limit get better, however,
since there are  1024 symbols possible for the encoder (since in this 
case symbols are spans of zeros that need to fit in a file that is 1 GB in
size).  So is actually real worst case is much closer to 1 bit per page of 
the dataset or ~10 bits per page of physical memory.  The real performance
I see with huffman is more like 1.3 bits per page of physical memory.  All the 
encoding decoding is actually very fast.  zlib would actually compress even 
better than huffman, but huffman encoder/decoder is actually pretty good and
very straightforward code.

I would like to integrate something like this into PG or perhaps even into
something like rsync, but its was written as proof of concept and I haven't 
had time work on it recently.

Garick

 -- 
 Cédric Villemain               2ndQuadrant
 http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support
 
 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Streaming base backups

[Garick Hamlin]

On Fri, Jan 07, 2011 at 10:26:29AM -0500, Garick Hamlin wrote:
 On Thu, Jan 06, 2011 at 07:47:39PM -0500, Cédric Villemain wrote:
  2011/1/5 Magnus Hagander mag...@hagander.net:
   On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine dimi...@2ndquadrant.fr 
   wrote:
   Magnus Hagander mag...@hagander.net writes:
   * Stefan mentiond it might be useful to put some
   posix_fadvise(POSIX_FADV_DONTNEED)
     in the process that streams all the files out. Seems useful, as long 
   as that
     doesn't kick them out of the cache *completely*, for other backends 
   as well.
     Do we know if that is the case?
  
   Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
   not already in SHM?
  
   I think that's way more complex than we want to go here.
  
  
  DONTNEED will remove the block from OS buffer everytime.
  
  It should not be that hard to implement a snapshot(it needs mincore())
  and to restore previous state. I don't know how basebackup is
  performed exactly...so perhaps I am wrong.
  
  posix_fadvise support is already in postgresql core...we can start by
  just doing a snapshot of the files before starting, or at some point
  in the basebackup, it will need only 256kB per GB of data...
 
 It is actually possible to be more scalable than the simple solution you
 outline here (although that solution works pretty well).  
 
 I've written a program that syncronizes the OS cache state using
 mmap()/mincore() between two computers.  It haven't actually tested its
 impact on performance yet, but I was surprised by how fast it actually runs
 and how compact cache maps can be.
 
 If one encodes the data so one remembers the number of zeros between 1s 
 one, storage scale by the amount of memory in each size rather than the 

Sorry for the typos, that should read:

the storage scales by the number of pages resident in memory rather than the 
total dataset size.

 dataset size.  I actually played with doing that, then doing huffman 
 encoding of that.  I get around 1.2-1.3 bits / page of _physical memory_ 
 on my tests.
 
 I don't have my notes handy, but here are some numbers from memory...
 
 The obvious worst cases are 1 bit per page of _dataset_ or 19 bits per page
 of physical memory in the machine.  The latter limit get better, however,
 since there are  1024 symbols possible for the encoder (since in this 
 case symbols are spans of zeros that need to fit in a file that is 1 GB in
 size).  So is actually real worst case is much closer to 1 bit per page of 
 the dataset or ~10 bits per page of physical memory.  The real performance
 I see with huffman is more like 1.3 bits per page of physical memory.  All 
 the 
 encoding decoding is actually very fast.  zlib would actually compress even 
 better than huffman, but huffman encoder/decoder is actually pretty good and
 very straightforward code.
 
 I would like to integrate something like this into PG or perhaps even into
 something like rsync, but its was written as proof of concept and I haven't 
 had time work on it recently.
 
 Garick
 
  -- 
  Cédric Villemain               2ndQuadrant
  http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support
  
  -- 
  Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
  To make changes to your subscription:
  http://www.postgresql.org/mailpref/pgsql-hackers
 
 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] plperl arginfo

[Garick Hamlin]

On Thu, Oct 28, 2010 at 01:03:24PM -0400, Andrew Dunstan wrote:
 
 
 On 10/28/2010 12:34 PM, Tom Lane wrote:
 
  BTW, maybe we could have the best of both worlds?  Dunno about Perl,
  but in some languages it would be possible to instantiate the hash
  only if it's actually touched.  Passing the data as a hash definitely
  seems to fit with the spirit of things otherwise, so as long as it
  didn't cost cycles when not needed, I'd be in favor of that API.
 
 Maybe, but I think that's getting rather beyond my perlguts-fu. I think 
 we'd need to do that via PERL_MAGIC_tied, but it's new territory for me. 
 Anyone else want to chime in?

Warning, I don't know the plperl, I am just a perl coder.

I do think all the anonymous array are worth worrying about in terms of
performance.

I don't think that tie is necessarily good for performance.  tie() is not 
generally fast.  I think you'd likely be better off writing plain accessors 
or using a function to add type info.

Use an accessor for type information, like this?
$ref-typeof($key)

...
or perhaps use a special function?

add_type_info(\%args);

...
or if you want attibute based syntax sugar for the add_type_info() solution...

my %args : pg_record(add_type_info);

Again, these I don't know the plperl code, so I might be missing something
here.

Garick


 
 cheers
 
 andrew
 
 
 
 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SQL command to edit postgresql.conf, with comments

[Garick Hamlin]

On Wed, Oct 13, 2010 at 12:56:15PM -0400, Tom Lane wrote:
 Robert Haas robertmh...@gmail.com writes:
  But creating a separate file doesn't fix that problem.  It just moves
  it around.  If people will expect comments in postgresql.conf to get
  preserved, then why won't they also expect comments in
  postgresql.conf.auto to get preserved?
 
 Because postgresql.conf.auto will have a nice leading comment telling
 people (1) not to hand-edit the file, (2) if they do so anyway,
 not to expect comments to be preserved, and (3) the place to do manual
 editing of settings is postgresql.conf.
 
  If the answer is because postgresql.conf has always worked that way
  before, then add one more line to the proposed initial contents
  saying it's not true any more.
 
 Sorry, wrong answer.  The objection to this is not whether you tell
 people that you're taking away the ability to keep useful comments
 in postgresql.conf, it's that you're taking away the ability.
 
   regards, tom lane

I like this approach.  I was just wondering if there is a simple 
tweak to this schema to make it work more easily with standbys.  If
there was a GUC that controlled the 'auto filename' and it could expand
something like %h to hostname (or name if we had something like standby 
registration).  This would allow each standby to store its local settings 
in a different location and have something like a unified set of config
files.  

I suppose something like symlinking postgresql.auto hostname.auto on each 
machine might achieve a similar effect...

Garick

 
 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: SQL/MED(FDW) DDL

[Garick Hamlin]

On Wed, Sep 15, 2010 at 10:05:00PM -0400, Tom Lane wrote:
 Mark Kirkwood mark.kirkw...@catalyst.net.nz writes:
  On 16/09/10 13:22, Tom Lane wrote:
  What exactly do those get you that an ordinary index, or at worst an
  index-organized table, doesn't get you?
 
  It is pretty rare to see key value stores vs relational engines 
  discussed without a descent into total foolishiness, but this Wikipedia 
  page looks like a reasonable summary:
  http://en.wikipedia.org/wiki/NoSQL
 
 That doesn't do anything at all to answer my question.  I don't want
 to debate NoSQL versus traditional RDBMS here.  What I asked was:
 given that PG is a traditional RDBMS, what exactly are you hoping
 to accomplish by putting a key-value storage mechanism in it?  And
 if you did, how would that be different from an index-organized table?

One thing it would get is integration with existing infrastructure that
makes up many critical apps.  Many horizontal apps use things like memcached 
or redis to provide a distributed data layer for developing their applications. 
 
Sometimes that becomes the middle layer for an enterprise.  Being able to hook 
into that middle layer is very handy.  Shared login or session information is 
a good example of data that one might want put in a KVS.  Also, many 
enterprises 
are full of different departments, orgs, teams, systems, etc ...  KVS are 
simple 
and limited enough they might make a good choice for standardizing on how to 
share 
data in some places.

Isn't this what SQL/Med is about?

Garick

 
   regards, tom lane
 
 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] libpq should not be using SSL_CTX_set_client_cert_cb

[Garick Hamlin]

On Tue, May 25, 2010 at 10:29:07PM -0400, Tom Lane wrote:
 I've been experimenting with SSL setups involving chains of CA
 certificates, ie, where the server or client cert itself is signed by
 an intermediate CA rather than a trusted root CA.  This appears to work
 well enough on the server side if you configure the server correctly
 (see discussion of bug #5468).  However, libpq is not able to work with
 a client certificate unless that cert is directly signed by a CA that
 the server trusts (ie, one listed directly in the server's root.crt file).
 This is because there is no good way to feed back any intermediate CA
 certs to the server.  The man page for SSL_CTX_set_client_cert_cb says
 in so many words that the client_cert_cb API is maldesigned:

I don't understand the problem (I don't know the code).  

However, requiring intermediate certs be prepopulated in a trust store will 
effectively make use cases like Bridged PKI not practical.  (and think that 
use cases like that will be very interesting as things like various PIV 
Cards projects are implemented).  

InCommon is also rolling out personal and machine certificates soon and 
there are some potentially similar use cases there for educational 
communities.  Their offering is not a bridge (although some users could
join a bridge), but there will likely be many different intermediate CA
created and likely created on a regular basis so the same issue exists.

Ideally, I think the client code should be able to compute a path to an
offered Trust Anchor and present it.  The server should not have to store 
and keep up-to-date the entire bridge's repository to authenticate any user. 
This would be the right way, IMO.

It should be easy for the client to build a path: they have the Trust Anchor 
in their store along with whatever intermediate certs they need otherwise
their cert would be less than useful.  They just need to send that chain.

I am guessing the problem is that validating the presented chain is hard?  
or am I misunderstanding the reason to want to nail down all the 
intermediates ahead of time.  

Again, I am not very familiar with what that code does currently

Garick

 
   BUGS
 
   The client_cert_cb() cannot return a complete certificate chain,
   it can only return one client certificate. If the chain only has
   a length of 2, the root CA certificate may be omitted according
   to the TLS standard and thus a standard conforming answer can be
   sent to the server. For a longer chain, the client must send the
   complete chain (with the option to leave out the root CA
   certificate). This can only be accomplished by either adding the
   intermediate CA certificates into the trusted certificate store
   for the SSL_CTX object (resulting in having to add CA
   certificates that otherwise maybe would not be trusted), or by
   adding the chain certificates using the
   SSL_CTX_add_extra_chain_cert(3) function, which is only
   available for the SSL_CTX object as a whole and that therefore
   probably can only apply for one client certificate, making the
   concept of the callback function (to allow the choice from
   several certificates) questionable.
 
 It strikes me that we could not only fix this case, but make the libpq
 code simpler and more like the backend case, if we got rid of
 client_cert_cb and instead preloaded the ~/.postgresql/postgresql.crt
 file using SSL_CTX_use_certificate_chain_file().  Then, using an
 indirectly signed client cert would only require including the full cert
 chain in that file.
 
 So I'm wondering if there was any specific reason behind using the
 callback API to start with.  Anybody remember?
 
   regards, tom lane
 
 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] libpq should not be using SSL_CTX_set_client_cert_cb

[Garick Hamlin]

On Wed, May 26, 2010 at 10:54:42AM -0400, Tom Lane wrote:
 Garick Hamlin gham...@isc.upenn.edu writes:
  I am guessing the problem is that validating the presented chain is hard?  
 
 No, the problem is that the current libpq code fails to present the
 chain at all.  It will only load and send the first cert in the
 postgresql.crt file.  This works only when the client's cert is signed
 directly by one of the CAs trusted by the server.

Sorry, I just re-read your original message.  You were clear, but I read
it wrong.

This is much less limiting than what I thought was being suggested.  Having 
a user's credentials work with only one trust anchor isn't that bad.  I am 
not familiar enough with openssl to know if there is a specific pitfall to
the change you suggested (which I think was what you were asking)..

One could make it work with multiple TAs in a similar fashion if it also 
checked for the existence of a directory (like: ~/.postgresql/client_ta ) to 
store chains to each supported TA by fingerprint.  

That might not be worth the effort at this point...

Garick

 
   regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] pgbouncer + psql 9.0a4

[Garick Hamlin]

I was just trying out 9.0a4 and I noticed.  That I can't connect to 
pgbouncer with psql from 9.0a4 as a result of the set application_name 
changes to psql.

I imagine this is really mostly pgbouncer's problem, but I couldn't 
figure out how if it was possible to make psql not set application_name 
from the psql man page.  If not, at least from a users point of view psql 
9.0a4 looks incompatible with existing versions of pgbouncer in at least 
in this configuration that works with an older versions of psql.

... or, is that a crazy way to look at it?

Garick

Details are below..

My setup (Solaris 10 / sparc )

[ psql 9.0a4 ] - [ pgbouncer 1.3.1 ] - [ postgres 9.0a4 ]

$ /usr/local/var/postgres/9.0a4/bin/psql -h 127.0.0.1 -p 6543
psql: ERROR:  Unknown startup parameter

..pgbouncer's log..
2010-02-25 21:56:29.721 6979 WARNING C-71d48: (nodb)/(nouser)@127.0.0.1:54000 
unsupported startup parameter: application_name=psql
2010-02-25 21:56:29.721 6979 LOG C-71d48: (nodb)/(nouser)@127.0.0.1:54000 
closing because: Unknown startup parameter (age=0)
2010-02-25 21:56:29.722 6979 LOG C-71d48: (nodb)/(nouser)@127.0.0.1:54000 
Pooler Error: Unknown startup parameter




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pgbouncer + psql 9.0a4

[Garick Hamlin]

On Fri, Feb 26, 2010 at 11:38:20AM -0500, Jaime Casanova wrote:
 On Fri, Feb 26, 2010 at 11:26 AM, Garick Hamlin gham...@isc.upenn.edu wrote:
 
  I imagine this is really mostly pgbouncer's problem
 
 yes it is, but you can workaround it using ignore_startup_parameters
 parameter in pgbouncer.ini
 http://pgbouncer.projects.postgresql.org/doc/config.html#toc4

Thanks!
I am new to pgbouncer, sorry if this was obvious.

Garick
 
 -- 
 Atentamente,
 Jaime Casanova
 Soporte y capacitación de PostgreSQL
 Asesoría y desarrollo de sistemas
 Guayaquil - Ecuador
 Cel. +59387171157

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Make standby server continuously retry restoring the next WAL

[Garick Hamlin]

On Thu, Feb 11, 2010 at 01:22:44PM -0500, Kevin Grittner wrote:
 Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote:
  
  I think 'rsync' has the same problem.
  
 There is a switch you can use to create the problem under rsync, but
 by default rsync copies to a temporary file name and moves the
 completed file to the target name.
  
 -Kevin

I don't use PITR, So I don't know any of the well understood facts about 
PITR with postgres, but my understanding with rsync is ...

It doesn't fsync data before rename, its something like: open / write / 
close / rename, which could lead to zero length files on some filesystems.  
(are there other anomalies to worry about here?)

Would that be a concern?  

Garick

 
 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Status of plperl inter-sp calling

[Garick Hamlin]

On Wed, Jan 06, 2010 at 11:14:38AM -0500, Tom Lane wrote:
 Andrew Dunstan and...@dunslane.net writes:
  Tom Lane wrote:
  spi_rv = SPI_execute(query, current_call_data-prodesc-fn_readonly,
  ^^^
 
  OK, but won't that automatically supply the value from the function 
  called from postgres, which will be the right thing?
 
 My point was that that is exactly the wrong thing.  If I have a function
 declared stable, it must not suddenly start behaving as volatile because
 it was called from a volatile function.  Nor vice versa.
 
 Now as I mentioned upthread, there might be other ways to get the
 correct value of the readonly parameter.  One that comes to mind is
 to somehow attach it to the spi call at compile time, whatever that
 means in the perl world.  But you can't just have it be determined by
 the outermost active function call.

If you want 'a perl compile time hook', those are called attributes.
http://search.cpan.org/~dapm/perl-5.10.1/lib/attributes.pm

You can define attributes to effect how a given syntax compiles.
perl.  

my $var :foo;

or 

sub bar :foo;

The subroutine or variable is compiled in a way defined by the
':foo' attribute.

This might be a clean way around the type dispatch issues
as well.  One could include the invokant type information in the 
perl declaration.  

sub sp_something :pg_sp ('bigint bigint');

sp_something (12,0);

Anyway, that looks like a nice interface to me...

Although, I don't understand the Pg internals problem faced here
so ... I'm not sure my suggestion is helpful.


Garick
 
   regards, tom lane
 
 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Get rid of the need for manual maintenance of the initial

[Garick Hamlin]

On Tue, Jan 05, 2010 at 02:02:51PM -0500, Tom Lane wrote:
 Stefan Kaltenbrunner ste...@kaltenbrunner.cc writes:
  did that and it seems the problem is in the loop that does:
 
  foreach my $row (@$data)
  {
 
   # To construct fmgroids.h and fmgrtab.c, we need to inspect some
   # of the individual data fields.  Just splitting on whitespace
 
 Huh.  It's weird that I don't see a leak in either 5.8.7 or 5.10.1,
 which are the two closest perl versions I have handy here.  I think
 this may be a platform-specific Perl bug.  Still, it would be nice
 to work around it if we can.
 
 I'm not nearly good enough in Perl to be sure about the semantics
 of this loop.  Is it possible that it's changing the global contents
 of the @$data structure, rather than just hacking a local copy of
 each row before pushing some values into @fmgr?
Yes, that is what would happen.  
I have not read this script at all but that is how perl works...

$row is an aliased to each scalar in the list '(@$data)'

There is no copying.  Copying happens on list assignment, 
but not with for/foreach.

If you wanted a copy you need something like:
foreach my $row (@{[ @$data ]}) {

for a shallow copy.

Is that a problem? (I haven't read the script)

Garick

 
   regards, tom lane
 
 -- 
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Solaris ident authentication using unix domain sockets

[Garick Hamlin]

Hi,
I have a patch that I have been using to support postgresql's
notion of ident authentication when using unix domain sockets on
Solaris.  This patch basically just adds support for using 
getupeercred() on Solaris so unix sockets and ident auth works just
like it does on Linux and elsewhere.

This was my first attempt wrestling with automake.  I've 
tested it builds properly after it is applied and autoreconf is run
on RHEL4/Linux/x86.  I am using the patch currently on Solaris 10 / 
x86.

Garick

diff -cr postgresql_CVS/configure.in postgresql/configure.in
*** postgresql_CVS/configure.in Tue Jun 24 15:52:30 2008
--- postgresql/configure.in Tue Jun 24 15:57:22 2008
***
*** 1095,1101 
  AC_FUNC_ACCEPT_ARGTYPES
  PGAC_FUNC_GETTIMEOFDAY_1ARG
  
! AC_CHECK_FUNCS([cbrt dlopen fcvt fdatasync getpeereid getrlimit memmove poll 
pstat readlink setproctitle setsid sigprocmask symlink sysconf towlower utime 
utimes waitpid wcstombs])
  
  AC_CHECK_DECLS(fdatasync, [], [], [#include unistd.h])
  AC_CHECK_DECLS(posix_fadvise, [], [], [#include fcntl.h])
--- 1095,1101 
  AC_FUNC_ACCEPT_ARGTYPES
  PGAC_FUNC_GETTIMEOFDAY_1ARG
  
! AC_CHECK_FUNCS([getpeerucred cbrt dlopen fcvt fdatasync getpeereid getrlimit 
memmove poll pstat readlink setproctitle setsid sigprocmask symlink sysconf 
towlower utime utimes waitpid wcstombs])
  
  AC_CHECK_DECLS(fdatasync, [], [], [#include unistd.h])
  AC_CHECK_DECLS(posix_fadvise, [], [], [#include fcntl.h])
diff -cr postgresql_CVS/src/backend/libpq/hba.c 
postgresql/src/backend/libpq/hba.c
*** postgresql_CVS/src/backend/libpq/hba.c  Tue Jun 24 15:52:32 2008
--- postgresql/src/backend/libpq/hba.c  Tue Jun 24 15:53:00 2008
***
*** 25,30 
--- 25,33 
  #include sys/uio.h
  #include sys/ucred.h
  #endif
+ #if defined(HAVE_GETPEERUCRED) 
+ #include ucred.h
+ #endif
  #include netinet/in.h
  #include arpa/inet.h
  #include unistd.h
***
*** 1500,1505 
--- 1503,1539 
strlcpy(ident_user, pass-pw_name, IDENT_USERNAME_MAX + 1);
  
return true;
+ #elif defined(HAVE_GETPEERUCRED) /* Solaris  10 */
+   uid_t   uid;
+   gid_t   gid;
+   struct passwd   *pass;
+   int ucred_ok=1;
+   ucred_t *ucred = NULL;
+   if (getpeerucred(sock, ucred) == -1)
+   ucred_ok = 0;
+   if (ucred_ok  (uid = ucred_geteuid(ucred)) == -1 )
+   ucred_ok = 0;
+   if (ucred_ok  (gid = ucred_getrgid(ucred)) == -1 )
+   ucred_ok = 0;
+   if (ucred)
+   ucred_free(ucred);
+   if (!ucred_ok) {
+   /* We didn't get a valid credentials struct. */
+   ereport(LOG, (
+could not get peer credentials: %s,
+   strerror(errno)));
+   return false;
+   }
+   pass = getpwuid(uid);
+   if (pass == NULL)
+   {
+   ereport(LOG,
+   (errmsg(local user with ID %d does not exist,
+   (int) uid)));
+   return false;
+   }
+   strlcpy(ident_user, pass-pw_name, IDENT_USERNAME_MAX + 1);
+   return true;
  #elif defined(HAVE_STRUCT_CMSGCRED) || defined(HAVE_STRUCT_FCRED) || 
(defined(HAVE_STRUCT_SOCKCRED)  defined(LOCAL_CREDS))
struct msghdr msg;
  

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Solaris ident authentication using unix domain sockets

[Garick Hamlin]

On Thu, Jul 03, 2008 at 02:01:22PM -0400, Tom Lane wrote:
 Garick Hamlin [EMAIL PROTECTED] writes:
I have a patch that I have been using to support postgresql's
  notion of ident authentication when using unix domain sockets on
  Solaris.  This patch basically just adds support for using
  getupeercred() on Solaris so unix sockets and ident auth works just
  like it does on Linux and elsewhere.
 
 Cool.
 
  + #if defined(HAVE_GETPEERUCRED)
  + #include ucred.h
  + #endif
 
 But this is not cool.  There might be systems out there that have
 getpeerucred() but not ucred.h, and this coding would cause a compile
 failure (even if they actually wouldn't be trying to use getpeerucred()
 because they have some other way to do it).  You need an explicit
 configure probe for the header file too, I think.
Ok, I can fix that.
 
 Also, what is the rationale for putting this before the
 HAVE_STRUCT_CMSGCRED case instead of after?  Again, that seems like it
 could cause unexpected behavioral changes on platforms that work fine
 now (consider possibility that getpeerucred is there but broken).
Good Point, It should be the other way.
 
 regards, tom lane

Thanks,

Garick

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers