Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Terry Laurenzo
Good points. In addition, any binary format needs to support object property traversal without having to do a deep scan of all descendants. BSON handles this with explicit lengths for document types (objects and arrays) so that entire parts of the tree can be skipped during sibling traversal. It

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Itagaki Takahiro
On Wed, Oct 20, 2010 at 6:39 AM, Terry Laurenzo wrote: > The answer may be to have both a jsontext and jsonbinary type as each will > be optimized for a different case. I want to choose one format for JSON rather than having two types. It should be more efficient than other format in many cases,

Re: [HACKERS] max_wal_senders must die

2010-10-19 Thread Greg Smith
Josh Berkus wrote: Well, now that you mention it, I also think that "hot standby" should be the default. Yes, I know about the overhead, but I also think that the number of our users who want easy replication *far* outnumber the users who care about an extra 10% WAL overhead. I think this w

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Itagaki Takahiro
Sorry. I missed v5 patch and other new ones. Some of the issues might have been already fixed in the new version. On Wed, Oct 20, 2010 at 12:19 PM, Itagaki Takahiro wrote: > On Tue, Oct 19, 2010 at 9:33 PM, Dimitri Fontaine >> Fixed in v4, attached. > > Source code > * It still has a wa

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Itagaki Takahiro
On Wed, Oct 20, 2010 at 12:58 PM, Alvaro Herrera wrote: >> * xml2 module has a different extension name from the directory name. >> The directory name is 'xml2', but the extension name is 'pgxml'. > > Lets rename the directory. Hmmm, but we call it 'xml2' in the doc. There is no 'pgxml' at all in

Re: [HACKERS] PostgreSQL and HugePage

2010-10-19 Thread daveg
On Wed, Oct 20, 2010 at 04:08:37PM +1300, Mark Kirkwood wrote: > On 20/10/10 16:05, Mark Kirkwood wrote: > > > > > >shmget and friends are hugetlbpage aware, so it seems it should 'just > >work'. > > > > Heh - provided you specify > > SHM_HUGETLB > > > in the relevant call that is :-) I had

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Alvaro Herrera
Excerpts from Itagaki Takahiro's message of mié oct 20 00:19:44 -0300 2010: > * xml2 module has a different extension name from the directory name. > The directory name is 'xml2', but the extension name is 'pgxml'. > I'm not sure whether we should change the names. Or, merging xml2 module > into c

Re: [HACKERS] ECPG regression tests need .gitignore update?

2010-10-19 Thread Tom Lane
Robert Haas writes: > On Tue, Oct 19, 2010 at 11:27 PM, Tom Lane wrote: >> This seems like a build-process bug, not a .gitignore oversight. >> Those shouldn't be there at all (and I notice "make clean" doesn't >> get rid of them). > I don't even know what they are. "Outboard" debug symbols. So

Re: [HACKERS] ECPG regression tests need .gitignore update?

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 11:27 PM, Tom Lane wrote: > Robert Haas writes: >> After running "make installcheck-world", git status shows a bunch of >> stuff that looks like this: > >> #   src/interfaces/ecpg/test/sql/desc.dSYM/ >> #   src/interfaces/ecpg/test/sql/describe.dSYM/ >> #   src/interfaces/

Re: [HACKERS] ECPG regression tests need .gitignore update?

2010-10-19 Thread Tom Lane
Robert Haas writes: > After running "make installcheck-world", git status shows a bunch of > stuff that looks like this: > # src/interfaces/ecpg/test/sql/desc.dSYM/ > # src/interfaces/ecpg/test/sql/describe.dSYM/ > # src/interfaces/ecpg/test/sql/dynalloc.dSYM/ > # src/interfaces/ecpg/test

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Itagaki Takahiro
On Tue, Oct 19, 2010 at 9:33 PM, Dimitri Fontaine wrote: > Fixed in v4, attached. It works as expected in basic functionality, so I pick up edge case issues. Some of them might be a problem for the patch; They might come from our non-standardized module design. Source code * It still h

Re: [HACKERS] security hook on authorization

2010-10-19 Thread Robert Haas
On Wed, Oct 13, 2010 at 2:13 AM, Robert Haas wrote: > 2010/8/24 KaiGai Kohei : >> I tried to revise the patch. It allows plugins to get control next to >> client authentication, but before returning the status to users. >> >> This change enables plugins which should be invoked on authentication >>

Re: [HACKERS] PostgreSQL and HugePage

2010-10-19 Thread Mark Kirkwood
On 20/10/10 16:05, Mark Kirkwood wrote: shmget and friends are hugetlbpage aware, so it seems it should 'just work'. Heh - provided you specify SHM_HUGETLB in the relevant call that is :-) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your s

Re: [HACKERS] PostgreSQL and HugePage

2010-10-19 Thread Mark Kirkwood
On 20/10/10 15:10, Robert Haas wrote: On Tue, Oct 19, 2010 at 8:39 PM, Hsien-Wen Chu wrote: I want to use hugepage function on Linux platform, my question is if PostgreSQL supports hugepage in default, if not, what's the code need to be modified? Unfortunately, I don't think this is

Re: [HACKERS] ISN patch that applies cleanly with git apply

2010-10-19 Thread Robert Haas
On Mon, Oct 18, 2010 at 2:17 PM, Robert Haas wrote: > On Mon, Oct 18, 2010 at 1:57 PM, Tom Lane wrote: >> Robert Haas writes: >>> I may be in the minority here, but I'm inclined to just apply this and >>> move on. >> >> FWIW, I agree with applying the code patch as-is, but I think we need to >>

[HACKERS] Re: [COMMITTERS] pgsql: Restructure the pg_upgrade code to use several global structures

2010-10-19 Thread Bruce Momjian
Itagaki Takahiro wrote: > On Wed, Oct 20, 2010 at 6:38 AM, Bruce Momjian wrote: > > Restructure the pg_upgrade code to use several global structures rather > > than packing everything into 'ctx' and passing that to every function. > > --- > > http://git.postgresql.org/gitweb?p=postgresql.git;a

[HACKERS] ECPG regression tests need .gitignore update?

2010-10-19 Thread Robert Haas
After running "make installcheck-world", git status shows a bunch of stuff that looks like this: # src/interfaces/ecpg/test/sql/desc.dSYM/ # src/interfaces/ecpg/test/sql/describe.dSYM/ # src/interfaces/ecpg/test/sql/dynalloc.dSYM/ # src/interfaces/ecpg/test/sql/dynalloc2.dSYM/ # src/inte

Re: [HACKERS] PostgreSQL and HugePage

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 8:39 PM, Hsien-Wen Chu wrote: > I want to use hugepage function on Linux platform, my question is if > PostgreSQL supports hugepage in default, if not, what's the code need to be > modified? Unfortunately, I don't think this is too simple. PostgreSQL uses sysv shared memo

Re: [HACKERS] Domains versus arrays versus typmods

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 9:17 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Oct 19, 2010 at 6:14 PM, Tom Lane wrote: >>> I think that what we ought to do about it is to stop exposing typelem >>> in domains' pg_type rows.  If you want to subscript a domain value, you >>> should have to dril

[HACKERS] Re: [COMMITTERS] pgsql: Restructure the pg_upgrade code to use several global structures

2010-10-19 Thread Itagaki Takahiro
On Wed, Oct 20, 2010 at 6:38 AM, Bruce Momjian wrote: > Restructure the pg_upgrade code to use several global structures rather > than packing everything into 'ctx' and passing that to every function. > --- > http://git.postgresql.org/gitweb?p=postgresql.git;a=commitdiff;h=e13f7e9a711c9117e438

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 9:15 PM, Terri Laurenzo wrote: > I hear ya.  It might be a premature optimization but I still think there may > be benefit for the case of large scale extraction and in- database > transformation of large JSON datastructures.  We have terabytes of this stuff > and I'd li

Re: [HACKERS] leaky views, yet again

2010-10-19 Thread Robert Haas
On Wed, Oct 13, 2010 at 12:52 PM, Tom Lane wrote: > Robert Haas writes: >> On Wed, Oct 13, 2010 at 11:45 AM, Tom Lane wrote: >>> That's all true, but you have to consider how much the obstacle actually >>> gets in their way versus how painful it is on your end to create and >>> maintain the obst

Re: [HACKERS] ask for review of MERGE

2010-10-19 Thread Boxuan Zhai
Hi, I considered the empty source table situation again. I think it is correct to upsert nothing in this case. Back to the original logic of MERGE command, it is main purpose is to add the supplementary data from the source table into the target table. Thus, an empty source table means no input d

Re: [HACKERS] Domains versus arrays versus typmods

2010-10-19 Thread Tom Lane
Robert Haas writes: > On Tue, Oct 19, 2010 at 6:14 PM, Tom Lane wrote: >> I think that what we ought to do about it is to stop exposing typelem >> in domains' pg_type rows.  If you want to subscript a domain value, you >> should have to drill down to its base type with getBaseTypeAndTypmod, >> wh

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Terri Laurenzo
I hear ya. It might be a premature optimization but I still think there may be benefit for the case of large scale extraction and in- database transformation of large JSON datastructures. We have terabytes of this stuff and I'd like something between the hip nosql options and a fully structure

Re: [HACKERS] WIP: extensible enums

2010-10-19 Thread Andrew Dunstan
On 10/19/2010 08:51 PM, Robert Haas wrote: On Tue, Oct 19, 2010 at 5:42 PM, Andrew Dunstan wrote: Well a bit more testing shows some benefit. I've sorted out a few kinks, so this seems to work. In particular, with the above tables, the version imported from 9.0 can create have an index create

Re: [HACKERS] Serializable snapshot isolation patch

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 6:28 PM, Kevin Grittner wrote: > One thing that would work, but I really don't think I like it, is > that a request for a snapshot for such a transaction would not only > block until it could get a "clean" snapshot (no overlapping > serializable non-read-only transactions w

Re: [HACKERS] WIP: extensible enums

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 5:42 PM, Andrew Dunstan wrote: > Well a bit more testing shows some benefit. I've sorted out a few kinks, so > this seems to work. In particular, with the above tables, the version > imported from 9.0 can create have an index created in about the same time as > on the fresh

Re: [HACKERS] Domains versus arrays versus typmods

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 6:14 PM, Tom Lane wrote: > In bug #5717, Richard Huxton complains that a domain declared like >        CREATE DOMAIN mynums numeric(4,2)[1]; > doesn't work properly, ie, the typmod isn't enforced in places where > it reasonably ought to be.  I dug into this a bit, and found

[HACKERS] PostgreSQL and HugePage

2010-10-19 Thread Hsien-Wen Chu
Dear All I want to use hugepage function on Linux platform, my question is if PostgreSQL supports hugepage in default, if not, what's the code need to be modified? Thank you for your greate support Hsien-Wen

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 6:56 PM, Tom Lane wrote: > Greg Stark writes: >> The elephant in the room is if the binary encoded form is smaller then >> it occupies less ram and disk bandwidth to copy it around. > > It seems equally likely that a binary-encoded form could be larger > than the text form

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Alvaro Herrera
Excerpts from David E. Wheeler's message of mar oct 19 16:36:20 -0300 2010: > On Oct 19, 2010, at 12:17 PM, Robert Haas wrote: > > > I think we should take a few steps back and ask why we think that > > binary encoding is the way to go. We store XML as text, for example, > > and I can't remember

Re: [HACKERS] pg_rawdump

2010-10-19 Thread Roberto Mello
On Tue, Oct 19, 2010 at 6:13 PM, Stephen R. van den Berg wrote: > Greg Stark wrote: > >> But I'm not sure how useful. I mean, you can't really decipher >>everything properly without the data in the catalog -- and you have to >>premise this on the idea that you've lost everything in the catalog >>b

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-19 Thread Greg Stark
On Tue, Oct 19, 2010 at 4:12 PM, Tom Lane wrote: > ... and relpages and reltuples ... Those are hardly very convincing examples. Neither are values that need to be updated in transactions and neither relates to the definition of the table. They could be moved from pg_class to some internal data s

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-19 Thread Tom Lane
Greg Stark writes: > On Tue, Oct 19, 2010 at 3:45 PM, Tom Lane wrote: >> Relcache entries alone are not gonna work.  There is way too much stuff >> that assumes that tables are correctly represented in the system >> catalogs. > Well we're talking about multiple things now. In the global temporar

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Terry Laurenzo
On Tue, Oct 19, 2010 at 4:51 PM, Tom Lane wrote: > Terry Laurenzo writes: > > After spending a week in the morass of this, I have to say that I am less > > certain than I was on any front regarding the text/binary distinction. > I'll > > take some time and benchmark different cases. My hypothe

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-19 Thread Greg Stark
On Tue, Oct 19, 2010 at 3:45 PM, Tom Lane wrote: > Relcache entries alone are not gonna work.  There is way too much stuff > that assumes that tables are correctly represented in the system > catalogs. > Well we're talking about multiple things now. In the global temporary table case they *are* p

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Tom Lane
Greg Stark writes: > The elephant in the room is if the binary encoded form is smaller then > it occupies less ram and disk bandwidth to copy it around. It seems equally likely that a binary-encoded form could be larger than the text form (that's often true for our other datatypes). Again, this i

Re: [HACKERS] pg_rawdump

2010-10-19 Thread Tom Lane
"Stephen R. van den Berg" writes: > In order to simplify recovery at this point (enormously), it would > have been very helpful (at almost negligible cost), to have the name > of the table, the name of the columns, and the types of the > columns available. > Why don't we insert that data into the

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Tom Lane
Terry Laurenzo writes: > After spending a week in the morass of this, I have to say that I am less > certain than I was on any front regarding the text/binary distinction. I'll > take some time and benchmark different cases. My hypothesis is that a well > implemented binary structure and convers

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-19 Thread Tom Lane
Greg Stark writes: > On Tue, Oct 19, 2010 at 12:03 PM, Robert Haas wrote: >> The trick is that it would require us to have two pg_class tables, two >> pg_attribute tables, two pg_attrdef tables, etc.: in each case, one >> permanent and one temporary.  I am not sure how complex that will turn >> o

Re: [HACKERS] Serializable snapshot isolation patch

2010-10-19 Thread Kevin Grittner
Jeff Davis wrote: > I briefly looked into this when I woke up this morning, and I > think I'm close. I can reproduce it every time, so I should be > able to fix this as soon as I can find some free time (tomorrow > night, probably). OK, I'll focus on other areas. > I might also be able to he

[HACKERS] Domains versus arrays versus typmods

2010-10-19 Thread Tom Lane
In bug #5717, Richard Huxton complains that a domain declared like CREATE DOMAIN mynums numeric(4,2)[1]; doesn't work properly, ie, the typmod isn't enforced in places where it reasonably ought to be. I dug into this a bit, and found that there are more worms in this can than I first expec

Re: [HACKERS] pg_rawdump

2010-10-19 Thread Stephen R. van den Berg
Greg Stark wrote: >On Tue, Oct 19, 2010 at 1:12 PM, Stephen R. van den Berg wrote: >> In order to simplify recovery at this point (enormously), it would >> have been very helpful (at almost negligible cost), to have the name >> of the table, the name of the columns, and the types of the >> columns

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Pavel Stehule
2010/10/19 Greg Stark : > On Tue, Oct 19, 2010 at 12:17 PM, Robert Haas wrote: >> I think we should take a few steps back and ask why we think that >> binary encoding is the way to go.  We store XML as text, for example, >> and I can't remember any complaints about that on -bugs or >> -performance

Re: [HACKERS] pg_rawdump

2010-10-19 Thread Greg Stark
On Tue, Oct 19, 2010 at 1:12 PM, Stephen R. van den Berg wrote: > In order to simplify recovery at this point (enormously), it would > have been very helpful (at almost negligible cost), to have the name > of the table, the name of the columns, and the types of the > columns available. > > Why don

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Pavel Stehule
> > After spending a week in the morass of this, I have to say that I am less > certain than I was on any front regarding the text/binary distinction.  I'll > take some time and benchmark different cases.  My hypothesis is that a well > implemented binary structure and conversions will add minimal

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Greg Stark
On Tue, Oct 19, 2010 at 12:17 PM, Robert Haas wrote: > I think we should take a few steps back and ask why we think that > binary encoding is the way to go.  We store XML as text, for example, > and I can't remember any complaints about that on -bugs or > -performance, so why do we think JSON will

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-19 Thread Pavel Stehule
2010/10/19 Greg Stark : > On Tue, Oct 19, 2010 at 12:03 PM, Robert Haas wrote: >> The trick is that it would require us to have two pg_class tables, two >> pg_attribute tables, two pg_attrdef tables, etc.: in each case, one >> permanent and one temporary.  I am not sure how complex that will turn

[HACKERS] pg_rawdump

2010-10-19 Thread Stephen R. van den Berg
I spent yesterday writing a new tool pg_rawdump (which will be released as open source in due course), which takes the table files in an arbitrary pgsql database, and is able to transform those back into tables (including toast values). In the course of doing this (a customer needed it because he

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Terry Laurenzo
On Tue, Oct 19, 2010 at 2:46 PM, Robert Haas wrote: > On Tue, Oct 19, 2010 at 3:40 PM, Joseph Adams > wrote: > > On Tue, Oct 19, 2010 at 3:17 PM, Robert Haas > wrote: > >> I think we should take a few steps back and ask why we think that > >> binary encoding is the way to go. We store XML as t

Re: [HACKERS] gist DatumGetPointer returns pointer to corrupted data

2010-10-19 Thread Marios Vodas
On Tue, Oct 19, 2010 at 10:23 PM, Robert Haas wrote: > Is pg_type.typlen set correctly? > Are you refering to table pg_type? If yes, those type structures exist only in c I didn't write any in-out functions, so they don't exist in sql level. I pass a different data type (which exists in sql) to

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-19 Thread Greg Stark
On Tue, Oct 19, 2010 at 12:03 PM, Robert Haas wrote: > The trick is that it would require us to have two pg_class tables, two > pg_attribute tables, two pg_attrdef tables, etc.: in each case, one > permanent and one temporary.  I am not sure how complex that will turn > out to be. Tom suggested u

Re: [HACKERS] max_wal_senders must die

2010-10-19 Thread Josh Berkus
>> Yes. This isn't just a numeric parameter; it's also a boolean >> indicating "do I want to pay the overhead to be prepared to be a >> replication master?". Since this is the first time I've heard of the overhead, it would be hard for me to have taken that into consideration. If there was di

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 3:40 PM, Joseph Adams wrote: > On Tue, Oct 19, 2010 at 3:17 PM, Robert Haas wrote: >> I think we should take a few steps back and ask why we think that >> binary encoding is the way to go.  We store XML as text, for example, >> and I can't remember any complaints about tha

Re: [HACKERS] max_wal_senders must die

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 3:32 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Oct 19, 2010 at 12:18 PM, Josh Berkus wrote: >>> On 10/19/2010 09:06 AM, Greg Smith wrote: I think Magnus's idea to bump the default to 5 triages the worst of the annoyance here, without dropping the feat

Re: [HACKERS] knngist plans

2010-10-19 Thread Oleg Bartunov
On Tue, 19 Oct 2010, Marios Vodas wrote: On Sat, Oct 16, 2010 at 8:42 PM, Oleg Bartunov wrote: Marios, you're right. There are several reasons for poor documentation, but of course, no excuse, we do need good docs any way ! It's very nice you're willing to write one, since it's always better

Re: [HACKERS] max_wal_senders must die

2010-10-19 Thread Rob Wultsch
On Tue, Oct 19, 2010 at 12:32 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Oct 19, 2010 at 12:18 PM, Josh Berkus wrote: >>> On 10/19/2010 09:06 AM, Greg Smith wrote: I think Magnus's idea to bump the default to 5 triages the worst of the annoyance here, without dropping the fea

Re: [HACKERS] plan time of MASSIVE partitioning ...

2010-10-19 Thread Tom Lane
Boszormenyi Zoltan writes: > The problem is with the two functions in path/equivclass.c, > as process_equivalance() and those functions are all walk > the tree, and the current RBTree code can only deal with > one walk at a time. We need to push/pop the iterator state > to be able to serve more th

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Alvaro Herrera
Excerpts from Dimitri Fontaine's message of mar oct 19 13:09:47 -0300 2010: > Tom Lane writes: > > You could argue that either way I guess. The script knows what it > > needs, but OTOH just about every extension there is will probably > > be generating useless NOTICEs unless something is done, so

Re: [HACKERS] plan time of MASSIVE partitioning ...

2010-10-19 Thread Boszormenyi Zoltan
Hi, Boszormenyi Zoltan írta: > There is one problem with the patch, it doesn't survive > "make check". One of the regression tests fails the > Assert(!cur_em->em_is_child); > line in process_equivalence() in equivclass.c, but I couldn't > yet find it what causes it. The "why" is vaguely clear:

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Joseph Adams
On Tue, Oct 19, 2010 at 11:22 AM, Terry Laurenzo wrote: > Perhaps we should enumerate the attributes of what would make a good binary > encoding? Not sure if we're discussing the internal storage format or the binary send/recv format, but in my humble opinion, some attributes of a good internal f

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread David E. Wheeler
On Oct 19, 2010, at 12:17 PM, Robert Haas wrote: > I think we should take a few steps back and ask why we think that > binary encoding is the way to go. We store XML as text, for example, > and I can't remember any complaints about that on -bugs or > -performance, so why do we think JSON will be

Re: [HACKERS] max_wal_senders must die

2010-10-19 Thread Tom Lane
Robert Haas writes: > On Tue, Oct 19, 2010 at 12:18 PM, Josh Berkus wrote: >> On 10/19/2010 09:06 AM, Greg Smith wrote: >>> I think Magnus's idea to bump the default to 5 triages the worst of the >>> annoyance here, without dropping the feature (which has uses) or waiting >>> for new development

Re: [HACKERS] gist DatumGetPointer returns pointer to corrupted data

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 1:46 PM, Marios Vodas wrote: > I index these structures in gist: > >> typedef struct { >>     uint8 type_flag; >>     float8 xi; >>     float8 yi; >>     Timestamp ti; >>     float8 xe; >>     float8 ye; >>     Timestamp te; >>     int32 id; >> } typ_s_flagged; >> >> typede

Re: [HACKERS] Issues with two-server Synch Rep

2010-10-19 Thread Simon Riggs
On Tue, 2010-10-19 at 09:36 -0700, Josh Berkus wrote: > >> Absolutely. For a synch standby, you can't tolerate any standby delay > >> at all. This means that anywhere from 1/4 to 3/4 of queries on the > >> standby would be cancelled on any high-traffic OLTP server. Hence, > >> "useless". > > > >

Re: [HACKERS] knngist - 0.8

2010-10-19 Thread Robert Haas
2010/10/19 Teodor Sigaev : >>> Thinking about it that way, perhaps we could add an integer column >>> amop_whats_it_good_for that gets used as a bit field.  That wouldn't >>> require changing the index structure, although it might break some >>> other things. > >> OPERATOR strategy_number ( op_type

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Tom Lane
Robert Haas writes: > On Tue, Oct 19, 2010 at 12:09 PM, Dimitri Fontaine > wrote: >> Either way is the key here too, so please find attached a revised (v5) >> patch which will force log_min_messages and client_min_messages to >> WARNING while the script is run. > It seems good to do this in the

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 11:22 AM, Terry Laurenzo wrote: > Agreed.  BSON was born out of implementations that either lacked arbitrary > precision numbers or had a strong affinity to an int/floating point way of > thinking about numbers.  I believe that if BSON had an arbitrary precision > number ty

Re: [HACKERS] Serializable snapshot isolation patch

2010-10-19 Thread Jeff Davis
On Mon, 2010-10-18 at 22:12 -0500, Kevin Grittner wrote: > Hmmm... When Joe was looking at the patch he exposed an intermittent > problem with btree indexes which turned out to be related to improper > handling of the predicate locks during index page clean-up caused by a > vacuum. Easy to fix on

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 3:01 PM, Martijn van Oosterhout wrote: > On Tue, Oct 19, 2010 at 02:52:01PM -0400, Robert Haas wrote: >> Well, temp tables really want a separate set of XIDs with a separate >> CLOG, too.  Admittedly, they don't necessarily need WAL, if you can >> make them work without cat

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-19 Thread Martijn van Oosterhout
On Tue, Oct 19, 2010 at 02:52:01PM -0400, Robert Haas wrote: > Well, temp tables really want a separate set of XIDs with a separate > CLOG, too. Admittedly, they don't necessarily need WAL, if you can > make them work without catalog entries, but that's not so easy either. At one point there was

Re: [HACKERS] max_wal_senders must die

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 12:18 PM, Josh Berkus wrote: > On 10/19/2010 09:06 AM, Greg Smith wrote: >> >> I think Magnus's idea to bump the default to 5 triages the worst of the >> annoyance here, without dropping the feature (which has uses) or waiting >> for new development to complete.  I'd be in

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Robert Haas
On Tue, Oct 19, 2010 at 12:09 PM, Dimitri Fontaine wrote: > Tom Lane writes: >> You could argue that either way I guess.  The script knows what it >> needs, but OTOH just about every extension there is will probably >> be generating useless NOTICEs unless something is done, so maybe >> the extens

Re: [HACKERS] Creation of temporary tables on read-only standby servers

2010-10-19 Thread Robert Haas
On Mon, Oct 18, 2010 at 6:05 PM, Greg Stark wrote: > On Mon, Oct 18, 2010 at 1:27 PM, Robert Haas wrote: >>> It'd be kinda cool if we had it, but the work required to get there >>> seems far out of proportion to the benefits ... >> >> I agree.  I think that's backing into the problem from the wro

Re: [HACKERS] Simplifying replication

2010-10-19 Thread Josh Berkus
Greg, > The way things stand you *always* need archived logs. Even if you have > streaming set up it might try to use archived logs if it falls too far > behind. Actually, you don't. If you're willing to accept possible desynchronization and recloning of the standbys, then you can skip the archi

Re: [HACKERS] Simplifying replication

2010-10-19 Thread Greg Stark
On Tue, Oct 19, 2010 at 9:16 AM, Josh Berkus wrote: > Well, one thing to be addressed is separating the PITR functionality from > replication.  PITR needs a lot of features -- timelines, recovery stop > points, etc. -- which replication doesn't need or want.  I think that > focussing on streaming

[HACKERS] gist DatumGetPointer returns pointer to corrupted data

2010-10-19 Thread Marios Vodas
I index these structures in gist: typedef struct { > uint8 type_flag; > float8 xi; > float8 yi; > Timestamp ti; > float8 xe; > float8 ye; > Timestamp te; > int32 id; > } typ_s_flagged; > > typedef struct { > uint8 type_flag; > float8 xl; > float8 yl; >

Re: [HACKERS] Timeline in the light of Synchronous replication

2010-10-19 Thread Robert Haas
On Mon, Oct 18, 2010 at 4:31 AM, Fujii Masao wrote: > But, even though we will have done that, it should be noted that WAL in > A might be ahead of that in B. For example, A might crash right after > writing WAL to the disk and before sending it to B. So when we restart > the old master A as the s

Re: [HACKERS] knngist plans

2010-10-19 Thread Marios Vodas
On Sat, Oct 16, 2010 at 8:42 PM, Oleg Bartunov wrote: > Marios, > > you're right. There are several reasons for poor documentation, but of > course, > no excuse, we do need good docs any way ! It's very nice you're willing to > write one, since it's always better seen from outside of development.

[HACKERS] Location for "makeWholeRowVar()"?

2010-10-19 Thread Tom Lane
The cause of bug #5716 is that preprocess_targetlist is trying to get away with creating a whole-row variable marked with type RECORDOID, even in cases where a specific composite type is known for the referenced RTE. This fails because now we might have non-equal Vars representing the same RTE att

Re: [HACKERS] WIP: extensible enums

2010-10-19 Thread Andrew Dunstan
On 10/19/2010 12:21 AM, Andrew Dunstan wrote: On 10/18/2010 10:52 AM, Tom Lane wrote: We could possibly deal with enum types that follow the existing convention if we made the cache entry hold a list of all the original, known-to-be-sorted OIDs. (This could be reasonably compact and cheap t

Re: [HACKERS] Issues with two-server Synch Rep

2010-10-19 Thread Josh Berkus
Absolutely. For a synch standby, you can't tolerate any standby delay at all. This means that anywhere from 1/4 to 3/4 of queries on the standby would be cancelled on any high-traffic OLTP server. Hence, "useless". Don't agree with your numbers there and you seem to be assuming no workaroun

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Dimitri Fontaine
Alvaro Herrera writes: > Hmm, this needs some cleanup; the comments still talk about the old > function name; and about just the recovery.conf file. Ah yes, thinking it's an easy patch is not helping. Please find attached a revised version of it. Regards, -- Dimitri Fontaine http://2ndQuadrant.

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Dimitri Fontaine
Alvaro Herrera writes: > The error message wording needs some work; maybe > errmsg("file \"%s\" could not be executed", filename) [...] > I happened to notice that there are several pieces of code that are > calling SPI_connect and SPI_finish without checking the return value, > notably the

Re: [HACKERS] max_wal_senders must die

2010-10-19 Thread Josh Berkus
On 10/19/2010 09:06 AM, Greg Smith wrote: I think Magnus's idea to bump the default to 5 triages the worst of the annoyance here, without dropping the feature (which has uses) or waiting for new development to complete. I'd be in favor of just committing that change right now, before it gets for

Re: [HACKERS] Simplifying replication

2010-10-19 Thread Josh Berkus
Dimitri, Greg, I want to say a big big +1 here. The way replication and PITR setup are implemented now are a very good prototype, it's time to consolidate and get to something usable by normal people, as opposed to PostgreSQL full time geeks. Well, one thing to be addressed is separating the P

Re: [HACKERS] O_DSYNC broken on MacOS X?

2010-10-19 Thread Bruce Momjian
A.M. wrote: > > On Oct 19, 2010, at 11:22 AM, Bruce Momjian wrote: > > > Greg Smith wrote: > >> A.M. wrote: > >>> Perhaps a simpler tool could run a basic fsyncs-per-second test and > >>> prompt the DBA to check that the numbers are within the realm of > >>> possibility. > >>> > >> > >> This

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Alvaro Herrera
Excerpts from Dimitri Fontaine's message of vie oct 15 16:15:23 -0300 2010: > The cfparser patch didn't change, the current version is still v1. Hmm, this needs some cleanup; the comments still talk about the old function name; and about just the recovery.conf file. -- Álvaro Herrera The Postg

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Alvaro Herrera
Excerpts from Dimitri Fontaine's message of dom oct 17 16:30:47 -0300 2010: > The bulk of it is now short enough to be inlined in the mail, and if you > have more comments I guess they'll be directed at this portion of the > patch, so let's make it easy: > > /* > * We abuse some internal

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Dimitri Fontaine
Tom Lane writes: > You could argue that either way I guess. The script knows what it > needs, but OTOH just about every extension there is will probably > be generating useless NOTICEs unless something is done, so maybe > the extension management code should take care of it for them. Either way

Re: [HACKERS] max_wal_senders must die

2010-10-19 Thread Greg Smith
Josh Berkus wrote: Under what bizarre set of circumstances would anyone have runaway connections from replicas to the master? Cloud computing deployments where additional replicas are brought up automatically in response to demand. It's easy to imagine a situation where a standby instance is

Re: [HACKERS] max_wal_senders must die

2010-10-19 Thread Josh Berkus
Stefan, Dimitri, > I disagree - limiting the maximum number of replication connections is > important for my usecases. Can you explain more? I clearly don't understand your use case. If we want something fixed *now*, should we perhaps just bump the *default* value for max_wal_senders to 5 or

Re: [HACKERS] Simplifying replication

2010-10-19 Thread Greg Smith
Josh Berkus wrote: *shrug*. Robert asked me to write it up for the list based on the discussions around synch rep. Now you're going to bash me for doing so? Sorry, next time I'll make sure to bash Robert too. I don't have any problems with the basic ideas you're proposing, just concerns abo

Re: [HACKERS] create tablespace fails silently, or succeeds improperly

2010-10-19 Thread Bruce Momjian
Tom Lane wrote: > Bruce Momjian writes: > > Tom Lane wrote: > >> Bruce Momjian writes: > >>> Looking at the pg_upgrade code some more, I found that it was not > >>> removing the PG_VERSION file when deleting <= 8.4 tablespace files. > >>> This might confuse administrators so the attached patch a

Re: [HACKERS] Simplifying replication

2010-10-19 Thread Josh Berkus
You should enjoy one of the patches we're furiously working on then, which is aiming at some of the administration and monitoring pieces here. Great, glad to hear it! Would you be willing to go into detail? I have my own grand vision of how easy replication should be to setup too. So, sha

Re: [HACKERS] Extensions, this time with a patch

2010-10-19 Thread Tom Lane
Dimitri Fontaine writes: > Tom Lane writes: >> Only if the script is intentionally noisy. The right fix here is >> probably to bump up the min message level while running the script. > You mean doing that from the SQL script itself (using SET) or in the > pg_execute_from_file() code? My guess i

Re: [HACKERS] knngist - 0.8

2010-10-19 Thread Teodor Sigaev
combination of parameters is one that it can handle. I'm thinking perhaps in lieu of a boolean, we can add another indexam method which, if not InvalidOid, gets called when we're wondering about whether a given clause is something that the index can order by. Although knngist focuses on a the OR

Re: [HACKERS] O_DSYNC broken on MacOS X?

2010-10-19 Thread A.M.
On Oct 19, 2010, at 11:22 AM, Bruce Momjian wrote: > Greg Smith wrote: >> A.M. wrote: >>> Perhaps a simpler tool could run a basic fsyncs-per-second test and prompt >>> the DBA to check that the numbers are within the realm of possibility. >>> >> >> This is what the test_fsync utility that alr

Re: [HACKERS] PL/JS

2010-10-19 Thread Greg
Cool, will take a look. Thanks! --- On Tue, 19/10/10, Itagaki Takahiro wrote: From: Itagaki Takahiro Subject: Re: [HACKERS] PL/JS To: "Terri Laurenzo" Cc: "Greg" , "pgsql-hackers@postgresql.org" Date: Tuesday, 19 October, 2010, 3:18 On Tue, Oct 19, 2010 at 9:04 AM, Terri Laurenzo wrote: >

  1   2   >