Re: [HACKERS] json api WIP patch

2013-01-07 Thread james
The processing functions have been extended to provide populate_record() and populate_recordset() functions.The latter in particular could be useful in decomposing a piece of json representing an array of flat objects (a fairly common pattern) into a set of Postgres records in a single pass.

Re: [HACKERS] Set visibility map bit after HOT prune

2013-01-07 Thread Pavan Deolasee
Sorry for a long pause on this thread. A new arrival at home kept me occupied all the time. This thread saw a lot of ideas and suggestions from different people. I don't think we had an agreement one way or the other on any of them, but let me summarize the discussion for archival and taking furth

[HACKERS] pg_upgrade with parallel tablespace copying

2013-01-07 Thread Bruce Momjian
Pg_upgrade by default (without --link) copies heap/index files from the old to new cluster. This patch implements parallel heap/index file copying in pg_upgrade using the --jobs option. It uses the same infrastructure used for pg_upgrade parallel dump/restore. Here are the performance results:

Re: [HACKERS] [PATCH] Patch to fix missing libecpg_compat.lib and libpgtypes.lib.

2013-01-07 Thread Peter Eisentraut
On Mon, 2012-11-19 at 14:03 +0800, JiangGuiqing wrote: > hi > > I install postgresql-9.1.5 from source code on windows successfully. > But there is not "libecpg_compat.lib" and "libpgtypes.lib" in the > installed lib directory . > If someone used functions of pgtypes or ecpg_compat library in ecpg

Re: [HACKERS] bad examples in pg_dump README

2013-01-07 Thread Peter Eisentraut
On Sat, 2013-01-05 at 15:34 +0100, Magnus Hagander wrote: > On Fri, Jan 4, 2013 at 3:36 AM, Josh Kupershmidt wrote: > > I propose slimming down the pg_dump README, keeping intact the > > introductory notes and details of the tar format. > > Do we need to keep it at all, really? Certainly the intr

[HACKERS] PL/Python result object str handler

2013-01-07 Thread Peter Eisentraut
For debugging PL/Python functions, I'm often tempted to write something like rv = plpy.execute(...) plpy.info(rv) which prints something unhelpful like By implementing a "str" handler for the result object, it now prints something like Patch attached for review. diff --git a/src/pl/plpytho

[HACKERS] lazy_vacuum_heap()'s removal of HEAPTUPLE_DEAD tuples

2013-01-07 Thread Noah Misch
Per this comment in lazy_scan_heap(), almost all tuple removal these days happens in heap_page_prune(): case HEAPTUPLE_DEAD: /* * Ordinarily, DEAD tuples would have been removed by

Re: [HACKERS] PATCH: optimized DROP of multiple tables within a transaction

2013-01-07 Thread Shigeru Hanada
Hi Tomas, On Tue, Jan 8, 2013 at 7:08 AM, Tomas Vondra wrote: >> * I found another extra space after asterisk. >> >> + RelFileNode * nodes; > > Thanks, fixed. check >> * Curly bracket which starts code block should be at the head of next line. >> >> + /* extend t

Re: [HACKERS] A very small typo in the comment

2013-01-07 Thread Bruce Momjian
On Mon, Jan 7, 2013 at 07:46:46PM -0500, Bruce Momjian wrote: > On Sat, Jan 5, 2013 at 12:13:27AM +0800, 孟庆钟 wrote: > > Hi, > > I am a student in China. I have read some source code of postgreSQL, > > though only a little. I think maybe there is typo in this file: src\backend\ > > catalog\i

Re: [HACKERS] A very small typo in the comment

2013-01-07 Thread Bruce Momjian
On Sat, Jan 5, 2013 at 12:13:27AM +0800, 孟庆钟 wrote: > Hi, > I am a student in China. I have read some source code of postgreSQL, > though only a little. I think maybe there is typo in this file: src\backend\ > catalog\index.c. At line 649 in v9.2.0, line 653 in V9.2.2. The sentence is " > *i

Re: [HACKERS] in-catalog Extension Scripts and Control parameters (templates?)

2013-01-07 Thread Dimitri Fontaine
Hi, Please find attached a preliminary patch following the TEMPLATE ideas, and thanks in particular to Tom and Heikki for a practical design about how to solve that problem! Tom Lane writes: >> - Extension Scripts are now stored in the catalogs, right? > > Only for these new-style thingies. I

Re: [HACKERS] psql \l to accept patterns

2013-01-07 Thread Robert Haas
On Mon, Jan 7, 2013 at 5:14 PM, Peter Eisentraut wrote: > We removed showing system functions and operators from \df and \do > without S. Those are needed all the time. This was controversial at > the time, but it's the way it is now. The definition of "S", I suppose, > is more like "is there a

Re: [HACKERS] psql \l to accept patterns

2013-01-07 Thread Peter Eisentraut
On 1/7/13 3:53 PM, Robert Haas wrote: > On Mon, Jan 7, 2013 at 7:14 AM, Peter Eisentraut wrote: >> > Here is a patch for psql's \l command to accept patterns, like \d >> > commands do. While at it, I also added an "S" option to show system >> > objects and removed system objects from the default

Re: [HACKERS] PATCH: optimized DROP of multiple tables within a transaction

2013-01-07 Thread Tomas Vondra
On 7.1.2013 10:07, Shigeru Hanada wrote: > Hi Tomas, > > I reviewed v6 patch, and found that several places need fix. > Sorry for extra nitpickings. > > * I found another extra space after asterisk. > > + RelFileNode * nodes; Thanks, fixed. > > * Curly bracket which starts code block shou

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Robert Haas
On Mon, Jan 7, 2013 at 4:19 PM, Tom Lane wrote: > Hm ... one of us is reading those results backwards, then. *looks* It's me. Sorry for the noise. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hacker

Re: [HACKERS] ALTER command reworks

2013-01-07 Thread Alvaro Herrera
Tom Lane escribió: > Robert Haas writes: > > On Mon, Jan 7, 2013 at 3:43 PM, Alvaro Herrera > > wrote: > >> If this is considered a problem, I think the way to fix it is to have a > >> getObjectDescriptionOids() variant that quotes the object name in the > >> output. > > > This sort of thing h

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Andrew Dunstan
On 01/07/2013 04:19 PM, Tom Lane wrote: Robert Haas writes: On Mon, Jan 7, 2013 at 11:16 AM, Tom Lane wrote: Why would that be a good tradeoff to make? Larger stored values require more I/O, which is likely to swamp any CPU savings in the compression step. Not to mention that a value once

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Merlin Moncure
On Mon, Jan 7, 2013 at 2:41 PM, Tom Lane wrote: > Merlin Moncure writes: >> On Mon, Jan 7, 2013 at 10:16 AM, Tom Lane wrote: >>> Takeshi Yamamuro writes: The attached is a patch to improve compression speeds with loss of compression ratios in backend/utils/adt/pg_lzcompress.c. > >>> W

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Tom Lane
Robert Haas writes: > On Mon, Jan 7, 2013 at 11:16 AM, Tom Lane wrote: >> Why would that be a good tradeoff to make? Larger stored values require >> more I/O, which is likely to swamp any CPU savings in the compression >> step. Not to mention that a value once written may be read many times, >>

Re: [HACKERS] psql \l to accept patterns

2013-01-07 Thread Tom Lane
Robert Haas writes: > On Mon, Jan 7, 2013 at 7:14 AM, Peter Eisentraut wrote: >> Here is a patch for psql's \l command to accept patterns, like \d >> commands do. While at it, I also added an "S" option to show system >> objects and removed system objects from the default display. This might >>

Re: [HACKERS] ALTER command reworks

2013-01-07 Thread Kohei KaiGai
2013/1/7 Tom Lane : > Robert Haas writes: >> On Mon, Jan 7, 2013 at 3:43 PM, Alvaro Herrera >> wrote: >>> I checked this patch. It needed a rebase for the changes to return >>> OIDs. Attached patch applies to current HEAD. In general this looks >>> good, with one exception: it's using getObje

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Robert Haas
On Mon, Jan 7, 2013 at 11:16 AM, Tom Lane wrote: > Why would that be a good tradeoff to make? Larger stored values require > more I/O, which is likely to swamp any CPU savings in the compression > step. Not to mention that a value once written may be read many times, > so the extra I/O cost coul

Re: [HACKERS] ALTER command reworks

2013-01-07 Thread Tom Lane
Robert Haas writes: > On Mon, Jan 7, 2013 at 3:43 PM, Alvaro Herrera > wrote: >> I checked this patch. It needed a rebase for the changes to return >> OIDs. Attached patch applies to current HEAD. In general this looks >> good, with one exception: it's using getObjectDescriptionOids() to buil

Re: [HACKERS] psql \l to accept patterns

2013-01-07 Thread Robert Haas
On Mon, Jan 7, 2013 at 7:14 AM, Peter Eisentraut wrote: > Here is a patch for psql's \l command to accept patterns, like \d > commands do. While at it, I also added an "S" option to show system > objects and removed system objects from the default display. This might > be a bit controversial, bu

Re: [HACKERS] ALTER command reworks

2013-01-07 Thread Robert Haas
On Mon, Jan 7, 2013 at 3:43 PM, Alvaro Herrera wrote: > Kohei KaiGai escribió: > >> The attached patch is a revised version. >> >> It follows the manner in ExecAlterObjectSchemaStmt(); that tries to check >> name duplication for object classes that support catcache with name-key. >> Elsewhere, it

Re: [HACKERS] recent ALTER whatever .. SET SCHEMA refactoring

2013-01-07 Thread Robert Haas
On Mon, Jan 7, 2013 at 2:14 PM, Alvaro Herrera wrote: > Kohei KaiGai escribió: > >> Function and collation are candidates of this special case handling; >> here are just two kinds of object. >> >> Another idea is to add a function-pointer as argument of >> AlterNamespace_internal for (upcoming) ob

Re: [HACKERS] ALTER command reworks

2013-01-07 Thread Alvaro Herrera
Kohei KaiGai escribió: > The attached patch is a revised version. > > It follows the manner in ExecAlterObjectSchemaStmt(); that tries to check > name duplication for object classes that support catcache with name-key. > Elsewhere, it calls individual routines for each object class to solve objec

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Tom Lane
Merlin Moncure writes: > On Mon, Jan 7, 2013 at 10:16 AM, Tom Lane wrote: >> Takeshi Yamamuro writes: >>> The attached is a patch to improve compression speeds with loss of >>> compression ratios in backend/utils/adt/pg_lzcompress.c. >> Why would that be a good tradeoff to make? Larger stored

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Merlin Moncure
On Mon, Jan 7, 2013 at 10:16 AM, Tom Lane wrote: > Takeshi Yamamuro writes: >> The attached is a patch to improve compression speeds with loss of >> compression ratios in backend/utils/adt/pg_lzcompress.c. > > Why would that be a good tradeoff to make? Larger stored values require > more I/O, wh

Re: [HACKERS] json api WIP patch

2013-01-07 Thread Pavel Stehule
2013/1/7 Merlin Moncure : > On Fri, Jan 4, 2013 at 3:03 PM, Pavel Stehule wrote: >> I understand - but hstore isn't in core - so it should not be precedent >> >> regexp_split_to_table >> >> I am not native speaker, it sounds little bit strange - but maybe >> because I am not native speaker :) > >

Re: [HACKERS] psql \l to accept patterns

2013-01-07 Thread Alvaro Herrera
Peter Eisentraut wrote: > Here is a patch for psql's \l command to accept patterns, like \d > commands do. While at it, I also added an "S" option to show system > objects and removed system objects from the default display. This might > be a bit controversial, but it's how it was decided some ti

Re: [HACKERS] psql \l to accept patterns

2013-01-07 Thread Fabrízio de Royes Mello
On Mon, Jan 7, 2013 at 10:14 AM, Peter Eisentraut wrote: > Here is a patch for psql's \l command to accept patterns, like \d > commands do. While at it, I also added an "S" option to show system > objects and removed system objects from the default display. This might > be a bit controversial,

Re: [HACKERS] recent ALTER whatever .. SET SCHEMA refactoring

2013-01-07 Thread Alvaro Herrera
Kohei KaiGai escribió: > Function and collation are candidates of this special case handling; > here are just two kinds of object. > > Another idea is to add a function-pointer as argument of > AlterNamespace_internal for (upcoming) object classes that takes > special handling for detection of na

Re: [HACKERS] [PERFORM] Slow query: bitmap scan troubles

2013-01-07 Thread Claudio Freire
On Mon, Jan 7, 2013 at 3:27 PM, Tom Lane wrote: > > One issue that needs some thought is that the argument for this formula > is based entirely on thinking about b-trees. I think it's probably > reasonable to apply it to gist, gin, and sp-gist as well, assuming we > can get some estimate of tree

Re: [HACKERS] [PERFORM] Slow query: bitmap scan troubles

2013-01-07 Thread Tom Lane
Simon Riggs writes: > On 7 January 2013 17:35, Tom Lane wrote: >> That gives a formula of >> cpu_operator_cost * log2(N) + cpu_operator_cost * 50 * (H+2) > Again, this depends on N and H, so thats good. > I think my retinas detached while reading your explanation, but I'm a > long way from

Re: [HACKERS] [PERFORM] Slow query: bitmap scan troubles

2013-01-07 Thread Simon Riggs
On 7 January 2013 17:35, Tom Lane wrote: > That gives a formula of > > cpu_operator_cost * log2(N) + cpu_operator_cost * 50 * (H+2) > > This would lead to the behavior depicted in the attached plot, wherein > I've modified the comparison lines (historical, 9.2, and HEAD behaviors) > to in

Re: [HACKERS] [PERFORM] Slow query: bitmap scan troubles

2013-01-07 Thread Tom Lane
I wrote: > Now, if we consider only CPU costs of index descent, we expect > about log2(P) comparisons to be needed on each of the H upper pages > to be descended through, that is we have total descent cost > cpu_index_tuple_cost * H * log2(P) > If we ignore the ceil() step as being a second-o

Re: [HACKERS] ALTER .. OWNER TO error mislabels schema as other object type

2013-01-07 Thread Robert Haas
On Wed, Jan 2, 2013 at 10:35 AM, Kohei KaiGai wrote: > The attached patch fixes this trouble. Thanks. Committed. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes t

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Tom Lane
Takeshi Yamamuro writes: > The attached is a patch to improve compression speeds with loss of > compression ratios in backend/utils/adt/pg_lzcompress.c. Why would that be a good tradeoff to make? Larger stored values require more I/O, which is likely to swamp any CPU savings in the compression s

Re: [HACKERS] json api WIP patch

2013-01-07 Thread Merlin Moncure
On Fri, Jan 4, 2013 at 3:03 PM, Pavel Stehule wrote: > I understand - but hstore isn't in core - so it should not be precedent > > regexp_split_to_table > > I am not native speaker, it sounds little bit strange - but maybe > because I am not native speaker :) it's common usage: http://api.jquery

Re: Review of "pg_basebackup and pg_receivexlog to use non-blocking socket communication", was: Re: [HACKERS] Re: [BUGS] BUG #7534: walreceiver takes long time to detect n/w breakdown

2013-01-07 Thread Boszormenyi Zoltan
2013-01-04 13:43 keltezéssel, Hari Babu írta: On January 02, 2013 12:41 PM Hari Babu wrote: On January 01, 2013 10:19 PM Boszormenyi Zoltan wrote: I am reviewing your patch. • Is the patch in context diff format? Yes. Thanks for reviewing the patch. • Does it apply cleanly to the current git

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Andres Freund
Hi, It seems worth rereading the thread around http://archives.postgresql.org/message-id/CAAZKuFb59sABSa7gCG0vnVnGb-mJCUBBbrKiyPraNXHnis7KMw%40mail.gmail.com for people wanting to work on this. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ Postgre

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread k...@rice.edu
On Mon, Jan 07, 2013 at 01:36:33PM +, Greg Stark wrote: > On Mon, Jan 7, 2013 at 10:21 AM, John R Pierce wrote: > > On 1/7/2013 2:05 AM, Andres Freund wrote: > >> > >> I think there should be enough bits available in the toast pointer to > >> indicate the type of compression. I seem to remembe

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread k...@rice.edu
On Mon, Jan 07, 2013 at 09:10:31AM +, Simon Riggs wrote: > On 7 January 2013 07:29, Takeshi Yamamuro > wrote: > > > Anyway, the compression speed in lz4 is very fast, so in my > > opinion, there is a room to improve the current implementation > > in pg_lzcompress. > > So why don't we use LZ4

Re: [HACKERS] Extra XLOG in Checkpoint for StandbySnapshot

2013-01-07 Thread Andres Freund
On 2013-01-07 19:03:35 +0530, Amit Kapila wrote: > On Monday, January 07, 2013 6:30 PM Simon Riggs wrote: > > On 7 January 2013 12:39, Amit Kapila wrote: > > > > > So We can modify to change this in function LogStandbySnapshot as > > below: > > > running = GetRunningTransactionData

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Simon Riggs
On 7 January 2013 13:36, Greg Stark wrote: > On Mon, Jan 7, 2013 at 10:21 AM, John R Pierce wrote: >> On 1/7/2013 2:05 AM, Andres Freund wrote: >>> >>> I think there should be enough bits available in the toast pointer to >>> indicate the type of compression. I seem to remember somebody even >>>

Re: [HACKERS] Extra XLOG in Checkpoint for StandbySnapshot

2013-01-07 Thread Simon Riggs
On 7 January 2013 13:33, Amit Kapila wrote: >> If we skip the WAL record in the way you suggest, we'd be unable to >> start quickly in some cases. > > If there are any operations happened which have generated WAL, then on next > checkpoint interval the checkpoint operation should happen. > Which

Re: [HACKERS] Re: Proposal: Store "timestamptz" of database creation on "pg_database"

2013-01-07 Thread Stephen Frost
* Fabrízio de Royes Mello (fabriziome...@gmail.com) wrote: > Understood... a "COMMENT" is a database object, then if we add a creation > time column to pg_description/shdescription tables how we track his > creation time? When it's NULL it "doesn't exist", in this case, when it transistions from N

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Greg Stark
On Mon, Jan 7, 2013 at 10:21 AM, John R Pierce wrote: > On 1/7/2013 2:05 AM, Andres Freund wrote: >> >> I think there should be enough bits available in the toast pointer to >> indicate the type of compression. I seem to remember somebody even >> posting a patch to that effect? >> I agree that it'

Re: [HACKERS] Extra XLOG in Checkpoint for StandbySnapshot

2013-01-07 Thread Amit Kapila
On Monday, January 07, 2013 6:30 PM Simon Riggs wrote: > On 7 January 2013 12:39, Amit Kapila wrote: > > > So We can modify to change this in function LogStandbySnapshot as > below: > > running = GetRunningTransactionData(); > > if (running->xcnt > 0) > >

[HACKERS] A very small typo in the comment

2013-01-07 Thread ??????
Hi, I am a student in China. I have read some source code of postgreSQL, though only a little. I think maybe there is typo in this file: src\backend\catalog\index.c. At line 649 in v9.2.0, line 653 in V9.2.2. The sentence is " *indexInfo: same info executor uses to insert into the index",

Re: [HACKERS] Extra XLOG in Checkpoint for StandbySnapshot

2013-01-07 Thread Simon Riggs
On 7 January 2013 12:39, Amit Kapila wrote: > So We can modify to change this in function LogStandbySnapshot as below: > running = GetRunningTransactionData(); > if (running->xcnt > 0) > LogCurrentRunningXacts(running); > > So this check wil

[HACKERS] Extra XLOG in Checkpoint for StandbySnapshot

2013-01-07 Thread Amit Kapila
Observation is that whenever a checkpoint happens and the wal_level configured is hot_standby then one standby snapshot XLOG gets written with the information of "running transaction". So if first time checkpoint happened at specified interval, it will create new XLOG in LogStandbySnapshot, due

[HACKERS] psql \l to accept patterns

2013-01-07 Thread Peter Eisentraut
Here is a patch for psql's \l command to accept patterns, like \d commands do. While at it, I also added an "S" option to show system objects and removed system objects from the default display. This might be a bit controversial, but it's how it was decided some time ago that the \d commands shou

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Andres Freund
On 2013-01-07 02:21:26 -0800, John R Pierce wrote: > On 1/7/2013 2:05 AM, Andres Freund wrote: > >I think there should be enough bits available in the toast pointer to > >indicate the type of compression. I seem to remember somebody even > >posting a patch to that effect? > >I agree that it's proba

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread John R Pierce
On 1/7/2013 2:05 AM, Andres Freund wrote: I think there should be enough bits available in the toast pointer to indicate the type of compression. I seem to remember somebody even posting a patch to that effect? I agree that it's probably too late in the 9.3 cycle to start with this. so an upgra

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Andres Freund
On 2013-01-07 09:57:58 +, Simon Riggs wrote: > On 7 January 2013 09:19, John R Pierce wrote: > > On 1/7/2013 1:10 AM, Simon Riggs wrote: > >> > >> On 7 January 2013 07:29, Takeshi Yamamuro > >> wrote: > >> > >>> >Anyway, the compression speed in lz4 is very fast, so in my > >>> >opinion, the

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Simon Riggs
On 7 January 2013 09:19, John R Pierce wrote: > On 1/7/2013 1:10 AM, Simon Riggs wrote: >> >> On 7 January 2013 07:29, Takeshi Yamamuro >> wrote: >> >>> >Anyway, the compression speed in lz4 is very fast, so in my >>> >opinion, there is a room to improve the current implementation >>> >in pg_lzc

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread John R Pierce
On 1/7/2013 1:10 AM, Simon Riggs wrote: On 7 January 2013 07:29, Takeshi Yamamuro wrote: >Anyway, the compression speed in lz4 is very fast, so in my >opinion, there is a room to improve the current implementation >in pg_lzcompress. So why don't we use LZ4? what will changing compression

Re: [HACKERS] Improve compression speeds in pg_lzcompress.c

2013-01-07 Thread Simon Riggs
On 7 January 2013 07:29, Takeshi Yamamuro wrote: > Anyway, the compression speed in lz4 is very fast, so in my > opinion, there is a room to improve the current implementation > in pg_lzcompress. So why don't we use LZ4? -- Simon Riggs http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] PATCH: optimized DROP of multiple tables within a transaction

2013-01-07 Thread Shigeru Hanada
Hi Tomas, I reviewed v6 patch, and found that several places need fix. Sorry for extra nitpickings. * I found another extra space after asterisk. + RelFileNode * nodes; * Curly bracket which starts code block should be at the head of next line. + /* extend t