Re: [HACKERS] using arrays within structure in ECPG

2014-03-24 Thread Ashutosh Bapat
On Mon, Mar 24, 2014 at 3:40 PM, Boszormenyi Zoltan  wrote:

>  2014-03-24 07:22 keltezéssel, Ashutosh Bapat írta:
>
>  Hi,
>  I tried using integer array within a structure array in ECPG code. But
> it resulted in some garbage values being printed from the table. Here are
> the details,
>
>  The ECPG program is attached (array_test.pgc). It tries to read the
> contents of table emp, whose structure and contents are as follows
> postgres=# \d+ emp
>Table "public.emp"
>  Column  |   Type| Modifiers | Storage  | Stats target |
> Description
>
> -+---+---+--+--+-
>  empno   | numeric(4,0)  |   | main |  |
>  ename   | character varying |   | extended |  |
>  job | character varying |   | extended |  |
>  arr_col | integer[] |   | extended |  |
> Has OIDs: no
>
> postgres=# select * from emp;
>  empno | ename  |   job   |  arr_col
> ---++-+
>   7900 | JAMES  | CLERK   | {1,2,7900}
>   7902 | FORD   | ANALYST | {1,2,7902}
>   7934 | MILLER | CLERK   | {1,2,7934}
> (3 rows)
>
>  You will notice that the last element of the arr_col array is same as
> the empno of that row.
>
>  The ECPG program tries to read the rows using FETCH in a structure emp
> defined as
>  15 struct employee {
>  16 int empno;
>  17char ename[11];
>  18char job[15];
>  19int  arr_col[3];
>  20 };
>
>  and then print the read contents as
>  39 /* Print members of the structure. */
>  40 for ( i = 0 ;i < 3; i++){
>  41 printf("empno=%d, ename=%s, job=%s, arr_col[2]=%d\n",
> emp[i].empno, emp[i].ename, emp[i].job, emp[i].arr_col[2]);
>  42
>  43 }
>
>  But garbage values are printed
> [ashutosh@ubuntu repro]./array_test
>
> +++
> empno=7900, ename=JAMES, job=CLERK, arr_col[2]=1
> empno=2, ename=� , job=ANALYST, arr_col[2]=32767
> empno=7934, ename=MILLER, job=CLERK, arr_col[2]=1719202336
>
>  Here are steps I have used to compile the ECPG program
> [ashutosh@ubuntu repro]make array_test
> ecpg -c -I/work/pg_head/build/include array_test.pgc
> cc -I/work/pg_head/build/include -g   -c -o array_test.o array_test.c
> cc -g  array_test.o  -L/work/pg_head/build/lib -lecpg -lpq -o array_test
> rm array_test.o array_test.c
>
>  where /work/pg_head/build is the directory containing the postgresql
> build (essentially argument to the --prefix option to configure).
>
>  The programs compiles and links fine.
>
>  Without the arr_col member, the program works fine. So, it seems to be a
> problem with array within structure array.
>
>  In array_test.c I see that the ECPGdo statement corresponding to the
> FETCH command is as follows
>  87 /* Fetch multiple columns into one structure. */
>  88 { ECPGdo(__LINE__, 0, 1, NULL, 0, ECPGst_normal, "fetch 3 from
> cur1", ECPGt_EOIT,
>  89 ECPGt_int,&(emp->empno),(long)1,(long)14,sizeof( struct employee ),
>  90 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
>  91 ECPGt_char,&(emp->ename),(long)11,(long)14,sizeof( struct employee
> ),
>  92 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
>  93 ECPGt_char,&(emp->job),(long)15,(long)14,sizeof( struct employee ),
>  94 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
>  95 ECPGt_int,(emp->arr_col),(long)1,(long)3,sizeof(int),
>  96 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L, ECPGt_EORT);
>
>  For all the members of struct employee, except arr_col, the size of
> array is set to 14 and next member offset is set of sizeof (struct
> employee). But for arr_col they are set to 3 and sizeof(int) resp. So, for
> the next row onwards, the calculated offset of arr_col member would not
> coincide with the real arr_col member's address.
>
>  Am I missing something here?
>
>
> ECPG (I think intentionally) doesn't interpret or parse array fields.
> You need to pass a character array and parse the contents by yourself.
>
>
That doesn't seem to be the case with bare arrays (not included in anything
else). I added following lines to the program attached in my first mail,
and they worked perfectly fine.

 47 /* Test only arrays */
 48 EXEC SQL DECLARE cur2 CURSOR FOR select arr_col from emp;
 49
 50 EXEC SQL OPEN cur2;
 51
 52 EXEC SQL FETCH 1 FROM cur2 into :emp[0].arr_col;
 53
 54 printf("\n+++\n");
 55
 56 /* Print members of the array fetched. */
 57 for ( i = 0 ;i < 3; i++)
 58 {
 59 printf("arr_col[%d] = %d\n", i, emp[0].arr_col[i]);
 60 }
 61 EXEC SQL CLOSE cur2;

Anyway, if that's a restriction, shouldn't there be an error during
compilation?

> Best regards,
> Zoltán Böszörményi
>
>
>   --
> Best Wishes,
> Ashutosh Bapat
> EnterpriseDB Corporation
> The Postgres Database Company
>
>
>
>


-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporat

Re: [HACKERS] About adding an attribute to pg_attibute

2014-03-24 Thread Craig Ringer
On 03/25/2014 01:44 AM, Tanmay Deshpande wrote:
> We are trying add columnar support to postgreSQL and for that we need
> identify each column in each relation by an oid. So my doubt is how do
> we add an attribute/ a column to catalog pg_attribute ?

Is this some kind of class project?

I've seen at least three people post short, uninformative questions
about somewhat similar topics to this mailing list, all from gmail
addresses, in the last couple of days.

Rajashree Mandaogane 
Swapnil Bhoite 
Tanmay Deshpande 

Ah, yes, it is something like that:

http://www.postgresql.org/message-id/capn0l0oxp9afnr7uhwzkytvzzyy8z-aqgfnpnabk-cgscge...@mail.gmail.com

To make it easier to understand your questions and their context, in
future please:

- Use an email signature that says something about what you're working
  on, so we know who you are. There are lots of people doing lots of
  things on this list. Or mark your emails with a subject tag like
  [COLSTORE] or whatever you call your project.

- When somebody replies to you, and you have a new question as a
  result, reply to their reply. Do not make a new email thread.

- If someone replies to you, respond. Don't just ignore their reply.
  If you didn't understand it, that's OK, we're all learning here.
  Say so. If you found it helpful, you can always follow up with a
  quick off-list email saying "thanks".

Finally, please remember that we don't know much about what you're
working on. When you're communicating by email, context is key. Don't
just say "how do I do X". Say "Is X the best way to achive my goal, Y?
If it is, any advice on how to do X?" . That will help us know whether
to say "Oh, do X this way" or "No, don't do that! You should do Z instead!".

Putting more effort into your questions will get you better answers.

-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Why MarkBufferDirtyHint doesn't increment shared_blks_dirtied

2014-03-24 Thread Amit Kapila
MarkBufferDirty() always increment BufferUsage counter
(shared_blks_dirtied) for dirty blocks whenever it dirties any
block, whereas same is not true for MarkBufferDirtyHint().
Is there any particular reason for not incrementing
shared_blks_dirtied in MarkBufferDirtyHint()?



With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Global flag

2014-03-24 Thread Craig Ringer
On 03/24/2014 05:53 PM, Swapnil Bhoite wrote:
> Hi,
> 
> I want to set a *global flag* with which I can decide whether to use my
> code or not
> in modified source code.
> How I can do that?

Please reply to existing mailing list threads. Don't make a new message
for every post. It's confusing and causes people to loose track of what
others have already told you.

Reply-all to somebody's reply to your original message (or reply-list,
if your mail client supports it) if you're following up on an existing
topic.

Imagine it like a forum. You don't make a new thread for every post, do
you? No, you *reply* to an existing thread.

-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Craig Ringer
On 03/25/2014 07:05 AM, Tom Lane wrote:
> Jim Nasby  writes:
>> Wait... I thought that was one of the objections... that we wanted to
>> leave a BOM in something like a COPY untouched?
> 
> I think most of us are okay with stripping a BOM that appears at the
> *beginning* of a text file (assuming there's reason to believe the file
> is in UTF8 encoding).  BOM sequences embedded later in the file are a lot
> more debatable, and I for one don't want to assume those can be dropped.
> I don't know of any legitimate usage of such cases, and think it's
> probably better to report an encoding error.

Yep, it's absolutely an encoding error IMO.

Something like:

ERROR: Invalid UTF-8 - probable UTF-8 byte-order mark detected
mid-data-stream.
HINT: Multiple files with byte-order marks were probably concatenated
with a tool that is not Unicode-aware.




-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Craig Ringer
On 03/25/2014 02:50 AM, Jim Nasby wrote:
> So instead of trying to handle this on the psql side[1], I think we need
> to handle it in the backend; specifically in the parser. Is there an
> easy way to get the parser to ignore the BOM character in the context of
> commands (but not in strings)?

I disagree. The server deals with SQL statements and client data
streams, it doesn't deal with files.

The BOM is an artifact of *files*. You don't expect to see a BOM in a
UTF-8 string passed to a function call in a library; nor should you
expect one to be passed to you on a network protocol that isn't about
exchanging files.

-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [bug fix] PostgreSQL fails to start on Windows if it crashes after tablespace creation

2014-03-24 Thread Amit Kapila
On Mon, Mar 24, 2014 at 7:49 PM, MauMau  wrote:
> A bit contrived example is:
>
> 1. After the directory is created by TablespaceCreateDbspace(), recovery is
> stopped (e.g. due to power outage).  The directory remains.
> 2. Restart the server, redoing CREATE TABLESPACE during recovery, which
> executes create_tablespace_directories().

I don't understand how after step-1, step-2 can occur in recovery for same
tablespace path.

Function TablespaceCreateDbspace() would have called for CREATE TABLE.
Now Step-2 can only occur if there is a Drop Tablespace command in-between
step-1 and step-2, else CREATE TABLESPACE can't be successful during
command execution so will not get recorded in WAL.  Basically Create Table
cannot happen on a particular directory without having some
CREATE TABLESPACE before it, so in the above example taken by you,
there must be some Create TableSpace before step-1 or it's on default
tablespace location which means you cannot perform step-2 for same
tablespace path as step-1 without having DROP TABLESPACE in-between
step-1 and step-2.

If you think that above scenario is not possible, then you just need to
modify comment:
"!  * Remove old symlink in recovery"

One more minor point about patch:
+ struct stat st;

if (InRecovery)
{
struct stat st;

Defining stat struct two times in same function in different ways doesn't
seem to be good, we can do the same way for new usage as is already
done in code or may be declare it once.


With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] About adding a column to an existing system catalog

2014-03-24 Thread Tom Lane
Tanmay Deshpande  writes:
> while adding a column to an existing system catalog I am getting following
> error while initdb

> FATAL:  incorrect number of columns in row (expected 22, got 21)

> where do I have to make the changes ?

I'd say you missed updating relnatts in the hard-coded row for
pg_attribute in pg_class.h.

The more general answer is to look through our git history for a previous
commit that did something like what you want, and then read through it to
see if you missed anything.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Dynamic background workers & docs question

2014-03-24 Thread Michael Paquier
On Mon, Mar 24, 2014 at 11:03 PM, Magnus Hagander  wrote:
> That's what I thought. Can a dynamic background worker start *another*
> dynamic background worker, or can they only be started from "first level"
> background workers?
I have never really tried by myself, but I don't see any reason why it
wouldn't work as it is only a matter of doing what is for example in
worker_spi_launch:worker_spi.c. Btw, a bgworker could also behave like
a "regular backend" as mentioned in the docs, so a regular backend is
just a subclass of a bgworker :)

>> > Also:
>> > "Background workers are expected to be continuously running; if they
>> > exit
>> > cleanly, postgres will restart them immediately. "
>> >
>> > This doesn't apply to dynamic ones, which we might want to clarify. Do
>> > we
>> > have a "term" for non-dynamic background workers? "static workers"?
>> In the code or the documentation, there is no explicit
>> differentiation, bgworkers are either called plainly "bgworker", or
>> "dynamic bgworker". Perhaps the solution here is simply to say
>> "background workers started by the postmaster are expected blabla".
> That, or we need to invite a term for it?
Hm... Seems like an overkill. The main difference between a
non-dynamic and dynamic bgworker is the way they are registered.
"Static" bgworkers use RegisterBackgroundWorker that can only be
called in _PG_init when a module is loaded with
shared_preload_libraries. Dynamic bgworkers use
RegisterDynamicbackgroundWorker. And this differentiation is clearly
done in the 2nd paragraph.

So perhaps the solution here is simply to write "Background workers
registered with RegisterBackgroundWorker are expected...".

I am not a native English speaker, but "static" sounds like a daemon
process that has to restart, and a bgworker could perform a one-time
task as well.
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] About adding a column to an existing system catalog

2014-03-24 Thread Tanmay Deshpande
while adding a column to an existing system catalog I am getting following
error while initdb

FATAL:  incorrect number of columns in row (expected 22, got 21)

where do I have to make the changes ?


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Andrew Dunstan


On 03/24/2014 08:28 PM, Tatsuo Ishii wrote:

The code would probably be pretty trivial, *if* we had consensus on
what the behavior ought to be.  I'm not sure if we do.  People who
only use Unicode would probably like it if BOMs were unconditionally
swallowed, whether or not psql thinks the client_encoding is UTF8.
(And I seem to recall somebody even proposing that finding a BOM
be cause to switch the client_encoding to UTF8.)

This is a bad idea. ISO 8859-1 uses 0xfe and 0xff (BOM) for some
characters.





Yeah, I think there is no consensus to do anything unless the client 
encoding is UTF8.


cheers

andrew


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] About adding an attribute to a system catalog

2014-03-24 Thread Tanmay Deshpande
My doubt is what changes does one have to make in the source code if he/she
is trying to add an attribute to the existing system catalogs table ?


Re: [HACKERS] Useless "Replica Identity: NOTHING" noise from psql \d

2014-03-24 Thread Bruce Momjian
On Mon, Mar 24, 2014 at 01:35:20PM -0300, Alvaro Herrera wrote:
> Bruce Momjian wrote:
> > On Mon, Mar 24, 2014 at 05:06:25PM +0100, Andres Freund wrote:
> > > On 2014-03-22 23:47:57 -0400, Bruce Momjian wrote:
> > > > test=> \d+ test
> > > >  Table "public.test"
> > > >  Column |  Type   | Modifiers | Storage | Stats target | 
> > > > Description
> > > > 
> > > > +-+---+-+--+-
> > > >  x  | integer |   | plain   |  |
> > > > Replica Identity: full
> > > > Has OIDs: no
> > > > 
> > > > I used lower-case for the value, rather than all-caps.
> > > 
> > > Why? CLUSTER, PRIMARY KEY, etc. are displayed all caps, and replica
> > > identity is similarly set via ALTER TABLE ... REPLICA IDENITY?
> > 
> > Oh, good points;  I had not considered PRIMARY KEY.  Updated patch
> > attached.
> 
> In the "INDEX" case, should the output mention specifically which index
> is being considered?

Ah, good idea.  Updated patch attached.  The output is now:

test=> \d+ test
 Table "public.test"
 Column |  Type   | Modifiers | Storage | Stats target | Description
+-+---+-+--+-
 x  | integer | not null  | plain   |  |
Indexes:
"test_pkey" PRIMARY KEY, btree (x) REPLICA IDENTITY
"i_test2" btree (x)
--> Replica Identity: USING INDEX "test_pkey"
Has OIDs: no

However, now that I look at it, it seems redundant as REPLICA IDENTITY
is already marked on the actual index.  Ideas?

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
new file mode 100644
index a194ce7..2230968
*** a/src/bin/psql/describe.c
--- b/src/bin/psql/describe.c
*** describeOneTableDetails(const char *sche
*** 2345,2358 
  			printTableAddFooter(&cont, buf.data);
  		}
  
! 		if ((tableinfo.relkind == 'r' || tableinfo.relkind == 'm') &&
! 			tableinfo.relreplident != 'd' && tableinfo.relreplident != 'i')
  		{
  			const char *s = _("Replica Identity");
  
! 			printfPQExpBuffer(&buf, "%s: %s",
! 			  s,
! 			  tableinfo.relreplident == 'n' ? "NOTHING" : "FULL");
  			printTableAddFooter(&cont, buf.data);
  		}
  
--- 2345,2387 
  			printTableAddFooter(&cont, buf.data);
  		}
  
! 		if (verbose && (tableinfo.relkind == 'r' || tableinfo.relkind == 'm') &&
! 			strcmp(schemaname, "pg_catalog") != 0)
  		{
  			const char *s = _("Replica Identity");
  
! 			/* find USING INDEX index? */
! 			if (tableinfo.relreplident == 'i')
! 			{
! printfPQExpBuffer(&buf,
!   "SELECT r.relname\n"
!   "FROM pg_catalog.pg_class r,\n"
!   " pg_catalog.pg_index i\n"
!   "WHERE i.indrelid = %s AND "
!   "  i.indisreplident AND "
!   "  i.indexrelid = r.oid;",
!   oid);
! result = PSQLexec(buf.data, false);
! if (!result)
! 	goto error_return;
! else if (PQntuples(result) != 1)
! {
! 	PQclear(result);
! 	goto error_return;
! }
! 	
! printfPQExpBuffer(&buf, _("%s: USING INDEX \"%s\""), s,
!   PQgetvalue(result, 0, 0));
! PQclear(result);
! 			}
! 			else
! printfPQExpBuffer(&buf, "%s: %s",
!   s,
!   tableinfo.relreplident == 'd' ? "DEFAULT" :
!   tableinfo.relreplident == 'f' ? "FULL" :
!   tableinfo.relreplident == 'n' ? "NOTHING" :
!   "???");
! 
  			printTableAddFooter(&cont, buf.data);
  		}
  
diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out
new file mode 100644
index 5f29b39..feb6c93
*** a/src/test/regress/expected/create_table_like.out
--- b/src/test/regress/expected/create_table_like.out
*** CREATE TABLE ctlt12_storage (LIKE ctlt1
*** 115,120 
--- 115,121 
   a  | text | not null  | main |  | 
   b  | text |   | extended |  | 
   c  | text |   | external |  | 
+ Replica Identity: DEFAULT
  Has OIDs: no
  
  CREATE TABLE ctlt12_comments (LIKE ctlt1 INCLUDING COMMENTS, LIKE ctlt2 INCLUDING COMMENTS);
*** CREATE TABLE ctlt12_comments (LIKE ctlt1
*** 125,130 
--- 126,132 
   a  | text | not null  | extended |  | A
   b  | text |   | extended |  | B
   c  | text |   | extended |  | C
+ Replica Identity: DEFAULT
  Has OIDs: no
  
  CREATE TABLE ctlt1_inh (LIKE ctlt1 INCLUDING CONSTRAINTS INCLUDING COMMENTS) INHERITS (ctlt1);
*** NOTICE:  merging constraint "ctlt1_a_che
*** 140,145 
--- 142,148 
  Check constraints:
  "ctlt1_a_check" CHECK (length(a) > 2)
 

Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Tatsuo Ishii
> The code would probably be pretty trivial, *if* we had consensus on
> what the behavior ought to be.  I'm not sure if we do.  People who
> only use Unicode would probably like it if BOMs were unconditionally
> swallowed, whether or not psql thinks the client_encoding is UTF8.
> (And I seem to recall somebody even proposing that finding a BOM
> be cause to switch the client_encoding to UTF8.)

This is a bad idea. ISO 8859-1 uses 0xfe and 0xff (BOM) for some
characters.

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] About adding an attribute to pg_attibute

2014-03-24 Thread Michael Paquier
On Tue, Mar 25, 2014 at 2:44 AM, Tanmay Deshpande
 wrote:
> We are trying add columnar support to postgreSQL and for that we need
> identify each column in each relation by an oid. So my doubt is how do we
> add an attribute/ a column to catalog pg_attribute ?
You can identify already uniquely columns in pg_attribute with the
couple (attrelid,attnum) or even (attrelid,attname). Using an OID to
do what already exists will just add complication in your application.
Regards,
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Tatsuo Ishii
>> Just a quick comment on this. Yes, pgAdmin always added a BOM in every
>> SQL files it wrote.
> 
> From 
> http://stackoverflow.com/questions/2223882/whats-different-between-utf-8-and-utf-8-without-bom:
> 
> According to the Unicode standard, the BOM for UTF-8 files is not recommended:
> 
> 2.6 Encoding Schemes
> 
> ... Use of a BOM is neither required nor recommended for UTF-8, but may be 
> encountered in contexts where UTF-8 data is converted from other encoding 
> forms that use a BOM or where the BOM is used as a UTF-8 signature. See the 
> “Byte Order Mark” subsection in Section 16.8, Specials, for more information.

Right. I think unconditionally adding BOM to a file is evil.

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Only first XLogRecData is visible to rm_desc with WAL_DEBUG

2014-03-24 Thread Alvaro Herrera
Robert Haas wrote:
> On Mon, Mar 24, 2014 at 7:14 AM, Tom Lane  wrote:

> >> 3. Remove the feature altogether, so that enabling wal_debug doesn't
> >> cause all insertions to be logged anymore (no changes to the logging
> >> during replay). It's a lot less interesting now that we have pg_xlogdump.
> >
> > I think the main use-case for rm_desc anymore is making CONTEXT lines
> > for errors reported during WAL replay.  I guess that situation does not
> > have the same problem, since we've already loaded the complete WAL record.
> >
> > However, I'm not sure how easy it's going to be for WAL_DEBUG to make the
> > data look the same as the replay case: in particular, substitution of
> > full-page-images for data would be tough to predict in advance (and moving
> > the printout into the critical section seems like a bad answer).
> >
> > I'd be okay with removing WAL_DEBUG, I think, particularly in view of the
> > fact that there have been no requests to make it a compiled-by-default
> > feature.
> 
> I've found WAL_DEBUG quite useful in the past, when working on
> scalability, and have indeed wished for it to be
> compiled-in-by-default.
> 
> I don't know whether I'm the only one, though.

You are not.  I would rather have it fixed than removed, if possible.  I
don't really care too much about getting a performance hit to palloc the
records, really; being able to actually read what's happening is much
more useful.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] GSoC application: MADlib k-medoids clustering

2014-03-24 Thread Robert Haas
On Thu, Mar 20, 2014 at 12:30 PM, Maxence Ahlouche
 wrote:
> My proposal is now available on Google melange website:
> http://www.google-melange.com/gsoc/proposal/public/google/gsoc2014/viod/5668600916475904
> There seems to be a formatting issue: half of the text is a link to the page
> I mentionned during my registration on my website. I don't know how to fix
> it though.

As far as I know, this mailing list isn't the right place to discuss
anything related to MADlib.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Only first XLogRecData is visible to rm_desc with WAL_DEBUG

2014-03-24 Thread Robert Haas
On Mon, Mar 24, 2014 at 7:14 AM, Tom Lane  wrote:
> Heikki Linnakangas  writes:
>> As we all know, when you compile with WAL_DEBUG, and enable wal_debug
>> GUC, you get output like this in the log for every inserted WAL record:
>
>> LOG:  INSERT @ 0/5407E578: prev 0/5407E4D0; xid 0; len 32: Standby -
>> running xacts: nextXid 774 latestCompletedXid 771 oldestRunningXid 772;
>> 2 xacts: 7877929 0
>
>> What I did *not* realize before is that the redo routine only gets
>> passed the first XLogRecData struct in the chain. Because of that, the
>> "xacts: 7877929 0" information above is garbage. That's probably not the
>> only rm_desc routine that didn't get the memo.
>
> Yeah, I recall having fixed at least one such bug in the past.
>
>> There are a few alternatives on how to fix that:
>
>> 1. Do nothing. Add a comment somewhere explaining that rm_redo cannot
>> safely look at data beyond what's inserted in the first XLogRecData.
>
> We'll probably just make the same mistake again :-(.  And I think there
> already is such a comment someplace.
>
>> 2. Reconstruct the WAL data from the XLogRecData entries in a palloc'd
>> buffer, and pass that to rm_redo. That would be fairly expensive, but
>> you probably don't care about that if you've enabled wal_debug.
>
>> 3. Remove the feature altogether, so that enabling wal_debug doesn't
>> cause all insertions to be logged anymore (no changes to the logging
>> during replay). It's a lot less interesting now that we have pg_xlogdump.
>
> I think the main use-case for rm_desc anymore is making CONTEXT lines
> for errors reported during WAL replay.  I guess that situation does not
> have the same problem, since we've already loaded the complete WAL record.
>
> However, I'm not sure how easy it's going to be for WAL_DEBUG to make the
> data look the same as the replay case: in particular, substitution of
> full-page-images for data would be tough to predict in advance (and moving
> the printout into the critical section seems like a bad answer).
>
> I'd be okay with removing WAL_DEBUG, I think, particularly in view of the
> fact that there have been no requests to make it a compiled-by-default
> feature.

I've found WAL_DEBUG quite useful in the past, when working on
scalability, and have indeed wished for it to be
compiled-in-by-default.

I don't know whether I'm the only one, though.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Tom Lane
Jim Nasby  writes:
> Wait... I thought that was one of the objections... that we wanted to
> leave a BOM in something like a COPY untouched?

I think most of us are okay with stripping a BOM that appears at the
*beginning* of a text file (assuming there's reason to believe the file
is in UTF8 encoding).  BOM sequences embedded later in the file are a lot
more debatable, and I for one don't want to assume those can be dropped.
I don't know of any legitimate usage of such cases, and think it's
probably better to report an encoding error.

> Uh... could we just treat BOM as another whitespace character?

A BOM is *most certainly not* whitespace.  The only even semi-legitimate
usage it has in UTF8 is as a file encoding marker.  You can bet that the
user whose text editor made the file did not think he had whitespace at
the front.  Anyway, your proposition that leading whitespace is ignorable
fails completely for data files.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Jim Nasby

On 3/24/14, 1:59 PM, Andrew Dunstan wrote:

It occurs to me that we're going about this the wrong way...

The error here isn't being generated by psql; it's generated by the backend. In 
the context of a statement (and not, say, a COPY command).

So instead of trying to handle this on the psql side[1], I think we need to 
handle it in the backend; specifically in the parser. Is there an easy way to 
get the parser to ignore the BOM character in the context of commands (but not 
in strings)?

[1]: Obviously, BOM could still screw up a psql command like \d. We'd want to 
address that as well; but I suspect backends are the more common scenario.



But what about COPY files? I don't see why there is less of a case for eating a 
leading BOM for a COPY file (or COPY stdin for that matter, given that it can 
come from \copy) than for an SQL file.


Wait... I thought that was one of the objections... that we wanted to leave a 
BOM in something like a COPY untouched? If that's not the case, why not just 
strip BOM wherever it shows up in psql input? (Granted, not good for \copy or 
copy; performance, so we might want to special case those, but that doesn't 
seem unreasonable...)


I suspect suspect trying to do this in the parser will be quite messy. This 
needs to happen before the input is converted to the server encoding, I think.


My hope was that there's a point in the parser where we know whether we're 
dealing with a command strong or raw data, and that we'd be able to only strip 
this from command strings... or better yet, get the code that looks for a 
command string to simply ignore BOM when it's parsing.

Uh... could we just treat BOM as another whitespace character? ISTM the case is basically 
the same: we don't want " INSERT ... VALUES( '  extra  spaces  ' )  ;  " to 
blow up because of extra white space, but obviously '  extra  spaces  ' needs to stay 
intact
--
Jim C. Nasby, Data Architect   j...@nasby.net
512.569.9461 (cell) http://jim.nasby.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [pgsql-advocacy] [HACKERS] First draft of update announcement

2014-03-24 Thread Josh Berkus
On 03/19/2014 02:16 PM, Darren Duncan wrote:
> On 2014-03-18, 2:42 PM, Josh Berkus wrote:
>> Other PostgreSQL 9.3 only fixes in this update include:
>>
>> * Add read-only data_checksum parameter
> 
> I recall being told last fall that this would not be added to 9.3.x
> (9.3.1 at the time I think) and only to 9.4.x because such a feature
> addition was something only allowed for major releases and not minor
> ones which were just supposed to be security and bug fixes.
> 
> So what changed that it is added in 9.3.x after all?

Enough people reported operational problems with not having the parameter.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Merlin Moncure
On Mon, Mar 24, 2014 at 2:37 PM, Merlin Moncure  wrote:
> psql -1 already requires '-f' to work

actually, it doesn't.  this was fixed recently.

merlin


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Merlin Moncure
On Mon, Mar 24, 2014 at 2:16 PM, Tom Lane  wrote:
> Andrew Dunstan  writes:
>> I suspect suspect trying to do this in the parser will be quite messy.
>> This needs to happen before the input is converted to the server
>> encoding, I think.
>
> Indeed --- what if the server isn't using utf8 internally?
>
> And a larger point is that the server has no idea where the file
> boundaries are.  If we were to do this server-side, we'd essentially
> end up discarding BOM anywhere, which is more libertine than I care
> to be.

Right -- I had a feeling you'd say that.  That's why the best solution
ISTM is to allow psql to be invoked in such a way that it *does* know
the file boundaries for consolidated scripts; this means better
handling of multiple file arguments.  psql -1 already requires '-f' to
work (vs cat foo.sql | psql) and that's pretty reasonable.  BOM
handling fixes should probably be injected in cases where the precise
beginning points of the file are known, which AFAICT are \i and -f.
So, in short, it seems prudent to:

1. make multiple -f invocation work (with -1 spanning)
2. strip BOM from -f or \i foo.sql if it's there

That will fix all non redirection usages.  Cases involving redirection
are not psql's bailiwick.

merlin


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Tom Lane
Andrew Dunstan  writes:
> I suspect suspect trying to do this in the parser will be quite messy. 
> This needs to happen before the input is converted to the server 
> encoding, I think.

Indeed --- what if the server isn't using utf8 internally?

And a larger point is that the server has no idea where the file
boundaries are.  If we were to do this server-side, we'd essentially
end up discarding BOM anywhere, which is more libertine than I care
to be.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Andrew Dunstan


On 03/24/2014 02:50 PM, Jim Nasby wrote:

On 3/22/14, 11:26 AM, Jim Nasby wrote:

On 3/21/14, 4:54 PM, Tom Lane wrote:

Merlin Moncure  writes:

There is no way for psql to handle that case though unless you'd strip
*all* BOMs encountered.  Compounding this problem is that there's no
practical way AFAIK to send multiple file to psql via single command
line invocation.  If you pass multiple -f arguments all but one is
ignored.


Well, that seems like a solvable but rather independent problem.
I guess one issue is how you'd define the meaning of --single ...
one transaction per run, or one per file?


Well, if you're catting multiple files into psql -1, you'd get all 
the files in one transaction, right? So I'd say that's what should 
happen.


It occurs to me that we're going about this the wrong way...

The error here isn't being generated by psql; it's generated by the 
backend. In the context of a statement (and not, say, a COPY command).


So instead of trying to handle this on the psql side[1], I think we 
need to handle it in the backend; specifically in the parser. Is there 
an easy way to get the parser to ignore the BOM character in the 
context of commands (but not in strings)?


[1]: Obviously, BOM could still screw up a psql command like \d. We'd 
want to address that as well; but I suspect backends are the more 
common scenario.



But what about COPY files? I don't see why there is less of a case for 
eating a leading BOM for a COPY file (or COPY stdin for that matter, 
given that it can come from \copy) than for an SQL file.


I suspect suspect trying to do this in the parser will be quite messy. 
This needs to happen before the input is converted to the server 
encoding, I think.


cheers

andrew



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql blows up on BOM character sequence

2014-03-24 Thread Jim Nasby

On 3/22/14, 11:26 AM, Jim Nasby wrote:

On 3/21/14, 4:54 PM, Tom Lane wrote:

Merlin Moncure  writes:

There is no way for psql to handle that case though unless you'd strip
*all* BOMs encountered.  Compounding this problem is that there's no
practical way AFAIK to send multiple file to psql via single command
line invocation.  If you pass multiple -f arguments all but one is
ignored.


Well, that seems like a solvable but rather independent problem.
I guess one issue is how you'd define the meaning of --single ...
one transaction per run, or one per file?


Well, if you're catting multiple files into psql -1, you'd get all the files in 
one transaction, right? So I'd say that's what should happen.


It occurs to me that we're going about this the wrong way...

The error here isn't being generated by psql; it's generated by the backend. In 
the context of a statement (and not, say, a COPY command).

So instead of trying to handle this on the psql side[1], I think we need to 
handle it in the backend; specifically in the parser. Is there an easy way to 
get the parser to ignore the BOM character in the context of commands (but not 
in strings)?

[1]: Obviously, BOM could still screw up a psql command like \d. We'd want to 
address that as well; but I suspect backends are the more common scenario.
--
Jim C. Nasby, Data Architect   j...@nasby.net
512.569.9461 (cell) http://jim.nasby.net


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Global flag

2014-03-24 Thread Jeff Janes
On Mon, Mar 24, 2014 at 2:53 AM, Swapnil Bhoite wrote:

> Hi,
>
> I want to set a *global flag* with which I can decide whether to use my
> code or not
> in modified source code.
> How I can do that?
>

edit src/backend/utils/misc/guc.c to add an external variable declaration
near "XXX these should appear in other modules' header file", and then add
a block for the variable down where it belongs based on the type of the
variable, by copying and modifying a related block.  Avoid copying a block
with a special validation hook, unless of course you need those.

Then where you want to use it, just add a declaration near the top of the
file, and use it where it needs to be used.

Now you can set it the same way you can set other configuration variables.
 If the block you copied and changed had "PGC_USERSET", you will be able to
change the setting inside each connection dynamically.

It is very easy and powerful once you get used to it.  I find the hardest
part is remembering which directory guc.c lives in.

Cheers,

Jeff


[HACKERS] About adding an attribute to pg_attibute

2014-03-24 Thread Tanmay Deshpande
We are trying add columnar support to postgreSQL and for that we need
identify each column in each relation by an oid. So my doubt is how do we
add an attribute/ a column to catalog pg_attribute ?


Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

2014-03-24 Thread Heikki Linnakangas
I came up with the attached patch, to reduce the WAL volume of GIN 
insertions. It become fairly large, but I guess that's not too 
surprising as the old WAL-logging method was basically to dump the whole 
page to WAL record. This is now a lot more fine-grained and smarter. I 
separated constructing the WAL record from copying the changes back to 
the disk page, which IMHO is a readability improvement even though it's 
more code.


There are two parts to this patch:

* leafRepackItems has been rewritten. The previous coding basically 
searched for the first modified item, and decoded and re-encoded 
everything on the page that after that. Now it tries harder to avoid 
re-encoding segments that are still reasonably sized (between 128 and 
384 bytes, with the target for new segments being 256 bytes). This ought 
to make random updates faster as a bonus, but I didn't performance test 
that.


* Track more carefully which segments on the page have been modified. 
The in-memory structure used to manipulate a page now keeps an action 
code for each segment, indicating if the segment is completely new, 
deleted, or replaced with new content, or if just some new items have 
been added to it. These same actions are WAL-logged, and replayed in the 
redo routine.


This brings the WAL volume back to the same ballpark as 9.3. Or better, 
depending on the operation.


Fujii, Alexander, how does this look to you?

- Heikki
diff --git a/src/backend/access/gin/gindatapage.c b/src/backend/access/gin/gindatapage.c
index 313d53c..56b456e 100644
--- a/src/backend/access/gin/gindatapage.c
+++ b/src/backend/access/gin/gindatapage.c
@@ -22,17 +22,17 @@
 #include "utils/rel.h"
 
 /*
- * Size of the posting lists stored on leaf pages, in bytes. The code can
- * deal with any size, but random access is more efficient when a number of
- * smaller lists are stored, rather than one big list.
- */
-#define GinPostingListSegmentMaxSize 256
-
-/*
- * Existing posting lists smaller than this are recompressed, when inserting
- * new items to page.
+ * Min, Max and Target size of posting lists stored on leaf pages, in bytes.
+ *
+ * The code can deal with any size, but random access is more efficient when
+ * a number of smaller lists are stored, rather than one big list. If a
+ * posting list would become larger than Max size as a result of insertions,
+ * it is split into two. If a posting list would be smaller than minimum
+ * size
  */
-#define GinPostingListSegmentMinSize 192
+#define GinPostingListSegmentMaxSize 384
+#define GinPostingListSegmentTargetSize 256
+#define GinPostingListSegmentMinSize 128
 
 /*
  * At least this many items fit in a GinPostingListSegmentMaxSize-bytes
@@ -55,12 +55,32 @@ typedef struct
 	dlist_node *lastleft;		/* last segment on left page */
 	int			lsize;			/* total size on left page */
 	int			rsize;			/* total size on right page */
+
+	bool		oldformat;		/* page is in pre-9.4 on disk */
 } disassembledLeaf;
 
 typedef struct
 {
 	dlist_node	node;		/* linked list pointers */
 
+	/*-
+	 * 'action' indicates the status of this in-memory segment, compared to
+	 * what's on disk. It is one of the GIN_SEGMENT_* action codes:
+	 *
+	 * UNMODIFIED	no changes
+	 * DELETED		the segment is to be removed. 'seg' and 'items' are
+	 *ignored
+	 * INSERT		this is a completely new segment
+	 * REPLACE		this replaces an existing segment with new content
+	 * ADDITEMS		like REPLACE, but we track in detail what items have
+	 *been added to this segment, in 'modifieditems'
+	 *-
+	 */
+	char		action;
+
+	ItemPointerData *modifieditems;
+	int			nmodifieditems;
+
 	/*
 	 * The following fields represent the items in this segment.
 	 * If 'items' is not NULL, it contains a palloc'd array of the items
@@ -72,8 +92,6 @@ typedef struct
 	GinPostingList *seg;
 	ItemPointer items;
 	int			nitems;			/* # of items in 'items', if items != NULL */
-
-	bool		modified;		/* is this segment on page already? */
 } leafSegmentInfo;
 
 static ItemPointer dataLeafPageGetUncompressed(Page page, int *nitems);
@@ -87,9 +105,9 @@ static bool leafRepackItems(disassembledLeaf *leaf, ItemPointer remaining);
 static bool addItemsToLeaf(disassembledLeaf *leaf, ItemPointer newItems, int nNewItems);
 
 
-static void dataPlaceToPageLeafRecompress(Buffer buf,
-			  disassembledLeaf *leaf,
-			  XLogRecData **prdata);
+static XLogRecData *constructLeafRecompressWALData(Buffer buf,
+			   disassembledLeaf *leaf);
+static void dataPlaceToPageLeafRecompress(Buffer buf, disassembledLeaf *leaf);
 static void dataPlaceToPageLeafSplit(Buffer buf,
 		 disassembledLeaf *leaf,
 		 ItemPointerData lbound, ItemPointerData rbound,
@@ -563,15 +581,21 @@ dataPlaceToPageLeaf(GinBtree btree, Buffer buf, GinBtreeStack *stack,
 	if (!needsplit)
 	{
 		/*
-		 * Great, all the items fit on a single page. Write the segments to
-		 * the page, and WAL-log appropriately.
+		 * Great, all the items fit on a single page. Construct a 

Re: [HACKERS] Useless "Replica Identity: NOTHING" noise from psql \d

2014-03-24 Thread Alvaro Herrera
Bruce Momjian wrote:
> On Mon, Mar 24, 2014 at 05:06:25PM +0100, Andres Freund wrote:
> > On 2014-03-22 23:47:57 -0400, Bruce Momjian wrote:
> > >   test=> \d+ test
> > >Table "public.test"
> > >Column |  Type   | Modifiers | Storage | Stats target | Description
> > >   +-+---+-+--+-
> > >x  | integer |   | plain   |  |
> > >   Replica Identity: full
> > >   Has OIDs: no
> > > 
> > > I used lower-case for the value, rather than all-caps.
> > 
> > Why? CLUSTER, PRIMARY KEY, etc. are displayed all caps, and replica
> > identity is similarly set via ALTER TABLE ... REPLICA IDENITY?
> 
> Oh, good points;  I had not considered PRIMARY KEY.  Updated patch
> attached.

In the "INDEX" case, should the output mention specifically which index
is being considered?



-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Useless "Replica Identity: NOTHING" noise from psql \d

2014-03-24 Thread Bruce Momjian
On Mon, Mar 24, 2014 at 05:06:25PM +0100, Andres Freund wrote:
> On 2014-03-22 23:47:57 -0400, Bruce Momjian wrote:
> > test=> \d+ test
> >  Table "public.test"
> >  Column |  Type   | Modifiers | Storage | Stats target | Description
> > +-+---+-+--+-
> >  x  | integer |   | plain   |  |
> > Replica Identity: full
> > Has OIDs: no
> > 
> > I used lower-case for the value, rather than all-caps.
> 
> Why? CLUSTER, PRIMARY KEY, etc. are displayed all caps, and replica
> identity is similarly set via ALTER TABLE ... REPLICA IDENITY?

Oh, good points;  I had not considered PRIMARY KEY.  Updated patch
attached.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
new file mode 100644
index a194ce7..f369e0e
*** a/src/bin/psql/describe.c
--- b/src/bin/psql/describe.c
*** describeOneTableDetails(const char *sche
*** 2345,2358 
  			printTableAddFooter(&cont, buf.data);
  		}
  
! 		if ((tableinfo.relkind == 'r' || tableinfo.relkind == 'm') &&
! 			tableinfo.relreplident != 'd' && tableinfo.relreplident != 'i')
  		{
  			const char *s = _("Replica Identity");
  
  			printfPQExpBuffer(&buf, "%s: %s",
  			  s,
! 			  tableinfo.relreplident == 'n' ? "NOTHING" : "FULL");
  			printTableAddFooter(&cont, buf.data);
  		}
  
--- 2345,2362 
  			printTableAddFooter(&cont, buf.data);
  		}
  
! 		if (verbose && (tableinfo.relkind == 'r' || tableinfo.relkind == 'm') &&
! 			strcmp(schemaname, "pg_catalog") != 0)
  		{
  			const char *s = _("Replica Identity");
  
  			printfPQExpBuffer(&buf, "%s: %s",
  			  s,
! 			  tableinfo.relreplident == 'd' ? "DEFAULT" :
! 			  tableinfo.relreplident == 'f' ? "FULL" :
! 			  tableinfo.relreplident == 'i' ? "INDEX" :
! 			  tableinfo.relreplident == 'n' ? "NOTHING" :
! 			  "???");
  			printTableAddFooter(&cont, buf.data);
  		}
  
diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out
new file mode 100644
index 5f29b39..feb6c93
*** a/src/test/regress/expected/create_table_like.out
--- b/src/test/regress/expected/create_table_like.out
*** CREATE TABLE ctlt12_storage (LIKE ctlt1
*** 115,120 
--- 115,121 
   a  | text | not null  | main |  | 
   b  | text |   | extended |  | 
   c  | text |   | external |  | 
+ Replica Identity: DEFAULT
  Has OIDs: no
  
  CREATE TABLE ctlt12_comments (LIKE ctlt1 INCLUDING COMMENTS, LIKE ctlt2 INCLUDING COMMENTS);
*** CREATE TABLE ctlt12_comments (LIKE ctlt1
*** 125,130 
--- 126,132 
   a  | text | not null  | extended |  | A
   b  | text |   | extended |  | B
   c  | text |   | extended |  | C
+ Replica Identity: DEFAULT
  Has OIDs: no
  
  CREATE TABLE ctlt1_inh (LIKE ctlt1 INCLUDING CONSTRAINTS INCLUDING COMMENTS) INHERITS (ctlt1);
*** NOTICE:  merging constraint "ctlt1_a_che
*** 140,145 
--- 142,148 
  Check constraints:
  "ctlt1_a_check" CHECK (length(a) > 2)
  Inherits: ctlt1
+ Replica Identity: DEFAULT
  Has OIDs: no
  
  SELECT description FROM pg_description, pg_constraint c WHERE classoid = 'pg_constraint'::regclass AND objoid = c.oid AND c.conrelid = 'ctlt1_inh'::regclass;
*** Check constraints:
*** 162,167 
--- 165,171 
  "ctlt3_a_check" CHECK (length(a) < 5)
  Inherits: ctlt1,
ctlt3
+ Replica Identity: DEFAULT
  Has OIDs: no
  
  CREATE TABLE ctlt13_like (LIKE ctlt3 INCLUDING CONSTRAINTS INCLUDING COMMENTS INCLUDING STORAGE) INHERITS (ctlt1);
*** Check constraints:
*** 177,182 
--- 181,187 
  "ctlt1_a_check" CHECK (length(a) > 2)
  "ctlt3_a_check" CHECK (length(a) < 5)
  Inherits: ctlt1
+ Replica Identity: DEFAULT
  Has OIDs: no
  
  SELECT description FROM pg_description, pg_constraint c WHERE classoid = 'pg_constraint'::regclass AND objoid = c.oid AND c.conrelid = 'ctlt13_like'::regclass;
*** Indexes:
*** 198,203 
--- 203,209 
  "ctlt_all_expr_idx" btree ((a || b))
  Check constraints:
  "ctlt1_a_check" CHECK (length(a) > 2)
+ Replica Identity: DEFAULT
  Has OIDs: no
  
  SELECT c.relname, objsubid, description FROM pg_description, pg_index i, pg_class c WHERE classoid = 'pg_class'::regclass AND objoid = i.indexrelid AND c.oid = i.indexrelid AND i.indrelid = 'ctlt_all'::regclass ORDER BY c.relname, objsubid;
diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out
new file mode 100644
index c84c435..7f2eeea
*** a/src/test/regress/expected/inherit.out
--- b/src/test/regress/expected/inherit.out
*** ALTER TABLE inhts 

Re: [HACKERS] Useless "Replica Identity: NOTHING" noise from psql \d

2014-03-24 Thread Andres Freund
On 2014-03-22 23:47:57 -0400, Bruce Momjian wrote:
>   test=> \d+ test
>Table "public.test"
>Column |  Type   | Modifiers | Storage | Stats target | Description
>   +-+---+-+--+-
>x  | integer |   | plain   |  |
>   Replica Identity: full
>   Has OIDs: no
> 
> I used lower-case for the value, rather than all-caps.

Why? CLUSTER, PRIMARY KEY, etc. are displayed all caps, and replica
identity is similarly set via ALTER TABLE ... REPLICA IDENITY?

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Useless "Replica Identity: NOTHING" noise from psql \d

2014-03-24 Thread Bruce Momjian
On Sun, Mar 23, 2014 at 11:49:37AM -0400, Tom Lane wrote:
> Bruce Momjian  writes:
> > Is this the patch you had in mind?  I kept the pg_catalog filter.  Do we
> > want to always show the replica identity line for \d+?
> 
> Doesn't seem like a great idea to remove the filter tests for replident
> values and then not fix the display code to cope with those values.
> 
> I think this display code is well south of minimal acceptability anyhow:
> if the column contains anything other than what it expects, it will print
> a lie, meaning it's entirely not future-proof.  I'd suggest a switch()
> that prints "???" in the default: case.

Oh, good points.  I have updated the attached patch.

-- 
  Bruce Momjian  http://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
new file mode 100644
index a194ce7..8542b93
*** a/src/bin/psql/describe.c
--- b/src/bin/psql/describe.c
*** describeOneTableDetails(const char *sche
*** 2345,2358 
  			printTableAddFooter(&cont, buf.data);
  		}
  
! 		if ((tableinfo.relkind == 'r' || tableinfo.relkind == 'm') &&
! 			tableinfo.relreplident != 'd' && tableinfo.relreplident != 'i')
  		{
  			const char *s = _("Replica Identity");
  
  			printfPQExpBuffer(&buf, "%s: %s",
  			  s,
! 			  tableinfo.relreplident == 'n' ? "NOTHING" : "FULL");
  			printTableAddFooter(&cont, buf.data);
  		}
  
--- 2345,2362 
  			printTableAddFooter(&cont, buf.data);
  		}
  
! 		if (verbose && (tableinfo.relkind == 'r' || tableinfo.relkind == 'm') &&
! 			strcmp(schemaname, "pg_catalog") != 0)
  		{
  			const char *s = _("Replica Identity");
  
  			printfPQExpBuffer(&buf, "%s: %s",
  			  s,
! 			  tableinfo.relreplident == 'd' ? "default" :
! 			  tableinfo.relreplident == 'f' ? "full" :
! 			  tableinfo.relreplident == 'i' ? "index" :
! 			  tableinfo.relreplident == 'n' ? "nothing" :
! 			  "???");
  			printTableAddFooter(&cont, buf.data);
  		}
  
diff --git a/src/test/regress/expected/create_table_like.out b/src/test/regress/expected/create_table_like.out
new file mode 100644
index 5f29b39..6fe51c0
*** a/src/test/regress/expected/create_table_like.out
--- b/src/test/regress/expected/create_table_like.out
*** CREATE TABLE ctlt12_storage (LIKE ctlt1
*** 115,120 
--- 115,121 
   a  | text | not null  | main |  | 
   b  | text |   | extended |  | 
   c  | text |   | external |  | 
+ Replica Identity: default
  Has OIDs: no
  
  CREATE TABLE ctlt12_comments (LIKE ctlt1 INCLUDING COMMENTS, LIKE ctlt2 INCLUDING COMMENTS);
*** CREATE TABLE ctlt12_comments (LIKE ctlt1
*** 125,130 
--- 126,132 
   a  | text | not null  | extended |  | A
   b  | text |   | extended |  | B
   c  | text |   | extended |  | C
+ Replica Identity: default
  Has OIDs: no
  
  CREATE TABLE ctlt1_inh (LIKE ctlt1 INCLUDING CONSTRAINTS INCLUDING COMMENTS) INHERITS (ctlt1);
*** NOTICE:  merging constraint "ctlt1_a_che
*** 140,145 
--- 142,148 
  Check constraints:
  "ctlt1_a_check" CHECK (length(a) > 2)
  Inherits: ctlt1
+ Replica Identity: default
  Has OIDs: no
  
  SELECT description FROM pg_description, pg_constraint c WHERE classoid = 'pg_constraint'::regclass AND objoid = c.oid AND c.conrelid = 'ctlt1_inh'::regclass;
*** Check constraints:
*** 162,167 
--- 165,171 
  "ctlt3_a_check" CHECK (length(a) < 5)
  Inherits: ctlt1,
ctlt3
+ Replica Identity: default
  Has OIDs: no
  
  CREATE TABLE ctlt13_like (LIKE ctlt3 INCLUDING CONSTRAINTS INCLUDING COMMENTS INCLUDING STORAGE) INHERITS (ctlt1);
*** Check constraints:
*** 177,182 
--- 181,187 
  "ctlt1_a_check" CHECK (length(a) > 2)
  "ctlt3_a_check" CHECK (length(a) < 5)
  Inherits: ctlt1
+ Replica Identity: default
  Has OIDs: no
  
  SELECT description FROM pg_description, pg_constraint c WHERE classoid = 'pg_constraint'::regclass AND objoid = c.oid AND c.conrelid = 'ctlt13_like'::regclass;
*** Indexes:
*** 198,203 
--- 203,209 
  "ctlt_all_expr_idx" btree ((a || b))
  Check constraints:
  "ctlt1_a_check" CHECK (length(a) > 2)
+ Replica Identity: default
  Has OIDs: no
  
  SELECT c.relname, objsubid, description FROM pg_description, pg_index i, pg_class c WHERE classoid = 'pg_class'::regclass AND objoid = i.indexrelid AND c.oid = i.indexrelid AND i.indrelid = 'ctlt_all'::regclass ORDER BY c.relname, objsubid;
diff --git a/src/test/regress/expected/inherit.out b/src/test/regress/expected/inherit.out
new file mode 100644
index c84c435..0083b4f
*** a/src/test/regress/expected/inherit.out
--- b/src/test/regress/expected/inherit.out
*** ALTER TABLE inhts RENAME d TO dd;
*** 913,918 
--- 913,919 
   dd 

Re: [HACKERS] [bug fix] PostgreSQL fails to start on Windows if it crashes after tablespace creation

2014-03-24 Thread MauMau

From: "Amit Kapila" 

1. Assume a tablespace tbs already exists.
2. Create table t1(c1 int) tablespace tbs;
3. drop table t1;
4. Drop tablespace tbs;
5. Do immediate shutdown (pg_ctl stop -mi);
6. During recovery it will create a table in directory (in function
   TablespaceCreateDbspace) which needs to be removed by
   destroy_tablespace_directories().

I am neither aware of, nor could think of such a case for
create_tablespace_directories(). Do you have any such case in mind
which I might be missing?


A bit contrived example is:

1. After the directory is created by TablespaceCreateDbspace(), recovery is 
stopped (e.g. due to power outage).  The directory remains.
2. Restart the server, redoing CREATE TABLESPACE during recovery, which 
executes create_tablespace_directories().



By saying above, I don't mean that your current patch has any
problem; even if there is no such scenario, I think your code is
right as stat/isdir check seems to be okay to identify junction
points and it avoids ifdef WIN32 check (which I personally think
is bit annoying and we should try to avoid such code unless it
is must or provides any significant advantage).


I think so, too.

Regards
MauMau



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Only first XLogRecData is visible to rm_desc with WAL_DEBUG

2014-03-24 Thread Tom Lane
Heikki Linnakangas  writes:
> As we all know, when you compile with WAL_DEBUG, and enable wal_debug 
> GUC, you get output like this in the log for every inserted WAL record:

> LOG:  INSERT @ 0/5407E578: prev 0/5407E4D0; xid 0; len 32: Standby - 
> running xacts: nextXid 774 latestCompletedXid 771 oldestRunningXid 772; 
> 2 xacts: 7877929 0

> What I did *not* realize before is that the redo routine only gets 
> passed the first XLogRecData struct in the chain. Because of that, the 
> "xacts: 7877929 0" information above is garbage. That's probably not the 
> only rm_desc routine that didn't get the memo.

Yeah, I recall having fixed at least one such bug in the past.

> There are a few alternatives on how to fix that:

> 1. Do nothing. Add a comment somewhere explaining that rm_redo cannot 
> safely look at data beyond what's inserted in the first XLogRecData.

We'll probably just make the same mistake again :-(.  And I think there
already is such a comment someplace.

> 2. Reconstruct the WAL data from the XLogRecData entries in a palloc'd 
> buffer, and pass that to rm_redo. That would be fairly expensive, but 
> you probably don't care about that if you've enabled wal_debug.

> 3. Remove the feature altogether, so that enabling wal_debug doesn't 
> cause all insertions to be logged anymore (no changes to the logging 
> during replay). It's a lot less interesting now that we have pg_xlogdump.

I think the main use-case for rm_desc anymore is making CONTEXT lines
for errors reported during WAL replay.  I guess that situation does not
have the same problem, since we've already loaded the complete WAL record.

However, I'm not sure how easy it's going to be for WAL_DEBUG to make the
data look the same as the replay case: in particular, substitution of
full-page-images for data would be tough to predict in advance (and moving
the printout into the critical section seems like a bad answer).

I'd be okay with removing WAL_DEBUG, I think, particularly in view of the
fact that there have been no requests to make it a compiled-by-default
feature.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Dynamic background workers & docs question

2014-03-24 Thread Magnus Hagander
On Mon, Mar 24, 2014 at 12:20 PM, Michael Paquier  wrote:

> On Mon, Mar 24, 2014 at 5:54 PM, Magnus Hagander 
> wrote:
> > I was looking at
> http://www.postgresql.org/docs/devel/static/bgworker.html
> > with a client today.
> >
> > It says:
> > "Unlike RegisterBackgroundWorker, which can only be called from within
> the
> > postmaster,RegisterDynamicBackgroundWorker must be called from a regular
> > backend."
> >
> > Is that the correct restriction? In particular, don't we allow calling
> > RegisterDynamicBackgroundWorker from another background worker? (In the
> > launcher/worker kind of scenario, like AutoVacuum).
> Yes, you can start a dynamic background worker from another background
> worker, have a look for example at contrib/worker_spi. Perhaps the
> correct wording would be "RegisterDynamicBackgroundWorker must be
> called from a regular backend or another background worker".
>
>
That's what I thought. Can a dynamic background worker start *another*
dynamic background worker, or can they only be started from "first level"
background workers?



> > Also:
> > "Background workers are expected to be continuously running; if they exit
> > cleanly, postgres will restart them immediately. "
> >
> > This doesn't apply to dynamic ones, which we might want to clarify. Do we
> > have a "term" for non-dynamic background workers? "static workers"?
> In the code or the documentation, there is no explicit
> differentiation, bgworkers are either called plainly "bgworker", or
> "dynamic bgworker". Perhaps the solution here is simply to say
> "background workers started by the postmaster are expected blabla".
>

That, or we need to invite a term for it?

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


Re: [HACKERS] Command line argument for Server

2014-03-24 Thread Tom Lane
Swapnil Bhoite  writes:
> I want to set or reset a global flag.
> 1) What currently I'm thinking is, I'll declare that global flag in
> *postgres.h*
> 2) I'll set that depending on my custom flag given at server startup
> 3) I encountered function *getopt* but I couldn't understand it

> So my question is how should I process command line argument for server?
> Or is there any another way to do so using *postgresql.conf*?

Forget about inventing your own flag mechanism.  Create a new GUC
parameter instead: there is lots and lots of existing infrastructure
you won't have to reinvent.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] New Vacancy for Permanent C# Developer in Kentish Town : ref:VAC-973549

2014-03-24 Thread Michael Glaze
Hi,

I'm currently looking for candidates who might be suitable for a C# Developer 
position based in Kentish Town and I wondered if you might know anyone who is 
interested. Salaries are around the £55000 level but might be flexible 
dependent on experience.

Software Developer / Programmer

London NW5 or central Bristol

£40,000 - £70,000 DOE plus benefits

The Company is an extremely successful and vibrant software company offering 
bespoke software development services in a diverse range of sectors including 
media, security, online retail, travel, finance, publishing and insurance. Our 
client base includes companies such as the BBC, Telegraph Media Group, Lonely 
Planet, Serco, Thales, McAfee, Panasonic and Allianz to name just a few.

We are currently looking for software developers with ideally some prior 
commercial experience to work in our offices based in London (NW5) and Bristol. 
We have placed in the top twenty of the Sunday Times Best Small Companies 
awards for the past three years running, and employees will tell you that this 
is a truly great - and unique - place to work. We combine a genuinely relaxed 
and friendly environment with a superb all-round package of benefits.

- Varied and interesting work that will keep you on your toes day in, day out.

- Great remuneration - all employees reap the rewards of our joint efforts

- Flexible working time and conditions and no pressure to regularly work long 
hours.

- Focused training, rapid personal development, and the opportunity to take on 
elevated levels of responsibility early in your career

- Lively and frequent social events and company outings.

If there's one thing we know, it's how to do software development properly. 
That means not just having the brightest staff but also great processes. All 
our managers are software developers, or ex-developers, and we make extensive 
use of Agile and test-driven development approaches. Every piece of output, 
including every line of code and every design, is peer reviewed to make sure 
not only that it meets our high standards, but that valuable experience is 
shared among our teams.

It's not easy getting into the company - we have over 30 applicants for every 
place offered - so you should be ready for a challenge. Our employees will tell 
you that it's worth it. The work will stretch you, there are ample career 
opportunities to take you where you want to go (tech lead or tech guru, project 
manager, line manager, tech consultant) and we hand 60% of profits back to our 
staff.

Applicants must have demonstrated excellence in previous employment and possess 
impeccable academic records including a good degree from a top university. You 
will need stand-out ability and enthusiasm in order to be successful.

I appreciate it is a long shot, but it occurs to me that good people know good 
people, and as I don't think you are looking I thought it would be worth 
running past you. Anyway, salaries are around the £55000 level but might be 
flexible dependent on experience.

If the match isn't right for you or anyone you then then I apologise in 
advance. Emails such as these are generated from a combination of keywords on 
your CV and other information that I've input against your record. If you 
aren't looking, are a contractor, have a higher salary, don't want to do .net 
development or are not happy with the location, then please let me know and 
I'll do my best to make sure that you aren't included in any further mailshots. 
Dealing with spam responses slows me down as well so I'm not deliberately 
mailshotting everyone on the planet.

Kind regards,

Michael Glaze

Michael Glaze
Head of Software Development
t: 0161 923 8171
e: michael.gl...@northpointrecruitment.com
w: www.northpointrecruitment.com
linkedin: 
http://www.linkedin.com/pub/michael-glaze/22/b78/618

This message is confidential. It may also be privileged or protected by other 
legal rules. If you have received this communication in error, please let us 
know by replying to Email Support then destroy it. You should not use, print, 
copy the message or disclose its contents to anyone. All ideas and concepts 
enclosed in this email are copyright of North Point Recruitment unless 
otherwise stated and cannot be used, copied or implemented without the express 
permission of North Point Recruitment and on payment of the agreed fees. North 
Point Recruitment, its directors and staff retain all intellectual property 
rights. e-mail is subject to possible data corruption, is not secure, and its 
content does not necessarily represent the opinion of North Point Recruitment. 
No representation or warranty is made as to the accuracy or completeness of the 
information and no liability can be accepted for any loss arising from its use. 
This e-mail and any attachments are not guaranteed to be free from so-called 
computer viruses, it is recommended

[HACKERS] Command line argument for Server

2014-03-24 Thread Swapnil Bhoite
Hi,

I want to set or reset a global flag.
1) What currently I'm thinking is, I'll declare that global flag in
*postgres.h*
2) I'll set that depending on my custom flag given at server startup
3) I encountered function *getopt* but I couldn't understand it

So my question is how should I process command line argument for server?
Or is there any another way to do so using *postgresql.conf*?

Thank you,
Swapnil


Re: [HACKERS] Patch for CREATE RULE sgml -- Was in: [DOCS]

2014-03-24 Thread Fujii Masao
On Sat, Mar 22, 2014 at 12:56 AM, Emanuel Calvo
 wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
>
>
> Hi guys,
>
> I realized that the output of the CREATE RULE has not a detailed
> output for the "events" parameter.
>
> But the question here is that I'm not sure which format follow:
>
> { INSERT | UPDATE | DELETE | SELECT}
>
> or
>
> INSERT
> UPDATE
> DELETE
> SELECT
> - --
>
>
> I attach a patch for each one.

Though I'm not sure the right policy of the format in synopsis, ISTM that
the following format is suitable in this case, i.e., if the value list
is very simple.
Patch attached.

SELECT | INSERT | UPDATE | DELETE

Regards,

-- 
Fujii Masao
*** a/doc/src/sgml/ref/create_rule.sgml
--- b/doc/src/sgml/ref/create_rule.sgml
***
*** 24,29  PostgreSQL documentation
--- 24,33 
  CREATE [ OR REPLACE ] RULE name AS ON event
  TO table_name [ WHERE condition ]
  DO [ ALSO | INSTEAD ] { NOTHING | command | ( command ; command ... ) }
+ 
+ where event can be one of:
+ 
+ SELECT | INSERT | UPDATE | DELETE
  
   
  

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Global flag

2014-03-24 Thread Craig Ringer
On 03/24/2014 06:28 PM, Rajashree Mandaogane wrote:
> I need to set a global flag in such a way that only if the flag is on
> then my modified code will get executed, so how can I do that?

PostgreSQL is single-threaded, multi-processing. So if you need to set
this flag only within a given backend, just make it a global variable.

If you want it to be global across all back-ends, you may want a GUC
(see the docs & source code). However, IIRC GUC changes aren't visible
to all backends immediately.

If you need something that's immediately visible, I imagine you'll want
a lock or mutex.

-- 
 Craig Ringer   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Standby server won't start

2014-03-24 Thread Fujii Masao
On Sat, Mar 22, 2014 at 9:33 AM, Tatsuo Ishii  wrote:
>> That's because the parameter is checked at the beginning of recovery
>> (i.e. at standby start) before XLOG_PARAMETER_CHANGE is received and
>> applied on the standby.  Please see CheckRequiredParameterValues() in
>> StartupXLOG().
>>
>> To persist the max_connections change:
>>
>> 1) stop primary
>> 2) change max_connections on the primary
>> 3) start primary
>> 4) watch pg_stat_replication to wait until the standby is sync with
>> the primary (XLOG_PARAMETER_CHANGE is applied)
>> 5) stop standby
>> 6) change max_connections on the standby
>> 7) start standby
>
> Unfotunately this did not work for me. pg_stat_replication showed
> replay_location and sent_location are identical, and I assume the
> standby is sync with the primary in step #4. Still the standby did not
> start in #7 with same error message I showed. This is PostgreSQL
> 9.3.3. Also pg_controldata  showed the old
> max_connections at #7. So I guess XLOG_PARAMETER_CHANGE has not been
> sent for some reason. Will look into this.

ISTM that's because WAL has not been flushed after XLOG_PARAMETER_CHANGE
is generated.  Attached patch fixes this problem.

Regards,

-- 
Fujii Masao
*** a/src/backend/access/transam/xlog.c
--- b/src/backend/access/transam/xlog.c
***
*** 8904,8909  XLogReportParameters(void)
--- 8904,8910 
  		{
  			XLogRecData rdata;
  			xl_parameter_change xlrec;
+ 			XLogRecPtr	recptr;
  
  			xlrec.MaxConnections = MaxConnections;
  			xlrec.max_worker_processes = max_worker_processes;
***
*** 8917,8923  XLogReportParameters(void)
  			rdata.len = sizeof(xlrec);
  			rdata.next = NULL;
  
! 			XLogInsert(RM_XLOG_ID, XLOG_PARAMETER_CHANGE, &rdata);
  		}
  
  		ControlFile->MaxConnections = MaxConnections;
--- 8918,8925 
  			rdata.len = sizeof(xlrec);
  			rdata.next = NULL;
  
! 			recptr = XLogInsert(RM_XLOG_ID, XLOG_PARAMETER_CHANGE, &rdata);
! 			XLogFlush(recptr);
  		}
  
  		ControlFile->MaxConnections = MaxConnections;

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Archive recovery won't be completed on some situation.

2014-03-24 Thread Fujii Masao
On Thu, Mar 20, 2014 at 3:43 PM, Kyotaro HORIGUCHI
 wrote:
> Hello,
>
> At Wed, 19 Mar 2014 19:34:10 +0900, Fujii Masao wrote
>> > Agreed. Attached patches do that and I could "recover" the
>> > database state with following steps,
>>
>> Adding new option looks like new feature rather than bug fix.
>> I'm afraid that the backpatch of such a change to 9.3 or before
>> is not acceptable.
>
> Me too. But on the other hand it simplly is a relief for the
> consequence of the behavior of server (altough it was ill
> operation:), and especially it is needed for at least 9.1 which
> seems cannot be saved without it. Plus it has utterly no impact
> on servers' behavior of any corresponding versions. So I hope it
> is accepted.

Even in 9.1, we can think that problematic situation as database corruption
and restart the server from the backup which was successfully taken before.
No?

Regards,

-- 
Fujii Masao


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Review: plpgsql.extra_warnings, plpgsql.extra_errors

2014-03-24 Thread Simon Riggs
On 24 March 2014 10:58, Petr Jelinek  wrote:

> Docs updated.

OK, it looks to me that all outstanding comments have been resolved.

I'll be looking to commit this later today, so last call for comments.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Dynamic background workers & docs question

2014-03-24 Thread Michael Paquier
On Mon, Mar 24, 2014 at 5:54 PM, Magnus Hagander  wrote:
> I was looking at http://www.postgresql.org/docs/devel/static/bgworker.html
> with a client today.
>
> It says:
> "Unlike RegisterBackgroundWorker, which can only be called from within the
> postmaster,RegisterDynamicBackgroundWorker must be called from a regular
> backend."
>
> Is that the correct restriction? In particular, don't we allow calling
> RegisterDynamicBackgroundWorker from another background worker? (In the
> launcher/worker kind of scenario, like AutoVacuum).
Yes, you can start a dynamic background worker from another background
worker, have a look for example at contrib/worker_spi. Perhaps the
correct wording would be "RegisterDynamicBackgroundWorker must be
called from a regular backend or another background worker".

> Also:
> "Background workers are expected to be continuously running; if they exit
> cleanly, postgres will restart them immediately. "
>
> This doesn't apply to dynamic ones, which we might want to clarify. Do we
> have a "term" for non-dynamic background workers? "static workers"?
In the code or the documentation, there is no explicit
differentiation, bgworkers are either called plainly "bgworker", or
"dynamic bgworker". Perhaps the solution here is simply to say
"background workers started by the postmaster are expected blabla".
-- 
Michael


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Review: plpgsql.extra_warnings, plpgsql.extra_errors

2014-03-24 Thread Petr Jelinek

On 23/03/14 19:38, Pavel Stehule wrote:



doc should be enhanced by:





Docs updated.

--
 Petr Jelinek  http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index bddd458..2a9b327 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -4711,6 +4711,54 @@ a_output := a_output || $$ if v_$$ || referrer_keys.kind || $$ like '$$
   
 
   
+  
+   Additional compile-time checks
+
+   
+To aid the user in finding instances of simple but common problems before
+they cause harm, PL/PgSQL provides additional
+checks. When enabled, depending on the configuration, they
+can be used to emit either a WARNING or an ERROR
+during the compilation of a function.
+   
+
+ 
+  These additional checks are enabled through the configuration variables
+  plpgsql.extra_warnings for warnings and 
+  plpgsql.extra_errors for errors. Both can be set either to
+  a comma-separated list of checks, "none" or "all".
+  The default is "none". Currently the list of available checks
+  includes only one:
+  
+   
+shadowed_variables
+
+ 
+  Checks if a declaration shadows a previously defined variable. 
+ 
+
+   
+  
+
+  The following example shows the effect of plpgsql.extra_warnings
+  set to shadowed_variables:
+
+SET plpgsql.extra_warnings TO 'shadowed_variables';
+
+CREATE FUNCTION foo(f1 int) RETURNS int AS $$
+DECLARE
+f1 int;
+BEGIN
+RETURN f1;
+END
+$$ LANGUAGE plpgsql;
+WARNING:  variable "f1" shadows a previously defined variable
+LINE 3: f1 int;
+^
+CREATE FUNCTION
+
+ 
+ 
  
 
   
diff --git a/src/pl/plpgsql/src/pl_comp.c b/src/pl/plpgsql/src/pl_comp.c
index 5afc2e5..12ac964 100644
--- a/src/pl/plpgsql/src/pl_comp.c
+++ b/src/pl/plpgsql/src/pl_comp.c
@@ -352,6 +352,9 @@ do_compile(FunctionCallInfo fcinfo,
 	function->out_param_varno = -1;		/* set up for no OUT param */
 	function->resolve_option = plpgsql_variable_conflict;
 	function->print_strict_params = plpgsql_print_strict_params;
+	/* only promote extra warnings and errors at CREATE FUNCTION time */
+	function->extra_warnings = forValidator ? plpgsql_extra_warnings : 0;
+	function->extra_errors = forValidator ? plpgsql_extra_errors : 0;
 
 	if (is_dml_trigger)
 		function->fn_is_trigger = PLPGSQL_DML_TRIGGER;
@@ -849,6 +852,9 @@ plpgsql_compile_inline(char *proc_source)
 	function->out_param_varno = -1;		/* set up for no OUT param */
 	function->resolve_option = plpgsql_variable_conflict;
 	function->print_strict_params = plpgsql_print_strict_params;
+	/* don't do extra validation for inline code as we don't want to add spam at runtime */
+	function->extra_warnings = 0;
+	function->extra_errors = 0;
 
 	plpgsql_ns_init();
 	plpgsql_ns_push(func_name);
diff --git a/src/pl/plpgsql/src/pl_gram.y b/src/pl/plpgsql/src/pl_gram.y
index c0cb585..91186c6 100644
--- a/src/pl/plpgsql/src/pl_gram.y
+++ b/src/pl/plpgsql/src/pl_gram.y
@@ -727,6 +727,21 @@ decl_varname	: T_WORD
 			  $1.ident, NULL, NULL,
 			  NULL) != NULL)
 			yyerror("duplicate declaration");
+
+		if (plpgsql_curr_compile->extra_warnings & PLPGSQL_XCHECK_SHADOWVAR ||
+			plpgsql_curr_compile->extra_errors & PLPGSQL_XCHECK_SHADOWVAR)
+		{
+			PLpgSQL_nsitem *nsi;
+			nsi = plpgsql_ns_lookup(plpgsql_ns_top(), false,
+	$1.ident, NULL, NULL, NULL);
+			if (nsi != NULL)
+ereport(plpgsql_curr_compile->extra_errors & PLPGSQL_XCHECK_SHADOWVAR ? ERROR : WARNING,
+		(errcode(ERRCODE_DUPLICATE_ALIAS),
+		 errmsg("variable \"%s\" shadows a previously defined variable",
+$1.ident),
+		 parser_errposition(@1)));
+		}
+
 	}
 | unreserved_keyword
 	{
@@ -740,6 +755,21 @@ decl_varname	: T_WORD
 			  $1, NULL, NULL,
 			  NULL) != NULL)
 			yyerror("duplicate declaration");
+
+		if (plpgsql_curr_compile->extra_warnings & PLPGSQL_XCHECK_SHADOWVAR ||
+			plpgsql_curr_compile->extra_errors & PLPGSQL_XCHECK_SHADOWVAR)
+		{
+			PLpgSQL_nsitem *nsi;
+			nsi = plpgsql_ns_lookup(plpgsql_ns_top(), false,
+	$1, NULL, NULL, NULL);
+			if (nsi != NULL)
+ereport(plpgsql_curr_compile->extra_errors & PLPGSQL_XCHECK_SHADOWVAR ? ERROR : WARNING,
+		(errcode(ERRCODE_DUPLICATE_ALIAS),
+		 errmsg("variable \"%s\" shadows a previously defined variable",
+$1),
+		 parser_errposition(@1)));
+		}
+
 	}
 ;
 
diff --git a/src/pl/plpgsql/src/pl_handler.c b/src/pl/plpgsql/src/pl_handler.c
index f21393a..e659f8e 100644
--- a/src/pl/plpgsql/src/pl_handler.c
+++ b/src/pl/plpgsql/src/pl_handler.c
@@ -25,6 +25,11 @@
 #include "utils/lsyscache.h"
 #include "utils/syscache.h"
 
+
+static bool plpgsql_extra_checks_check_hook(char **newvalue, void **extra, GucSource source);
+static void plpgsql_extra_warnings_assign_hook(const char *newvalue, void *extra

[HACKERS] Global flag

2014-03-24 Thread Rajashree Mandaogane
I need to set a global flag in such a way that only if the flag is on then
my modified code will get executed, so how can I do that?

Thank you.


Re: [HACKERS] using arrays within structure in ECPG

2014-03-24 Thread Boszormenyi Zoltan

2014-03-24 07:22 keltezéssel, Ashutosh Bapat írta:

Hi,
I tried using integer array within a structure array in ECPG code. But it resulted in 
some garbage values being printed from the table. Here are the details,


The ECPG program is attached (array_test.pgc). It tries to read the contents of table 
emp, whose structure and contents are as follows

postgres=# \d+ emp
   Table "public.emp"
 Column  |   Type| Modifiers | Storage  | Stats target | Description
-+---+---+--+--+-
 empno   | numeric(4,0)  |   | main |  |
 ename   | character varying |   | extended |  |
 job | character varying |   | extended |  |
 arr_col | integer[] |   | extended |  |
Has OIDs: no

postgres=# select * from emp;
 empno | ename  |   job   |  arr_col
---++-+
  7900 | JAMES  | CLERK   | {1,2,7900}
  7902 | FORD   | ANALYST | {1,2,7902}
  7934 | MILLER | CLERK   | {1,2,7934}
(3 rows)

You will notice that the last element of the arr_col array is same as the empno of that 
row.


The ECPG program tries to read the rows using FETCH in a structure emp defined 
as
 15 struct employee {
 16 int empno;
 17char ename[11];
 18char job[15];
 19int  arr_col[3];
 20 };

and then print the read contents as
 39 /* Print members of the structure. */
 40 for ( i = 0 ;i < 3; i++){
 41 printf("empno=%d, ename=%s, job=%s, arr_col[2]=%d\n", emp[i].empno, 
emp[i].ename, emp[i].job, emp[i].arr_col[2]);

 42
 43 }

But garbage values are printed
[ashutosh@ubuntu repro]./array_test

+++
empno=7900, ename=JAMES, job=CLERK, arr_col[2]=1
empno=2, ename=�, job=ANALYST, arr_col[2]=32767
empno=7934, ename=MILLER, job=CLERK, arr_col[2]=1719202336

Here are steps I have used to compile the ECPG program
[ashutosh@ubuntu repro]make array_test
ecpg -c -I/work/pg_head/build/include array_test.pgc
cc -I/work/pg_head/build/include -g   -c -o array_test.o array_test.c
cc -g  array_test.o  -L/work/pg_head/build/lib -lecpg -lpq -o array_test
rm array_test.o array_test.c

where /work/pg_head/build is the directory containing the postgresql build (essentially 
argument to the --prefix option to configure).


The programs compiles and links fine.

Without the arr_col member, the program works fine. So, it seems to be a problem with 
array within structure array.


In array_test.c I see that the ECPGdo statement corresponding to the FETCH command is as 
follows

 87 /* Fetch multiple columns into one structure. */
 88 { ECPGdo(__LINE__, 0, 1, NULL, 0, ECPGst_normal, "fetch 3 from cur1", 
ECPGt_EOIT,
 89 ECPGt_int,&(emp->empno),(long)1,(long)14,sizeof( struct employee ),
 90 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
 91 ECPGt_char,&(emp->ename),(long)11,(long)14,sizeof( struct employee ),
 92 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
 93 ECPGt_char,&(emp->job),(long)15,(long)14,sizeof( struct employee ),
 94 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L,
 95 ECPGt_int,(emp->arr_col),(long)1,(long)3,sizeof(int),
 96 ECPGt_NO_INDICATOR, NULL , 0L, 0L, 0L, ECPGt_EORT);

For all the members of struct employee, except arr_col, the size of array is set to 14 
and next member offset is set of sizeof (struct employee). But for arr_col they are set 
to 3 and sizeof(int) resp. So, for the next row onwards, the calculated offset of 
arr_col member would not coincide with the real arr_col member's address.


Am I missing something here?


ECPG (I think intentionally) doesn't interpret or parse array fields.
You need to pass a character array and parse the contents by yourself.

Best regards,
Zoltán Böszörményi


--
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company






[HACKERS] Global flag

2014-03-24 Thread Swapnil Bhoite
Hi,

I want to set a *global flag* with which I can decide whether to use my
code or not
in modified source code.
How I can do that?

Thank you
-Swapnil


[HACKERS] Only first XLogRecData is visible to rm_desc with WAL_DEBUG

2014-03-24 Thread Heikki Linnakangas
As we all know, when you compile with WAL_DEBUG, and enable wal_debug 
GUC, you get output like this in the log for every inserted WAL record:


LOG:  INSERT @ 0/5407E578: prev 0/5407E4D0; xid 0; len 32: Standby - 
running xacts: nextXid 774 latestCompletedXid 771 oldestRunningXid 772; 
2 xacts: 7877929 0


What I did *not* realize before is that the redo routine only gets 
passed the first XLogRecData struct in the chain. Because of that, the 
"xacts: 7877929 0" information above is garbage. That's probably not the 
only rm_desc routine that didn't get the memo.


There are a few alternatives on how to fix that:

1. Do nothing. Add a comment somewhere explaining that rm_redo cannot 
safely look at data beyond what's inserted in the first XLogRecData.


2. Reconstruct the WAL data from the XLogRecData entries in a palloc'd 
buffer, and pass that to rm_redo. That would be fairly expensive, but 
you probably don't care about that if you've enabled wal_debug.


3. Remove the feature altogether, so that enabling wal_debug doesn't 
cause all insertions to be logged anymore (no changes to the logging 
during replay). It's a lot less interesting now that we have pg_xlogdump.


Thoughts?

- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Dynamic background workers & docs question

2014-03-24 Thread Magnus Hagander
I was looking at
http://www.postgresql.org/docs/devel/static/bgworker.htmlwith a client
today.

It says:
"Unlike RegisterBackgroundWorker, which can only be called from within the
postmaster,RegisterDynamicBackgroundWorker must be called from a regular
backend."

Is that the correct restriction? In particular, don't we allow calling
RegisterDynamicBackgroundWorker from another background worker? (In the
launcher/worker kind of scenario, like AutoVacuum).

Also:
"Background workers are expected to be continuously running; if they exit
cleanly, postgres will restart them immediately. "

This doesn't apply to dynamic ones, which we might want to clarify. Do we
have a "term" for non-dynamic background workers? "static workers"?



--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


Re: [HACKERS] About adding a column to system catalog pg_attribute

2014-03-24 Thread Ashutosh Bapat
Why do you want to add OID to pg_attribute? The relid and attnum would be
able to uniquely identify each row of pg_attribute and hence each column of
any relation in the database.


On Mon, Mar 24, 2014 at 1:10 PM, Tanmay Deshpande
wrote:

> I want to add a column to system catalog pg_attribute for getting an oid
> for a column also. How do we add a column into pg_attribute ?
>



-- 
Best Wishes,
Ashutosh Bapat
EnterpriseDB Corporation
The Postgres Database Company


[HACKERS] About adding a column to system catalog pg_attribute

2014-03-24 Thread Tanmay Deshpande
I want to add a column to system catalog pg_attribute for getting an oid
for a column also. How do we add a column into pg_attribute ?


Re: [HACKERS] Triggers on foreign tables

2014-03-24 Thread Ronan Dunklau
Le dimanche 23 mars 2014 02:44:26 Noah Misch a écrit :
> On Tue, Mar 18, 2014 at 09:31:06AM +0100, Ronan Dunklau wrote:
> > Le mardi 18 mars 2014 03:54:19 Kouhei Kaigai a écrit :
> > > > (1) To acquire the old tuple for UPDATE/DELETE operations, the patch
> > > > closely
> >  
> >  parallels our handling for INSTEAD OF triggers on views.  It
> >  
> > > > adds a wholerow resjunk attribute, from which it constructs a
> > > > HeapTuple
> > > > before calling a trigger function.  This loses the system columns, an
> > > > irrelevant
> > > > consideration for views but meaningful for foreign tables. 
> > > > postgres_fdw
> > > > maintains the "ctid" system column (t_self), but triggers will always
> > > > see
> > > > an invalid t_self for the old tuple.  One way to fix this is to pass
> > > > around
> >  
> >  the old tuple data as ROW(ctid, oid, xmin, cmin, xmax, cmax,
> >  
> > > > tableoid, wholerow).  That's fairly close to sufficient, but it
> > > > doesn't
> > > > cover t_ctid. Frankly, I would like to find a principled excuse to not
> > > > worry about making foreign table system columns accessible from
> > > > triggers.
> > > > 
> > > >  Supporting system columns dramatically affects the mechanism, and
> > > >  what
> > > > 
> > > > trigger is likely to care?  Unfortunately, that argument seems too
> > > > weak.
> > > > Does anyone have a cleaner idea for keeping track of the system column
> > > > values or a stronger argument for not bothering?
> > > 
> > > Regarding to the first suggestion,
> > > I think, it is better not to care about system columns on foreign
> > > tables,
> > > because it fully depends on driver's implementation whether FDW fetches
> > > "ctid" from its data source, or not.
> > > Usually, system columns of foreign table, except for tableoid, are
> > > nonsense.
> >  
> >  Because of implementation reason, postgres_fdw fetches "ctid" of
> >  
> > > remote tables on UPDATE / DELETE, it is not a common nature for all FDW
> > > drivers. For example, we can assume an implementation that uses primary
> > > key
> > > of remote table to identify the record to be updated or deleted. In this
> > > case, local "ctid" does not have meaningful value.
> > > So, fundamentally, we cannot guarantee FDW driver returns meaningful
> > > "ctid"
> > > or other system columns.
> > 
> > I agree, I think it is somewhat clunky to have postgres_fdw use a feature
> > that is basically meaningless for other FDWs. Looking at some threads in
> > this list, it confused many people.
> 
> My own reasoning for accepting omission of system columns is more along the
> lines of Robert's argument.  Regardless, three folks voting to do so and
> none against suffices for me.  I documented the system columns limitation,
> made the returningList change I mentioned, and committed the patch.

Thank you, I'm glad the patch found its way to the repository !

> 
> > This is off-topic, but maybe we could devise an API allowing for local
> > "system attributes" on foreign table. This would allow FDWs to carry
> > attributes that weren't declared as part of the table definition. This
> > could then be used for postgres_fdw ctid, as well as others foreign data
> > wrappers equivalent of an implicit "tuple id".
> 
> We could, but I discourage it.  System columns are a legacy feature; I doubt
> we'd choose that design afresh today.  On the off-chance that you need the
> value of a remote system column, you can already declare an ordinary
> foreign table column for it.  I raised the issue because it's inconsistent
> for RETURNING to convey system columns while tg_trigtuple/tg_newtuple do
> not, not because acquiring system columns from foreign tables is notably
> useful.

The idea here was to allow an FDW author to add columns which are not part of 
the table definition, for example colums which are required to identify the 
tuple remotely. Without system columns, a postgres_fdw user would have to 
declare the ctid column on every table for a tuble to be identifiable. The 
proposal would allow postgres_fdw to automatically inject an hidden (system ?) 
column on every table for this ctid.

-- 
Ronan Dunklau
http://dalibo.com - http://dalibo.org

signature.asc
Description: This is a digitally signed message part.


[HACKERS] Comment in src/backend/commands/vacuumlazy.c

2014-03-24 Thread Amit Langote
Hi,

Note the following comment in src/backend/commands/vacuumlazy.c:lazy_scan_heap()

1088 /* If no indexes, make log report that lazy_vacuum_heap
would've made */
1089 if (vacuumed_pages)
1090 ereport(elevel,

Just wondering if it would read better as:

1088 /* Make the log report that lazy_vacuum_heap would've made
had there been no indexes */

Is that correct?

--
Amit


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers