date:20140904

Re: [HACKERS] Re: proposal: ignore null fields in not relation type composite type based constructors

2014-09-04 Thread Pavel Stehule

I am sory

too much patches

Regards

Pavel



2014-09-04 7:35 GMT+02:00 Jeevan Chalke jeevan.cha...@enterprisedb.com:

 Hi Pavel,

 You have attached wrong patch.


 Thanks
 --
 Jeevan B Chalke
 Principal Software Engineer, Product Development
 EnterpriseDB Corporation
 The Enterprise PostgreSQL Company


commit eb67c8d3e5e443d9cad1ef08fe2b4747eac933d9
Author: Pavel Stehule pavel.steh...@gooddata.com
Date:   Sat Jun 28 17:40:47 2014 +0200

as function with default parameters

diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 722640b..0d915c1 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -10294,11 +10294,12 @@ table2-mapping
   /row
   row
entry
- literalrow_to_json(record [, pretty_bool])/literal
+ literalrow_to_json(rowval record [, pretty bool [, ignore_nulls bool] ])/literal
/entry
entry
  Returns the row as a JSON object. Line feeds will be added between
- level-1 elements if parameterpretty_bool/parameter is true.
+ level-1 elements if parameterpretty_bool/parameter is true. Ignore
+ NULL when parameterignore_nulls/parameter is true.
/entry
entryliteralrow_to_json(row(1,'foo'))/literal/entry
entryliteral{f1:1,f2:foo}/literal/entry
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 1bde175..02cf965 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -867,3 +867,10 @@ RETURNS interval
 LANGUAGE INTERNAL
 STRICT IMMUTABLE
 AS 'make_interval';
+
+CREATE OR REPLACE FUNCTION
+  row_to_json(rowval record, pretty boolean DEFAULT false, ignore_nulls boolean DEFAULT false)
+RETURNS json
+LANGUAGE INTERNAL
+STRICT STABLE
+AS 'row_to_json';
diff --git a/src/backend/utils/adt/json.c b/src/backend/utils/adt/json.c
index 494a028..9f445ff 100644
--- a/src/backend/utils/adt/json.c
+++ b/src/backend/utils/adt/json.c
@@ -79,7 +79,8 @@ static void report_invalid_token(JsonLexContext *lex);
 static int	report_json_context(JsonLexContext *lex);
 static char *extract_mb_char(char *s);
 static void composite_to_json(Datum composite, StringInfo result,
-  bool use_line_feeds);
+  bool use_line_feeds,
+  bool ignore_nulls);
 static void array_dim_to_json(StringInfo result, int dim, int ndims, int *dims,
   Datum *vals, bool *nulls, int *valcount,
   JsonTypeCategory tcategory, Oid outfuncoid,
@@ -1362,7 +1363,7 @@ datum_to_json(Datum val, bool is_null, StringInfo result,
 			array_to_json_internal(val, result, false);
 			break;
 		case JSONTYPE_COMPOSITE:
-			composite_to_json(val, result, false);
+			composite_to_json(val, result, false, false);
 			break;
 		case JSONTYPE_BOOL:
 			outputstr = DatumGetBool(val) ? true : false;
@@ -1591,7 +1592,8 @@ array_to_json_internal(Datum array, StringInfo result, bool use_line_feeds)
  * Turn a composite / record into JSON.
  */
 static void
-composite_to_json(Datum composite, StringInfo result, bool use_line_feeds)
+composite_to_json(Datum composite, StringInfo result, bool use_line_feeds,
+bool ignore_nulls)
 {
 	HeapTupleHeader td;
 	Oid			tupType;
@@ -1630,6 +1632,12 @@ composite_to_json(Datum composite, StringInfo result, bool use_line_feeds)
 		if (tupdesc-attrs[i]-attisdropped)
 			continue;
 
+		val = heap_getattr(tuple, i + 1, tupdesc, isnull);
+
+		/* Don't serialize NULL field when we don't want it */
+		if (isnull  ignore_nulls)
+			continue;
+
 		if (needsep)
 			appendStringInfoString(result, sep);
 		needsep = true;
@@ -1638,8 +1646,6 @@ composite_to_json(Datum composite, StringInfo result, bool use_line_feeds)
 		escape_json(result, attname);
 		appendStringInfoChar(result, ':');
 
-		val = heap_getattr(tuple, i + 1, tupdesc, isnull);
-
 		if (isnull)
 		{
 			tcategory = JSONTYPE_NULL;
@@ -1721,34 +1727,19 @@ array_to_json_pretty(PG_FUNCTION_ARGS)
 }
 
 /*
- * SQL function row_to_json(row)
+ * SQL function row_to_json(rowval record, pretty bool, ignore_nulls bool)
  */
 extern Datum
 row_to_json(PG_FUNCTION_ARGS)
 {
 	Datum		array = PG_GETARG_DATUM(0);
-	StringInfo	result;
-
-	result = makeStringInfo();
-
-	composite_to_json(array, result, false);
-
-	PG_RETURN_TEXT_P(cstring_to_text_with_len(result-data, result-len));
-}
-
-/*
- * SQL function row_to_json(row, prettybool)
- */
-extern Datum
-row_to_json_pretty(PG_FUNCTION_ARGS)
-{
-	Datum		array = PG_GETARG_DATUM(0);
 	bool		use_line_feeds = PG_GETARG_BOOL(1);
+	bool		ignore_nulls = PG_GETARG_BOOL(2);
 	StringInfo	result;
 
 	result = makeStringInfo();
 
-	composite_to_json(array, result, use_line_feeds);
+	composite_to_json(array, result, use_line_feeds, ignore_nulls);
 
 	PG_RETURN_TEXT_P(cstring_to_text_with_len(result-data, result-len));
 }
diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h
index 5176ed0..5aeadc3 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -4203,10 +4203,8 @@ DATA(insert

Re: [HACKERS] PQputCopyEnd doesn't adhere to its API contract

2014-09-04 Thread David G Johnston

On Wed, Sep 3, 2014 at 6:25 PM, Bruce Momjian [via PostgreSQL]
ml-node+s1045698n5817646...@n5.nabble.com wrote:

On Fri, May 9, 2014 at 12:03:36PM -0400, Robert Haas wrote:

On Thu, May 8, 2014 at 5:21 PM, Tom Lane [hidden email]
http://user/SendEmail.jtp?type=nodenode=5817646i=0 wrote:
Perhaps the text should be like this:

The result is 1 if the termination message was sent; or in nonblocking
mode, this may only indicate that the termination message was
successfully
queued. (In nonblocking mode, to be certain that the data has been
sent,
you should next wait for write-ready and call functionPQflush/,
repeating until it returns zero.) Zero indicates that the function
could
not queue the termination message because of full buffers; this will
only
happen in nonblocking mode. (In this case, wait for write-ready and
try
the PQputCopyEnd call again.) If a hard error occurs, -1 is returned;
you
can use functionPQerrorMessage/function to retrieve details.

That looks pretty good. However, I'm realizing this isn't the only
place where we probably need to clarify the language. Just to take
one example near at hand, PQputCopyData may also return 1 when it's
only queued the data; it seems to try even less hard than PQputCopyEnd
to ensure that the data is actually sent.

Uh, where are we on this?

The comment for PQsetnonblocking in 31.4 says:

In the nonblocking state, calls to PQsendQuery, PQputline, PQputnbytes,
and PQendcopy will not block but instead return an error if they need to be
called again.

This is apparently false for PQendcopy - I did not go look at the others

I stand by my belief that someone who is using Non-Blocking Mode on read
should
understand how to proceed if they receive a
1 or 0
result from one of the put calls. A cross-reference to the relevant
section of the docs may be in order if that assumption is felt to be too
optimistic.
While it could possibly be more prominent the last sentence in 31.4
states:

After sending any command or data on a nonblocking connection, call
PQflush. If it returns 1, wait for the socket to be write-ready and call it
again; repeat until it returns 0. Once PQflush returns 0, wait for the
socket to be read-ready and then read the response as described above.

That said, I imagine a tip section for PQputCopyData may be in order if
the caller can gain efficiencies by filling up the queue before going and
polling with PQflush. I imagine this is the exact reason that the potential
for a 0 result exists. From the comment in 31.4 each call to PQputCopyData
should be followed by the call to PQflush...

As is my usual I decided to use my fresh perspective to see if I could
organize the material in a more structured, and in this case less
repetitive, way. Sending a diff/patch and a PDF of the result of make
html

I did not look at the 31.9.3 - Obsolete Functions section

Is there any particular rule-of-thumb for choosing 0 or zero that I
should consider? I tended to pick 0 in almost all cases and even fixed a
few otherwise untouched blocks of text.

The adjective network in ...to read from or write to the network
connection used by libpq. seems redundant...

David J.

P.S. I am also curious as to why the documents generated using make html
do not more closely (and there is quite a gap) match the style of the
online website

libpq_sgml_31_9_functions_associated_with_the_copy_command.diff (14K)
http://postgresql.1045698.n5.nabble.com/attachment/5817691/0/libpq_sgml_31_9_functions_associated_with_the_copy_command.diff
libpq_sgml_31_9_functions_associated_with_the_copy_command.pdf (263K)
http://postgresql.1045698.n5.nabble.com/attachment/5817691/1/libpq_sgml_31_9_functions_associated_with_the_copy_command.pdf

--
View this message in context:
http://postgresql.1045698.n5.nabble.com/PQputCopyEnd-doesn-t-adhere-to-its-API-contract-tp5803240p5817691.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-09-04 Thread Ants Aasma

On Sat, Aug 30, 2014 at 8:50 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Andres Freund and...@2ndquadrant.com writes:
On 2014-08-27 19:23:04 +0300, Heikki Linnakangas wrote:
A long time ago, Itagaki Takahiro wrote a patch sort the buffers and write
them out in order
(http://www.postgresql.org/message-id/flat/20070614153758.6a62.itagaki.takah...@oss.ntt.co.jp).
The performance impact of that was inconclusive, but one thing that it
allows nicely is to interleave the fsyncs, so that you write all the buffers
for one file, then fsync it, then next file and so on.

...
So, *very* clearly sorting is a benefit.

pg_bench alone doesn't convince me on this. The original thread found
cases where it was a loss, IIRC; you will need to test many more than
one scenario to prove the point.

The same objection came up last time I tried to push for sorted
checkpoints. I did not find any reference to where it caused a loss,
nor was I able to come up with a case where writing out in arbitrary
order would be better than writing out in file sequential order. In
fact if we ask for low latency this means that the OS must keep the
backlog small eliminating any chance of write combining writes that
arrive out of order.

I have a use case where the system continuously loads data into time
partitioned indexed tables, at every checkpoint all of the indexes of
the latest partition need to be written out. The only way I could get
the write out to happen with sequential I/O was to set
checkpoint_completion_target to zero and ensure OS cache allows for
enough dirty pages to absorb the whole checkpoint. The fsync that
followed did obviously nasty things to latency. Whereas sorted
checkpoints were able to do sequential I/O with checkpoint spreading
and low latency tuned OS virtual memory settings. I can create a
benchmark that shows this behavior if you need additional data points
to pgbench's OLTP workload to convince you that sorting checkpoint
writes is a good idea.

I did just come up with a case where plain sorting might cause an
issue. If the writes go to different I/O devices then naive sorting
will first use one device then the other, whereas arbitrary writing
will load balance between the devices. Assuming that separate
tablespaces are used for separate I/O devices, it should be enough to
just interleave writes of each tablespace, weighed by the amount of
writes per tablespace.

Regards,
Ants Aasma
--
Cybertec Schönig Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] postgresql latency bgwriter not doing its job

2014-09-04 Thread Ants Aasma

On Thu, Sep 4, 2014 at 12:36 AM, Andres Freund and...@2ndquadrant.com wrote:
 It's imo quite clearly better to keep it allocated. For one after
 postmaster started the checkpointer successfully you don't need to be
 worried about later failures to allocate memory if you allocate it once
 (unless the checkpointer FATALs out which should be exceedingly rare -
 we're catching ERRORs). It's much much more likely to succeed
 initially. Secondly it's not like there's really that much time where no
 checkpointer isn't running.

In principle you could do the sort with the full sized array and then
compress it to a list of buffer IDs that need to be written out. This
way most of the time you only need a small array and the large array
is only needed for a fraction of a second.

Regards,
Ants Aasma
-- 
Cybertec Schönig  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Re: [Fwd: Re: [HACKERS] proposal: new long psql parameter --on-error-stop]

2014-09-04 Thread Pavel Stehule

Hi

here is a second variant with support --help=variables

Regards

Pavel


2014-09-04 4:25 GMT+02:00 Robert Haas robertmh...@gmail.com:

 On Thu, Aug 28, 2014 at 11:20 AM, Andres Freund and...@2ndquadrant.com
 wrote:
  * How about making it --help=variables instead of --help-variables?
 
  -1, help is not a variable to be assigned imho
 
  I don't think variable assignment is a good mental model for long
  commandline arguments. And it's not like I'm the first to come up with
  an extensible --help. Check e.g. gcc.
 
  But anyway, I guess I've lost that argument.

 I think it mostly depends on how far we think we might extend it.  I
 mean, --help-variables is fine as a parallel to --help.  But if we're
 eventually going to have help for 12 things, --help=TOPIC is a lot
 better than 12 separate switches.  So +0.5 for your proposal from me.

 --
 Robert Haas
 EnterpriseDB: http://www.enterprisedb.com
 The Enterprise PostgreSQL Company

commit cac76b6135ae9f4b3c1eab1ce6bc34b43a7506ef
Author: Pavel Stehule pavel.steh...@gooddata.com
Date:   Wed Aug 27 22:47:07 2014 +0200

access to help_variables and usage from psql via psql command

diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml
index db314c3..05a0f01 100644
--- a/doc/src/sgml/ref/psql-ref.sgml
+++ b/doc/src/sgml/ref/psql-ref.sgml
@@ -560,11 +560,18 @@ EOF
 varlistentry
   termoption-?//term
   termoption--help//term
+  termoption--help=replaceable class=parametertopic/replaceable/option/term
   listitem
   para
   Show help about applicationpsql/application command line
   arguments, and exit.
   /para
+
+  para
+  option--help=variables/show help about applicationpsql/application variables,
+  and exit.
+  /para
+
   /listitem
 /varlistentry
 
@@ -2572,10 +2579,12 @@ testdb=gt; userinput\setenv LESS -imx4F/userinput
 
 
   varlistentry
-termliteral\?/literal/term
+termliteral\? [ options | variables ]/literal/term
 listitem
 para
-Shows help information about the backslash commands.
+Shows help information about the backslash commands.  This command can have a
+option variables or options to take help for psql configuration variables
+or psql command line options.
 /para
 /listitem
   /varlistentry
diff --git a/src/bin/psql/command.c b/src/bin/psql/command.c
index e16b4d5..987a79f 100644
--- a/src/bin/psql/command.c
+++ b/src/bin/psql/command.c
@@ -1503,7 +1503,19 @@ exec_command(const char *cmd,
 
 	/* \? -- slash command help */
 	else if (strcmp(cmd, ?) == 0)
-		slashUsage(pset.popt.topt.pager);
+	{
+		char	   *opt0 = psql_scan_slash_option(scan_state,
+	OT_NORMAL, NULL, false);
+
+		if (!opt0)
+			slashUsage(pset.popt.topt.pager);
+		else if (strcmp(opt0, variables) == 0)
+			help_variables(pset.popt.topt.pager);
+		else if (strcmp(opt0, options) == 0)
+			usage(pset.popt.topt.pager);
+		else
+			slashUsage(pset.popt.topt.pager);
+	}
 
 #if 0
 
diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c
index f8f000f..0ada015 100644
--- a/src/bin/psql/help.c
+++ b/src/bin/psql/help.c
@@ -46,11 +46,12 @@
 #define ON(var) (var ? _(on) : _(off))
 
 void
-usage(void)
+usage(unsigned short int pager)
 {
 	const char *env;
 	const char *user;
 	char	   *errstr;
+	FILE	   *output;
 
 	/* Find default user, in case we need it. */
 	user = getenv(PGUSER);
@@ -64,77 +65,82 @@ usage(void)
 		}
 	}
 
-	printf(_(psql is the PostgreSQL interactive terminal.\n\n));
-	printf(_(Usage:\n));
-	printf(_(  psql [OPTION]... [DBNAME [USERNAME]]\n\n));
+	output = PageOutput(59, pager);
+
+	fprintf(output, _(psql is the PostgreSQL interactive terminal.\n\n));
+	fprintf(output, _(Usage:\n));
+	fprintf(output, _(  psql [OPTION]... [DBNAME [USERNAME]]\n\n));
 
-	printf(_(General options:\n));
+	fprintf(output, _(General options:\n));
 	/* Display default database */
 	env = getenv(PGDATABASE);
 	if (!env)
 		env = user;
-	printf(_(  -c, --command=COMMANDrun only single command (SQL or internal) and exit\n));
-	printf(_(  -d, --dbname=DBNAME  database name to connect to (default: \%s\)\n), env);
-	printf(_(  -f, --file=FILENAME  execute commands from file, then exit\n));
-	printf(_(  -l, --list   list available databases, then exit\n));
-	printf(_(  -v, --set=, --variable=NAME=VALUE\n
-			set psql variable NAME to VALUE\n));
-	printf(_(  -V, --versionoutput version information, then exit\n));
-	printf(_(  -X, --no-psqlrc  do not read startup file (~/.psqlrc)\n));
-	printf(_(  -1 (\one\), --single-transaction\n
+	fprintf(output, _(  -c, --command=COMMANDrun only single command (SQL or internal) and exit\n));
+	fprintf(output, _(  -d, --dbname=DBNAME  database name to connect to (default: \%s\)\n), env);
+	fprintf(output, _(  -f, --file=FILENAME  execute commands from file, then exit\n));
+	fprintf(output, _(

Re: [HACKERS] implement subject alternative names support for SSL connections

2014-09-04 Thread Alexey Klyukin

On Wed, Sep 3, 2014 at 11:50 AM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:

 * It's ugly that the caller does the malloc and memcpy, and the
 certificate_name_entry_validate_match function then modifies its name
 argument. Move the malloc+memcpy inside the function.

For the case of CN the caller has to do the malloc anyway, since
X509_NAME_get_text_by_NID expects the already allocated buffer.
This means that 'movable' malloc + memcpy occurs only once, and while
it would still make sense to move it into the function, it would also
mean we would do an unnecessary malloc for the case of CN.


 * The error message in certificate_name_entry_validate_match says SSL
 certificate's common name contains embedded null even though it's also used
 for SANs.

Will fix, thank you.



 The tricky part is the error
 message if no match was found: initially, it only listed a single
 common name, but now tracking all DNS names just for the sake of the
 error message makes the code more bloated, so I'm wondering if simply
 stating that there was no match, as implemented in the attached patch,
 would be good enough?


 Hmm. It would still be nice to say something about the certificate that was
 received. How about:

   server certificate with common name %s does not match host name %s

We cannot guarantee at this point that the common name is present in
the certificate. And if it is not, which name should we pick instead?
One way to solve this is to take the first non-NULL name, but it if
there is no common name, but then the wording of the error message
would depend on the availability of CN.
Another is to show all available names, but I do not like collecting
them just for the sake of displaying in the error message.
And last one is to just show the error without mentioning names,
that's what I've chosen to be the most consistent.

Regards,
-- 
Alexey Klyukin


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

1 2 >

1 - 100 of 128 matches

Mail list logo