Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?

2012-08-25 Thread Bruce Momjian
On Fri, Aug 17, 2012 at 02:01:25PM -0400, Bruce Momjian wrote:
 On Fri, Aug 17, 2012 at 06:55:14PM +0100, Richard Huxton wrote:
  Well, it'd be nice to allow substitution there ...
  
  What we can't easily do is to allow quotes to prevent variable
  substitution in these whole-line commands because we can't process the
  quotes because that will remove them.
  
  ... but if there is then no way to prevent it, that's absolutely
  unacceptable.
  
  If I'm understanding this correctly, \copy parsing just passes the
  query part unaltered as part of a COPY statement back into the
  top-level parser. Likewise with the \!shell stuff (but presumably to
  execve).
  
  To handle variable-substitution correctly for \copy we'd need to
  duplicate the full parsing for COPY. For \! we'd need something
  which understood shell-syntax (for the various shells out there).
  Ick.
  
  Or you'd need a separate variable-bracketing {{:x}} syntax that
  could work like reverse dollar-quoting. Also Ick.
  
  As far as we know this has only inconvenienced one person (me) badly
  enough to report a maybe-bug. Thanks for trying Bruce, but I fear
  this is one itch that'll go unscratched.
  
  Rest assured I'm not about to storm off and replace all my
  installations with MySQL :-)
 
 Good analysis.  Basically we can't hope to fully understand COPY or
 shell quoting syntax well enough to properly replace only unquoted psql
 variable references.
 
 Therefore, unless I hear otherwise, I will just document the limitation
 and withdraw the patch.

Patch withdrawn.  Seems documentation was already in place --- I
clarified \! limitations match \copy.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?

2012-08-17 Thread Tom Lane
Bruce Momjian br...@momjian.us writes:
 On Fri, Oct 21, 2011 at 05:31:41PM -0400, Robert Haas wrote:
 I'm not sure whether that's a bug per se, but I can see where a
 behavior change might be an improvement.

 I did some research on this and learned a little more about flex rules.

 Turns out we can allow variable substitution in psql whole-line
 commands, like \copy and \!, by sharing the variable expansion flex
 rules with the code that does argument processing.  

Well, it'd be nice to allow substitution there ...

 What we can't easily do is to allow quotes to prevent variable
 substitution in these whole-line commands because we can't process the
 quotes because that will remove them.

... but if there is then no way to prevent it, that's absolutely
unacceptable.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?

2012-08-17 Thread Richard Huxton

On 17/08/12 18:38, Tom Lane wrote:

Bruce Momjianbr...@momjian.us  writes:

On Fri, Oct 21, 2011 at 05:31:41PM -0400, Robert Haas wrote:

I'm not sure whether that's a bug per se, but I can see where a
behavior change might be an improvement.



I did some research on this and learned a little more about flex rules.



Turns out we can allow variable substitution in psql whole-line
commands, like \copy and \!, by sharing the variable expansion flex
rules with the code that does argument processing.


Well, it'd be nice to allow substitution there ...


What we can't easily do is to allow quotes to prevent variable
substitution in these whole-line commands because we can't process the
quotes because that will remove them.


... but if there is then no way to prevent it, that's absolutely
unacceptable.


If I'm understanding this correctly, \copy parsing just passes the query 
part unaltered as part of a COPY statement back into the top-level 
parser. Likewise with the \!shell stuff (but presumably to execve).


To handle variable-substitution correctly for \copy we'd need to 
duplicate the full parsing for COPY. For \! we'd need something which 
understood shell-syntax (for the various shells out there). Ick.


Or you'd need a separate variable-bracketing {{:x}} syntax that could 
work like reverse dollar-quoting. Also Ick.


As far as we know this has only inconvenienced one person (me) badly 
enough to report a maybe-bug. Thanks for trying Bruce, but I fear this 
is one itch that'll go unscratched.


Rest assured I'm not about to storm off and replace all my installations 
with MySQL :-)


--
  Richard Huxton
  Archonet Ltd


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?

2012-08-17 Thread Bruce Momjian
On Fri, Aug 17, 2012 at 06:55:14PM +0100, Richard Huxton wrote:
 Well, it'd be nice to allow substitution there ...
 
 What we can't easily do is to allow quotes to prevent variable
 substitution in these whole-line commands because we can't process the
 quotes because that will remove them.
 
 ... but if there is then no way to prevent it, that's absolutely
 unacceptable.
 
 If I'm understanding this correctly, \copy parsing just passes the
 query part unaltered as part of a COPY statement back into the
 top-level parser. Likewise with the \!shell stuff (but presumably to
 execve).
 
 To handle variable-substitution correctly for \copy we'd need to
 duplicate the full parsing for COPY. For \! we'd need something
 which understood shell-syntax (for the various shells out there).
 Ick.
 
 Or you'd need a separate variable-bracketing {{:x}} syntax that
 could work like reverse dollar-quoting. Also Ick.
 
 As far as we know this has only inconvenienced one person (me) badly
 enough to report a maybe-bug. Thanks for trying Bruce, but I fear
 this is one itch that'll go unscratched.
 
 Rest assured I'm not about to storm off and replace all my
 installations with MySQL :-)

Good analysis.  Basically we can't hope to fully understand COPY or
shell quoting syntax well enough to properly replace only unquoted psql
variable references.

Therefore, unless I hear otherwise, I will just document the limitation
and withdraw the patch.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?

2012-08-16 Thread Bruce Momjian
On Fri, Oct 21, 2011 at 05:31:41PM -0400, Robert Haas wrote:
 On Fri, Oct 21, 2011 at 7:24 AM, Richard Huxton d...@archonet.com wrote:
  It looks like \copy is just passing the text of the query unadjusted to
  COPY. I get a syntax error on :x with the \copy below on both 9.0 and
  9.1
 
  === test script ===
  \set x '''HELLO'''
  -- Works
  \echo :x
  -- Works
  \o '/tmp/test1.txt'
  COPY (SELECT :x) TO STDOUT;
  -- Doesn't work
  \copy (SELECT :x) TO '/tmp/test2.txt'
  === end script ===
 
 I'm not sure whether that's a bug per se, but I can see where a
 behavior change might be an improvement.

I did some research on this and learned a little more about flex rules.

Turns out we can allow variable substitution in psql whole-line
commands, like \copy and \!, by sharing the variable expansion flex
rules with the code that does argument processing.  

What we can't easily do is to allow quotes to prevent variable
substitution in these whole-line commands because we can't process the
quotes because that will remove them.

Here are some examples;  \copy and \! behave the same:

test= \set x abc
test= \echo :x
abc
test= \echo :x
-- :x
test= \! echo :x
abc
test= \! echo :x
-- abc

Notice the last line has expanded :x even though it is in quotes.

So, what do we want?  The attached patch is pretty short.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +
diff --git a/src/bin/psql/psqlscan.l b/src/bin/psql/psqlscan.l
new file mode 100644
index 1208c8f..3732dc5
*** a/src/bin/psql/psqlscan.l
--- b/src/bin/psql/psqlscan.l
*** other			.
*** 934,949 
  
  }
  
! xslasharg{
  	/*
  	 * Default processing of text in a slash command's argument.
  	 *
  	 * Note: unquoted_option_chars counts the number of characters at the
  	 * end of the argument that were not subject to any form of quoting.
  	 * psql_scan_slash_option needs this to strip trailing semicolons safely.
  	 */
  
! {space}|\\	{
  	/*
  	 * Unquoted space is end of arg; do not eat.  Likewise
  	 * backslash is end of command or next command, do not eat
--- 934,956 
  
  }
  
! 
  	/*
  	 * Default processing of text in a slash command's argument.
+ 	 * It shares token actions with xslasharg and xslashwholeline.
  	 *
  	 * Note: unquoted_option_chars counts the number of characters at the
  	 * end of the argument that were not subject to any form of quoting.
  	 * psql_scan_slash_option needs this to strip trailing semicolons safely.
  	 */
  
! xslashwholeline{space}+	{
! 	/* process entire line, but suppress leading whitespace */
! 	if (output_buf-len  0)
! 		ECHO;
! }
! 
! xslasharg{space}|\\	{
  	/*
  	 * Unquoted space is end of arg; do not eat.  Likewise
  	 * backslash is end of command or next command, do not eat
*** other			.
*** 957,982 
  	return LEXRES_OK;
  }
  
! {quote}			{
! 	*option_quote = '\'';
! 	unquoted_option_chars = 0;
! 	BEGIN(xslashquote);
! }
! 
! `{
  	backtick_start_offset = output_buf-len;
  	*option_quote = '`';
  	unquoted_option_chars = 0;
  	BEGIN(xslashbackquote);
  }
  
! {dquote}		{
  	ECHO;
  	*option_quote = '';
  	unquoted_option_chars = 0;
  	BEGIN(xslashdquote);
  }
  
  :{variable_char}+	{
  	/* Possible psql variable substitution */
  	if (option_type == OT_NO_EVAL)
--- 964,1005 
  	return LEXRES_OK;
  }
  
! xslasharg`		{
! /* Only in xslasharg, so backticks are potentially passed to the shell */
  	backtick_start_offset = output_buf-len;
  	*option_quote = '`';
  	unquoted_option_chars = 0;
  	BEGIN(xslashbackquote);
  }
  
! xslasharg{quote}			{
! 	*option_quote = '\'';
! 	unquoted_option_chars = 0;
! 	BEGIN(xslashquote);
! }
! 
! xslasharg{dquote}		{
  	ECHO;
  	*option_quote = '';
  	unquoted_option_chars = 0;
  	BEGIN(xslashdquote);
  }
  
+ xslasharg{other}	{
+ 	unquoted_option_chars++;
+ 	ECHO;
+ }
+ 
+ xslashwholeline{other}	{ ECHO; }
+ 
+ 	/*
+ 	 *	This code allows variable processing in slasharg and wholeline
+ 	 *	modes.  wholeline does not allow quoting to prevent variable
+ 	 *	subtitution because quote detection would remove the quotes.
+ 	 */
+  
+ xslasharg,xslashwholeline{
+ 
  :{variable_char}+	{
  	/* Possible psql variable substitution */
  	if (option_type == OT_NO_EVAL)
*** other			.
*** 1044,1054 
  	ECHO;
  }
  
- {other}			{
- 	unquoted_option_chars++;
- 	ECHO;
- }
- 
  }
  
  xslashquote{
--- 1067,1072 
*** other			.
*** 1115,1133 
  
  }
  
- xslashwholeline{
- 	/* copy everything until end of input line */
- 	/* but suppress leading whitespace */
- 
- {space}+		{
- 	if (output_buf-len  0)
- 		

Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?

2011-10-21 Thread Robert Haas
On Fri, Oct 21, 2011 at 7:24 AM, Richard Huxton d...@archonet.com wrote:
 It looks like \copy is just passing the text of the query unadjusted to
 COPY. I get a syntax error on :x with the \copy below on both 9.0 and
 9.1

 === test script ===
 \set x '''HELLO'''
 -- Works
 \echo :x
 -- Works
 \o '/tmp/test1.txt'
 COPY (SELECT :x) TO STDOUT;
 -- Doesn't work
 \copy (SELECT :x) TO '/tmp/test2.txt'
 === end script ===

I'm not sure whether that's a bug per se, but I can see where a
behavior change might be an improvement.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers