Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?
On Fri, Aug 17, 2012 at 02:01:25PM -0400, Bruce Momjian wrote:
> On Fri, Aug 17, 2012 at 06:55:14PM +0100, Richard Huxton wrote:
> > >Well, it'd be nice to allow substitution there ...
> > >
> > >>What we can't easily do is to allow quotes to prevent variable
> > >>substitution in these whole-line commands because we can't process the
> > >>quotes because that will remove them.
> > >
> > >... but if there is then no way to prevent it, that's absolutely
> > >unacceptable.
> >
> > If I'm understanding this correctly, \copy parsing just passes the
> > query part unaltered as part of a COPY statement back into the
> > top-level parser. Likewise with the \!shell stuff (but presumably to
> > execve).
> >
> > To handle variable-substitution correctly for \copy we'd need to
> > duplicate the full parsing for COPY. For \! we'd need something
> > which understood shell-syntax (for the various shells out there).
> > Ick.
> >
> > Or you'd need a separate variable-bracketing {{:x}} syntax that
> > could work like reverse dollar-quoting. Also Ick.
> >
> > As far as we know this has only inconvenienced one person (me) badly
> > enough to report a maybe-bug. Thanks for trying Bruce, but I fear
> > this is one itch that'll go unscratched.
> >
> > Rest assured I'm not about to storm off and replace all my
> > installations with MySQL :-)
>
> Good analysis. Basically we can't hope to fully understand COPY or
> shell quoting syntax well enough to properly replace only unquoted psql
> variable references.
>
> Therefore, unless I hear otherwise, I will just document the limitation
> and withdraw the patch.
Patch withdrawn. Seems documentation was already in place --- I
clarified \! limitations match \copy.
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?
On Fri, Aug 17, 2012 at 06:55:14PM +0100, Richard Huxton wrote:
> >Well, it'd be nice to allow substitution there ...
> >
> >>What we can't easily do is to allow quotes to prevent variable
> >>substitution in these whole-line commands because we can't process the
> >>quotes because that will remove them.
> >
> >... but if there is then no way to prevent it, that's absolutely
> >unacceptable.
>
> If I'm understanding this correctly, \copy parsing just passes the
> query part unaltered as part of a COPY statement back into the
> top-level parser. Likewise with the \!shell stuff (but presumably to
> execve).
>
> To handle variable-substitution correctly for \copy we'd need to
> duplicate the full parsing for COPY. For \! we'd need something
> which understood shell-syntax (for the various shells out there).
> Ick.
>
> Or you'd need a separate variable-bracketing {{:x}} syntax that
> could work like reverse dollar-quoting. Also Ick.
>
> As far as we know this has only inconvenienced one person (me) badly
> enough to report a maybe-bug. Thanks for trying Bruce, but I fear
> this is one itch that'll go unscratched.
>
> Rest assured I'm not about to storm off and replace all my
> installations with MySQL :-)
Good analysis. Basically we can't hope to fully understand COPY or
shell quoting syntax well enough to properly replace only unquoted psql
variable references.
Therefore, unless I hear otherwise, I will just document the limitation
and withdraw the patch.
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?
On 17/08/12 18:38, Tom Lane wrote:
Bruce Momjian writes:
On Fri, Oct 21, 2011 at 05:31:41PM -0400, Robert Haas wrote:
I'm not sure whether that's a bug per se, but I can see where a
behavior change might be an improvement.
I did some research on this and learned a little more about flex rules.
Turns out we can allow variable substitution in psql whole-line
commands, like \copy and \!, by sharing the variable expansion flex
rules with the code that does argument processing.
Well, it'd be nice to allow substitution there ...
What we can't easily do is to allow quotes to prevent variable
substitution in these whole-line commands because we can't process the
quotes because that will remove them.
... but if there is then no way to prevent it, that's absolutely
unacceptable.
If I'm understanding this correctly, \copy parsing just passes the query
part unaltered as part of a COPY statement back into the top-level
parser. Likewise with the \!shell stuff (but presumably to execve).
To handle variable-substitution correctly for \copy we'd need to
duplicate the full parsing for COPY. For \! we'd need something which
understood shell-syntax (for the various shells out there). Ick.
Or you'd need a separate variable-bracketing {{:x}} syntax that could
work like reverse dollar-quoting. Also Ick.
As far as we know this has only inconvenienced one person (me) badly
enough to report a maybe-bug. Thanks for trying Bruce, but I fear this
is one itch that'll go unscratched.
Rest assured I'm not about to storm off and replace all my installations
with MySQL :-)
--
Richard Huxton
Archonet Ltd
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?
Bruce Momjian writes: > On Fri, Oct 21, 2011 at 05:31:41PM -0400, Robert Haas wrote: >> I'm not sure whether that's a bug per se, but I can see where a >> behavior change might be an improvement. > I did some research on this and learned a little more about flex rules. > Turns out we can allow variable substitution in psql whole-line > commands, like \copy and \!, by sharing the variable expansion flex > rules with the code that does argument processing. Well, it'd be nice to allow substitution there ... > What we can't easily do is to allow quotes to prevent variable > substitution in these whole-line commands because we can't process the > quotes because that will remove them. ... but if there is then no way to prevent it, that's absolutely unacceptable. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?
On Fri, Oct 21, 2011 at 05:31:41PM -0400, Robert Haas wrote:
> On Fri, Oct 21, 2011 at 7:24 AM, Richard Huxton wrote:
> > It looks like \copy is just passing the text of the query unadjusted to
> > "COPY". I get a syntax error on ":x" with the \copy below on both 9.0 and
> > 9.1
> >
> > === test script ===
> > \set x '''HELLO'''
> > -- Works
> > \echo :x
> > -- Works
> > \o '/tmp/test1.txt'
> > COPY (SELECT :x) TO STDOUT;
> > -- Doesn't work
> > \copy (SELECT :x) TO '/tmp/test2.txt'
> > === end script ===
>
> I'm not sure whether that's a bug per se, but I can see where a
> behavior change might be an improvement.
I did some research on this and learned a little more about flex rules.
Turns out we can allow variable substitution in psql whole-line
commands, like \copy and \!, by sharing the variable expansion flex
rules with the code that does argument processing.
What we can't easily do is to allow quotes to prevent variable
substitution in these whole-line commands because we can't process the
quotes because that will remove them.
Here are some examples; \copy and \! behave the same:
test=> \set x abc
test=> \echo :x
abc
test=> \echo ":x"
--> ":x"
test=> \! echo :x
abc
test=> \! echo ":x"
--> abc
Notice the last line has expanded :x even though it is in quotes.
So, what do we want? The attached patch is pretty short.
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ It's impossible for everything to be true. +
diff --git a/src/bin/psql/psqlscan.l b/src/bin/psql/psqlscan.l
new file mode 100644
index 1208c8f..3732dc5
*** a/src/bin/psql/psqlscan.l
--- b/src/bin/psql/psqlscan.l
*** other .
*** 934,949
}
! {
/*
* Default processing of text in a slash command's argument.
*
* Note: unquoted_option_chars counts the number of characters at the
* end of the argument that were not subject to any form of quoting.
* psql_scan_slash_option needs this to strip trailing semicolons safely.
*/
! {space}|"\\" {
/*
* Unquoted space is end of arg; do not eat. Likewise
* backslash is end of command or next command, do not eat
--- 934,956
}
!
/*
* Default processing of text in a slash command's argument.
+ * It shares token actions with xslasharg and xslashwholeline.
*
* Note: unquoted_option_chars counts the number of characters at the
* end of the argument that were not subject to any form of quoting.
* psql_scan_slash_option needs this to strip trailing semicolons safely.
*/
! {space}+ {
! /* process entire line, but suppress leading whitespace */
! if (output_buf->len > 0)
! ECHO;
! }
!
! {space}|"\\" {
/*
* Unquoted space is end of arg; do not eat. Likewise
* backslash is end of command or next command, do not eat
*** other .
*** 957,982
return LEXRES_OK;
}
! {quote} {
! *option_quote = '\'';
! unquoted_option_chars = 0;
! BEGIN(xslashquote);
! }
!
! "`"{
backtick_start_offset = output_buf->len;
*option_quote = '`';
unquoted_option_chars = 0;
BEGIN(xslashbackquote);
}
! {dquote} {
ECHO;
*option_quote = '"';
unquoted_option_chars = 0;
BEGIN(xslashdquote);
}
:{variable_char}+ {
/* Possible psql variable substitution */
if (option_type == OT_NO_EVAL)
--- 964,1005
return LEXRES_OK;
}
! "`" {
! /* Only in xslasharg, so backticks are potentially passed to the shell */
backtick_start_offset = output_buf->len;
*option_quote = '`';
unquoted_option_chars = 0;
BEGIN(xslashbackquote);
}
! {quote} {
! *option_quote = '\'';
! unquoted_option_chars = 0;
! BEGIN(xslashquote);
! }
!
! {dquote} {
ECHO;
*option_quote = '"';
unquoted_option_chars = 0;
BEGIN(xslashdquote);
}
+ {other} {
+ unquoted_option_chars++;
+ ECHO;
+ }
+
+ {other} { ECHO; }
+
+ /*
+ * This code allows variable processing in slasharg and wholeline
+ * modes. wholeline does not allow quoting to prevent variable
+ * subtitution because quote detection would remove the quotes.
+ */
+
+ {
+
:{variable_char}+ {
/* Possible psql variable substitution */
if (option_type == OT_NO_EVAL)
*** other .
*** 1044,1054
ECHO;
}
- {other} {
- unquoted_option_chars++;
- ECHO;
- }
-
}
{
--- 1067,1072
*** other .
*** 1115,1133
}
- {
- /* copy everything until end of input line */
- /* but suppress leading whitespace */
-
- {space}+ {
- if (output_buf->len > 0)
- ECHO;
- }
-
- {other} { ECHO; }
-
- }
-
{
/* at end of command, eat a double backslash, bu
Re: [HACKERS] psql \set vs \copy - bug or expected behaviour?
On Fri, Oct 21, 2011 at 7:24 AM, Richard Huxton wrote: > It looks like \copy is just passing the text of the query unadjusted to > "COPY". I get a syntax error on ":x" with the \copy below on both 9.0 and > 9.1 > > === test script === > \set x '''HELLO''' > -- Works > \echo :x > -- Works > \o '/tmp/test1.txt' > COPY (SELECT :x) TO STDOUT; > -- Doesn't work > \copy (SELECT :x) TO '/tmp/test2.txt' > === end script === I'm not sure whether that's a bug per se, but I can see where a behavior change might be an improvement. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] psql \set vs \copy - bug or expected behaviour?
It looks like \copy is just passing the text of the query unadjusted to "COPY". I get a syntax error on ":x" with the \copy below on both 9.0 and 9.1 === test script === \set x '''HELLO''' -- Works \echo :x -- Works \o '/tmp/test1.txt' COPY (SELECT :x) TO STDOUT; -- Doesn't work \copy (SELECT :x) TO '/tmp/test2.txt' === end script === -- Richard Huxton Archonet Ltd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
