Re: [HACKERS] YAML Was: CommitFest status/management
Tom Lane wrote: Greg Smith g...@2ndquadrant.com writes: To be clear about which version we're talking about: http://archives.postgresql.org/message-id/20091130123456.4a03.52131...@oss.ntt.co.jp is the candidate for commit that includes the cleanup you've already done. I suppose this is subject to the same auto_explain problem already noticed for JSON, but I would suggest that we commit this first and then fix both together, rather than create merge issues. OK, I've committed the YAML stuff, so who is working on the auto-explain bug? Robert? I'd like that fixed before I add in the query text to auto-explain output. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
On Thu, Dec 10, 2009 at 8:39 PM, Andrew Dunstan and...@dunslane.net wrote:
OK, I've committed the YAML stuff, so who is working on the auto-explain
bug? Robert? I'd like that fixed before I add in the query text to
auto-explain output.
I'm going to propose fixing this in what seems to me to be the
simplest possible way, per the attached. Anyone want to argue?
...Robert
diff --git a/contrib/auto_explain/auto_explain.c b/contrib/auto_explain/auto_explain.c
index 75ac9ca..f0d907d 100644
--- a/contrib/auto_explain/auto_explain.c
+++ b/contrib/auto_explain/auto_explain.c
@@ -223,7 +223,9 @@ explain_ExecutorEnd(QueryDesc *queryDesc)
es.verbose = auto_explain_log_verbose;
es.format = auto_explain_log_format;
+ ExplainBeginOutput(es);
ExplainPrintPlan(es, queryDesc);
+ ExplainEndOutput(es);
/* Remove last line break */
if (es.str-len 0 es.str-data[es.str-len - 1] == '\n')
diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c
index 0437ffa..2067636 100644
--- a/src/backend/commands/explain.c
+++ b/src/backend/commands/explain.c
@@ -91,8 +91,6 @@ static void ExplainCloseGroup(const char *objtype, const char *labelname,
bool labeled, ExplainState *es);
static void ExplainDummyGroup(const char *objtype, const char *labelname,
ExplainState *es);
-static void ExplainBeginOutput(ExplainState *es);
-static void ExplainEndOutput(ExplainState *es);
static void ExplainXMLTag(const char *tagname, int flags, ExplainState *es);
static void ExplainJSONLineEnding(ExplainState *es);
static void ExplainYAMLLineStarting(ExplainState *es);
@@ -1791,7 +1789,7 @@ ExplainDummyGroup(const char *objtype, const char *labelname, ExplainState *es)
* This is just enough different from processing a subgroup that we need
* a separate pair of subroutines.
*/
-static void
+void
ExplainBeginOutput(ExplainState *es)
{
switch (es-format)
@@ -1822,7 +1820,7 @@ ExplainBeginOutput(ExplainState *es)
/*
* Emit the end-of-output boilerplate.
*/
-static void
+void
ExplainEndOutput(ExplainState *es)
{
switch (es-format)
diff --git a/src/include/commands/explain.h b/src/include/commands/explain.h
index ab48825..ba2ba08 100644
--- a/src/include/commands/explain.h
+++ b/src/include/commands/explain.h
@@ -65,6 +65,8 @@ extern void ExplainOnePlan(PlannedStmt *plannedstmt, ExplainState *es,
extern void ExplainPrintPlan(ExplainState *es, QueryDesc *queryDesc);
+extern void ExplainBeginOutput(ExplainState *es);
+extern void ExplainEndOutput(ExplainState *es);
extern void ExplainSeparatePlans(ExplainState *es);
#endif /* EXPLAIN_H */
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Robert Haas robertmh...@gmail.com wrote: On Thu, Dec 10, 2009 at 8:39 PM, Andrew Dunstan and...@dunslane.net wrote: OK, I've committed the YAML stuff, so who is working on the auto-explain bug? Robert? I'm going to propose fixing this in what seems to me to be the simplest possible way, per the attached. Anyone want to argue? +1 to the fix. Typical usage of explain functions will be: 1. ExplainInitState() 2. (setup ExplainState) 3. ExplainBeginOutput() 4. ExplainXXX() except ExplainQuery() 5. ExplainEndOutput() Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Can I ask the final decision whether the YAML formatter should be applied or rejected? As far as I read the discussion, we can apply it because serveral users want it and we don't have a plan to support extensible formatters in the core. Greg Smith g...@2ndquadrant.com wrote: -The patch is small to apply -Having one more legacy format to satisfy would actually improve the odds that a future modular EXPLAIN would be robustly designed -While it's questionable whether it's strictly a majority on voting here, it's close, which suggests there is plenty of support for wanting this feature -Since nothing is removed the people who aren't in favor of this format aren't negatively impacted by it being committed Comments from additional discussion: - YAML format is the second (or the best for some people) human-unreadabe format for EXPLAIN output. - JSON is not always a subset of YAML. It is not recommended to generate/parse YAML with a JSON generator/parser or vice versa. - We won't add any core hooks and plug-in to format EXPLAIN output. Client tools may convert the output if needed. (ex. using XLST) - The spec of YAML might be changed in the feature, but we use only the basic parts. The parts will not be changed. Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Takahiro Itagaki wrote: Can I ask the final decision whether the YAML formatter should be applied or rejected? As far as I read the discussion, we can apply it because serveral users want it and we don't have a plan to support extensible formatters in the core. The path I thought made sense at this point was to mark the patch ready for a committer, since it sounds like everyone is done with it now, and have another committer besides yourself do a final review as part of that. At this point, I think we've justified the feature and confirmed the feature works. Given the controversy, I think another set of eyes to make sure it's not going to be a maintenance headache moving forward should (as usual) have the final say on whether the code goes in or not, because that's only drawback to it left to committing it I see at this point. To be clear about which version we're talking about: http://archives.postgresql.org/message-id/20091130123456.4a03.52131...@oss.ntt.co.jp is the candidate for commit that includes the cleanup you've already done. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Greg Smith wrote: Takahiro Itagaki wrote: Can I ask the final decision whether the YAML formatter should be applied or rejected? As far as I read the discussion, we can apply it because serveral users want it and we don't have a plan to support extensible formatters in the core. The path I thought made sense at this point was to mark the patch ready for a committer, since it sounds like everyone is done with it now, and have another committer besides yourself do a final review as part of that. That brings us back to where this conversation started ;-) I'll pick it up. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Greg Smith g...@2ndquadrant.com writes: To be clear about which version we're talking about: http://archives.postgresql.org/message-id/20091130123456.4a03.52131...@oss.ntt.co.jp is the candidate for commit that includes the cleanup you've already done. I suppose this is subject to the same auto_explain problem already noticed for JSON, but I would suggest that we commit this first and then fix both together, rather than create merge issues. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
2009/12/7 Itagaki Takahiro itagaki.takah...@oss.ntt.co.jp: Tom Lane t...@sss.pgh.pa.us wrote: It was written and submitted by one person who did not bother to ask first whether anyone else thought it was worthwhile. So its presence on the CF list should not be taken as evidence that there's consensus for it. Should we have Needs Discussion phase before Needs Review ? Reviews, including me, think patches with needs-review status are worthwhile. In contrast, contributers often register their patches to CF without discussions just because of no response; they cannot find whether no response is silent approval or not. +1. Sometimes a reviewer waits for the consensus in the community when someone else waits for review (, because it is marked as Needs Review). Regards, -- Hitoshi Harada -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Hi, Greg Smith g...@2ndquadrant.com writes: Robert Haas wrote: The main point here for me is that the JSON format is already parseable by YAML parsers, and can probably be turned into YAML using a very short Perl script - possibly even using a sed script. I think that it's overkill to support two formats that are that similar. It's not the case that JSON can be turned into YAML or that it just happens that it can be parsed by YAML parsers. While there was some possible divergence in earlier versions, a JSON 1.2 document *is* in YAML format already. JSON is actually a subset of YAML that uses one of the many possible YAML styles--basically, YAML accepts anything in JSON format, along with others. This means that by providing JSON output, we've *already* provided YAML output, too. Just not the nice looking output people tend to associate with YAML. Well we have JSON and agreed it was a good idea to have it. Now JSON is a subset of YAML and some would prefer another YAML style (me included). If the problem is supporting 2 formats in core rather than 3, what about replacing the current JSON support with the YAML one? At a later point we could even have JSON support back by having the YAML printer able to output different YAML styles, but I guess that's not where we are now. Vote: +1 for YAML even if that means dropping JSON. Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
On mån, 2009-12-07 at 17:14 +0900, Hitoshi Harada wrote: 2009/12/7 Itagaki Takahiro itagaki.takah...@oss.ntt.co.jp: Tom Lane t...@sss.pgh.pa.us wrote: It was written and submitted by one person who did not bother to ask first whether anyone else thought it was worthwhile. So its presence on the CF list should not be taken as evidence that there's consensus for it. Should we have Needs Discussion phase before Needs Review ? Reviews, including me, think patches with needs-review status are worthwhile. In contrast, contributers often register their patches to CF without discussions just because of no response; they cannot find whether no response is silent approval or not. +1. Sometimes a reviewer waits for the consensus in the community when someone else waits for review (, because it is marked as Needs Review). Yes, I would have had use for this myself a couple of times. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
-BEGIN PGP SIGNED MESSAGE- Hash: RIPEMD160 Tallying up the votes on this patch First, I would hope that you don't overlook the author of the patch (me) as an aye vote. :) Additionally, if you are going to tally, please consider the positive responses from people on -hackers and -general when the patch was first posted and there was a question about what the format would look like. YAML is designed to be machine-parseable (just like XML) *AND* human-readable (obstensibly like our current output). JSON tries to overcome the shortcomings of XML, but doesn't quite get it right. YAML does: it's small, intuitive, and only as verbose as needed. Human readable, yet structured enough to be read by machines as well. As far as the arguments for making an extensible system for supporting other formats, but I think this is putting the cart before the horse. While this feature is only in cvs at the moment, I've not heard a single person state another format they would like to see. I cannot think of one myself (machine parseable anyway: I'd love to see an abbreviated version that takes out the explain bits of explain analyze and changes rows= to R=, loops= to L=, etc.). This is a small patch, easy to maintain, and useful to more people than just myself, judging from the responses to the earlier thread. ObPartingShot: I hope that those in the machine readable output should never be seen by human eyes camp will support my upcoming patch to remove all extra whitespace, including newlines, from the XML format. :) - -- Greg Sabino Mullane g...@turnstep.com End Point Corporation PGP Key: 0x14964AC8 200912070727 http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8 -BEGIN PGP SIGNATURE- iEYEAREDAAYFAksc9QMACgkQvJuQZxSWSshJXwCfeHrspp39mqGJD5Z86121F1Ap VJEAoMIXrXe8/vS5rPnzyfbX04fJjp36 =H5BM -END PGP SIGNATURE- -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
* Dimitri Fontaine: Well we have JSON and agreed it was a good idea to have it. Now JSON is a subset of YAML and some would prefer another YAML style (me included). YAML is rather difficult to parse, and most parsers assume a trusted document source because they support arbitrary class instantiations (and it's not possible to switch this off, or it's rather involved to do so). Plain JSON doesn't have this issue. -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Itagaki Takahiro itagaki.takah...@oss.ntt.co.jp writes: Tom Lane t...@sss.pgh.pa.us wrote: It was written and submitted by one person who did not bother to ask first whether anyone else thought it was worthwhile. So its presence on the CF list should not be taken as evidence that there's consensus for it. Should we have Needs Discussion phase before Needs Review ? Reviews, including me, think patches with needs-review status are worthwhile. In contrast, contributers often register their patches to CF without discussions just because of no response; they cannot find whether no response is silent approval or not. Hm, I guess the question would be: what is the condition for getting out of that state? It's clear who is supposed to move a patch out of 'Needs Review', 'Waiting for Author', or 'Ready for Committer' respectively. I don't know who's got the authority to decide that something has or has not achieved community consensus. Right at the moment we handle this sort of problem in a very informal way, but if it's going to become part of the commitfest state for a patch I think we need to be a bit less informal. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Dimitri Fontaine wrote: If the problem is supporting 2 formats in core rather than 3, what about replacing the current JSON support with the YAML one? That's a step backwards. By providing JSON format, we've also satisfied people who want YAML. Ripping out JSON would mean we *only* support YAML. There are far many more JSON parsers than YAML parsers, which is why I thought the current code committed was good enough. Anyway, the fact that I have a technical opinion suggests to me I've caught up with the discussion now, so let's talk about where we're at. I think that the ongoing discussion here is likely to consume more resources than the expected future maintenance of this small bit of code. I believe the following to be true: -The patch is small to apply -It would also be easy to remove in the future should a more modular EXPLAIN implementation replace it -Having one more legacy format to satisfy would actually improve the odds that a future modular EXPLAIN would be robustly designed -There is no way a modular explain will be written any time soon -While it's questionable whether it's strictly a majority on voting here, it's close, which suggests there is plenty of support for wanting this feature -Since nothing is removed the people who aren't in favor of this format aren't negatively impacted by it being committed All that suggests to me that we've cleared the usual do we want it? hurdles that would normally go along with deciding whether a patch should go to a committer or not. That leaves code quality then. Are there any open issues with that? There's some notes about line-breaks in the CF app. Once we have a patch with those issues resolved, this should go to a committer for final veto power on its inclusion, and then we're done here. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Greg Smith wrote: That's a step backwards. By providing JSON format, we've also satisfied people who want YAML. Ripping out JSON would mean we *only* support YAML. There are far many more JSON parsers than YAML parsers, which is why I thought the current code committed was good enough. XML parsers are common enough IMHO the other computer readable formats can't be that important from a computer-readability perspective, leaving their main benefit as being human friendly. I like YAML output; but I think the most compelling arguments against the patch are that if so many people want different formats it may be a good use case for external modules. And far more than yaml output, I'd like to see a flexible module system with an equivalent of cpan install yaml or gem install yaml. I suppose one could argue that instead of YAML we design a different human-oriented format for loosely structured data; but that seems even harder. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Florian Weimer escribió: * Dimitri Fontaine: Well we have JSON and agreed it was a good idea to have it. Now JSON is a subset of YAML and some would prefer another YAML style (me included). YAML is rather difficult to parse, and most parsers assume a trusted document source because they support arbitrary class instantiations (and it's not possible to switch this off, or it's rather involved to do so). That's irrelevant because EXPLAIN YAML is already a trusted document source. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Robert Haas escribió: Tallying up the votes on this patch, we have Tom Lane, Andrew Dunstan, Josh Drake, and myself arguing against including this in core, and Josh Berkus and Itagaki Takahiro arguing for it. Ron Mayer seemed somewhat in favor of it in his message on-list but sent a message off-list taking the opposite position. Brendan Jurd cast aspersions on the human-readability of the text format but didn't explicitly take a position on the core issue, Please add my vote for YAML, because of the human-unreadability of the default text format. In fact I'd argue we could rip out the default text format and replace it with this one. -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
On Mon, 2009-12-07 at 14:52 -0300, Alvaro Herrera wrote: Robert Haas escribió: Tallying up the votes on this patch, we have Tom Lane, Andrew Dunstan, Josh Drake, and myself arguing against including this in core, and Josh Berkus and Itagaki Takahiro arguing for it. Ron Mayer seemed somewhat in favor of it in his message on-list but sent a message off-list taking the opposite position. Brendan Jurd cast aspersions on the human-readability of the text format but didn't explicitly take a position on the core issue, Please add my vote for YAML, because of the human-unreadability of the default text format. In fact I'd argue we could rip out the default text format and replace it with this one. That would almost correct. My standing is we should have one format that is parsed. The current text explain output certainly does not qualify. The XML one or YAML one does. Pick one, then parse it into whatever you want from the client. Sincerely, Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
On Dec 7, 2009, at 9:52 AM, Alvaro Herrera wrote: Tallying up the votes on this patch, we have Tom Lane, Andrew Dunstan, Josh Drake, and myself arguing against including this in core, and Josh Berkus and Itagaki Takahiro arguing for it. Ron Mayer seemed somewhat in favor of it in his message on-list but sent a message off-list taking the opposite position. Brendan Jurd cast aspersions on the human-readability of the text format but didn't explicitly take a position on the core issue, Please add my vote for YAML, because of the human-unreadability of the default text format. In fact I'd argue we could rip out the default text format and replace it with this one. +1 David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
* Alvaro Herrera: Florian Weimer escribió: * Dimitri Fontaine: Well we have JSON and agreed it was a good idea to have it. Now JSON is a subset of YAML and some would prefer another YAML style (me included). YAML is rather difficult to parse, and most parsers assume a trusted document source because they support arbitrary class instantiations (and it's not possible to switch this off, or it's rather involved to do so). That's irrelevant because EXPLAIN YAML is already a trusted document source. Uhm, really? Do I have to expect code execution on the client when I connect to a database? I hope not. -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Josh Berkus j...@agliodbs.com wrote: Having compared the JSON and YAML output formats, I think having YAML as a 2nd human-readable format might be valuable, even though it adds nothing to machine-processing. Sure. YAML is human readable and can print more information that is too verbose in one-line text format. +1 to add YAML format to EXPLAIN. Again, if there were a sensible way to do YAML as a contrib module, I'd go for that, but there isn't. Some ideas to export formatters: 1. Provides register_explain_format(name, handler) function. Contrib modules can register their handlers when LOADed. 2. Provides a method to specify a filter function. XML output is passed to the function and probably converted with XSLT. BTW, an extensible formatter is surely useful, but what will we do about XML and JSON formats when we have it? Will we remove them from core? If we have a plan to the direction, we should complete it before 8.5.0 release. We will be hard to revert them after the release. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
On Sun, Dec 6, 2009 at 8:34 PM, Itagaki Takahiro itagaki.takah...@oss.ntt.co.jp wrote: Josh Berkus j...@agliodbs.com wrote: Having compared the JSON and YAML output formats, I think having YAML as a 2nd human-readable format might be valuable, even though it adds nothing to machine-processing. Sure. YAML is human readable and can print more information that is too verbose in one-line text format. +1 to add YAML format to EXPLAIN. Again, if there were a sensible way to do YAML as a contrib module, I'd go for that, but there isn't. Some ideas to export formatters: 1. Provides register_explain_format(name, handler) function. Contrib modules can register their handlers when LOADed. 2. Provides a method to specify a filter function. XML output is passed to the function and probably converted with XSLT. BTW, an extensible formatter is surely useful, but what will we do about XML and JSON formats when we have it? Will we remove them from core? If we have a plan to the direction, we should complete it before 8.5.0 release. We will be hard to revert them after the release. I sent a previous message about this, but just realized I forgot to copy the list. I thought about the possibility of a pluggable system for formats when I wrote the original machine-readable explain patch. It seemed to me at the time that if we went this route any contrib module would have to duplicate significant portions of explain.c, no matter where we put the hooks. We'd presumably feel obliged to update the contrib module when making any changes to explain.c, so I think the maintenance burden would actually be higher that way than with everything in core. The main point here for me is that the JSON format is already parseable by YAML parsers, and can probably be turned into YAML using a very short Perl script - possibly even using a sed script. I think that it's overkill to support two formats that are that similar. Tallying up the votes on this patch, we have Tom Lane, Andrew Dunstan, Josh Drake, and myself arguing against including this in core, and Josh Berkus and Itagaki Takahiro arguing for it. Ron Mayer seemed somewhat in favor of it in his message on-list but sent a message off-list taking the opposite position. Brendan Jurd cast aspersions on the human-readability of the text format but didn't explicitly take a position on the core issue, and Euler Taveira de Oliveira suggested that writing a converter would be easier than writing a module framework (which I think is almost certainly true) but also didn't explicitly take a position on the core issue. Overall, that sounds to me like a slight preponderance of no votes. Anyone think I've mis-tallied or want to weigh in? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Robert Haas wrote: The main point here for me is that the JSON format is already parseable by YAML parsers, and can probably be turned into YAML using a very short Perl script - possibly even using a sed script. I think that it's overkill to support two formats that are that similar. It's not the case that JSON can be turned into YAML or that it just happens that it can be parsed by YAML parsers. While there was some possible divergence in earlier versions, a JSON 1.2 document *is* in YAML format already. JSON is actually a subset of YAML that uses one of the many possible YAML styles--basically, YAML accepts anything in JSON format, along with others. This means that by providing JSON output, we've *already* provided YAML output, too. Just not the nice looking output people tend to associate with YAML. Accordingly, there is really no basis for this patch to exist from the perspective of helping a typical tool author. If you want to parse YAML robustly, you're going to grab someone's parsing library to do it rather than writing it yourself, and if you do that it will accept the existing JSON output just fine too. Basically this patch lives or dies by whether it looks so much nicer to people as to justify its code weight. Given the above, I don't understand why writing this patch was deemed worthwhile in the first place, but I hate to tell people they can't have something they find visually appealing just because I don't think it's an improvement. Consider me a neutral vote, although I suspect the above may sway some people who were on the fence toward disapproval. -- Greg Smith2ndQuadrant Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Greg Smith g...@2ndquadrant.com writes: Given the above, I don't understand why writing this patch was deemed worthwhile in the first place, It was written and submitted by one person who did not bother to ask first whether anyone else thought it was worthwhile. So its presence on the CF list should not be taken as evidence that there's consensus for it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Tom Lane t...@sss.pgh.pa.us wrote: It was written and submitted by one person who did not bother to ask first whether anyone else thought it was worthwhile. So its presence on the CF list should not be taken as evidence that there's consensus for it. Should we have Needs Discussion phase before Needs Review ? Reviews, including me, think patches with needs-review status are worthwhile. In contrast, contributers often register their patches to CF without discussions just because of no response; they cannot find whether no response is silent approval or not. Regards, --- ITAGAKI Takahiro NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Josh Berkus wrote: ... YAML for easier readability ... almost as good ... I agree with Kevin that it's more readable. Again, if there were a sensible way to do YAML as a contrib module, I'd go for that, but there isn't. Perhaps that's be a direction this could take? What would it take for this feature to be a demo/example for some future modules system. It seems like there have been a few recent features related to decorating output (UTF8 tables, YAML explain, \d... updates). While there's no great way to make this a contrib module today, would it make sense to add such hooks for an eventual module system? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Ron Mayer escreveu: While there's no great way to make this a contrib module today, would it make sense to add such hooks for an eventual module system? I don't think so. It's easier to code a converter tool. -- Euler Taveira de Oliveira http://www.timbira.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
On Wed, Dec 2, 2009 at 4:24 PM, Tom Lane t...@sss.pgh.pa.us wrote: Ron Mayer rm...@cheapcomplexdevices.com writes: Tom Lane wrote: Hmm. So the argument for it is let's make a machine-readable format more human-readable? I'm not getting the point. People should look at the regular text output. IMHO YAML beats the regular text format for human-readability - at least for people with narrow terminal windows, and for novices. Greg posted examples comparing regular-text vs yaml vs json here: http://archives.postgresql.org/pgsql-hackers/2009-08/msg02090.php Mph. Maybe I've been looking at the traditional format too long, but I don't find the YAML version better --- it's so verbose that you could only see a small fraction of a query at a time. I've been thinking about this a little more and I think I am going to vote (if I get a vote) to reject this patch. I think Andrew hit the crucial point upthread: the latest version of YAML is a superset of JSON, which means that the only possible use cases for this patch are: - people using older YAML parsers that can't handle the latest version (because if they can handle the latest version then they can just use that on the existing JSON format), and - people who want to use the YAML format as a substitute for text format on grounds of readability. The first target doesn't seem worth aiming at. Admittedly the latest version of the YAML 1.2 spec - whose stated goal is JSON-compatibilty - is only two months old, so there may be many YAML users who don't have parsers that are completely JSON-compatible. But presumably this problem will resolve itself over time. With respect to the second one, I am not going to argue that the current text format is ideal in all ways. In particular, on complex plans, I find it difficult to match up the plans for the inner and outer sides of any given node, which may be far enough apart vertically that it's difficult to tell exactly which bits are in the same column. Furthermore, the main output row for each node is wider than I would like, especially when using EXPLAIN ANALYZE. Even on relatively simple plans, I have to maximize my terminal window to forestall wrapping, and on complex plans, wrapping is inevitable even if I do maximize the window. For all of that, in the nearly-two-years I've been reading pgsql-hackers, I can't remember one complaint about the visual presentation of the EXPLAIN output. It's possible that my memory is faulty, but I think I would remember if there had been very many. On top of that, if you did want YAML for easier readability, what aspect of the output is more readable in YAML than it is in text format? The only answer I can think of is that you like having each data element on a separate line, so that the plan is much longer but somewhat narrower. But if that's what you want, the JSON output is almost as good - the only difference is a bit of extra punctuation. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
On top of that, if you did want YAML for easier readability, what aspect of the output is more readable in YAML than it is in text format? The only answer I can think of is that you like having each data element on a separate line, so that the plan is much longer but somewhat narrower. But if that's what you want, the JSON output is almost as good - the only difference is a bit of extra punctuation. almost as good ... I agree with Kevin that it's more readable. The whole patch just adds 144 lines. It doesn't look to me like there's significant maintenance burden involved, but of course I need to defer to the more experienced. It's even possible that we could reduce the size of the patch still further if we really looked at it as just a differently punctuated JSON. Having compared the JSON and YAML output formats, I think having YAML as a 2nd human-readable format might be valuable, even though it adds nothing to machine-processing. Again, if there were a sensible way to do YAML as a contrib module, I'd go for that, but there isn't. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
On Fri, Dec 4, 2009 at 7:42 PM, Josh Berkus j...@agliodbs.com wrote: On top of that, if you did want YAML for easier readability, what aspect of the output is more readable in YAML than it is in text format? The only answer I can think of is that you like having each data element on a separate line, so that the plan is much longer but somewhat narrower. But if that's what you want, the JSON output is almost as good - the only difference is a bit of extra punctuation. almost as good ... I agree with Kevin that it's more readable. The whole patch just adds 144 lines. It doesn't look to me like there's significant maintenance burden involved, but of course I need to defer to the more experienced. It's even possible that we could reduce the size of the patch still further if we really looked at it as just a differently punctuated JSON. Having compared the JSON and YAML output formats, I think having YAML as a 2nd human-readable format might be valuable, even though it adds nothing to machine-processing. Again, if there were a sensible way to do YAML as a contrib module, I'd go for that, but there isn't. I don't think the maintenance burden is the issue, per se. It's more that I don't like the idea of putting in a bunch of formats that are only trivially different from each other, and if there were ever a case where we should reject a new format because it is too similar to an existing one, this seems to be it. If that's a bad reason for rejecting a new format, then I don't have a second one, but we may end up with a lot of formats - and that WILL be a maintenance burden, especially if we ever want to make non-trivial extensions to the output format. Frankly, just adding new fields (perhaps controlled by new options) is never going to be that big of a deal, but there will certainly come a day when someone wants to do something more novel, like dumping parse-tree representations of expressions or something along the line of Tom Raney's visual explain tool, which dumped out every path the planner considered. I don't really want to be the person who has to tell the person who writes that patch sorry, but we have to reject your patch until it supports every one of our numerous slightly different output formats. One possibility for contrib-module-izing this, and perhaps other output formats that people might like to see, is to write a function that takes the JSON or XML output as input and does the appropriate translation. For something like YAML, whose semantics are so close to JSON, this should be pretty simple. One of the reasons why I was hot to get JSON support into the initial version of machine-readable EXPLAIN is because JSON maps very cleanly onto the type of data structures that are common in scripting languages: everything is lists and hashes, nested inside each other, and text and numeric scalars. So you can read a JSON object into a script written in Perl, PHP, Python, Ruby, JavaScript, and probably half a dozen other languages and get a native object. From there, it's very easy to write the data back out in whatever format you happen to prefer just by walking the data structure. I suspect that a JSON-to-YAML converter in Perl would be less than 50 lines. (The XML format can also be transformed using things like XSLT, but I'm less familiar with those tools.) ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Josh Berkus wrote: On 11/30/09 8:17 PM, Andrew Dunstan wrote: Do we have consensus yet that we want YAML? It seemed, well, yet another format without all that much advantage over what's there. Well, what's the code count? What dependencies, if any, does it add? The patch itself is quite small. There are no extra external dependencies. YAML and JSON are pretty much interchangeable for our purposes. According to Wikipedia, Both functionally and syntactically, JSON is effectively a subset of YAML. See http://en.wikipedia.org/wiki/JSON#YAML So the YAML parsers should be able to handle the JSON output. The only thing we'd be buying with this patch is making a bit happier some people who prefer reading the YAML syntax. For machine readability we'd be gaining precisely nothing. I guess the question is this: when are we going to say No more output formats. We have enough.? One consideration is this: the more formats we support the dumber the output will be. Already the XML output is arguably dumber than it should be, because XML elements are two-dimensional (they can have property lists (attributes) and child elements) but JSON/YAML nodes are one-dimensional, so we have made some things that one might normally expect to be attributes in XML into child elements. While adding YAML won't impose any additional burden of that kind, because its semantics are so close to those of JSON, other output formats well might. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Andrew Dunstan and...@dunslane.net writes: YAML and JSON are pretty much interchangeable for our purposes. According to Wikipedia, Both functionally and syntactically, JSON is effectively a subset of YAML. See http://en.wikipedia.org/wiki/JSON#YAML So the YAML parsers should be able to handle the JSON output. The only thing we'd be buying with this patch is making a bit happier some people who prefer reading the YAML syntax. For machine readability we'd be gaining precisely nothing. Hmm. So the argument for it is let's make a machine-readable format more human-readable? I'm not getting the point. People should look at the regular text output. One consideration is this: the more formats we support the dumber the output will be. Already the XML output is arguably dumber than it should be, because XML elements are two-dimensional (they can have property lists (attributes) and child elements) but JSON/YAML nodes are one-dimensional, so we have made some things that one might normally expect to be attributes in XML into child elements. While adding YAML won't impose any additional burden of that kind, because its semantics are so close to those of JSON, other output formats well might. I tend to look at it the other way around: having to support output formats that have significantly different data models is a Good Thing because it forces you to design sufficiently general code mechanisms. If YAML had yet another data model it might actually be a useful exercise to get the code to handle that. However, if it's not teaching us anything we didn't learn from JSON, there's no gain from that viewpoint either. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: YAML... Hmm. So the argument for it is let's make a machine-readable format more human-readable? I'm not getting the point. People should look at the regular text output. IMHO YAML beats the regular text format for human-readability - at least for people with narrow terminal windows, and for novices. Greg posted examples comparing regular-text vs yaml vs json here: http://archives.postgresql.org/pgsql-hackers/2009-08/msg02090.php I think it's more human-readable for novices since it explicitly spells out what values refer to startup values vs totals. I think it's more human-readable to me because the current text format frequently wraps for me on even a modestly complex query, and I find scrolling down easier than scrolling both ways. None of the other machine-intended formats seem to suit that purpose well because they're dominated by a lot of markup. That said, though, it's not that big a deal. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
All, If some people want it, and there's no significant maintenance burden associated with YAML output, then why not? Mind you, if it was practical, I'd suggest that YAML ... and all additional Explain formats ... should be a contrib module. Anything other than XML and JSON will be fairly marginal. --Josh Berkus -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
On Wed, 2009-12-02 at 10:45 -0800, Josh Berkus wrote: All, If some people want it, and there's no significant maintenance burden associated with YAML output, then why not? Mind you, if it was practical, I'd suggest that YAML ... and all additional Explain formats ... should be a contrib module. Anything other than XML and JSON will be fairly marginal. That would be my take... have explain kick out XML (or whatever) and then parse it into anything you want. That way it isn't additional burden into core. Joshua D. Drake --Josh Berkus -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
2009/12/3 Ron Mayer rm...@cheapcomplexdevices.com: Tom Lane wrote: Hmm. So the argument for it is let's make a machine-readable format more human-readable? I'm not getting the point. People should look at the regular text output. IMHO YAML beats the regular text format for human-readability - at least for people with narrow terminal windows, and for novices. Agreed. Calling the regular text output of EXPLAIN human readable is an exaggeration. Cheers, BJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
Ron Mayer rm...@cheapcomplexdevices.com writes: Tom Lane wrote: Hmm. So the argument for it is let's make a machine-readable format more human-readable? I'm not getting the point. People should look at the regular text output. IMHO YAML beats the regular text format for human-readability - at least for people with narrow terminal windows, and for novices. Greg posted examples comparing regular-text vs yaml vs json here: http://archives.postgresql.org/pgsql-hackers/2009-08/msg02090.php Mph. Maybe I've been looking at the traditional format too long, but I don't find the YAML version better --- it's so verbose that you could only see a small fraction of a query at a time. The main strike against the traditional format IME is exactly what Greg alludes to in that message: it's prone to being rendered totally unreadable by email line-wrapping. However, I'm unconvinced that YAML would be any better; it looks like it's still critically dependent on the location and amount of whitespace in order to be readable. The lines might be a bit shorter on average, but you're still going to hit a narrow window's right margin pretty quick in any complicated plan. In any case, the real killer is email clients that feel no obligation to preserve whitespace layout at all, and this would certainly not look much better after that treatment. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] YAML Was: CommitFest status/management
On 11/30/09 8:17 PM, Andrew Dunstan wrote: Do we have consensus yet that we want YAML? It seemed, well, yet another format without all that much advantage over what's there. Well, what's the code count? What dependencies, if any, does it add? --Josh -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
