Re: Images in the official documentation
Hi Jürgen, On 11/14/18 8:36 AM, Jürgen Purtz wrote: > After many mails, F2F-discussions and some experiments - everything > without a concrete result - I hereby start a new initiative concerning > SVG tools. The attached PDF file describes my recommendation. The two > SVG files are examples for working in this sense. It looks like you put a lot of effort into this, thanks! Those SVGs do look great, and it's IMO it's a much needed feature to support images in our documentation :) I will leave it to others to discuss if the generated SVGs are a technical fit for the documentation code. I would recommend that you transfer the contents of your proposal to the PostgreSQL wiki (https://wiki.postgresql.org/) as it would make it easier for others to collaborate on and contribute to it. If you need help with it, please feel free to ping me. The one thing that I see is if we would like to have a style guide, primarily around colors, for images we put into the repo. I know that somehow correlates with the definition of "bike shed" but it could also help reduce that in the future. As we do have a PostgreSQL color palette, I also see that as an opportunity to reduce potential bikeshedding as well. Thanks! Jonathan signature.asc Description: OpenPGP digital signature
Re: Images in the official documentation
Because I strongly support the involvement of SVG into our documentation, I welcome every activity to establish this goal. The mail to which Pavel had replied (https://www.postgresql.org/message-id/4ea1bacb-02ca-e967-31d7-d2a6db30abff%40purtz.de) contained the two graphics /PageLayout.svg/ "Page Layout" and /pgDump.svg/ "pg_dump, psql, pg_restore". Actually they are *not* part of our documentation. They are only part of a proof-of-concept to generate SVG and include the result into HTML and PDF output. The two examples can be used as references for comparisons with any tool. Kind regards, Jürgen On 20.07.2018 18:14, Pavel Golub wrote: Hello, Jürgen. You wrote: JP> On 19.07.2018 14:06, Pavel Golub wrote: I disagree. From what I heard, GraphViz is the winner for now. I can give you my two cents: plantuml is another good choice JP> Ok, please give us an example - possibly the two previous graphics. Fair enough. Let's try different formats. Sorry, what exactly previous graphics we're talking about? JP> Kind regards, Jürgen
Re: Images in the official documentation
Hello, Jürgen. You wrote: JP> On 19.07.2018 14:06, Pavel Golub wrote: >> I disagree. From what I heard, GraphViz is the winner for now. I can >> give you my two cents: plantuml is another good choice JP> Ok, please give us an example - possibly the two previous graphics. Fair enough. Let's try different formats. Sorry, what exactly previous graphics we're talking about? JP> Kind regards, Jürgen -- With best wishes, Pavel mailto:pa...@gf.microolap.com
Re: Images in the official documentation
On 19.07.2018 14:06, Pavel Golub wrote: I disagree. From what I heard, GraphViz is the winner for now. I can give you my two cents: plantuml is another good choice Ok, please give us an example - possibly the two previous graphics. Kind regards, Jürgen
Re: Images in the official documentation
Hello, Jürgen. You wrote: JP> Our discussion about grafics in the documentation reached to the JP> conclusion that we shall use SVG, the importance to 'diff-ability' JP> is rated differently, and there is no consensus about tools. I disagree. From what I heard, GraphViz is the winner for now. I can give you my two cents: plantuml is another good choice JP> To push the issue forward I modify my original proposal to use JP> plain svg files in a standard editor as follows: JP> We define a 'Simplified SVG format' (SSVG) We create libraries JP> where complex elements are predefined and can be JP> referenced We write the source in ssvg-format A compiler JP> (written in bison or xslt) converts ssvg-files to JP> svg-files We extend the sgml-files to include the svg-files The JP> ssvg and svg-files are located in a new svg directory, JP> Makefile copies them to sgml and html directory A JP> proof-of-concept is performed in 11beta2 for HTML and PDF JP> generation. The ssvg-format may be XML (as used in the examples), JP> JSON, C-style function calls JP> JP>PRO: JP> JP> JP> SVG 1.x has many restrictions and SVG 2.x does not make JP> progress in the last years. Tools and Browsers support different JP> ranges of the specification. The planned compiler cuts JP> everything down to the basic language level, where a broad JP> support is possible. Predefinded elements and default values JP> reduce the ssvg file to a small and clear source file. JP> You can embed original svg commands into ssvg files. When you use JP> an editor and a browser in parallel, you get the visual JP> result with few clicks. Everything is diff-able. The Makefile JP> needs only slightly amendments: additional cp commands and JP> some target-dependencies. We need no new tool. JP> JP> CON: JP> JP> JP> The development is done in a non-wysiwyg editor and without JP> mouse. You have to count pixel. JP> JP>Example: JP>PageLayout.ssvg: written in the new language JP>PageLayout.svg: the generated svg file (actually by hand, JP> the compiler is not yet implemented) JP>storage.sgml: an additional paragraph to refer to the svg-file JP> JP> JP> JP> JP> JP> JP> JP> scalefit="1" width="100%" contentdepth="100%"/> JP> JP> JP> JP> JP>PageLayoutHtml.png: the HTML result JP>PageLayoutPdf.png: the PDF result JP> JP>A second example: pgDump.svg within backup.sgml JP> JP> JP>Kind regards, Jürgen Purtz JP> JP> JP> JP> -- With best wishes, Pavel mailto:pa...@gf.microolap.com
Re: Images in the official documentation
Peter Eisentraut wrote: > On 2/26/18 20:02, Craig Ringer wrote: > > If I were maintaining the docs in a vacuum, I'd use graphviz for > > something like that, because it's a figure that does need regular > > updates and changes. And because > > the list of fun things to do in my life definitely does not include > > hand-writing SVG. Not that tweaking GraphViz .dot is fun, but it's the > > default tool for a reason. > > > > I'd be awfully tempted to generate the node-map part of the catalog > > relationship .dot file from a query, too. > > I think graphviz would be a great fit for what we are discussing here. > Certainly more so then some-person-on-github's latest idea for how to > convert ASCII art into diagrams. ... particularly so if said idea involves PHP, Haskell, Go, Python, or any other language that we don't currently have as requirement in our build chain. GraphViz gets my vote, too. It may not produce the most elegant diagrams in the universe, but the source format is as good as we can get. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Images in the official documentation
On 2/26/18 20:02, Craig Ringer wrote: > If I were maintaining the docs in a vacuum, I'd use graphviz for > something like that, because it's a figure that does need regular > updates and changes. And because > the list of fun things to do in my life definitely does not include > hand-writing SVG. Not that tweaking GraphViz .dot is fun, but it's the > default tool for a reason. > > I'd be awfully tempted to generate the node-map part of the catalog > relationship .dot file from a query, too. I think graphviz would be a great fit for what we are discussing here. Certainly more so then some-person-on-github's latest idea for how to convert ASCII art into diagrams. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Images in the official documentation
On Mon, Feb 26, 2018 at 10:23 PM, Tom Lane wrote: > Craig Ringer writes: >> On 26 February 2018 at 12:16, Tom Lane wrote: >>> How can we resolve these issues? > >> Question the assumptions and requirements. Why do we actually _need_ >> diffable, mergeable images? Sure, it'd be *nice*, but what's the real world >> impact if we don't have it? > > Well, I'll tell you exactly why I'm being sticky about this: we've been > down this road before. We used to have some figures in .gif format, > and one of the problems with them was they were too hard to update. > I don't buy the "they won't need updates" argument for a second, either. > For instance, I recall that one of the images we had was a diagram of > the system catalog cross-references, and it was constantly out of date > because of the difficulty of updating it. > > Admittedly, this was 15+ years ago. Maybe the state of the art in > figure editors has advanced to the point where it won't be so hard. > But color me suspicious. In case you missed, a couple of years ago we discussed this on pgcon: Heikki's version: https://wiki.postgresql.org/wiki/Figures_%26_Pics_in_Docs Emre suggested to use Markdeep (BSD license), http://casual-effects.com/markdeep/ http://www.sai.msu.su/~megera/postgres/gin-ascii-v2.md.html It looks good for small diagrams, but will not work for complex stuff, such as pg_catalog structure. > > regards, tom lane >
Re: Images in the official documentation
On 27 February 2018 at 03:23, Tom Lane wrote: > Craig Ringer writes: > > On 26 February 2018 at 12:16, Tom Lane wrote: > >> How can we resolve these issues? > > > Question the assumptions and requirements. Why do we actually _need_ > > diffable, mergeable images? Sure, it'd be *nice*, but what's the real > world > > impact if we don't have it? > > Well, I'll tell you exactly why I'm being sticky about this: we've been > down this road before. We used to have some figures in .gif format, > and one of the problems with them was they were too hard to update. > I don't buy the "they won't need updates" argument for a second, either. > For instance, I recall that one of the images we had was a diagram of > the system catalog cross-references, and it was constantly out of date > because of the difficulty of updating it. > Yeah, that sounds painful. I can certainly see your objection when framed in terms of things like illustrations of catalogs and catalog relationships. If I were maintaining the docs in a vacuum, I'd use graphviz for something like that, because it's a figure that does need regular updates and changes. And because the list of fun things to do in my life definitely does not include hand-writing SVG. Not that tweaking GraphViz .dot is fun, but it's the default tool for a reason. I'd be awfully tempted to generate the node-map part of the catalog relationship .dot file from a query, too. I still advocate for relaxing the policy for images that are *not* likely to need frequent updates, but also for being conservative about how/when we include images. Does this add real value to the docs, is it worth any maintenance hassle? Then, for things that will change more, like catalogs, using a tool like graphviz. If we object to adding a docs build-dependency, we could always commit generated files like we already do for the 'configure' script, and make sure there's a committer/maintainer Make target that warns if the sources are newer than the docs. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: Images in the official documentation
Craig Ringer writes: > On 26 February 2018 at 12:16, Tom Lane wrote: >> How can we resolve these issues? > Question the assumptions and requirements. Why do we actually _need_ > diffable, mergeable images? Sure, it'd be *nice*, but what's the real world > impact if we don't have it? Well, I'll tell you exactly why I'm being sticky about this: we've been down this road before. We used to have some figures in .gif format, and one of the problems with them was they were too hard to update. I don't buy the "they won't need updates" argument for a second, either. For instance, I recall that one of the images we had was a diagram of the system catalog cross-references, and it was constantly out of date because of the difficulty of updating it. Admittedly, this was 15+ years ago. Maybe the state of the art in figure editors has advanced to the point where it won't be so hard. But color me suspicious. regards, tom lane
Re: Images in the official documentation
On 26 February 2018 at 12:16, Tom Lane wrote: > Craig Ringer writes: > > Yeah, I think it'd just effectively preserve the status quo by rendering > > anyone who's willing to add images and designs to the docs unable - or > > unlikely to be willing - to do so. > > This is an entirely reasonable complaint. But I don't see how we'd cope > with patches that rewrite an entire SVG file because the patch author's > WYSIWG editor emits its output in a style randomly different from the tool > the previous patch author used. It seems like such patches would be > effectively unreviewable, and certainly incapable of being merged. > Well, they definitely couldn't be merged in any situation entailing conflicts, no. Patch review would entail displaying the new and (if present) old SVGs, possibly in the context of a build of the docs, or possibly standalone. This is always going to be the case for anything but the most trivial SVG changes anyway. After all, even small textual changes can cause elements to overlap, break out of their expected bounaries, or otherwise look wrong. So IMO whether it's SVG or a raster image format, the net effect isn't that different: you have to review the rendered result not the source. Personally I'd just mark svg as binary in .gitattributes to stop it from spamming noise in diffs. Github offers a cool tool for side-by-side compares of svg diffs ( https://github.com/blog/1902-svg-viewing-diffing) but that likely won't help us much. There's diffsvg (https://github.com/jrsmith3/diffsvg), which I haven't tried but looks interesting. Combined with filters in .gitattributes this might offer improved visibility into change history if we really need it. Personally, I don't think images will be changing that often and they should just be tracked as blobs. > Maybe we could improve matters a bit by insisting that everyone use the > same version of the same SVG-editing tool. But that's not too practical. > Worse, from what I've seen, even that would not really fix the problem. > The tools simply don't give a damn about comparability of their dump > files. I don't blame their authors exactly (try diffing Postgres data > file changes :-() but that doesn't mean it isn't a problem for us. > > How can we resolve these issues? > Question the assumptions and requirements. Why do we actually _need_ diffable, mergeable images? Sure, it'd be *nice*, but what's the real world impact if we don't have it? -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: Images in the official documentation
Craig Ringer writes: > On 26 February 2018 at 04:12, Steve Atkins wrote: >> Writing SVG by hand maybe doesn't seem the best idea. >> I understand the attraction to people who want to store everything as >> diffable text, but images of this sort are unlikely to get updated by >> others, which means they're unlikely to be maintained as the things they're >> intended to document change. It also means that the people best suited to >> generating diagrams are the least likely to do so, and vice-versa. > Yeah, I think it'd just effectively preserve the status quo by rendering > anyone who's willing to add images and designs to the docs unable - or > unlikely to be willing - to do so. This is an entirely reasonable complaint. But I don't see how we'd cope with patches that rewrite an entire SVG file because the patch author's WYSIWG editor emits its output in a style randomly different from the tool the previous patch author used. It seems like such patches would be effectively unreviewable, and certainly incapable of being merged. Maybe we could improve matters a bit by insisting that everyone use the same version of the same SVG-editing tool. But that's not too practical. Worse, from what I've seen, even that would not really fix the problem. The tools simply don't give a damn about comparability of their dump files. I don't blame their authors exactly (try diffing Postgres data file changes :-() but that doesn't mean it isn't a problem for us. How can we resolve these issues? regards, tom lane
Re: Images in the official documentation
> On Feb 25, 2018, at 6:45 PM, Craig Ringer wrote: > > On 26 February 2018 at 04:12, Steve Atkins wrote: > > Writing SVG by hand maybe doesn't seem the best idea. > > I understand the attraction to people who want to store everything as > diffable text, but images of this sort are unlikely to get updated by others, > which means they're unlikely to be maintained as the things they're intended > to document change. It also means that the people best suited to generating > diagrams are the least likely to do so, and vice-versa. > > > > Yeah, I think it'd just effectively preserve the status quo by rendering > anyone who's willing to add images and designs to the docs unable - or > unlikely to be willing - to do so. Yup. I do think that graphics would be nice in a few places, and that SVG is likely the best format for them. There are quite a few tools that could be integrated with varying amounts of effort into the documentation generation workflow. # ASCII language or ascii art to SVG # asciitosvg https://github.com/dhobsd/asciitosvg Inspired by ditaa, similar in functionality ## blockdiag, seqdiag, actdiag, nwdiag http://blockdiag.com Generates various box and arrow diagrams from a DOT-ish input language. ## ditaa It takes ascii art of box and arrow diagrams and turns them into nice svg. Also supports boxes that look like "storage" cylinders, documents, clouds and computers. ## erd https://github.com/BurntSushi/erd Entity relationship diagrams, from a plain text input. Uses DOT and graphviz under the covers. ## Markdeep http://casual-effects.com/markdeep/ In-browser javascript rendering of ascii art, particularly boxes and arrows. Alternate implementation, https://github.com/blampe/goat, converts to SVG via CLI. ## Mermaid Flowcharts and sequence diagrams from a markdown-esque input. ## mscgen Message sequence chart inputs in a DOT-ish language to SVG ## plantuml http://plantuml.com It supports a human-editable descriptive text input language and generates from it: sequence diagrams various box + arrow style diagrams flowcharts state diagrams etc. ## shaape https://github.com/christiangoltz/shaape Converts ascii art to SVG. Rather nice. ## svgbob https://github.com/ivanceras/svgbobrus Ascii art to SVG. Likely good for boxes and arrows. ## syntrax https://kevinpt.github.io/syntrax/ Railroad diagrams (like the ones SQLite docs are known for). ## umlet http://www.umlet.com Pointy-clicky editor, but also converts plain text to uml, sequence, activity diagrams # Interactive editor ## SVG-Edit https://github.com/SVG-Edit/svgedit Open source, browser based interactive SVG editor. Seems to generate fairly clean SVG. Cheers, Steve
Re: Images in the official documentation
On 26 February 2018 at 04:12, Steve Atkins wrote: > > Writing SVG by hand maybe doesn't seem the best idea. > > I understand the attraction to people who want to store everything as > diffable text, but images of this sort are unlikely to get updated by > others, which means they're unlikely to be maintained as the things they're > intended to document change. It also means that the people best suited to > generating diagrams are the least likely to do so, and vice-versa. > > > Yeah, I think it'd just effectively preserve the status quo by rendering anyone who's willing to add images and designs to the docs unable - or unlikely to be willing - to do so. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services
Re: Images in the official documentation
> On Feb 25, 2018, at 4:00 AM, Jürgen Purtz wrote: > > As an addition to my mail from January 2016 concerning graphics > (https://www.postgresql.org/message-id/568A9148.30303%40purtz.de) I propose > to use SVG (after switching to XML) - but not an SVG which is generated by > Inkscape or similar tools. Those editors generate very ugly and chatty > commands. This form is not easy to read or understand. Therefore we shall use > nothing but a simple text editor and write every line by our self. The > process is divided into two parts: > As a basis we shall develop an SVG library containing a bunch of "atomic" > symbols of simple graphical elements (rectangle, arrow, ...) up to complex > elements (magnetic disc, laptop, cloud, UML-elements, ...). The SVG routines > creating those symbols shall accept parameters for position, size, rotation, > colour, ... . This library shortens the individual SVG files, it ensures a > consistent rendering of common graphical elements, it is diff-able, and it > will reach a stable state - some day. > > The real graphics shall use the elements of the library and add individual > SVG elements. The rules for this part are the same as above: create SVG > commands with vi (or similar), store it in git. > If such an approach works (we must distribute the docs across a wide range of > different systems, a proof-of-system is necessary) and the community accepts > my proposal, I would like to work on the library-part - starting after > finishing my actual project in about 6 weeks from now. The attached file > contains a very first draft as of Jan. 2016. Writing SVG by hand maybe doesn't seem the best idea. I understand the attraction to people who want to store everything as diffable text, but images of this sort are unlikely to get updated by others, which means they're unlikely to be maintained as the things they're intended to document change. It also means that the people best suited to generating diagrams are the least likely to do so, and vice-versa. Cheers, Steve
Re: Images in the official documentation
As an addition to my mail from January 2016 concerning graphics (https://www.postgresql.org/message-id/568A9148.30303%40purtz.de) I propose to use SVG (after switching to XML) - but not an SVG which is generated by Inkscape or similar tools. Those editors generate very ugly and chatty commands. This form is not easy to read or understand. Therefore we shall use nothing but a simple text editor and write every line by our self. The process is divided into two parts: As a basis we shall develop an SVG library containing a bunch of "atomic" symbols of simple graphical elements (rectangle, arrow, ...) up to complex elements (magnetic disc, laptop, cloud, UML-elements, ...). The SVG routines creating those symbols shall accept parameters for position, size, rotation, colour, ... . This library shortens the individual SVG files, it ensures a consistent rendering of common graphical elements, it is diff-able, and it will reach a stable state - some day. The real graphics shall use the elements of the library and add individual SVG elements. The rules for this part are the same as above: create SVG commands with vi (or similar), store it in git. If such an approach works (we must distribute the docs across a wide range of different systems, a proof-of-system is necessary) and the community accepts my proposal, I would like to work on the library-part - starting after finishing my actual project in about 6 weeks from now. The attached file contains a very first draft as of Jan. 2016. Kind regards Jürgen Purtz On 23.02.2018 22:14, Daniel Westermann wrote: Hi %, I am working with PostgreSQL documentation quite a few years now and I am almost happy. What I think is completely missing (especially if you compare to commercial product documentation) are pictures that illustrate a topic, e.g. the relation of instance->database->user/role->schema->objects. Is there an agreement not to include that pictures for any reason? I can not promise that I find time for that in the near future but if that will be appreciated I am willing to spend time on that to make the documentation even better. Regards Daniel
Re: Images in the official documentation
Oleg Bartunov writes: > On Sat, Feb 24, 2018 at 4:04 AM, Peter Eisentraut > wrote: >> The issue of how to manage the sources is still the same, though. > SVG format is ascii based vector format. Yeah. I think where the last discussion left this was that we'd be willing to accept SVG-format figures, but we were having a hard time figuring out what tools to recommend for editing them, because available editors have no respect for readability or preserving small deltas when rewriting SVG files. regards, tom lane
Re: Images in the official documentation
On Sat, Feb 24, 2018 at 4:04 AM, Peter Eisentraut wrote: > On 2/23/18 11:21, Tom Lane wrote: >> In the distant >> past, as I recall, we had a GIF or two; but we abandoned that on the >> grounds that it was unmaintainable and also incompatible with some >> documentation output formats. I'm not too sure what the state of >> play is on the latter point, now that we've switched to XML. > > The complications with the image formats in the past were mainly around > what ((pdf)jade)tex would accept. The tools have shifted a bit now, and > the zoo formats is a different one. Nothing that a few make rules > couldn't address, though, I think. > > The issue of how to manage the sources is still the same, though. SVG format is ascii based vector format. We made experimental pdf with pictures http://www.sai.msu.su/~megera/postgres/files/postgres-11-diagram.pdf (GIN AM diagram, Appendix L). Appendix L also demonstrates our sample database with step-by-step introduction to Postgres for beginners. We have a separate book for beginners, which we released under BSD license and it's available on russian/english languages. Our experience shows, that people really appreciate it. I hope we will have time at PGCon to discuss documentation somehow. > > -- > Peter Eisentraut http://www.2ndQuadrant.com/ > PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >
Re: Images in the official documentation
On 2/23/18 11:21, Tom Lane wrote: > In the distant > past, as I recall, we had a GIF or two; but we abandoned that on the > grounds that it was unmaintainable and also incompatible with some > documentation output formats. I'm not too sure what the state of > play is on the latter point, now that we've switched to XML. The complications with the image formats in the past were mainly around what ((pdf)jade)tex would accept. The tools have shifted a bit now, and the zoo formats is a different one. Nothing that a few make rules couldn't address, though, I think. The issue of how to manage the sources is still the same, though. -- Peter Eisentraut http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Re: Images in the official documentation
Daniel Westermann writes: > Is there an agreement not to include that pictures for any reason? This has been discussed repeatedly (see the archives). We'd surely like the ability to include figures, but we've not found any tools that met the varying requirements people wanted to set --- mainly, that it be possible to commit readable source text for figures into our git repo and have updates that were reviewable (ie, didn't amount to a 100% replacement of one set of gibberish with another one). In the distant past, as I recall, we had a GIF or two; but we abandoned that on the grounds that it was unmaintainable and also incompatible with some documentation output formats. I'm not too sure what the state of play is on the latter point, now that we've switched to XML. regards, tom lane