RE: [computer-go] XML alternatives to SGF
Some critics of an XML-based go format seem to be involved in a paranoid fantasy that they are going to be forced by evil goblins to use it against their will. No, Jennifer, that's not the case. Sure, if the format becomes popular they may end up having to deal with it, but XML-formatted game records and SGF will be round-trippable in the blink of an eye. Many of those complaining about XML don't seem to really know too much about it. Griping that it's too big is roughly equivalent to wanting to go back to six-bit character encodings. Carping that it's not readable misses the point. Those who wonder if their favorite platform/language might not support XML haven't checked recently. The point is that XML offers an incredibly rich environment of transformability and extensibility and interoperability. Go programmers are mostly interested in move sequence data, and that's natural, but we should remember that there are lots of other pieces to the overall puzzle, including commentary, organization of problem sets, and how diagrams are handled. Let's take just a few examples. Say we want to store metadata in or alongside SGF files, and/or retrieve/search/index the metadata already in them, such as the name of the source. If the metadata is in XML, probably in a well-established format such as Dublin Core or an extension thereof, it can be discovered and processed by any search engine or, in the not too distant future, reasoning engine, to answer queries such as find all games played by Shuko in 1971. In addition, XML has a built-in mechanism for extending vocabularies (namespaces). This allows information specific to a particular application to be included in a document, with well-understood characteristics that allow other applications to ignore the extra stuff. DocBook is an example of an XML-based document format for articles and books, technical and otherwise, For instance, O'Reilly uses a variant of DocBook for all its publishing. Using XML to represent go information would make it much easier to integrate with document formats such as DocBook. As someone pointed out, using XML would lay to rest once and for all any questions about character encodings. It also provides the built-in xml:lang mechanism to represent parallel textual information in different languages, very useful for Oriental players' names, to give just one example. Many people think of XML documents only as text files, but in fact they can take any form, including being stored in databases which are optimized for performance in executing e.g. XQuery queries. How're ya gonnna do that with SGF? Many go applications are going to require additional types of information, such as the threaded commentaries mentioned by Bill S. Certainly compared to the option of forking SGF into a dozen proprietary formats which don't interoperate, or stuffing random s**t into the C[] field, would it not make sense to take the opportunity to upgrade to a single yet extensible model What say ye, architects of the computing universe? The XML formats that have been proposed thus far for go, unfortunately, lack imagination; they are little more than SGF with a thin XML veneer. In particular, they have the problem that they hang information such as commentary and diagrams off the game tree. What we need is a new format defined ground-up from an XML perspective. Realistically, putting up white-tower proposals is not going to be successful. What is needed is real-world proposals on which real-world applications are built, so that people can see the real-world benefit. Bob Myers -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Stuart A. Yeates Sent: Thursday, October 25, 2007 1:04 AM To: computer-go Subject: Re: [computer-go] XML alternatives to SGF I sat down and read the DTD and the documentation and have some direct feedback on it. I'm aware that the DTD is quite old, and some of the ideas and solutions I'm going to suggest might not have been available (or as popular) when the DTD was written. Lines starting with ! are quotes from the DTD. !-- P is the Paragraph of HTML -- !ELEMENT P (#PCDATA) Referencing HTML in this way doesn't allow validation. Defining the standard using schemas allow importing of concepts such as Paragraph of HTML directly from an appropriate HTML standard. !-- Each Go file consists of one or several GoGame -- !ELEMENT Go (GoGame*) I believe it is a mistake not to have a protocol version number here. !ELEMENT Application (#PCDATA) !ATTLIST Application format CDATA #IMPLIED It seems unfortunate that there is no explicit version number here and no url link to the application website. !ELEMENT Date (#PCDATA) !ATTLIST Date format CDATA #IMPLIED It would be great to define this in terms of a standard format (i.e. ISO date format), since more than once I've had to infer the formatting of a date an SGF file. !ELEMENT User (#PCDATA) The user tag is ambiguous
Re: [computer-go] XML alternatives to SGF
Bob I'm sorry if you read my message as saying that I'm not in favour of an XML file format for go. I'm actually very much in favour of such a thing, which is why I spent two hours getting to understand the current contender and pointing out some of the issues that need to be fixed. cheers stuart On 10/27/07, Bob Myers [EMAIL PROTECTED] wrote: Some critics of an XML-based go format seem to be involved in a paranoid fantasy that they are going to be forced by evil goblins to use it against their will. No, Jennifer, that's not the case. Sure, if the format becomes popular they may end up having to deal with it, but XML-formatted game records and SGF will be round-trippable in the blink of an eye. Many of those complaining about XML don't seem to really know too much about it. Griping that it's too big is roughly equivalent to wanting to go back to six-bit character encodings. Carping that it's not readable misses the point. Those who wonder if their favorite platform/language might not support XML haven't checked recently. The point is that XML offers an incredibly rich environment of transformability and extensibility and interoperability. Go programmers are mostly interested in move sequence data, and that's natural, but we should remember that there are lots of other pieces to the overall puzzle, including commentary, organization of problem sets, and how diagrams are handled. Let's take just a few examples. Say we want to store metadata in or alongside SGF files, and/or retrieve/search/index the metadata already in them, such as the name of the source. If the metadata is in XML, probably in a well-established format such as Dublin Core or an extension thereof, it can be discovered and processed by any search engine or, in the not too distant future, reasoning engine, to answer queries such as find all games played by Shuko in 1971. In addition, XML has a built-in mechanism for extending vocabularies (namespaces). This allows information specific to a particular application to be included in a document, with well-understood characteristics that allow other applications to ignore the extra stuff. DocBook is an example of an XML-based document format for articles and books, technical and otherwise, For instance, O'Reilly uses a variant of DocBook for all its publishing. Using XML to represent go information would make it much easier to integrate with document formats such as DocBook. As someone pointed out, using XML would lay to rest once and for all any questions about character encodings. It also provides the built-in xml:lang mechanism to represent parallel textual information in different languages, very useful for Oriental players' names, to give just one example. Many people think of XML documents only as text files, but in fact they can take any form, including being stored in databases which are optimized for performance in executing e.g. XQuery queries. How're ya gonnna do that with SGF? Many go applications are going to require additional types of information, such as the threaded commentaries mentioned by Bill S. Certainly compared to the option of forking SGF into a dozen proprietary formats which don't interoperate, or stuffing random s**t into the C[] field, would it not make sense to take the opportunity to upgrade to a single yet extensible model What say ye, architects of the computing universe? The XML formats that have been proposed thus far for go, unfortunately, lack imagination; they are little more than SGF with a thin XML veneer. In particular, they have the problem that they hang information such as commentary and diagrams off the game tree. What we need is a new format defined ground-up from an XML perspective. Realistically, putting up white-tower proposals is not going to be successful. What is needed is real-world proposals on which real-world applications are built, so that people can see the real-world benefit. Bob Myers -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Stuart A. Yeates Sent: Thursday, October 25, 2007 1:04 AM To: computer-go Subject: Re: [computer-go] XML alternatives to SGF I sat down and read the DTD and the documentation and have some direct feedback on it. I'm aware that the DTD is quite old, and some of the ideas and solutions I'm going to suggest might not have been available (or as popular) when the DTD was written. Lines starting with ! are quotes from the DTD. !-- P is the Paragraph of HTML -- !ELEMENT P (#PCDATA) Referencing HTML in this way doesn't allow validation. Defining the standard using schemas allow importing of concepts such as Paragraph of HTML directly from an appropriate HTML standard. !-- Each Go file consists of one or several GoGame -- !ELEMENT Go (GoGame*) I believe it is a mistake not to have a protocol version number here. !ELEMENT Application (#PCDATA) !ATTLIST Application format CDATA
Re: [computer-go] XML alternatives to SGF
Bob Myers [EMAIL PROTECTED] writes: Many of those complaining about XML don't seem to really know too much about it. The problem with XML is, that most people using it don't know anything about it and possible alternatives, using XML for everything, even when there are better alternatives available. XML as an technologie is not very interesting because it doesn't solve any problem, that wasn't solved before. The single advantage of XML is quite a bit of libraries and tools (but hey, there are formats with much more available libraries and tools, say S-Expressions) and that XML is taught to everyone today. By the way, XML does have quite some overhead and there are many platforms like embedded systems, where handling XML is much too expensive. And there is also quite some processing overhead at runtime. Not very much for a single application on modern PC hardware, but there is a cost and there are alternatives in most cases that gives you everything that XML gives you without those additional costs. So it's always good to know your tools and there alternatives. XML for everything ist just brain-dead. Griping that it's too big is roughly equivalent to wanting to go back to six-bit character encodings. That's bullshit. There are file formats with all possibilities of XML but without overhead. If you use XML only via tools and everyone knows, it's not humand readable, why not use binary formats? Yes, you can use XML and compress it, but only on disk. In memory you still have all this overhead and quite some big and complex libraries. The point is that XML offers an incredibly rich environment of transformability and extensibility and interoperability. S-Expressions offers an much richer environment of transformability and extensibility and interoperability. So what? a well-established format such as Dublin Core or an extension thereof To embed metadata of any standard in a file format you don't have to use XML. In addition, XML has a built-in mechanism for extending vocabularies (namespaces). This allows information specific to a particular application to be included in a document, with well-understood characteristics that allow other applications to ignore the extra stuff. The problem is not the syntax, that's easy and solved long ago in various other formats. Nothing new here. The real problem is semantics and there XML is of no help. Yes, XML has it's uses. It's a reasonable format for technical documents and that's why DocBook is really nice (if you really need different representations -- I would never use it if I have only need for one representation, say papers in journals; in this cases specialized format like LaTeX are much more powerful and expressive and also easier to handle). And in quite some cases XML does a good job as intermediate format for data exchange (but is often quite a mess as wire format in network protocols -- worst of two worlds in this case). As someone pointed out, using XML would lay to rest once and for all any questions about character encodings. Huh? You think this would only be possible with XML? It's a really simple problem with quite some solutions, all much simpler than running to XML. Many people think of XML documents only as text files, but in fact they can take any form, including being stored in databases which are optimized for performance in executing e.g. XQuery queries. Oh yes, go back in time to the 1960. Yea, hierachical databases rules, relational databases do suck so hard. Wait? When and why are RDBMS developed? Hmmm... there were problems with hierachical databases? But XML is so cool, there can't be any problems or drawbacks with all these cool tools available... What we need is a new format defined ground-up from an XML perspective. Please, n! -- Until the next mail..., Stefan. pgpDD73fn0wAG.pgp Description: PGP signature ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
In message [EMAIL PROTECTED], Bob Myers [EMAIL PROTECTED] writes Some critics of an XML-based go format seem to be involved in a paranoid fantasy that they are going to be forced by evil goblins to use it against their will. No, that is not the problem. The problem is that if there are two standards, programmers will split their efforts between them, supporting neither correctly, and causing far more incompatibility problems than we have at present. Even now, with only one format, there are enough problems in moving an SGF record from one application to another. No, Jennifer, that's not the case. Sure, if the format becomes popular they may end up having to deal with it, but XML-formatted game records and SGF will be round-trippable in the blink of an eye. Will this round-tripping be done by some magic bug-free program that you are able to provide? If so, could you please start with the much simpler task of providing one that will round-trip SGF-SGF, taking out all the incompatibilities with the standard and preserving all the information? Nick P.S. Bob, could you please configure your intelligentgo.org domain so that it no longer forwards to me all the emails send to its webmaster? -- Nick Wedd[EMAIL PROTECTED] ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
Many of those complaining about XML don't seem to really know too much about it. Dude. It's a file format. File formats don't solve problems. Data structures solve problems. XML is not a data structure, it is a very loosely specified way to arrange tags. By becoming so multipurpose it has the advantage of being able to be modified to do most anything, and the disadvantage of convincing many, many people that if they put it into XML that you'll be able to read it with your XML tools. I think that the 3-character extension is so easy to recognize that people tend to assume that it represents a very stable, steady and reliable way of moving information around. That'd be an error. s. __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
RE: [computer-go] XML alternatives to SGF
Bob Myers wrote: Many of those complaining about XML don't seem to really know too much about it. That is exactly my point. I don't know and I don't want to know! SGF is fine. It has been stable for years because there is no problem at all. Should we find a problem, there is a straightforward solution: a new SGF version. At the end, either creating your own XML parser (by experience, the best option) or entering the nightmare of depending on someone else's library, bugs, updates, limitations, etc. costs A LOT OF TIME. For what? For nothing. Again, sgf works fine. Nothing would be as stupid as having go games split in 2 incompatible formats. Two formats for the same thing is just a tax on go development: Its not enough with one effort for making your program compatible you have to do it twice. Who wins anything with that? I can't guess. Jacques. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
On 23/10/2007, Gunnar Farnebäck [EMAIL PROTECTED] wrote: A potential problem with an XML library is the internal representation of the game tree. For debugging purposes it's not unusual to dump reading trees containing literally millions of moves, sometimes up to the limit of the available RAM. If an XML tree requires more bytes per move, the functionality would suffer. Does anybody know how big a node would become in expat for a move tag? There are two widespread models for parsing XML, DOM and SAX, SAX does not require you to be able to store the whole XML file in memory, it's a streaming model. Next problem is of course the file size of the game records. If they are 5 or 10 times as large we're talking 9 MB or 18 MB for the game records. Not a huge amount by itself but when considering the number of copies of GNU Go being distributed it sums up. I'm not sure about C, but in Java it's normal to pipe XML through gzip (which is included in the Java standard libraries), as this has been found to increase read/write speed (i.e. the cpu hit of (de-)compression is less than the speed up of writing fewer bytes to the disk). I've not studied it deeply, but I imagine a compressed XML file would be smaller than an SGF file. So what are the benefits? So far I haven't seen anything that is relevant for GNU Go. The readability is not really an issue, it's almost never possible to visualize a game record without a graphical viewer anyway, regardless of coordinate representation, and from the examples I've seen XML has been worse off than sgf on readability. Character sets are a non-issue for GNU Go, information about players is simply ignored. Version control conflicts have never happened with game records and I don't foresee it for the future. Where I would see a win for GnuGo might be: (*) a standard notation for move X should have been played A rather then B which would allow clients to provide direct, machine readable, feedback to developers and a potential format for regression tests. (*) a standard notation for representing compile-time and runtime arguments. (*) a standard notation for representing runtime information such as the top N moves considered. ... But I can provide a hint for something I would find useful. If it's something I'm missing in today's sgf viewers it's a good way to dump and inspect a transposition table. It's possible to expand the transpositions into a big tree with duplicate subtrees but that makes it very difficult to traverse it efficiently. Alternatively the tree is cut off when the same position is reached again but then there's no easy way to find where the position was first reached, which is needed to follow the continuations. My program doesn't use transposition tables, so I don't understand them enough to know whether this is practical. cheers stuart ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
I'm not an authority on SGF, but I am the owner of one of the larger SGF libraries (the KGS archives, 19,599,303 games as of this moment), and author one of the more common SGF editors (CGoban 3). Everybody wants something different from a SGF replacement. The syntax of SGF sucks, but once you write the parser your motivation to see it change drops to near zero. If it had to be replaced, I think JSON would be better than XML just because JSON is lighter and go doesn't really need the features that XML provides. I don't much care about the coordinates. I can't read c5 or cd either, I always use a program to parse. In general human readability isn't something that interests me at all, when I want to see a go game on paper I use the numbered diagrams. What would make my life significantly easier would be a way to searialize comments and tag them with the author instead of one C[] block per node. Breaking up the C[] into comments and figuring out who originally wrote the comment is helpful in some cases on KGS, and the current system is very clumsy. Another thing that would be nice (which a few other people here have mentioned) is a way to specify a path through an SGF tree. The KGS lessons are actually XML files that have bits of SGF embedded in them, describing how to incrementally build the SGF tree you get at the end. The audio in the lesson files is in blocks of base64-encoded SPEEX data. The whole XML file gets put into a ZIP file, packing it down very nicely...whenever people complain about the size of XML, I don't really see the issue, it compresses so nicely that you can easily get rid of the file size problem by a couple calls to zlib or java.util.compress or whatever the equivalent for your language is. So really there are only two things that would make my life easier, but whenever I look at it, and think about all the backward compatibility issues...well, I always decide to move on to something more useful. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
I sat down and read the DTD and the documentation and have some direct feedback on it. I'm aware that the DTD is quite old, and some of the ideas and solutions I'm going to suggest might not have been available (or as popular) when the DTD was written. Lines starting with ! are quotes from the DTD. !-- P is the Paragraph of HTML -- !ELEMENT P (#PCDATA) Referencing HTML in this way doesn't allow validation. Defining the standard using schemas allow importing of concepts such as Paragraph of HTML directly from an appropriate HTML standard. !-- Each Go file consists of one or several GoGame -- !ELEMENT Go (GoGame*) I believe it is a mistake not to have a protocol version number here. !ELEMENT Application (#PCDATA) !ATTLIST Application format CDATA #IMPLIED It seems unfortunate that there is no explicit version number here and no url link to the application website. !ELEMENT Date (#PCDATA) !ATTLIST Date format CDATA #IMPLIED It would be great to define this in terms of a standard format (i.e. ISO date format), since more than once I've had to infer the formatting of a date an SGF file. !ELEMENT User (#PCDATA) The user tag is ambiguous, is this a person's name? a user name? a user name on what server? !ELEMENT Copyright (P+) It would be great to use a URL here to the licence under which the file is being distributed, for example, the creative commons licences on a lot of web content these days. !ELEMENT Rules (#PCDATA) !ATTLIST Rules format CDATA #IMPLIED Using a url to a ruleset here would be great. Even better would be a machine-interpretable ruleset, but I'm not counting on that anytime soon. !ELEMENT Black ((at)*) Using schemas allows the content of tags to be restricted. See also discussion in the docs. !-- This is to take care of SGF tags, which are not translated -- !ELEMENT SGF (Arg*) !ATTLIST SGF type CDATA #REQUIRED !ELEMENT Arg (#PCDATA) This introduces ambiguity into the file format, since it is unclear what the precedence is. If the XML says one thing and the embedded SGF tags say another, which has precedence. cheers stuart On 10/22/07, Jason House [EMAIL PROTECTED] wrote: An XML alternative [1] to SGF has recently come to my attention. What do others think of this alternative? Personally, the effect of a tag affecting the previous tag seems kind of strange to me. PS: I found out about this from [2], a recently closed GoGui feature request to write more sane sgf files that contain the standard algebraic notation used in all GUIs. [1] http://www.rene-grothmann.de/jago/Documentation/xml.html [2] https://sourceforge.net/tracker/?func=detailatid=489967aid=1752711group_id=59117 ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
RE: [computer-go] XML alternatives to SGF
GMP (go modem protocol) was invented for direct computer to computer play using a 300 baud modem, before the internet existed. It was used in tournaments since it was easy to connect up serial ports to emulate modems. GTP solves a completely different problem, of go engines communicating with a GUI or referee. GMP is no longer interesting since no one does direct computer to computer connection using modems today. GTP may replace GMP as a tournament method, but it is in no way a replacement for GMP. GMP is simply obsolete, since the problem it solves no longer exists. Replacing sgf with xml is very different, since they do the same thing. David -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jeff Nowakowski Sent: Tuesday, October 23, 2007 5:57 AM To: computer-go Subject: Re: [computer-go] XML alternatives to SGF On Tue, 2007-10-23 at 08:42 -0400, Don Dailey wrote: GTP pretty much replace GMP.A lot of resistance because GMP was the defacto standard at the time. It would have been foolish to insist on being backwards compatible. GTP was a huge change in protocol with clear benefits. What's being quibbled over now is minor change in the coordinate system at the cost of breaking all existing tools, with the exception of a couple that have implemented this incompatible change. The benefit does not outweight the cost. -Jeff ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
RE: [computer-go] XML alternatives to SGF
There is a standard file format like PGN for Go, that uses standard go notation. It's the Ishi Press Go Format used by the original Many Faces of go, and still supported by Many Faces. You might still find files out there with a .go extension. It was invented before sgf, but the go commuinity preferred sgf, I think because the files were smaller. I think Anders supports using Korshelt notation in sgf (E4, etc, with no I). David I'd just like to see adoption of a standard file format for go that is human readable. To me, that requires standard algebraic notation such as E4.I'd definitely prefer to see an sgf-like or PGN-like variant over XML. I can't yet commoent on JSON or YAML. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
RE: [computer-go] XML alternatives to SGF
Anders Kierulf, who created the sgf standard for the go program he wrote for his Ph.D thesis in 1990. Search for Smart Go to see his current go program. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Jason House Sent: Thursday, October 25, 2007 11:45 AM To: computer-go Subject: Re: [computer-go] XML alternatives to SGF On 10/25/07, David Fotland [EMAIL PROTECTED] wrote: I think Anders supports using Korshelt notation in sgf (E4, etc, with no I). Who or what is Anders? ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
I tried to manually compose a JSON example to roughly match the xml
example Stuart Yates gave. I don't know if I did it right because I'm
not an expert on JSON although I've used it a little bit in javascript
programming:
{
White: John Doe,
Black: Fred Johnson,
BoardSize: 19,
Moves: [ D16, E16, H13, D15, E12, C16, G15 ]
}
The moves can be on separate lines if you want them to.
JSON is designed to be read directly into a data structure in your
program. Square brackets are for lists, strings are quoted and colons
represent records or hashes.
This would be directly read into a data structure in your programming
language of choice.
I'm guessing here because I haven't looked at JSON in C, but it might
place this information in the following data structure:
typedef struct
{
char *White;
char *Black;
int boardsize;
char **Moves;
};
There would have to be some agreement on what to call the names of the
various elements.
Variations can be expressed rescursively with JSON, just as in SGF, so
you can make trees if you want to.
- Don
Stuart A. Yeates wrote:
I've been looking further at the jago xml format, and for a very
simple game it looks like:
?xml version=1.0 encoding=utf-8?
?xml-stylesheet href=go.xsl type=text/xsl?
!DOCTYPE Go SYSTEM go.dtd
Go
GoGame name=*
Information
ApplicationJago:Version 4.7/Application
BoardSize19/BoardSize
/Information
Nodes
Node/
Black number=1 at=D16/
White number=2 at=E16/
Black number=3 at=H13/
White number=4 at=D15/
Black number=5 at=E12/
White number=6 at=C16/
Black number=7 at=G15/
White number=8 at=D17/
/Nodes
/GoGame
/Go
cheers
stuart
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
I saw this on the web a JSON library for C:
JSON-C implements a reference counting object model that allows you to
easily construct JSON objects in C, output them as JSON formatted
strings and parse JSON formatted strings back into the C representation
of JSON objects.
I think I will change CGOS over to using JSON for the game records
:-)Just kidding!
But seriously, a simple system just for storing games would be to use
the same tags that the
xml version uses and use it pretty much like I have below - assuming
that is valid JSON.
Don't worry, I'm not going to push for yet another standard - but I do
hate SGF.
- Don
Don Dailey wrote:
I tried to manually compose a JSON example to roughly match the xml
example Stuart Yates gave. I don't know if I did it right because I'm
not an expert on JSON although I've used it a little bit in javascript
programming:
{
White: John Doe,
Black: Fred Johnson,
BoardSize: 19,
Moves: [ D16, E16, H13, D15, E12, C16, G15 ]
}
The moves can be on separate lines if you want them to.
JSON is designed to be read directly into a data structure in your
program. Square brackets are for lists, strings are quoted and colons
represent records or hashes.
This would be directly read into a data structure in your programming
language of choice.
I'm guessing here because I haven't looked at JSON in C, but it might
place this information in the following data structure:
typedef struct
{
char *White;
char *Black;
int boardsize;
char **Moves;
};
There would have to be some agreement on what to call the names of the
various elements.
Variations can be expressed rescursively with JSON, just as in SGF, so
you can make trees if you want to.
- Don
Stuart A. Yeates wrote:
I've been looking further at the jago xml format, and for a very
simple game it looks like:
?xml version=1.0 encoding=utf-8?
?xml-stylesheet href=go.xsl type=text/xsl?
!DOCTYPE Go SYSTEM go.dtd
Go
GoGame name=*
Information
ApplicationJago:Version 4.7/Application
BoardSize19/BoardSize
/Information
Nodes
Node/
Black number=1 at=D16/
White number=2 at=E16/
Black number=3 at=H13/
White number=4 at=D15/
Black number=5 at=E12/
White number=6 at=C16/
Black number=7 at=G15/
White number=8 at=D17/
/Nodes
/GoGame
/Go
cheers
stuart
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
Heikki Levanto wrote: Even if a new proposed standard would have many benefits, obvious to everyone (which I have not yet seen), I would stuill urge people to consider those benefits carefully, and to weigh them against the problems that arise from having two incompatible standards. I agree 100%. Unfortunately we (all of us, I guess) all have few time for go programming. The most crazy idea to invest that time in is yet another file format. Personally, I won't do anything against sgf. I just expect that the XML idea fails. If it doesn't, that may be a problem in a few years. Jacques. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
On Tue, 2007-10-23 at 22:31 +0200, Gunnar Farnebäck wrote: But I can provide a hint for something I would find useful. If it's something I'm missing in today's sgf viewers it's a good way to dump and inspect a transposition table. It's possible to expand the transpositions into a big tree with duplicate subtrees but that makes it very difficult to traverse it efficiently. Alternatively the tree is cut off when the same position is reached again but then there's no easy way to find where the position was first reached, which is needed to follow the continuations. There have been proposals for this on the SGF mailinglist, IIRC. For a similar purpose (a joseki library), I have used the Node Name (N property) for this purpose, which is acceptable for 'private use'; but for real functionality the SGF-browser should be aware of this usage. A real extention should either implement 'unique node names', or use a label+goto -like mechanism. An advantage of using the N[] property is that it will not break existing implementations. Another feature request (by Frank de Groot of moyogo fame, IIRC) was the introduction of 'compressed movelists', eg ;BW [ddppdp] instead of ;B[dd] ;W[pp] ;B[dp] This would allow for a more compact representation of games. (for storing them in fixed-size database records, etc.) Adriaan van Kessel ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
On Mon, Oct 22, 2007 at 10:03:56PM -0700, Phil G wrote: To my surprise, GoGui can already read SGF with standard coordinates! :) I think you are muddying the waters by calling non-standard extensions 'standard' coordinates. I don't see any reason why [C4] should be harder to read than [cd]. Not for humans, and not for computers. I see many good reasons for sticking to *one* standard, and everyone using the same. At least for exchange of data - what you use internally in your programs is of course your own business. Even if a new proposed standard would have many benefits, obvious to everyone (which I have not yet seen), I would stuill urge people to consider those benefits carefully, and to weigh them against the problems that arise from having two incompatible standards. Just my personal opinion, of course. - Heikki -- Heikki Levanto In Murphy We Turst heikki (at) lsd (dot) dk ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
From: Heikki Levanto [EMAIL PROTECTED] I don't see any reason why [C4] should be harder to read than [cd]. Not for humans, and not for computers. Actually, C4 is not [cd]. C4 is actually [cf] on a 9x9 board. It's something else on a 19x19 or 13x13 board. The row character depends on the board size. SGF coordinate for a row are labeled top to bottom. Go boards are exactly opposite. In my opinion, it is SGF that uses a non-standard coordinate system to represent positions on a Go board. As a community, I believe we can improve SGF by extending the specification slightly to allow points to also be encoded in standard coordinates and depreciated, admittedly slowly, the use of the old coordinate system. We already see Go programs (SmartGo, GoGui and others) supporting this format. Maybe this can be a key point in the proposed FF[5] specification. Phil___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
On Tue, 2007-10-23 at 01:16 -0700, Phil G wrote: As a community, I believe we can improve SGF by extending the specification slightly to allow points to also be encoded in standard coordinates and depreciated, admittedly slowly, the use of the old coordinate system. We already see Go programs (SmartGo, GoGui and others) supporting this format. Maybe this can be a key point in the proposed FF[5] specification. As a community, I don't think we should support *breaking* an existing, well-established data format over something as trivial as the coordinate system used. There are much bigger issues with sgf, such as not being able to follow a teaching review that jumps around nodes. What I would support is a new standard that was backwards compatible with existing tools. -Jeff ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
Heikki Levanto wrote: On Mon, Oct 22, 2007 at 10:03:56PM -0700, Phil G wrote: To my surprise, GoGui can already read SGF with standard coordinates! :) I think you are muddying the waters by calling non-standard extensions 'standard' coordinates. I don't see any reason why [C4] should be harder to read than [cd]. Not for humans, and not for computers. I have a few books on GO and there are many articles on the web and they all seem to use one standard. Are you saying that the standard is really 'cd' and not 'C4' ? If we are to stick to one standard what are you saying it should be? I see many good reasons for sticking to *one* standard, and everyone using the same. At least for exchange of data - what you use internally in your programs is of course your own business. Even if a new proposed standard would have many benefits, obvious to everyone (which I have not yet seen), I would stuill urge people to consider those benefits carefully, and to weigh them against the problems that arise from having two incompatible standards. Just my personal opinion, of course. - Heikki ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
There is a lot to be said about standards, but common sense should prevail. Very few things have remained the same in computers, such as the program language you use.I've programmed for years and C is not the same as it was when I started and will not compile on the original compilers. Is that a bad thing? Heck no! I much prefer the advancements over slavish devotion to remain compatible at all costs. On the other hand I don't think standards should be broken at the drop of a hat. Microsoft breaks standards on purpose as a matter of course just to remain incompatible and to distinguish themselves and discourage people from moving to superior operating system. GTP pretty much replace GMP.A lot of resistance because GMP was the defacto standard at the time. It would have been foolish to insist on being backwards compatible. I think this is a good move. Make your software read both formats seamlessly, and make it write either format optionally. Provide conversion utilities that convert in both directions. It's a relatively painless improvement to SGF. - Don Jeff Nowakowski wrote: On Tue, 2007-10-23 at 01:16 -0700, Phil G wrote: As a community, I believe we can improve SGF by extending the specification slightly to allow points to also be encoded in standard coordinates and depreciated, admittedly slowly, the use of the old coordinate system. We already see Go programs (SmartGo, GoGui and others) supporting this format. Maybe this can be a key point in the proposed FF[5] specification. As a community, I don't think we should support *breaking* an existing, well-established data format over something as trivial as the coordinate system used. There are much bigger issues with sgf, such as not being able to follow a teaching review that jumps around nodes. What I would support is a new standard that was backwards compatible with existing tools. -Jeff ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
On Tue, 2007-10-23 at 08:42 -0400, Don Dailey wrote: GTP pretty much replace GMP.A lot of resistance because GMP was the defacto standard at the time. It would have been foolish to insist on being backwards compatible. GTP was a huge change in protocol with clear benefits. What's being quibbled over now is minor change in the coordinate system at the cost of breaking all existing tools, with the exception of a couple that have implemented this incompatible change. The benefit does not outweight the cost. -Jeff ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
to be fair, most KR code will compile on modern compilers, if you ask nicely. s. - Original Message From: Don Dailey [EMAIL PROTECTED] To: computer-go computer-go@computer-go.org Sent: Tuesday, October 23, 2007 8:42:15 AM Subject: Re: [computer-go] XML alternatives to SGF There is a lot to be said about standards, but common sense should prevail. Very few things have remained the same in computers, such as the program language you use.I've programmed for years and C is not the same as it was when I started and will not compile on the original compilers. Is that a bad thing? Heck no! I much prefer the advancements over slavish devotion to remain compatible at all costs. On the other hand I don't think standards should be broken at the drop of a hat. Microsoft breaks standards on purpose as a matter of course just to remain incompatible and to distinguish themselves and discourage people from moving to superior operating system. GTP pretty much replace GMP.A lot of resistance because GMP was the defacto standard at the time. It would have been foolish to insist on being backwards compatible. I think this is a good move. Make your software read both formats seamlessly, and make it write either format optionally. Provide conversion utilities that convert in both directions. It's a relatively painless improvement to SGF. - Don Jeff Nowakowski wrote: On Tue, 2007-10-23 at 01:16 -0700, Phil G wrote: As a community, I believe we can improve SGF by extending the specification slightly to allow points to also be encoded in standard coordinates and depreciated, admittedly slowly, the use of the old coordinate system. We already see Go programs (SmartGo, GoGui and others) supporting this format. Maybe this can be a key point in the proposed FF[5] specification. As a community, I don't think we should support *breaking* an existing, well-established data format over something as trivial as the coordinate system used. There are much bigger issues with sgf, such as not being able to follow a teaching review that jumps around nodes. What I would support is a new standard that was backwards compatible with existing tools. -Jeff ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
But that's backward compatibility.We don't write code that compiles on old compliers even though we can compile older code on new compilers. Which is exactly what is being suggested by me. We can improve SGF without losing the ability to read older SGF files with no sweat. - Don steve uurtamo wrote: to be fair, most KR code will compile on modern compilers, if you ask nicely. s. - Original Message From: Don Dailey [EMAIL PROTECTED] To: computer-go computer-go@computer-go.org Sent: Tuesday, October 23, 2007 8:42:15 AM Subject: Re: [computer-go] XML alternatives to SGF There is a lot to be said about standards, but common sense should prevail. Very few things have remained the same in computers, such as the program language you use.I've programmed for years and C is not the same as it was when I started and will not compile on the original compilers. Is that a bad thing? Heck no! I much prefer the advancements over slavish devotion to remain compatible at all costs. On the other hand I don't think standards should be broken at the drop of a hat. Microsoft breaks standards on purpose as a matter of course just to remain incompatible and to distinguish themselves and discourage people from moving to superior operating system. GTP pretty much replace GMP.A lot of resistance because GMP was the defacto standard at the time. It would have been foolish to insist on being backwards compatible. I think this is a good move. Make your software read both formats seamlessly, and make it write either format optionally. Provide conversion utilities that convert in both directions. It's a relatively painless improvement to SGF. - Don Jeff Nowakowski wrote: On Tue, 2007-10-23 at 01:16 -0700, Phil G wrote: As a community, I believe we can improve SGF by extending the specification slightly to allow points to also be encoded in standard coordinates and depreciated, admittedly slowly, the use of the old coordinate system. We already see Go programs (SmartGo, GoGui and others) supporting this format. Maybe this can be a key point in the proposed FF[5] specification. As a community, I don't think we should support *breaking* an existing, well-established data format over something as trivial as the coordinate system used. There are much bigger issues with sgf, such as not being able to follow a teaching review that jumps around nodes. What I would support is a new standard that was backwards compatible with existing tools. -Jeff ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
I find it interesting the position I am taking on this.I'm a bit of believer in not making your parser too forgiving, or gratuitously breaking standards because it encourages sloppiness. In the days when PGN files for chess were just starting to become popular, it was horrible what you would find in the PGN files. Castling could appear as o-o, O-O, 0-0 and e1g1 among other things.Instead of Nbd2 you might have the incorrect disambiguation of N1d2 - which describes the move un-ambiguously, but is non-standard (unless there is also a knight on b3 in which case N1d2 is correct.) So many people made their parsers forgiving because many of the problems were easily corrected and it was easy to tell what move was intended.The idea was to make you parser strict on export, forgiving on import. The problem with being forgiving on input is that it doesn't encourage others to strictly adhere to the standards. Suddenly new defacto standards are acceptable which can complicate things. Once everyone else forgives, you must forgive or suddenly your software becomes non-standard in a defacto way. I think the right way to deal with this is to always be strict but provide cleaner tools. If a tool can correct a non-standard format, make it clear that it is broken and needs correction.Then the file can be passed on in a properly working state. However, I differently with this issue of allowing standard notation with SGF. I advocate defining it as acceptable. One of the primary reasons for having non-binary formats such as SGF and XML is to make it human readable.It's supposed to be made easy to edit these files manually without absolutely requiring special tools, not to mention the ease of debugging. I would rather look at a text file than a hex dump and is why we use text formats instead of binary formats. Despite posts to the contrary, ae is harder for humans to read than E5 for two reasons: 1. We are used to E5 notation. This IS the standard used by everything except SGF. Esperanto may be just as easy to learn as English but nobody speaks it (or only about 1 million out of 6.5 billion do.) 2. I would argue that having a letter and a number is more clear than having 2 letters or 2 numbers. This is an aid to distinguishing file and rank, or row and column. If I say the A edge, it means something with e5 notation but it could mean 2 different things with aa notation.Number is always row, Letter is always column. No big deal, but nice. - Don Jeff Nowakowski wrote: On Tue, 2007-10-23 at 08:42 -0400, Don Dailey wrote: GTP pretty much replace GMP.A lot of resistance because GMP was the defacto standard at the time. It would have been foolish to insist on being backwards compatible. GTP was a huge change in protocol with clear benefits. What's being quibbled over now is minor change in the coordinate system at the cost of breaking all existing tools, with the exception of a couple that have implemented this incompatible change. The benefit does not outweight the cost. -Jeff ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
On Oct 23, 2007, at 9:01 AM, Stuart A. Yeates wrote: Much of the discussion in this thread has focused very narrowly on using an XML format to replace SGF, I believe that if an XML format is to take off, it should offer capabilities beyond what are possible in SGF, conversion to XML for XMLs sake is pointless. Possibilities include: * A method for presenting translated comments, i.e. the same comments in different languages, so a program can display only the appropriate ones. cheers stuart I'm surprised no one has brought up an obvious benefit of XML: well defined character encoding. A prime weakness of SGF is its unspecified character encoding. I thought that SGF is not well accepted in Asia due to this limitation. Re: coordinate normalization. This seems really trivial to me, easily something that could be incorporated as an option into the existing SGF standard. The code change would be minor. (I also presume that any proposal would retain the quirk of skipping the letter i in the horizontal coordinates?) Ian ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
What is used in Asia? Terry McIntyre [EMAIL PROTECTED] They mean to govern well; but they mean to govern. They promise to be kind masters; but they mean to be masters. -- Daniel Webster - Original Message From: Ian Osgood [EMAIL PROTECTED] To: computer-go computer-go@computer-go.org Sent: Tuesday, October 23, 2007 9:26:25 AM Subject: Re: [computer-go] XML alternatives to SGF On Oct 23, 2007, at 9:01 AM, Stuart A. Yeates wrote: Much of the discussion in this thread has focused very narrowly on using an XML format to replace SGF, I believe that if an XML format is to take off, it should offer capabilities beyond what are possible in SGF, conversion to XML for XMLs sake is pointless. Possibilities include: * A method for presenting translated comments, i.e. the same comments in different languages, so a program can display only the appropriate ones. cheers stuart I'm surprised no one has brought up an obvious benefit of XML: well defined character encoding. A prime weakness of SGF is its unspecified character encoding. I thought that SGF is not well accepted in Asia due to this limitation. Re: coordinate normalization. This seems really trivial to me, easily something that could be incorporated as an option into the existing SGF standard. The code change would be minor. (I also presume that any proposal would retain the quirk of skipping the letter i in the horizontal coordinates?) Ian ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
I'm surprised no one has brought up an obvious benefit of XML: well defined character encoding. A prime weakness of SGF is its unspecified character encoding. I thought that SGF is not well accepted in Asia due to this limitation. I'm also a firm believer in UTF-8 only. This is an example of insanity, having so many different encodings. But please let's not talk about that one here. Re: coordinate normalization. This seems really trivial to me, easily something that could be incorporated as an option into the existing SGF standard. The code change would be minor. (I also presume that any proposal would retain the quirk of skipping the letter i in the horizontal coordinates?) That's how I feel. It's trivial and highly benefical. Not everyone would benefit because most people never directly look at SGF. However it's supposed to be human readable friendly and this is a no-brainer. - Don Ian ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
I'm surprised no one has brought up an obvious benefit of XML: well defined character encoding. A prime weakness of SGF is its unspecified character encoding. I thought that SGF is not well accepted in Asia due to this limitation. I'm also a firm believer in UTF-8 only. This is an example of insanity, having so many different encodings. But please let's not talk about that one here. Re: coordinate normalization. This seems really trivial to me, easily something that could be incorporated as an option into the existing SGF standard. The code change would be minor. (I also presume that any proposal would retain the quirk of skipping the letter i in the horizontal coordinates?) That's how I feel. It's trivial and highly benefical. Not everyone would benefit because most people never directly look at SGF. However it's supposed to be human readable friendly and this is a no-brainer. - Don Ian ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
On Mon October 22 2007 18:24, Don Dailey wrote: On Mon October 22 2007 10:15, Don Dailey wrote: it also seems to be hard to write an SGF file without bugs. 20% of the games or 20% of the sources? 20% of the games could have come from a single source. from different sources. You can check it yourself. Most of the files came from freely available sources linked from here: http://www.u-go.net/links/gamerecords/ I modified GoGui to output non-critical warnings for this experiment. Usually it warns only about problems that could cause information loss. For example KM[] or KM[ 6.5 ] (with spaces) are both invalid according to the standard, even if a forgiving SGF reader will accept them. So are you arguing that you should have no control over properties? Is this a strength of XML that you cannot define properties in a flexible way? yes. Allowing everyone to add non-standard properties means that you cannot validate the files in a meaningful way anymore. Also, I haven't really seen convincing use cases for non-standard properties. SGF defines (more than) enough, even if some of them are a bit underspecified (like OT allowing arbitrary text, which makes this part of the time settings practically unparsable for programs). That's the problem. The constant pressure to add more and more libraries to do simple things in more complicated ways. I don't want to link in yet another library to my code. its not about adding more and more. Its about selecting a few best-practice norms and conventions. XML is a standard that is used by a large number of projects and it handles problems on an intermediate layer that every complicated file format will run into. IMO, reinventing the wheel in this case is inferior coding practice. But I am not pushing for a new standard here. It was Jason who wrote the initial mail about the topic. Remembering earlier discussions on the list, it is not surprising to see the usual amount of XML hate mails. I simply wanted to support an XML-based file format in GoGui and I selected Jago's XML format, because it already seems to be supported by some existig programs: Jago, qGo, glGo (read support only?), GoSVG. The next major version of GoGui will also include a converter program that allows conversion from SGF to XML and backwards. - Markus ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
I've been looking further at the jago xml format, and for a very simple game it looks like: ?xml version=1.0 encoding=utf-8? ?xml-stylesheet href=go.xsl type=text/xsl? !DOCTYPE Go SYSTEM go.dtd Go GoGame name=* Information ApplicationJago:Version 4.7/Application BoardSize19/BoardSize /Information Nodes Node/ Black number=1 at=D16/ White number=2 at=E16/ Black number=3 at=H13/ White number=4 at=D15/ Black number=5 at=E12/ White number=6 at=C16/ Black number=7 at=G15/ White number=8 at=D17/ /Nodes /GoGame /Go cheers stuart ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
On Tue, 23 Oct 2007, Markus Enzenberger wrote: its not about adding more and more. Its about selecting a few best-practice norms and conventions. XML is a standard that is used by a large number of projects and it handles problems on an intermediate layer that every complicated file format will run into. IMO, reinventing the wheel in this case is inferior coding practice. It is not re-inventing the wheel! SGF exists and is well supported for many years. Christoph ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
Jason House wrote: An XML alternative [1] to SGF has recently come to my attention. What do others think of this alternative? Personally, the effect of a tag affecting the previous tag seems kind of strange to me. For use in GNU Go it would need to have quite compelling benefits to become interesting. Let's look at numbers. GNU Go 3.7.10 roughly consists of 2.4 MB C code (83000 lines), 1.4 MB pattern data, 0.45 MB testcase files, 1.8 MB sgf game records (1500 sgf files), and 2 MB documentation. Of the C code 2600 lines come from the sgf library. If we want to use an available C library for XML, expat seems like a possible choice. The whole distribution is 2.5 MB but maybe it's possible to get away with the 400 kB (13000 lines) C code in the lib directory. Five times bigger than our sgf library but manageable. (That cannot be said of libxml2 though, with some 14 lines of code.) A potential problem with an XML library is the internal representation of the game tree. For debugging purposes it's not unusual to dump reading trees containing literally millions of moves, sometimes up to the limit of the available RAM. If an XML tree requires more bytes per move, the functionality would suffer. Does anybody know how big a node would become in expat for a move tag? Next problem is of course the file size of the game records. If they are 5 or 10 times as large we're talking 9 MB or 18 MB for the game records. Not a huge amount by itself but when considering the number of copies of GNU Go being distributed it sums up. So what are the benefits? So far I haven't seen anything that is relevant for GNU Go. The readability is not really an issue, it's almost never possible to visualize a game record without a graphical viewer anyway, regardless of coordinate representation, and from the examples I've seen XML has been worse off than sgf on readability. Character sets are a non-issue for GNU Go, information about players is simply ignored. Version control conflicts have never happened with game records and I don't foresee it for the future. But I can provide a hint for something I would find useful. If it's something I'm missing in today's sgf viewers it's a good way to dump and inspect a transposition table. It's possible to expand the transpositions into a big tree with duplicate subtrees but that makes it very difficult to traverse it efficiently. Alternatively the tree is cut off when the same position is reached again but then there's no easy way to find where the position was first reached, which is needed to follow the continuations. /Gunnar ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
yes. Allowing everyone to add non-standard properties means that you cannot validate the files in a meaningful way anymore. Also, I haven't really seen convincing use cases for non-standard properties. SGF defines (more than) enough, even if some of them are a bit underspecified ... I've been using the C[] tag whenever I've wanted to extend SGF with a non-standard property. It has the great advantage that just about every SGF viewer allows viewing and editing of comments; the same cannot be said for support of some of the more exotic commands. Re: XML vs. SGF, neither is particularly easy to parse, but sgf has the market share, so I think ultimately the argument is pointless. Re: changing SGF coord syntax: does anyone seriously read or edit sgf by hand? Get yourself an sgf viewer. The only people who ever have to deal with B[dc] are the programmers on this list, and I'm sure we're all capable of writing board_height-y. And, as I'm here spouting opinions, I think the bicycle shed should be painted chrome, so it looks all futuristic. Darren ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
[computer-go] XML alternatives to SGF
An XML alternative [1] to SGF has recently come to my attention. What do others think of this alternative? Personally, the effect of a tag affecting the previous tag seems kind of strange to me. PS: I found out about this from [2], a recently closed GoGui feature request to write more sane sgf files that contain the standard algebraic notation used in all GUIs. [1] http://www.rene-grothmann.de/jago/Documentation/xml.html [2] https://sourceforge.net/tracker/?func=detailatid=489967aid=1752711group_id=59117 ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I personally hate XML. It's crazy to reject SGF because of it's readability only to use XML which is verbose and only somewhat readable if you carefully construct it to be.What is crazy to me is that it was desgined to be HUMAN readable, and yet the argument to use it is that there are lots of libraries to create it and parse it. It's almost impossible to write XML manually without bugs. As far as I'm concerned you might as well use a much more efficient binary format if it's designed to be used with libraries for reading and writing. Just my opinion, don't want to start any wars here ... :-) Web pages that send xmlHttpRequests to a server are moving away from XML to the more readable and much more easily parsable JSON. The main advantage of JSON in my opinon is that it is a subset of the far more sane YAML Of course YAML and XML are not direct replacements for each other. They both have strengths and weaknesses. YAML is a data serialization language - you could describe a GO game in it, and basically have the data structure to describe this game to whatever language you write in. JSON looks like javascript data declarations and you can just directly eval it. XML has the advantage that it's an industry standard.So is JSON (which is YAML) but less so.YAML has the advantage that it's much easier to parse - software runs much faster if it's doing signficant parsing in YAML vs XML. I looked at the web site and it says near the top: XML is far more human readable than SGF I looked at the example XML game and it was horrific to read and incredibly verbose: ?xml version=1.0 encoding=utf-8? Go GoGame Information BoardSize19/BoardSize /Information CommentA test game/Comment Black at=Q16/ White at=Q4/ Black at=D4/ White at=D16/ Black at=F17/ White at=C14/ Black at=D18/ White at=C17/ Black at=J17/ /GoGame /Go I'm not suggesting that we use JSON, it has the same drawbacks as XML. Just my opinon, but if you were to suggest an alternative format the EASIEST to read and very easy to parse is PGN - yes, I know, the same format chess games are in. PGN is designed specifically to be easy to read. There is huge difference between being human readable and easy to read and pgn is definitely easiest of all formats to read. Here is an example: [Event Autotest Sequence] [Site Local Site] [Date 2007.02.08] [Round 4] [Black bas] [BlackTime 00:01:59] [White mog] [WhiteTime 00:05:44] [Boardsize 9x9] [Komi 7.5] [Rules Tromp/Taylor] [Final 41-40] [Result 0-1] 1. e5 f7 2. d4 g6 3. c5 g4 4. c7 f3 5. e3 f2 6. f5 g5 7. e2 e8 8. d8 d7 9. c8 e1 10. d6 g8 11. d1 f1 12. f4 d2 13. g3 g2 14. h3 h2 15. c2 c1 16. d3 j3 17. h4 h5 18. j2 j4 19. f9 e9 20. e7 d1 21. f8 g9 22. f6 a3 23. d9 f8 24. h7 g7 25. c3 b7 26. h3 j1 27. h1 g1 28. j8 j6 29. h6 h8 30. h4 b6 31. j7 c4 32. j5 j2 33. j6 h9 34. b2 a8 35. b9 b1 36. a4 b4 37. b8 j9 38. j5 j7 39. h6 g3 40. a9 a7 41. a5 b3 42. a1 a6 43. b5 h4 44. a2 c4 45. j6 h7 46. a3 j5 47. b4 h6 48. c6 b7 49. a7 b6 50. a8 pass 51. a6 b7 52. b6 pass 53. pass You can include tree information by using parenthesis to denote alternative lines. There are also provisions for comments and other such things. I know PGN doesn't address the issues mentioned in the article which is to have a common format for all applications. If that is incredibly important, a better choice is the YAML format. YAML parsers exist for every major programming language, it's more efficient to process, it's more readable and much less verbose. Yaml parses directly into the data structure you are using in your language, so you are ready to go. Having said that and despite the fact that I would prefer YAML (or JSON) over XML, I don't like it for the same reasons I don't like XML, it's not a NATURAL format for human consumption. I think it comes down to how important this is. If you care very little about readability and efficiency, use XML, it's a standard and many libraries exist. If you want readability, use PGN or SGF but with standard algebraic notation (it was a silly mistake not doing this from the start with SGF.) If you want a common format for all your applications use YAML which is superior to XML in readability, efficiency and simplicity. Interesting web site: http://www.xml.com/lpt/a/2002/07/24/yaml.html Starts out like this: XML combines all the inefficiency of text-based formats with most of the unreadability of binary formats. -- Oren Tirosh, comp.lang.python For a GO format, I would personally stick with the JSON subset of YAML, it's dirt simple, you can make your own parser in a few minutes and it's heavily used now with web sites and AJAX. WAY more sane than XML. - - Don Jason House wrote: An XML alternative [1] to SGF has recently come to my attention. What do others think of this alternative? Personally, the effect of a tag affecting the previous tag seems kind of strange to me. PS: I found
Re: [computer-go] XML alternatives to SGF
XML is overkill with little to no additional benefits. I can't read XML better than SGF. Plus it could be difficult to implemented on platforms which do not have pre-made libraries for reading/writing XML documents, and writing an XML parser from scratch is not a trivial task. However, I am in favor of changing the SGF format to allow coordinate encoding using the standard coordinates system rather than the one created just for SGF; i.e., a1 vs aa. This would make SGF files more readable by a human (and computer since it wouldn't need to know the board size to properly interpret the coordinates). To start, we just need the leading Go programs to read either encoding format (so they are backwards compatible). This should be somewhat trivial since you can tell which format just by looking at the coordinate encoding. And then sometime in the near future, Go programs switch to writing SGF files with standard coordinates. Phil - Original Message From: Jason House [EMAIL PROTECTED] To: computer-go computer-go@computer-go.org Sent: Monday, October 22, 2007 8:09:17 AM Subject: [computer-go] XML alternatives to SGF An XML alternative [1] to SGF has recently come to my attention. What do others think of this alternative? Personally, the effect of a tag affecting the previous tag seems kind of strange to me. PS: I found out about this from [2], a recently closed GoGui feature request to write more sane sgf files that contain the standard algebraic notation used in all GUIs. [1] http://www.rene-grothmann.de/jago/Documentation/xml.html [2] https://sourceforge.net/tracker/?func=detailatid=489967aid=1752711group_id=59117___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Of all the possibilities, I think what Phil says is the most sensible
thing.
- - Don
Phil G wrote:
!-- DIV {margin:0px;} --
XML is overkill with little to no additional benefits. I can't
read XML better than SGF. Plus it could be difficult to implemented on
platforms which do not have pre-made libraries for reading/writing XML
documents, and writing an XML parser from scratch is not a trivial task.
However, I am in favor of changing the SGF format to allow
coordinate encoding using the "standard" coordinates system rather than
the one created just for SGF; i.e., "a1" vs "aa". This would make SGF
files more readable by a human (and computer since it wouldn't need to
know the board size to properly interpret the coordinates). To start,
we just need the leading Go programs to read either encoding format (so
they are backwards compatible). This should be somewhat trivial since
you can tell which format just by looking at the coordinate encoding.
And then sometime in the near future, Go programs switch to writing SGF
files with "standard" coordinates.
Phil
- Original Message
From: Jason House [EMAIL PROTECTED]
To: computer-go computer-go@computer-go.org
Sent: Monday, October 22, 2007 8:09:17 AM
Subject: [computer-go] XML alternatives to SGF
An XML alternative [1] to SGF has recently come to my attention.
What do others think of this alternative? Personally, the effect of a
tag affecting the previous tag seems kind of strange to me.
PS: I found out about this from [2], a recently closed GoGui
feature request to write more sane sgf files that contain the standard
algebraic notation used in all GUIs.
[1] http://www.rene-grothmann.de/jago/Documentation/xml.html
[2]
https://sourceforge.net/tracker/?func=detailatid=489967aid=1752711group_id=59117
-
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHHPQ3DsOllbwnSikRAg8KAKDahJmWlV0lFTj9h3ShWHc5fjp+gQCeP82L
d3tJCM7NS8AvSQKbPk8c8ns=
=pe/r
-END PGP SIGNATURE-
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
On 10/22/07, Phil G [EMAIL PROTECTED] wrote: To start, we just need the leading Go programs to read either encoding format (so they are backwards compatible). This should be somewhat trivial since you can tell which format just by looking at the coordinate encoding. In my go bubble, the list of leading programs would be: * GoGui (which already supports reading the alternate format) * KGS To a lesser extent, it should probably include popular bots (when using the gtp loadsgf command): * Gnu Go * MogoBot I wouldn't mind providing a patch for gnu go. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
Also, if converters were supplied it would be easy. A simple utility to go either way. - Don Jason House wrote: On 10/22/07, *Phil G* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: To start, we just need the leading Go programs to read either encoding format (so they are backwards compatible). This should be somewhat trivial since you can tell which format just by looking at the coordinate encoding. In my go bubble, the list of leading programs would be: * GoGui (which already supports reading the alternate format) * KGS To a lesser extent, it should probably include popular bots (when using the gtp loadsgf command): * Gnu Go * MogoBot I wouldn't mind providing a patch for gnu go. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
I'll take a look at GoGui tonight and see if I can provide a patch (unless Markus does this first). - Original Message From: Jason House [EMAIL PROTECTED] In my go bubble, the list of leading programs would be: * GoGui (which already supports reading the alternate format) * KGS To a lesser extent, it should probably include popular bots (when using the gtp loadsgf command): * Gnu Go * MogoBot I wouldn't mind providing a patch for gnu go.___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
On Mon October 22 2007 10:15, Don Dailey wrote: almost impossible to write XML manually without bugs. it also seems to be hard to write an SGF file without bugs. I recently run a test on a collection of about 5000 SGF files from various sources on the web and more than 20% of them generated a warning when reading with GoGui. The most frequent problems are duplicate properties or root-properties in non-root nodes. Another problem I have with SGF is that it allows user-defined properties. While this is very appealing at first, because it allows to preserve unknown properties, it creates problems in practice. One is the potential of conflicting property keys, because there are no namespaces. Also, if a program does not know, what a property means, it cannot do much apart from preserving it when saving the file. But of other properties of a node were changed, the unknown property could have lost its context in which it is meaningful. I agree that reading an XML file without using a library is complicated, but there are enough libraries existing and even part of the standard libraries of some languages. The big advantage of an XML-based file format is that it leverages the power of meta-tools. Like it is a big advantage to use line-oriented text-based file formats on Unix, because you can do a lot of tasks simply using standard tools, like grepping, transforming, diffing and merging. But line-oriented text-based file formats reach their limits when you have data with a more complex structure, like trees. Think of resolving a conflict in a Go game file in your version control system. If the file format was XML-based, a meta-tool could merge changes like version control systems do right now with text-based files. Do you want to write a different conflict-resolvers for every file format? Or hava a look at this XSL style sheet: http://homepage.ntlworld.com/daniel.gilder/gosvg.html It creates a nice looking SVG vector graphic from a Jago XML file and the whole transformation is done by a standard tool with the transformation again described in an XML data file. - Markus ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
Markus Enzenberger wrote: On Mon October 22 2007 10:15, Don Dailey wrote: almost impossible to write XML manually without bugs. it also seems to be hard to write an SGF file without bugs. 20% of the games or 20% of the sources? 20% of the games could have come from a single source. In either case, I wasn't comparing SGF to XML, I was comparing it to JSON. I recently run a test on a collection of about 5000 SGF files from various sources on the web and more than 20% of them generated a warning when reading with GoGui. The most frequent problems are duplicate properties or root-properties in non-root nodes. Another problem I have with SGF is that it allows user-defined properties. While this is very appealing at first, because it allows to preserve unknown properties, it creates problems in practice. One is the potential of conflicting property keys, because there are no namespaces. Also, if a program does not know, what a property means, it cannot do much apart from preserving it when saving the file. But of other properties of a node were changed, the unknown property could have lost its context in which it is meaningful. So are you arguing that you should have no control over properties? Is this a strength of XML that you cannot define properties in a flexible way? I agree that reading an XML file without using a library is complicated, but there are enough libraries existing and even part of the standard libraries of some languages. That's the problem. The constant pressure to add more and more libraries to do simple things in more complicated ways. I don't want to link in yet another library to my code. The big advantage of an XML-based file format is that it leverages the power of meta-tools. That's one way to look at it, but it also demands the use of these tools. Suddenly it becomes problematic to use simple tools like an editor. My primary problem with XML is that a big deal is made out of it being human readable, calling this a virtue but then it's treated as if it were a binary format, requiring the use of libraries and other bloat-ware to deal with it. Not only is it verbose and bloated, but now you must add libraries and more bloat to deal with it. Like it is a big advantage to use line-oriented text-based file formats on Unix, because you can do a lot of tasks simply using standard tools, like grepping, transforming, diffing and merging. But line-oriented text-based file formats reach their limits when you have data with a more complex structure, like trees. There we go again - XML was supposed to be a text-based file format. I don't know what this has to do with using pipes in unix. The whole web thing with html tree's began with unix before most OS's even existed - but how is this relevant? Think of resolving a conflict in a Go game file in your version control system. If the file format was XML-based, a meta-tool could merge changes like version control systems do right now with text-based files. Do you want to write a different conflict-resolvers for every file format? What I don't want to do is become a hammer and think everything is a nail. Example: there are those who advocate XML as a programming language. Now this is just about as silly as it gets. There is a language called Superx++ which is written in XML. Superx++ is a buzzword compliant language if I ever saw one. Here is hello world written with this hammer: xpp xoutHello World!/xout /xpp I can't think of any reason I would use this language and I don't believe having to use meta-tools would be enough of an enticement to make we want to use this language. - Don Or hava a look at this XSL style sheet: http://homepage.ntlworld.com/daniel.gilder/gosvg.html It creates a nice looking SVG vector graphic from a Jago XML file and the whole transformation is done by a standard tool with the transformation again described in an XML data file. - Markus ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] XML alternatives to SGF
- Original Message From: Phil G [EMAIL PROTECTED] I'll take a look at GoGui tonight and see if I can provide a patch (unless Markus does this first). To my surprise, GoGui can already read SGF with standard coordinates! :)___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
