Re: [abcusers] ABC parser output data structure.
Christian wrote: The core of our abc subsystem (please, someone suggest a name for this thing) could be released as a library (a dll under Win32) that can be I question the idea of it being a .dll... the purpose of this is to be a universal parser for software written in any language, and on any platform. That's why I promote the use of a scripting language. Let me push Lua for a while. Lua is multiplatform. It has been compiled on any Unix flavour I've never heard of, on Palm, Windows, WinCE, Mac, PS2, XBOX, (don't know about Game Cube), this gives a boost about platform independence (both HW and SO). Lua has its native interface with C/C++ but other mechanisms are available: LuaJava, LuaAda, LuaCOM (for MS COM objects) and if someone needs a new interface it can count on the very active Lua community. Lua is widespread in the gaming industry (Grim Fandango, Impossible Creatures, MDK2, Homeworld 2, Escape from Monkey Island, ...) where speed, robustness and ease of use are a must. That's a guarantee for me! Anyway, I said that on Windows it COULD be distributed as a DLL but you are not forced to do so. Since we want to be able to embed ABCp (let's stick with this name for now) in other applications, we have no choice but structuring it as a library that could be linked statically or dinamically to the host application. If the linkage is dinamic, it means a .dll on Windows, a .so on Linux and whatever other mechanism is in place on the targeted platform. The command line tools will access the same libraries, again either statically or dinamically linked. The hard part, IMHO, is that of creating a language-independent interface (or worst, maintain a different interfacing mechanism for each language). Scripting languages as Lua (but also Python and Ruby, to be fair) already solved it . I think we are a bit off topic here with this technical issues, we risk to annoy someone! May I propose to move the technical discussion to the list on sourceforge (I'm going to subscribe right now) and coming back to this list for advice on how internally represent an ABC file, criticisms and other things more abc-ish? Bye, Remo To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
Steven Bennett wrote: I was going to let this idea die quietly... sigh If we were talking about creating a data interchange format, I'd agree 100%. But we're talking about creating a general purpose front-end parser that can be linked into assorted ABC programs so they don't have to write their own parser. Having the output of that be a text format defeats the whole purpose, because each application would then need to write it's own parser for *that* format. This SPECIFICALLY is what has confused me regarding this whole discussion... When I proposed the idea of a universal parser, I stated with NO AMBIGUITY that it was not to generate text. It was to have a control API which would could be asked questions and return answers. Once the file is parsed, a possible method would be list_titles(1,10) to return the primary titles of the first 10 tunes. This is just an example... not a well thought out API function... Anyways... an API of useful commands to glean information from the file, as well as commands to alter it. I would not object if it had a function to generate_musicxml() or something like that, but that would not be it's primary function. It's been suggested that such a second-stage parser would be a trivial job to write. As a programmer who has written a number of parsers of various complexity over my career, and looking at the amount and complexity of the data we're talking about, I'd have to say that trivial is doubtful. It's probably not a complex task, assuming the text format is designed properly, is totally unambiguous, comprehensive, and flexible enough to do the job. But it's bound to be tedious and time consuming. I suggest anyone who thinks this is a trivial job should actually design such an intermediate text format, and then try and write a parser for that format. Let me know how long it takes. ;-) As a final thought - if anyone *really* wants to see text output from such a parser, let the parser be written to output C structures or whatever the programmers want, then they can write their own back end to the parser to convert the output into any text output format desired. It should be relatively easy to do. And everyone can be happy. --Steve Bennett Jeff Szuhay wrote: On Wednesday, July 28, 2004, at 03:43 pm, Bernard wrote: [snip] The maximum is ascii. You can even read it without a computer. Flexibility is maximum in ascii. A new keyword is added and the old software doesn't understand the keyword and will ignore it. The problem of upgrading software is old software which won't read the new software's output at all. [snip] I agree completely with this. In fact, for the past 10 years, the whole of computing has moved towards ascii-based (character-based) data interchange standards away from binary data formats. To wit, SGML, HTML and its variants, XML, as well as scripting languages which remain text based and uncompiled (binary data). As an example, Perl, Python, Javascript run on more platforms than I know. A powerful database environment I worked in years go made every record available as text (it lives on in Universe and multi-value databases). LaTex and PostScript ... text based. The only (bad) reason to not use ascii for text based data IMNSHO is when a vendor wants to maintain control of their proprietary data format. Good for them but a real PITA for everyone else. To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html -- || Christian Marcus Cepel | And the wrens have returned [EMAIL PROTECTED] icq:12384980 | are nesting; In the hollow of 371 Crown Point, Columbia, MO| that oak where his heart once 65203-2202 573.999.2370 | had been; And he lifts up his Computer Support Specialist, Sr. | arms in a blessing; For being University of Missouri-Columbia | born again. --Rich Mullins To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
On Wednesday, July 28, 2004, at 03:43 pm, Bernard wrote: [snip] The maximum is ascii. You can even read it without a computer. Flexibility is maximum in ascii. A new keyword is added and the old software doesn't understand the keyword and will ignore it. The problem of upgrading software is old software which won't read the new software's output at all. [snip] I agree completely with this. In fact, for the past 10 years, the whole of computing has moved towards ascii-based (character-based) data interchange standards away from binary data formats. To wit, SGML, HTML and its variants, XML, as well as scripting languages which remain text based and uncompiled (binary data). As an example, Perl, Python, Javascript run on more platforms than I know. A powerful database environment I worked in years go made every record available as text (it lives on in Universe and multi-value databases). LaTex and PostScript ... text based. The only (bad) reason to not use ascii for text based data IMNSHO is when a vendor wants to maintain control of their proprietary data format. Good for them but a real PITA for everyone else. jw To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
That's what I was thinking. Unfortunately, I probably won't have a moment of free time until Sunday evening. umm - perhaps the best approach, then, would be to write the abc parser front end as part of the libmusicxml project. Then you can take advantage of their work... wil Paul Rosen wrote: I've been poking around the web a little bit and I came upon http://libmusicxml.sourceforge.net/ Paul Rosen --- Life is a musical, every once in a while the plot stops and you start singing and dancing --- http://home.earthlink.net/~catharsis.music/ http://home.earthlink.net/~theplums/ To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
I've been poking around the web a little bit and I came upon http://libmusicxml.sourceforge.net/ This is an open source project that I think is really similar to what we're thinking about, except that it is written for MusicXML. If we use the same structure that they've used, then any package that uses that can also import ABC with very little work, and alternately, any package we write for ABC will also be able to import MusicXML. I haven't looked deeply at their package yet, but I think they haven't done any cross language support, so we would have to add that. I plan on studying it and seeing if there is a way to use their code as a library and we would just need to concentrate on the parsing itself. Paul Rosen --- Life is a musical, every once in a while the plot stops and you start singing and dancing --- http://home.earthlink.net/~catharsis.music/ http://home.earthlink.net/~theplums/ To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
In message [EMAIL PROTECTED], John Chambers [EMAIL PROTECTED] writes I hope you've found the FreeTTS project. Yes. Although we haven't tested it yet for various reasons. Stephen -- Stephen Kellett Object Media Limitedhttp://www.objmedia.demon.co.uk RSI Information:http://www.objmedia.demon.co.uk/rsi.html To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
My concept of a parser is totally different. It's a standalone program which takes one file and converts to another, more generally usable. It would be called invisibly by any program needing it. That program would then read the simpler file in an obvious form and presumably delete it. I call that a Conversion Program. The parser is the difficult half of that. Once you have the output from the parser it should be trivial to write it to disk in whatever format you want. I imagine that would be useful on UNIX systems too, where the ABC is stdin and your format is stdout. Paul Rosen --- Life is a musical, every once in a while the plot stops and you start singing and dancing --- http://home.earthlink.net/~catharsis.music/ http://home.earthlink.net/~theplums/ To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
On 30 Jul 2004, at 13:34, John Chambers wrote: | In message [EMAIL PROTECTED], John Chambers | [EMAIL PROTECTED] writes | I hope you've found the FreeTTS project. | | Yes. Although we haven't tested it yet for various reasons. And, to make it relevant to this group, you'll have to start a sub-project to control the length and pitch of each syllable, so you can write a program that takes abc files with words and sings them in any of the available voices. (For those who wonder what I'm talking about, FreeTTS is an project that's producing an open-source text-to-speech package, all written in java.) Myriad, the people who make Melody/Harmony assistant are working on a browser plugin which displays, plays and even sings MusicXML. There's a mock-up demo of it here: http://www.myriad-online.com/misc/forum/testmusicxml/ (You'll have to download the plugin, but it seems fairly automatic.) Since they already have code to import abc, I wonder if they'll be making this capable of handling our format? Phil Taylor To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
easily, as is of course the parsing since this is very easy to write. (for information it's www.visiv.co.uk and the author provides documentation for the ascii file structure on the web site). I looked at the web site and checked out the format. It is not exactly the information we have to convey, it is much more lower level. For instance, it looked like you can place note stem flags where ever you want. That's not something in the scope of the parser. See my other message for more on file formats. I think we can find an output that would suit you. In fact, if you had an exact spec for a format that you'd like, that would influence me greatly. To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
To add my two cents to this discussion: While I have my doubts that any single parser design could fit the needs of more than maybe half (being generous) of the possible applications out there, it's still a worthwhile project. What projects are there? My imagination leads me to the following. I'd love to know how else ABC could be used. Standard Notation Tablature MIDI Player Transposer Indexer Pattern Matcher Conversion to another format Instruction (probably needing fragments) I would avoid producing output in any format which needs further text parsing, such as a text or XML format. Which leaves a C structure-based design a good Least Common Denominator -- most languages can access C structures in some fashion. But the problem with C structures is that they aren't easily extendible, which makes it difficult to add features and change things. Another huge problem with C structures is that if they contain pointers, those pointers become invalid when the structure is moved through memory. (Such as might happen on some platforms inter-process communications, or when moved to another language.) However, with proper planning, you can make the returned format highly extensible and highly portable by taking a couple of steps early on: lots of great ideas snipped I've been thinking along the same lines, although I haven't thought as much about exactly how to express the structures as you have. I'm a little worried that I'll come up with something that is clever in C++, but is a pain in other languages. One idea that I'm leaning towards is having multiple formats to pick from instead of trying to please everyone. In other words, have a set of functions like this (I don't like the names, but they are descriptive for now): void ParseAbcWithContainers(const string strAbcFormattedTune, StlBasedClass clsOutput, ErrorInfo clsErrors); void ParseAbcForFlatStruct(char* szAbcFormattedTune, HandmadeListStruct* pOutput, ErrorInfo* pErrors); void ParseAbcToText(char* szAbcFormattedTune, char* szOutput, char* szErrorInfo); This is not a big deal. Actually there are two very different issues with the output format. 1) What data belongs in it, and what is the relationship to other data? For instance, a rhythm needs to be associated with a note, because if there are three notes playing at the same time, they could all have different rhythms. However, a bowing mark needs to be associated with the entire vertical set of notes. This is a pretty big issue, and is crucial to get right. This has nothing to do with the physical representation of the data. 2) The physical representation of the data. There are a number of issues, including extensibility and ease of use in other languages, but in general, I see this as easier to deal with. There is no reason why we couldn't support multiple output formats. In fact, I could implement the population of the data as a templated class, so any C++ programmer could extend it. Perhaps there is a way to take that idea further and allow programmers of non-OO languages to create their own formats. Paul Rosen --- Life is a musical, every once in a while the plot stops and you start singing and dancing --- http://home.earthlink.net/~catharsis.music/ http://home.earthlink.net/~theplums/ To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
Paul Rosen wrote: What projects are there? My imagination leads me to the following. I'd love to know how else ABC could be used. Standard Notation Tablature MIDI Player Transposer Indexer Pattern Matcher Conversion to another format Instruction (probably needing fragments) The obvious missing one that comes to mind is an Editor. I'd have a lot of trouble seeing how a general purpose parser as you've laid it out could work for an application like Barfly, for example. I have designed and partly wrote an Objective C based ABC parser (currently shelved indefinitely), and tried to make it general purpose enough to work for an editor app. Reworking my design for that was both a frustrating and educational experience, and gave me a great appreciation for what Phil Taylor has accomplished with Barfly. (Originally, I started writing my parser because I wanted four things: a notation based ABC editor, something which would generate PDF output directly instead of Postscript, something which would generate a new style of Hammered Dulcimer tablature I had worked out, and finally, something which would serve as a reference design. My Mac can now read Postscript files directly into Preview and convert them to PDF trivially, so I now use ABCM2PS for printing. My HD skills have passed beyond the point where I need tab anymore, but it would still be a good teaching tool. And what little ABC editing I do nowadays I've gotten used to doing with Barfly. So my motivations have waned considerably. But maybe I'll get back to it at some point because I still think it's a good reference work, and I have ideas for a player program... ;) --Steve Bennett To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
Bernard wrote (portions snipped): I disagree entirely on the maximise portability. The maximum is ascii. You can even read it without a computer. ... Sorry, but it seems archaic to me (in a situation such as we are talking about) not to write the file in ascii. First off, we're talking about a general purpose parser here, not a file conversion program. In all likelihood, most, if not all, of the programs it will be used with will probably link directly to it, and there will be no intermediate file or storage involved - just a data buffer of some sort returned from a function call. The output isn't intended for an end user - it's intended for a program. The whole idea and purpose of a parser is to put the data in a machine readable format that the rest of the program can actually make use of in a non-sequential fashion. Even with ABC, most of the possible applications out there needs to maintain some information for non-sequential access. A notation program needs to access the key, clef, and time signature as it starts each staff. A player program needs to access the tempo, and will need to loop through repeated notes. Format conversion programs may need to access everything in a completely different order. An editor program might need to randomly access any individual range of notes. And so on. Not to mention that all of them need to access numerical values as actual numbers, not strings. Converting ABC into yet another ASCII based format means that you need to create yet another parser for *that* text format to get it into something your program can actually *use* in a non-sequential format. (And what would the output of *that* parser be, eh?) You could possibly write that secondary parser to simulate random access - a call to get the key of Tune 15 as of the 23rd measure, for example. But that would be prohibitively expensive in terms of processing time as you reparse the ASCII stuff over and over again. (Yes, even with today's faster processors...) Not to mention the complexity of writing functions to return every conceivable bit of data. Honestly, it makes absolutely no sense to me whatsoever to be making an intermediate ASCII text format the output of a general purpose parser. It defeats the entire purpose of there *being* a parser in the first place. Mind you, I agree that C structures are somewhat archaic, but they do the job. Ideally the output should be a set of objects, but then you'd lose cross-language portability. --Steve Bennett To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: Re: [abcusers] ABC parser output data structure.
easily, as is of course the parsing since this is very easy to write. (for information it's www.visiv.co.uk and the author provides documentation for the ascii file structure on the web site). I looked at the web site and checked out the format. It is not exactly the information we have to convey, it is much more lower level. For instance, it looked like you can place note stem flags where ever you want. That's not something in the scope of the parser. You misunderstand me. That format is not useful for our purposes, but is an example of a structured ascii format which could well be a model but with completely different fields. Bernard Hill Braeburn Software To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: Re: [abcusers] ABC parser output data structure.
This is not a big deal. Actually there are two very different issues with the output format. 1) What data belongs in it, and what is the relationship to other data? For instance, a rhythm needs to be associated with a note, because if there are three notes playing at the same time, they could all have different rhythms. However, a bowing mark needs to be associated with the entire vertical set of notes. This is a pretty big issue, and is crucial to get right. This has nothing to do with the physical representation of the data. How about this as a first approximation: A score consists of': Name Author Tempo A number of systems A system consists of A number of staves A stave consists of Instrument name MIDI patch number Number of lines in stave (normally 5) Size (normal or cue sized) An array of musical objects A musical object consists of Type (Clef, time sig, key sig, chord, text, barline) A clef consists of type (F,G,C, neutral) position (with respect to centre line) A time sig consists of type (normal, C, C/) number on top number on bottom A key sig consists of an array of accidentals An accidental consists of a type (bb,b,n,#,x) and pitch Text consists of Function (chord name, lyric syllable, general text) Text string Barline consists of type (single, double, final, repeat left, right, both) Size (full, half, dotted) Other (such as fermata over) Chord consists of Staccato type (none, staccato, staccatissimo) Accent (none, , -, fermata, ...) Start to play time (in crotchets from an agreed point) array of notes Beam ID Number of beams (including any stubs) to the left Number of beams to the right Note consists of Rest (true or false) Duration (4,2,1,0.5,0.25... in crotchets) Duration modifier (eg 2/3 for triplets) Number of dots (0,1,2,3) Pitch (octave and name) Accidental (none, bb,b,n,#,x) Stem direction (up, down, none) (In most abc works a chord consists of a single note) How's that for a start? I can see holes in it such as whether chords should have stem direction or notes should. Now converting abc to and from a format like that would be a useful and non-trivial exercise. The midi player can use the parts of relevance, the notation writer can use almost all of the data. 2) The physical representation of the data. There are a number of issues, including extensibility and ease of use in other languages, but in general, I see this as easier to deal with. There is no reason why we couldn't support multiple output formats. In fact, I could implement the population of the data as a templated class, so any C++ programmer could extend it. Perhaps there is a way to take that idea further and allow programmers of non-OO languages to create their own formats. Multiple formats is interesting. Presumably you'd stipulate things like this either with command switches or an initialisation file which the program reads on starup. However for me there is no real contest. Producing ascii allows the programmer to check for errors in the program with greater ease, and allows the format above to grow as development moves from single part music up to full information. And if anyone says that MXML is better than my list then that's fine by me. I have no quarrel at the present time with MXML. I simply don't know enough about it yet. Bernard Hill Braeburn Software To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
On Thu, Jul 29, 2004 at 11:59:44AM -0400, Steven Bennett wrote: Bernard wrote (portions snipped): I disagree entirely on the maximise portability. The maximum is ascii. You can even read it without a computer. ... Sorry, but it seems archaic to me (in a situation such as we are talking about) not to write the file in ascii. First off, we're talking about a general purpose parser here, not a file conversion program. In all likelihood, most, if not all, of the programs it will be used with will probably link directly to it, and there will be no intermediate file or storage involved - just a data buffer of some sort returned from a function call. The output isn't intended for an end user - it's intended for a program. I suppose callbacks might be an alternative ? Honestly, it makes absolutely no sense to me whatsoever to be making an intermediate ASCII text format the output of a general purpose parser. It defeats the entire purpose of there *being* a parser in the first place. Of course, one of the things you could use a parser for, would be in an application that would generate, say, MusicXML, or other ascii, output ... -- Richard Robinson The whole plan hinged upon the natural curiosity of potatoes - S. Lem To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
In message [EMAIL PROTECTED], Wil Macaulay [EMAIL PROTECTED] writes with in your target language. I can say quite honestly that C datastructures in memory are of no use to me as a Java developer You can access them. Write a JNI wrapper, then you can read them and store them any way you desire. JNI isn't hard, all the tools are there to help you. Stephen -- Stephen Kellett Object Media Limitedhttp://www.objmedia.demon.co.uk RSI Information:http://www.objmedia.demon.co.uk/rsi.html To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
More to the point if java is being chosen as a simple means for cross platform deployment then jni suddenly makes this much more complicated. You suddenly go from one distribution to n of them :( Guy Wil Macaulay wrote: Yes, I know I _can_ access them. They are no advantage to me. My Java objects are lightweight, and the same jar file can run on any platform without requiring separate compilation of JNI wrappers. I'm not trying to diminish the theoretical value of cross-language support, I'm simply suggesting that it would be low on my list of requirements (spoken as a product manager...) wil Stephen Kellett wrote: In message [EMAIL PROTECTED], Wil Macaulay [EMAIL PROTECTED] writes with in your target language. I can say quite honestly that C datastructures in memory are of no use to me as a Java developer You can access them. Write a JNI wrapper, then you can read them and store them any way you desire. JNI isn't hard, all the tools are there to help you. Stephen To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
not to overstate java's cross platform attributes - but at least I can put all the platform specific stickiness in one module and compile it all on the Mac... wil Guy Gascoigne - Piggford wrote: More to the point if java is being chosen as a simple means for cross platform deployment then jni suddenly makes this much more complicated. You suddenly go from one distribution to n of them :( Guy Wil Macaulay wrote: Yes, I know I _can_ access them. They are no advantage to me. My Java objects are lightweight, and the same jar file can run on any platform without requiring separate compilation of JNI wrappers. I'm not trying to diminish the theoretical value of cross-language support, I'm simply suggesting that it would be low on my list of requirements (spoken as a product manager...) wil Stephen Kellett wrote: In message [EMAIL PROTECTED], Wil Macaulay [EMAIL PROTECTED] writes with in your target language. I can say quite honestly that C datastructures in memory are of no use to me as a Java developer You can access them. Write a JNI wrapper, then you can read them and store them any way you desire. JNI isn't hard, all the tools are there to help you. Stephen To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
To add my two cents to this discussion: While I have my doubts that any single parser design could fit the needs of more than maybe half (being generous) of the possible applications out there, it's still a worthwhile project. I would avoid producing output in any format which needs further text parsing, such as a text or XML format. Which leaves a C structure-based design a good Least Common Denominator -- most languages can access C structures in some fashion. But the problem with C structures is that they aren't easily extendible, which makes it difficult to add features and change things. Another huge problem with C structures is that if they contain pointers, those pointers become invalid when the structure is moved through memory. (Such as might happen on some platforms inter-process communications, or when moved to another language.) However, with proper planning, you can make the returned format highly extensible and highly portable by taking a couple of steps early on: 1) Your parsing routine should return the data as a single linear buffer. While parsing, you can keep everything in separately allocated structures, but at the very end, when all the data is in structure form, you should calculate the size of the final output buffer, allocate space for it, and copy all the structure data into it. 2) There can be NO pointers in the buffer. Instead, you can specify offset values from the start of the buffer, which will serve the same purpose. Those values can be declared in structures as a pointer and used as such right up to the point where the structure is copied into the output buffer. Use this in combination with #1, and you become pretty much language agnostic. You can even turn the parser into a UNIX filter tool, or save the buffer as a file. 3) Place, at the beginning of each structure definition, a trio of variables which identify the type of structure, length of the structure, and a general purpose linked list pointer/offset value. This gives you extensibility -- the caller can simply ignore any structure it encounters whose type it doesn't know, and you can add fields to a structure later which callers can check for if the length is long enough. (In the past, I used to put a version number in the structure instead of a length, but I found that only rarely would I revise a structure -- I'd more often add a new structure type -- and almost always the revision would be to make it longer, so having the length was more convenient, especially when it came down to calculating the buffer size.) 4) Rather than putting a lot of data into a single large structure, break it up into many smaller chunks of data, each in it's own structure, in a linked list. This, in combination with #3, gives you a whole lot of flexibility and extensibility. (For example, all your tune background info -- history, author, copyright, discography, origin, etc... -- could be in a single linked list of structures pointed to by an info pointer/offset field in the Tune structure. And down the road, if someone, say, added an ABC field for Similar Tunes, you could just create another structure type and stick a copy of that structure in the info linked list.) 5) Avoid arrays of structures, or anything which assumes that structure Y follows structure X in the buffer. Every bit of data should be accessible by following a pointer/offset chain down from the first structure. Use the general purpose linked list pointer to group things together, and pointer/offsets inside the structures to point to other linked lists. You can use arrays of pointers/offsets as well, but the array should be fixed length. I've done this kind of thing on a number of projects (not ABC related), and it's always paid off nicely. I'm not saying you have to do it this way, but if you do, you will maximize your portability, and gain a lot of flexibility. --Steve Bennett To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: Re: [abcusers] ABC parser output data structure.
What language is Music Publisher 5 written in? What OS? Delphi. Windows 32. I ask because I'd like to make the output format convenient for everyone who wants to read ABC. Besides, I think we are talking about slightly different things. Here's my vision for the parser: Depending on the OS and language, there will probably be a different structure for how this fits together. 1) If the language is C or C++, you can just compile and link the parser into your program. You would use it by calling the parsing function with an ASCII string containing ABC, and it would return something that would be easy to iterate through. 2) If your linker can link in a C module with your program, you would link and use it as above. Some kind soul might write some access functions to make it more convenient to use in your language. 3) If you can't link the module in, but are running on Windows, there would be a stand alone parser program that you can access with COM. you would use it the same as above, once you connect to it. 4) If none of the above applies, then perhaps some kind soul can write the glue to make it work. Option 2 is fine for Delphi. C DLLs can be accessed easily. (However it rather breaks the DLL-less style which I use!) Now, the parser function itself would not load or save files. It is purely a memory based function. It would be up to the client to do something with the output. Here are some possible clients: 1) Conversion Program: read an ABC file from disk, run it though the parser, iterate through the parser's output to write a different format to a new file. 2) Notational Program: read an ABC file from disk, run it though the parser, iterate though the parser's output to create sheet music. 3) Player Program: read an ABC file from disk, run it though the parser, iterate though the parser's output to create a MIDI file. 4) Analyzer Program: read an ABC file from disk, run it though the parser, iterate though the parser's output to collect whatever statistics are of interest. A simple example is just getting the title for a table of contents. A more complex example is to look for simularities between tunes. A transposing program is another example. It is also possible to create more than one type of output if a single structure is not adaquate. For instance, MusicXML files are either score-wise or part-wise. That is probably a good thing to emulate. We can also create different output for different languages if that turns out to be easier. For C++, the output most naturally would be some kind of STL container, for Pascal and C a structure might be best, perhaps an INI-file-like format would be easier for you. All that is possible but, I repeat, not as flexible as ascii for both debugging and expansion. FWIW I already use a module along the lines of the abc parser except it is an engine for OCR. It takes a bitmap file and a standard Windows (short) .ini file containing the options and provides a structured ascii file. It is called as a background program and sends messages back to a windows handle provided in the .ini file. Debugging and expansion are handled very easily, as is of course the parsing since this is very easy to write. (for information it's www.visiv.co.uk and the author provides documentation for the ascii file structure on the web site). Bernard Hill Braeburn Softwar To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: Re: [abcusers] ABC parser output data structure.
[snip] I've done this kind of thing on a number of projects (not ABC related), and it's always paid off nicely. I'm not saying you have to do it this way, but if you do, you will maximize your portability, and gain a lot of flexibility. I disagree entirely on the maximise portability. The maximum is ascii. You can even read it without a computer. Flexibility is maximum in ascii. A new keyword is added and the old software doesn't understand the keyword and will ignore it. The problem of upgrading software is old software which won't read the new software's output at all. You say it's hard to write. What's simpler than if keyword = this then readvalue otherwise if keyword = that then readvalue otherwise if keyword = the_other then readvalue... ... otherwise skipThisKeyword Sorry, but it seems archaic to me (in a situation such as we are talking about) not to write the file in ascii. Maybe in database applications where storage really is multiplied by millions, but who cares if a music tune takes 5K or 50K on a PC? Bernard Hill Braeburn Software To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: Re: [abcusers] ABC parser output data structure.
I was thinking that the output would be a C struct, which should be readable by any language. I'd like some input about limitations in your favorite language. Linkage is an issue, too. I'm most familar with C++ and Windows, so I need to avoid my natural instinct here. I'd be against that. The output should be a structured ascii file with keywords and in this case it is simplicity itself to add features and create readers which will still read future versions of the program output. I've given examples in a previous contribution. The trouble with ascii output is that it's slow to generate, takes up a lot of disk space or memory, Well my current version of Music Publisher can save files in its own (proprietory) binary file; or as ascii. I can' tell the difference in speed either reading or writing and I typically run an older system working at 800MHz. As to size, it certainly is 4-10 times as large but the advantages are far outweighed by the compatibility issues which occur as the data fields inevitably expand. and slow to reas it has to be parsed again. See above. (Also if you're going to use an ascii file you might as well use MusicXML, which can express everything in abc (apart from the Hp key sig) I'm not against MHML, just that it contains much more than we need. Using a C struct would be fine by me, since I use Pascal, and C structs are directly equivalent to Pascal records. Linking to C code is no problem either, as in the Metrowerks development system I use the C and Pascal compilers are just front ends to a common code generator, and as long as you keep them in different files you can mix and match the two languages. Linking to C++ is a bit more difficult (something to do with name mangling or somesuch) and you have to write some glue code in C, but that's not too difficult. Other problems you may encounter are to do with endianness and the availability of floating point formats on different processors. As long as you don't make any assumptions about the order of bytes within a number and avoid using real numbers (or at least stick to the common float types) it shouldn't cause too much trouble. The format doesn't worry me. But the expandability does. A simpler structured ascii format is far better than abc or any fixed binary format. But what they heck, I don't care much anyway: Music Publisher 5 supports a subset of natural abc anyway and it's not really aimed at the abc user anyway so if the format is too much trouble I won't use it. It just seems crazy to me to use any binary format in these days of large and fast storage; fast processors and particularly in a multi-platform format. I was simply offering advice. Bernard Hill Braeburn Software To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
But what they heck, I don't care much anyway: Music Publisher 5 supports a subset of natural abc anyway and it's not really aimed at the abc user anyway so if the format is too much trouble I won't use it. It just seems crazy to me to use any binary format in these days of large and fast storage; fast processors and particularly in a multi-platform format. What language is Music Publisher 5 written in? What OS? I ask because I'd like to make the output format convenient for everyone who wants to read ABC. Besides, I think we are talking about slightly different things. Here's my vision for the parser: Depending on the OS and language, there will probably be a different structure for how this fits together. 1) If the language is C or C++, you can just compile and link the parser into your program. You would use it by calling the parsing function with an ASCII string containing ABC, and it would return something that would be easy to iterate through. 2) If your linker can link in a C module with your program, you would link and use it as above. Some kind soul might write some access functions to make it more convenient to use in your language. 3) If you can't link the module in, but are running on Windows, there would be a stand alone parser program that you can access with COM. you would use it the same as above, once you connect to it. 4) If none of the above applies, then perhaps some kind soul can write the glue to make it work. Now, the parser function itself would not load or save files. It is purely a memory based function. It would be up to the client to do something with the output. Here are some possible clients: 1) Conversion Program: read an ABC file from disk, run it though the parser, iterate through the parser's output to write a different format to a new file. 2) Notational Program: read an ABC file from disk, run it though the parser, iterate though the parser's output to create sheet music. 3) Player Program: read an ABC file from disk, run it though the parser, iterate though the parser's output to create a MIDI file. 4) Analyzer Program: read an ABC file from disk, run it though the parser, iterate though the parser's output to collect whatever statistics are of interest. A simple example is just getting the title for a table of contents. A more complex example is to look for simularities between tunes. A transposing program is another example. It is also possible to create more than one type of output if a single structure is not adaquate. For instance, MusicXML files are either score-wise or part-wise. That is probably a good thing to emulate. We can also create different output for different languages if that turns out to be easier. For C++, the output most naturally would be some kind of STL container, for Pascal and C a structure might be best, perhaps an INI-file-like format would be easier for you. Also, Christian, I went to SourceForge, but the only way I could see of subscribing was to subscribe to SourceForge itself. That appears to cost some money. Is there a way to just be part of the AlphabetSoup group for free? Paul Rosen --- Life is a musical, every once in a while the plot stops and you start singing and dancing --- http://home.earthlink.net/~catharsis.music/ http://home.earthlink.net/~theplums/ To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
RE: [abcusers] ABC parser output data structure.
I'm definitely confused. I'll just keep reading and maybe I'll catch on to the goal. ABC is a text format and it seems like most things associated with ABC might fit better if text, but then again I don't exactly know what is being proposed. If it is to read ABC files and convert them to other formats for a variety of OS and software, seems like Java (stand-alone) would be something to consider. Isn't that software totally portable to other platforms? I'm back to the reading mode ... To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
I've finally gotten a few free moments of time, and started thinking about Hey, if you'd like to volunteer to head up the project, it's still looking for a project head to get it started. Thanks, but, did you notice the word few ;-) I'm not sure how much time I can devote to it. I'll see what I can do. Just go to sourceforge.org and search for the Alphabet Soup project and register to be a project member, and I'll promote you to head it up. I'll register. I'm a little reluctant to take the conversation over there, though, until everyone who's interested signs up. Are we boring anyone here? Paul Rosen --- Life is a musical, every once in a while the plot stops and you start singing and dancing --- http://home.earthlink.net/~catharsis.music/ http://home.earthlink.net/~theplums/ To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
Re: [abcusers] ABC parser output data structure.
This is a big job, and not for the faint of heart - I wish you luck! Skink is written in Java, and uses a parser written in Javacc. (so a c++ set of datastructures would have done me little good...) I actually started off with a C++ version using Flex++ and Bison, but found Java and never looked back... Some coments below, for what it's worth. I was thinking that the output would be a C struct, which should be readable by any language. I'd like some input about limitations in your favorite language. Linkage is an issue, too. I'm most familar with C++ and Windows, so I need to avoid my natural instinct here. 5. The purpose of this structure is to completely separate the parser, the formatter, and the player. Each one of these three functions can be developed independently, and different developers can have their pieces snap in. This is more-or-less what I did with Skink, though the intermediate representation is a set of objects, not a file. I have defined an interface for my renderer (I have two, for different versions of Java) and for my player, (although I only have one at the moment.) The intermediate could be memory or a file. I was originally thinking that if we define the structure without pointers, it would be more likely readable by other languages, and we could blast the entire thing out to disk if we wanted to, but we wouldn't have to. 7. This structure is for a single tune. To do multiple tunes, you can simply have an array of these. Not quite - there is additional content that is not inside any tune, and some of it might want to be displayed. In particular, creating tables-of-contents, indexes, and pagenumbers needs some back-and-forth between the tune and the display. I seem to remember that there is some data that is global over all tunes until it is changed. (That was causing an issue with concatinating files). That data needs to be passed along with the tune, for sure. As far as TOC, etc, I think that is not quite the scope of what I'm doing. It would be simple to feed the parser one tune at a time and get the info you need, though. 8. Everything possible should be interpreted as much as possible. In other words, a chord of Gm would be stored as a G minor chord, not the text Gm. This allows the player to not need to do any parsing. I'm not sure I agree with this - the chord notation has been much abused. That's one reason for the paragraph below. I was thinking that, if the parser sees a quote, it tries to recognize the chord. If it can recognize it, it represents it internally. If not, it puts a place holder in there meaning unknown chord. In either case, it saves the exact text also in case the client wants it. A MIDI player probably wouldn't want it. A formatter might have a choice about whether to display all the chords it recognizes in a similar format for consistancy between tunes, or display them as originally written. A formatter could also optionally highlight in red the chords it doesn't recognize. 9. however, the original wording should be available if needed. In other words, GM and G are both a G major chord. Some formatting programs may wish to use one or the other exclusively, or may choose to parrot whatever was specified. 11. The structure should have a simple, fixed format so that it can be saved to disk easily, or passed between programs written in different languages. The saved version would be something of a compiled version, that is, our equivalent of an OBJ file. This means that there can't be any pointers in the structure. I think this restriction essentially ends up being true only for an intermediate form. For example, if you have a | abcd | dead | cabb :| age (a purposely wrong, but often found construct) you will need to figure out how to point back to the first barline for the repeat. In other words, the player will have to construct an intermediate form that does have pointers. I didn't mean pointer in that sense. I meant in the sense of an absolute memory location. In the sense you mean, I think the pointer would the ordinal number of the element. 12. A C++ class will encapsulate the structure, so that users of C++ can easily gain access to the data in a natural way. Java? Perl? Ruby? python? ... I'd love to have someone encapsulate the structure for all common languages. And suggest a good way to pass the data. If we were only worried about Windows I'd suggest COM. For C++, there'll be source code that you can link into your app directly that has a public function that does the parsing. I need some guidance about the best way to link to other languages. It may be that there needs to be a little glue for each one we support, I don't know. You will need to define an 'extent' of notes that have certain characteristics. Some items will cause changes in the player and/or the display context. For example K:D |: abcd || [K:Bb] def abc :| Hmmm... it looks like the key signature,
Re: [abcusers] ABC parser output data structure.
Yes, absolutely, the discussion should stay here. The real purpose of sourceforge is simply to enable collaboration with CVS, and project tracking and such. Chuck Boody wrote: You are not boring me. I haven't much to contribute right now, but am following the discussion and may have comments later. I would hope the discussion stays here. Chuck Boody On Monday, July 26, 2004, at 08:52 PM, Paul Rosen wrote: I've finally gotten a few free moments of time, and started thinking about Hey, if you'd like to volunteer to head up the project, it's still looking for a project head to get it started. Thanks, but, did you notice the word few ;-) I'm not sure how much time I can devote to it. I'll see what I can do. Just go to sourceforge.org and search for the Alphabet Soup project and register to be a project member, and I'll promote you to head it up. I'll register. I'm a little reluctant to take the conversation over there, though, until everyone who's interested signs up. Are we boring anyone here? Paul Rosen --- Life is a musical, every once in a while the plot stops and you start singing and dancing --- http://home.earthlink.net/~catharsis.music/ http://home.earthlink.net/~theplums/ To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html -- || Christian Marcus Cepel | And the wrens have returned [EMAIL PROTECTED] icq:12384980 | are nesting; In the hollow of 371 Crown Point, Columbia, MO| that oak where his heart once 65203-2202 573.999.2370 | had been; And he lifts up his Computer Support Specialist, Sr. | arms in a blessing; For being University of Missouri-Columbia | born again. --Rich Mullins To subscribe/unsubscribe, point your browser to: http://www.tullochgorm.com/lists.html
