On 12/11/2011 12:34 AM, Gabriel Scherer wrote: > A summary to this lengthy mail: > (1) Why type-enriched Camlp4 is an unreasonable idea > (2) We should extract the typedtree; why it's hard > (3) A fictional narrative of the camlp4/camlp5 history > (4) Why you don't want to become Camlp4 maintainer > (5) How we could try not to use Camlp4 in the future > (6) Syntax extension survival advices
Sorry, it's too long to read entirely for a sunday morning, so I reply just after reading points (1) and (2): (1) many extensions in Camlp4 are used to manipulate types (most of them based on type-conv). So, having the compiler knowledge of types at a point where you want to expanse a macro is actually very interesting for them. I wrote a patch to 3.12.0 a long time ago called "ocaml-template" that would exactly do that: it would call a plugin function at some points in the program, giving the function both the string to parse and the typing environment, so that it is possible (to some extend, since typing might only be partial at the point where it is called) to know the type of an identifier, and the complete description of the type. Here is the link to the patch: http://www.ocamlpro.com/files/ocaml-template-0.1-for-3.12.0.tar.gz It is not documented much, but there are some simple examples of plugins and uses. (2) exporting the typedtree is not really a problem. The main problem is to add enough information in the typedtree for all the tools that could use it (so that ocamlspotter and refactoring tools can use it). We are currently in the process of polishing our final patch to submit it for integration, now that we have tested it with enough tools. --Fabrice > > # (1) Why type-enriched Camlp4 is an unreasonable idea > > Wojciech, your idea of having type information at the Camlp4 level is > absolutely unreasonable. You are not speaking about a "minor change" > here, but a major rewrite that would affect the compiler internals as > well. It would really be a new (and interesting) project. > > Camlp4 is, and I guess will remain, a syntax-level preprocessing > tool. You have to accept the fact that you can't use type information > at this level (but you can certainly "interact" in some way with the > type system by producing/transforming pieces of code in a way that you > know will have interesting typing effects; for example, you may want > to generate code that is purposedly ill-typed in some cases, to > disallow certain uses of your syntax extension). I'm not even sure > what it would mean to access type information at the camlp4 level, as > you're producing and transforming untyped AST; would you want > partially typed ASTs? How is the typer supposed to work on the part > that you haven't transformed yet, and therefore are not valid OCaml > syntax? I suppose you could have a "preprocessing and transformation" > tool at the typedtree level, but that would be a different tool with > different uses, distinct from the syntactic preprocessing part (though > you may develop "extensions" that act on both fronts). > > I'm not aware of so much Camlp4 situations that would really require > typing information. I would be interested in good examples if you have > some. One problem that I have had with Camlp4 is that you don't have > identifier resolution information (eg. you don't know if the > identifier "(@)" you're seeing is really list concatenation, or has > been redefined/shadowed in the context); this makes uses of Camlp4 for > inlining, for example, quirky and fragile. That's still a simpler > problem than type information. > > > > # (2) We should extract the typedtree; why it's hard > > If you really want to play with type information and scope-resolved > identifiers, AST-manipulating tools is probably not the way to go: you > indeed want full access to the typedtree. Currently this is only > possible by hacking the compiler, and this is what for example Jun > Furuse's Ocamlspotter project does. Those kind of tools could be made > less intrusive if it was possible to pass typedtree-like information > in and out of the compiler. > > I remember reading that some people (OcamlPro, I suppose) have this on > their target list. The problem however is that the current internal > compiler's typedtree representation is not at all adapted for external > communication. If you want a kind of tool that is robust and > future-proof in any sense (you could probably get something working by > just marshalling the current typedtree, but then it could break > awfully after minor language changes, make the compiler choke, etc.; > I certainly wouldn't want to use that), you have to design a clean and > efficient representation for OCaml programs after the type inference > phase. Having a solid proposal on this topic wil be an awful lot of work. > > > > (3) A fictional narrative of the camlp4/camlp5 history > > Jérémie Dimino wrote: >> But there is something I don't understand here. Why is there camlp4 and >> camlp5 ? These two projects do exactly the same thing and are >> incompatible. So i don't see the point of maintaining them both. We >> should at least deprecate one. > > Let me repeat the story as I know it (with possible mistakes, I was > still a caml baby in the <3.10 times) in a hopefully compact form for > those on the list who have no idea about it. DISCLAIMER: this is only > a fictional storytelling, meant to give a reasonable idea (or at least > my vision) of the situation. I may be wrong about the events > chronology, people name, hard facts, and of course english spelling > and grammar. The story is complicated and I don't know the gory > details. > If you know a better story, feel free to add important precisions, > correct the obvious mistakes, etc. I also welcome suggestions to make > it a funny, entertaining read; finally, a few romantic details could > clearly turn it into a blockbuster. > > > The original Camlp4 tool was mostly developped by Daniel de > Rauglaudre. Apparently, personal relations between Daniel and the > OCaml team were not easy, and Camlp4 was gradually becoming more and > more external to the OCaml distribution (in the past, the stream > syntax was available as part of the core language, but it was moved to > Camlp4; the Oreilly book was written before that move) and its > maintainance status incertain. > > In the 3.09/3.10 transition, Nicolas Pouillard started working on > a refactoring of the Camlp4 codebase (which was mostly a silent, > non-moving animal at that time) to make it more easy to evolve and > maintain. The refactoring maybe went "a bit too far", in that it > brought a number of changes to the external interface and, in > particular, broke existing Camlp4 extension, as well as the (quite > good) Camlp4 documentation. Of course they also were advantages, in > that the new design was modular and, for example, the bootstrapping > process was made easier, and Nicolas Pouillard could maintain the tool > as offered by the distribution. The 3.10 transition was however very > painful for people using existing Camlp4 extensions (I'm thinking of > eg. Martin Jambon, which had extensive Camlp4 extensions, and the Coq > team which has user-defined notations using Camlp4 and, huh, I really > don't want to know the details); basically they didn't upgrade to > 3.10 -- instead of porting the extensions, as was originally hoped. > > Personal note: I learned Camlp4 in this period, just after the release > of OCaml 3.10. I'm honestly unaware of how pre-3.10 Camlp4 was (though > I guess it is not too difficult to move from one to the other) so my > interpretations of previous times are all based on reading the > mailing-list. > > Daniel, which apparently did not agree with some of the changes made, > relatively suddenly restarted developpment of "his" branch of Camlp4, > taken from the old sources, before refactoring. This was done as > a separate project, outside the OCaml distribution (apparently Daniel > and the OCaml team prefer not to work together). In a quite creative > move, he named his version "camlp5" so that it could be easily > distinguished from the "upstream camlp4", the incompatible new version > being distributed with OCaml. Camlp5 is therefore the continuation of > the *old* camlp4. Development, however, continued (while still > preserving or mostly preserving compatibility with pre-3.10 > extensions, ensuring grateful thanks from the users) with > non-neglectible changes (eg. addition of a library of non-destructive > streams), and is currently ongoing (as is maintainance of pre-3.10 > camlp4 in the OCaml distribution). > http://cristal.inria.fr/~ddr/camlp5/CHANGES > > Of the projects that relied on Camlp4 before 3.10, some of them ported > they extension to >=3.10 Camlp4, and lived happily ever after, and > some of them jumped on camlp5 as a lower-cost migration opportunity > and, I suppose, also lived happily ever after; or at least, as happily > as you can knowing that your codebase depends on camlp{4,5}. > > I make no technical judgments of which version (Camlp4 or Camlp5) is > "better". I know and (try not to) use Camlp4. I also understand the > position of Camlp5 users for compatibility reasons. I would advise > newcomers to try Camlp4 first as it has a larger user base (being > distributed with the OCaml distribution). Anyway, see at the end of > this mail why you may not want to use camlp* anyway. > > > # (4) Why you don't want to become Camlp4 maintainer > >> I am volunteer for the maintenance of Camlp4. > > Jérémie, I am deeply impressed by your sense of sacrifice. > > Camlp4 is a piece of devil beauty. It does incredibly clever things, > and is incredibly complex inside: Daniel is clearly a remarkable > hacker, but his code is not easy to understand. I know that > maintaining the whole thing is very hard; and that is the reason why > Camlp4 tends to have problems to bump from one version to another, > when non-neglectible syntaxic changes are made to the language. > > The core internals of Camlp4 are quite imperative: parsing is done by > erasing tokens from the input stream (which makes for lovely Stream > debugging), but, more importantly, grammar extensions are destructive > mutations of the grammar. It is sometimes painful when using Camlp4, > and I suppose it must be hell maintaining it. > > > # (5) How we could try not to use Camlp4 in the future > > I think Alain's idea of moving out of Camlp4 is actually a good > thing. After having used Camlp4 extensively, I came out with the > general feeling that allowing general extension of the grammar is not > a good thing, because it is too complex and the solutions are too > fragile. On the contrary, the quotation mechanism of Camlp4 is > a wonderful tool, that acts as a *controlled* extension point for the > grammar, and is therefore much more robust. The other excellent use of > Camlp4 is generating OCaml code out of "annotations" on certain AST > nodes, as the 'type-conv' extension does; this is also a form of > controlled extensibility that could, I think, be taken into account in > the base language or a simple tool, and doesn't require full-fledged > grammar mutation. > > I think we should isolate such restricted uses of syntax extensibility > and allow them through simpler, more robust tools. Alain's idea, and > the reactions from Camlp4 users (Martin Jambon, Nicolas Pouillard), > can be found on his blog: > http://www.lexifi.com/blog/syntax-extensions-without-camlp4 > > I'm personally not sure his solution is quite ready to replace Camlp4 > yet; or at least the state in which it was at this time. He has an > annotation mechanism, but it is somehow not restricted enough to > guarantee a reasonable behaviour; as long as people are tempted to use > "annotations" to define try..finally, we will have fragile extensions > floating around. > > (And I'm not sure it's a good idea to move Camlp4 out of the > distribution as long as we don't have a viable alternative proposal > and some users have started moving to it. Camlp4 will still need to be > supported by someone anyway, and it needs to evolve in lockstep with > OCaml language changes.) > > Yet I do agree that "without Camlp4" is the future. It may take some > time, but what I would personally like to see in a few years is an > OCaml world where we don't need Camlp4 anymore, because we have other, > simpler tools to do the reasonable things, and have learned to live > without the rest. The whole "maintain Camlp4" entreprise will still be > useful, necessary until that time, but it won't stay, I hope, at the > center of attentions. > > > # (6) Syntax extension survival advices > > To the reader considering use of a new syntax extension in his next project: > > - don't > > - if you really must, try to make sure that your code is also > reasonable to use *without* a syntax extension (eg. by producing > a library with a clean interface, making your extension desugar to > uses of it, but also making sure that it can be used by the human > user) > > - if you really must, try to get it in the form of a quotation; the > rest is fragile > > - alternatively, try to branch yourself on existing "flexible" syntax > extensions such as Markus Mottl's 'type-conv' and Jeremy Yallop's > 'patterns': you are relatively safe if you don't write any line of > code modifying OCaml's syntax yourself. > > http://hg.ocaml.info/release/type-conv > http://code.google.com/p/ocaml-patterns/ > > - do not hesitate to send for code review and/or ask for help on the > list > > - don't > > > On Sat, Dec 10, 2011 at 10:40 PM, Wojciech Meyer > <wojciech.me...@googlemail.com> wrote: >> Jérémie Dimino <jere...@dimino.org> writes: >> >>> Le samedi 10 décembre 2011 à 19:10 +0000, Wojciech Meyer a écrit : >>>> I'm aware that these are huge changes to Camlp4, but it would make >>>> meta programming more powerful and push Camlp4 to the next level. >>> >>> Sure. But it seems that the next version of OCaml will have runtime >>> types, see http://www.lexifi.com/blog/runtime-types , so maybe it is not >>> needed to add this to camlp4. >> >> It's interesting and I didn't know about it. However, the problem is >> slightly different, I would like to know the typing of a freshly >> generated piece of code by Camlp4 in the previous phase. Then, have >> pattern matching against these meta types in annotated AST and produce >> another AST, which in turn have most likely another typing and pass to >> the next phase etc. I would say that Camlp4 is fine for the simpler >> syntax extensions and majority of small DSLs but when you start composing >> syntax extensions and macros it quickly becomes a problem. >> >>> >>> Also they are problems that i don't know how to solve with the camlp4 >>> approach. For example consider: >>> >>> let x = 1 >>> type int = A >>> let y = A >>> >>> The typer knows that x has the type (int, 1) and y has the type (int, >>> 42). But what you send to ocaml is a parse tree, and you cannot make >>> this difference in the parse tree. >> >> Yes, you would need a type information in the parse tree as mentioned >> before, so you want to feed up the compiler with AST start unrolling >> macros top-down and then follow up with the inferred types bottom-up. >> >> Cheers; >> Wojciech >> >> >> -- >> Caml-list mailing list. Subscription management and archives: >> https://sympa-roc.inria.fr/wws/info/caml-list >> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners >> Bug reports: http://caml.inria.fr/bin/caml-bugs >> > > -- Caml-list mailing list. Subscription management and archives: https://sympa-roc.inria.fr/wws/info/caml-list Beginner's list: http://groups.yahoo.com/group/ocaml_beginners Bug reports: http://caml.inria.fr/bin/caml-bugs
<<attachment: fabrice_le_fessant.vcf>>
