OK, I'm still pretty new here so I might be missing something, or perhaps I'm not explaining things as clearly as I thought (sincerely - while I sometimes degrade to sarcasm, it's rarely subtle when I do ; ) I believe in what is happening here. Having worked for a vendor of commercial C++ libraries from 1995 until a few months ago, it was amazing to discover how much the Boost community has accomplished. The last thing I want to do is communicate disrespect to those (like Joel in this case) whom I've rapidly come to respect deeply, having clearly established themselves as experts and made substantial contributions to Boost. However, for reasons which will hopefully become apparent over time, I am very interested in Boost's success. That enthusiasm may sometimes incite me to boldness in arenas where I have not yet earned the right to be so. With that being said...Why at startup? Why not actually load (include) dynamic syntax definitions explicitly, something like "[sourcemode foo]" pulling in "*/foo/.highlight_def_extension*"? This could look first in the local directory, then search relative to a some standard lookup path (perhaps $BOOST/tools/QuickBook/highlighters ?). It might allow for more intelligent feedback on errors (i.e., dependency on a definition file that isn't found). I think it could even be leveraged to allow BBv2 to treat referenced definitions as dependencies, triggering a QuickBook "rebuild" if the definition is updated (which might be much harder to do otherwise, short of any change to any definition file triggering every .qbk to get reprocessed whether or not it used that definition).
Too fancy, I think. But you do have a point. Maybe we can start simple and build our way there.
Why? Why too fancy? Or from another angle, might this be a sensible approach in general for Boost, but not for QuickBook?
As I understand it, Boost's charter is to provide a community for the development and vetting of peer-reviewed libraries and establish "existing practice" relevant to potential enhancements of the C++ Standard. The concept of "starting simple and build our way there" is extremely sensible in this context. Those involved in the early stages of a new project should expect it to change and grow as they experiment, push the edge, and seek practical elegance. Concerns like consistency and backwards compatibility, whether with code written for earlier versions or with aging compilers unable to support modern C++, can become secondary and in some cases dismissed altogether if they obscure the primary focus. That focus is on the future, which sometimes requires letting go of the past.
IMHO Boost has a third goal. Perhaps it came about implicitly, as various contributers added bits and pieces to facilitate collaboration on various projects, and was never really established as a significant and fundamental goal on it's own. Maybe it was planned all along. Since I missed most of the history, how it came to be is still a bit opaque, but here's what I see now : that third goal is the creation of a robust infrastructure to support the use and development of the Boost library projects. The current realization of this goal is the "Boost Tools", including the Boost build system, BoostBook, the regression testing process, and so on.
The quality of the infrastructure of supporting tools is, in my experience (and I ask you to trust for now the moment that I have sufficient depth of experience in this area) one of the most significant and least regarded aspects of C++ development. When it works well, it fades into the background and everybody just takes it for granted. When it obviously breaks, the average project team may send some unlucky junior developer into the byzantine bowels of the build system to hack away until things seemingly work again. When everything seems to work but doesn't, that's where the real fun begins. Countless hours can circle the drain, swirling away along with the assumption that the foundation was solid. Consider how much frustration has been felt over the years by developers combing through their source, trying to fix some problem which was nothing more than an accidental dependency on both debug and release versions of the MSVC runtime, or some change that hadn't compiled yet because the "automatic" header dependency checking tripped up, or a "clean" target which forgot to blow away the template instantiation cache files. I might even go so far as to say that, given the challenges involved in compiling and linking complex C++ applications, the lack of a solid build infrastructure has been a significant part of the perception that C++ is "too hard", right up there with raw pointers running wild and the absurd expectation of actually managing memory allocation without a garbage collector ; ) Coming back on topic, I believe the Boost community's efforts in addressing the need for such an infrastructure is one of the fundamental causes for the past, present, and future success of Boost.
My apologies for the appearance of overkill here, as I'm rambling on about (hopefully obviously) more than a reply to Joel. I've been thinking about this for a while, and now seemed like a good time to try and gather my thoughts in and write them down. Joel's comment filled in one of those nagging blank spots where I knew I knew something that I couldn't quite identify - the necessary tension in intention between Boost "Tools" and Boost "Libraries". Where the library projects are dynamic, likely to change frequently until reaching a satisfying equilibrium of "established practice", a similar approach to the "tools" can wreak havoc. The library projects experiment with new ideas, explore the unknown, and bring deeper insight to the hidden things of C++. The tools need to provide a stable, reliable - perhaps sometimes even boring - foundation: the base from which the library projects can safely and sanely launch their expeditions.
Some of the reasons for treating the development of libraries and tools differently are simple, like the impact of change. Changes to a Boost library impact the users of that library. Changes to the Boost tools can potentially impact the users, and authors, of _every_ Boost library. This creates a second tension for tools. On one hand, refinement of the tools should be one of our highest priorities, because in theory the benefits from enhancing a tool can propagate rapidly through the entire Boost community. On the other hand, the need to provide consistency over time (which can actually hinder achieving Boost's intentions for a library project, especially in the early stages), becomes paramount for tools, and any changes must be made with extreme care. In one sense, the most important "source code" for a tool isn't the code in the tool itself, but all the code out in the Boost libraries depending on that tool, sometimes extending to the users of Boost as well. For example, justifying a change to Boost Build that even trivially breaks most existing Jamfiles should require incredibly strong arguments. Justification for a change to a Boost library project may be simply the discovery of a better way to do things, even when it must break existing code.
I'm sure that little (if any) of this is new or surprising to experienced C++ practitioners. In fact, there are already ways in which the history of the Boost tools implies awareness of these concepts, such as developing and testing BBv2 while continuing to use and maintain the current Boost Build system. But since I have rather, er, strong opinions on this topic and haven't found these concepts explicitly articulated in relation to Boost, well... here it is.
So now, with a bit more context ;) we return to the beginning:
There are a few different parts here, which I didn't really clarify. One is the concept of adding a new QuickBook command to explicitly load a syntax definition. The second is all the interesting "extras" which could eventually build on the first. Taken as a whole, perhaps that is a bit fancy, but perhaps the basic idea of explicitly loading a syntax definition, without doing all those extras, qualifies as starting simple?Why at startup? Why not actually load (include) dynamic syntax definitions explicitly, something like "[sourcemode foo]" pulling in "*/foo/.highlight_def_extension*"? This could look first in the local directory, then search relative to a some standard lookup path (perhaps $BOOST/tools/QuickBook/highlighters ?). It might allow for more intelligent feedback on errors (i.e., dependency on a definition file that isn't found). I think it could even be leveraged to allow BBv2 to treat referenced definitions as dependencies, triggering a QuickBook "rebuild" if the definition is updated (which might be much harder to do otherwise, short of any change to any definition file triggering every .qbk to get reprocessed whether or not it used that definition).
Too fancy, I think. But you do have a point. Maybe we can start simple and build our way there.
The third part, which is perhaps the most important, I realize I simply assumed without saying anything. What is the intended visibility of these syntax definitions? If it's only those working on QuickBook itself, as an easier way to provide built-in syntax highlight parsing, then this is an interesting implementation detail but ultimately of little significance beyond QuickBook developers. However, if the idea is (and here's where I might have jumped to an assumption) that the ability to create custom grammars would be exposed and provided to QuickBook users writing documentation using QuickBook (but not necessarily involved in QuickBook development), then this directly impacts the "interface" QuickBook provides to it's users.
There are several ways in which I can envision productive uses for "project local" syntax highlighting definitions. For example, it would be pretty neat to have syntax highlighting for QuickBook's own syntax in the QuickBook documentation, but there wouldn't be much use for that format outside of quickbook.qbk. For this to work, there needs to be a very clear, reliable mechanism for a project to express such customizations in a way which future improvements to QuickBook can easily avoid breaking. The idea of adding an explicit command to QuickBook for loading custom syntax highlighting definitions was based on this idea of projects being able to create their own, local definitions, not for built-in formatting definitions which may happen to be implemented by a dynamic definition file. Having a tool like QuickBook implicitly find such local, external dependencies, IMHO, seriously increases the risk of future changes introducing problems with existing documentation using QuickBook. But again, this only applies if the intention is to expose such abilities available to QuickBook users so they can create custom, project-specific syntax highlight definitions.
If you're still reading after all this, it's probably obvious that I consider getting things "right" when creating or changing interfaces for anything in the "Boost Tools" family to be a very, very serious matter... and since I think supporting project-specific customization for the syntax highlighting in QuickBook is a great idea, and that means defining such an interface, this grew into a bit more than I had planned. Thanks for taking it... seriously ; )
- james
-- __________________________________________________________ James Fowler, Open Sea Consulting http://www.OpenSeaConsulting.com, Marietta, Georgia, USA Do C++ Right. http://www.OpenCpp.org, opening soon!
------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Boost-docs mailing list [email protected] Unsubscribe and other administrative requests: https://lists.sourceforge.net/lists/listinfo/boost-docs
