[Response intentionally CC'd to the original poster, on the off chance he's not already subscribed to the mailing list]
On Mon, Mar 19, 2018 at 7:42 PM, Offray Vladimir Luna Cárdenas < [email protected]> wrote: This is just kind of a next semester project, but I would like to start > exploring the options. > > I'm using Fossil mainly to store documentation (using the superb > embedded documentation feature) for my Grafoscopio[1] project. I think > that SQLite and Fossil are good examples of what I call pocket > infrastructures (simple, self-contained and working well on-line and > off-line) that can help to democratize fields like data visualization > and storytelling[2]. But, despite its advantages, Fossil is starting to > feel limited, particularly with the Yaml metadata blocks that the Pandoc > Markdown variant uses[3]. So I would like to explore the possibility to > use Lua (which is already integrated in Pandoc and easily integrable in > other programs) to read such metadata y show it in a proper way. I also > think that the support for Lua should open a lot of scriptability > options and ecosystem, far beyond TH1 and Tcl, by using a more wide > spread scripting language with a bigger ecosystem. > > [1] http://mutabit.com/grafoscopio/index.en.html > [2] http://mutabit.com/offray/blog/en/entry/panama-papers-1 > [3] https://pandoc.org/MANUAL.html#extension-yaml_metadata_block > > What should be done to explore such Lua extensions to Fossil? Where can > I start? Dear Mr. Cardenas (my apologies for misspelling your name, but I'm just Yet Another Damned Monolingual Yankee), I saw your question above when you first sent it to the fossil-users list, and meant to respond to it, at least to give "one bystander's opinion". My apologies for taking so long to respond. I *do* personally think it's an interesting idea, but for reasons I'll give below, it might not be as interesting to the core Fossil developers, at least at this point. Before I continue, I need to emphasize to the utmost extent that I am *not* a core Fossil developer or committer to this project; I am *not* an active external developer for it submitting patches or modules to be integrated with it; nor am I even just someone actively using it and perhaps submitting bug reports, documentation, user feedback, etc. (Perhaps I will be some of these things at some point if I ever get around to getting myself a real system suitable for development, but at this time I am not.) I have, however, been following the project and this list for a fair while as an outside bystander and observer. The following should be taken with that in mind; obviously, anything I say can and should be superseded by an opinion by someone with more activity and status within this project. For anyone reading this message who's not already aware of it, Lua (the site https://www.lua.org/ is the canonical home for it) is a small, relatively new (compared to others anyway) programming language originally coming from academia in Brazil but now in use at least somewhat in commercial and production environments internationally, especially in certain niches. It was intentionally designed to be embedded within larger applications to serve as a scripting or programming language for those applications, as I understand TCL was as well. It is under active development, and has significant existing documentation (including in dead trees form) and a user community. As such, it doesn't appear to me at least at first glance (again, solely based from the viewpoint of an external bystander / observer of the Fossil project) to be a completely unreasonable notion to integrate Lua as an alternative scripting / programming language for Fossil. That being said, Mr. Cardenas, I have to warn you that in my opinion there's going to be ... I won't say an active objection by the core Fossil development team to doing this, but at the least a significant amount of inertia and friction (*not* necessarily maliciously or consciously intended!) to the idea. I'd say much of this will have to do with the history surrounding the creation and development of Fossil and of its sister and Siamese twin project, the embedded database SQLite. My apologies if the following brief synopsis and analysis of said history is incorrect or misleading; I eagerly welcome corrections or augmentations of what I've written. As I understand it, SQLite was originally created by Dr. D. Richard Hipp as an add-on for the TCL programming language, and only later subsequently became an independent entity in its own right able to be used without reference to TCL. Fossil then was developed by Dr. Hipp both as a tool for use in the development of SQLite (presumably the version control systems commonly available at the time Fossil was created were deemed by him to be unsuitable) and also as an example of the use of SQLite by a third-party active project, insofar as Fossil project repositories are created as SQLite databases, etc. Due to the historical origins of SQLite springing out of TCL, and Fossil springing out of SQLite (and thus at second-hand out of TCL), both of these projects have naturally considered TCL to be a, or even the only, first-class embedded scripting and programming language for their project. (As I recall there's also TC1, but as I understand it that's derived in its design from TCL, and it's not nearly as capable as TCL.) The architects and developers of SQLite and Fossil were originally extremely experienced with, and had comfort using, TCL and already were (and are) presumably expert in its use, and can use it to achieve anything they feel necessary to achieve. As such, presumably they have seen no great need to support alternative embedded scripting or programming languages for SQLite or Fossil to date. Given this, I'd say you have (at least) two major problems to solve in your goal of being able to use Lua with or within Fossil: (1) Assumptions within the Fossil code base that there is only the TCL (and perhaps TC1) embedded programming language available and needing to be accommodated; objects and concepts which are "natural" for TCL but might not be natural for Lua; etc. (2) Unwillingness by the core Fossil development team to include whatever integration for Lua you might achieve into the primary sources and distribution for Fossil which *they* will have to subsequently maintain and support, especially if they do not see any need for Fossil to support Lua but rather feel that Fossil's support of TCL provides everything needed by any likely user of Fossil, and also especially if they do not feel capable of providing adequate maintenance and support for this capability, the use of Lua within Fossil by people, etc. The first point is, obviously, primarily if not solely a technical issue; the second has both technical and social aspects. For the first point, obviously I have no idea what if any problems you might encounter, as I am entirely unfamiliar at this time with the Fossil code base. I'd say that the first obvious step is to examine the Fossil sources, to see what is enabled or disabled when TCL support is activated or disabled at compile time. (You'll want to see what Makefile switches are used to enable or disable this support; they should be well documented in the documentation for Fossil.) See what if anything should be abstracted out of the enabling of support for embedded TCL to allow for enabling of support for embedded Lua. (This would presumably also allow later for others to more easily embed other suitable scripting / programming languages if they find the need to do so.) I'd say that you should watch for problems caused both by the enabling of Lua support and the disabling of TCL support, and also problems caused by the enabling of support for both embedded languages simultaneously; I see no obvious reason why there might not be people wanting to have both Lua *and* TCL capability at the same time, especially given the historical background of Fossil, so you should make sure that can be done. And, this will aid later with the problems of point (2) mentioned above -- people familiar with TCL do not have to give it up if support for Lua is also enabled. Make sure to create a suitable test suite for Fossil with embedded Lua (and simultaneous embedded TCL and Lua), to demonstrate that the basic functionality is working properly. And you'll want to make sure that any changes you make do not break any existing Fossil test suite functionality. There are people on this list specifically interested in that point, and again it will aid later with the problems of point (2) above. And of course you'll need to create adequate documentation for your changes, both how to enable embedded Lua support and also on how to actually *use* this support once it is enabled! I wouldn't say you need to write much if any documentation on how to use Lua itself (tho providing pointers to such documentation maintained elsewhere, such as by the Lua project itself, is probably desirable), or at least no more than might already exist within the Fossil documentation concerning use of TCL. But, you *will* want to document how Lua concepts, data objects, etc are exposed and available within the embedded Lua support in Fossil, and how Fossil concepts, data objects, etc are made available to scripts using the embedded Lua support. Oh, and as an aside, you should make sure that any support for embedded Lua that you develop is capable of using *both* Lua support compiled statically within the Fossil binary (which will be important especially for the Microsoft Windows platform, and possibly for Apple MacOS/X) and provided by Lua sources incorporated within the source code for Fossil, *and* *also* provided by a separate system library and compilation header files for Lua provided by the operating system and dynamically linked in with the Fossil binary, and not making use of any sources included with the source code for Fossil itself. This latter point will be important to most if not all Linux and Unix distributions which provide Fossil; they typically as a matter of policy strip out any embedded libraries in the source code for any application they provide and require said distribution to use the distribution-provided version of the library, its compilation headers, etc. (This isn't necessarily the approach that the developers of Fossil favor, mind you, but it's not something they can control for any Fossil binary provided by a O/S distribution as opposed to by they themselves.) In particular, you should not assume that the version of Lua *you* provide, especially if it's a "bleeding edge" version, will be the version of Lua available to a Fossil with Lua support enabled which is provided by a O/S distribution; you'll need to incorporate checks that whatever version of Lua is provided contains the minimum capabilities you need for your embedded support. To this end, you'll want to look at whatever the primary sources for Fossil do with respect to using SQLite, making sure it is available and has the minimum functionality needed, etc, and also with whatever the Fossil sources for representative members of the Unix / Linux ecosystem do. I'd suggest looking at the sources provided by Debian (and or Ubuntu), Fedora, perhaps also OpenSUSE or Arch or Gentoo, and also those provided by one or two BSD derivatives, perhaps FreeBSD and or OpenBSD. (IIRC, the person who maintains Fossil and I guess also SQLite for OpenBSD is active on this list. I think some of the maintainers of Fossil for various Linux distributions are also active here.) Note that if you *don't* do this yourself, you'll just make the maintainers of Fossil for the various Unix / Linux distributions have to work harder to do this themselves (and perhaps they'll do it in a way you disagree with), because they *will* do it and their having to do it themselves may impede their ability to provide versions of Fossil which include embedded Lua support, which will adversely affect you with respect to point (2) above. (Since the core Fossil developers want their new versions of Fossil adopted by O/S distributions in a timely fashion, and anything which impedes that will be viewed in a negative light by them, I believe.) So, you might as well do it this way to begin with, to make sure it gets done right. This isn't meant to be an exhaustive list of things to look out for, of course. It's just my first quick pass over the notion. For the second point, you have to remember that while *you* see the desirability if not need for the capability to embed and use Lua within Fossil (and, this is NOT in any way insignificant!), it's not yet obvious to others, particularly to others who are not experienced users of Lua or familiar with its capabilities, that this is a necessary or even desirable capability. In particular, if you desire for your work to be integrated into the primary sources of Fossil where they will be potentially available to anyone who uses such a version of Fossil, as opposed to merely being a fork of Fossil which you yourself will have to maintain and keep up to date with respect to the primary version of Fossil and which will only be available to people who are aware of and seek out your fork, then you'll need to convince the core developers of Fossil that not only is it at the least desirable (if not necessary) for the primary version of Fossil they develop, maintain, and support to have the capability of using Lua as an embedded programming language with status equal to that of TCL (which they are already intimately familiar with), but that this can be done at a cost in terms of increased cost of maintenance and support which they can afford and are willing to pay, and that the benefits that end users gain by their doing this are worth the cost to them of doing it. (After all, if they don't see why it's worth the cost to do it, why should they?) Note that I *assume* you will ultimately desire and have as a goal for your work to be integrated into the primary sources of Fossil if you can achieve it. If that is unimportant to you, then obviously you can ignore all the following. To this end, you'll need to keep in mind that the core developers of Fossil are *not* necessarily expert in, or even familiar with, Lua. As such, I think it likely they will initially see any request to support embedded Lua in Fossil as something with a potential high or even unbounded cost upon them in terms of requirements for future maintenance and support of this capability, and with minimal or even no benefit to users of Fossil. (After all, they *themselves* use Fossil and don't use embedded Lua with it!) Therefore, you'll need to demonstrate the benefit of having embedded Lua available, not just in theory but with one or more concrete demonstrations of it. You'll need to also demonstrate that any additional cost upon them to maintain and support this capability will be no greater than they are willing to pay, and preferably as minimal as possible. Ideally, you'll need to show that there are others, preferably a significant number of others, besides yourself who would benefit from this capability, again ideally by demonstrating that they are already actively *using* the capability. (E.g. that it's worth the Fossil developers paying the cost of maintenance for this capability because it won't be going to waste.) You may also need to revise your approach to embedding Lua within Fossil in response to feedback from the Fossil developers as to how they see your work. Remember that while you're paying the upfront cost of development of this capability, you're asking them to pay the cost of maintaining and supporting it into the indefinite future. As such, if there's two or more approaches to doing something where they think the cost to them of the approach you choose will be greater long term than that of a different approach, even if the different approach costs more up front in development (e.g. to you), they may well ask you to take the different approach. You may need to negotiate with them concerning this. In this aspect, especially if you follow Linux Kernel News, it's much as if you were a Linux kernel developer who's come up with some wonderful new feature which you want accepted into the main kernel sources (e.g. the version controlled by Linus Torvalds). But, when you submit your proposed feature on the appropriate development mailing list (be it the main Linux Kernel Mailing List or a subsystem- or topic-specific list), you get responses back that you should have considered approach B instead of A, or why didn't you consider approach C, or did you consider doing something more generally, or or or or. Often there will be several or *many* rounds of feedback, where changes to the feature will be made in response to prior feedback, and then there'll be feedback to those changes which result in still more changes. The goal here isn't to subject a developer with a new feature to having to run a gauntlet or gratuitously jump through hoops just for the fun of it, but to make sure the new feature is the best that it can be and to make as sure as possible that once the feature is incorporated into a released kernel that nothing will come back to bite people in terms of too expensive maintenance cost, technical debt which has to be paid later, etc. On that same vein, if and when you are ready to propose having this capability included into the primary sources for Fossil, set things up much as a new capability or feature for the Linux kernel would ideally be presented. Rather than have one giant patch that does everything in one fell swoop, have several patches that each individually accomplish one sub-goal or way point in terms of enabling the capability, and which build on one another incrementally. It'll be a lot easier for the Fossil developers to follow what you're doing step by step and be convinced that what's being done is reasonable and makes sense that way, I think. Further and in particular, I'd say you need to consider how to handle having Fossil having an external optional dependency (e.g. on Lua) on a project which the core Fossil developers do not themselves control or are already intimately involved with. (SQLite doesn't count for this as it's maintained by the same people who maintain Fossil and the two are kept well in sync. TCL is an external dependency, but presumably the core Fossil developers are already intimately involved with the development of TCL due to the history of the projects involved. Lua will be in neither category.) If Lua has a "stable" version, or better yet a "long term support" version, available to choose from in terms of what you choose to select to embed within Fossil, I would advise choosing that version if at all possible. It'll likely reduce the cost of maintenance both for the primary Fossil developers and maintainers, and also for the third party maintainers of Fossil as provided by the various Unix / Linux distributions. Oh, and remember the obvious things like following whatever coding style guidelines exist for Fossil, etc, to make it as easy to merge if and when that comes to pass as possible. Also, there's some legal hoops you'll have to jump through before the core Fossil developers will agree to accept your changes into the primary Fossil sources. You can find out more about all of this on the primary web site for Fossil (it should also be already included in your own personal copy of Fossil) in the information about how to contribute to Fossil. In short, even if and when you get something *you* find to be suitable in terms of providing a capability for embedded Lua within Fossil, it may be a good while longer before others agree with you and until then you may have to maintain and support your changes to Fossil by yourself. And you may have to change how you do things one or more times before the core Fossil developers agree to incorporate your feature into the primary sources for Fossil. Keep in mind that "slow and steady, and persistence, wins the race". I realize this is a very long response to your question, and that it may come across as at least a little negative towards what you want to do. Please don't think I am trying to dissuade you from attempting to do this thing; I am not, in the least. But, I *am* trying to give my opinion, as best as is possible, as to what I think you will need to do to achieve what I (perhaps incorrectly) believe your ultimate goals to be, so that you will have as accurate a picture as you can get of the effort you'll need to make, and so you can decide if you really want to make the effort or not. I'd hate for you to put a lot of work into this only to become discouraged because it wasn't going as easily as you believed it would go and end up giving up and wasting all the time you'd spent to that point. Other than that, I hope this is of some use and interest to you. And I thank you for your time in reading this message. Also, my apologies in advance if I've come across as, I don't know, condescending or dismissive in any way. That was not my intent or desire in any way whatsoever. Unfortunately, I have Male Answer Syndrome, and sometimes it expresses itself in an undesirable fashion. Anyway. I wish you the very best of luck and success with this effort, if you do decide to go ahead with it. I'm sure you'll find it interesting if you do decide to pursue it. Be well. Joseph
_______________________________________________ fossil-users mailing list [email protected] http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
