Do you have a customer/user? I have always found it extraordinarily difficult to write software for other people to use unless I can find some. Turns out I'm not a typical user. :-)
Regards David M Bennett FACS -----Original Message----- From: Darren Duncan [mailto:dar...@darrenduncan.net] Sent: Sunday, 25 January 2015 1:35 PM To: muldis-db-users@mm.darrenduncan.net; t...@thethirdmanifesto.com Subject: [TTM] status in 2015 of Muldis database projects On 2015 Jan 1 I had announced here that I had updated my Perl module Set::Relation to reduce is dependency load without cutting functionality. Now on 2015 Jan 24 I will briefly update you on the status of the more central project, the Muldis D language itself. For the last few years I had been doing a lot of brainstorming and note-taking and made a lot of improvements to the design of the Muldis D language and its ecosystem, a lot of this being put unorganized in a TODO_DRAFT file on Github and a lot not making it online myself; that was more to help me remember. Starting a couple years ago I switched to an implementation-driven process which saw a few periods of movement and also long periods of none, as I was quite busy with some other concerns. So in 2015 ... For various reasons, I am now making an explicit push to actually get things implemented, where I try and spend an hour each day on something tangible. I feel that with continued focus, people should be able to execute Muldis D code within maybe 2 months of now, a milestone, and things can then snowball. ====== The first "for real" implementation is https://github.com/muldis/Muldis-D-Ref-Perl5 or the "Muldis D Reference Engine", in terms of the public-facing Perl 5 module Muldis::D::RefEng (not yet released on CPAN). This has a few main components: 1. Low_Level, which is a Perl library corresponding directly to the Muldis_D::Low_Level system-defined package. That package defines the standard foundation of types and routines which is easily implementable in or mappable to typical general purpose languages, Perl or otherwise. The rest of the higher level Muldis_D system-defined package, say 90% of the built-ins that user code can invoke, would be written in Muldis D itself and can be reused, while just the Low_Level part is expected to be rewritten from Perl for each implementation langauge. This is the primary extent of the bootstrapping. The 9 mutually disjoint low-level types are: Boolean, Integer, Array, String, Bag, (named) Tuple, Capsule, Identifier, External. Almost all other types one is likely to see in Muldis D would be subtypes of Capsule, which corresponds to the concept of generic objects each belonging to a named class, making it easy to partition types from each other in the face of polymorphism. In ! contrast, Tuple is analogous to a classless object; or put another way, a Capsule is a tagged Tuple. The most complicated type implementation-wise by far is Bag, which is the homogenous collection type over which Set and Relation are defined; the complexity relates to all the infrastructure related to element indexing and identity/sameness determination involved. Low_Level also does some functionality related to memory management. A Perl ::Value object is used to represent each Muldis D value regardless of type, and is value-immutable. 2. Plain_Text implements the Muldis D Plain Text (MDPT) parser, which takes Muldis D Plain Text source code, typically in .mdpt text files (as per .pl or .java or .sql etc files), and maps it into Muldis D's homoiconic native representation. The latter is analogous to the "information schema" of SQL but that all routine and type definitions are completely broken down, analogous to an abstract syntax tree that also remembers most concrete syntax details as metadata, for lossless round-tripping of all the details users would consider important, including in translation to other languages. The internal representation is composed of a tree of types/::Value provided by Low_Level, typically Capsule subtypes corresponding to different types of expression/etc nodes. (Due to how Muldis D works, we don't actually have to declare a higher level type to use it, as higher level types are just constraints on or unions of the Low_Level ones, so we can use all the AST types without them ! having to be "declared" first, so Low_Level is kept simple.) This parser is procedural and stream-oriented, so it works with very little RAM no matter how big the source file is, particularly important for those that are database dumps. 3. A compiler component will take the homoiconic native Muldis D code and generate equivalent Perl 5 code from it. Generally there is a 1:1 correspondence between each Muldis D routine and a generated Perl 5 routine, as well as between a Muldis D package and a Perl 5 package. The generated Perl 5 code will include calls to any other glue code provided by RefEng so that it has the desired runtime semantics such as the dispatch mechanism. It is expected in practice that the "compiled" (Perl 5 source) modules will then be cached, at least in memory but often also in the filesystem, as Perl 5 packages, maybe under the Muldis::D::RefEng::Compiled::* namespace, so that as long as further data-definition activities aren't happening, runtime performance will be more reasonable due to avoiding unnecessary repeated work. 4. A runtime and/or memory component will implement a virtual machine supporting multiple concurrent processes and transactional memory. (The multi-process model is virtual, and all exists within a single actual Perl process.) Each process would correspond to either a "database connection" or an "autonomous transaction" or other things in SQL DBMSs or other programming languages. This would provide the main event loop(s) in which all the regular compiling and runtime activities happen. Shared access between processes to possibly-common resources such as "databases" is managed here, processes can start and end others, and they can communicate using message passing. 5. Fundamentally Muldis::D::RefEng is an embeddable library, but a binary "muldisdre" would be included that wraps it. One can treat that binary in exactly the same manner as one treats "perl" or "python" or whatever to run Muldis D Plain Text source files, they don't even have to know it is written in Perl. This is part of a model of portability between Muldis::D::RefEng and say ports written in NQP or Python or Ruby or C or Go or Haskell or whatever, users can just run it without knowing what its written in. Another precedent for this is Perl 6 and its user-interchangeable Rakudo backends, MoarVM, JVM, and Parrot. 6. A wrapper/SDK for Muldis::D::RefEng, or more likely that would be the Perl 5 package name of the wrapper, would be invokable from a general Perl 5 program in the same way that DBI et al are. It would implement API-level input checking from user code as well as provide Perl users a second way to write Muldis D code, which is as nested Perl data structures analogous to what various Perl ORMs or abstractions take, so in that context Perl programs don't have to generate string form (Plain Text) Muldis D at all. It would be particularly suited for a backend of DBIx::Class or the like, where they can just map Perl data structures and no messy details of string serialization is involved like with DBI+SQL today. For status, the above RefEng 6 components are all loosely designed, and: 1. Component 1 / Low_Level is about 40-80% coded depending how you measure it, and the working proof-of-concept Set::Relation Perl module will have a lot of its guts adapted to flesh this out more. 2. Component 2 / PlainText is about 10% coded, currently under file names like StreamDecoder/StreamLexer/more. 3. Component 3 / the compiler is pending under RefEng but some precursors were released incomplete on CPAN years ago in the form of modules like "Rosetta" et al, those having generated a mixture of Perl and SQL. Likewise for component 6 / the SDK for embedding in Perl. Components 4 and 5 don't have enough written yet to commit. ====== Besides Muldis::D::RefEng, also on GitHub is https://github.com/muldis/Muldis-D-Standard which is the repository for the self-hosted portion of Muldis D, intended to house the 90% of the system library code written in Muldis D itself, as well as the standard test suite for implementations, and example code, and so on. This would be commonly shared by Muldis::D::RefEng and any other implementations. Currently this is maybe 1-5% written depending on how you measure it. A precedent being that a majority of the Perl 6 standard library is written in Perl 6, that having its own shared repository. Note that while Muldis::D::RefEng releases outside GitHub conceptually bundle Muldis-D-Standard, what it will actually ship to CPAN with is pre-compiled-as-Perl versions of the standard library, so Perl users can just install that Perl module and go. A loose analogy is that the DBD::SQLite Perl module ships to CPAN with the sqlite.h and sqlite.c but the latter don't canonically live in the former's version control. ====== Finally, I have started rewriting/revising the Muldis D language specification, at https://github.com/muldis/Muldis-D and started making periodic CPAN releases (where it is formatted) at http://search.cpan.org/dist/Muldis-D/ for the first time since 2011. This rewrite both considerably simplifies the spec and incorporates my last few years of brainstorming. In particular this is also partially implementation-driven and kept in sync with RefEng developments. I jumped the spec version ahead to 0.200, from 0.148. The root file Muldis::D was brought up to date and most of the other files were renamed into Muldis::D::Outdated::* to more clearly indicate their status; over time replacements for the latter will appear under typically new Muldis::D::* names. One of the more significant changes is that the new version will have concrete Muldis D Plain Text code examples, that is actual source code examples, throughout it just like typical programming language documentation. Learning by example is generally what users want and is more effective. Previously almost all the files had no code examples and just described things. At the time a reason for this was partly that I was trying to de-emphasize the plain text syntax as just 1 of a variety of forms, unlike now where it is the canonical form, and another reason is that originally I had conceived the features long before actually figuring out the syntax, unlike now. ====== So my current plan is to co-develop Muldis::D::RefEng and the spec up until the point that a reasonable starter subset of the language works, and then release RefEng to CPAN for the first time. It is expected that the first release will be just enough that you can invoke "muldisdre" and compile/run a hello world program as well as do some basic math or string operations, as well a relational operation (probably relational join), basically something that differentiates it from all the other general purpose languages. The goalpost may be moved for the first release, but it should have a substantially complete RefEng infrastructure in place, the ability to parse and run at least the simplest Muldis D programs. This will maybe come out by the end of March if I'm lucky. After that, subsequent releases would be mainly about implementing more system types and routines, adding examples and tests. This includes a lot of Low_Level routines needed for a normal language but not for a "get something running" release. Then other projects in the ecosystem such as ports to other languages or other / eg SQL backends. Until then or afterwards, you can follow some of my progress on Github. Thank you again for your interest and I'm sorry for taking so long to get to an executable. -- Darren Duncan _______________________________________________ muldis-db-users mailing list muldis-db-users@mm.darrenduncan.net http://mm.darrenduncan.net/mailman/listinfo/muldis-db-users
